Lab · Module 15 · BAA 15.0

Monitoring &
Orchestration

Building an agent is day one. Running it reliably at scale for 90 days is the real job. This lab shifts from building to maintaining — you'll set up performance dashboards, read and annotate reasoning logs, diagnose and fix a failed response, and evaluate whether your system can actually scale.

📊

Read a Live Dashboard

Identify which metric has breached its alert threshold and understand why it matters.

🔍

Analyze Agent Logs

Read a real reasoning trace, find the failure entry, and annotate it with the root cause.

🔬

Diagnose the Failure

Classify the failure type and trace it to its root cause using the reasoning trace as evidence.

✏️

Write the Prompt Fix

Review a before/after diff and select the correct configuration change to prevent recurrence.

📈

Evaluate Scalability

Match 4 real scaling scenarios to the correct architectural mitigation strategy.

Lab Roadmap

—

Brief — ContentPilot goes to production

What changes when an agent serves 12,000 users per month

1

Live Dashboard — Identify the anomaly

Read 6 live metrics and click the one that has breached its alert threshold

2

Log Analysis — Find the failed response

Review 8 reasoning trace entries — one contains a failure. Identify and annotate it.

3

Diagnose the root cause

Classify the failure type and identify which prompt rule was violated or missing

4

Write the prompt fix

Choose the correct configuration change from the diff — and explain why it prevents recurrence

5

Scalability evaluation — 4 scenarios

Match each scaling failure mode to the correct architectural solution

—

Knowledge Check · 7 questions · 70% to pass

The Production Reality

ContentPilot Is Live

ContentPilot v1.1 shipped 6 weeks ago. It's now running at 12,400 requests/month across 847 active users. The honeymoon phase is over. Three things are now true that weren't on launch day — and they change everything about how you work.

🔴

Failures Are Silent

When your agent produces a bad response, most users won't tell you — they'll just leave. Without logging, you only learn about failures from churn data, which is weeks too late.

⚡

Scale Amplifies Bugs

A prompt rule that works 97% of the time produces 372 broken outputs per month at 12,400 requests. A 1-in-1,000 edge case becomes a daily occurrence.

📊

Drift Is Invisible

Model providers silently update base models. User input patterns evolve. Your agent's quality drifts without any code change on your end. Only dashboards catch it.

The Monitoring Stack You Need

What You Measure

📊 Output quality metrics — specificity score, format adherence, tone alignment

⏱ Latency percentiles — P50, P95, P99. P99 is where users rage-quit.

💰 Token cost per request — signals prompt bloat and output verbosity drift

🚨 Error rate by mode — failures cluster in specific modes, not evenly

🔄 Memory hit rate — are users' past contexts being retrieved correctly?

What You Log Per Request

          run_id: "cp_20250312_084821_a3f"

          mode: "post_ideas"

          topic: "[input hash, not raw text]"

          latency_ms: 2847

          input_tokens: 412

          output_tokens: 289

          quality_flags: ["specificity_low"]

          pipeline_stages: [...]

💡 The audit trail principle: Every agent action that affects a real user or moves real data should produce a log entry you can trace back to. Not just the final output — the reasoning path. If your agent makes a decision and you can't explain why from the logs alone, the system is not auditable.

Section 01 · Performance Dashboard

ContentPilot Live Metrics

You arrive at work and open the dashboard. Six metrics are live. One has breached its alert threshold and needs immediate attention. Click the metric that is in a critical state to investigate — then answer the diagnostic question.

ContentPilot · Production · Last updated 12:00:00

1H

6H

24H

7D

Alert Feed

Section 02 · Log Analysis

Reasoning Trace Review

Below is ContentPilot's reasoning log for the last hour — 8 entries. One of these requests failed silently: the user received a response, but the agent violated a core system prompt rule. Click through each entry to read the full trace. Find the failure and mark it.

⚠️ Silent failures are the hardest to catch — the agent returned HTTP 200 and the user got something. The failure is in the reasoning trace: a rule was skipped, a wrong path was taken, or a constraint was not honored. Expand each log entry to read the full trace.

contentpilot · reasoning-trace.log · 2026-03-12

ALL

ERROR

WARN

INFO

Section 03 · Root Cause Analysis

Diagnose the Failure

You've found the failed log entry. Now apply structured root cause analysis — the same method used in production incident reviews. Answer both questions to build a complete RCA.

The Failed Entry (Log #6)

      run_id: "cp_20260312_113042_e7b"

      mode: "summary"

      user_input: "Can you summarize the AI safety report?"

      input_text_provided: false

      stage[3]: LLM_CALL initiated

      stage[3] reasoning: "User requested summary. Proceeding with general knowledge about AI safety."

      output: "AI safety focuses on ensuring that artificial intelligence systems behave as intended…"

      quality_flags: ["hallucination_risk", "rule_violation_no_text_check"]

      rule_checked: "ASK_FOR_TEXT_IF_MISSING": NOT_EVALUATED

QUESTION 1 OF 2 · Failure Classification

Which failure category best describes what happened in this trace?

Section 04 · Prompt Refinement

Write the Configuration Fix

You've diagnosed the failure. Now prescribe the fix. Below are three proposed changes to ContentPilot's pipeline and system prompt. Only one correctly prevents the failure from recurring without breaking the working modes. Select it, then review the diff.

🔮 Good fixes are surgical. The best prompt fix is the smallest change that closes the gap — not a full rewrite. Over-engineering a fix for one failure mode often introduces new failure modes. Read each option carefully.

Section 05 · Scalability Evaluation

Match Failures to Mitigations

As ContentPilot grows to 100,000 requests/month, four new failure modes emerge — each requiring a different architectural response. Select a mitigation strategy from the pool, then click a scenario to apply it. Match all 4 correctly.

💡 Strategy: Click a strategy chip to select it (it turns red), then click the scenario you want to apply it to. If wrong, the chip returns to the pool. Match all 4 to proceed.

Strategy Pool — Select a strategy, then click a scenario

Knowledge Check

7 Questions

Covering dashboards, log analysis, root cause diagnosis, prompt refinement, and scalability. 70% to pass.

Question 1 of 7Score: 0 / 0

Lab Debrief