Multi-Agent Systems for Business

A practical next step for teams that already understand single AI agents. Learn when several agents genuinely beat one, the core orchestration patterns, how to design roles, handoffs and shared state, and how to keep the system safe, observable, affordable and compliant. A worked quote-to-order example runs through every lesson and ends in a 90-day pilot plan.

Course structure

• Module 1: When One Agent Is Not Enough (4 lessons). A short recap of single agents, the four good reasons to use several, the real costs of adding them, and the quote-to-order example used throughout the course.
• Module 2: Orchestration Patterns (5 lessons). The core ways to coordinate agents: supervisor-worker, pipelines, routers, parallel fan-out, generator-evaluator loops and hierarchical teams, and how to choose between them.
• Module 3: Designing the Agent Team (5 lessons). Roles and a RACI for agents, structured message contracts, shared state and context, tool and agent protocols, and human checkpoints.
• Module 4: Reliability, Safety and Governance (5 lessons). How multi-agent systems fail, the safeguards that contain them, tracing, evaluation at agent and system level, and security, audit and PIPEDA basics.
• Module 5: Run, Choose and Pilot (3 lessons). Managing cost and latency, deciding whether to build or use a framework, and a 90-day plan for a first multi-agent pilot.

Interactive labs

• Which Design Fits?: Sort twelve business scenarios by the design they need, then check the explanations. It builds the judgement to avoid both under- and over-engineering.
• Prioritize Multi-Agent Use Cases: Rate each candidate use case on coordination complexity and risk against business value, then see where it lands. Rename or add your own cases to build a shortlist.
• Multi-Agent Token Cost Estimator: Estimate the monthly model cost of a multi-agent workflow. Enter the combined calls of all agents per task, so you see how each agent's calls multiply the total.

How it works

• Work through 22 short lessons at your own pace. Each has key takeaways, an exercise for your own workplace and a quick knowledge check.
• Try the three interactive labs as you reach them in the course.
• Pass the 20-question final exam (70% or higher, unlimited retakes) to unlock your Chmltech certificate of completion.
• Free, with no sign-up. Your progress is saved in this browser.

This course assumes you already know what a single AI agent is. Here is a one-lesson recap so we share the same vocabulary, then we move on to systems where several agents work together.

The single agent in one paragraph

An AI agent is a language model given a goal, instructions, a set of tools it may call and some memory of the case so far. It runs a loop: understand the situation, plan the next step, act through a tool, observe the result, and repeat until the goal is met, a limit is reached or it hands off to a person. The model decides the path at run time, within the limits you set.

• Goal and instructions: what the agent is for and how it should behave.
• Tools: the actions it may take, such as looking up an order or drafting an email.
• Memory and context: what it can see while it works, limited by the model's context window.
• Guardrails: permissions, approvals and stop conditions.

Where one agent starts to struggle

Single agents work well for focused, multi-step cases. They begin to strain when one agent is asked to do too much:

A quick example

Imagine a 45-person industrial distributor in the Fraser Valley that sells valves, fittings and hose assemblies. Its first agent reads customer emails and drafts quotes. Over time, the team asks it to also check export rules, apply customer-specific pricing, verify stock and review its own work. The prompt grows, the tool list grows, and reviewers notice it occasionally skips the compliance check on busy days. That is the moment to ask whether the work should be split.

Splitting is not automatically the answer. The rest of this module helps you decide when several agents genuinely beat one, and when they only add cost and moving parts.

A multi-agent system is two or more agents, each with its own instructions, tools and limits, coordinated to complete a larger task. Done well, it is like moving from one overworked generalist to a small, well-organized team. There are four good reasons to make that move.

1. Specialization

Each agent gets a narrow job, a short set of instructions and only the tools it needs. A pricing agent knows price lists and discount rules; it does not need to know how to read a drawing. Narrow agents are easier to write, test and improve, because you can change one without disturbing the others.

2. Parallelism

Independent subtasks can run at the same time. Checking stock at three warehouses, or reviewing a contract for payment terms, insurance and liability clauses, can be split across workers and merged. Total elapsed time falls, even though total work does not.

3. Separation of duties

The same principle that stops one clerk from both creating and approving a payment applies to agents. A separate reviewer agent, with different instructions and no ability to change the draft itself, is more likely to catch errors than the author checking its own work.

4. Context limits

Every model has a finite context window. Splitting work means each agent sees only what it needs. The intake agent reads the full messy email thread; the pricing agent receives a clean, structured summary. Less noise usually means better decisions.

A test for justification

A multi-agent design is justified when at least one of these reasons is strong and you can name the specific benefit in business terms: fewer quote errors, faster turnaround on large requests, or an audit trail showing an independent check. If you cannot name the benefit, you are probably adding agents because the idea is appealing, not because the work needs it.

Notice that none of these reasons is "agents are smarter in groups". The gains come from structure: clear roles, clean inputs and independent checks, the same things that make human teams work.

Every agent you add brings real costs. Before committing to a multi-agent design, weigh them honestly. Many processes that look like they need a team of agents are better served by one well-scoped agent, a simple workflow, or no AI at all.

The costs of adding agents

• Model cost: each agent makes its own model calls, and each call carries its own instructions and context. Costs add up per agent, per step.
• Latency: sequential handoffs add waiting time. A five-agent chain is slower than one agent unless the work runs in parallel.
• More failure points: every handoff is a place where information can be lost, misread or corrupted.
• Coordination errors: agents can disagree, repeat each other's work or loop.
• Harder debugging: when the answer is wrong, you must find which agent, and which handoff, caused it.
• More to govern: more instructions to version, more permissions to scope, more logs to review.

Three better alternatives to check first

A worked comparison

Consider a clinic's intake desk that wants to sort incoming referral faxes. One proposal uses four agents: a reader, a classifier, a urgency scorer and a reviewer. A second uses one agent that extracts fields and suggests a category, with a fixed rule that anything marked urgent goes to a nurse immediately.

The second design has fewer handoffs, lower cost and one place to look when something goes wrong. The urgency decision, the only truly high-stakes step, stays with a person in both designs. The four-agent version adds complexity without adding a control that matters. That is the pattern to watch for: extra agents that do not add specialization, parallelism, separation of duties or context relief.

A simple rule of thumb

Start with the simplest design that could work. Add an agent only when you can point to a failure or limit in the simpler design that the new agent fixes. Then measure whether it did.

Throughout this course we will use one worked example so you can see each idea applied to the same business. It is illustrative and generic, but it reflects a very common SME workflow.

The business

Imagine a 45-person industrial distributor selling valves, fittings and hose assemblies to fabrication shops, municipal utilities and oil and gas contractors in BC and Alberta. Customers send requests for quotation by email, often with attachments, part lists and loose wording such as "same as last time but in stainless". Two inside sales reps turn these into quotes and, once accepted, into orders in the ERP system. On busy weeks, quotes wait days and errors slip through.

The proposed agent team

The people in the loop

• An inside sales rep approves every quote before it goes to the customer during the pilot.
• The sales manager approves any discount above the rep's authority.
• Only after customer acceptance, and a person's confirmation, is an order written to the ERP.

Why this example justifies several agents

It passes the tests from the previous lessons. Intake, pricing and compliance follow different rulebooks (specialization). Pricing and compliance can run at the same time once intake is done (parallelism). The reviewer did not write the quote (separation of duties). And the pricing agent receives a structured summary rather than a long email thread (context limits).

It is also a case where agents could go wrong in costly ways: a wrong price, a missed certification, an order placed twice. That makes it a good teaching example for the safeguards, tracing and evaluation lessons later in the course.

As you go, keep a parallel example from your own business in mind. Each lesson's exercise asks you to apply the idea to it.

The supervisor pattern is the most common starting point for multi-agent systems. One orchestrator agent receives the task, breaks it into subtasks, assigns each to a worker agent, collects the results and decides what happens next.

How it works

1. The orchestrator reads the request and decides which workers are needed.
2. It sends each worker a clear task with the inputs it needs.
3. Workers do their job and return structured results.
4. The orchestrator checks the results, asks for rework if needed, and assembles the output.
5. It decides whether the case is complete, needs a person, or must stop.

Strengths and weaknesses

Worked example: the quote orchestrator

A request arrives for 40 stainless ball valves and 12 hose assemblies, needed in three weeks. The orchestrator calls the intake agent, which returns a structured list of items and the customer account. The orchestrator then sends the item list to the pricing agent and the compliance-check agent. When both return, it passes the assembled quote to the reviewer. The reviewer flags that one hose assembly lacks a pressure rating in the request. The orchestrator does not guess: it marks the case "needs clarification" and routes a drafted question to the sales rep.

Design tips

• Keep the orchestrator's job about coordination, not doing the work itself.
• Give it an explicit list of workers, what each does, and what each returns.
• Set a maximum number of rounds before it must stop or escalate.
• Consider whether parts of the orchestration can be fixed code instead of model decisions. If the order of steps never changes, a simple workflow can do the coordinating more cheaply and predictably, with agents only for the judgement steps.

That last tip matters. Many effective business systems are a deterministic workflow that calls agents at specific points, rather than an agent deciding everything.

Two simple patterns cover a surprising share of business needs. A pipeline passes work through a fixed sequence of agents. A router looks at each incoming item and sends it to the right specialist.

Sequential pipeline

Each agent does one stage and hands its output to the next, like stations on a production line. The order is fixed in advance.

• Good for: work with clear stages, such as extract, then enrich, then format.
• Watch for: errors early in the line flow downstream, and total time is the sum of every stage.
• Control tip: validate the output of each stage before passing it on, just as you would inspect between stations on a shop floor.

Router (dispatcher)

A router classifies each incoming item and sends it to one of several specialist agents or workflows. It does not do the work itself.

• Good for: mixed inbound streams, such as a shared inbox with quotes, order changes, complaints and invoices.
• Watch for: misrouting. Always include an "unclear" route that goes to a person.
• Control tip: the router can often be a simple classifier or even rules; it does not always need a full agent.

Worked example: combining both

The distributor's shared sales inbox receives everything. A router reads each email and tags it as a new quote request, a change to an existing order, a delivery question or other. Quote requests go into the quote pipeline: intake, then pricing and compliance, then review. Delivery questions go to a single agent that looks up shipment status and drafts a reply. Anything tagged "other", or where the router's confidence is low, goes to a person's queue.

This combination is often all an SME needs. It is easy to explain, easy to trace and easy to test, because each item follows one visible path.

Choosing between them

If every item goes through the same stages, use a pipeline. If items differ in kind and need different handling, put a router in front. If items differ and the stages vary per item, you may need a supervisor, covered in the previous lesson.

In the fan-out pattern, one task is split into independent pieces that run at the same time, and a merge step combines the results. It is the pattern to reach for when elapsed time matters and the pieces do not depend on each other.

How it works

1. Split: divide the task into independent subtasks.
2. Fan out: send each subtask to a worker agent at the same time.
3. Wait: collect results, with a timeout for any worker that does not return.
4. Merge: combine the results using clear rules, and flag conflicts.

Worked example: checking a large quote

A municipal utility sends a request with 60 line items. Once intake has produced a structured list, the orchestrator fans out three tasks at the same time:

• The pricing agent prices every line from the contract price list.
• The compliance-check agent confirms material and pressure-rating requirements for each line.
• A stock agent checks availability at the Surrey and Calgary branches.

If each check takes about two minutes, running them in sequence takes about six; running them in parallel takes about two, plus the merge. The merge step builds one table per line item and flags any line where, for example, the cheapest stocked item fails the compliance check.

Merge rules matter most

When fan-out does not help

Parallelism reduces waiting, not work. Total model cost stays the same or rises, because every worker still makes its calls. It also does not help when subtasks depend on each other: you cannot price items before you know what the items are. And if one slow worker always finishes last, the whole fan-out waits for it.

A practical rule: fan out only where the subtasks are genuinely independent and someone actually cares about the time saved, such as a customer waiting for a quote.

In the generator-evaluator pattern, one agent produces a draft and another evaluates it against explicit criteria. If the draft falls short, the evaluator's feedback goes back to the generator for a revision. The loop repeats until the draft passes or a limit is reached.

Why it works

Checking is often easier than creating. An evaluator with a clear checklist and a fresh context can spot omissions the generator missed. It mirrors a well-known quality practice: the person who inspects is not the person who produced the part.

Making the evaluator useful

• Explicit criteria: give the evaluator a checklist, not "is this good?". For a quote: every requested item priced, quantities match, lead times stated, required certifications listed, discount within authority.
• Structured verdicts: pass or fail per criterion, with a short reason.
• Different instructions: the evaluator should not share the generator's prompt; otherwise it tends to share its blind spots.
• No editing rights: the evaluator returns findings; the generator revises.

Worked example

The risks

1. Endless loops: the generator and evaluator can disagree forever. Set a maximum number of rounds, commonly two or three, then escalate to a person with both views attached.
2. Rubber-stamping: an evaluator that passes everything adds cost without value. Test it with drafts that contain known errors and confirm it catches them.
3. False confidence: an AI reviewer is a filter, not a final authority. For high-stakes outputs, a person still approves.

Used well, this pattern raises quality before a person ever sees the draft, which means the human reviewer spends time on judgement rather than catching typos and missing lines.

In a hierarchical design, a top-level supervisor manages team leads, and each lead manages its own workers. It mirrors an organization chart: a general manager, department heads and staff.

When hierarchy helps

• The overall task spans several distinct domains, each with its own sub-team.
• One supervisor would need too many workers and too much context.
• Different sub-teams need different permissions or data access.

For example, a larger distributor might have a sales team (intake, pricing, compliance, reviewer) and a fulfilment team (allocation, shipping, documentation), each with a lead, under one case supervisor.

The cost of hierarchy

Every layer adds model calls, latency and places for messages to be misunderstood. Instructions passed down two levels can drift from the original intent, much as they do in human organizations. Most SMEs should not start here. Reach for hierarchy only after a flat design is working and clearly hitting its limits.

Choosing a pattern

Patterns combine

Real systems mix patterns. The quote-to-order example uses a router at the inbox, a supervisor for each quote, a fan-out for pricing and compliance, and a generator-evaluator loop at the end. That is fine, as long as each pattern earns its place and you can draw the whole design on one page.

The one-page test

If you cannot draw the design on one page, showing every agent, every handoff and every human checkpoint, it is too complicated for a first pilot. Simplify until you can. A design that fits on one page is also one that your team, your auditors and your future self can understand.

A RACI chart sets out who is Responsible, Accountable, Consulted and Informed for each activity. The same discipline works for agents, with one firm rule: only a person can be Accountable. Agents can be responsible for doing work; people answer for the outcome.

Write a role card for every agent

• Purpose: one sentence on what the agent is for.
• Inputs: what it receives and from whom.
• Outputs: what it returns, in what format.
• Tools and permissions: exactly what it may read or change.
• Out of scope: what it must never do, stated plainly.
• Escalation: when it must stop and pass the case to a person.
• Owner: the named person who maintains its instructions and reviews its performance.

Worked example: RACI for the quote process

Common role design mistakes

1. Overlap: two agents both "check pricing", so neither does it fully, or they contradict each other.
2. Gaps: nobody owns an activity, such as confirming the customer's ship-to address.
3. Vague boundaries: "help with the quote" is not a role. "Return a price for each line from the contract list, or flag it" is.
4. No human owner: an agent with no named maintainer slowly drifts out of date.

A good test is to read each activity row and confirm it has exactly one R and exactly one A, and that every A is a person. Then read each agent column and confirm the role card matches what the chart says it does.

Keep the RACI and role cards with the agents' instructions under version control. When a role changes, update the card, the chart and the permissions together, so the documents your team reads always match what the system actually does.

Most multi-agent failures happen at the handoffs. One agent writes a loose paragraph, the next one misreads it, and the error travels on. The fix is a message contract: an agreed, structured format for what each agent sends and receives.

Structured outputs over free text

Instead of "here is what I found", an agent returns data in a defined shape, usually JSON that follows a schema. A schema lists the fields, their types, which are required and which values are allowed. Many model providers can be instructed to return output that conforms to a schema, and your code should still validate it before passing it on.

Worked example: the intake-to-pricing contract

What a good contract includes

• Required versus optional fields, and what to do when information is missing.
• An explicit place for uncertainty, such as open questions and a confidence level, so agents do not fill gaps by guessing.
• Identifiers that let every step be traced to the same case.
• A version number, so you can change the contract without breaking older cases.

Validate at every boundary

Treat each handoff like a receiving inspection. Before the pricing agent starts, code checks the intake output against the schema. If a required field is missing or a quantity is negative, the case goes back or to a person. It does not go forward and hope. This one habit prevents many cascading errors.

Keep the message lean

Pass what the next agent needs, not everything the previous agent saw. The pricing agent needs items and the customer account, not the full email thread with signatures and disclaimers. Lean messages cost less and give the next agent less to misread.

Agents in a team need a common picture of the case. Without it, they repeat work, contradict each other or act on stale information. Designing that shared picture is one of the most important decisions in a multi-agent system.

Three kinds of memory

Use a shared case record, not a long chat

A common early design lets every agent read one growing conversation. It works for demos and fails in production: the transcript gets long, costs rise, and agents latch onto outdated statements. A better design keeps a structured case record, such as a database row or document, that the orchestrator updates. Each agent receives only the slice it needs and returns structured results that are written back.

Rules for shared state

• One writer per field. Only the pricing agent writes prices; only the compliance agent writes compliance results. This prevents silent overwrites.
• Record where each fact came from and when, so you can trace a wrong price back to its source.
• Keep history. Store changes rather than overwriting, so you can see how the case evolved.
• Retrieve, do not stuff. Pull relevant reference documents when needed instead of pasting whole manuals into every prompt.

Context management in practice

Each agent's context should contain its instructions, its slice of the case record and any retrieved references, nothing more. For long cases, summarize earlier steps into a short structured note. Check summaries carefully: a summary that drops a key detail, such as "customer requires mill test certificates", is a quiet source of errors.

Privacy and memory

Shared state often holds personal information, such as contact names and phone numbers. Decide how long case records are kept, who can read them, and what is excluded from long-term memory. We return to this in the governance lesson.

Agents are only as useful as the tools and data they can reach. As systems grow, connecting each agent to each system by hand becomes slow and fragile. Open protocols aim to make these connections more standard.

Tools in general terms

A tool is a defined action an agent may request, such as "look up price for part number" or "create draft order". Each tool has a name, a description the model reads, defined inputs and defined outputs. Your code, not the model, actually executes the tool and enforces permissions.

• Prefer narrow tools ("get contract price for account and part") over broad ones ("run any database query").
• Separate read tools from write tools, and give write tools stricter permissions and approvals.
• Write tool descriptions as carefully as instructions; the model chooses tools based on them.

The Model Context Protocol

The Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data sources. A system exposes its capabilities through an MCP server, for example "search catalogue" or "read specification sheet", and any compatible AI application can connect to it as a client. The benefit is reuse: you describe a connection once rather than rebuilding it for every agent or platform.

MCP does not make a tool safe by itself. You still decide which agent may use which server, with what credentials, and which actions need approval. Treat third-party MCP servers like any other software you install: check who maintains them and what they can access.

Agent-to-agent communication

MCP connects an agent to tools. A separate, emerging set of standards addresses how agents talk to other agents, including agents built by different vendors or running in different organizations. An example is the Agent2Agent (A2A) protocol. In general, these standards describe how an agent advertises what it can do, how tasks are sent and tracked, and how results are returned.

For most SME pilots, all agents live inside one system you control, and simple internal handoffs are enough. Agent-to-agent standards become relevant when you need to work with agents outside your own system. Because these standards are still evolving, check current documentation before relying on them.

Multi-agent systems can move fast, which makes it more important, not less, to decide exactly where people stay in control. A checkpoint is a point where the system pauses for a person to review, approve, correct or reject before work continues.

Where to put checkpoints

• Before irreversible actions: sending to a customer, writing an order to the ERP, issuing a payment.
• Above authority limits: discounts, credit, quantities or values above a set threshold.
• When agents disagree: the reviewer fails a quote twice, or pricing and compliance conflict.
• On low confidence: any agent marks its result as low confidence or has open questions.
• On sensitive data: cases involving personal or confidential information beyond the usual.

Worked example: the quote checkpoints

Design the review screen, not just the rule

A checkpoint only works if the reviewer can decide quickly and well. Show the evidence, not just the answer: the source email, the price list line used, the compliance rule applied and any reviewer flags. Make approve, edit and reject equally easy. Record who decided, when, and what they changed.

Avoid automation complacency

When agents are right most of the time, reviewers start clicking approve without reading. Counter this by highlighting what changed or was flagged, keeping review queues manageable, occasionally sampling approved cases for a second look, and tracking how often reviewers edit. An edit rate that falls to zero may mean the system is excellent, or that nobody is really reviewing.

Relax checkpoints with evidence

Start with more checkpoints than you think you need. As evidence accumulates that a category of case is handled reliably, for example repeat orders from contract customers under a set value, you can move from approving every case to sampling. Make that a documented decision by the accountable owner, not a quiet drift.

Multi-agent systems inherit every failure mode of a single agent and add new ones that come from agents interacting. Knowing the patterns helps you design against them and recognize them quickly in your logs.

Five characteristic failure modes

Why injection is worse with several agents

In a single agent, untrusted text enters once. In a team, text extracted by one agent can be passed to others as if it were trusted data. A downstream agent may not know that a "note" originally came from an outside email. Treat any content that came from outside your organization as untrusted throughout the system, label it as such in messages, and never let it change instructions or permissions.

Quieter failures

• Lost information: a summary drops a requirement, and nobody notices.
• Responsibility gaps: each agent assumes another checked something.
• Drift: one agent's instructions are updated and its outputs no longer fit the next agent's expectations.
• Silent partial failure: a worker times out and the merge proceeds as if all was well.

A worked diagnosis

A customer complains that a quote listed the wrong valve pressure class. Tracing the case shows intake extracted the right class, but the pricing agent substituted a similar part number with a lower class because it was in stock. The compliance agent checked the original item, not the substituted one. The root cause is a handoff gap: substitutions were not sent back for compliance checking. The fix is a rule, not a better prompt: any substitution triggers a fresh compliance check.

Many multi-agent problems are process design problems. Root cause analysis tools you already know, such as the five whys, work well here.

Safeguards are the engineering controls that keep a multi-agent system within safe bounds even when an agent misbehaves. They belong in code and configuration, not only in instructions, because a model can ignore or misread an instruction but cannot bypass a limit enforced outside it.

The core safeguards

Permission scoping in practice

• Give each agent its own credentials rather than one shared account, so its actions can be limited and traced.
• Default to read-only. Grant write access only for specific actions, and pair write actions with checkpoints.
• Limit data, not just actions: the pricing agent does not need customer contact details.
• Review permissions whenever an agent's role changes.

Other useful controls

1. Idempotency: design write actions so that repeating them does not repeat the effect, for example an order is keyed to the case ID so it cannot be created twice.
2. Output validation: check every structured output before it is used.
3. Untrusted-content labelling: mark outside text so downstream agents treat it as data, never as instructions.
4. Rate limits: cap how many cases or messages run per minute so a fault cannot flood customers or systems.

Worked example: what happens when a loop starts

The reviewer and pricing agent disagree on a lead time. Round three ends without a pass. The step limit stops the loop, the case is marked "needs human review", both agents' last outputs are attached, and the sales rep sees it in their queue. Total cost is capped, the customer gets an answer from a person, and the trace shows exactly why. That is a safeguard working as designed.

When a multi-agent system gives a wrong answer, you need to know which agent, which handoff and which input caused it. Observability is the ability to see inside the system: what happened, in what order, with what inputs and outputs, at what cost.

Traces, spans and logs

• Trace: the complete record of one case from start to finish, tied together by a case or trace ID.
• Span: one step within the trace, such as one agent call or one tool call, with its start time, end time and result.
• Logs: the detailed records attached to each span: inputs, outputs, errors and decisions.

What to record for every step

Worked example: reading a trace

A quote took 14 minutes instead of the usual three. The trace shows intake took 40 seconds, pricing 50 seconds, and the compliance agent made eleven calls to the specification tool, each timing out and retrying. The cause is a slow tool, not a confused agent. The fix is in the tool connection and the retry rule, not the prompt.

Dashboards and alerts

Beyond individual traces, track a few measures over time: cases completed, cases escalated, average cost and duration per case, review edit rates and validation failures by agent. Set alerts for spikes, such as a sudden rise in cost per case or a step-limit stop rate above normal.

Logs contain sensitive data

Traces often include customer names, emails and prices. Protect them like any business record: restrict access, set retention periods, and mask personal information where full text is not needed. Many agent frameworks and monitoring tools offer tracing features; whichever you use, confirm where the trace data is stored.

A single agent can be tested by checking its outputs. A multi-agent system needs testing at two levels: each agent on its own, and the whole system end to end. Good evaluation is what lets you change one agent without fear of breaking the rest.

Build a golden dataset

A golden dataset is a set of realistic cases with known correct outcomes, checked by people who know the work. For the quote system, that could be 50 past requests with their correct extracted fields, prices, compliance results and final quotes. Include:

• Typical, clean cases that represent most of the volume.
• Messy cases: vague wording, missing quantities, attachments only.
• Edge cases: substitutions, contract customers, discounts near the authority limit.
• Adversarial cases: emails containing instructions aimed at the agents.

Remove or mask personal information where it is not needed for the test.

Two levels of testing

Why both levels matter

Every agent can pass its own tests while the system still fails, for example when a substitution is never re-checked. And the system can pass end to end by luck while one agent is weak and another compensates. Per-agent tests tell you where problems are; end-to-end tests tell you whether the business outcome is right.

Score what the business cares about

1. Correctness: right items, quantities, prices and certifications.
2. Safe behaviour: escalates when it should, never acts on injected instructions, never exceeds authority.
3. Efficiency: steps, cost and time per case.
4. Reviewer effort: how much a person had to edit.

Some checks are exact, such as a price matching the list. Others, such as whether a clarification question is clear, may need a scoring rubric applied by a person or by a model used as a grader. If you use a model as a grader, check a sample of its scores against human judgement.

Run the tests on every change

Rerun the golden dataset whenever you change an instruction, a model, a tool or a contract. Compare results with the previous run before going live. Add every real failure you find in production to the dataset, so the same mistake is caught next time.

A multi-agent system is a set of software workers acting on your behalf. Govern it like you would any team with access to customers, money and records: clear authority, limited access, records of what was done, and named people answerable for it.

Least privilege per agent

• Each agent has its own identity and credentials, with only the permissions its role card lists.
• Write access is rare, specific and paired with a checkpoint.
• Secrets such as passwords and API keys are stored in a secure vault, never in prompts.
• Permissions are reviewed on a schedule and whenever a role changes.

Audit trails

An audit trail lets you show, after the fact, who or what did what, when and on what basis. For the quote system it records each agent action, each tool call that changed data, each human approval and each change to instructions, models or permissions, with versions and dates. The traces from the observability lesson are the raw material; governance decides how long they are kept and who may see them.

Governance roles

PIPEDA basics for multi-agent systems

The Personal Information Protection and Electronic Documents Act (PIPEDA) governs how many private-sector organizations in Canada handle personal information in commercial activity; some provinces, including BC, have their own private-sector privacy laws that apply in place of it in certain cases. Its fair information principles apply directly to agent design:

1. Accountability: designate a person responsible for privacy compliance, including for AI systems.
2. Identifying purposes and consent: use personal information only for purposes people would reasonably expect or have agreed to.
3. Limiting collection, use and retention: pass each agent only the personal information it needs, and keep case records and logs no longer than necessary.
4. Safeguards: protect information with security appropriate to its sensitivity, including with AI service providers.
5. Openness and access: be able to explain how information is used and respond to individuals' access requests.

Also check where each AI provider processes and stores data, what their contract says about using your data, and how you would detect and respond to a breach. PIPEDA requires reporting breaches that create a real risk of significant harm. This lesson is general business guidance, not legal advice; confirm your obligations with a qualified privacy professional.

In a multi-agent system, every agent makes its own model calls, and every call carries instructions, context and output. Costs multiply with the number of agents, the number of steps and the size of each context. So does waiting time.

Where the cost comes from

Model providers typically charge per token, with separate prices for input and output tokens. Prices vary widely by provider and model and change over time, so always check your provider's current price list. A rough per-case cost is the sum, for every agent, of its calls multiplied by the tokens per call and the price per token.

Worked example: one quote

That is 11 calls and about 39,000 input and 6,000 output tokens per quote, many times what a single drafting call would use. Multiply by quotes per day and working days per month to see the monthly figure. The calculator lab in this module lets you try your own numbers.

Levers to reduce cost

• Fewer agents and steps: remove agents that do not earn their place; let fixed code orchestrate where possible.
• Smaller contexts: pass lean structured messages, not full transcripts.
• Right-size models: use smaller, cheaper models for routing, extraction and checks; save larger ones for hard reasoning.
• Caching: reuse results for repeated lookups, and use provider prompt caching for long, stable instructions where available.
• Budgets: per-case caps stop runaway loops.

Levers to reduce latency

1. Run independent agents in parallel.
2. Cut sequential handoffs that add little.
3. Use faster models for simple steps.
4. Set timeouts on slow tools.
5. Show progress to users so a short wait feels predictable.

Compare cost with value

Cost only matters relative to value. If each quote's model cost is a small fraction of the time a rep would spend, and quality holds, the system pays for itself. Track cost per completed case, not just the monthly bill, and compare it with the labour and error costs it replaces.

You can build a multi-agent system with your own code calling model APIs directly, or use an agent framework that provides building blocks for agents, tools, handoffs, state and tracing. Both are reasonable. The right choice depends on your team, your systems and how much control you need.

What frameworks typically provide

• Ways to define agents, their instructions and tools.
• Patterns for orchestration, such as supervisors, graphs of steps or role-based teams.
• Shared state and memory handling.
• Support for structured outputs, retries and tool connections, often including MCP.
• Tracing and debugging features.

Well-known open-source examples include LangGraph, CrewAI, Microsoft's AutoGen and Semantic Kernel, and LlamaIndex's agent features. Major model providers also offer their own agent toolkits. These are named as neutral examples only; capabilities change quickly, so evaluate current documentation against your own needs.

Comparing the options

Questions to answer before choosing

1. Can it express the patterns you actually need, and nothing much more?
2. Can you enforce step limits, budgets, timeouts and per-agent permissions?
3. Does it produce traces you can store where you choose?
4. Can you test agents individually and end to end?
5. Does it connect to your systems, directly or through standard connectors?
6. Who maintains it, and what happens if you need to move away?

A practical SME approach

For the distributor's pilot, a sensible path is a small proof of concept in one framework, with the router and checkpoints written as plain code and the agents' contracts kept in your own schemas. That way, the business logic and test data belong to you, and switching frameworks later means rewriting glue, not rethinking the design. Whatever you choose, keep instructions, schemas and golden datasets under version control.

This final lesson pulls the course together into a practical 90-day plan. The goal is not a large system, but one well-chosen multi-agent workflow running safely, with evidence on whether it works.

Before you start: pick the right pilot

• A real, repeated process with a named business owner.
• A clear reason for multiple agents: specialization, parallelism, separation of duties or context limits.
• Moderate risk, with natural human checkpoints before anything irreversible.
• Data and systems you can access through read-only tools at first.

Days 1 to 30: design

1. Map the current process and measure a baseline: time per case, error rate, rework.
2. Choose the simplest pattern that fits and draw it on one page.
3. Write role cards and a RACI, with a person accountable for every activity.
4. Define message contracts and the shared case record.
5. Build a golden dataset of 30 to 50 real cases with verified answers.
6. Agree safeguards, checkpoints and the privacy review.

Days 31 to 60: build and test

1. Build a prototype in a sandbox with read-only tools.
2. Test each agent, each handoff and the full system against the golden dataset.
3. Turn on tracing, budgets, step limits and timeouts from day one.
4. Run in shadow mode: agents prepare outputs while people continue the real work, and you compare.

Days 61 to 90: controlled live use

1. Go live for a defined slice of cases, with a person approving every output.
2. Review traces, costs and edit rates weekly.
3. Add every failure to the golden dataset and retest after each change.
4. At day 90, compare results with the baseline and decide: scale, adjust or stop.

Measures for the day-90 decision

A decision to stop is a valid, useful result. It tells you something real about the process and avoids sinking more money into a design that does not fit. If the pilot succeeds, expand one step at a time, and keep the same discipline of tests, traces and checkpoints as the system grows.