Most orchestration-platform evaluations fail the same way: a team compares feature lists, picks the longest one, and learns six months later that the decisive questions were about data boundaries, log export, and who owns the bill.
This is a screening guide, not a ranking. It explains what evidence to require and when buying is the wrong decision. My procurement rule: reject candidates that cannot make their controls, costs, and exit path inspectable before the pilot.
What Is an Agent Orchestration Platform?
An agent orchestration platform packages agent coordination into hosting, accounts, billing, a UI, and a control layer for models, tools, persistent state, and approvals.
The distinction that matters for procurement is this: the layer is the control plane; the platform is what you buy. A platform contains a layer. A layer does not require a platform because a team can assemble equivalent control from a harness, a configuration repository, and CI.
That is enough category definition for procurement. Screen the following four areas before comparing features; finding a failure after migration is the expensive version of the same evaluation.
Four Checks Before Feature Comparison
1. Deployment Model and Data Boundaries
Deployment may be multi-tenant SaaS, single-tenant hosted, self-hosted, or air-gapped. Availability can vary by plan, region, or feature. Ask which capabilities are unavailable in the mode you would buy; that answer is more useful than four checkmarks.
Then map the data. Prompts, code context, tool results, embeddings, run logs, and configuration are separate data types. Put five questions in writing:
- Where is each data type processed, and where is it stored?
- What is the retention period for each type, and is it configurable?
- Which subprocessors see which types?
- Does deletion remove the data, or only mark it as deleted?
- Is any data used for model training or product improvement by default?
The last question still needs a written answer when the marketing site addresses it. The site is not the contract.
For coding workflows, an agent that reaches a repository may also reach credentials exposed to its execution context; repository access alone does not imply access to every credential the repository can use. GitHub Actions makes the distinction concrete: a workflow can read a secret only when the workflow explicitly includes it, and environment secrets can remain unavailable until a required reviewer approves. Ask how the candidate enforces scope for repositories, secrets, and environments, and whether that boundary lives in the platform or only in your setup conventions.
2. Reconstructability and Log Export
The standard is reconstructability. After a failed run, can you answer:
- Which model handled each step, at what settings?
- Which tools were available, and which were called?
- What state entered and left each step?
- Who approved or overrode the run, when, and under what identity?
This is not an exotic telemetry request. OpenTelemetry's GenAI attributes already distinguish agent identity, conversation IDs, model requests, tool definitions, tool-call arguments, and tool results, while also warning that recorded content may be sensitive. A vendor does not have to use that exact schema, but its export should give you equivalent fields in a machine-readable form.
Can exported records join with your incident data? How long are transcripts, tool calls, and audit events retained? Does deleting a project delete its logs? If the answers live only in the vendor console, your incident review depends on that console after the contract ends.
3. Cost Controls That Survive Real Workloads
Pricing can combine seats, credits, and model usage. The headline rate is rarely the hard part. Check four operating details:
Attribution. Can spend be broken down by team, repository, or workflow? Without that, the first large bill produces an argument instead of a decision.
Caps and limit behavior. Does a run stop, queue, or switch models? A silent fallback to a cheaper model changes the evaluation conditions.
Long-run economics. Agentic work can produce large prompts and repeated tool loops. OpenAI, for example, publishes separate short- and long-context rates for GPT-5.6 Fast mode, with long context triggered above an estimated 272K prompt tokens. Ask whether threshold repricing applies only above the line or to the whole request, then model the pilot with the same context and retry limits you expect in production.
Budget ownership. Name the owner before the pilot. Platform costs sit between engineering and finance; an unowned cost is eventually resolved by whoever is most annoyed, not by policy.
4. Vendor Evidence on Your Repository
Vendor benchmarks tell you where to look. They are not evidence about your codebase.
Run the same task, prompt, tool set, time limit, token budget, and human scoring rubric across every finalist. Different harnesses can produce different outcomes with the same model, so an unmatched pilot compares sales processes more than systems.
Also require a written list of exclusions, named failure modes with recovery paths, and references from teams shaped like yours. A fifty-person success says little about a four-person team, and vice versa.
The recurring red flags are easy to recognize:
- Benchmark numbers without run configuration
- "Enterprise-ready" without a named deployment mode
- No machine-readable export for logs
- Roadmap features presented as current capability
- Pricing that cannot be modeled before the pilot
Set Governance Before the Pilot
Governance failures rarely look like outages. The system keeps working while nobody can say who is responsible for it. NIST's AI Risk Management Framework treats documented roles, human oversight, ongoing review, and safe decommissioning as lifecycle controls, not paperwork to add after deployment.
Settle five questions early:
Who approves exception paths? Check whether the platform provides a way to bypass or relax its controls, such as a permissive mode, skip flag, or admin override. Decide who can use it and what gets recorded. GitHub protected branches show why the detail matters: required reviews and status checks can be paired with an explicit rule that does not allow bypassing those controls. A gate that privileged users can silently disable is a default, not a gate.
Who owns each workflow? An unowned workflow accumulates dependencies and becomes difficult to remove.
What triggers external communication? Decide which failures require notifying a customer or stakeholder. The useful line is whether the incident crossed a boundary you said you controlled.
How do you leave? Ask what is exportable and what becomes unreadable after the subscription ends. Offboarding is a design question, not a renewal dispute.
Where does configuration live? Control configuration held only in a vendor UI cannot be reviewed, diffed, or restored. Prefer a repository-backed configuration or a complete export.
Decide Whether to Buy, Build, or Wait
An agent orchestration platform fits when several people run agents against shared repositories, permissions must be consistent across users, audit evidence must leave the engineering team, or parallel branches and pre-merge review have become the bottleneck. The license should also cost less than building and maintaining equivalent control.
Build your own layer when one or two developers work in separate repositories, a harness and versioned configuration provide enough control, or your constraints would force continuous workarounds. Building also makes more sense while the workflow itself is still changing; buying too early freezes a design you have not settled.
Wait when you cannot name the acceptance test the platform must pass. That is not a vendor problem. It means the evaluation would produce a preference rather than a decision.
Verdent fits when several agents work in parallel on one project, with plans reviewed before execution and changes isolated before integration. Its workflow separates read-only planning, parallel workspaces, and reviewable rebase paths. That is different from choosing a single harness for one developer.
Turn Every Vendor Claim Into a Test
Split responsibility before an incident. The vendor owns platform availability and the documented behavior of its controls. Your team owns configuration, granted permissions, and the workflows built on top. The gray zone is behavior that is technically documented but not expected; shrink it by recording the behaviors your pilot relies on and naming an internal incident owner regardless of fault.
Convert each claim before it enters the decision document. "Reduces review time" becomes "run these ten tasks and compare verification minutes with our current process." "Reliable" becomes "show the failure mode and recovery path for a tool timeout mid-run." Keep each claim beside one evidence status: verified by our pilot, verified by current documentation, or unverified.
Claims that remain unverified should not enter the business case. If removing them empties the case, that is the finding.
FAQ
Who Should Have Access to Agent Run Logs?
Give access at least to the people who debug runs and answer audits. Decide whether code-bearing logs leave engineering and whether developers can see other teams' runs. Verify both scopes in the product.
Who Should Approve Exceptions?
Use the workflow owner or a security reviewer, not the person who is blocked. Make approval time-bounded and record the reason. An exception renewed three times is either policy or a control the team does not need.
What Should the Pilot Report Contain?
Record what was tested, what passed, what failed, what was not tested, and what the pilot could not determine. Have someone who did not run the pilot read the report before procurement. A second reader catches the gap between "this worked" and "this worked for the person who wanted it to."
Related Reading
- For the control-plane boundary, start with what an agentic orchestration layer controls.
- Once several people use the system, decide who owns an agentic workflow across a software team.
- For the merge gate, treat automated review as queue triage rather than final approval.
- Use this DeepSeek Harness checklist to gather pre-adoption evidence.
- When a vendor leads with a benchmark score, check what SWE-bench Pro can and cannot establish.
