How to hire an AI evaluation company

01

Write down the question

• Capability: can the model do this job to a professional standard?
• Comparison: which model or vendor should we choose for this use case?
• Safety: does the model produce harmful, deceptive, or policy-breaking output?
• Security: can attackers jailbreak it, inject prompts, or extract data through tools and retrieval?
• Dangerous capability: does the model raise cyber, autonomy, or other frontier risks?
• Regression: did the new version get worse on what we care about?

Write one sentence on the decision the result will inform, and who must accept the result (your team, a regulator, customers, or the public).

02

Choose an engagement type

• Managed evaluation: the vendor designs tasks, recruits graders, runs the evaluation, and reports results.
• Domain-expert grading: credentialed experts score outputs against rubrics you agree in advance.
• Custom or commissioned benchmark: a new test set built for your domain, often kept private to avoid contamination.
• Red-team engagement: time-boxed adversarial testing with findings, reproductions, and remediation guidance.
• Pre-deployment assessment: an independent evaluation of a model before release, sometimes published in a system card.
• Self-serve participant platform: you design the study and pay per participant; you own the QA.

03

Decide how much independence you need

Ask whether the evaluator also sells you training data, owns a product that competes with yours, or receives funding from the developer it evaluates. Some organizations decline payment from the companies they assess; others publish their results by default. If the result must convince outsiders, agree in writing what the evaluator may publish, what you may redact, and how disagreements about findings are recorded.

04

Check who does the evaluating

Ask how graders or red teamers are vetted (credentials, tests, past results), how many qualified people exist for your domain and language, whether the same people stay across rounds, and how rubric disagreements are adjudicated. For security work, ask about the team’s background in adversarial ML and application security.

05

Agree the deliverables

Specify the output before you sign: a written report with methods and limitations, raw scores and transcripts, severity ratings, reproducible attack traces, a reusable test suite or harness you can rerun, and a retest after fixes. Ask whether the test set stays private and whether you receive it.

06

Security and confidentiality evidence

Evaluators often see unreleased models and sensitive prompts. Ask for current reports, not badges: SOC 2 Type II report, ISO/IEC 27001 certificate and scope, HIPAA terms if health data is involved, FedRAMP status for US government work, model-access controls, data retention and deletion, and subprocessors. A certification on one product page may not cover the evaluation team.

07

Commercial terms

Clarify the unit of pricing (fixed engagement, per task, per expert hour, per participant, or retainer), timeline to first results, rework and retest policy, IP ownership of test sets and results, publication rights, and exclusivity. Very few evaluation providers publish prices; most quote per engagement.