Handshake AI

Overview

Handshake AI connects model developers with human contributors and domain experts for training data, evaluation, and environment work. Its AI program sits alongside a research effort examining professional tasks and the methods used to grade them. This combination puts Handshake in both expert-data supply and agent-evaluation infrastructure.

One concrete example is Gandalf the Grader, published in May 2026. Handshake describes it as an agentic verifier that operates inside the task environment and inspects files and tool state. Its associated BankerVerifierBench evaluates agreement between automated graders and expert judgments on banking-task results.

The distinction between an agent benchmark and a verifier benchmark is important. The former measures task performance; the latter measures whether the scoring system recognizes good and bad work. Handshake reports favorable cost and quality results for Gandalf in its experiments, but those findings should be read with their task distribution and comparison setup.

Why the combination matters

Our assessment is that expert data and grading research address complementary problems. Experts can establish what good work looks like, while automated checks can help apply criteria repeatedly. Neither removes the need to examine whether the evaluation reflects the intended job.

For an illustrative procurement task, a correct summary would be insufficient if the agent updated the wrong supplier record. An evaluator needs access to the consequential state, and a rubric needs to say why that state matters. This makes the design of the grading process part of the product being assessed.

Evaluating fit

Ask separately about contributor sourcing, managed data production, environment construction, and evaluation tooling. For expert work, inspect qualifications, review procedures, and the handling of disagreement. For automated grading, request comparisons against independent human judgments and examples of both false acceptance and false rejection. A pilot should test the buyer's own artifact types and workflow constraints, since a verifier that performs well in one domain may need adaptation in another. Define acceptance criteria around usable data and trustworthy evaluation, with clear ownership and reuse terms for the resulting tasks and labels.