Run your own AI benchmark

Step by step

1. 1

   Create a suite

   Open the benchmark tool in your Raq.com account and click New Suite. Name it after the job you're testing for, such as customer replies or monthly reports.

2. 2

   Add your prompts

   Click Add Prompt and paste each one exactly as you'd send it. With a lot of them, paste them into Import Cases as JSONL, JSON or CSV. A suite holds up to 200 prompts.

3. 3

   Choose how each one is marked

   Under How to grade, pick Vibes to rate the answers yourself, Exact match when there's one right answer, or AI judges to have a judge model score each answer against criteria you write.

4. 4

   Pick the models

   Click Start a Run and tick up to 25 models, including several versions or reasoning levels of the same one. Attempts per model repeats each prompt, up to ten times.

5. 5

   Check the size, then start

   The dialog shows how many answers the run will produce and the AI usage it will take. Results fill in as the answers arrive.

6. 6

   Compare and share

   The run shows each model's score, cost and speed, and you can open any answer to read it. Publish a leaderboard, or share the run with a link, an optional password and an end date.

What we learned running ours

• Put the answer on its own line

  For questions with one right answer we asked each model to finish with a line starting "Answer:" and checked only that line, so a model couldn't pass by mentioning the answer in passing.

• Try the prompts on a few models first

  Our everyday round was too easy: 459 of the 460 checked answers were right. For the harder round we tried 43 prompts on six models first. Most of the maths, document and UK questions still got full marks, so we wrote 16 harder ones, tried those on three models and kept 36 in all.

• Run the code

  For the coding prompts we ran every answer against hidden tests instead of asking a judge whether the code looked right, then recorded the share of tests passed as that answer's score.

• Use a judge from outside the test

  Kimi K3.1 Pro from Moonshot AI marked the writing in both rounds, because Moonshot had no model in the test.

• Ask each prompt more than once

  Models don't always give the same answer twice, even at temperature 0. With two tries you can see which slips were one-offs.

• Include a trap

  In the everyday round one prompt asked about a novel that doesn't exist, and all ten models said so. In the harder round a bank statement prompt asks for an overdraft limit the statement never gives. A model that makes one up can't be trusted with your documents.

• Put deductions in the marking scheme

  Tell the judge what to look for and how much to take off for each problem. Ours took 0.2 off for any detail the prompt didn't give, and that was the most common reason models lost marks.

Questions