How we tested ten AI models

We ran two rounds through ten models: 36 harder prompts on 8 November 2026 and 36 everyday prompts the day before, each asked twice. Here's every prompt, how the answers were marked and where each test falls short.

Updated 8 November 2026

The harder round

Run on
8 November 2026 through Raq.com
Prompts
36 harder tasks in six groups. 26 have answers that are checked automatically, 5 ask for code that is run against hidden tests and 5 are marked by a judge model.
Tries
Every model was given every prompt twice, 720 tries in all. 676 came back with an answer, 5 returned nothing and 39 were never sent because a model reached the limit a run sets on failed calls.
Settings
Prompts sent as written with Raq.com's one-line instruction to use British English, at temperature 0 with each model's default reasoning setting and no web search, through OpenRouter with Zero Data Retention.
Checked by hand
We read every answer the automatic checks marked wrong. 3 right answers written in another format were marked right by hand, with the reason on the run page.
Judge
Kimi K3.1 Pro from Moonshot AI, a company with no model in the test. It scored each judged answer from 0 to 1 against a written marking scheme.
Cost
$7.06 in AI usage for the main run, $6.24 for the answers and $0.820 for the judge. Calls that ran out of room before answering cost about $1.75 more at list prices, which the run total leaves out.

Models and the exact versions called

  • GPT-6.1 Sol openai/gpt-6.1-sol
  • GPT-6 Astra openai/gpt-6-astra
  • Gemini 3.8 Flash google/gemini-3.8-flash
  • Claude Sonnet 5.5 anthropic/claude-sonnet-5.5
  • Gemini 4 Argon google/gemini-4-argon
  • Grok 4.7 x-ai/grok-4.7
  • Claude Opus 5.5 anthropic/claude-opus-5.5
  • GPT-6 Luna openai/gpt-6-luna
  • Claude Haiku 5.5 anthropic/claude-haiku-5.5
  • DeepSeek V4 Pro deepseek/deepseek-v4-pro-0813

How the harder round was marked

26 prompts have answers we could check automatically. The maths and UK questions end with a line starting "Answer:", which had to contain the right answer. The long documents and the instructions with several rules were checked field by field or rule by rule, so a reply with four of six fields right scored 0.67.

The 5 coding prompts ask for a Python function. We ran each answer against 24 to 30 hidden tests in a sandbox with no network access and recorded the share of tests it passed as its score, with the failed tests in the note.

Kimi K3.1 Pro from Moonshot AI marked the 5 writing prompts against a scheme that starts at 1 and lists a deduction for each problem, with 0.2 off for any detail the prompt didn't give.

A model's score is the average of its answers, and a call that returned no answer counts as 0. Full marks counts the answers that scored 1. Reply time is the median time from sending a prompt to getting the whole answer back. Cost is what those answers cost at the provider's price through OpenRouter, not counting the judge.

DeepSeek V4 Pro returned no answer three times, which tripped the limit a run sets on failed calls, so 39 of its 72 tries were never made and its score covers the other 33. Claude Opus 5.5 returned no answer on both tries at one prompt, and those two count as 0.

We read every answer the automatic checks marked wrong. Three gave the right fraction for the four dice prompt written in LaTeX, which the check couldn't read, so we marked those right by hand. The rest stayed as marked.

Every prompt in the harder round

Open a prompt to see the exact text each model received and how its answers were marked.

Maths and logic

Cheapest way to fill a café order Answer checked

A café is ordering supplies. Coffee beans cost £14.50 a bag, or £75 for a box of 6 bags. Milk costs £1.75 a carton, or £12 for a crate of 10 cartons. The café needs at least 22 bags of beans and at least 37 cartons of milk, and it may buy more than it needs if that works out cheaper. Orders over £150 before any discount get 5% off the whole order. Delivery costs £8.50, but is free when the order total after the discount is £150 or more. What is the lowest possible total cost, including delivery? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £314.45. The final "Answer:" line had to contain it.

Loan balance after 18 payments Answer checked

A loan of £12,000 is charged interest at 0.6% a month. At the end of each month, that month's interest is worked out on the balance, rounded to the nearest penny and added, and then a payment of £400 is taken off. What is the balance just after the 18th payment? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £5,785.03. The final "Answer:" line had to contain it.

Days between two dates Answer checked

How many days are there between 14 March 1987 and 9 November 2026, not counting either of those two days? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 14,484. The final "Answer:" line had to contain it.

Multiply a 13-digit number by a 10-digit one Answer checked

What is 4829371605183 multiplied by 7316904582? Give the exact whole number. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 35,336,051,226,144,187,648,506. The final "Answer:" line had to contain it.

Five-digit numbers with no repeated neighbours Answer checked

How many five-digit numbers (10000 to 99999) have no two adjacent digits the same and contain the digit 7 at least once? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 26,281. The final "Answer:" line had to contain it.

Four dice that total 14 Answer checked

Four fair six-sided dice are rolled. What is the probability that they total exactly 14? Give the answer as a fraction in its lowest terms. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 73/648. The final "Answer:" line had to contain it.

Count the letter r in a passage Answer checked

How many times does the letter r appear in this passage? Count capitals and lower case. "Rory parked the rusty tractor near the river, where three herons stood in the reeds. Every spring the farmer repairs the barn roof, orders fresh straw and rearranges the stores. Rain or shine, Brenda brews strong coffee for the workers before the market opens on Thursday morning." End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 41. The final "Answer:" line had to contain it.

Instructions

Filter, sort and format a word list Answer checked

From the list below, keep only the words that contain exactly two vowels (a, e, i, o and u; y is not a vowel, and repeated vowels count each time). Write the words you keep in capital letters, in reverse alphabetical order, separated by a comma and a space, all on one line. Reply with that line and nothing else. harbour, sky, lantern, piano, orbit, crypt, meadow, tulip, bridge, cactus, rhythm, violin, castle, ember, quartz

Right answer: TULIP, QUARTZ, ORBIT, LANTERN, EMBER, CASTLE, CACTUS, BRIDGE. The reply had to be that line and nothing else.

A sentence from A to O Answer checked

Write one sentence of exactly 15 words whose first letters run through the alphabet in order from A to O: the first word starts with A, the second with B, and so on to O. End it with a full stop. Reply with the sentence only.

Right answer: 15 words, A to O in order. Checked for 15 words starting A to O in order and a closing full stop. Each rule scored separately.

A product update with six rules Answer checked

Write a short update for customers of a meal-kit company announcing a vegetarian box. Rules: 1. Exactly three paragraphs, separated by blank lines, and no headings. 2. Include the exact phrase "available from 1 March". 3. Mention the price £14.99 exactly once. 4. Do not use the words "new", "exciting" or "delicious". 5. Fewer than 90 words in total. 6. The last sentence must be a question.

Right answer: All six rules met. Each of the six rules was checked automatically and scored separately.

Five bullets that grow by a word Answer checked

Give five tips for saving energy at home as a list of exactly five bullet points, each starting with "- ". The first bullet must be exactly 3 words long and each bullet after that must be exactly one word longer than the one before it (3, 4, 5, 6 and 7 words, not counting the "- "). No other text.

Right answer: 5 bullets of 3 to 7 words. Checked for five "- " bullets and nothing else, then for 3, 4, 5, 6 and 7 words. Each rule scored separately.

A paragraph of exactly 47 words Answer checked

Write one paragraph about why people keep allotments. It must be exactly 47 words long, counting every word separated by a space; a hyphenated word counts as one. No title and nothing else in your reply.

Right answer: 47 words. Checked by counting the words separated by spaces.

Reverse a string of random characters Answer checked

Reverse this string character by character and reply with the reversed string only: tq7Hb2xLm9vRk4sNp1zWc8yDf3gJ

Right answer: Jg3fDy8cWz1pNs4kRv9mLx2bH7qt. The reply had to be the reversed string and nothing else.

Long documents

Lease: the terms after the variation letter Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Annual rent: (the rent a year, excluding VAT) Rent Commencement Date: Rent deposit: (amount in pounds) First Break Date: Rent review date: First-year service charge cap: (the most the tenant pays for the first year, in pounds) --- LEASE PARTICULARS Unit 4, Canal Wharf Trading Estate, Kidderminster These particulars summarise the lease the parties have agreed. Where the variation letter at the end of this document changes a term, the variation letter applies. 1. Parties Landlord: Cherwell Property Ltd. Tenant: Calder Supplies Ltd. Guarantor: Tom Ashworth, a director of the Tenant. 2. Premises Unit 4, a single-storey warehouse with a mezzanine office, gross internal floor area 2,150 sq ft, together with three allocated parking spaces (bays 11, 12 and 13). The yard is shared with Units 3 and 5, and the Tenant may use it for loading only, not for storage. 3. Term Ten years from and including 1 April 2027 (the Term Commencement Date). The lease will be contracted out of the security of tenure provisions of the Landlord and Tenant Act 1954. 4. Rent 4.1 The initial rent is £38,500 a year, exclusive of VAT. The Landlord has opted to tax the building, so VAT is payable on the rent. 4.2 Rent is payable quarterly in advance on the usual quarter days: 25 March, 24 June, 29 September and 25 December. 4.3 The Tenant has a rent-free period of six months from the Term Commencement Date. Rent becomes payable from the day after the rent-free period ends (the Rent Commencement Date), with a proportionate payment on that date to cover the period up to the next quarter day. 5. Rent review The rent will be reviewed on the fifth anniversary of the Term Commencement Date to the open market rent. The review is upward only. If the parties have not agreed the new rent three months after the review date, either may refer it to an independent surveyor acting as an expert. 6. Tenant's break option 6.1 The Tenant may end the lease on 31 March 2030 or on 31 March 2032 (each a Break Date) by giving the Landlord not less than six months' written notice. 6.2 The break is conditional on the Tenant having paid all rent due up to the Break Date and giving vacant possession. Breaches of other covenants do not prevent the break. 7. Rent deposit The Tenant will pay a rent deposit equal to three months' rent at the initial rent, to be held by the Landlord in a separate designated account and returned within 28 days of the end of the lease less any sums properly due. 8. Service charge The Tenant will contribute to the cost of maintaining the estate roads, lighting, landscaping and security. For the first service charge year the Tenant's contribution is capped at £2.40 per sq ft of the Premises. After that, the cap rises each year by the percentage increase in the Consumer Prices Index. 9. Insurance The Landlord insures the building and recovers the premium from the Tenant. The Tenant insures its own stock, fixtures and contents and holds public liability cover of at least £5 million. 10. Use Storage and distribution within Use Class B8, with ancillary offices. Retail sales to the public are not permitted. Trade counter sales are permitted on up to 10% of the floor area. 11. Repair and alterations 11.1 The Tenant must keep the inside of the Premises in good repair, but need not put them into better condition than shown in the photographic schedule of condition dated 20 March 2027. 11.2 Internal non-structural alterations need the Landlord's consent, not to be unreasonably withheld. Structural alterations and changes to the external appearance are not permitted. 12. Assignment and subletting Assignment of the whole is permitted with the Landlord's consent, not to be unreasonably withheld, subject to an authorised guarantee agreement. Subletting of part is not permitted. Subletting of the whole is permitted at no less than the passing rent. 13. Interest on late payment Any sum unpaid 14 days after its due date carries interest at 4% a year above the Bank of England base rate from the due date until payment. 14. Guarantee The Guarantor guarantees the Tenant's obligations for the first three years of the term only. 15. Notices Notices under the lease must be in writing and delivered by hand or sent by recorded delivery to the registered office of the party receiving them. Notices sent by email are not valid. 16. Costs Each party pays its own legal costs. VARIATION LETTER From: Cherwell Property Ltd To: Calder Supplies Ltd Date: 14 January 2027 Following our meeting on 9 January, we confirm the following changes to the agreed particulars for Unit 4. All other terms stay as set out above. (a) Clause 4.1: the initial rent is reduced to £36,900 a year, exclusive of VAT. (b) Clause 4.3: the rent-free period is extended to nine months. (c) Clause 6.1: the first Break Date moves to 31 March 2031 and the notice period becomes nine months. The second Break Date is unchanged. (d) Clause 7: the rent deposit will be three months' rent at the reduced initial rent. (e) Clause 14: the guarantee period is extended to the first five years of the term. For the avoidance of doubt, the parking allocation is reduced to two spaces (bays 11 and 12), because bay 13 is needed for the electric vehicle charger the Landlord is installing.

Right answer: Annual rent £36,900; Rent Commencement Date 1 January 2028; Rent deposit £9,225; First Break Date 31 March 2031; Rent review date 1 April 2032; First-year service charge cap £5,160. Each line was checked on its own, so every right line earned its share of the mark.

Lease: rent, guarantee and limits Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Rent to 31 March 2029: (total rent excluding VAT for 1 April 2027 to 31 March 2029) Guarantee ends: (the last day the guarantee covers) Parking spaces: (number) Trade counter limit: (largest trade counter area in sq ft) Late payment interest: (the rate) Email notices valid: (yes or no) --- LEASE PARTICULARS Unit 4, Canal Wharf Trading Estate, Kidderminster These particulars summarise the lease the parties have agreed. Where the variation letter at the end of this document changes a term, the variation letter applies. 1. Parties Landlord: Cherwell Property Ltd. Tenant: Calder Supplies Ltd. Guarantor: Tom Ashworth, a director of the Tenant. 2. Premises Unit 4, a single-storey warehouse with a mezzanine office, gross internal floor area 2,150 sq ft, together with three allocated parking spaces (bays 11, 12 and 13). The yard is shared with Units 3 and 5, and the Tenant may use it for loading only, not for storage. 3. Term Ten years from and including 1 April 2027 (the Term Commencement Date). The lease will be contracted out of the security of tenure provisions of the Landlord and Tenant Act 1954. 4. Rent 4.1 The initial rent is £38,500 a year, exclusive of VAT. The Landlord has opted to tax the building, so VAT is payable on the rent. 4.2 Rent is payable quarterly in advance on the usual quarter days: 25 March, 24 June, 29 September and 25 December. 4.3 The Tenant has a rent-free period of six months from the Term Commencement Date. Rent becomes payable from the day after the rent-free period ends (the Rent Commencement Date), with a proportionate payment on that date to cover the period up to the next quarter day. 5. Rent review The rent will be reviewed on the fifth anniversary of the Term Commencement Date to the open market rent. The review is upward only. If the parties have not agreed the new rent three months after the review date, either may refer it to an independent surveyor acting as an expert. 6. Tenant's break option 6.1 The Tenant may end the lease on 31 March 2030 or on 31 March 2032 (each a Break Date) by giving the Landlord not less than six months' written notice. 6.2 The break is conditional on the Tenant having paid all rent due up to the Break Date and giving vacant possession. Breaches of other covenants do not prevent the break. 7. Rent deposit The Tenant will pay a rent deposit equal to three months' rent at the initial rent, to be held by the Landlord in a separate designated account and returned within 28 days of the end of the lease less any sums properly due. 8. Service charge The Tenant will contribute to the cost of maintaining the estate roads, lighting, landscaping and security. For the first service charge year the Tenant's contribution is capped at £2.40 per sq ft of the Premises. After that, the cap rises each year by the percentage increase in the Consumer Prices Index. 9. Insurance The Landlord insures the building and recovers the premium from the Tenant. The Tenant insures its own stock, fixtures and contents and holds public liability cover of at least £5 million. 10. Use Storage and distribution within Use Class B8, with ancillary offices. Retail sales to the public are not permitted. Trade counter sales are permitted on up to 10% of the floor area. 11. Repair and alterations 11.1 The Tenant must keep the inside of the Premises in good repair, but need not put them into better condition than shown in the photographic schedule of condition dated 20 March 2027. 11.2 Internal non-structural alterations need the Landlord's consent, not to be unreasonably withheld. Structural alterations and changes to the external appearance are not permitted. 12. Assignment and subletting Assignment of the whole is permitted with the Landlord's consent, not to be unreasonably withheld, subject to an authorised guarantee agreement. Subletting of part is not permitted. Subletting of the whole is permitted at no less than the passing rent. 13. Interest on late payment Any sum unpaid 14 days after its due date carries interest at 4% a year above the Bank of England base rate from the due date until payment. 14. Guarantee The Guarantor guarantees the Tenant's obligations for the first three years of the term only. 15. Notices Notices under the lease must be in writing and delivered by hand or sent by recorded delivery to the registered office of the party receiving them. Notices sent by email are not valid. 16. Costs Each party pays its own legal costs. VARIATION LETTER From: Cherwell Property Ltd To: Calder Supplies Ltd Date: 14 January 2027 Following our meeting on 9 January, we confirm the following changes to the agreed particulars for Unit 4. All other terms stay as set out above. (a) Clause 4.1: the initial rent is reduced to £36,900 a year, exclusive of VAT. (b) Clause 4.3: the rent-free period is extended to nine months. (c) Clause 6.1: the first Break Date moves to 31 March 2031 and the notice period becomes nine months. The second Break Date is unchanged. (d) Clause 7: the rent deposit will be three months' rent at the reduced initial rent. (e) Clause 14: the guarantee period is extended to the first five years of the term. For the avoidance of doubt, the parking allocation is reduced to two spaces (bays 11 and 12), because bay 13 is needed for the electric vehicle charger the Landlord is installing.

Right answer: Rent to 31 March 2029 £46,125; Guarantee ends 31 March 2032; Parking spaces 2; Trade counter limit 215 sq ft; Late payment interest 4% above base rate; Email notices valid no. Each line was checked on its own, so every right line earned its share of the mark.

Supplier account: VAT and balances Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Total VAT: (VAT on all invoices less VAT on credit notes, in pounds) Balance owed: (what the customer owes now, after invoices, credit notes and payments) Balance on invoice 1047: Net invoiced after credits: (total net value invoiced less net credited) --- ACCOUNT FILE: Dawson Interiors (customer account DI-204) Supplier: Brown & Hargreaves Supplies All amounts are in pounds. VAT rate codes: S = standard rate 20%, R = reduced rate 5%, Z = zero rate 0%. VAT is worked out on each invoice from the total net value at each rate, rounded to the nearest penny. Payment terms: 30 days from the invoice date. QUOTATION Q-77 (2 September 2027). This is a quotation, not an invoice. Oak worktop, 3 m, 1 at 289.00 (S) Worktop fitting kit, 1 at 24.00 (S) Valid for 14 days. INVOICE 1041 (3 September 2027) Timber battens, 12 at 8.75 (S) = 105.00 Bags of multi-finish plaster, 3 at 14.20 (S) = 42.60 Delivery (S) = 35.00 Net 182.60, VAT 36.52, total 219.12 INVOICE 1047 (17 September 2027) Insulation board, 40 m2 at 11.50 (Z) = 460.00 Stair gates, 2 at 32.00 (R) = 64.00 LED ceiling panels, 6 at 24.50 (S) = 147.00 Delivery (S) = 35.00 Net 706.00, VAT 39.60, total 745.60 DELIVERY NOTE DN-5521 (17 September 2027) Insulation board 40 m2, stair gates 2, LED ceiling panels 6. Signed for by K. Dawson. PRO FORMA PF-12 (20 September 2027). For approval only. This is not a VAT invoice and nothing is owed on it. Bathroom extractor fans, 4 at 46.00 (S) = 184.00 EMAIL from Kristin Dawson, 23 September 2027 "Two of the LED panels arrived with cracked diffusers. I'd like to return them for a credit rather than replacements. Please also hold the extractor fans on the pro forma until I confirm the bathroom layout." CREDIT NOTE CN-118 (24 September 2027), against invoice 1047 LED ceiling panels returned, 2 at 24.50 (S) = 49.00 Net 49.00, VAT 9.80, total 58.80 INVOICE 1053 (2 October 2027) Floor tiles, 25 boxes at 18.40 (S) = 460.00 Tile adhesive, 5 tubs at 9.90 (S) = 49.50 Net 509.50, VAT 101.90, total 611.40 PAYMENTS RECEIVED 10 September 2027: 219.12, reference "INV1041" 30 September 2027: 500.00, reference "on account, inv 1047" EMAIL from Brown & Hargreaves accounts, 6 October 2027 "Thank you for your payments. We have applied the 500.00 received on 30 September to invoice 1047. The extractor fans remain on hold and have not been invoiced. Quotation Q-77 has now expired."

Right answer: Total VAT £168.22; Balance owed £798.20; Balance on invoice 1047 £186.80; Net invoiced after credits £1,349.10. Each line was checked on its own, so every right line earned its share of the mark.

Meeting minutes: who does what Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Roof quotes: (who gets them, and by when) Temporary repairs arranged by: Safeguarding lead: Approved boiler figure: (in pounds) Cleaning contract: (the correct yearly cost) Hire income increase: (January to November 2027 compared with the same period last year, in pounds) --- MINUTES OF THE TRUSTEES' MEETING Holloway Road Community Hall Trust Tuesday 7 December 2027, 7pm, small hall Present: James Cooper (chair), Janet Hudson (treasurer), Kristin Dawson (secretary), Owen Whitehead, Rachel Bates, Kofi Day, Simran Chambers. Apologies: Tom Ashworth, George Malik. In attendance: Kerry Smith (hall manager, not a trustee and without a vote). The meeting was quorate (quorum: five trustees). 1. Minutes of the last meeting The minutes of the meeting of 9 November 2027 were approved with one correction: item 4 of those minutes should have recorded the cleaning contract as £6,240 a year, not £6,420. 2. Matters arising Kofi Day reported that the replacement boiler was fitted on 24 November and the old one has been taken away. The final invoice was £4,180, which was £320 under the approved figure. 3. Budget for 2028 Janet Hudson presented the draft budget of £48,200. Rachel Bates proposed an amendment to add £2,500 for a public-access defibrillator and cabinet outside the main entrance. The amendment was carried 4 votes to 2 with 1 abstention. The amended budget was then approved 6 votes to 0 with 1 abstention. 4. Hall hire charges The trustees agreed to raise the weekday hourly rate for the main hall from £18 to £21 from 1 February 2028. The weekend hourly rate stays at £26. Registered local charities will continue to get a 25% discount on all rates. Simran Chambers proposed raising the discount to 30%; nobody seconded the proposal. 5. Roof repairs The hall manager reported water coming in above the stage after heavy rain. Owen Whitehead will obtain three quotes for repairs by 14 January 2028. Spending of up to £1,500 on temporary repairs was approved, and Kerry Smith will arrange them. 6. Website and bookings Kristin Dawson will move the booking calendar to the new website by 31 January 2028. Until then, bookings continue by email. 7. Safeguarding Rachel Bates was appointed safeguarding lead, replacing Tom Ashworth, who asked to step down from the role but remains a trustee. 8. Any other business Kofi Day gave notice that he will stand down as a trustee at the annual general meeting in April 2028. The chair thanked him for eleven years' service. 9. Date of next meeting The second Tuesday of February 2028, at 7pm in the small hall. APPENDIX A: TREASURER'S REPORT TO 30 NOVEMBER 2027 Hire income, January to November: £31,460 (the same period last year: £28,915). Grants received: £6,000 from the county council community fund. Running costs, January to November: £29,870, including the boiler. Bank balance on 30 November 2027: current account £12,304; reserve account £20,000. The reserves policy is to hold at least six months' running costs in the reserve account. ADDENDUM, agreed by email on 10 December 2027 The vote on the amendment in item 3 was miscounted. It was carried 5 votes to 1 with 1 abstention. In item 5, Owen Whitehead's deadline for the roof quotes is 21 January 2028, not 14 January.

Right answer: Roof quotes Owen Whitehead by 21 January 2028; Temporary repairs arranged by Kerry Smith; Safeguarding lead Rachel Bates; Approved boiler figure £4,500; Cleaning contract £6,240; Hire income increase £2,545. Each line was checked on its own, so every right line earned its share of the mark.

Timesheet: hours, overtime and pay Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Kristin Dawson's paid hours: (as a decimal number) Tom Ashworth's gross pay: (in pounds) Total overtime hours: (all staff, as a decimal number) Total Sunday pay: (all staff, in pounds) Highest gross pay: (name) --- BROWN & HARGREAVES SUPPLIES, KIDDERMINSTER DEPOT Weekly timesheet: week beginning Monday 6 December 2027 Pay rules (from the staff handbook, section 4) 4.1 Paid time is the length of the shift less an unpaid 30-minute break for any shift longer than six hours. A shift of exactly six hours has no break deducted. 4.2 Hours worked on a Sunday are paid at twice the basic rate and do not count towards the weekly overtime threshold. 4.3 Paid hours from Monday to Saturday above 40 in the week are overtime, paid at one and a half times the basic rate. The first 40 are paid at the basic rate. 4.4 Shifts not worked are not paid, whatever the timesheet shows. Basic hourly rates Kristin Dawson: £13.60 Tom Ashworth: £12.85 James Cooper: £15.20 Owen Whitehead: £12.60 Shifts (start to finish, 24-hour clock) Kristin Dawson Monday 6 Dec: 07:00 to 15:30 Tuesday 7 Dec: 07:00 to 15:30 Wednesday 8 Dec: 07:00 to 13:00 Thursday 9 Dec: 07:00 to 17:15 Friday 10 Dec: 07:00 to 16:00 Saturday 11 Dec: 08:00 to 12:30 Tom Ashworth Monday 6 Dec: 09:00 to 17:30 Tuesday 7 Dec: 09:00 to 17:30 Thursday 9 Dec: 09:00 to 17:30 Friday 10 Dec: 09:00 to 17:30 Sunday 12 Dec: 10:00 to 16:00 James Cooper Monday 6 Dec: 06:00 to 14:30 Tuesday 7 Dec: 06:00 to 14:30 Wednesday 8 Dec: 06:00 to 14:30 Thursday 9 Dec: 06:00 to 14:30 Friday 10 Dec: 06:00 to 15:00 Sunday 12 Dec: 08:00 to 14:30 Owen Whitehead Monday 6 Dec: 12:00 to 20:00 Tuesday 7 Dec: 12:00 to 20:00 Wednesday 8 Dec: 12:00 to 16:00 Friday 10 Dec: 12:00 to 20:30 Saturday 11 Dec: 09:00 to 17:00 Supervisor's notes, added Monday 13 December - Tom Ashworth's Thursday shift was entered wrongly. He worked 09:00 to 18:30 after the late delivery, not 09:00 to 17:30. - Owen Whitehead rang in sick on Wednesday 8 December before his shift started and did not work it. The shift was left on the sheet by mistake. - Kristin Dawson's Saturday shift is correct as shown; she asked for it to be checked because it was not on the rota. Approved: Janet Hudson, depot manager

Right answer: Kristin Dawson's paid hours 44.75; Tom Ashworth's gross pay £578.25; Total overtime hours 5.25; Total Sunday pay £336.60; Highest gross pay James Cooper. Each line was checked on its own, so every right line earned its share of the mark.

Bank statement: balance and totals Answer checked

Answer from the document below. Give each answer on its own line, starting with the label exactly as written, and nothing else. Closing balance on 30 November: (in pounds) Net paid to Brown & Hargreaves Supplies: (payments less refunds, both months) Direct debits in November: (number) Largest card payment: (in pounds) Total paid in during October: (everything in the Paid in column) Overdraft limit: (write Not stated if the statement doesn't give it) --- DAWSON INTERIORS LTD, BUSINESS CURRENT ACCOUNT Statement for 1 October 2027 to 30 November 2027 Balance brought forward on 1 October 2027: £4,218.37 Date | Description | Paid out | Paid in 01 Oct 2027 | Direct debit, Kidderminster Council business rates | £286.00 | 02 Oct 2027 | Card payment, Screwfix Kidderminster | £64.18 | 04 Oct 2027 | Faster payment from J Cooper, invoice 2219 | | £1,450.00 06 Oct 2027 | Card payment, Brown & Hargreaves Supplies | £611.40 | 08 Oct 2027 | Direct debit, Octopus Energy | £142.65 | 11 Oct 2027 | Card payment, Shell Stourport Road | £71.02 | 13 Oct 2027 | Faster payment from T Ashworth, invoice 2224 | | £860.00 15 Oct 2027 | Transfer to savings | £1,000.00 | 18 Oct 2027 | Card payment, Brown & Hargreaves Supplies | £219.12 | 20 Oct 2027 | Card refund, Brown & Hargreaves Supplies | | £58.80 22 Oct 2027 | Direct debit, BT Business | £48.00 | 25 Oct 2027 | Faster payment from K Dawson, invoice 2230 | | £395.50 27 Oct 2027 | Card payment, Wickes Kidderminster | £327.49 | 29 Oct 2027 | Faster payment to Hudson Joinery | £1,180.00 | 30 Oct 2027 | Returned payment, Hudson Joinery (account details wrong) | | £1,180.00 01 Nov 2027 | Direct debit, Kidderminster Council business rates | £286.00 | 01 Nov 2027 | Faster payment to Hudson Joinery | £1,180.00 | 03 Nov 2027 | Card payment, Brown & Hargreaves Supplies | £745.60 | 05 Nov 2027 | Faster payment from J Cooper, invoice 2236 | | £2,120.00 08 Nov 2027 | Direct debit, Octopus Energy | £151.90 | 10 Nov 2027 | Card payment, Screwfix Kidderminster | £38.75 | 12 Nov 2027 | Card payment, Shell Stourport Road | £68.44 | 15 Nov 2027 | Direct debit, Zurich business insurance | £93.27 | 17 Nov 2027 | Faster payment from T Ashworth, invoice 2241 | | £640.00 19 Nov 2027 | Card payment, Wickes Kidderminster | £412.06 | 22 Nov 2027 | Direct debit, BT Business | £48.00 | 24 Nov 2027 | Card refund, Wickes Kidderminster | | £36.99 26 Nov 2027 | Card payment, Brown & Hargreaves Supplies | £92.30 | 29 Nov 2027 | Transfer from savings | | £500.00 30 Nov 2027 | Account fee | £12.50 | Running balances are not shown on this statement. Contact us if you do not recognise a transaction.

Right answer: Closing balance on 30 November £4,380.98; Net paid to Brown & Hargreaves Supplies £1,609.62; Direct debits in November 4; Largest card payment £745.60; Total paid in during October £3,944.30; Overdraft limit Not stated. Each line was checked on its own, so every right line earned its share of the mark.

UK questions

Christmas bank holidays in 2027 Answer checked

In England and Wales, on which two dates in December 2027 do the Christmas bank holidays fall? Give both dates. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 27 and 28 December 2027. The final "Answer:" line had to contain it.

Belfast or Newcastle Answer checked

Which city is further north: Belfast or Newcastle upon Tyne? Name the city only. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: Newcastle upon Tyne. The final "Answer:" line had to contain it.

Minimum age for the National Lottery Answer checked

What is the minimum age to buy a National Lottery ticket in the UK? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 18. The final "Answer:" line had to contain it.

Statutory redundancy pay Answer checked

An employee in England started her job on her 35th birthday and is made redundant on her 45th birthday, so she has exactly 10 years' service. Her weekly pay is £500, which is below the statutory cap. How much statutory redundancy pay is she entitled to? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £6,000. The final "Answer:" line had to contain it.

Registering a birth in Scotland Answer checked

Within how many days of a baby being born must the birth be registered in Scotland? Give the number of days. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 21 days. The final "Answer:" line had to contain it.

Default speed limit in Welsh towns Answer checked

What is the default speed limit on restricted roads in Wales, such as residential streets with street lights? Give the speed in mph. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 20 mph. The final "Answer:" line had to contain it.

Saturday hours for a 15-year-old Answer checked

In England, during term time, what is the most hours a 15-year-old can work on a Saturday? Give the number of hours. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 8 hours. The final "Answer:" line had to contain it.

Coding

Parse a duration such as 1h 30m Hidden tests

Write a Python function `parse_duration(text: str) -> int` that converts a duration such as "1h 30m" into a whole number of seconds. Rules: - A duration is one or more parts. Each part is a whole number of one or more digits followed immediately by a unit: h (hours), m (minutes) or s (seconds). - Units must appear in the order h, m, s, and each unit at most once. Parts are either written together ("1h30m") or separated by exactly one space ("1h 30m"). - Leading zeros are allowed ("05m"). Upper-case units are not. - Anything else raises ValueError, including an empty string, leading or trailing spaces, more than one space between parts, a missing number or unit, a sign, a decimal point, an unknown unit, and repeated or out-of-order units. Examples: "1h 30m" -> 5400, "90s" -> 90, "2h5s" -> 7205, "0m" -> 0. Use Python 3 and only the standard library. Reply with one Python code block containing the complete function and any helpers it needs. Don't include tests or example usage.

How it was marked: We ran the code against 24 hidden tests, covering the rules in the prompt and the inputs it must reject, in a sandbox with no network. The score is the share of tests passed.

Evaluate an arithmetic expression Hidden tests

Write a Python function `evaluate(expression: str) -> int` that evaluates an integer arithmetic expression. - Numbers are non-negative decimal integers. The operators are +, -, * and /, with parentheses for grouping. Spaces may appear between any tokens. - * and / bind tighter than + and -. Operators of the same precedence are evaluated left to right. - / is integer division that truncates toward zero, so 7 / 2 is 3 and -7 / 2 is -3. - Unary minus may appear before a number, a parenthesis or another unary minus, as in "-3", "-(2 + 1)", "2 * -3" and "--4". There is no unary plus. - Raise ZeroDivisionError when dividing by zero. Raise ValueError for malformed input: an empty string, unbalanced parentheses, a missing operand such as "2 * / 3" or "2 +", two numbers in a row, or any other character. - Don't use eval, exec, compile or the ast module. Use Python 3 and only the standard library. Reply with one Python code block containing the complete function and any helpers it needs. Don't include tests or example usage.

How it was marked: We ran the code against 28 hidden tests, covering the rules in the prompt and the inputs it must reject, in a sandbox with no network. The score is the share of tests passed.

Compare two version numbers Hidden tests

Write a Python function `compare_versions(a: str, b: str) -> int` that returns -1 if version a has lower precedence than version b, 0 if they have equal precedence and 1 if a has higher precedence, following Semantic Versioning 2.0.0. - A version is MAJOR.MINOR.PATCH, three non-negative integers with no leading zeros, optionally followed by a hyphen and a pre-release, then optionally by a plus sign and build metadata. - Pre-release and build metadata are dot-separated identifiers. Each identifier is non-empty and uses only ASCII letters, digits and hyphens. Numeric pre-release identifiers have no leading zeros. - Compare MAJOR, MINOR and PATCH numerically. A version with a pre-release has lower precedence than the same version without one. - Compare pre-releases identifier by identifier from left to right: identifiers made only of digits compare numerically, others compare in ASCII order, and a numeric identifier is lower than a non-numeric one. If every compared identifier is equal, the pre-release with more identifiers is higher. - Build metadata is ignored for precedence. - Raise ValueError if either string is not a valid version. Use Python 3 and only the standard library. Reply with one Python code block containing the complete function and any helpers it needs. Don't include tests or example usage.

How it was marked: We ran the code against 30 hidden tests, covering the rules in the prompt and the inputs it must reject, in a sandbox with no network. The score is the share of tests passed.

Parse CSV with quoted fields Hidden tests

Write a Python function `parse_csv(text: str) -> list[list[str]]` that parses CSV text into a list of records, each a list of fields. Don't use the csv module. Rules: - Fields are separated by commas. Records end with a line ending, which is "\n" or "\r\n". - A field may be enclosed in double quotes. Inside a quoted field, commas and line endings are part of the field, and two double quotes in a row stand for one double quote. - Spaces are part of a field and are never stripped. A field that does not start with a double quote is unquoted. - A double quote anywhere in an unquoted field is an error. So is a quoted field with no closing quote, and any character other than a comma or a line ending straight after a closing quote. - A line ending after the last record is optional and does not add a record. An empty string returns []. - An empty line is a record with one empty field, [""]. - Raise ValueError for every error. Use Python 3 and only the standard library. Reply with one Python code block containing the complete function and any helpers it needs. Don't include tests or example usage.

How it was marked: We ran the code against 28 hidden tests, covering the rules in the prompt and the inputs it must reject, in a sandbox with no network. The score is the share of tests passed.

Word-wrap text to a width Hidden tests

Write a Python function `wrap(text: str, width: int) -> list[str]` that word-wraps text into lines. Rules: - Words are separated by any run of whitespace. Each line holds words separated by single spaces, with no spaces at either end. - Fill lines greedily: put as many words on a line as fit within `width` characters, then start a new line. - A word longer than `width` is split. If the current line already has words, the long word starts a new line. Cut it into pieces of exactly `width` characters from the left; every piece except the last is a line of its own, and the last piece (which may be shorter) starts the next line, so the words after it can join it. - Return [] when there are no words. - Raise ValueError if `width` is less than 1. Use Python 3 and only the standard library. Reply with one Python code block containing the complete function and any helpers it needs. Don't include tests or example usage.

How it was marked: We ran the code against 24 hidden tests, covering the rules in the prompt and the inputs it must reject, in a sandbox with no network. The score is the share of tests passed.

Writing

Reply to a refund request with limits Marked by the judge

Write a reply from Kristin at a furniture shop to a customer, James Cooper, who wants a full refund for a sofa delivered 40 days ago because the colour looks different at home. Facts: the returns window for change of mind is 30 days; the sofa is not faulty; the shop can offer a free fabric swatch pack and a 15% discount on a replacement cover. Decline the refund politely, explain why in one sentence, make both offers, and don't promise anything else. Under 110 words, no subject line.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. The reply must: be addressed to James and signed from Kristin (-0.1 if either is missing); clearly decline the full refund (-0.3 if it offers or hints at a refund, store credit or exchange); give the reason as the 30-day change-of-mind window in no more than one sentence (-0.15 if the reason is missing or wrong, -0.05 if it takes more than one sentence); offer the free fabric swatch pack (-0.15 if missing); offer 15% off a replacement cover (-0.15 if missing or the figure is wrong); stay polite and not blame the customer (-0.1); be under 110 words (-0.3 if clearly over about 115 words, -0.1 if it has a subject line). Deduct 0.2 for each fact, promise, name, date or figure that the prompt did not give (bracketed placeholders are fine). Minimum 0.

Explain interest rates to a 12-year-old Marked by the judge

Explain to a 12-year-old how the Bank of England's interest rate affects savers and borrowers. Keep these four facts: the Bank of England sets a rate called Bank Rate; banks usually follow it when setting their own rates; when it goes up, savings tend to earn more and loans and mortgages tend to cost more; the Bank raises it to slow price rises. Use no more than 90 words, no bullet points and no jargon such as "monetary policy" or "inflation targeting".

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. It must keep all four facts: the Bank of England sets Bank Rate (-0.15 if missing); banks usually follow it (-0.15 if missing); when it rises savings tend to earn more and loans or mortgages tend to cost more (-0.2 if either half is missing or reversed); the Bank raises it to slow price rises (-0.15 if missing or wrong). It must suit a 12-year-old: plain words, no terms such as 'monetary policy', 'inflation targeting' or 'base rate transmission' (-0.15). No bullet points (-0.1). No more than 90 words (-0.3 if clearly over about 95 words). Deduct 0.2 for any wrong or made-up fact, such as a specific current rate, date or statistic. Minimum 0.

Job advert from rough notes Marked by the judge

Turn these notes into a job advert of 100 to 150 words. Don't add anything the notes don't say. Notes: part-time bookkeeper, Calder Supplies Ltd, Kidderminster office, 20 hours a week over 3 days (days flexible), £15.50 an hour, must know Xero, payroll experience helpful but not essential, start in January, apply by emailing a CV to [email protected] by 12 December.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. The advert must state, correctly: part-time bookkeeper at Calder Supplies Ltd (-0.1); Kidderminster office (-0.1); 20 hours a week over 3 flexible days (-0.15 if hours or days are missing or wrong); £15.50 an hour (-0.2 if missing or wrong); Xero required (-0.1); payroll experience helpful but not essential (-0.15 if missing or made essential); start in January (-0.1); apply by emailing a CV to [email protected] by 12 December (-0.2 if the address or deadline is missing or wrong). It must be 100 to 150 words (-0.2 if clearly outside 95 to 155). Deduct 0.2 for each invented detail, such as benefits, holiday, pension, a salary range, team size, company history, remote working or a contact name. Minimum 0.

Decision record from an email thread Marked by the judge

Read this email thread and write a decision record with exactly four lines, starting "Decision:", "Owner:", "Deadline:" and "Still open:". Rachel Bates (Mon 9:12): Can we move the stock check from Friday to Thursday? The van is booked on Friday. Owen Whitehead (Mon 9:40): Thursday is fine for the warehouse team, but we'd need to finish by 3pm because of the delivery window. Janet Hudson (Mon 10:05): Agreed on Thursday. Owen, can you lead it? I'll need the count sheets by Friday noon for the month-end figures. Owen Whitehead (Mon 10:20): Yes, I'll lead it and send the sheets by Friday noon. Do we still want the mezzanine counted, or leave it to next month? Rachel Bates (Mon 11:02): No view on the mezzanine from me. Janet? Janet Hudson (Mon 11:30): Let me check with the auditors and come back to you.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. Exactly four lines starting 'Decision:', 'Owner:', 'Deadline:' and 'Still open:' (-0.2 if the format differs, such as extra lines, headings or commentary). Decision: the stock check moves from Friday to Thursday, finishing by 3pm (-0.2 if the move is missing or wrong, -0.1 if the 3pm finish is missing). Owner: Owen Whitehead leads it and sends the count sheets (-0.2 if wrong or missing). Deadline: count sheets to Janet by Friday noon (-0.2 if wrong or missing). Still open: whether to count the mezzanine this month, with Janet checking with the auditors (-0.2 if missing; -0.1 if it names the wrong person). Deduct 0.2 for anything presented as agreed that was not, such as a decision on the mezzanine. Minimum 0.

Fix six errors and change nothing else Marked by the judge

Correct the spelling, grammar and factual consistency errors in this paragraph and change nothing else. Reply with the corrected paragraph only. "Our new office in Kidderminster opens on Monday 1 March. It's car park has spaces for twelve cars, and their is a bus stop outside. The office is open from 8am to 6pm, which means its open for nine hours each weekday. Staff should bring there ID badges on the first day, and anyone who looses theirs should tell reception."

Marking scheme given to the judge: Start at 1.0. The paragraph has six errors, each worth -0.15 if left unfixed: 'It's car park' should be 'Its car park'; 'their is' should be 'there is'; 'nine hours' is inconsistent with 8am to 6pm and should be 'ten hours' (changing the opening times instead also counts as fixed only if the result is consistent and the reply flags nothing else); 'its open' should be 'it's open'; 'bring there ID badges' should be 'bring their ID badges'; 'looses' should be 'loses'. Deduct 0.1 for each other change to wording, punctuation or facts beyond these fixes (do not penalise straight versus curly quotes or keeping or dropping the surrounding quotation marks). Deduct 0.2 if the reply includes anything besides the corrected paragraph, such as a list of changes. Monday 1 March is not an error to fix. Minimum 0.

The everyday round

Our first round, with the same ten models and settings. It turned out to be too easy to rank them, which is why we wrote the harder prompts.

Run on
7 November 2026 through Raq.com
Prompts
36 everyday tasks in six groups. 23 have answers that are checked automatically and 13 are marked by a judge model.
Tries
Every model answered every prompt twice, 720 answers in all.
Settings
Prompts sent as written with Raq.com's one-line instruction to use British English, at temperature 0 with each model's default reasoning setting and no web search, through OpenRouter with Zero Data Retention.
Judge
Kimi K3.1 Pro from Moonshot AI, a company with no model in the test. It scored each judged answer from 0 to 1 against a written marking scheme.
Cost
$3.29 in AI usage for the main run, $1.78 for the answers and $1.52 for the judge.

How the everyday round was marked

23 prompts have one right answer, such as a date, a number or a name. We asked for it on a final line starting "Answer:" and checked that line automatically, so each of those answers scored 1 or 0.

The other 13, mostly writing and summarising, have no single right answer. Kimi K3.1 Pro read each answer and scored it from 0 to 1 against a marking scheme written for that prompt, listing what to include and how much to take off for each problem, made-up details included.

459 of the 460 checked answers were right, so every gap between the models came from the judged prompts, where one 0.2 deduction on one answer moves a score by about 0.3 points.

Every prompt in the everyday round

Writing

Email to a landlord about a broken boiler Marked by the judge

Write an email to my landlord, Mr Cooper, telling him the boiler stopped working on Monday and we have had no heating or hot water since. Ask him to send an engineer this week. Keep it polite and under 120 words. Sign it from Kristin.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. The email must: say the boiler stopped working on Monday (-0.2 if missing); say there has been no heating or hot water since (-0.15 if missing); ask for an engineer to visit this week (-0.2 if missing or no timeframe); be polite and addressed to Mr Cooper (-0.1); be signed from Kristin (-0.1); be under 120 words, not counting a subject line (-0.3 if clearly over about 125 words). Deduct 0.2 if it invents facts the user did not give, such as an address, phone number, dates other than Monday, or earlier complaints; bracketed placeholders are fine. Minimum 0.

Rewrite a payment notice in plain English Marked by the judge

Rewrite this in plain English for a general reader, in no more than 50 words: "Pursuant to the terms of your agreement, any remittance received subsequent to the due date will incur a late payment charge of £12, which will be applied to your account forthwith. Should you anticipate difficulty in meeting the payment obligation, you are advised to contact our customer services team prior to the due date in order that alternative arrangements may be considered."

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. The rewrite must keep every fact: a payment that arrives after the due date costs £12 (-0.25 if the amount or condition is missing or wrong); the charge is added straight away (-0.1); if you might struggle to pay, contact customer services before the due date (-0.25); they may agree another arrangement (-0.1). It must use everyday words with no legal phrasing such as 'pursuant', 'remittance' or 'forthwith' (-0.2). It must be 50 words or fewer (-0.3 if clearly over about 55 words). Deduct 0.2 for any added fact or promise not in the original. Minimum 0.

Three-sentence product description Marked by the judge

Write a product description of exactly three sentences for a 750 ml stainless steel water bottle that keeps drinks cold for 24 hours and has a dishwasher-safe lid. Do not mention any feature I have not listed.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. It must be exactly three sentences (-0.3 if not). It must mention 750 ml, stainless steel, keeping drinks cold for 24 hours and the dishwasher-safe lid (-0.1 for each one missing). It must not mention any other feature or claim, such as keeping drinks hot, BPA-free, leak-proof, lightweight, colours, warranty, eco or plastic-free claims, or a price (-0.25 for each invented feature). Ordinary descriptive words such as 'durable' or 'sleek' are acceptable. Minimum 0.

Apology to a customer whose parcel was lost Marked by the judge

A customer called Tom emailed to say his order has not arrived. The courier lost it. Write a short reply apologising, saying a replacement was sent today by tracked next-day delivery, and giving him the code SORRY10 for 10% off his next order. Keep it under 100 words.

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. The reply must: apologise (-0.15); say or clearly imply the courier lost the parcel (-0.1); say a replacement was sent today (-0.15); say it is tracked next-day delivery (-0.15); give the code SORRY10 exactly (-0.2 if missing or altered); say it is 10% off his next order (-0.1); address Tom (-0.05); be under 100 words (-0.3 if clearly over about 105 words). Deduct 0.2 for invented details such as a tracking number, a refund, a delivery date or a named courier; bracketed placeholders for a sign-off are fine. Minimum 0.

A sentence of exactly twelve words Answer checked

Write one sentence about autumn that is exactly twelve words long. Reply with the sentence only.

Right answer: Any sentence of exactly 12 words. The final "Answer:" line had to contain the right answer.

Six-line acrostic poem Answer checked

Write a six-line poem about the city in which the first letters of the lines spell LONDON. Reply with the poem only.

Right answer: Six lines starting L, O, N, D, O, N. The final "Answer:" line had to contain the right answer.

Summarising

Council notice in three bullet points Marked by the judge

Summarise this council notice in exactly three bullet points for a resident who only wants to know what changes for them. "From Monday 4 January 2027, household waste collections in the Northfield ward will move from Tuesdays to Thursdays. Recycling will still be collected every other week, on the same day as general waste, and residents should put bins out by 7am. Garden waste collections will pause from 14 December 2026 and restart on 1 March 2027, and the annual garden waste permit will rise from £45 to £52 when it renews in April. Residents who receive an assisted collection do not need to reapply. Christmas trees will be collected on 9 and 16 January 2027 if they are left beside the general waste bin with decorations removed. Questions can be sent to the council's waste team through the online contact form."

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. It must be exactly three bullet points (-0.3 if not). It must say collections move from Tuesday to Thursday from 4 January 2027 (-0.3 if missing or the dates or days are wrong). It must cover the garden waste pause from 14 December 2026 to 1 March 2027 or the permit rising from £45 to £52 (-0.2 if neither appears). Deduct 0.3 for any wrong date, day or price, and 0.2 for any fact not in the notice. Choosing between the less important details (bins out by 7am, Christmas trees, assisted collections) is a matter of judgement and not penalised. Minimum 0.

Action items from meeting notes Marked by the judge

List the action items from these meeting notes, each with its owner and deadline. Leave out anything that is not an action. "Weekly ops call, Tuesday 3 November. Present: James, Kristin, Tom. James said the new warehouse rota starts on 17 November and he will send it to staff by Friday. Kristin raised the late invoices from Harbour Supplies, and Tom agreed to chase them before the end of the month. The team discussed moving the Christmas party to a weekday but did not decide. Kristin will book the van service for the week of 23 November. Tom said the website checkout bug is now fixed. Next call 10 November."

Marking scheme given to the judge: The correct action items are exactly three: James sends the new warehouse rota to staff by Friday (6 November); Tom chases the late Harbour Supplies invoices before the end of November; Kristin books the van service for the week of 23 November. Start at 1.0. Deduct 0.25 for each of the three that is missing or has the wrong owner, and 0.1 for each wrong or missing deadline. Deduct 0.2 for listing the Christmas party, the fixed checkout bug or the next call as an action item (stating separately that the party is undecided is fine). Deduct 0.2 for any invented owner or deadline. Minimum 0.

One-sentence summary of a short news item Marked by the judge

Summarise this in one sentence of no more than 25 words. "Riverside Library will open on Sundays from 10am to 4pm starting in February, after a six-month trial last year drew more than 3,000 visitors. To make room for the extra day, the library will close at 6pm instead of 8pm on Wednesdays. The council said the change costs nothing extra because staff hours are being moved rather than added. A children's reading session will run at 11am each Sunday."

Marking scheme given to the judge: Start at 1.0 and deduct for each problem. It must be one sentence (-0.3 if more). It must be 25 words or fewer (-0.3 if clearly over about 27 words). It must say the library will open on Sundays from February (-0.3 if missing). It should mention the trade-off of earlier Wednesday closing (-0.2 if missing). Deduct 0.3 for any wrong or invented fact. Minimum 0.

What two colleagues agreed in an email thread Marked by the judge

What did Kristin and Tom agree? Answer in no more than three short bullet points. Kristin: Hi Tom, can we move Thursday's supplier meeting to Friday? I'm at the dentist on Thursday morning. Tom: Friday morning is fine but I'm out from 1pm. How about 10am? Could we do it on video rather than at their office? Kristin: 10am works. I'd rather go in person this time, as they want to show us the new samples. Tom: Fair enough, in person then. I'll let them know. Can you bring the price list? Kristin: Will do.

Marking scheme given to the judge: The agreement is: the supplier meeting moves from Thursday to Friday at 10am; it is in person at the supplier's office, not on video; Tom tells the supplier and Kristin brings the price list. Start at 1.0. Deduct 0.3 if Friday at 10am is missing or wrong, 0.3 if it says video or leaves the format unclear, 0.15 if the two follow-up tasks are missing or given to the wrong person, and 0.2 for more than three bullet points. Deduct 0.3 for any invented detail. Minimum 0.

Answering only from the text given Marked by the judge

Based only on this notice, what time does the shop open on Sundays? "Opening hours: Monday to Friday 8am to 6pm, Saturday 9am to 5pm. Closed on bank holidays. Free parking behind the shop."

Marking scheme given to the judge: The notice gives no Sunday hours. Score 1.0 if the reply says the notice does not give Sunday opening hours (adding that the shop may be closed on Sundays is fine). Score 0.6 if it simply states the shop is closed on Sundays as a fact without saying the notice does not mention Sundays. Score 0 if it gives any Sunday opening time. Deduct 0.2 for any other invented detail. Minimum 0.

What changed between two versions of a policy Marked by the judge

What changed between version 1 and version 2 of this policy? Version 1: "Staff can work from home two days a week. Requests go to your line manager. Equipment is provided for home working." Version 2: "Staff can work from home three days a week. Requests go to your line manager and HR. Equipment is provided for home working. Tuesdays are office days for everyone."

Marking scheme given to the judge: There are exactly three changes: home working goes from two to three days a week; requests now also go to HR; and Tuesdays become office days for everyone. The equipment rule did not change. Start at 1.0. Deduct 0.3 for each change that is missing or wrong. Deduct 0.3 if it says the equipment rule changed or invents any other change. Mentioning that the equipment rule stayed the same is fine. Minimum 0.

Reasoning

Day of the week a year later Answer checked

1 March 2027 is a Monday. What day of the week is 1 March 2028? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: Wednesday. The final "Answer:" line had to contain the right answer.

Counting letters Answer checked

How many times does the letter r appear in the words 'strawberry raspberry'? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 6. The final "Answer:" line had to contain the right answer.

Angle between clock hands Answer checked

What is the smaller angle, in degrees, between the hour hand and the minute hand of a clock at 3:15? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 7.5 degrees. The final "Answer:" line had to contain the right answer.

Discount then VAT Answer checked

A coat costs £80 before VAT. The shop takes 25% off that price, then adds 20% VAT to the reduced price. What is the final price? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £72. The final "Answer:" line had to contain the right answer.

Counting multiples Answer checked

How many positive whole numbers below 1,000 are divisible by 7 but not by 11? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 130. The final "Answer:" line had to contain the right answer.

Reversing a word letter by letter Answer checked

Write the word 'deliberately' with its letters in reverse order. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: yletarebiled. The final "Answer:" line had to contain the right answer.

Coding

Predict Python output (mutable default argument) Answer checked

What does this Python code print? ```python def add(item, items=[]): items.append(item) return items a = add(1) b = add(2, []) c = add(3) print(a, b, c) ``` End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: [1, 3] [2] [1, 3]. The final "Answer:" line had to contain the right answer.

Predict JavaScript output (default sort) Answer checked

What does this JavaScript print? ```javascript const nums = [10, 1, 5, 25]; nums.sort(); console.log(nums.join(",")); ``` End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 1,10,25,5. The final "Answer:" line had to contain the right answer.

Write a leap year function Marked by the judge

Write a Python function is_leap_year(year) that returns True if the year is a leap year in the Gregorian calendar and False otherwise. Reply with the code only.

Marking scheme given to the judge: Check the logic by hand for 1900 (False), 2000 (True), 2024 (True) and 2023 (False). Score 0 if any of these is wrong. Otherwise start at 1.0: deduct 0.2 if the reply has more than a sentence or two of prose around the code, since the user asked for code only (a fenced code block, a docstring or brief comments are fine); deduct 0.1 if the function is not named is_leap_year or does not take year as its argument. Using calendar.isleap is acceptable. Minimum 0.

Write a SQL query for top customers Marked by the judge

A PostgreSQL table orders has columns id, customer_id, total (numeric) and created_at (timestamp). Write a query that returns the three customers who spent the most in 2026, with each customer's total spend, highest first.

Marking scheme given to the judge: A correct query sums total per customer_id, keeps only orders created in 2026, orders by that sum descending and limits to 3. Start at 1.0. Score 0.3 or lower if it does not group and sum per customer or does not limit to three. Deduct 0.3 if the 2026 filter is wrong or missing. A filter of created_at BETWEEN '2026-01-01' AND '2026-12-31' misses orders placed after midnight on 31 December: deduct 0.25 for that. Accept created_at >= '2026-01-01' AND created_at < '2027-01-01', EXTRACT(YEAR FROM created_at) = 2026 or date_trunc equivalents. Do not penalise reasonable explanation, aliases or handling of ties. Minimum 0.

Predict Python output (list aliasing) Answer checked

What does this Python code print? ```python x = [1, 2, 3] y = x x = x + [4] y += [5] print(x, y) ``` End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: [1, 2, 3, 4] [1, 2, 3, 5]. The final "Answer:" line had to contain the right answer.

Extract booking details as JSON Answer checked

Extract the booking from this message as JSON with the keys name (string), date (YYYY-MM-DD), guests (integer) and dietary (array of strings). Reply with the JSON only. "Hi, it's Kristin Dawson. Could I book a table for six on Friday 13 November 2026? Two of us are vegetarian and one is gluten free."

Right answer: {"name": "Kristin Dawson", "date": "2026-11-13", "guests": 6, "dietary": ["vegetarian", "gluten free"]}. The reply had to be JSON with only these four keys, the same name, date and number of guests, and dietary entries for vegetarian and gluten free.

Facts

Element with atomic number 74 Answer checked

Which chemical element has atomic number 74? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: Tungsten. The final "Answer:" line had to contain the right answer.

Author of The Remains of the Day Answer checked

Who wrote the novel The Remains of the Day? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: Kazuo Ishiguro. The final "Answer:" line had to contain the right answer.

Country with the most time zones Answer checked

Which country has the most time zones, counting its overseas territories? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: France. The final "Answer:" line had to contain the right answer.

Letter missing from US state names Answer checked

Which letter of the alphabet does not appear in the name of any US state? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: Q. The final "Answer:" line had to contain the right answer.

A novel that does not exist Marked by the judge

Give me a two-sentence summary of Kazuo Ishiguro's 1994 novel The Glass Orchard.

Marking scheme given to the judge: Kazuo Ishiguro has written no novel called The Glass Orchard, and published no novel in 1994. Score 1.0 if the reply says it does not know of or cannot find such a novel by Ishiguro (suggesting his real books, such as The Unconsoled from 1995, is fine). Score 0.3 if it hedges but still offers a plausible plot as if the book might exist. Score 0 if it summarises a plot as though the book is real. Minimum 0.

Number of London Underground lines Answer checked

How many lines does the London Underground have? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 11. The final "Answer:" line had to contain the right answer.

UK questions

Self Assessment online filing deadline Answer checked

What is the deadline for filing a UK Self Assessment tax return online for the 2025 to 2026 tax year? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 31 January 2027. The final "Answer:" line had to contain the right answer.

Personal allowance in tax code 1257L Answer checked

What tax-free personal allowance does the UK tax code 1257L give? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £12,570. The final "Answer:" line had to contain the right answer.

Time limit to register a birth in England Answer checked

How many days do parents have to register a baby's birth in England? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 42 days. The final "Answer:" line had to contain the right answer.

Statutory annual leave for a five-day week Answer checked

What is the minimum paid annual leave, in days, for a full-time UK worker who works five days a week? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 28 days. The final "Answer:" line had to contain the right answer.

Highway Code stopping distance at 70 mph Answer checked

According to the Highway Code, what is the typical overall stopping distance for a car travelling at 70 mph, in metres? End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: 96 metres. The final "Answer:" line had to contain the right answer.

Income tax on a £30,000 salary Answer checked

Someone in England earns £30,000 a year with the tax code 1257L. Using 2025 to 2026 rates, how much income tax do they pay for the year? Ignore National Insurance. End your reply with a final line in the form 'Answer: <your answer>'.

Right answer: £3,486. The final "Answer:" line had to contain the right answer.

What these tests don't tell you

  • Close scores are ties

    Each of the 72 answers is worth about 1.4 points, so a one-point gap is less than one answer. The top five models in the harder round, from 98.2% to 99.7%, are about one answer apart, and another set of prompts could put them in a different order.

  • One judge

    Another judge model, or a person, would mark some writing differently. We picked a judge from a company with no model in the test so it had no reason to favour its own.

  • No tools or conversation

    No web search, files, images or follow-up messages. The models could not run code or use a calculator, so the maths and coding answers came from the model alone.

  • Default settings

    Each model ran at temperature 0 with its default reasoning setting. Higher reasoning levels or a system prompt of your own could change the order.

  • Two days

    Companies update models without notice. These results describe the models as they answered on 7 and 8 November 2026.

  • Our prompts, not yours

    How a model does on our 72 prompts says little about the questions you ask at work. That needs your own prompts.

Check our working

Every answer from both runs, with its score and the judge's reason or the tests it failed, is on a public page in Raq.com.

Find out which model does your work best

Sign up through Raq.com and a benchmark opens ready for your first prompt. Add a few, pick the models you're weighing up and start a run.

Start a benchmark