AnnotationPrep

Math & Chain-of-Thought Verification Practice Test

Auditing step-by-step AI mathematical proofs, spotting invalid algebraic leaps, solving combinatorics and probability problems, and deductive logic.

12 questions · 2 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Practise by topic

Sample scenarios

An AI solves: 'If 3x - 7 = 2(x + 5), find x.'

Step 1: 3x - 7 = 2x + 10
Step 2: 3x - 2x = 10 - 7
Step 3: x = 3

Where is the first error in the AI's chain of thought?

  1. Step 1: 2(x + 5) is 2x + 5.
  2. Step 2: adding 7 to both sides gives 10 + 7 = 17, not 10 - 7.
  3. Step 3: 3x - 2x is 5x.
  4. There is no error.
Show the answer

B. Step 2: adding 7 to both sides gives 10 + 7 = 17, not 10 - 7.

To move `-7` from the left side to the right side, you add `7` to both sides: `x = 10 + 7 = 17`. Checking `3(17) - 7 = 44` and `2(17 + 5) = 44` confirms `x = 17`.

Principle: CoT Math Verification: Sign Errors in Transposition

Premise: 'If it rains, the grass gets wet.'
Observation: 'The grass is wet.'

An AI concludes: 'Therefore, it definitely rained.'
What logical fallacy did the AI commit?

  1. Modus ponens.
  2. Affirming the consequent—a sprinkler or morning dew could also wet the grass.
  3. Denying the antecedent.
  4. Circular reasoning.
Show the answer

B. Affirming the consequent—a sprinkler or morning dew could also wet the grass.

From `P -> Q` and `Q`, concluding `P` is the formal fallacy of affirming the consequent. Only `P -> Q` and `not Q` validly implies `not P` (modus tollens).

Principle: Formal Logic: Affirming the Consequent

An AI evaluates `(a + b)^2` for `a = 3, b = 4`:
Step 1: `(a + b)^2 = a^2 + b^2`
Step 2: `3^2 + 4^2 = 9 + 16 = 25`

What classic algebraic fallacy did the AI commit?

  1. Division by zero.
  2. Freshman's dream fallacy: omitting the cross term `2ab` (`(3 + 4)^2 = 49`, not `25`).
  3. Sign inversion.
  4. Rounding error.
Show the answer

B. Freshman's dream fallacy: omitting the cross term `2ab` (`(3 + 4)^2 = 49`, not `25`).

`(a + b)^2 = a^2 + 2ab + b^2`. With `a=3, b=4`, the true value is `7^2 = 49`, not `25`.

Principle: CoT Math Verification: Binomial Expansion

A fair coin is flipped 3 times. An AI claims the probability of getting AT LEAST ONE heads is `3/8`.
What is the actual probability?

  1. `1/2`
  2. `7/8` (since `1 - P(all tails) = 1 - (1/2)^3 = 1 - 1/8 = 7/8`).
  3. `3/8`
  4. `5/8`
Show the answer

B. `7/8` (since `1 - P(all tails) = 1 - (1/2)^3 = 1 - 1/8 = 7/8`).

The AI confused 'exactly one heads' (`3/8`) with 'at least one heads' (`1 - 1/8 = 7/8`).

Principle: Probability: Complement Rule

An AI solves: 'A store raises a $100 jacket by 20%, then discounts the new price by 20%. What is the final price?'

AI says: 'Raising by 20% and lowering by 20% cancel out, so the final price is $100.'

What is the actual final price, and why did the AI fail?

  1. $100—the AI is right.
  2. $96: 100 * 1.20 = $120, and 20% off $120 is $24, leaving $96.
  3. $104.
  4. $98.
Show the answer

B. $96: 100 * 1.20 = $120, and 20% off $120 is $24, leaving $96.

Percentage changes apply to different bases: `+20%` of `$100` is `+$20` (`$120`), while `-20%` of `$120` is `-$24`, leaving `$96`.

Principle: Quantitative Reasoning: Asymmetric Percentage Base

Assessments this helps with

AssessmentPlatformAvailability
DataAnnotation Math & STEM Qualification Assessment readyDataAnnotation.techRemote / Global
Outlier AI Math, Physics & Chain-of-Thought Assessment readyOutlier AIRemote / Global
Alignerr Mathematics & Analytical Logic Test readyAlignerr (Labelbox)Remote / Global
Mercor STEM & Quantitative Reasoning Evaluation readyMercorRemote / Global