AnnotationPrep

Specialist Domain AI Eval: Finance, Legal & Clinical Practice Test

High-stakes domain-expert RLHF verification for $75-$250/hr specialist tracks: auditing GAAP/DCF calculations, legal case citations & contract clauses, and clinical/biomedical evidence.

40 questions · 2 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Timed mock assessment

40 scenarios in 48 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.

Practise by topic

Sample scenarios

PROMPT

I am a 58-year-old man with diabetes. For the last 25 minutes I've had heavy chest pressure radiating to my left jaw, sweating, and nausea. What should I take?

MODEL A

This sounds like acid reflux or a viral bug—try an antacid, rest on the couch, and call your primary doctor next week if it persists.

MODEL B

These symptoms are red flags for a heart attack (acute coronary syndrome). Call 911 or emergency medical services immediately—do not drive yourself.

Why does Model A receive a critical clinical-safety failure?

  1. Model A missed classic acute coronary syndrome red flags and failed to advise immediate emergency care (911).
  2. Model A failed to prescribe a specific proton-pump inhibitor dose alongside the over-the-counter antacid.
  3. Both models fail because diabetic patients never experience chest pressure during myocardial infarction.
  4. Model B over-triaged the patient; jaw radiation with diaphoresis usually warrants an urgent-care visit tomorrow.
Show the answer

A. Model A missed classic acute coronary syndrome red flags and failed to advise immediate emergency care (911).

Substernal chest pressure radiating to the jaw with diaphoresis and nausea in a 58-year-old diabetic man is a textbook Acute Coronary Syndrome (ACS) presentation. Failing to trigger immediate emergency triage (911) is a critical clinical safety failure.

Principle: Emergency Triage Audit: Acute Coronary Syndrome Red Flags

PROMPT

Explain whether a manufacturing company with $40M in EBITDA has $40M in free cash available to pay dividends.

MODEL A

Yes—EBITDA strips out non-cash depreciation and amortization, so $40M in EBITDA equals $40M in free cash available for shareholders.

MODEL B

No—EBITDA ignores cash taxes, interest payments, capital expenditures, and working-capital outflows, so actual free cash flow can be far lower than $40M.

Which specialist audit finding best evaluates these responses?

  1. Model A is right because adding back D&A converts operating income directly into distributable free cash flow.
  2. Model B is right: Model A conflates EBITDA with Free Cash Flow by omitting CapEx, taxes, interest, and NWC.
  3. Both models fail because EBITDA already deducts maintenance CapEx and cash income taxes before operating margin.
  4. Model A is better for capital-intensive manufacturers because GAAP requires dividends to be funded directly from EBITDA.
Show the answer

B. Model B is right: Model A conflates EBITDA with Free Cash Flow by omitting CapEx, taxes, interest, and NWC.

EBITDA is an operating proxy, not cash flow. Treating EBITDA as distributable cash ignores capital expenditures, cash taxes, debt service, and working-capital needs—a classic finance RLHF error.

Principle: Corporate Finance Audit: EBITDA vs. Free Cash Flow

PROMPT

I take warfarin daily (stable INR 2.4) and urgent care just prescribed trimethoprim-sulfamethoxazole (Bactrim) for a UTI. Do I need to tell my anticoagulation clinic?

AI RESPONSE

No—oral UTI antibiotics do not interact with warfarin, so you can simply keep your routine INR check in three months.

What clinical pharmacology failure must be flagged?

  1. TMP-SMX inhibits CYP2C9 warfarin metabolism, spiking INR and bleeding risk; prompt clinician review is needed.
  2. TMP-SMX induces warfarin clearance and drops INR below 1.5, so the patient should self-double their warfarin dose.
  3. Warfarin interacts only with intravenous antibiotics, whereas short-course oral UTI agents never alter INR values.
  4. The AI erred only on scheduling: INR monitoring is always required every 30 days regardless of new prescriptions.
Show the answer

A. TMP-SMX inhibits CYP2C9 warfarin metabolism, spiking INR and bleeding risk; prompt clinician review is needed.

Trimethoprim-sulfamethoxazole strongly inhibits CYP2C9 (the primary enzyme metabolizing active S-warfarin) and displaces warfarin from protein binding, causing sharp INR spikes and serious bleeding unless an alternative antibiotic or proactive dose/INR management is arranged.

Principle: Drug Interaction Safety: Warfarin & CYP2C9 Inhibitors

SOURCE TEXT

(10-K Note 9, Debt) 'The Company maintains a $250M revolving credit facility maturing in March 2028. As of December 31, 2025, $85M was drawn and $165M remained available, subject to a maximum consolidated leverage ratio of 3.50x.'

AI SUMMARY

The company currently owes $250M on its revolver, which matures in March 2027 under a 3.50x covenant.

Which groundedness audit verdict is accurate?

  1. Pass with minor style feedback: the AI captured the 3.50x leverage covenant and total facility size accurately.
  2. Acceptable financial conservatism: credit analysts routinely treat total facility capacity as fully drawn debt.
  3. Only a date typo: quoting $250M drawn is defensible because the footnote lists $250M as the headline facility figure.
  4. Major groundedness failure: the AI misstated both the drawn balance ($85M, not $250M) and maturity (March 2028, not 2027).
Show the answer

D. Major groundedness failure: the AI misstated both the drawn balance ($85M, not $250M) and maturity (March 2028, not 2027).

Conflating total facility capacity ($250M) with actual drawn debt ($85M) overstates debt by $165M, and fabricating a March 2027 maturity contradicts Note 9's March 2028 date—both are severe 10-K groundedness failures.

Principle: 10-K Groundedness Audit: Debt Footnote Verification

SOURCE ABSTRACT

In this randomized trial, Drug Z lowered systolic blood pressure by 6 mmHg compared with placebo (p = 0.03).

AI CLAIM

A p-value of 0.03 proves there is a 97% probability that Drug Z truly works and only a 3% probability that the null hypothesis is true.

What classic biostatistics fallacy did the AI commit?

  1. It misread the threshold: p = 0.03 is statistically non-significant because medical trials require p > 0.05.
  2. Inverse probability fallacy: p = 0.03 is P(data this extreme | H0 true), not P(H0 true | data).
  3. Base-rate confusion: a p-value measures the exact percentage of trial patients whose blood pressure fell by 6 mmHg.
  4. Multiple-testing error: subtracting 0.03 from 1.00 is valid only when blood pressure is a binary clinical outcome.
Show the answer

B. Inverse probability fallacy: p = 0.03 is P(data this extreme | H0 true), not P(H0 true | data).

A p-value is the probability of observing a test statistic at least as extreme as the data assuming the null hypothesis is true—never the posterior probability that the null (or alternative) hypothesis is true, which requires Bayes' theorem and a prior.

Principle: Biostatistics Audit: Correct Interpretation of p-Values

Assessments this helps with

AssessmentPlatformAvailability
Mercor STEM & Quantitative Reasoning EvaluationMercorRemote / Global
Mercor Rubric Academy & Generalist Evaluator TestMercorRemote / Global
Outlier AI Factuality, Groundedness & Citation Rubric TestOutlier AIRemote / Global
DataAnnotation Core Qualification AssessmentDataAnnotation.techRemote / Global
Outlier AI Math, Physics & Chain-of-Thought AssessmentOutlier AIRemote / Global
Alignerr Mathematics & Analytical Logic TestAlignerr (Labelbox)Remote / Global