James Seegel · MIS 752 · Lab 7
Catch the AI Lying
Ten gold questions, two scoring methods, and an audit of the judge.
Accuracy
| Scorer | Passed | Accuracy |
|---|---|---|
| Exact match | 6/10 | 60% |
| Judge pass A | 7/10 | 70% |
| Judge pass B | 7/10 | 70% |
Judge reliability
| comparison | n | agreement | disagreements | kappa |
|---|---|---|---|---|
| exact match vs judge (pass A) | 10 | 0.9 | 1 | 0.7826 |
| judge pass A vs judge pass B | 10 | 1.0 | 0 | 1.0000 |
Mean absolute score difference between judge passes: 0.00 points across 10/10 paired scores.
Pass/fail results
| Item | Gold answer | Model answer | Exact | Judge A | Score A | Judge B | Score B |
|---|---|---|---|---|---|---|---|
| icd10_t2dm | E11.9 | E11.9 | PASS | PASS | 10 | PASS | 10 |
| icd10_ami | I21.9 | I20.0 | FAIL | FAIL | 0 | FAIL | 0 |
| snomed_t2dm | 44054006 | 40298007 | FAIL | FAIL | 0 | FAIL | 0 |
| acetaminophen_max_g | 4 | 4 | PASS | PASS | 10 | PASS | 10 |
| a1c_8_eag_mgdl | 183 | 183 | PASS | PASS | 10 | PASS | 10 |
| hypo_level1_mgdl | 70 | 70 | PASS | PASS | 10 | PASS | 10 |
| hipaa_safe_harbor | 18 | 6 | FAIL | FAIL | 0 | FAIL | 0 |
| warfarin_reversal_agent | 4-factor prothrombin complex concentrate | Prothrombin complex concentrate (PCC) | FAIL | PASS | 10 | PASS | 10 |
| sjw_warfarin_yesno | NO | NO | PASS | PASS | 10 | PASS | 10 |
| atorva80_intensity | high | high | PASS | PASS | 10 | PASS | 10 |
Takeaways
I evaluated ten gold question-and-answer pairs. Exact match scored 6/10 (60%), while both judge passes scored 7/10 (70%). I trust the judge on the PCC answer because the notebook accepts both “PCC” and “prothrombin complex concentrate.” Adding the abbreviation in parentheses changed the text without changing its meaning under this rubric.
The judge agreed with itself on 10/10 items (100%), with zero contradictions, Cohen’s kappa of 1.0, and a mean absolute score difference of 0.0. One initial response was missing after reaching the 2,000-token limit. Both passes for that item were retried at an 8,000-token limit and returned PASS. The missing response was not a changed verdict.
Before relying on this evaluation, I would add more expert-reviewed questions, unit conversions, alternative wording, ambiguous cases, and plausible wrong answers.
How the results were measured
The supplied gold set contains 10 questions. The answer model was liquid/lfm-2.5-2.6b:free at temperature 0.0. The judge was apodex/apodex-1.1-mini:free, with pass A at 0.7 and pass B at 1.1. This report was rebuilt from saved responses without new model requests. Replay does not add another reliability trial. One changed verdict moves accuracy by 10 percentage points. Agreement on these ten items does not establish broader reliability. Optional Lab 4 rescoring was not performed.
Recovery note
The initial PCC response in judge pass B returned no verdict at the 2,000-token limit. Both passes for that item were retried at 8,000 tokens. Other items used 2,000 tokens. The missing response was not treated as a changed verdict. Earlier failed Nvidia attempts are excluded from the final Apodex evaluation.
AI Assistance Disclosure
I worked through this lab in Google Colab, ran the code and self-checks, reviewed the outputs, and reran the completed notebook to verify the saved results. I used ChatGPT/Codex to help develop and debug code, troubleshoot API errors, interpret outputs, and draft and revise the written responses. The reported findings came from the recorded model answers and judge responses, including the documented retry.
Recorded evidence
gold_qa.csv · lab07_agreement.csv · lab07_answers.csv · lab07_confusion.csv · lab07_judge_runs.csv · lab07_verdicts.csv