Evaluating answers

In plain English

Evaluating an answer means checking whether it is true, not whether it sounds good. For code, run it and add an edge case the happy path would miss, such as an empty input or an error response. For factual claims, open the source yourself rather than trusting the citation, because a hallucination can include a made-up reference. When you have two candidate answers, compare them against the same tests and pick the one you can explain, not the longer or more confident one. Reject output whenever you cannot verify it, when it invents an API or function, or when it changes requirements you did not state.

Step by step

1. 1Ask the model for the assumptions and edge cases it considered.
2. 2Run the generated code with one input it never mentioned.
3. 3Open any cited source and confirm it exists and says what was claimed.
4. 4Generate a second answer to the same prompt and compare both.
5. 5Write down the one reason you accepted or rejected the output.

BEFORE YOU ACCEPT AN ANSWER
1. Run the code and add one edge case it never mentioned.
2. Open every cited source and confirm it says what was claimed.
3. Get a second answer to the same prompt and compare both.
4. Ask: can I explain this line by line?

REJECT WHEN
- it invents an API, function or flag that does not exist
- it changes a requirement you never stated
- you cannot reproduce or verify the result

Common mistakes

• Accepting code because it compiles is wrong, since syntax can be valid while the logic is broken.
• Trusting a citation without opening it lets a hallucinated reference slip through.
• Choosing the longer answer as the better one is a bad rule, because extra text can hide errors.
• Keeping output you cannot explain means you now own code you cannot fix when it fails.