Short answer: An AI project is not evaluated by a single impressive example. Use representative cases, define what counts as a correct outcome, record failures, and explain the trade-offs among quality, reliability, cost, and safety.
Create a representative test set
Include ordinary cases, edge cases, ambiguous inputs, unsupported requests, and expected failures. Keep the dataset small enough to review manually and large enough to reveal repeated problems.
Measure the right outcome
A support assistant may be judged by grounded answers and correct escalation. A classifier may be judged by precision and recall. A user-facing tool also needs latency, clarity, and recovery from errors.
Publish limitations
State where the system is likely to fail, which inputs it does not support, what data it uses, and when a person should take over. Honest limits make a portfolio more credible.
Build a small scorecard
Create a spreadsheet or JSON test file with an ID, input, expected behaviour, actual output, pass/fail, error category, and reviewer note. Choose measures that match the task: exactness for extraction, precision and recall for classification, grounded citations for RAG, or successful completion and safe handoff for an agent.
Run the same set after meaningful changes. A project can improve one metric while becoming slower, more expensive, or less safe. Keeping the scorecard makes these trade-offs visible and stops you from selecting only the best screenshots.
Treat failures as first-class results
Group failures into categories: data problem, retrieval miss, model misunderstanding, tool error, unclear interface, unsafe action, or unsupported request. Fixing one category at a time is more useful than randomly changing prompts. Report recurring failures in the README.
Safety evaluation does not need to be dramatic. Test whether the project protects secrets, avoids unsupported claims, handles missing inputs, respects its own scope, and tells the user when a person should take over. These are practical habits for every AI system.
Evaluation turns a demonstration into evidence
A demonstration proves that a system can work once. Evaluation asks whether it works across the kinds of inputs a real user will provide. That difference is especially important for AI projects, where a fluent response can hide an unsupported answer, a missed edge case, or an unsafe action. A small, carefully reviewed test set is far better than no evidence at all.
Keep the evaluation honest. Record failures, explain the scoring method, and avoid changing the test cases just to make a new version look better. If the project is a portfolio piece, those imperfect results can become the most interesting part of your explanation: they show how you recognised a problem and what you would improve next.
Run an evaluation that tells you something useful
Take a small collection of representative cases and review them one by one. For a document extractor, compare every required field with the source. For a RAG assistant, verify that the cited text supports the answer. For an agent, verify that it selected the permitted action or escalated at the right time. The goal is not to create a grand benchmark; it is to make errors visible enough to improve.
When a result fails, write down why before changing the system. Was the source missing? Was the input ambiguous? Did retrieval select the wrong passage? Did the model ignore a clear instruction? These categories guide better fixes than randomly rewriting prompts. They also give you an honest limitations section for the project report or README.
Repeat the same cases after significant changes. If a new version is more accurate but takes too long to respond, costs too much, or fails more badly on unsupported inputs, the trade-off should be part of the result. Good evaluation helps you make those trade-offs consciously.