Your AI Passed the Exam Because It Saw the Answer Sheet

Suppose I give you 1,000 maths questions. Then I give you an exam containing 200 of those exact same questions. You score 98%. Did you learn maths — or did you see the answer sheet?

Machine-learning teams accidentally build versions of this problem all the time. The resulting model looks incredible in development, clears every dashboard metric and then collapses when real users arrive.

Training data is the textbook

The training dataset is the information from which a model learns patterns. Take a simple helmet detector. You show it thousands of labelled images of people wearing helmets and not wearing helmets. During training, the model adjusts its internal parameters to become better at distinguishing the categories.

If almost every helmet in the dataset is black, the model may accidentally treat “black object around head” as a useful shortcut. Then somebody arrives wearing a bright yellow helmet and the real world exposes what your dataset forgot.

Testing data is the exam

You therefore keep some examples away from training. The model does not learn from them. You use those examples later to ask whether the system works on data it has not already seen.

Training teaches. Validation helps you make development decisions. Testing gives you an independent check.

The sneakiest problem is data leakage

Imagine you are building a model that predicts whether a customer will cancel a subscription. Someone accidentally includes a field called cancellation_processed_at.

The model becomes extraordinarily accurate. Of course it does: you gave it information that only exists after the event you are trying to predict.

“99% accuracy” should never end a technical conversation. The next question is: on what data?

More data is not automatically better data

One million poor examples can be less useful than 50,000 representative ones. Train the helmet detector almost entirely on daytime footage and deploy it on highway cameras at night. Lighting changes. Camera angles change. Weather changes. Resolution changes.

The model did not suddenly become stupid. You changed the world around it.

Training data is not the same as your company documents

Businesses often tell us, “We want to train ChatGPT on our documents.” Most of the time, they actually mean: we want the AI to answer using our information.

Those are different engineering problems. If you have policies, product documents, menus, SOPs or internal manuals, you often do not need to retrain a foundation model. You can retrieve the relevant information at request time and give it to the model as context.

Fine-tuning still needs an exam

Fine-tuning can be useful for consistent formatting, specialised classification or domain-specific behaviour. But the school analogy still applies: you need examples the model learns from and genuinely separate examples for evaluation.

If your team keeps tuning against the same test set, that test set slowly stops being independent. Eventually you need a fresh exam.

The number we care about most

When someone shows us an AI model with impressive performance, the first question is not always accuracy. We want to understand the gap between development conditions and production conditions.

• Where did the data come from?
• How old is it?
• Which users or environments are underrepresented?
• What cases are missing?
• What will change after deployment?
• How will you detect when production inputs stop looking like training inputs?

Your model learns the world you show it. Production introduces it to the world you forgot.

References & further reading

• Google ML Crash Course — Dividing Datasets
• Google ML Crash Course — Production Monitoring