Overfitting & Honest Evaluation
This is the lesson that separates people who use ML from people who just ran a tutorial. Overfitting is when a model scores brilliantly on training data and poorly on new data. It memorised instead of generalising.
Spot it in one line
Compare train accuracy vs test accuracy:
- Deep tree: train ~1.00, test lower → the gap is the overfitting.
- Small tree: train and test close together → it generalises.
A model that's perfect on training and mediocre on test hasn't learned the pattern; it's learned the noise. When you see a big train-test gap, simplify (shallower tree, fewer features, more regularisation) or get more data.
Read a confusion matrix
Accuracy is one number; a confusion matrix shows where the mistakes are. Row = true class, column = predicted class. The diagonal is correct; everything off-diagonal is an error you can name: "it confuses versicolor for virginica twice." That's the difference between "78% accurate" and knowing which cases to fix.
The workflow, start to finish
- Split train/test. 2. Fit on train. 3. Predict on test. 4. Compare train vs test to check for overfitting. 5. Read the confusion matrix to find which errors. 6. Simplify or add data. Repeat.
Try it
- Add
max_depth=1— does it now underfit (train accuracy drops too)? - In the confusion matrix, which two species does the model mix up? Why might that be (hint: petal sizes)?
3 graded exercises
Write the code, click Check. We’ll tell you exactly what to fix.
- Exercise 1
Write
gap(model, X_tr, y_tr, X_te, y_te)returningtrain_accuracy - test_accuracy(the overfitting gap). - Exercise 2
Write
errors_off_diagonal(y_true, y_pred)returning the number of misclassified samples, computed from the confusion matrix. - Exercise 3
A model scores 1.00 on training and 0.72 on test. Set
diagnosisto the single word"overfitting".