What do we call the error that occurs when a model performs poorly on unseen data?

The term used to describe the error that occurs when a model performs poorly on unseen data is generalization error. This concept refers to the difference between the model's performance on the training data versus its performance on new, unseen datasets. When a model generalizes well, it successfully captures the underlying patterns in the training data and applies them to make accurate predictions on unseen data.

In contrast, if a model exhibits high generalization error, it indicates that it has not learned to generalize effectively and has likely memorized the training data instead of learning from it. This situation can arise particularly when a model is too complex and has been trained in such a way that it successfully minimizes training error but fails to perform well on data it hasn't encountered before.

The other terms mentioned have distinct meanings and contexts. Training error refers to the error observed on the training dataset, validation error pertains to the model’s performance during the validation phase using a separate set of data to tune hyperparameters, and overfitting error reflects a situation where a model learns noise and details in the training data to an extent that it negatively impacts its performance on new data. However, it is the generalization error that specifically encapsulates the issue of poor performance on unseen data.