5. Dataset and subsets:

 

Training Set:

·       Training set is used for learning the parameters of the model.

·       Training set has an optimistic bias.

·       The training set is usually the biggest one of three sets and used to build the model.

Test Set:

·       Test set is used to get a final, unbiased estimate of how well the learning method works.

·       We expect that this estimate to be worse than training/validation set.

·       Test set is not biased and this distinguishes test set from training set.

·       Similar to Training set, Test set is a finite sample and bound to have variance due to sample size, but the test set doesn’t have an optimistic or pessimistic bias.

·       The test set doesn’t effect the outcome of learning process which only uses training set.

Validation Set:

·       Validation set is not used for learning but is for avoiding overfitting.

·       The idea of ‘Validation Set’ is identical to ‘Test Set’ but there is a difference.

·       We remove a subset from data and this subset is not used for training.

·       Although validation set is not directly used for training, it will be used in making certain choices in the learning process. As the set affects the learning process, it is no longer a test set.

·       The more we use validation set to fine tune the model, the more validation set becomes like a training set.

·       The validation and test sets are roughly the same sizes, much smaller than the size of the training set. The learning algorithm cannot use examples from these two subsets to build the model.

·       Validation set is used for selecting the model complexity.

In the past, the rule of thumb was to use 70% of the dataset for training, 15% for validation and 15% for testing. However, in the age of big data, datasets often have millions of examples. In such cases, it could be reasonable to keep 95% for training and 2.5%/2.5% for validation/testing.