Out of sample

Out of sample is the part of the data that was set aside before a strategy was built and is only looked at once, at the end, to judge it. The years where the rule was shaped are in sample: whatever was tuned, chosen or discarded was chosen against those years. Picture a student who wrote the exam and then took it. The in-sample score is that exam. The out-of-sample score is a different exam, sealed until the day.

It matters because fitting is easy and forecasting is hard. Give any rule enough knobs and it will explain the past it was shown. The only evidence that it found something real is what it does on the past it was not shown. A rule that earns in sample and nothing out of sample did not find an edge; it memorized a stretch of history.

In the logbook, the out-of-sample result is the number that decides. It is quoted first, its bar is written before the run, and no later adjustment is allowed to touch it. Once the sealed exam has been opened, it cannot be sealed again, which is why each idea gets one look.