Why a Random Forest Beats a Genius: Ensemble Methods for Indian Equities

After three posts of careful preparation — dollar bars for honest sampling, triple-barrier labels that respect the path, sample weights that account for overlap, fractional differentiation for memory — we finally get to put a model on top. Chapter 6 is about which kind of model, and it delivers two counter-intuitive verdicts: the clever method (boosting) is usually the wrong one for markets, and the dumb-sounding method (averaging hundreds of mediocre trees) is usually right. This post explains why, with numbers.

Where prediction error comes from

Every model's error is three things. Bias is being systematically wrong — too simple to see the pattern, like an archer who clusters tightly but always to the left. Variance is being unstable — retrain on slightly different data and you get a very different model, like an archer scattering shots all around the target. Noise is irreducible — tomorrow's NIFTY move is not fully written in today's tape, and no model erases that.

Ensembles are two distinct tricks. Bagging attacks variance. Boosting attacks bias. Picking the right one means knowing which problem you actually have — and in Indian equities, as in all markets, the dominant problem is noise, which changes everything.

Bagging: the wisdom of independent crowds

Bagging trains many copies of a model, each on a different random resample of the data, and averages them. The intuition is the wisdom of crowds: one analyst is noisy, but the average of many cancels the individual noise.

The fine print that López de Prado underlines: averaging only helps to the degree the models are independent. If every analyst on your desk reads the same three brokerage notes, averaging them tells you nothing new — their errors are correlated and don't cancel. A random forest adds a second trick to force independence: at each split, each tree may only consider a random subset of features — like making every analyst decide while blindfolded to a random handful of the indicators. That forced disagreement is what makes the average robust.

Verdict 1: in low signal-to-noise, bagging beats boosting

Boosting works differently. It trains models in sequence, each one focusing on the examples the previous models got wrong — relentlessly chasing its own mistakes. That makes it brilliant at reducing bias in clean problems. But in markets the "hardest, most-often-misclassified" examples are usually noise, not signal — and boosting will faithfully contort itself to fit that noise.

We measured it. Take a 4,000-sample classification task, then progressively corrupt the training labels with noise (flipping a rising fraction of them) to simulate the falling signal-to-noise of real market labels. Random forest (bagging) vs gradient boosting:

On clean data boosting is marginally ahead — that's its bias-fighting superpower. But the instant noise enters, the lines diverge, and by 40% corrupted labels bagging leads by a full 8 percentage points. This is the chapter's core finance argument in one table: the noisier the labels, the more boosting overfits, and Indian-equity labels are extremely noisy. A triple-barrier label on a NIFTY smallcap is a coin-flip dressed up as a classification target far more often than we'd like — exactly the regime where the right column collapses.

The practical reading for the NIFTY 500: reach for a bagged random forest as the default, not XGBoost. (Add that bagging is trivially parallel while boosting is inherently sequential, and the engineering case agrees with the statistical one.)

Verdict 2: redundant labels make your model look better than it is

Here is where Chapter 6 collides with the overlap problem from Chapter 4. A bagged model's out-of-bag (OOB) score is the free cross-validation estimate everyone quotes. But it assumes the resampled rows are independent — and triple-barrier labels are not, because adjacent events share most of their outcome window.

We simulated the overlap directly: duplicate every training row k times (with a whisker of jitter), mimicking labels that are near-copies of their neighbours, and watch the OOB estimate versus true held-out accuracy:

With no overlap the OOB estimate is honest (off by 0.1 pp). With 3× or 6× redundancy the OOB score sails to a perfect 1.000 while the true accuracy never budges from ~0.896. The model hasn't learned anything more — it's just that every bootstrap "test" row has a near-identical twin sitting in the training set, so the OOB estimate is grading the model on data it has effectively already seen. The redundancy doesn't make the model better; it makes the scoreboard lie by ten points.

Recall the uniqueness numbers from the labeling post: NIFTY 50, Smallcap 250, HDFC and APAR all showed ~6–7× effective redundancy in their triple-barrier labels. That is squarely in the regime where the right-hand bars above diverge. If we trained a forest on those events with a naive bootstrap and quoted the OOB score, we'd be reporting a fantasy.

The fix is the machinery we already built

This is why the earlier chapters were not detours. The remedy for both problems is the Chapter 4 toolkit feeding the Chapter 6 model:

The sequential bootstrap draws low-overlap (high-uniqueness) samples, so the trees see genuinely different data, decorrelate, and bagging's variance reduction actually works — and the OOB estimate stops lying. The concrete knob López de Prado prescribes is max_samples = average uniqueness: never let a single bootstrap draw pull in more rows than the data's real independent content (~1/6th of the events, for our ~0.16 uniqueness). On top of that, balanced class weights stop a lopsided +1/0/−1 label mix from dominating, and the purged, embargoed cross-validation (Chapter 7, already in our src/ml/) replaces the OOB score with a leakage-free estimate for the final verdict.

How does it look

Where it sits in the pipeline

This is the model node at the end of the spine:

dollar bars (Ch 2) → FFD features (Ch 5) → triple-barrier labels (Ch 3) → sample weights + sequential bootstrap (Ch 4) → bagged Random Forest (Ch 6) → purged-CV evaluation (Ch 7) → feature audit (Ch 8)

The sequential bootstrap isn't a nice-to-have bolted onto the forest — it is the component that makes the Chapter 6 ensemble statistically valid on overlapping NSE labels. And the recurring lesson holds for a fourth chapter running: the damage in financial ML is rarely done by a weak model. It's done by feeding a reasonable model redundant, noisy, leaky data and trusting the optimistic number it prints. Chapter 6's quiet advice — average many decorrelated trees, size the bags to the real information content, and never trust an OOB score on overlapping labels — is one more instance of refusing to lie to the model, and refusing to let it lie to you.