Your Model Has Amnesia: Fractional Differentiation for Indian Equities

Every quant tutorial begins the same way: "first, convert prices to returns."

We do it so reflexively that we never ask what it costs. The justification is stationarity — ML models need inputs whose statistical properties don't drift, and raw prices drift relentlessly. NIFTY 50 at 8,000 in 2016 and 23,600 in 2026 is not a series any classifier can learn from directly. So we difference it, get returns, and move on.

López de Prado in his book Advances in Financial Machine Learning stops at exactly this step and asks the uncomfortable question: how much memory did you just erase?

The answer, measured on ten years of Indian market data below: almost all of it. And almost all of it was erased for nothing.

The dilemma: stationarity vs memory

Think of a price series as having two properties in tension.

Memory is everything the current value tells you about where the series has been — the level, the distance from old highs, the shape of the trend. Price (d=0, undifferenced) has perfect memory. It is also hopelessly non-stationary: its mean wanders, and a model trained on the 12,000-level NIFTY learns nothing useful about the 23,000-level NIFTY.

Returns (d=1, first difference) are the opposite. Beautifully stationary — and amnesiac. Yesterday's return tells you essentially nothing about the price level, the trend, or how far we are from the 52-week high. We measured it: the correlation between NIFTY 50 daily returns and the NIFTY 50 price level over ten years is 0.007. Statistical white noise with respect to where the market actually is.

The standard workflow treats differencing as a binary switch: 0 or 1, memory or stationarity, pick one. In Prado's Book the Chapter 5's insight is that the switch is actually a dial. There are fractional values of d — 0.25, 0.3, 0.4 — and somewhere on that dial is the minimum amount of differencing that achieves stationarity. Everything beyond that point destroys memory and buys you nothing.

The intuition: a fading echo

What does it even mean to difference a series by 0.25?

A fractionally differenced value is a weighted combination of today's price and its own past — where the weights fade gradually the further back you look. A return is "today minus yesterday": an echo exactly one day long. A price is the full, unfaded history. Fractional d sits in between: today's value, plus a fading echo of the past several hundred days. Small d, long slow-fading echo, lots of memory. Large d, short sharp echo, little memory. At d=1 the echo collapses to a single day and you have ordinary returns.

Standard vs fixed-width window — the part that trips people up

Chapter 5 presents two implementations, and the difference matters in practice.

The standard (expanding window) method lets every point listen to the entire history back to the first observation in your dataset. Two problems follow. Early observations have only a short history available, so they're computed with a shorter echo than later ones — the series isn't comparable with itself. And because the weights never actually reach zero, every value quietly depends on where your dataset happens to start: re-run the same code with data starting in 2014 instead of 2016 and the value for the same date changes. Features that depend on an arbitrary sample boundary are a quiet source of drift and irreproducibility.

The fixed-width window (FFD) method makes one pragmatic cut: once the weights fade below a small threshold (we used 1e-4), truncate them. Now every point listens to the same fixed stretch of past — same echo length for everyone. The series is internally consistent, independent of sample start, and you only sacrifice the first window-length of observations, which were unreliable anyway. FFD is what the book recommends and what everything below uses.

What we measured

Ten years of daily closes (June 2016 – June 2026) from the Kite historical API. For each series, we ran d from 0 to 1, applied FFD, and recorded two numbers: the ADF test statistic (is it stationary yet?) and the correlation with the original price (how much memory survives?). d* is the smallest d that passes ADF at 95% confidence.

Read the last column first. At d*, each FFD value is built from a fading echo of roughly one to two years of history. Returns use two days. That is the memory gap in its rawest form: the standard workflow compresses a year of context into "today minus yesterday."

Now read the d* column top to bottom, because the ordering is the whole point.

HDFC Bank, d = 0.10.* The gentlest differencing in the table. HDFC Bank spent long stretches of the decade range-bound — the 2021–23 sideways grind that frustrated every holder. Range-bound behaviour has a natural pull toward its own mean, so the series barely needs scrubbing to pass a stationarity test. You keep 98% of the memory.

NIFTY 50, d = 0.25.* A diversified large-cap index: trends, but with the churn of fifty constituents partially offsetting each other.

NIFTY Smallcap 250, d = 0.30.* Smallcaps trend harder and longer — the 2020–22 surge, the 2023–24 melt-up. More persistent trends mean a stronger unit-root character, so the dial needs to turn further. At NIFTY 50's d* of 0.25, the smallcap index still fails the ADF test outright (p = 0.16).

APAR Industries, d = 0.35.* A single hard-trending smallcap — roughly a 20x move across the decade with brutally persistent runs. The trendiest series in the table needs the most differencing.

The pattern is clean: d is a measurable fingerprint of how persistently a series trends.* Efficient, range-bound large-caps sit near 0.1. Diversified indices near 0.25. Momentum-heavy smallcap segments at 0.3 and above. This is why Chapter 5 insists on estimating d* per instrument rather than assuming one number for the universe: a 3.5x spread separates HDFC Bank from APAR in the same market, same decade.

And every row shares the same punchline: the d* needed for stationarity is nowhere near 1. Differencing to d=1 — the thing every tutorial does on line one — over-differences Indian equities by a factor of three to ten, destroying 90%+ of recoverable memory to purchase stationarity that d=0.1–0.35 already delivers.

Understanding the Graphs

Both charts have the same three-panel layout, so once you read one you can read the other. Let me walk through each panel and then contrast the two stocks.

Top panel — price (blue) vs FFD series (red)

Blue is the raw closing price on the left axis; red is the fractionally-differenced series on the right axis. The thing to look for is how closely red tracks the shape of blue while living in a stable horizontal band instead of drifting upward.

For HDFC Bank the red line is almost a shadow of the blue one — they rise and fall together so tightly that the correlation is 0.98. That's because d* is only 0.10: barely any differencing was applied, so almost all the price's memory survives. Yet the red series no longer trends off to the top-right; it oscillates around a level. Stationary, but it still "knows" the price.

For APAR the red still follows the broad contour — you can see it respond to the huge 2021-onward run-up — but it's visibly noisier and the link to blue is looser (correlation 0.94). That's the cost of needing d* = 0.35: more differencing was required to tame APAR's powerful trend, and stronger differencing always sands away some memory.

Middle panel — daily returns (d=1)

This is the same series differenced all the way to d=1, the conventional "convert to returns" step, shown for contrast. For both stocks it looks like a stationary band of noise centered on zero with volatility clusters (you can spot the COVID-2020 spike in both). The point of this panel is that returns are stationary but carry essentially no memory of the price level — correlation with price ≈ 0.01. It's the "amnesiac" extreme.

Bottom panel — the tradeoff curve

This is the most important panel. The x-axis is d from 0 to 1. The green line (left axis) is how much memory survives — correlation with price — and it falls as you difference more. The red line (right axis) is the ADF stationarity statistic; the dotted horizontal line is the 5% critical value, and a series is "stationary enough" once the red curve drops below it. The black vertical dashed line marks d*, the smallest d where that happens.

For HDFC Bank the black line sits far to the left at d* = 0.10. The green memory curve is still way up near the top there — you cross into stationarity while keeping 98% of the memory. HDFC barely needed any treatment because it spent much of the decade range-bound, which is naturally close to stationary already.

For APAR the black line sits further right at d* = 0.35. APAR's relentless multi-year trend has a stronger unit-root character, so the red ADF curve takes longer to drop below the threshold — you have to turn the dial further, sliding further down the green curve and giving up more memory in the process.

So the two bottom panels side by side are the whole argument of the chapter in one picture: the trendier the stock, the further right its d*, and the more memory you must spend to reach stationarity. HDFC at 0.10 and APAR at 0.35 are the two ends of that spectrum within the same market and same decade — which is exactly why d* has to be estimated per instrument rather than assumed.

What this means for a feature pipeline

For our NIFTY 500 work, the takeaways are concrete.

FFD series, not returns, become the price-based inputs to anything that should know about levels and trends — meta-labeling features, regime inputs, trend filters. Returns remain fine for what they're good at (volatility estimation, risk), but as a memory-bearing feature they are the wrong tool.

d* must be estimated on the training window only, then frozen. It's a fitted parameter; estimating it on the full sample leaks future information into your features, exactly the sin Chapters 7–8 of the book are devoted to preventing.

And d* deserves re-checking per instrument and occasionally per era. A stock that migrates from smallcap-momentum behaviour to largecap-rangebound behaviour (plenty of Indian examples) will see its fingerprint drift.

The deeper lesson rhymes with the dollar-bars note that preceded this one. Both chapters attack a default transformation that everyone applies without thinking — time bars there, first-differencing here — and both show the default quietly destroys information the model needed. The fix in both cases is not a fancier model. It's stopping the destruction.