The experiment design, step by step

The question, and why benchmarks can't answer it

A multivariate forecaster can read each variable from its own history (channel-independent, CI) or attend across all variables at once (channel-dependent, CD). CD costs far more compute, and the question is when it earns that cost.

• Scope: PatchTST with flattened-token CD, the generative families tested, one T4 GPU budget, horizon 96.
• Real datasets (ETTh1, ECL) appear as context only; the synthetic families carry the evidence.

Three questions the paper keeps apart

Dependence in the data does not by itself mean extra forecasting information. Two series can move together because of shared noise that neither can see coming, and then the other series' past tells you nothing new. The whole design exists to separate the second row of that table from the first.

Four synthetic families, each isolating one thing

Every family uses the same persistence φ = 0.8, the same five seeds {42, 123, 456, 789, 1011}, horizon 96, and z-scoring fitted on the training split only. Why 0.8, and what it costs, is worked through in the ceilings section.

The patch sweep's generator differs from the other three: iid innovations at σ = 0.1, 20,000 steps and a 70/10/20 split, against compound-symmetry innovations at ρ = 0.5, 14,400 steps and 60/20/20. Its absolute MSE is never set against the other families; only the CD/CI ratio inside the sweep is read.

The controls, and what breaks without each

A null result is convincing only if the setup could have found an effect and nothing else could explain what it found. Each control below closes one alternative explanation.

The statistics: pairs, seeds and what the design could detect

Each point below is one paired difference, CD minus CI, for one seed in one AR(1) cell. Above zero means CI was better. Pick a seed to see why the seed is the unit of replication.

Why the interval is clustered by seed

The same five seeds appear in all nine cells, so a seed's luck moves nine rows together. Treating the 45 rows as independent understates the uncertainty. Average each seed across cells first, then build the interval over five seed means.

Two things widen the clustered interval: the standard error is built from five numbers instead of 45, and the multiplier comes from a t distribution with four degrees of freedom instead of 44.

Five seeds means df = 4

With few seeds the spread is estimated from little data, so the multiplier is larger and intervals are wider. That makes a tight interval at df = 4 genuinely tight.

The detection floor

• A true CD advantage of 0.0063 would just push a typical cell's interval off zero: roughly a 50%-power threshold. At 80% power the minimum detectable effect is about 0.0085 MSE (0.83%), still under the 1% band; the 0.941 is the t quantile that leaves 20% in the lower tail.
• It is a pooled, typical-cell figure: the noisiest cell (C = 84, ρ = 0.9) has SD 0.0111, so its own floor is about 0.0138.
• The 1% band was fixed before the robustness analyses were run; the paper calls it pre-specified. It is an operational cut-off on effect size and makes no formal equivalence claim. On the grand mean a two-one-sided test would pass, since even the 95% interval sits inside the band.

The explanations tested

In the leader-follower data, CD got relatively worse as coupling rose (slope +0.0072 per unit γ, p = 0.004, point estimates inside the band, though at γ = 0.9 the interval's upper edge reaches about 1.16%). Three experiments asked why.

Was CD simply trained differently? A matched update budget

Where does the widening live? A cross-variate head

The withdrawn claim: was it checkpoint selection?

Instrumented reruns at γ = 0.6 showed CD peaking early (epoch 4 to 7) and degrading, while CI plateaued (epoch 16 to 20). An earlier draft said the gap was "largely a selection artefact". It could not be: the protocol already restored the best checkpoint, visible because total steps = best epoch × steps per epoch in every row. The change was decomposed instead:

• The run-set gap is reported as unexplained. Seeds, batch sizes, steps and stopping rule were identical, which rules out configuration; code-level nondeterminism stays open.
• Its consequence is bounded: shifting every synthetic cell by +0.77% (the original γ = 0.6 gap) leaves all but one cell inside the band.

Real data, as context

The ceilings: what any model could achieve

Because the generating process is known, the best possible forecast error can be computed for each mode, with no training at all.

• Diagonal families (AR(1) grid, block covariance, leaders, isolate): a channel's own past is all that helps, whatever the noise correlation, so the CI and CD ceilings are identical at every horizon.
• Followers: the CD ceiling uses the full state; the CI ceiling is the steady-state Kalman predictor that sees only the follower's own history. Only the 10 followers of 21 channels have any gap at all.

The follower line comes from unrolling its equation h steps and taking expectations at time t. The leader's weight hφ^h−1 is 1 at one step, 2.05 at four and five steps, 0.29 at twenty and 6 × 10⁻⁸ at 96, which is why the absolute gap in the chart peaks near h = 5. The CI ceiling is lower than this term implies, since the Kalman predictor reads part of l_t back out of the follower's own past.

• At the h = 96 endpoint the two ceilings coincide: φ¹⁹² ≈ 2.5 × 10⁻¹⁹.
• Averaged over the forecast the most any model could gain is about 0.3% across all channels: at γ = 0.9 the follower gap averaged over 96 steps is 0.0065, and 0.0065 × 10/21 = 0.0031 once the eleven channels with no gap are counted, 0.31% of a CI MSE close to 1. That is under the 1% band and under the 0.0063 floor. The null holds, and parity there is a weak test of the architecture, since a model that ignores the signal and one that uses it would look the same.
• The opportunity lives at short horizons, so the informative next experiment is h ≤ 10.

Why φ = 0.8

Persistence φ controls two things at once: how much of a channel stays predictable far ahead, and how many effectively independent observations a fixed-length series contains. Raising one lowers the other, and the slider shows by how much.

• Below about 0.6 there is almost nothing to forecast: a 512-step lookback carries no information past a few steps.
• Above about 0.95 the effective sample size collapses: 222 effective observations per channel at 0.95 against 960 at 0.8, and 16 at 0.9964. Burn-in is fine at 0.95 and only starts to bite near 0.996.
• The only φ that keeps half the signal at h = 96 leaves about 16 effective observations. A fixed series length cannot give both, which is why the next experiment shortens the horizon instead of raising φ.

Limits, ranked, and what comes next

Ranked by how much each one could change the conclusion.

1. The unexplained run-set shift: every CD figure comes from one run per configuration, with rerun evidence at one cell.
2. h = 96 barely discriminates: about 0.3% available averaged over the forecast.
3. P = 2 untested: about 33 GPU-hours per seed with fused attention.
4. ECL is one seed, two of four horizons; descriptive only.
5. Narrow families: one leader-follower topology, two covariance structures, φ fixed at 0.8.
6. One CD formulation (flattened tokens) and one hardware budget.
7. MSE only, though MAE is logged in every results file.
8. ETTh1 test slice: the code scores a 5,900-row slice where the benchmark convention scores 2,880 rows, so its absolute MSE is not comparable with published tables.
9. No formal equivalence test and no multiplicity correction.
10. No independent retraining.

Next, in order

1. A short-horizon comparison, h ≤ 10, with the band restated against the ceilings.
2. P = 2 at that short horizon.
3. Five seeds on ECL.
4. A variate-token CD (iTransformer-style), which removes the quadratic cost.
5. Diebold-Mariano tests with a HAC variance on the real-data windows.

What would change my mind

At h = 5 the ceiling gap is 9 to 15 times the band. If CD still tied CI there, the architecture would be failing to use signal that exists. If it captured the gap, the h = 96 null is a horizon effect, which is what the theory predicts. Either answer is worth more than another run at 96 steps.