# Decoder Studio v2: held-out evaluation and audit

Completed November 10, 2026 (America/Chicago). This is an offline, same-person, later-run demonstration, not a population benchmark or a clinical model.

## Result

The prespecified validation procedure selected CSP + LDA. Its final held-out accuracy was **72/120 (60.0%)**, compared with **62/120 (51.7%)** for a deployable training-majority baseline. Mean participant balanced accuracy was **59.60%**, with a descriptive 95% participant t interval of **51.70% to 67.50%**. The paired difference from the training-majority baseline was +9.60 percentage points, with a descriptive interval of +1.70 to +17.50 points. These eight fixed, nonrandom participants and small per-person test sets do not establish broad reliability.

| Method | Validation mean participant balanced accuracy | Final correct / 120 | Final accuracy | Final mean participant balanced accuracy | Descriptive 95% participant interval |
|---|---:|---:|---:|---:|---:|
| Training-majority baseline | 50.00% | 62 | 51.67% | 50.00% | 50.00–50.00% |
| Bandpower + LDA | 58.82% | 65 | 54.17% | 54.13% | 38.00–70.26% |
| **CSP + LDA (frozen default)** | **62.17%** | **72** | **60.00%** | **59.60%** | **51.70–67.50%** |

The baseline always predicts each person's most common training label; ties resolve to left. Its balanced accuracy is exactly 50% for each person, so its zero-width interval is a property of that rule, not a claim of certain future accuracy. The pooled test-label majority reference also happens to be 51.67% (62 left, 58 right); it is a different, descriptive quantity derived from the test labels. Baseline confusion matrix: `[[47,15],[43,15]]`. CSP confusion matrix: `[[38,24],[24,34]]` (actual class by row, prediction by column).

Pooled balanced accuracy and mean participant balanced accuracy are distinct: CSP 59.96% versus 59.60%; bandpower 54.09% versus 54.13%; training-majority 50.83% versus 50.00%. Participant intervals apply to the participant means, not pooled trial metrics. Trials within a recording can be correlated, so no pooled independent-trial confidence interval is presented as population uncertainty.

## Every participant

All eight prespecified participants and all 15 eligible test epochs per person were retained. Training counts are 30 per person after refitting.

| Person | Test left / right | CSP correct / 15 | CSP balanced accuracy | Bandpower balanced accuracy | Training-majority accuracy |
|---|---:|---:|---:|---:|---:|
| S002 | 8 / 7 | 8 | 52.68% | 46.43% | 53.33% |
| S003 | 8 / 7 | 8 | 50.89% | 54.46% | 53.33% |
| S004 | 8 / 7 | 10 | 68.75% | 58.93% | 53.33% |
| S005 | 7 / 8 | 8 | 53.57% | 54.46% | 53.33% |
| S006 | 8 / 7 | 9 | 58.04% | 25.89% | 53.33% |
| S007 | 7 / 8 | 12 | 78.57% | 92.86% | 46.67% |
| S008 | 8 / 7 | 8 | 55.36% | 40.18% | 46.67% |
| S009 | 8 / 7 | 9 | 58.93% | 59.82% | 53.33% |

## Separation of fitting, selection and testing

The fixed cohort was the next eight dataset IDs after the exposed S001 example: S002–S009, selected without inspecting their performance. Each person's model first fit their own run 4 (120 training trials across people); run 8 supplied 120 validation trials. A shared default was chosen by mean participant balanced accuracy, with a fixed simplicity tie-break. The selected method and complete validation evidence were saved at `2026-11-11T00:54:46.785928+00:00` in `demo-selection.json`.

Every candidate was then refitted separately for each person on runs 4+8: 240 training trials, **including the earlier 120 validation trials**. Only after all fitting did the script download run 12 and evaluate all three methods once on its 120 trials. Final artifacts were written at `2026-11-11T01:01:23.297727+00:00`. There are 360 unique trials in total, not 480. The selection and final phases are separate commands and the final phase checks the frozen protocol hash. No final-test result changed the selected default, preprocessing, training settings, cohort or trial list.

Filtering, resampling and epoching occur within each run. Channel normalization is fitted on the applicable training run(s) only. Models receive no held-out labels. Inputs are rounded to six decimals before reference evaluation and publication, so Python and JavaScript consume the same arrays. See `DEMO_PROTOCOL.md` and `train_demo.py` for exact fixed settings; `demo-manifest.json` includes every EDF URL/hash, software version, model metric and asset hash.

## Audit of the earlier deep checkpoints

The existing S001 demonstration used 30 training trials from runs 4+8 and 15 test trials from run 12. EEGNet achieved 7/15 (46.7%) and predicted left for every trial; ShallowFBCSPNet achieved 6/15 (40.0%). Both underperformed the 8/15 (53.3%) test-label majority reference.

The source audit checked the run-specific T1/T2 mapping, microvolt conversion, electrode order, per-run 4–38 Hz filtering, resampling to 128 Hz, 0.5–3.5 s epoch boundaries, train-only normalization, fixed 60-epoch training, and export conventions. No label/preprocessing/export mismatch was found. The existing manifest reports maximum PyTorch/ONNX error of approximately 2.98e-7 for EEGNet and 1.43e-6 for ShallowFBCSPNet, consistent with numerical export agreement rather than an export failure. Limited data and the fixed training recipe are plausible contributors; this audit does not establish a unique cause.

The already inspected S001 test was not used for tuning. Original checkpoints, manifest, data and reproduction script remain available as a clearly separate legacy diagnostic. The current Windows ARM64 scientific Python environment had no PyTorch/Braindecode/ONNX installation, so new deep checkpoints were not trained. The new classical result must **not** be described as beating retrained deep models on the same cohort. It is a better-supported default demonstration with a separate fixed validation/test procedure and a measured simple baseline.

## Verification

`node --test demo-inference.test.mjs` passed all five tests. Every decision score for all three methods and all 120 test trials agrees with the independent Python calculation within 1e-7, with identical classifications. The tests also verify label independence, participant routing, malformed-input failures, default selection from validation alone, complete fixed cohort, and all recorded asset hashes. An independent source review confirmed the split and transformations; NumPy bandpower indexing agreed with an explicit per-trial/channel/bin calculation within 4.44e-16.

## Limits and sources

This is individually calibrated later-run decoding. It does not test a new person, a new day, a different device, online causal operation, arbitrary EEG or general thought reading. Zero-phase filters use future samples within a run. Eight nonrandom participants give imprecise descriptive t intervals, which assume independent and approximately normal participant scores/differences. LDA decision scores are uncalibrated preferences. Imported data require their own defensible split and evaluation.

- Dataset: Schalk, G. (2009). *EEG Motor Movement/Imagery Dataset*, v1.0.0. PhysioNet. https://doi.org/10.13026/C28G6P . Source: https://physionet.org/content/eegmmidb/1.0.0/ . License: Open Data Commons Attribution License v1.0, https://opendatacommons.org/licenses/by/1-0/ . Derived arrays are attributed to this source and retain its license notice.
- Instrumentation: Schalk et al. (2004). *BCI2000: A General-Purpose Brain-Computer Interface (BCI) System*. IEEE TBME 51(6), 1034–1043. https://doi.org/10.1109/TBME.2004.827072 .
- LDA implementation: https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html . Fixed solver `lsqr`, automatic shrinkage.
- CSP background and conventions: https://mne.tools/stable/generated/mne.decoding.CSP.html . This demonstration uses the explicitly documented covariance/eigendecomposition implementation in `train_demo.py`, not an assertion of byte-equivalence to MNE CSP.

Source and license checked November 10, 2026. Application/training code is MIT; data and third-party packages retain their own licenses.