Study 002 · How much calibration helps on the next recording day?¶
Daybreak Neural research project · 8 November 2026
Exploratory development study, not peer reviewed. Participants 1–4 only. Participants 5–9 remain reserved.
Completed result. Adding alignment did not demonstrate an advantage over the same supervised update at the preselected 24-original-trial budget. The four-person mean was 66.73% for the combined method and 67.47% for update-only. The primary difference was -0.74 percentage points, with a 95% conditional paired run-bootstrap interval [-2.29, +0.66]. All four participant-level primary intervals included zero.
The study used 893 accepted test trials, the same five later runs per person at every budget. The 24-trial opportunity yielded 22, 24, 22 and 20 accepted calibration trials. Participants 5–9 were not loaded or evaluated.
The short-prefix result is a development warning: for participant 3, alignment-only at eight original trials fell from 80.43% to 49.41%; the combined method scored 52.00%, while update-only scored 79.53%. This is an observed failure case, not a population estimate or proof of its cause.
Context and methods¶
The primary question is whether covariance alignment adds value beyond the same supervised update at a budget of 24 original trials. Four related methods share a source-trained representation: frozen, alignment only, supervised update only, and alignment plus update. Frozen bandpower–LDA, CSP–LDA, and raw tangent-space–LR are additional reference pipelines.
All accepted day-one task trials train a separate model for each person. The first day-two task run supplies the chronological calibration prefixes 0, 8, 16, 24, and 48. Artifact exclusions happen after those boundaries. The same five later runs supply the accepted test trials for every method and budget. Nothing from those later trials enters a fitted normalization, spatial filter, tangent reference, calibration transform, or classifier.
Key assumptions and limits. Source artifact flags define an offline accepted-trial analysis. Each prediction uses a full four-second imagery epoch. Alignment with zero labels still consumes calibration EEG. The protocol was fixed locally before this computation, after Study001 had already examined these participants; it is not externally preregistered. Day-one training and day-two test sets differ from Study001, so scores across the two studies are not an adaptation comparison. No result demonstrates live control, future-day generalization, clinical benefit or a new algorithm.
Execution record. All code cells were executed top-to-bottom in a fresh Python namespace, with tables and figures saved. This environment blocks Jupyter kernel sockets, so execution used an in-process Python runner. The interactive Jupyter interface was not tested here.
Setup¶
Select Python 3 (ipykernel). A first rerun may download eight public MATLAB source files; cached files are checked against recorded SHA-256 digests. Only participants 1–4 are allowed by the loader. Do not modify this restriction while developing the method.
The accompanying bundle includes pinned requirements.txt. Install missing dependencies with %pip install numpy scipy pandas scikit-learn mne pyriemann matplotlib threadpoolctl and restart the kernel if needed. Exact versions used for this execution are recorded in the outputs.
The NEURAL_DATA_DIR environment variable optionally selects an existing data cache. Otherwise the notebook uses eeg_data under its working directory. No private account, hosted service, or API key is needed.
Data directory: /workspace/scratch/b15a265229b0/tmp/eeg_data Result directory: /workspace/scratch/b15a265229b0/calibration_study/results
Fixed protocol record¶
This is the locally timestamped design fixed before execution. The supervised update retains source data and assigns target mixture fraction n/(n+48), where n is accepted calibration count. Total training weight remains equal to the source sample count at every budget. This fixed weight is a design choice, not a validated optimum.
{
"study": "002",
"frozen_at_utc": "2026-11-08T19:54:33.940268+00:00",
"scope": "exploratory development, not externally preregistered",
"subjects": [
1,
2,
3,
4
],
"reserved_subjects": [
5,
6,
7,
8,
9
],
"budgets_original_trials": [
0,
8,
16,
24,
48
],
"source_training": "all accepted day-one task trials",
"calibration": "accepted subset of original chronological prefix of first day-two task run",
"test": "all accepted trials in day-two task runs2\u20136, fixed for all budgets/methods",
"primary": {
"budget": 24,
"contrast": "combined minus update_only",
"endpoint": "equal-participant mean balanced accuracy",
"unit": "percentage points"
},
"bootstrap_replicates": 5000,
"bootstrap_seed": 20261108,
"target_mixture_weight": "n_accepted / (n_accepted + 48)",
"total_sample_weight": "N_source",
"filter": "Butterworth4thorder8\u201330Hz causal sosfilt peroriginaltrial [0,6)s, crop [2,6)s; no inter-trial state",
"covariance": "OAS pertrial, followed by covariance congruence for alignment variant",
"tangent_space": {
"metric": "riemann",
"tsupdate": false,
"reference": "source-only"
},
"methods": [
"bandpower_LDA",
"CSP_LDA",
"raw_tangent_LR",
"frozen",
"alignment_only",
"update_only",
"combined"
],
"adaptation_setting": "fixed at end of prefix; no later online updates"
}
Reproducible implementation¶
The following collapsed definition cells contain the complete implementation; expand them to inspect individual functions. The spatial/covariance branches use causal 8–30 Hz filtering within each original six-second trial, retaining the four-second imagery interval. OAS covariance is computed before covariance recentering. Tangent references and standardizers are fitted on source data only, with prediction-batch reference updates disabled.
Covariance recentering follows the Euclidean-alignment idea but is not an exact reproduction of raw-signal alignment with covariance shrinkage recomputed afterward.
Data and execution¶
Run the complete fixed study. All candidate settings are specified before test scoring. Progress is printed once per participant. Assertions check the data boundaries and zero-budget equivalence; convergence failures stop execution.
Participant 1/4: loading verified recordings and fitting fixed methods Participant 2/4: loading verified recordings and fitting fixed methods Participant 3/4: loading verified recordings and fitting fixed methods Participant 4/4: loading verified recordings and fitting fixed methods Study computation completed.
| Original budget | 0 | 8 | 16 | 24 | 48 |
|---|---|---|---|---|---|
| subject | |||||
| 1 | 0 | 8 | 16 | 22 | 45 |
| 2 | 0 | 8 | 16 | 24 | 46 |
| 3 | 0 | 7 | 14 | 22 | 46 |
| 4 | 0 | 7 | 13 | 20 | 35 |
| subject | n_source | n_test | |
|---|---|---|---|
| 3 | 1 | 273 | 236 |
| 38 | 2 | 270 | 237 |
| 73 | 3 | 270 | 227 |
| 108 | 4 | 262 | 193 |
| subject | class_1_accepted | class_2_accepted | class_3_accepted | class_4_accepted | |
|---|---|---|---|---|---|
| 21 | 1 | 7 | 7 | 3 | 5 |
| 56 | 2 | 8 | 7 | 3 | 6 |
| 91 | 3 | 7 | 6 | 3 | 6 |
| 126 | 4 | 8 | 6 | 3 | 3 |
Result tables: uncertainty.csv, trial_ledger.csv, leave_one_participant_out.csv, predictions.csv, scores.csv, calibration_counts.csv
Results¶
Balanced accuracy is the mean recall across four classes. The dashed horizontal reference is 25%; this is a reference for uniform guessing, not an empirical significance threshold. The curves show a single observed calibration prefix per participant and budget, with later test trials held constant.
Exact comparison at the primary 24-trial budget¶
The four-person mean gives each participant equal weight, irrespective of retained trial count. The additional frozen baselines provide context; the predeclared primary contrast is combined minus update-only.
| subject | 1 | 2 | 3 | 4 | Equal-person mean |
|---|---|---|---|---|---|
| CSP + LDA | 74.86 | 55.43 | 74.55 | 60.99 | 66.46 |
| Alignment only | 78.09 | 52.50 | 73.75 | 58.98 | 65.83 |
| Bandpower + LDA | 59.76 | 35.17 | 60.87 | 51.95 | 51.94 |
| Alignment + update | 78.52 | 49.01 | 74.64 | 64.73 | 66.73 |
| Frozen tangent decoder | 77.39 | 50.47 | 80.43 | 55.74 | 66.00 |
| Raw tangent + LR | 78.27 | 50.47 | 79.98 | 55.74 | 66.11 |
| Supervised update only | 77.49 | 49.42 | 76.45 | 66.51 | 67.47 |
Paired uncertainty¶
The 95% percentile intervals resample whole test runs, preserving pairing between methods. The four-person mean resamples runs independently within each fixed participant. These are conditional intervals for the observed people, calibration prefixes, fitted models and one test session; they are not intervals for future days or a participant population. Five test runs per participant provide limited resampling information.
| subject | budget | contrast | prespecified_role | estimate_pp | ci_low_pp | ci_high_pp | repetitions | interval_type | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 24 | combined_minus_update_only | primary | 1.030 | -2.390 | 4.120 | 5000 | conditional paired whole-run percentile; unadj... |
| 1 | 1 | 24 | combined_minus_frozen | secondary | 1.125 | -2.500 | 5.103 | 5000 | conditional paired whole-run percentile; unadj... |
| 2 | 1 | 24 | combined_minus_alignment_only | secondary | 0.424 | -2.610 | 3.032 | 5000 | conditional paired whole-run percentile; unadj... |
| 3 | 2 | 24 | combined_minus_update_only | primary | -0.402 | -2.903 | 1.770 | 5000 | conditional paired whole-run percentile; unadj... |
| 4 | 2 | 24 | combined_minus_frozen | secondary | -1.451 | -6.786 | 3.969 | 5000 | conditional paired whole-run percentile; unadj... |
| 5 | 2 | 24 | combined_minus_alignment_only | secondary | -3.491 | -7.184 | -0.163 | 5000 | conditional paired whole-run percentile; unadj... |
| 6 | 3 | 24 | combined_minus_update_only | primary | -1.808 | -4.033 | 0.652 | 5000 | conditional paired whole-run percentile; unadj... |
| 7 | 3 | 24 | combined_minus_frozen | secondary | -5.781 | -11.781 | 0.219 | 5000 | conditional paired whole-run percentile; unadj... |
| 8 | 3 | 24 | combined_minus_alignment_only | secondary | 0.893 | -3.139 | 3.397 | 5000 | conditional paired whole-run percentile; unadj... |
| 9 | 4 | 24 | combined_minus_update_only | primary | -1.773 | -5.546 | 1.350 | 5000 | conditional paired whole-run percentile; unadj... |
| 10 | 4 | 24 | combined_minus_frozen | secondary | 8.997 | -0.711 | 16.594 | 5000 | conditional paired whole-run percentile; unadj... |
| 11 | 4 | 24 | combined_minus_alignment_only | secondary | 5.754 | 3.566 | 8.203 | 5000 | conditional paired whole-run percentile; unadj... |
| 12 | mean | 24 | combined_minus_update_only | primary | -0.738 | -2.292 | 0.657 | 5000 | conditional paired whole-run percentile; unadj... |
| 13 | mean | 24 | combined_minus_frozen | secondary | 0.722 | -2.583 | 3.689 | 5000 | conditional paired whole-run percentile; unadj... |
| 14 | mean | 24 | combined_minus_alignment_only | secondary | 0.895 | -0.674 | 2.312 | 5000 | conditional paired whole-run percentile; unadj... |
Takeaways¶
- The added alignment component has not earned a performance claim. The primary estimate is negative and its conditional interval includes zero. Leaving out any one participant keeps the primary mean negative (range -1.33 to -0.38 pp); these are descriptive sensitivity checks.
- Updating can help some people and hurt others. At 24 trials, update-only averages 67.47% against the frozen decoder's 66.00%. The mean gain is descriptive and is uneven across participants. The largest mean among tested update-only budgets is 70.17% at 48; this is a secondary curve observation, not an independently validated optimal budget.
- Short-prefix alignment can fail badly. Participant 3's seven accepted trials at the nominal eight-trial budget contained three left-hand, three right-hand, one feet, and no tongue trial. Unrepresentative class composition is one plausible contributor to the covariance shift; this experiment does not isolate the cause. Participants 1 and 2 also lacked tongue examples at this budget without the same collapse.
- Established representations matter. Frozen CSP–LDA (66.46%) and raw tangent–LR (66.11%) outperform bandpower–LDA (51.94%) on the same test trials. Their preprocessing and representation differ; this is not an isolated algorithm effect or a new-method claim.
- Next research decision: retain supervised update-only as the comparator to beat; investigate conservative, prefix-dependent covariance recentering on development data before freezing any new rule. Keep participants 5–9 reserved. More sessions and independent data are needed before claiming reliable calibration savings.
This is an offline development ablation of existing ingredients. It does not establish a novel decoder, live control benefit, clinical usefulness, commercial readiness, or reduced real-world calibration time.
References and reproducibility¶
- BNCI2014_001 / BCI Competition IV 2a description
- He & Wu: Euclidean-space data alignment
- MNE CSP documentation
- pyRiemann TangentSpace documentation
- scikit-learn LogisticRegression documentation
The accompanying study bundle contains the frozen protocol, source hashes, pinned package versions, standalone engine, plots, trial-level predictions, calibration ledger, results, and independent validation record. Public raw EEG is retrieved from the source rather than redistributed in the bundle.