Study 004 · Evaluation on reserved participants¶
Daybreak Neural · 8 November 2026
Completed fixed evaluation on five previously reserved participants. At 24 original calibration trials, blended alignment plus supervised updating reached 62.99% mean balanced accuracy, versus 63.76% for supervised updating alone. The primary paired difference was -0.765 percentage points, with a model-based 95% participant t interval of -2.250 to +0.720 points. Balanced accuracy was higher for 1 participant, lower for 4, and tied for 0.
The protocol and engine were frozen before the first requests for participants 5–9. All methods and budgets were retained without retuning. Each person supplied their own day-one training data. This tests the fixed training/adaptation procedure on new people within the same dataset; it is not zero-shot transfer of one pretrained model or prospective clinical validation.
What is being tested?¶
Study 003 did not demonstrate an added average benefit of conservative covariance alignment over the same supervised update alone at the fixed 24-trial budget on four development participants. Here we keep the procedures unchanged and evaluate five people who did not inform that development.
Primary contrast: blended alignment plus supervised update minus supervised update only, at 24 original calibration trials. The endpoint is the equal-participant mean difference in four-class balanced accuracy, measured in percentage points. A positive difference favors adding the covariance blend.
The fixed blend is lambda = n / (n + 48), with n the accepted calibration count, and R_blend = (1 - lambda) R_source + lambda R_prefix. Its symmetric inverse square root transforms target covariances. The source tangent-space reference and scaler remain fixed. The separate supervised fitting weights retain the original source data, with total fitting weight equal to the source sample count and target fraction n / (n + 48). No blend strength, classifier parameter, feature, budget or participant is selected using the new results.
All nine earlier methods and all five budgets are retained. The four focal methods are plotted for clarity; the complete scores remain available.
Execution record. All code cells were executed top-to-bottom in a fresh Python namespace, with tables and figures saved. This environment blocks Jupyter kernel sockets, so execution used an in-process Python runner. The interactive Jupyter interface was not tested here.
Data boundaries and interpretation¶
For each person, accepted day-one trials train the source models. The first day-two task run supplies chronological prefixes of 0, 8, 16, 24 and 48 original trials; artifact exclusions occur after each boundary, so the accepted counts may be smaller. Accepted trials from day-two runs 2–6 form the same test set for every method and budget. Test trials never enter fitting, alignment estimation or a shared preprocessing reference.
This remains an offline, accepted-trial analysis within one dataset, with one evaluated day per person. It cannot establish clinical utility, live control performance, future-day reliability, calibration-time savings or a commercially useful gain. After this evaluation, all nine participants have been examined within this project; further tuning on these results becomes development work.
Setup and portable rerun¶
Use a separate Python 3.12 environment and select its registered kernel in Jupyter. If your current Python 3.12 (Neural Study 003) kernel already reproduced Study003 successfully, it can be used here. This notebook includes the engine and plotting definitions, so the accompanying .py files are not required to run it.
The bundle includes requirements.txt for the reference environment and requirements_windows_arm64.txt for native Windows ARM64. The latter uses the pandas version that previously reproduced Study003 in that environment. If dependencies are missing, run the recommended %pip install ... command printed by the next cell in a new notebook code cell, wait for successful completion, restart the kernel, and then run all cells. Installation is not part of the scientific computation.
The first evaluation or rerun may download ten public MATLAB files. Every read must match the recorded SHA-256 manifest. Those hashes were measured on the initial downloads; they are not publisher-provided checksums. NEURAL_DATA_DIR can point to an existing cache, otherwise eeg_data is used. Results go into results under the notebook's working directory.
Python: 3.12.14 | System: Linux | Machine: x86_64 If dependencies are missing, install in a new cell: %pip install --only-binary=:all: -r requirements.txt Then restart the kernel before running the analysis. Data: /workspace/scratch/b15a265229b0/tmp/study004_eeg Results: /workspace/scratch/b15a265229b0/heldout_study/results
Frozen protocol and implementation¶
The primary uncertainty summary uses five paired participant differences, with a model-based two-sided 95% Student t interval (four degrees of freedom). This assumes independent, approximately normal differences and is fragile with only five people. An exhaustive two-sided sign-flip sensitivity uses all 32 sign patterns under a symmetric-null assumption. Its smallest possible nonzero p value is 0.0625; it is not a binary success gate.
Secondary whole-run bootstrap intervals use 5,000 paired draws and remain conditional on these five people, fitted models, fixed prefixes and sessions. They do not include new-person, new-day, refitting or calibration-prefix uncertainty. If an accepted test run lacks a class, the fixed fallback reports conditional intervals unavailable without dropping people, runs or classes. Primary scoring still uses valid aggregate class coverage.
Expand the definition cells to inspect the data loading, fit boundaries, unchanged methods and summaries.
Design frozen at: 2026-11-08T21:24:31.571742+00:00
Protocol SHA-256: 78e92c96f3ad20676c7e519060c9a4e514f07e8a461b813666acad6aca3db355
Evaluation engine SHA-256: 2f0e0c0e160d405db60792f87df01f2f4f873a1e1b6b9cd4c08441c83bd08e40
Primary: {'budget': 24, 'contrast': 'blended_combined minus update_only', 'endpoint': 'equal-participant mean balanced accuracy difference', 'unit': 'percentage points', 'participant_count': 5}
Evaluation participants: [5, 6, 7, 8, 9]
Execute the fixed comparison¶
The computation checks source hashes, cohort membership, finite features, covariance validity, convergence, test-set pairing, zero-budget equality and fitting access boundaries. No later-trial updates occur. A repeat run is a computational reproduction; it does not provide another independent participant sample.
Participant 5: loading verified recordings and fitting frozen procedures Participant 6: loading verified recordings and fitting frozen procedures Participant 7: loading verified recordings and fitting frozen procedures Participant 8: loading verified recordings and fitting frozen procedures Participant 9: loading verified recordings and fitting frozen procedures Study004 completed.
| budget | 0 | 8 | 16 | 24 | 48 |
|---|---|---|---|---|---|
| subject | |||||
| 5 | 0 | 6 | 12 | 18 | 42 |
| 6 | 0 | 7 | 15 | 22 | 40 |
| 7 | 0 | 8 | 15 | 23 | 47 |
| 8 | 0 | 8 | 15 | 22 | 45 |
| 9 | 0 | 8 | 15 | 23 | 45 |
| budget | 0 | 8 | 16 | 24 | 48 |
|---|---|---|---|---|---|
| subject | |||||
| 5 | 0.0 | 0.111 | 0.200 | 0.273 | 0.467 |
| 6 | 0.0 | 0.127 | 0.238 | 0.314 | 0.455 |
| 7 | 0.0 | 0.143 | 0.238 | 0.324 | 0.495 |
| 8 | 0.0 | 0.143 | 0.238 | 0.314 | 0.484 |
| 9 | 0.0 | 0.143 | 0.238 | 0.324 | 0.484 |
Accepted calibration counts by original budget: Actual target covariance weight:
Results across calibration budgets¶
Balanced accuracy averages recall over the four classes. An increase of one percentage point means, for example, 60% to 61%, not a 1% relative increase. Different budgets use identical later test trials. These curves describe the fixed budgets rather than selecting the best one.
Primary result: 24 original calibration trials¶
The following table reports all nine methods. Each person contributes equally to the final mean, regardless of their number of accepted test trials. The individual paired differences make any improvement or harm visible.
| subject | 5 | 6 | 7 | 8 | 9 | Mean |
|---|---|---|---|---|---|---|
| Bandpower + LDA | 33.47 | 42.93 | 60.24 | 61.32 | 65.23 | 52.64 |
| CSP + LDA | 37.33 | 43.01 | 53.98 | 66.09 | 76.86 | 55.46 |
| Raw tangent + LR | 41.24 | 50.33 | 57.20 | 66.50 | 64.15 | 55.88 |
| Frozen tangent decoder | 40.81 | 50.32 | 56.75 | 66.96 | 64.15 | 55.80 |
| Alignment only | 43.20 | 47.94 | 70.08 | 76.05 | 65.58 | 60.57 |
| Supervised update only | 43.42 | 56.64 | 68.52 | 75.37 | 74.84 | 63.76 |
| Full alignment + update | 41.16 | 55.47 | 68.77 | 75.73 | 70.73 | 62.37 |
| Blended alignment only | 46.31 | 50.85 | 64.37 | 74.47 | 64.19 | 60.04 |
| Blended alignment + update | 42.99 | 56.05 | 68.98 | 74.86 | 72.07 | 62.99 |
| subject | budget | update_only_ba_percent | blended_combined_ba_percent | difference_pp | |
|---|---|---|---|---|---|
| 0 | 5 | 24 | 43.418 | 42.994 | -0.424 |
| 1 | 6 | 24 | 56.636 | 56.053 | -0.583 |
| 2 | 7 | 24 | 68.523 | 68.976 | 0.453 |
| 3 | 8 | 24 | 75.366 | 74.864 | -0.502 |
| 4 | 9 | 24 | 74.842 | 72.072 | -2.770 |
Mean paired difference: -0.7649 percentage points Model-based 95% t interval: [-2.2502, +0.7204] pp; df=4 Two-sided exact sign-flip sensitivity: p=0.1875; 32 patterns Positive / negative / tied participants: 1 / 4 / 0 Interval status: model_based
Descriptive sensitivity and conditional run intervals¶
Leave-one-person-out means show how much the five-person average changes when a single person is omitted for this diagnostic only; every participant remains in the primary result. Conditional run-bootstrap intervals answer a narrower question about repeated draws of these recorded runs. They are secondary to the participant-level summary and cannot be treated as population intervals.
The four additional comparisons were specified before reserved data were accessed. They are descriptive and have no multiple-comparison adjustment.
| omitted_subject | estimate_pp | |
|---|---|---|
| 0 | 5 | -0.8502 |
| 1 | 6 | -0.8105 |
| 2 | 7 | -1.0695 |
| 3 | 8 | -0.8307 |
| 4 | 9 | -0.2636 |
| budget | contrast | prespecified_role | estimate_pp | ci_low_pp | ci_high_pp | status | |
|---|---|---|---|---|---|---|---|
| 5 | 8 | blended_combined_minus_combined | secondary | 5.9948 | 3.9925 | 8.0697 | available |
| 26 | 24 | blended_combined_minus_update_only | primary | -0.7649 | -1.3368 | -0.1951 | available |
| 27 | 24 | blended_combined_minus_combined | secondary | 0.6216 | -0.7052 | 1.8873 | available |
| 28 | 24 | blended_combined_minus_frozen | secondary | 7.1956 | 5.4431 | 8.8552 | available |
| 29 | 24 | blended_combined_minus_blended_alignment | secondary | 2.9523 | 1.4257 | 4.5406 | available |
| method | blended_combined | frozen | blended_combined_minus_frozen_pp |
|---|---|---|---|
| subject | |||
| 5 | 42.994 | 40.815 | 2.180 |
| 6 | 56.053 | 50.319 | 5.734 |
| 7 | 68.976 | 56.747 | 12.230 |
| 8 | 74.864 | 66.955 | 7.909 |
| 9 | 72.072 | 64.146 | 7.926 |
Leave-one-participant-out primary means (descriptive): Conditional run-bootstrap means: Per-participant context: blended + update versus the frozen source model at 24 trials
Interpretation and next decision¶
The conservative blend did not demonstrate added benefit over supervised updating alone. On the five reserved participants, its primary mean difference was −0.765 percentage points. Balanced accuracy was lower for four participants and higher for one. The participant-level 95% t interval spans −2.250 to +0.720 points, so the study neither establishes a general disadvantage nor proves equivalence. Every leave-one-person-out mean remained negative (−1.069 to −0.264 points), making the direction of the observed average less dependent on one participant, while leaving the small-sample limitation intact.
The narrower conditional interval answers a different question. Resampling recorded runs gave −1.337 to −0.195 points for the same primary contrast. Its exclusion of zero does not override the participant-level interval: it keeps the people, fitted models, prefixes and sessions fixed and omits uncertainty about new people and days. The participant-level summary is the prespecified primary analysis, but five reserved participants from one dataset do not establish reliability for new people or future sessions.
Supervised calibration remains the more promising component in this benchmark. At 24 original trials, update-only averaged 63.76% versus 55.80% for the frozen source model, a descriptive difference of +7.96 points. All five participants improved on that comparison, by 2.60 to 11.78 points. It used 18–23 accepted calibration trials per person. This is a descriptive comparison of fixed methods, not a newly designated primary endpoint, an optimized calibration budget or a demonstration of live time savings. It supports retaining update-only as the simpler research baseline.
Conservative alignment still mitigated aggressive alignment in one prespecified setting. At eight original trials, blended alignment plus updating exceeded full alignment plus updating by 5.99 points on average. The conditional run-bootstrap interval was +3.99 to +8.07 points. However, update-only already reached 61.20%, close to the blend's 61.43%. Improving on full alignment at this small prefix does not establish added value over the simpler update-only procedure. At 48 trials the blend exceeded update-only by 0.56 points descriptively; changing the primary budget after seeing that result would be another development choice.
Research decision. Preserve this complete result, including the unfavorable primary comparison. Use update-only as the default comparator for subsequent research and retain the frozen blend as a documented experimental arm. The next major evidence step should use new data: an independently sourced dataset or prospective sessions from a research partner, with the endpoint and practical target fixed before evaluation. If developing a rule that decides when to update, align or leave a decoder unchanged, treat that as a new hypothesis with a separate development/validation split. All nine participants in the current dataset have now been examined within this project.
Business implication. The current concrete asset is an auditable calibration benchmark and a reproducible adaptation workflow. The observed benefit of labeled updating gives a specific question to investigate with an external team: can their decoder maintain useful performance with a small, predictable calibration burden across repeated sessions? A pilot should measure accepted and rejected trials, actual calibration duration, accuracy on later sessions, individual regressions and the team's operational costs. We have not established a superior proprietary decoder, prospective reliability, clinical utility or a market advantage. The present findings support an evidence-led evaluation pilot and continued method development, rather than marketing the covariance blend as a proven improvement.
Reproduce the computation¶
The final reference check, added after this initial execution, compares your run's scores and predicted labels with the saved reference. A successful rerun confirms computational agreement in another environment. It does not turn these same people or sessions into fresh validation data. Keep the fixed protocol and dependencies when checking reproducibility, and send back the final reference check plus any warnings or errors.
Sources and bundle contents¶
- BNCI dataset description and trial timing
- Euclidean alignment: He & Wu
- MNE CSP documentation
- pyRiemann TangentSpace documentation
- scikit-learn LogisticRegression documentation
The accompanying bundle contains the frozen protocol and hashes, acquisition record, standalone code, pinned dependencies, predictions, fit-access audit, numerical diagnostics, figures, reference results and independent validation. Public raw EEG files are downloaded separately.
REFERENCE CHECK Score rows: 225 / 225 expected All scores match within 1e-8: True Maximum score/count difference: 0 Prediction rows: 48780 / 48780 expected All predicted labels match exactly: True PASS: computation reproduces the saved reference.