Daybreak Neural • Study 006 • November 09, 2026

Can a small validation set tell us when to update?

The gate did not meet both development targets.

The fixed experiment

For each participant, 100 offline trials from day one train the source decoder. On day two, trials 1–16 train a candidate update, trials 17–24 compare it with the frozen model, trials 25–48 are unused, and trials 49–100 form the common test block. The gate accepts the candidate only when its validation balanced accuracy is strictly greater. Ties or a missing validation class keep the frozen model.

The gate accepted 22 updates and retained the frozen decoder for 31 participants. It used 16 labels for candidate training and 8 additional labels for selection. The candidate was not refitted after selection. This makes 24 labels the fair main budget, even though the selected candidate was trained on 16. Always-update16 separates the effect of the decision from the cost of reserving 8 labels for validation.

The eight validation labels never enter the candidate fit. The decision uses no test labels. All methods retain Study 005’s fixed filtering and per-trial covariance procedure. Whitening, the Riemannian tangent map and scaling are fitted on source-day trials only; classifier settings are unchanged.

Performance and observed losses

Gains, losses and ties are relative to the frozen decoder on the 52 later test trials. Mean loss burden averages over all 53 participants, not only those whose score falls. Frozen has zero loss by definition; its mean accuracy is therefore essential to interpreting this metric. The largest individual decrease was 11.52 pp for the gate versus 6.36 pp for always-update24. A lower average loss burden does not bound the worst individual loss.

Did it pass the development screen?

The fixed screen required at least 25% lower mean loss burden and no more than 1 pp lower mean accuracy than always-update24. Accuracy criterion: FAIL. Loss-burden criterion: FAIL. These point-estimate targets are provisional engineering choices, not clinical criteria or proof of noninferiority.

The primary paired loss-burden difference (gate minus always-update24) was -0.162 pp, with descriptive 95% t interval [-0.677, +0.353]. Negative values favor the gate. The corresponding accuracy difference was -4.138 pp, interval [-6.198, -2.077]. These intervals describe model-based uncertainty on the reused cohort; they are not confirmatory tests.

Compared with always-update16, selection changed mean balanced accuracy by -2.531 pp. This diagnostic helps distinguish the effect of choosing a model from the effect of training it with fewer labels.

What the early check predicts

Each point compares the 16-label candidate with the frozen model. Positive horizontal values trigger an update; positive vertical values mean that candidate later scored better. This plot is a post-decision diagnostic. It was not used to choose the rule or change its threshold. Individual values are noisy because validation uses 8 trials and testing uses 52.

Verification

• All 53 participants completed, with no exclusions or outcome-driven changes.
• Every source recording matched its Study 005 SHA256 and byte count.
• All 8,268 comparator predictions exactly reproduced Study 005 (frozen, update16 and update24).
• All 212 scores and all 53 gate decisions were independently recomputed from exported predictions.
• The 11,024 test prediction rows cover 2,756 distinct test trials;424 additional rows record the eight-trial decision checks.
• Fit and selection records were checked against their permitted trial IDs. Synthetic preflight checks confirmed test-label isolation, unused-trial isolation and the candidate’s exclusion of validation labels.
• The original protocol and computation files remain unchanged. Processed features are saved for fast classifier and gate reproduction.

Implication for the startup

Keep always-update24 as the current benchmark. Preserve this failed screen as a result rather than adjusting the threshold on the same outcomes. A revised selection method would need a new development protocol, followed by fresh confirmation if it earns that step.

The business hypothesis remains that BCI teams benefit from reliable recalibration with limited labels. This study evaluates one simple selection rule and its label-allocation cost. It does not establish live BCI utility, guaranteed individual safety, minutes saved or an advantage over competing products.

Limits and next confirmation

These are healthy participants performing offline left/right hand imagery, with one fixed day pair and a later test block separated from calibration by unused trials. No physiological artifact rejection was introduced. The result does not cover continuous operation, clinical users, new devices or repeated long-term use.

Kumar2024 remains a possible future longitudinal dataset: its documentation describes 18 participants across six separate days, with differing adaptation groups and online feedback. Only its documentation has been consulted; no recordings were downloaded for Study 006. It requires a separate protocol audit and freeze before any future confirmation.

Reproduce and trace the result

The bundle includes the frozen protocol, original code, source checksums, trial ledgers, predictions, gate decisions, independent validator and an executed results notebook. The results notebook runs without EEG downloads. The optional feature-cache archive supports full classifier and gate refitting under the fixed signal processing.

OpenBMI dataset · OpenBMI paper · Kumar2024 documentation

Freeze UTC: 2026-11-09T14:43:04.833784+00:00
Protocol SHA256: 546228436831409e32130bf7d0183666efbe3877d2ea9684f6dd1aaed16076cc