Why Higher Classification Accuracy Is Not Enough in BCI

A brain-computer interface can classify 90% of labelled trials correctly and still be a poor interface.

It may take too long to reach each decision. It may issue commands while the user intends to do nothing. Its confidence scores may be unreliable. It may work for the participants who generate the clearest signals while failing those who need it most. It may also require a new calibration session every time the headset is fitted.

None of those failures is visible in a single average-accuracy figure.

This article concerns classification-based brain-computer interfaces, particularly non-invasive electroencephalography (EEG) systems evaluated on discrete tasks. Continuous control, regression-based decoders and invasive interfaces require additional measures. Results from one sensing modality, task or participant population should not be transferred to another without direct evidence.

Accuracy answers a narrow question

Classification accuracy is the proportion of evaluated decisions assigned the correct label. That definition is useful only when its denominator and test conditions are clear.

An accuracy result should state the classes, their prevalence, the chance or majority-class baseline, the decision window and the validation split. A randomly shuffled trial split, a held-out recording session, a held-out user and a prospective online test answer different questions. They cannot support the same deployment claim.

The task protocol matters just as much. In many offline studies, the system receives a pre-segmented interval that is known to contain an intended action. The practical interface may instead need to determine when an intention begins, when it ends and whether the user intends to control the system at all. Accuracy calculated only on cued, intention-containing trials does not measure that problem.

Two systems can therefore report the same accuracy while behaving very differently. One may make a fast decision and reject uncertain inputs. The other may accumulate several seconds of data and always force a command. Their headline result is equal; their operational value is not.

Confidence must predict correctness

A classifier score is not automatically a probability. A system can rank one class above another without its reported confidence corresponding to the actual chance that the decision is correct.

A confidence estimate is calibrated when decisions assigned a given probability are correct at approximately that rate under the stated test conditions. If a system labels many decisions as 80% confident, roughly 80% of those decisions should be correct. Calibration and accuracy are different properties: a more accurate model can still be overconfident when it is wrong.

This distinction is established beyond BCI. Guo and colleagues showed empirically that modern neural networks can be poorly calibrated even when their classification performance is strong. Their definition of confidence calibration is directly relevant to any BCI that uses a score to accept a command, trigger a device or request confirmation. Guo et al., ICML (2017)

For BCI, confidence must be tested under the conditions in which it will control behaviour. A reliability result from pooled trials can conceal miscalibration for an individual user. A confidence mapping fitted and tested within one session may cease to be reliable after electrode placement, impedance, attention or signal statistics change.

Credible reporting should therefore include a proper scoring measure such as log loss or the Brier score, together with a reliability analysis. If an expected calibration error is reported, its binning rule and sample size should also be stated. These measures do not replace task performance; they test whether confidence is meaningful enough to support a decision policy.

A usable BCI must be allowed not to act

Forced classification assumes that every input belongs to one of the command classes. That assumption is inappropriate whenever the signal is too weak, corrupted or unrelated to an intended command.

A rejection or abstention outcome means that the system withholds a decision because the available evidence does not meet a defined acceptance condition. Rejection is not the same as an error. It is a controlled response to insufficient evidence.

The tradeoff must be reported transparently. Increasing the acceptance requirement may improve accuracy among accepted commands, but it will usually reduce coverage—the proportion of opportunities on which the system acts. Reporting accepted-decision accuracy without coverage can make a system appear reliable simply because it rejects most inputs. Reporting coverage without the error rate is equally incomplete.

There is also a distinct non-control problem. A self-paced interface must recognise periods in which the user is not attempting to issue any command. False commands during those periods should be measured per unit time, because a low percentage can still create an intolerable number of unintended actions during hours of use.

Nagel and Spüler demonstrated why this deserves its own metric. In an asynchronous visual-evoked-potential speller, they separately evaluated intentional-control and non-control states, reporting 0.075 erroneous classifications per minute during non-control, an average information-transfer rate of 122.7 bits per minute and 16.1 correct case-sensitive letters per minute. These figures describe one ten-participant speller study, not a universal BCI capability. Their value here is methodological: target accuracy, unintended activation and realised output rate were measured separately. Nagel and Spüler, Scientific Reports (2019)

For systems that can cause physical or consequential actions, the minimum reporting set should include accepted-command accuracy, rejection rate, missed intentions, false commands per unit time and the time required to stop or correct an erroneous action. The acceptable balance depends on the application. A spelling error, a missed rehabilitation cue and an unintended device movement do not carry the same cost.

Average performance hides user variability

EEG performance is not uniformly distributed across users. A group mean can describe the study sample while saying little about the people at the lower end of the distribution.

In a one-session sensorimotor-rhythm BCI study of 80 BCI-naive healthy participants, Blankertz and colleagues reported mean feedback accuracy of 74.4% with a standard deviation of 16.5%. Individual performance spanned the complete range from chance level at 50% to 100%. Performance also varied across runs within participants. This was a specific two-class, first-session study; it does not establish failure rates for other paradigms or clinical groups. It does show why an average alone is inadequate. Blankertz et al., NeuroImage (2010)

The public BCI Competition IV makes the same point under a session-to-session task. On Data Set 2b, the winning submission achieved a mean kappa of 0.60 across nine participants, while its participant-level values ranged from 0.21 to 0.95. Kappa was defined so that zero represented chance agreement. The competition report also noted that no submitted method achieved good results for every participant. Tangermann et al., Frontiers in Neuroscience (2012)

The correct conclusion is not that some people are “bad BCI users.” A weak or poorly separable measured signal can reflect physiology, sensor contact, environmental noise, fatigue, task design or a mismatch between the user and the control paradigm. The evaluation target is the complete human-device system.

Weak-signal users must remain visible in the evidence. A professional result should publish the participant-level distribution, median, dispersion, lower-tail performance and the number of users for whom the system did not meet a predeclared operating threshold. Participant exclusions and failed sessions should be reported with reasons. Removing difficult cases after seeing their performance changes the population to which the result applies.

Calibration burden is part of performance

User-specific calibration consumes time and effort before useful control begins. If it must be repeated because the signal changes between sessions, it becomes a recurring operating cost rather than a one-time laboratory procedure.

That burden has several components and they should not be collapsed into one number:

• physical setup time, including sensor placement and signal-quality checks;
• labelled calibration time, measured in both minutes and trials;
• user instruction or training time;
• computation or model-update time; and
• recalibration frequency, including the conditions that trigger it.

The phrase “zero calibration” is meaningful only when its boundary is explicit. It might mean no labelled data from the current session, no data from the target user, or no fitted parameters of any kind. Those are different claims. A system trained on other users or previous sessions can be zero-shot for the current user while still depending on substantial prior data.

Calibration-free operation is possible in particular paradigms, but it is not a generic property of BCI. Thielen and colleagues systematically reduced training data for a code-modulated visual-evoked-potential speller to a zero-training condition and tested that condition online. The result establishes feasibility for that task and design; it does not remove the need to measure calibration for motor-imagery, P300, clinical or other systems. Thielen et al., Journal of Neural Engineering (2021)

Useful-time efficiency should be reported alongside accuracy: the time from the start of setup to the first valid command, the fraction of a session spent producing useful output and the amount of intervention required from an operator.

Accuracy without time is not throughput

Longer observation windows often provide more evidence for classification. They also delay each decision. A system can improve accuracy by waiting longer while reducing the number of useful decisions a person can make.

A public 40-class steady-state visual-evoked-potential dataset illustrates the tradeoff directly. Across 40 participants and six sessions, one evaluated method's mean accuracy increased from 75.23% with a one-second EEG epoch to 96.37% with a three-second epoch. Under the study's definition of command time, its calculated information-transfer rate moved in the opposite direction: from 128.69 to 85.26 bits per minute. The result is specific to that dataset and evaluation, but the arithmetic is general: accuracy can be purchased with latency. Kim et al., Scientific Data (2025)

Information-transfer rate (ITR) is useful because it combines the number of available choices, classification accuracy and decision time. It is not a complete measure of practical utility. The standard calculation assumes a particular relationship among target frequency, independent selections and errors. Those assumptions may not describe natural language, error correction, hierarchical menus or continuous control.

Speier and colleagues re-evaluated 11 published BCI communication studies and found that the choice of metric could change system optimisation and study comparisons. Standard ITR assumptions about equally likely symbols and memoryless selections were particularly problematic for language systems. Speier et al., PLOS ONE (2013)

ITR should therefore be accompanied by task-level measures: correct selections or characters per minute, end-to-end task completion time, correction overhead, false activations and fatigue over sustained use. The metric must match what the user is trying to accomplish.

Cross-session stability is the deployment test

Within-session validation asks whether a system can classify data recorded under closely related conditions. Deployment usually asks a harder question: will it still work after the user returns, the sensors are fitted again and the signal distribution has changed?

A 2022 public motor-imagery dataset was designed specifically to expose that difference. It contains 25 participants, five sessions recorded on different days and 100 left- versus right-hand imagery trials per session. In the authors' subject-specific benchmarks, the best reported within-session condition reached 68.8% average accuracy, while cross-session classification fell to 53.7%. A cross-session adaptation condition reached 78.9%, but adaptation is a different operating claim from transfer without updates. Ma et al., Scientific Data (2022)

These numbers are not a ceiling for motor-imagery BCI. They quantify one dataset, several benchmark conditions and a particular validation design. Their importance is the size of the evaluation gap: testing on another day can answer a materially different question from splitting trials recorded on the same day.

Cross-session evidence should keep complete future sessions out of every data-dependent development step. The report should state whether labels from the new session were used, how much adaptation occurred, when performance was measured relative to adaptation and whether the operating threshold changed. “Cross-session” without those details is not reproducible as an evaluation claim.

A defensible BCI scorecard

There is no single replacement metric for accuracy. The required evidence is a scorecard tied to the intended use.

• Protocol: BCI paradigm, sensing modality, classes, class balance, cueing, no-control periods and chance baseline.
• Participants: population, sample size, participant-level results, exclusions and failed sessions.
• Validation: trial, session and user boundaries; prospective or retrospective testing; offline or closed-loop operation.
• Decision quality: confusion matrix, accepted-decision accuracy, rejection rate, coverage and missed intentions.
• Confidence: a proper scoring rule and reliability analysis on genuinely held-out data.
• Unintended control: false commands per unit time during representative non-control activity.
• Speed and utility: decision latency, ITR with its assumptions, correction cost and realised task output.
• Burden: setup, labelled calibration, user training, computation and recalibration frequency.
• Stability: degradation across sessions and time, together with the data and interventions used for adaptation.
• Robustness: signal-quality failures, artifacts and operating conditions that place the system outside its validated range.

This scorecard also prevents incomparable results from being compressed into a leaderboard. A two-class motor-imagery task, a 40-target visual speller and a continuous cursor-control system do not become comparable because each publishes an accuracy percentage.

What the public benchmarks do not prove

The studies above establish specific measurement problems. They do not establish the performance of every BCI.

Most cited results involve non-invasive EEG and controlled experimental tasks. Several use healthy participants. Offline replay does not reproduce the feedback, learning and behavioural adaptation of a closed-loop interface. Results from visual-evoked potentials do not transfer automatically to motor imagery, and neither establishes clinical effectiveness. Public benchmark performance also says nothing about a separate system unless that system is tested under a declared, comparable protocol.

For medical, rehabilitative or assistive use, evaluation must include the intended user population and the outcomes that matter to that use. Classification performance is an engineering measure, not by itself a clinical benefit.

The Mondren perspective

At Mondren, BCI and interpretable neural-signal systems are a frontier research area. We evaluate the problem as a human-machine decision system, not as a classifier leaderboard.

This article reports no Mondren BCI benchmark, clinical validation or deployment-readiness result. Every numerical result cited above comes from external public research. Any future performance claim from us should identify the users, sessions, task, sensing hardware, calibration, rejection policy, throughput measure and validation split that define it.

The public principle is straightforward: accuracy is one component of evidence. Confidence, abstention, user coverage, calibration burden, useful information rate and stability determine whether a result can support an interface. Those evaluation criteria can be public. Classifier construction and the implementation used to produce a decision remain proprietary.

The commercial implication

A buyer or development partner should not ask only, “What accuracy does the BCI achieve?” The better question is, “What useful action can the intended user complete, at what rate and burden, and what happens when the evidence is insufficient?”

That question forces the operating requirement to become explicit. It identifies the cost of a false command, a missed intention, a delayed output and a rejected decision. It exposes how much supervised setup is acceptable and how often it may recur. It also determines whether the relevant test is a same-day demonstration, repeated home use, clinical workflow or sustained control in another real environment.

Higher accuracy remains valuable when it is measured honestly. It is simply not sufficient.

A BCI is not a classifier with electrodes. It is a closed-loop system in which a person, a sensor, a decision process and an application must work together over time. The decisive result is not the best average obtained on labelled trials. It is reliable, usable control for the intended users under the conditions in which the interface is expected to operate.

This article discusses research and engineering evaluation. It is not medical advice and does not make a claim of clinical safety, efficacy or benefit.