BRAINMATTER Reference · Updated 2026-07-19
How modern neuroscience, cognitive science, and AI research actually get done — the tools, their strengths, and their limits, in plain English.
Frame a testable hypothesis about brain, cognition, or model behavior.
Choose methods whose temporal, spatial, and causal resolution match the question.
Lock the analysis plan before data collection to protect against p-hacking.
Run the experiment or training runs with documented protocols.
Apply the prespecified analysis; report effect sizes and uncertainty, not just p-values.
Independent replication is the actual signal that a finding is real.
Publish with data, code, and materials — findings without these are provisional.
What: Measures blood-oxygen-level changes as a proxy for neural activity across the whole brain.
Strengths: Non-invasive, whole-brain coverage, good spatial resolution (~1–3 mm).
Limits: Slow (seconds), indirect signal, expensive; interpretation depends heavily on task design.
What: Records electrical potentials at the scalp from populations of cortical neurons.
Strengths: Millisecond temporal resolution, cheap, portable, safe for repeated use.
Limits: Poor spatial resolution; signal dominated by superficial cortex; noisy.
What: Measures the magnetic fields produced by neural currents.
Strengths: Millisecond timing plus better spatial localization than EEG.
Limits: Requires shielded room; expensive; limited to cortex.
What: Electrodes placed in tissue record action potentials from individual or small groups of neurons.
Strengths: The gold standard for causal, cellular-resolution neural activity.
Limits: Invasive; small sample of neurons; mostly animal models or clinical BCI patients.
What: Genetically expressed light-sensitive channels let researchers activate or silence specific neurons with light.
Strengths: Causal control of specific cell types with millisecond precision.
Limits: Requires genetic modification; animal models only.
What: Reconstructing wiring diagrams of neural tissue from serial electron microscopy or expansion microscopy.
Strengths: Ground-truth circuit structure; enables constrained models.
Limits: Enormous compute and storage; snapshots only; alignment with function is hard.
What: Measuring what people or animals do — reaction times, error rates, choices — under controlled task conditions.
Strengths: Cheap, scalable, directly measures cognition.
Limits: Behavior alone underdetermines internal mechanism; needs converging methods.
What: Precise measurement of the relationship between physical stimuli and perceptual reports.
Strengths: Quantitative, replicable, decades of established paradigms.
Limits: Restricted to what can be operationalized as a discrimination task.
What: Random assignment of participants to intervention vs control to establish causal effects (drugs, therapies, digital interventions).
Strengths: Gold standard for causal inference in clinical brain research.
Limits: Expensive, slow; external validity limited by inclusion criteria.
What: Standardized datasets and scoring rubrics used to compare model capabilities (MMLU, ARC-AGI, HumanEval, GPQA, SWE-bench).
Strengths: Enable apples-to-apples comparison and track progress over time.
Limits: Overfitting, test-set contamination, and Goodhart's law — a benchmark measures the benchmark, not the underlying capability.
What: Empirical laws relating model performance to compute, parameters, and data.
Strengths: Highly predictive for pretraining loss; enables planning frontier runs.
Limits: Do not predict downstream capabilities well; extrapolation risks.
What: Adversarial testing for safety-relevant behavior (harmful outputs, deception, capability elicitation).
Strengths: Uncovers failure modes standard benchmarks miss.
Limits: Coverage depends on red-team creativity; negative results don't prove safety.
What: Selectively removing or replacing components (attention heads, neurons, layers, training data) to test their contribution.
Strengths: Directly identifies necessary components; the closest AI has to a causal method.
Limits: Confounded by redundancy and re-routing; ablated model behavior can differ subtly from the original circuit.
What: Reverse-engineering the algorithms implemented by neural networks (circuits, features, superposition).
Strengths: Provides mechanistic accounts, not just correlations; transfers ideas from systems neuroscience.
Limits: Scales slowly; results on small models may not generalize; heavy interpretive labor.
What: Recording and stimulating neural tissue to decode intent or restore function in humans.
Strengths: Direct human neural data at high channel counts; closes the loop with cognition.
Limits: Invasive; small patient populations; hardware constraints dominate what is measurable.
No single method captures brains or models fully. fMRI has spatial resolution but poor timing; EEG is the reverse. Benchmarks measure capability aggregates; ablations expose mechanism. Robust conclusions come from converging evidence across methods.
Look for preregistration, released data and code, independent replication, appropriate uncertainty estimates, and honesty about limits. Extraordinary claims from single studies without these are provisional at best.
Only directionally. Benchmarks are subject to Goodhart's law, test-set contamination, and prompt sensitivity. Treat a benchmark score as one data point in a portfolio that includes ablations, held-out evaluations, and mechanistic analysis.
fMRI, EEG, and observational studies produce correlations. Optogenetics, lesions, ablations, RCTs, and (imperfectly) interventions in models produce causal evidence. The strongest claims combine both.
