By: Arnaud Delorme, PhD, NITRC Domain Expert (https://doi.org/10.18116/7cg8-4951)

The "multiverse" approach in the context of Neuroscience analysis is a concept borrowed from statistical analysis and research methodologies, particularly in the fields of psychology and neuroscience. While specific to Neuroscience, the technique can be applied to the processing of fMRI, and EEG, MEG, and other physiological signals. In this article, we will focus on EEG as an example application. 

In research, a multiverse analysis refers to exploring multiple analytical scenarios for the same dataset. Instead of committing to a single method of analysis, researchers explore a range of different analytical choices to understand how these choices affect the study's outcomes. This approach acknowledges that different analytical paths can lead to different results, and it aims to provide a more comprehensive view of the data. For example, is it better to high-pass filter the data at 0.1 Hz or 0.5 Hz (see figure).

Why multiverse?

EEG data is complex and multidimensional, with many potential ways to preprocess, analyze, and interpret the signals. This includes choices like filtering methods, artifact rejection strategies, feature extraction techniques, and statistical analyses. In a multiverse approach, an EEG researcher might explore various preprocessing steps (e.g., different ways of handling artifacts like eye blinks or muscle movements), different methods of segmenting the data (e.g., time windows), and different statistical tests or machine learning algorithms for interpretation. By examining the outcomes across these different scenarios, researchers can better understand the robustness of their findings. If an outcome is consistent across many different analytical paths, it can be considered more reliable. If the outcome varies significantly with different methods, this variability needs to be understood and reported.

The importance of defining good metric

Defining good metrics is crucial because metrics are how we quantify the success, effectiveness, or quality of an experiment, model, or method. Taking the example from the paper titled "EEG is better left alone," (Delorme, 2022), where the authors used 'the number of significant channels' as a metric, we can understand the rationale behind selecting this specific metric. In EEG (Electroencephalography) studies, data is collected from multiple channels placed across the scalp. Each channel records electrical activity from different regions of the brain. The choice of using the number of significant channels as a metric likely stems from the goal of the study, which is to maximize significance. Other metrics could involve the amplitude of some ERPs or the deviation from the mean response (Clayson et al., 2021). Another metric could be how close automated data rejection and cleaning are to human rejection (Delorme and Martin, 2021).

Cross-validation 

Cross-validation is a vital technique in statistical modeling, particularly in fields like machine learning, where the performance of a model must be accurately assessed. At its core, cross-validation is about evaluating how well a model generalizes to an independent dataset. Traditionally, when a model is trained on a particular set of data, there's always the risk that it will overfit - that is, it becomes too tailored to the specific quirks of that data and fails to perform well on new, unseen data. Cross-validation addresses this problem by using different portions of the data to train and test the model in multiple rounds, ensuring that the model is robust and performs consistently across different data samples.

What does it mean in the context of a multiverse analysis? Cross-validation helps in tuning hyperparameters, which are the configuration settings used to optimize data processing performance. By evaluating the model's performance for various hyperparameters across multiple folds, one can find the most optimal set of hyperparameters that provide the best generalization performance. Concretely, it means that we will optimize processing on one set of data (the training data) and assess performance on another set of data (the testing data).

Conclusion

Multiverse analyses, when done properly with cross-validation, offer multiple advantages:

1. Transparency and robustness: This approach increases the transparency of the research process and helps identify the most robust and reproducible findings.
   
2. Reduces the risk of cherry-picking parameters or falling prey to confirmation biases, as researchers are not just presenting the analysis that worked best but are showing a range of possible outcomes. 
   

It does present some challenges, as conducting a multiverse analysis can be time-consuming and computationally intensive.

In summary, the multiverse approach in EEG analysis explores a wide range of analytical choices to understand how these choices impact the results. This approach ensures that the conclusions drawn from EEG data are robust and not overly dependent on specific analytical decisions.

References

A. Delorme and J. A. Martin, "Automated Data Cleaning for the Muse EEG," 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Houston, TX, USA, 2021, pp. 1-5, doi: 10.1109/BIBM52615.2021.9669415.

Delorme A. (2023). EEG is better left alone. Scientific reports, 13(1), 2372. https://doi.org/10.1038/s41598-023-27528...

Clayson, P. E., Baldwin, S. A., Rocha, H. A., & Larson, M. J. (2021). The data-processing multiverse of event-related potentials (ERPs): A roadmap for the optimization and standardization of ERP processing and reduction pipelines. NeuroImage, 245, 118712. https://doi.org/10.1016/j.neuroimage.202...

Quarterly Newsletter Article from February 2024