facebookresearch/brain2qwerty

Non-invasive decoding of typed sentences from MEG and EEG brain recordings using a convolutional encoder, transformer, and character-level language model.

What it solves

Brain2Qwerty addresses the challenge of decoding natural sentences from non-invasive brain recordings, allowing for the same-time reconstruction of text from brain activity while a person is typing.

How it works

The project uses non-invasive brain recordings (specifically MEG scanners) to interpret brain activity and translate it into typed sentences. It provides two versions of the code (v1 and v2) to implement these decoding processes.

Who it’s for

Researchers in neuroscience and AI who are interested in brain-computer interfaces (BCIs) and the decoding of human brain activity into text.

Highlights

  • Non-invasive decoding of typed sentences.
  • Based on research published in Nature Neuroscience.
  • Supports multiple versions of the decoding models (v1 and v2).
  • Integrates with NeuralSet and NeuralTrain infrastructure.

Related

  • Dispatch
    Meta Brain2Qwerty: Non-Invasive Brain-to-Text Communication

    Meta's Brain2Qwerty research demonstrates an improved non-invasive method for translating brain waves into text, utilizing open-source code and datasets to advance human-computer interaction.

  • Project
    braindecode/braindecode

    Braindecode is a Python toolbox that brings deep‑learning models to raw brain‑signal data (EEG/ECoG/MEG). It provides dataset loaders, preprocessing, several ready‑made neural‑network architectures, and visualisation utilities, all built on top of PyTorch and MNE‑Python. Installable via `pip install braindecode`, it targets neuroscientists and ML researchers working with neurophysiological data.

  • Project
    huggingface/sentence-transformers

    Sentence‑Transformers is a Python library that wraps Hugging Face Transformers to let you compute, fine‑tune, and use dense, sparse, or token‑level text embeddings and Cross‑Encoder rerankers. With a single `SentenceTransformer` or `CrossEncoder` object you can generate vectors, score similarity, and train custom models, leveraging a catalog of >15 000 pre‑trained models for many languages and tasks such as semantic search, paraphrase mining, clustering, and multilingual retrieval.

  • Project
    wenet-e2e/WeTextProcessing

    A production-ready text processing toolkit for text normalization and inverse text normalization, supporting multiple languages for speech-related AI applications.

  • Dispatch
    Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models – Hugging Face Blog Summary

    Hugging Face’s blog post explains how to warm-start encoder-decoder models using pre‑trained BERT, RoBERTa, or GPT2 checkpoints, showing that this approach matches the performance of large pre‑trained seq2seq models while cutting training cost.