Skip to main content

Research and Publications

Errors on the voice side follow this path back to the decoderTraining so there is no need to pair neural signals with voice annotations

Drag horizontally to see the full process →

Click on any stop to see what this step is doing

01cortical signalECoG

Electrode arrays are applied to the surface of the cortex to record high gamma activity while speaking. Each subject has a different electrode position, which is the first problem to be dealt with for this type of work.

icon buttonA neural speech decoding framework leveraging deep learning and speech synthesis(Nature Machine Intelligence, 2024). The parameter names and values are taken from the original text.

Chen Xupeng

Xupeng ChenCTO

Ph.D., Electrical Engineering, NYU

NYU Tandon School of Engineering · Advisors: Yao Wang, Adeen Flinker

Google Scholar

Top 5% of all research outputs scored by Altmetric

Publications

Research Atlas

Jump to the corresponding text at any point on the dot diagram

Neural signals and speechFrom cortical signals to spoken wordsMultimodality and evidenceLook in the right place, then answerReasoning and instruction followingMore reasoning does not always mean better instruction following20202021202220232024202520262024 A neural speech decoding framework leveraging deep learning and speech synthesis2026 Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models2026 Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering2026 Comparison of sEMG Encoding Accuracy Across Speech Modes Using Articulatory and Phoneme Features2025 VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI2025 Machine Learning-Based Prediction of Speech Arrest During Direct Cortical Stimulation Mapping2025 When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs2025 Transformer-based neural speech decoding from surface and depth electrode signals2024 R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest2024 A corollary discharge circuit in human speech2023 Distributed feedforward and feedback cortical processing supports human speech production2020 Stimulus Speech Decoding From Human Cortex with Generative Adversarial Network Transfer Learning

Horizontal drag to view full atlas →

2024Neural signals and speech

A neural speech decoding framework leveraging deep learning and speech synthesis

Nature Machine Intelligence · Vol. 6, No. 4, pp. 467–480

  • Journals and conferences 5 articles
  • 7 preprints
  • Award-winning or high-profile

12 articles · 2020 to 2026 · Three lines

  1. 2024Top 5% of all research outputs (Altmetric)

    A neural speech decoding framework leveraging deep learning and speech synthesis

    Xupeng Chen, Ran Wang, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker

    Nature Machine IntelligenceVol. 6, No. 4, pp. 467–480doi:10.1038/s42256-024-00824-8

    An ECoG decoder that translates cortical signals into interpretable speech parameters, paired with a differentiable speech synthesizer. Reproducible across 48 participants; the 3D ResNet decoder reaches PCC 0.804 against the original spectrogram, and stays high under causal-only operation as required for real-time neural prostheses.

    ECoG decoder outputs speech parameters, which are restored to the spectrum by a differentiable speech synthesizer, compared with the original speech spectrum, and then backpropagated frame diagram

    Drag horizontally to view full page →

    Fig. 1Closed loop of the whole frame: The ECoG decoder translates the high gamma signal of the cortex into an interpretable set of speech parameters, which can then be reduced to spectrum by the differentiable speech synthesizer. The synthesizer is differentiable, so the loss of the speech side can be transmitted back to the decoder - training does not require a pair of "neural signal — speech" annotations to hard align.Original picture ↗·CC BY 4.0
    The original speech spectrum is compared word by word with the decoded spectrum, and the harmonic structure of eight English words corresponds one by one.

    Drag horizontally to view full page →

    Fig. 2c–dAscending is the spectrum of the eight words actually spoken by the subject, and descending is the spectrum decoded only by the cortical signal. The harmonic structure, start and end time, and turbidity change were all correct - the correlation coefficient between the 3D ResNet decoder and the original spectrum reached 0.804 on 48 subjects.Original picture ↗·CC BY 4.0
    Heat map of the left cerebral cortex showing the decrease of the decoding correlation coefficient after masking each area, the heat zone is concentrated in the sensory motor cortex and the superior temporal gyrus

    Drag horizontally to view full page →

    Fig. 4Covering a certain cortex, how much quality is lost in decoding - the more it falls, the darker the color. The hot zone is concentrated in the sensory motor cortex and the superior temporal gyrus, and is stable under both decoders, causal and non-causal settings. This conclusion does not rely on model selection, indicating that the model has learned something that makes physiological sense.Original picture ↗·CC BY 4.0
  2. 2026

    Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

    Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

    arXiv preprintarXiv:2604.27720

    Asks why telling a vision-language model where to look often makes its answer worse. Benchmarks frontier VLMs on medical grounding, separates the two failure modes — the box lands in the wrong place, or the box is right but cropping to it throws away the context the answer needed — and shows what fine-tuning does and does not fix.

  3. 2026

    Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

    Xupeng Chen, Binbin Shi, Chenqian Le, Jiaqi Zhang, Kewen Wang, Ran Gong, Jinhan Zhang, Chihang Wang

    arXiv preprintarXiv:2604.27724

    MedVRAG retrieves over ~350K page images from the literature rather than over plain text, then decides for itself whether the evidence it pulled is enough — and goes back for more if it is not. Most questions settle in one round; the hard ones are exactly the ones that need a second and third.

  4. 2026

    Comparison of sEMG Encoding Accuracy Across Speech Modes Using Articulatory and Phoneme Features

    Chenqian Le, Ruisi Li, Beatrice Fumagalli, Yasamin Esmaeili, Xupeng Chen, Amirhossein Khalilian-Gourtani, Tianyu He, Adeen Flinker, Yao Wang

    arXiv preprintarXiv:2604.18920

    Surface EMG from the face and neck, compared across spoken, mimed and silently imagined speech. Asks how much of the articulatory signal survives when no sound is produced — the question any silent-speech interface has to answer before it can work.

  5. 2025

    VoxelFormer: Parameter-Efficient Multi-Subject Visual Decoding from fMRI

    Chenqian Le, Yilin Zhao, Nikasadat Emami, Kushagra Yadav, Xujin "Chris" Liu, Xupeng Chen, Yao Wang

    arXiv preprintarXiv:2509.09015

    Decodes what a person is looking at from fMRI, with one model shared across subjects instead of one model per brain. A token-merging encoder maps voxels into a CLIP embedding space, so adding a new subject costs a fraction of the parameters a per-subject model would.

  6. 2025

    Machine Learning-Based Prediction of Speech Arrest During Direct Cortical Stimulation Mapping

    Nikasadat Emami, Amirhossein Khalilian-Gourtani, Jianghao Qian, Antoine Ratouchniak, Xupeng Chen, Yao Wang, Adeen Flinker

    arXiv preprintarXiv:2509.08703

    Predicts which cortical sites will arrest speech when stimulated — the mapping a surgeon currently has to establish by stimulating the awake patient site by site. Same signal, read ahead of time instead of found the hard way.

  7. 2025

    When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs

    Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, Anurag Beniwal

    arXiv preprintarXiv:2505.11423

    Chain-of-thought prompting is supposed to make models better. On instruction following it frequently makes them worse: the model reasons its way past a constraint it would have obeyed if asked directly. Measures where the trade-off bites and what mitigates it.

  8. 2025

    Transformer-based neural speech decoding from surface and depth electrode signals

    Junbo Chen, Xupeng Chen, Ran Wang, Chenqian Le, Amirhossein Khalilian-Gourtani, Erika Jensen, Patricia Dugan, Werner Doyle, Orrin Devinsky, Daniel Friedman, Adeen Flinker, Yao Wang

    Journal of Neural EngineeringVol. 22, No. 1, 016017doi:10.1088/1741-2552/adab21

    SwinTW, a transformer that works with arbitrarily positioned electrodes by using their 3D cortical coordinates rather than a 2D grid. Subject-specific models on low-density 8×8 ECoG reach PCC 0.817 across 43 participants, freeing future speech prostheses from grid-only electrode layouts.

  9. 2024

    R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest

    Xupeng Chen, Zhixin Lai, Kangrui Ruan, Shichu Chen, Jiaxiang Liu, Zuozhu Liu

    arXiv preprintarXiv:2410.20327

    Injects the region a clinician would actually look at straight into the visual token stream, so the model reasons over the lesion instead of over the whole slide. The line of work that the 2026 grounding benchmark above grew out of, and argued with.

  10. 2024

    A corollary discharge circuit in human speech

    Amirhossein Khalilian-Gourtani, Ran Wang, Xupeng Chen, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker

    Proceedings of the National Academy of Sciences (PNAS)Vol. 121, No. 50, e2404121121doi:10.1073/pnas.2404121121

    Identifies a reproducible corollary discharge source in ventral speech motor cortex that fires before articulation and predicts how strongly auditory cortex is suppressed during speech — how the brain tells its own voice apart from the world.

  11. 2023

    Distributed feedforward and feedback cortical processing supports human speech production

    Ran Wang, Xupeng Chen, Amirhossein Khalilian-Gourtani, Leyao Yu, Patricia Dugan, Daniel Friedman, Werner Doyle, Orrin Devinsky, Yao Wang, Adeen Flinker

    Proceedings of the National Academy of Sciences (PNAS)Vol. 120, No. 42, e2300255120doi:10.1073/pnas.2300255120

    Uses causal, anticausal and noncausal convolutional architectures to disentangle motor control from sensory processing during speech, revealing a mixed feedforward/feedback recruitment across perisylvian cortex.

  12. 2020Best Paper Award Finalist

    Stimulus Speech Decoding From Human Cortex with Generative Adversarial Network Transfer Learning

    Ran Wang, Xupeng Chen, Amirhossein Khalilian-Gourtani, Zhaoxi Chen, Leyao Yu, Adeen Flinker, Yao Wang

    IEEE International Symposium on Biomedical Imaging (ISBI)

    Applies GAN transfer learning to decode heard speech from cortical recordings, addressing the scarcity of paired neural-speech data.

The 3 original drawings on the page are taken from the papers listed above, all of which are open access, licensed under CC BY 4.0, and the signed and licensed links are shown in the drawings. The layout is scaled by the width of the layout, and only the sub-panel of the individual diagrams has been taken, and the pixel color has not been changed. The remaining papers are copyrighted by the publisher and are not authorized to be reproduced.

From Papers to Products

Academic research and implementation

01

Leave unsaid words

Research

During his PhD, CTO and Chief Scientist Xupeng Chen focused on decoding speech from neural signals, producing interpretable speech outputs that reproduce stably. We carry the same direction forward: recovering thoughts that have formed before a person speaks. That work is part of Dimension Gate’s technical DNA.

fall on the product

This is where our phrase “eight bits per second” comes from. Today's products do not reach the cortex, but the direction is the same: MinuteX first retains the whole part of the speech, and then pushes it where it is vague, half said, and did not have time to say it at that time.

Corresponding paperNature Machine Intelligence 2024Journal of Neural Engineering 2025PNAS 2024arXiv:2604.18920arXiv:2509.08703

02

Make the big model’s ears and eyes clear

Research

In the past two years, this line has been multimodal: the model has been answered correctly, which does not mean that it has seen the right place. We separated and evaluated "where to look" and "what to answer" separately, and found that the frontier model was often wrong in medical images, but the answer was very similar; the retrieval was also changed from plain text to page image retrieval, so that the evidence could be pointed at.

fall on the product

Any conclusion, as long as it is read from the image, cannot bypass the same question: it must be able to point back to which piece it saw. This is the origin of the set of things traced by the evidence in the product. Can't trace the conclusion, we don't let it appear on the interface.

Corresponding paperarXiv:2604.27720arXiv:2604.27724arXiv:2410.20327arXiv:2509.09015

03

Being able to reason does not mean being able to understand words

Research

We want the model to think before answering, but this makes instruction compliance worse. This article reveals how the model's excessive reasoning time overturns the original constraints and how to alleviate the cognitive load caused by the model's excessive reasoning.

fall on the product

In a multi-agent system, one agent can act on its own and pass errors down the chain. We therefore make correction explicit: CONFIRM, CORRECT, DISMISS, and UPSERT. This is how frontier research becomes product practice.

Corresponding paperarXiv:2505.11423