Advances in Experimental Medicine and Biology 1101

Xiaoxiang Zheng Editor

Neural
Interface:
Frontiers and
Applications
Advances in Experimental Medicine
and Biology

Volume 1101

Series Editors
Wim E. Crusio, CNRS University of Bordeaux UMR 5287, Institut de
Neurosciences Cognitives et Intégratives d’Aquitaine, Pessac Cedex, France
John D. Lambris, University of Pennsylvania, Philadelphia, PA, USA
Nima Rezaei, Children’s Medical Center Hospital, Tehran University of
Medical Sciences, Tehran, Iran
More information about this series at http://www.springer.com/series/5584
Xiaoxiang Zheng
Editor

Neural Interface: Frontiers

and Applications
Editor
Xiaoxiang Zheng
Qiushi Academy for Advanced Studies
Zhejiang University
Hangzhou, Zhejiang, China

ISSN 0065-2598 ISSN 2214-8019 (electronic)

Advances in Experimental Medicine and Biology
ISBN 978-981-13-2049-1 ISBN 978-981-13-2050-7 (eBook)
https://doi.org/10.1007/978-981-13-2050-7

© Springer Nature Singapore Pte Ltd. 2019

This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

Neural interfaces interact directly with the central nervous system (CNS) or periph-
eral nervous system (PNS) to restore brain function impaired by neural diseases or
trauma or to enhance motor or cognitive functions for the abled individuals. The
technology interfacing the brain is also known as brain-computer interface (BCI) or
brain-machine interface (BMI). While the cochlear prosthesis and deep brain stim-
ulation for Parkinson’s disease have become reality, other neural interface applica-
tions are yet fully matured for decades. However, in recent years we have witnessed
the great advances in this field, and the advances in neuroscience and engineering are
speeding up neural interface technology, opening the door to assist, augment, repair,
or restore sensorimotor or other cognitive functions and thus improving the quality
of life for the individuals with disabilities. Medical applications such as NeuroPace
for seizure detection/suppression and spinal cord stimulation for pain management
are now becoming increasingly common. Neural interfaces are now being explored
in applications as diverse as rehabilitation, accessibility, gaming, education, recrea-
tion, robotics, and human enhancement.
Neural interfaces also represent a powerful tool to address fundamental questions
in neuroscience. We also have witnessed tremendous advancements in the field of
neural interface with high impact not only in the development of neuroprosthetics
but also in our basic understanding of brain function. Neural interface technology
can be seen as a bridge linking engineering and neuroscience.
This book focuses on frontiers of neural interface technology, including hard-
ware, software, neural decoding and encoding, control system, and system integra-
tion. It also covers the respects of applications in neuroprosthetics, neural diseases
treatment, neurorobotics, and also the research paradigms for basic neuroscience.
This book aims at providing the researchers, graduate students, and upper under-
graduate students in a wide range of disciplines with a cutting-edge and compre-
hensive summary of researches on neural interfaces.
Editing such a book is tough, yet delightful, challenging, and fundamentally a
team effort. Thus, I am in gratitude to a bevy of individuals without whom we could
not have hoped to succeed. First and foremost, I sincerely thank Dr. Peng Zhang, a
senior editor from Springer Nature, who invited me to edit this book, for his
v
vi Preface

understanding, support, and patience. I would like to thank all the authors who
shared their works and wrote the manuscripts of their chapters. Their great contri-
bution makes up this exciting work. I also would like to thank everyone who has
helped me along the way but whom I did not mention explicitly.

Hangzhou, China Xiaoxiang Zheng

Contents

1 Advances in Penetrating Multichannel Microelectrodes

Based on the Utah Array Platform . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Moritz Leber, Julia Körner, Christopher F. Reiche, Ming Yin,
Rajmohan Bhandari, Robert Franklin, Sandeep Negi, and
Florian Solzbacher
2 EEG-Based Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . 41
Yijun Wang, Masaki Nakanishi, and Dan Zhang
3 Invasive Brain Machine Interface System . . . . . . . . . . . . . . . . . . . . . 67
Yile Jin, Junjun Chen, Shaomin Zhang, Weidong Chen, and
Xiaoxiang Zheng
4 Peripheral Neural Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
Peng Zhang, Xiao Li, Dingyin Hu, Qiuxia Lai, Yuanyuan Wang,
Xuan Ma, Qi Xu, Wei Li, Jian Huang, and Jiping He
5 Brain-Machine Interface-Based Rat-Robot Behavior Control . . . . . . 123
Jiacheng Zhang, Kedi Xu, Shaomin Zhang, Yueming Wang,
Nenggan Zheng, Gang Pan, Weidong Chen, Zhaohui Wu,
and Xiaoxiang Zheng
6 Realizing Efficient EMG-Based Prosthetic Control Strategy . . . . . . . 149
Guanglin Li, Oluwarotimi Williams Samuel, Chuang Lin,
Mojisola Grace Asogbon, Peng Fang, and Paul Oluwagbengba Idowu
7 Neural Interface: Frontiers and Applications . . . . . . . . . . . . . . . . . . 167
Xiaoan Sun, Sui Huang, and Ningyuan Wang
8 Neuromodulation for Pain Management . . . . . . . . . . . . . . . . . . . . . . 207
Jing Wang and Zhe Chen
9 Future of Neural Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Farah Laiwalla and Arto Nurmikko

vii
About the Editor

Xiaoxiang Zheng is a professor at the Qiushi Academy for Advanced Studies,

Zhejiang University. She is currently a member of the academic board of Key
Laboratory for Biomedical Engineering of Ministry of Education at Zhejiang Uni-
versity and was the director of the College of Biomedical Engineering and Instru-
ment Science, Zhejiang University, and executive member of the council of the
Chinese Society of Biomedical Engineering. Dr. Zheng’s research interests include
neural engineering, microcirculation, and physiology of the cell. She has published
more than 200 articles in these fields, over 100 of which are indexed by SCI or EI.

ix
Contributors

Mojisola Grace Asogbon CAS Key Laboratory of Human-Machine Intelligence-

Synergy Systems, Shenzhen Institutes of Advanced Technology (SIAT), Chinese
Academy of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Shenzhen College of Advanced Technology, University of Chinese Academy of
Sciences, Shenzhen, China
Rajmohan Bhandari University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
Junjun Chen Qiushi Academy for Advanced Studies, Zhejiang University,
Hangzhou, China
Department of Biomedical Engineering, Zhejiang University, Hangzhou, China
Weidong Chen Qiushi Academy for Advanced Studies (QAAS), Zhejiang Univer-
sity, Hangzhou, People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education
Ministry, Zhejiang University, Hangzhou, People’s Republic of China
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China
Zhe Chen Department of Psychiatry, Department of Neuroscience and Physiology,
New York University School of Medicine, New York, NY, USA
Peng Fang CAS Key Laboratory of Human-Machine Intelligence-Synergy
Systems, Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy
of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Robert Franklin Blackrock Microsystems, Salt Lake City, UT, USA

xi
xii Contributors

Jiping He Neural Interface and Rehabilitation Technology Research Center,

School of Automation, Huazhong University of Science and Technology, Wuhan,
China
Beijing Advanced Innovation Center for Intelligent Robot and System, School of
Mechatronical Engineering, Beijing Institute of Technology, Beijing, China
Center for Neural Interface Design, School of Biological and Health Systems
Engineering, Arizona State University, Tempe, AZ, USA
Dingyin Hu Neural Interface and Rehabilitation Technology Research Center,
School of Automation, Huazhong University of Science and Technology, Wuhan,
China
Jian Huang Neural Interface and Rehabilitation Technology Research Center,
School of Automation, Huazhong University of Science and Technology, Wuhan,
China
Sui Huang Nurotron Biotechnology Inc., Irvine, CA, USA
Paul Oluwagbengba Idowu CAS Key Laboratory of Human-Machine Intelli-
gence-Synergy Systems, Shenzhen Institutes of Advanced Technology (SIAT),
Chinese Academy of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Shenzhen College of Advanced Technology, University of Chinese Academy of
Sciences, Shenzhen, China
Yile Jin Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou,
China
Department of Biomedical Engineering, Zhejiang University, Hangzhou, China
Julia Körner University of Utah, Salt Lake City, UT, USA
Qiuxia Lai Neural Interface and Rehabilitation Technology Research Center,
School of Automation, Huazhong University of Science and Technology, Wuhan,
China
Farah Laiwalla School of Engineering, Brown University, Providence, RI, USA
Moritz Leber University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
Guanglin Li CAS Key Laboratory of Human-Machine Intelligence-Synergy
Systems, Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy
of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Wei Li Neural Interface and Rehabilitation Technology Research Center, School of
Automation, Huazhong University of Science and Technology, Wuhan, China
Xiao Li Neural Interface and Rehabilitation Technology Research Center, School
of Automation, Huazhong University of Science and Technology, Wuhan, China
Contributors xiii

Chuang Lin CAS Key Laboratory of Human-Machine Intelligence-Synergy Sys-

tems, Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy of
Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Xuan Ma Neural Interface and Rehabilitation Technology Research Center, School
of Automation, Huazhong University of Science and Technology, Wuhan, China
Masaki Nakanishi Institute for Neural Computation, University of California San
Diego, San Diego, CA, USA
Sandeep Negi University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
Arto Nurmikko School of Engineering, Brown University, Providence, RI, USA
Gang Pan College of Computer Science and Technology, Zhejiang University,
Hangzhou, People’s Republic of China
Christopher F. Reiche University of Utah, Salt Lake City, UT, USA
Oluwarotimi Williams Samuel CAS Key Laboratory of Human-Machine Intelli-
gence-Synergy Systems, Shenzhen Institutes of Advanced Technology (SIAT),
Chinese Academy of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Florian Solzbacher University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
Xiaoan Sun Nurotron Biotechnology Inc., Irvine, CA, USA
Jing Wang Department of Anesthesiology, Perioperative Care and Pain Medicine,
Department of Neuroscience and Physiology, New York University School of
Medicine, New York, NY, USA
Ningyuan Wang Nurotron Biotechnology Inc., Irvine, CA, USA
Yijun Wang Institute of Semiconductors, Chinese Academy of Sciences, Beijing,
China
Yuanyuan Wang Neural Interface and Rehabilitation Technology Research Cen-
ter, School of Automation, Huazhong University of Science and Technology,
Wuhan, China
Yueming Wang Qiushi Academy for Advanced Studies (QAAS), Zhejiang Uni-
versity, Hangzhou, People’s Republic of China
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China
Zhaohui Wu College of Computer Science and Technology, Zhejiang University,
Hangzhou, People’s Republic of China
xiv Contributors

Kedi Xu Qiushi Academy for Advanced Studies (QAAS), Zhejiang University,

Hangzhou, People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education
Ministry, Zhejiang University, Hangzhou, People’s Republic of China
Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Tech-
nology and Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou,
People’s Republic of China
Qi Xu Neural Interface and Rehabilitation Technology Research Center, School of
Automation, Huazhong University of Science and Technology, Wuhan, China
Ming Yin University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
Dan Zhang Department of Psychology, Tsinghua University, Beijing, China
Jiacheng Zhang Qiushi Academy for Advanced Studies (QAAS), Zhejiang Uni-
versity, Hangzhou, People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education
Ministry, Zhejiang University, Hangzhou, People’s Republic of China
Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Tech-
nology and Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou,
People’s Republic of China
Peng Zhang Neural Interface and Rehabilitation Technology Research Center,
School of Automation, Huazhong University of Science and Technology, Wuhan,
China
Shaomin Zhang Qiushi Academy for Advanced Studies (QAAS), Zhejiang Uni-
versity, Hangzhou, People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education
Ministry, Zhejiang University, Hangzhou, People’s Republic of China
Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Tech-
nology and Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou,
People’s Republic of China
Nenggan Zheng Qiushi Academy for Advanced Studies (QAAS), Zhejiang Uni-
versity, Hangzhou, People’s Republic of China
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China
Xiaoxiang Zheng Qiushi Academy for Advanced Studies (QAAS), Zhejiang
University, Hangzhou, People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education
Ministry, Zhejiang University, Hangzhou, People’s Republic of China
Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Tech-
nology and Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou,
People’s Republic of China
Chapter 1
Advances in Penetrating Multichannel
Microelectrodes Based on the Utah Array
Platform

Moritz Leber, Julia Körner, Christopher F. Reiche, Ming Yin,

Rajmohan Bhandari, Robert Franklin, Sandeep Negi,
and Florian Solzbacher

Abstract The Utah electrode array (UEA) and its many derivatives have become a
gold standard for high-channel count bi-directional neural interfaces, in particular in
human subject applications. The chapter provides a brief overview of leading
electrode concepts and the context in which the UEA has to be understood. It goes
on to discuss the key advances and developments of the UEA platform in the past
15 years, as well as novel wireless and system integration technologies that will
merge into future generations of fully integrated devices. Aspects covered include
novel device architectures that allow scaling of channel count and density of
electrode contacts, material improvements to substrate, electrode contacts, and
encapsulation. Further subjects are adaptations of the UEA platform to support IR
and optogenetic simulation as well as an improved understanding of failure modes
and methods to test and accelerate degradation in vitro such as to better predict
device failure and lifetime in vivo.

Authors Moritz Leber, Julia Körner and Christopher F. Reiche have contributed equally to this
chapter.

M. Leber (*) · M. Yin · R. Bhandari · S. Negi · F. Solzbacher (*)

University of Utah, Salt Lake City, UT, USA
Blackrock Microsystems, Salt Lake City, UT, USA
e-mail: [email protected]; fl[email protected]
J. Körner · C. F. Reiche
University of Utah, Salt Lake City, UT, USA
R. Franklin
Blackrock Microsystems, Salt Lake City, UT, USA

© Springer Nature Singapore Pte Ltd. 2019 1

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_1
2 M. Leber et al.

Keywords Utah electrode array (UEA) · Neural interface materials · Wireless

technology · Accelerated aging · Advanced system integration

1.1 Application Areas and Challenges

Many neural interface electrode developments have been driven by government-

sponsored research programs that focused on the restoration of motor or sensory
function (neuroprosthetics). In recent years, increasing numbers of clinical research
studies under FDA investigative device exemption (IDE) approvals have been
carried out for a variety of interfaces. The Utah electrode array (UEA) is the only
penetrating high-channel count device of its kind to have received FDA 510
(k) clearance and to have been used in such studies to date. Up until recently, the
commercial sector however deemed this market to be too small and the technical
hurdles and risks as too large to warrant investment. Subchronic monitoring and
treatment of neurological disorders, such as epilepsy, and some peripheral and
autonomic nervous system applications appear to have gained momentum in the
commercial sector.
Key technological trends that can be observed are increasing channel count,
enhanced biocompatibility through materials advances (porosity, coatings, etc.),
the advent of devices for use in optogenetics and IR stimulation, advanced wireless
(implantable and external) and novel system integration, and packaging concepts for
these devices that, e.g., push the boundaries on electrode channel count into the
1000s of channels and beyond (the current Darpa NESD program targets 1 Mio
channels for an implantable wireless device covering about 2
2 cm2).
The clinical trends are driven by key opinion leaders with clinical research studies
in human patients at Brown University/Massachusetts General Hospital, Stanford,
University of Pittsburgh Medical Center, University of Pennsylvania, Chicago North-
western, Case Western Reserve University, Caltech, UCLA, Columbia, University of
Washington, and a growing number of centers in Europe. To an increasing extent,
epilepsy patients are being used as a recruiting group for a variety of clinical studies.
With increasing channel count on the hardware side, new challenges arise on the
software and data analysis side if one takes into account that the current state of the
art in analysis of the data is to search for specific features or carry out spike sorting
of action potential events. Data amounts can easily exceed terabytes from a single
subject within a 24-h period for devices in the 100-channel range. Assuming
50,000 channels, this would pose a serious challenge. Data collection, storage,
management, and also visualization, search, and analysis become intractable with
current methods. In addition, while the transition from nonhuman primates to
human subjects has great advantages, it also comes with a more stringent regula-
tory framework and Health Insurance Portability and Accountability Act (HIPPA)
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 3

regulations for protection of patient data. Future concepts for ultrahigh-channel

count neural interfaces will thus likely have to mimic some of the architecture used
in the human nervous system as well as in advanced sensor networks, where local
“processors” can receive and process neural data at high resolution and depending
on filters and parameters set by a central control unit can then either autonomously
decide to pass on, compact, or discard data. Smart contextual visualization and
search approaches could assume the role of a smart virtual assistant and become
support tools for exploration of data.

1.2 Overview of Electrodes in the Field

There is a plethora of penetrating neural micro electrodes that has been published
and demonstrated to varying degrees in vivo, mostly by academic institutions.
Microwire electrodes were among the first implantable penetrating microelectrodes
and were developed in the 1950s [25]. They are fabricated by cutting off the end of
an insulated metal wire or by laser ablation of the insulation material around the
sharpened tip of a wire [114]. With sufficient mechanical stiffness to withstand the
insertion process into neural tissue, common materials for the 25–50 μm diameter
wires are stainless steel, tungsten, iridium, and platinum-iridium alloys [25, 87,
114]. Several microwires can be assembled to an array, which is held together by
a cast material such as biocompatible epoxy [114]. Further development led to the
introduction of floating microelectrode arrays (FMAs), which are designed to move
with the brain by “floating” on the surface of the brain and being mechanically
decoupled from the skull. FMAs offer the advantage of being highly customizable
regarding individual electrode length, position, and spacing [77]. The different
electrodes are held together by lightweight substrates and are micro-welded to highly
flexible insulated wires leading to the connector (Fig. 1.1a).
Besides microwire and FMA probes, silicon micromachined electrodes and
arrays are prevalent in the field. The two main technologies include devices based
on the Michigan style shank electrodes and the Utah electrode array (UEA). Both
device types allow for high electrode density and electrode counts that are required
for neural prosthetic applications [114], and they exhibit the advantage of
micromachining batch process compatibility.
Michigan type microelectrodes consist of multiple electrode sites on a silicon
shank and are fabricated using standard microfabrication techniques such as dopant
diffusion, physical and chemical vapor deposition, photolithography, and etching
processes [34, 125]. Furthermore, it is possible to assemble individual silicon shanks
to three-dimensional array geometries [124].
In contrast, UEAs are immediately fabricated as a three-dimensional device. The
UEA technology was developed at the University of Utah in the early 1990s [22] and
is currently (as of 2019) the only FDA cleared microelectrode array for human
4 M. Leber et al.

Fig. 1.1 (a) A floating microelectrode array architecture. The 18–36 microelectrodes of arbitrary
lengths are held together by a lightweight ceramic platform and tethered to a connector by a thin and
lightweight cable. (b) The Utah electrode array: a 10
10 grid of insulated boron doped silicon
micro needles. Insulated gold wires are wire bonded before being encapsulated in medical grade
silicone to form the assembled Utah electrode array. The magnification shows an active platinum
coated electrode site at the tip of a parylene-C coated silicon shaft. (c) The Utah slant electrode
array: a modification of the UEA with variable shaft lengths to access different depths in the tissue.
(a. Image copyright and courtesy of Microprobes for Life Science. c. Image copyright and courtesy
of Blackrock Microsystems LLC)

research use. The arrays consist of typically 10
10 electrodes, which are formed by
cutting pillar geometries out of a highly p-doped silicon substrate with pre-structured
insulation sites. The contact pads for each electrode are deposited and patterned on
the back side of the wafer. A subsequent customized wet etching process sharpens
the silicon pillars and forms a dense array of silicon needles with lengths varying
from 500 to 1500 μm. The tips of the silicon needles are then metallized (typically
with platinum or iridium oxide) in a sputtering process to improve the electrochem-
ical charge transfer between the device and the surrounding biological media. To
ensure biocompatibility and to protect the array from the foreign body response
when implanted, the entire device is coated with parylene-C. Finally, the electrode
sites are formed by removing the parylene-C encapsulation coating from the upper
30–50 μm of the metallized silicon tips. Insulated gold wires are wire bonded to the
contact pads on the back of the UEA for each individual electrode and attach them to
a connector. The bond sites are subsequently encapsulated with medical grade
silicone.
Over two decades, the UEA was subject to multiple improvements and material
changes [9]. The early UEA electrodes were insulated from each other by p-n
junctions and the shafts were encapsulated with polyimide. The current UEA technol-
ogy, however, uses glass lines between the shaft sites for insulation instead of p-n
junctions and parylene-C instead of polyimide. As a modification of the standard UEA
geometry with equal shaft lengths, the Utah slant electrode array (USEA) exhibits
shaft lengths varying from 750 to 1500 μm. With the advantage of acting at different
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 5

depths in the tissue, the USEA is commonly implanted into the peripheral nervous
system [18]. Figure 1.1b displays an assembled standard UEA with constant shaft
lengths of 1000 μm and Fig. 1.1c a USEA with varying shaft lengths.
In addition to the Michigan probes and the UEA technology, recent research
efforts led to the introduction of high-density and high-channel silicon-based micro-
electrodes. Prominent technologies include the CMOS-based high-density micro-
probe arrays [100] developed at IMTEK/Germany and the neuropixels probes [56]
developed at Janelia Research Campus/USA.

1.3 Key Developments of the UEA Platform

1.3.1 Array Shape Variations

Over time, adjustments have been made to the standard UEA geometry to allow for a
variety of applications and to improve stimulation and recoding [98].
All shafts of the standard UEA have the same length (Fig. 1.2a). The arrays can be
fabricated with a widely varying number of electrodes, ranging from small 4
4
(16 electrodes) arrays to larger ones with 10
10 (100 electrodes). High-density
UEAs can comprise over 200 electrodes within the same dimensions as the standard
UEAs [106].
A commonly used variation of the UEA is the Utah slant electrode array (USEA),
which features electrodes with different lengths in one and constant length in the
other direction (Fig. 1.2b). These arrays are favorably used in stimulation and
recording in peripheral nerves as they allow to reach different axons in varying
depths within the nerve [11, 18]. Furthermore, USEA with particularly long elec-
trodes (1–9 mm compared to the usual 0.5–1.5 mm) have been reported (long
USEA), allowing specifically deep penetration into neural tissue [106].
Another development, especially for applications in retinal implants and periph-
eral nerves, are convoluted shape UEAs. In that case, although all electrodes have
the same lengths, they feature a varying penetration depth into the neural tissue due

Fig. 1.2 Different geometries of the Utah electrode array: (a) standard UEA, (b) Utah slant
electrode array, (c) and (d) convoluted shape UEA with concave and convex curvature, respectively
6 M. Leber et al.

to the electrode base being curved, either in a convex or a concave fashion (Fig. 1.2c,
d). According to Bhandari et al., this leads to a better integration of the array with the
nerve geometry, consequently less extra-neural volume where fibrotic tissue can
grow, and higher selectivity in stimulation as varying nerve sites can be reached [10].

1.3.2 Novel Architectures to Improve Tissue Response: The

Natural Buoyancy Utah Electrode Array

Recording and stimulation performance of microelectrodes as well as foreign body

response are critical factors limiting the longevity of neural interfaces and must be
addressed in the design process [3, 19, 23, 62]. Shape, size, electrode material,
tethering forces, and the implantation technique significantly influence the foreign
body response to implanted neural electrodes [35, 77].
As an example, blood pulsation and head movements cause permanent relative
motions of the brain with respect to the surrounding skull. Since the array must be
implanted in the small space between skull and brain, it is subjected to these
relative movements. Ideally, it would be advantageous for the electrodes to be
mechanically decoupled from one another and to be able to float on the cerebral
cortex. Minimizing the micromotions and resulting friction between the brain
tissue and the implanted electrodes would reduce the risk of inflammation and
the formation of an insulating sheath around the electrodes, leading to device
insulation and failure. Numerical studies have shown that causes for severe foreign
body response can be limited by flexible and/or soft implants [67, 109]. Hence, by
eliminating the rigid backplane of the UEA, the mechanical mismatch between
implant and tissue would be significantly reduced. Nevertheless, the backplane
holding all 100 electrodes of the UEA in place is indispensable for the device
handling and the high velocity insertion process during surgery. A dissolving and
biodegradable backplane offering the required mechanical strength prior to and
during the implantation process would therefore represent the ideal tradeoff solu-
tion. Once implanted, the dissolving backplane would decouple all electrodes from
one another. Although the use of dissolvable implantation shuttles showed prom-
ising results by mitigating the foreign body response [60, 127], the fabrication of a
high-channel count implant, such as the natural buoyancy Utah electrode array
(NBUA) [65], with a dissolving backplane is unique.
The NBUA represents a modification of the conventional UEA, where the matrix
of up to 10
10 microelectrodes is held together by the biocompatible dissolvable
polyethylene glycol (PEG) material. The PEG provides a temporary rigid mechan-
ical back-plane required for insertion of the array into the tissue. Once the 10
10
matrices of electrodes are inside the tissue, the PEG dissolves after it gets in contact
with the biological fluid. At that point each electrode is floating independently from
the others (Fig. 1.3).
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 7

Fig. 1.3 Cartoon image of the dissolvable UEA prior to insertion (a). At this point all the 100 shafts
are connected by PEG which represents the rigid backplane of the device required for the
simultaneous insertion of all 100 electrodes. After insertion, PEG dissolves resulting in mechani-
cally decoupled floating shafts (b, c). (Reprinted, with permission, from Leber et al. [65])

The fabrication of the NBUA comprises additional steps starting with a standard
unwired UEA. Although the main modification is the substitution of the glass traces
with PEG, the fabrication of single electrodes results in further challenges. The side
walls of single electrodes, previously covered and protected by the glass traces, need
to be encapsulated with parylene-C to avoid the exposure of silicon to the biological
environment. Furthermore, the packaging process, which includes wire bonding and
silicone encapsulation, needs to be adapted to decoupled single electrodes.
The first step in the fabrication process is the removal of the glass lines in
between the different shafts of the standard Utah array while keeping the array in
place. Glass lines are etched in hydrofluoric acid (HF). To keep the array in place
and to protect the shaft and tip metal from HF, the UEA is entirely molded in
photoresist with only the back side glass traces exposed. A polyethylene (PE) ring
surrounds the array and serves as mold for the photoresist. After the HF etch, the
photoresist is removed in acetone, releasing the singularized electrodes solely held
together by a 3 μm thick parylene-C layer, which will subsequently be removed.
Parylene-C is deposited again onto the array covering this time also the side walls
of the electrodes, which were previously protected by the glass traces. Laser
ablation disrupts the film connecting the single electrodes and forms the active
electrode sites at the tip of each shaft.
Finally, all electrodes embedded in PEG and insulated gold wires are used to wire
bond the bond sites on the back of each electrode to a connector. Silicone is applied
using a thin platinum wire onto each bond site in order to provide additional
mechanical strength to the bonds and to encapsulate the remaining exposed surface.
The PEG is then removed from the tip side of the assembly by dipping the assembly
into deionized water. The final device is shown in Fig. 1.4.
The PEG backplane has typically a mass of 300 mg, which dissolves in
phosphate-buffered saline at room temperature in less than 45 minutes (Fig. 1.5).
Electrical impedance spectroscopy experiments after dissolution yielded typical
UEA impedance values for the single electrodes.
8 M. Leber et al.

Fig. 1.4 Left: A standard UEA without dissolvable base for comparison. Right: The finalized
NBUA device consists of 100 separated electrodes held in place by PEG and connected by gold
wires to a connector. (© 2017 IEEE. Reprinted, with permission, from Leber et al. [65])

Fig. 1.5 A 4
4 device dissolving in PBS at room temperature. (a) Rear view of
PEG-encapsulated device right after immersion in PBS (t ¼ 0 min). (b) Rear view of device with
dissolved PEG base (t ¼ 45 min). (c) Same as (b) with deformed wire to demonstrate freely floating
electrodes. (d and e) Close-up pictures of dissolved UEA and single electrode. (Reprinted, with
permission, from Leber et al. [65])

It has not only been shown that the technology of a dissolving back plane is a
promising evolution of the UEA, but it also offers the potential for many additional
improvements. For once, PEG can be used as a carrier material for drug delivery to
reduce inflammation or enhance neuronal growth. Furthermore, contacting the
dissolving NBUA with highly flexible polymer-based ribbon cables will reduce
the tethering forces induced by the currently used gold wires and provide even
less mechanical mismatch between the implant and the tissue.
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 9

1.3.3 Ultrahigh-Channel Count Electrode Architectures

In order to reduce damage to and impact on the tissue in which stimulation and
recording take place, it is beneficial for neural probes to have a small footprint. In
case of the UEA, this requirement in combination with the 3D microelectrode
geometry limits the number of active sites as only a certain number of electrodes
fit within the device footprint due to the challenges of the dicing-based fabrication
process. However, there is substantial interest to increase the number of active sites
and their density to address larger fractions of the brain and nerve tissue as well as
submillimeter neuroanatomical structures without increasing discomfort and risk of
infection for the patient (Wark et al. 2013). Furthermore, the ability to simulta-
neously stimulate and record in a coordinated fashion at different locations is
desired.
In order to enhance both coverage and resolution of the UEA architecture,
different paths have been pursued:
1. Multiport devices that connect multiple UEAs to one percutaneous connector.
This has allowed up to 1024 channels (sixteen 64-channel UEAs) to be attached
to one connector (Fig. 1.6).
2. High-density (HD) UEAs that have a smaller pitch (200 μm, instead of 400 μm)
between electrodes. These have been originally developed for the use in the
peripheral nervous system as slant electrode arrays [118], but the technology is
easily transferable to regular UEAs with constant shaft lengths.
3. Utah multisite electrode arrays (UMEAs) which are UEAs with multiple elec-
trode sites per shank. This allows to increase the number of stimulation/recording
sites while maintaining the same array footprint and form factor as the standard
UEA [104]. Although initial demonstrators have shown up to 9 contacts per UEA

Fig. 1.6 Multiport UEA

with 1024 channels
(16 times 64 channel arrays
attached to one connector).
(Image copyright and
courtesy of Blackrock
Microsystems LLC)
10 M. Leber et al.

Fig. 1.7 (a) Concept of UMEA with up to nine active sites per electrode. (b) SEM micrographs of
FIB fabricated UMEA with varying configurations: (A) 2 μm metal traces with multiple sites along
shaft without tip electrode, (B) 5 μm diameter sites with 2 μm traces, (C) 2 pairs of 5 μm tetrodes.
(A. Reprinted, with permission, from Shandhi et al. [103])

shank, practical implementations allow typically 3–4 contacts per shank

(Fig. 1.7).
While the multiport and high-density devices are fabricated in the same way as
the standard UEA with only small changes to the process, the creation of the Utah
multisite electrode array requires significant modification and additional fabrica-
tion steps.
Each electrode of the UMEA has one active site at the electrode tip and up to eight
active sites along the shaft. This requires the bottom of the electrode to be divided
into nine isolated conductive squares which hold the contact pad for each site on the
back (the shaft occupies one square). All active sites are connected to the contact
pads via metal traces on the shaft’s side, as seen in Fig. 1.7a [103]. In order to
insulate all active sites from one another, a silicon nitride layer is deposited prior to
metal sputtering [104].
Two approaches have been pursued to fabricate UMEAs: the serial direct writing
by focused ion beam (FIB) patterning in combination with high-resolution scanning
electron microscopy (SEM) [103] and a shadow mask process compatible with batch
fabrication [104].
The FIB technique allows to deposit metal (platinum) patterns along the shafts of
the UEA with precise control and reproducible resolution of <1 μm. It enables the
creation of metal patterns of any shape (circles, square, ring, etc.), size, count, and
location for neural recording and stimulation, on the tip of the electrode or along the
shank of the Utah electrodes in various configurations [103].
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 11

Fig. 1.8 SEM micrograph of the nickel shadow mask fabricated by electroplating and laser
ablation on a sacrificial UEA. (Reprinted, with permission, from Shandhi et al. [104])

However, this approach is comparatively slow and expensive, as each active site
needs to be written individually. Shandhi et al. have developed a process which
addresses this challenge: first, a nickel shadow mask is created on a sacrificial UEA
by electroplating and laser ablation (Fig. 1.8). After dissolution of the sacrificial
UEA structure, this shadow mask can be used to deposit the metal traces and
electrode sites by sputtering of platinum or another desired metal on another UEA
during fabrication [104]. Although this approach requires additional fabrication
steps compared to the standard UEA, it enables batch fabrication of arrays with an
increased number of stimulation/recording sites while maintaining the footprint of a
standard UEA.
This novel technology to pattern 3D UEA architectures opens new possibilities
for commercial devices. For example, it enables the fabrication of different patterns
(such as active sites in tetrode and laminar configuration) and/or in situ sensors for
device and tissue health monitoring at the site of implantation. Figure 1.7b shows
SEM micrographs of some of the fabricated UMEA configurations.
12 M. Leber et al.

1.3.4 Improving the Stimulation and Recording Properties:

Iridium Oxide and Roughened Electrodes

Neural stimulation and recording electrodes are generally characterized by their

impedance, charge injection capacity (CIC), and surface area. It is common practice
in the field to express the electrodes’ impedance at the frequency of 1 kHz.
The CIC is defined as the highest amount of charge that can be delivered by an
electrode within the leading phase of a stimulation pulse without inducing electrol-
ysis of water, which is considered to have harmful effects on the surrounding tissue.
Although the CIC is often assumed as a “safe” stimulation parameter, material
degradation was observed at lower charge values [90, 91], and tissue damage can
be induced by overexcitation of neural tissue regardless of the CIC [30].
The surface area can be differentiated into geometric surface area (GSA) and real
surface area (RSA) that additionally considers surface roughness on a nanometer
scale. Both impedance and CIC depend on the RSA: the impedance decreases while
the CIC increases with increasing RSA [28]. Generally low impedance and high CIC
values are desirable for neural stimulation and recording electrodes [119]. Micro-
electrodes like the UEA ensure high selectivity and spatial resolution to discriminate
between single neurons. Consequently, they exhibit a rather small surface area
(ca. 2000 μm2 for UEA electrodes). Hence, a tradeoff must be made between
small electrode sizes on the one hand and low impedance along with high CIC
values on the other hand. As impedance and CIC depend not only on the GSA and
RSA, but also on the material specific electrochemical charge transfer, two
approaches to optimize impedance and CIC were pursued in research:
1. Exploration of various electrode coating materials regarding their charge transfer
properties.
2. Creation of rough electrode surfaces to increase the RSA/GSA ratio.
Both approaches were investigated for the UEA technology by introducing
sputtered iridium oxide as an alternative coating material to the well-established
platinum and by roughening the silicon substrate material.
The Introduction of Iridium Oxide as UEA Coating Material
Over the years, iridium oxide has been shown to be one of the most promising
electrode coating materials, especially due to its comparatively high CIC values
[28]. Iridium oxide films can be created by using several techniques such as
electrochemical activation by repeated oxidation and reduction in PBS to form
activated iridium oxide films (AIROF) [90, 91], thermal decomposition of iridium
salt on metal substrates [92], electrodeposition [72], or by sputtering processes
[59]. Sputtered iridium oxide films (SIROF) have the advantage of the possibility
to fine-tune process parameters to obtain the desired physical and electrical proper-
ties for neural stimulation and recording [80, 122]. Therefore, they were investigated
as an alternative coating material to platinum for the UEA electrodes.
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 13

Negi et al. [80] studied the effect of pulsed DC reactive sputtering parameters on
the resulting SIROF film quality [80]. The sputtering pressure was identified as the
critical deposition parameter, which has a major impact on the film morphology, as
well as the electrochemical properties of the coatings. At lower pressure values, the
SIROF surface exhibits a granular morphology, transitioning to dendritic structures
with increasing pressure. Furthermore, impedance and CIC values of the films were
measured to be lowest and highest, respectively, at lower pressure values. Further
studies confirmed the superior electrochemical properties of SIROF compared to
AIROF and platinum. SIROF coated UEA electrodes yielded CIC values of 2 mC/
cm2 compared to 1 mC/cm2 [83] and 0.3 mC/cm2 [81] for AIROF and platinum
coated electrodes, respectively. At the same time, the impedance values at 1 kHz for
SIROF electrodes were measured to be lowest with 6–10 kΩ compared to 120 kΩ
[83] and 125 kΩ [81] for AIROF and platinum, respectively. In addition to its
superior CIC and impedance values, SIROF was shown to be more resilient against
material degradation by analyzing the iridium content in PBS after continuous
pulsing, using mass spectrometry [82].
Due to its superior material properties described above, iridium oxide has been
validated as an alternative coating material for UEA electrodes and has demonstrated
its usefulness in sophisticated human experiments [32].
Silicon Substrate Material Roughening for Improved Electrode Properties
Although SIROF coatings have been shown to substantially increase the charge
injection properties and to decrease the impedance values of UEA electrodes,
platinum is the best studied material for neural electrodes to date, with numerous
reported acute and chronic in vivo studies. As a result, much research was conducted
on roughening platinum electrode surfaces for improved charge transfer properties.
The efforts to create high RSA/GSA ratio platinum surfaces include laser processing
[40, 41], electrochemical roughening [15, 115, 121], and electroplating of platinum
gray [123, 142] as well as platinum black [38, 71, 85] films. Whereas laser
processing is a time-consuming serial processing technique and presents difficulties
regarding the three-dimensional UEA geometry, electrochemical roughening and
deposition methods add to the process complexity and, in the case of platinum black,
raise concerns regarding their mechanical integrity. This led to a substantially new
approach to create high RSA/GSA ratio electrode surfaces by roughening the silicon
substrate instead of the coating material, allowing the conformal deposition of
platinum in a simple sputtering process.
Various physical and chemical roughening processes to alter the silicon surface of
UEA tips were investigated. Despite well-described silicon roughening techniques in
literature, the unusual high aspect ratio of the UEA architecture required a tailored
roughening process. A customized reactive ion etching (RIE) and Bosch-type deep
reactive ion etching (DRIE) recipe were developed to roughen the silicon tips of the
UEA before coating them with a 500-nm-thick platinum layer [64]. The initial RIE
step creates a shallow, well-defined microstructure on the silicon surface, whereas
the second DRIE step deepens the pores from the first RIE etch until a minimum
electrode impedance is reached, indicating maximum RSA increase. The process
14 M. Leber et al.

Fig. 1.9 Colored SEM images: (a) Standard platinum coated UEA electrode. (b) Roughened
platinum coated UEA electrode

exhibits high controllability and has the advantage of batch fabrication compatibil-
ity, allowing to process the UEAs on wafer level. Fig. 1.9 displays a standard
platinum coated UEA electrode and a roughened platinum coated UEA electrode.
Extensive soak testing in PBS over 500 days with regular impedance spectros-
copy and CIC measurements demonstrated a decrease of impedance at 1 kHz by a
factor of 0.32, and an increase of CIC by a factor of 1.8 of the roughened compared
to standard UEA electrodes [63]. Furthermore, the roughened electrodes demon-
strated an increased robustness toward stimulation-induced material failure. In a
stress test, during which 1 million biphasic charge balanced pulses were applied to
both roughened and standard electrodes, the roughened electrodes were found to be
less susceptible to the degradation and delamination of the platinum coating.
Furthermore, first in vivo tests with the roughened UEAs were carried out in
rodents. Although slightly higher platinum residuals in the neural tissue were
detected for the roughened electrodes, a negative effect on the foreign body response
and cytotoxicity could not be observed during a histological analysis of the neural
tissue after 8 weeks of implantation into the cerebral cortices of rodents [63].
The substantially novel method of substrate material alteration toward increased
RSA/GSA ratios of neural electrodes was only studied for platinum coatings to date.
SIROF coated roughened UEA electrodes remain to be studied but are expected to
even further increase the electrochemical properties such as the electrodes’ imped-
ance and CIC values.

1.3.5 Optrodes for Infrared and Optogenetic Stimulation

In the past, most devices that aim to interface with the nervous system and evoke
action potentials in neurons relied on electrical stimulation. This is still the case, but
there is an increasing interest in using electromagnetic radiation, both in infrared
(IR) and visible wavelengths for neural stimulation as these optical methods carry
the promise to address some of the challenges of electrical stimulation, such as
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 15

neuronal specificity, spatial resolution, stimulation-induced tissue damage, and

stimulation artifacts [86, 120].
IR stimulation-based approaches have the advantage that they can be employed
without preceding alterations to the neurons or their surroundings as the tissue
absorbs the radiative energy and the resulting rapid temperature change induces
stimulation due to many different processes and effects in the neural tissue [112].
In contrast, for the stimulation with visible wavelengths, the neurons have to be
modified via optogenetics. In this technique, chosen as Nature’s method of the year
2010 [79], virus vectors are used to genetically target specific neuron subtypes to
express light-activated proteins in their cell membrane. Depending on the protein
chosen, the thus genetically modified neurons can either be stimulated or inhibited
by light irradiation in a specific wavelength range of the visible and near-infrared
spectrum with very high specificity [86].
In order to take full advantage of the potential of optical stimulation, the efficient
delivery of radiation deep into the neuronal tissue with high spatiotemporal resolu-
tion is required in many use cases. Furthermore, a corresponding device should also
be able to record the resulting electrochemical response of the neurons to monitor the
effect of the stimulation. The simplest penetrating light delivery approach is given by
implanting an optical fiber into the neural tissue, often together with a penetrating or
surface electrode for electrophysiological measurements of the response of the
neural cells [5, 6, 39]. In more advanced approaches of integrated devices, either
Michigan-style penetrating electrodes are upgraded with optical waveguides [26] or
even whole micro light emitting diodes (μLED) [95, 126]. Alternatively, the UEA
platform is adapted to benefit the optical stimulation techniques as discussed in the
following.
A way to integrate a UEA-type device with a simple optical fiber approach is
given by removing one of the electrodes and creating a hole in the back plane of a
UEA via laser ablation. The missing electrode can then be replaced by aligning and
fixing a so-called optrode, a metal coated tapered optical fiber, in its place
(Fig. 1.10). This combines the relatively simple single site optical fiber stimulation
with the spatiotemporal multichannel recording capability of UEA-type devices.
Successful optogenetic stimulation with such a device was demonstrated both
in vitro and in vivo [117, 141].
Alternatively, the UEA geometry and batch compatible fabrication method can
be modified to create novel devices for multichannel light delivery. To create a
device for IR radiation transmission, undoped silicon is used as a base material in
the fabrication process and the contact pads of the UEA are replaced with an
aluminum mask with windows to facilitate the alignment of butt-coupled optical
fibers to the silicon needles that are acting as waveguides. Additionally, other
fabrication steps necessary for the electrostimulation such as the metal electrode
material deposition and the electrical insulation of the individual silicon needles
are omitted. The resulting IR optrode array (Fig. 1.10a) was able to efficiently and
selectively deliver IR radiation into tissue for stimulation [2, 27]. In order to further
enhance the efficiency of the IR transmission and to add the possibility of deliv-
ering visible light into tissue for optogenetic stimulation, the base material for the
16 M. Leber et al.

Fig. 1.10 Combination of a multi electrode array based on the Utah electrode array with a
sharpened and metal coated optical fiber-based optrode. (a) Schematic of the device and (b) optical
image with blue laser light being transmitted through the optrode. (c) Close-up optical image of the
device. The position of the optrode between the electrodes is indicated. (Reprinted, with permission,
from Zhang et al. [141])

device fabrication was changed from silicon to transparent silicate glass

[1, 2]. However, glass behaves differently than silicon especially during the
etching step of the UEA fabrication process and no sharp needles could be created
this way. Therefore, several additional steps had to be added to the standard UEA
fabrication process along with other smaller modifications to create a suitable
device geometry. These additional steps include the use of a tapered dicing blade
to pre-shape sharp optrode tips as well as a heat treatment step to smoothen the
rough glass surface after etching [2, 17, 96].
The resulting passive glass Utah optrode array devices (Fig. 1.11b, c) can be used
for light delivery using external stimulation sources. However, for neurological
research and future clinical and neuroprosthetic applications, a fully integrated
device, featuring both multichannel optical stimulation and electrical recording
capacity, would be beneficial. Such a device is currently under development by a
combination of the passive glass optrode array with a μLED array and an interposer
layer to reduce cross-talk between optrodes (Fig. 1.12). By adding electrical
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 17

Fig. 1.11 SEM images of optrode arrays fabricated from (a) undoped silicon, (b) fused silica glass,
and (c) optical image of an optrode array made from borosilicate glass. (a, b Reprinted, with
permission, from Abaya et al. [2])

Fig. 1.12 Images of a glass optrode array with a special interposer to prevent cross talk, coupled to
a μLED array. In (a), one of the μLEDs is turned on. (b) shows the emission profile of an optrode
when immersed in fluorescein solution. (Reprinted, with permission, from Scharf et al. [96])

recording functionality to the array, a highly flexible device for optogenetic research
applications will be created [96].

1.3.6 Advances in Encapsulation

An important requirement for implanted neural devices is the ability to perform their
function in vivo for an extended amount of time to justify the risk and cost associated
with implantation surgery. However, the physiological environment is hazardous to
devices. Water and different ionic species dissolved in body fluids lead to corrosion
and other electrochemical degradation processes of the various materials the device
consists of, resulting in leakage currents or short circuits of electrical components.
This will eventually lead to complete device failure. Therefore, the functional parts
of the device, except the active electrode sites that are supposed to interact directly
with the neural tissue, must be protected from the physiological environment by
encapsulation.
18 M. Leber et al.

Table 1.1 Methods and materials that can be used for the encapsulation of implantable neural
devices along with their advantages and disadvantages
Method and material Advantages Disadvantages
Hermetic can [55] Hermetic (airtight + waterproof), Large, assembly process complex-
long lifetime ity, need for feedthroughs
Silicon oxide Conformal, good insulation Dissolves in body fluids (except
([43, 78]) for thermal SiO2), needs to be
defect free – > capacitively
coupled electrodes
Silicon nitride Low WVTR, good dielectric prop- Pin holes, dissolves in body fluids
([97, 129]) erties, easy to deposit
Diamon-like carbon Low friction, low wear, chemically Poor adhesion, tends to delaminate
(DLC) [93] inert, biocompatible, deposition at
room temperature possible, doping
can control electrical properties
Ultrananocrystalline Chemically inert, biocompatible, Requires high deposition
diamond (UNCD) high wear resistance, low friction temperatures
([8, 128]) surface
Silicon carbide Very hard to dissolve/destroy, bio- Pin-holes, high temperature pro-
([29, 49, 66]) compatible, doping can control cess, nonuniformity, cracks on
electrical properties, low water polymer substrates
intake
PTFE ([54, 69]) Low friction, biostable, Nonuniformity, byproduct
biocompatible incorporation
Silicone [116] Biocompatible, cheap, widely used Nonuniformity, high WVTR
Parylene ([51, 70]) Conformal, low deposition temper- Relatively high WVTR, poor
ature, biocompatible, widely used adhesion
Polyimide [88] Biocompatible Nonuniformity, high water
absorption and WVTR

Any material that is used for encapsulation of implantable neural devices has to
provide chemical resistance against biological media and their diffusion into the
encapsulation material, biocompatibility, and long-term stability. It has to protect the
inner parts of the device over extended periods of time in vivo by creating a near-
hermetic isolation with a very low water vapor transmission rate (WVTR)
[107]. Additionally, it has to ensure the proper electrical functionality of the devices,
i.e. preferably should have high impedance and a low dielectric constant to reduce
signal noise and ensure high selectivity for interaction with neural tissue [51].
Furthermore, the encapsulation steps need to be compatible with the fabrication
or assembly processes, offer the potential for further miniaturization, and be com-
patible with wire-based or wireless data and power transmission. Identifying an
encapsulation method and corresponding materials that fully satisfy all these require-
ments at the same time is very difficult [134].
Consequently, many different materials have been investigated for encapsulation
purposes of implantable neural devices (Table 1.1). Methods can be divided into
hermetic and near-hermetic encapsulation. The former can be realized as capsules or
small implantable cans made of metal, glass, or ceramic and provide an airtight and
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 19

waterproof environment [129]. They are rather large which is a drawback for
miniaturized implantable devices and, furthermore, require feedthroughs for electri-
cal data and power transfer. Therefore, near-hermetic encapsulation using thin films,
which at least significantly slow down the ingress of body fluids, has significant
potential [129, 134].
Thin-film encapsulation materials can be grouped into polymeric
(e.g. polytetrafluoroethylene (PTFE), silicone elastomers, polyimide, and parylene)
and non-polymeric materials (e.g. silicon dioxide, silicon nitride, silicon carbide, and
diamond-like carbon (DLC)). Non-polymeric materials require chemical vapor
deposition (CVD) processes with rather high temperatures, which is not suitable
for neural devices with active electronics that would not sustain these high temper-
atures. Furthermore, some of them dissolve in saline solution over time, are prone to
structural defects (pin-holes), form non-conformal layers, and tend to delaminate.
Polymeric materials are very flexible, inexpensive, and require low process
temperatures. Disadvantages include a high WVTR, problems in creating conformal
and pin-hole free films, and material degradation as well as poor adhesion [129].
For implantable neural interfaces, many different encapsulation materials have
been studied and continuously developed for creating conformal and long-term
stable encapsulation. In the following, the ones relevant and used for the UEA will
be discussed, which include parylene-C, silicon carbide, and more advanced
parylene-C/Al2O3 bilayer structures.

1.3.6.1 Parylene-C

Parylene, a semicrystalline polymer for the family of thermoplasts, is a widely used

material for encapsulation and insulation of biomedical devices. It exists in several
variants, among which parylene-C and parylene-N are certified as class VI polymers
for use in biomedical devices by the Food and Drug Administration (FDA) [73, 108].
In the following, we will focus on parylene-C which is used for the UEA.
Information on other variants may be found elsewhere [36, 61].
Parylene-C has many advantageous properties: it is highly biocompatible,
non-toxic, chemically inert, an excellent ion barrier, has a low dielectric constant
(~3.15 at 60 Hz), relatively low water absorption rate (~0.1% in 24 h), high
resistivity, and dielectric stability in saline. Furthermore, due to the possibility of
employing a CVD process, a very conformal deposition down to the nanoscale and
also for complex geometries which include sharp edges and crevices is possible
[70, 73, 102, 105, 107, 130, 133].
Due to parylene-C being a polymer, it exhibits some of the corresponding
drawbacks as well, which include a relatively poor adhesion to inorganic and
metallic substrates and a certain permeability to water [73].
Parylene-C is deposited according to the Gorham method, which is a three-step
CVD process [140]. First, the precursor (di-para-xylene dimer) is sublimated at
~130  C. Next, by elevating the temperature to ~680  C, the dimer is cracked into
monomers. It is important to achieve a complete dissociation of dimers into monomers
20 M. Leber et al.

to reduce the number of dimers in the deposited film. Reduction of the temperature
(to at least <80  C) allows the polymerization of the monomers on all available
surfaces. The film grows due to a free radical chemical reaction process where first
new carbon chains are formed (initiation) and then extended to higher molecular
weight by attachment of monomers to the end of the chain [51, 101, 129]. Typical
film thicknesses deposited on the UEA are 3–6 μm. Resulting film properties depend
on the pressure, temperature, and deposition rate during the CVD process which
provides a means to tailor the film’s properties within a certain range [50].
The described CVD process has the additional advantage of not requiring any
solvents or additives, which reduces the contamination of the film to a minimum and
ensures almost defect free and conformal coverage, even of complex electrode
geometries [51].
A problem which needs to be addressed during film deposition is the poor
adhesion of parylene-C to metal and other inorganic surfaces. In case of the UEA,
it has to stick well to silicon and platinum or iridium oxide, depending on the type of
tip metallization used. Therefore, measures must be taken to increase the adhesion
which include silanization, PECVD deposition of an interlayer, plasma surface
treatment, heat treatment, and chemical modifications.
The use of silane (usually Silquest A-174) as adhesion promoter, which is
vaporized onto the substrate before deposition, is a common method for the UEA
[73, 105, 107]. The organo-silane creates a covalent bonding by functional groups
between substrate and parylene-C [45, 135]. It is inexpensive and integrates very
well in the process flow of UEA device fabrication.
Seymour et al. [102] developed a reactive parylene-C by adding a functional
group and demonstrated that it improves adhesion and short-term insulation perfor-
mance. Plasma surface treatment or chemical surface modifications of the substrate
before parylene-C deposition increases the number of radical sites for covalent
bonds, resulting in better adhesion [73]. This can even be enhanced by deposition
of a plasma polymer as a thin interlayer between substrate and parylene-C
[140]. Although that has been reported to be a very effective method, it requires
another complex step and is therefore not used for the UEA [129]. Finally, the
parylene-C coated UEA can be annealed to increase the physical interlock between
substrate and thin film [73].
All of these methods address the adhesion problem but do not prevent moisture
ingress over time.
The function of the UEA is based on the electrode tips being in contact with the
neural tissue; hence, the encapsulation has to be removed from this area. Several
methods have been proposed and studied, the most common one being the reactive
ion etching (RIE) with oxygen plasma by using aluminum foil or photoresist as a
mask [10, 107].
Yoo et al. employed laser ablation which is advantageous for mass production
and allows to selectively treat individual electrodes. However, this process creates
carbon debris and the amount thereof as well as the conformity of removal depend on
the laser fluence. The debris can be removed by a subsequent short (1 min) oxygen
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 21

plasma etch. However, the laser ablation requires a tip metal which can withstand the
laser power [138, 139].
Many experiments have been carried out (both in vivo and in vitro) to investigate
the long-term stability of parylene-C encapsulation to ensure the safe and reliable
performance of chronically implanted neural devices. Test devices include flat
interdigitated electrode structures and functional UEAs, which are submerged in
phosphate-buffered saline (PBS) solution for prolonged amounts of time (several
hundred days to more than a year). Typical temperatures range from 37  C (human
body temperature) up to 80  C to achieve accelerated aging processes.
Although parylene-C coating can withstand very harsh conditions, such as the use
of different sterilization methods and heat treatment [101], all experiments indicated
an increasing amount of failed test devices after being subject to physiological saline
solution for prolonged times. The failure mechanisms are not yet fully understood,
but investigations of failed devices found blistering, microcrack formation, micro-
scopic delamination, and thinning of the parylene-C film. One reason is believed to
be the steady water uptake of the material. Furthermore, imperfections at the
interface between the electrode and parylene-C coating may induce electrical
stresses leading to microcracks and delamination of the film. Once moisture and
possibly ions permeate though the encapsulation, chemical reactions degrade the
metal film underneath and eventually induce failure of the electrode [73, 130,
133]. These effects are observed for all variants of parylene, although with slightly
different characteristics. For example, parylene-N is prone to microcrack formation
but does not show delamination [73].
Failing electrodes exhibit an increased leakage current, decreased impedance
modulus for low frequencies, and an increased area under the cyclic voltammetry
curve used for testing the devices which indicate exposure of underlying metal
[73, 105, 130].
To overcome these problems, the single material encapsulation has been advanced
to a bilayer of parylene-C and Al3O3 which will be discussed in the next section.

1.3.6.2 Parylene-C/Al2O3 Bilayer Encapsulation

The use of a bilayer encapsulation evolved due to the observed long-term degrada-
tion of parylene-C films in physiological environment. The bilayer encapsulation
allows combining the favorable properties of both materials to form a better and
more durable encapsulation for neural implants. With parylene-C still being the outer
layer in direct contact with tissue, the ease of regulatory approval is ensured with its
proven performance in patients for up to 6 years.
Al2O3 thin films deposited by atomic layer deposition (ALD) feature the forma-
tion of conformal and pin-hole free layers, high hardness and abrasion resistance,
good biocompatibility, and perfect electrical performance and are a very good
moisture barrier with very low WVTR. However, the major drawback of ALD
Al2O3 is that it corrodes easily in water and more so in PBS solution [130].
22 M. Leber et al.

Table 1.2 Relevant properties for the UEA for both individual materials (ALD Al2O3 and
parylene-C) and the bilayer of these two materials (information taken from [[129], Xie et al.
2014, [20]])
Properties ALD Al2O3 Parylene-C Effects of bilayer
Water vapor transmission ~106 g/m2∙day [57] ~1 g/m2∙day Isolates water vapor from
rate (WVTR) (from SCS the device
coating)
Deposition temperature Room temperature Room Low temperature coating
to 200  C temperature process
Barrier to ions Yes Yes Good ion barrier
Biocompatible Yes Yes Biocompatible coating
Insulation 1014 ohm-cm 1015 ohm-cm High insulation
[133] impedance
Coating process Plasma-assisted CVD Pin-hole free, uniform
ALD coating
Dissolution in water Yes [4] No Prevents alumina
dissolution

By combining ALD Al2O3 and parylene-C, this disadvantage is mitigated as

parylene-C provides a good ion barrier. It therefore prevents ions from reaching and
dissolving the Al2O3 layer. Table 1.2 summarizes the properties relevant for use in
the UEA for both individual materials and the bilayer. Typical thicknesses used for
the UEA are ~50 nm Al2O3 and ~3–6 μm parylene-C [21, 131, 133].
Several studies have been carried out since the development of the bilayer encap-
sulation to investigate its long-term performance for test samples (flat glass IDEs)
and actual UEAs soaked in PBS at body temperature as well as for elevated
temperatures (for accelerated aging) for prolonged amounts of time (months to
more than a year) [21, 73, 130, 131, 134].
These investigations indicated that the bilayer on flat IDE test samples shows a
better long-term stability in physiological environment than a single parylene-C film.
A comprehensive study from Minnikanti et al. found a mean time to failure (MTTF)
of 8 months for single parylene-C encapsulation in saline solution at 37  C, while the
MTTF for the bilayer was 36 months – an increase by a factor of 4.6 and very close
to that of other materials like liquid crystal polymers (LCP; up to 2 years) [73]. The
MTTF for single parylene-C is consistent with values reported from other groups
which are 6–12 months. However, most of these tests were only performed on flat
IDE samples which does not reflect the complex geometry of the UEA.
Xie et al. and Caldwell et al. performed lifetime tests on actual UEAs encapsu-
lated with the bilayer. While Xie et al. reported an improved long-term stability even
under an increased bias voltage [134], the more comprehensive study (increased
number of samples/improved statistics) from Caldwell et al. did not find conclusive
evidence for an improved lifetime of the bilayer, neither on more complex flat test
samples with additional features nor for the full UEAs [21]. Furthermore, they
observed that, if the encapsulation fails, the bilayer is destroyed faster than the single
parylene-C film [21]. The rationale in that case is that solution ingress does not have
a strong effect for a single parylene-C film. For the bilayer, however, solution
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 23

Fig. 1.13 Comparison of

defect size for (a) single
parylene-C encapsulation
and (b) parylene-C/Al2O3
bilayer. (Reprinted, with
permission, from Caldwell
et al. [21])

permeating through the outer parylene-C layer reaches the underlying Al2O3 film
which dissolves, leading to the creation of an undercut and effectively to a growth of
the defect site (Fig. 1.13).
These results clearly show that imperfections of the parylene-C layer have a much
more severe effect in case of the bilayer. Hence, although the bilayer could be an
important improvement in the encapsulation of neural devices for long-term implan-
tation, the current materials combination still gives rise to problems and needs to be
developed further to allow for chronic implantation over several years.
Steps along that way have to include the development of failure models in order
to understand the causes for formation of imperfections and defects of the films.
Based on that, materials and deposition processes can be improved. A suggestion
made by Caldwell et al. is the use of higher deposition temperatures for the ALD
Al2O3 which would create less impurities, thereby reducing the probability of
dissolving the film [21]. Furthermore, it may become necessary to investigate the
feasibility of other materials such as silicon carbide or oxide-nitride-oxide layers in
the future [21].
As in the case of a single parylene-C film, the bilayer has to be removed from the
electrode tips to ensure contact between tip metallization and tissue. Xie et al.
proposed a three-step process similar to that of single parylene removal: (1) laser
ablation of the parylene-C layer, (2) reactive ion etching with oxygen plasma to
remove carbon debris and parylene-C residue, and (3) buffered oxide etch (BOE) to
remove the Al2O3 layer and expose the tip metal [132, 133].

1.3.6.3 Silicon Carbide as an Alternative Encapsulation Material

Another material which fulfills the criteria for an encapsulation material for implant-
able devices (i.e. high hardness, chemical inertness, biocompatibility, low dielectric
constant. and low moisture intake) is silicon carbide [29]. Hsu et al. developed a low
24 M. Leber et al.

Fig. 1.14 Scanning

electron micrographs from
a-SiCx:H films deposited on
UEA. The thickness of the
film was measured at
different sites along the
electrode shaft, and a wide
variation is observable.
(Reprinted, with permission,
from Hsu et al. [49])

temperature process for the deposition of hydrogenated amorphous silicon carbide

(a-SiCx:H) on test samples (IDEs) and UEAs and investigated the properties of the
resulting film with regard to stress, defect density, and dielectric constant. While for
the actual UEAs only the conformity and coating uniformity were studied with
electrode cross sections in scanning electron microscope (SEM), electrical tests
with coated IDEs soaked in saline solution were performed [49].
The a-SiCx:H films were deposited by plasma-enhanced CVD at temperatures
below 200  C and with hydrogen as dilution gas source and silane (SiH4) and
methane (CH4) as Si and C precursors. Through variation of process parameters
and subsequent characterization, it became evident that the quality of the deposited
film (strength of Si-C bond, film stress) can be optimized and tailored within a
certain range. However, while the results were promising for the flat IDE samples,
the encapsulation of the UEAs showed a rather large thickness variation along the
electrode shafts and a large pin-hole density (Fig. 1.14) [49]. Hence, the encapsula-
tion of UEA with silicon carbide was not viable.
Another issue that has been observed is that silicon carbide is hard to remove
which is crucial for the UEA as the metallized tips have to be exposed to get in
contact with tissue. Laser ablation and mechanical grinding have been reported but
led to alterations of the underlying metal film [29].
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 25

Despite these difficulties, silicon carbide stays in the focus as a possible material
due to its good biocompatibility and promising results in IDE soak tests. Currently, it
is being discussed if it is feasible to create an entire electrode from a-SiC allowing for
greater flexibility of electrode structures.

1.3.6.4 Accelerated Aging

In order to evaluate the performance of any encapsulation, its short- and long-term
performance in an in vivo environment and its possible failure modes have to be
investigated. However, in the development of new processes and exploration of new
materials, it is neither feasible nor reasonable to achieve this by chronic implantation
studies due to the time and resources necessary for such studies. Therefore, one
needs to test new encapsulation materials or processes in vitro in an experimental
setup that mimics the physiological environment. This also avoids the ethical
dilemma, high cost, and regulatory challenges associated with animal studies.
However, these experiments are still time-consuming if the long-term performance
has to be evaluated and therefore still pose a problem, especially since usually a large
number of iterations are involved in device development. Therefore, accelerated
aging methods have to be employed to estimate the lifetime performance of the
encapsulation and the whole device in a reasonable amount of time.
A basic in vitro method to test the lifetime of biomedical devices is to immerse
them in saline solution of physiological concentration at a temperature of 37  C to
mimic the human body temperature.
In order to induce accelerated aging of the device, a simple method is to increase
the temperature of the solution above 37  C as this will increase the rate at which
chemical reactions proceed [46]. For polymeric encapsulation materials and assum-
ing first-order reactions only, it can be shown by using chemical kinetics that
increasing the temperature speeds up the reaction rate and therefore the aging
process by a factor f:
 

R T  37 C
f ¼ exp
E act

where R being the universal gas constant (8.314 J mol1), T the elevated tempera-
ture, and Eact the activation energy of the aging reaction [52]. The application of this
relation, however, has the challenge of determining the activation energy. This is
avoided by employing the empirically determined “10-degree rule” which provides a
conservative estimate about the accelerated aging process. It states that a temperature
increase by 10  C roughly doubles the rate of many polymeric reactions and
therefore the aging process [46, 52]. For example, increasing the temperature by
20  C allows to test a device for an effective lifetime of 12 months in only 3 months
real time according to the “10-degree rule.” However, this is also only valid for first-
order chemical processes, and both approaches do not apply in cases where the
26 M. Leber et al.

Fig. 1.15 Leaked current of

planar interdigitated
electrode test structures that
were coated with either an
Al2O3/parylene-C bilayer
(A + P) or parylene-C only
(P) and soaked in saline
solution at various
temperatures for an
extended period of time. For
the parylene-C only devices,
the effect of the different
temperatures resulting in
different failure times and
the improved performance
of the bilayer are clearly
visible. (Reprinted, with
permission, from Xie et al.
[131])

elevated temperature induces processes that are not present in normal aging, such as
glass transitions or phase changes [52]. Consequently, any accelerated test protocol
has to be designed with great care [46].
The application of either the chemical reaction formula or the “10-degree rule”
enables the testing of encapsulation materials in a reduced time frame as this allows
to state an effective time the material or device was aged.
However, it would be desirable to always test the whole device in accelerated aging
studies. However, in case of complex devices with a high resource cost associated with
every sample (like the UEA), this is not economically feasible if entirely new
encapsulation materials or processes are studied. Therefore, one can employ more
simple test structures as for example planar interdigitated electrodes (IDEs) that are
coated with the materials to be studied and monitored with electrochemical impedance
spectrometry or leakage current measurements in strongly accelerated aging experi-
ments [131]. Figure 1.15 gives an example of how such test structures have been
employed to evaluate the performance of an Al2O3/parylene-C bilayer compared to
parylene-C only encapsulation in accelerated aging experiments.
While these simple planar test structures might give a first indication about
encapsulation longevity and performance, respective results have to be taken with
a grain of salt when generalizing them to the whole device. Even slightly increasing
the complexity of the test structures might lead to different results and makes
confirmatory accelerated aging testing of the whole device necessary [21].
Of course, the simple approach of evaluating device lifetime by soaking it in
heated saline solution does not capture the multitude of possible chemical pro-
cesses in an in vivo environment that involve the complex composition of body
fluids and the immune system’s foreign body response. The insufficiency of simple
immersion in (heated) saline solution is indicated by the inability to reproduce the
failure modes observed on UEAs that were chronically implanted in vivo, as
shown in Fig. 1.16.
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 27

Fig. 1.16 Characterization of explanted UEA. (A) Electrode tine of UEA explanted after 3 years in
feline nerve reveals silicon (green) exposed through the damaged parylene-C (red); inset shows
detail of parylene-C degradation and cratering over IrOx (white). Imaging XPS of inset area
(“explant”) shows evidence of C¼O and COO bonds, while identical measurements made on an
UEA aged in saline solution (“control”) only show nominal levels of C¼O binding energies.
(Images and data courtesy of Ryan Caldwell)

Fig. 1.17 Comparison of Utah electrode array that was subjected to RAA and a device which was
explanted after more than 3 years from a feline peripheral nerve. The colors were added to highlight
the different materials: red, parylene-C; white, IrOx; green, Si. (Images and data courtesy of Ryan
Caldwell)

In order to improve the predictive capabilities of accelerated aging experiments, it

was suggested to use a reactive accelerated aging (RAA) method involving the regular
addition of hydrogen peroxide to the saline solution [111]. This method tries to mimic
the environment that is created by activated immune cells in response to foreign body
intrusion, which includes the release of reactive oxygen species [7, 111].
This RAA method was applied to UEAs, and the damage to the arrays was
assessed visually from scanning electron microscopy (SEM) images and chemically
from energy dispersive X-ray spectroscopy (EDX) and X-ray photoelectron spec-
troscopy (XPS) measurements. It was found already from the visual examination that
new failure modes had appeared as compared to heated saline solution only devices.
Furthermore, comparison to respective measurements on UEAs that were removed
after being implanted in a feline peripheral nerve for more than 3 years revealed that
the visual damage modes looked very much alike for the explanted and the RAA
devices (Fig. 1.17) [20]. This demonstrates the benefit of the RAA method which
28 M. Leber et al.

enables the evaluation of device and encapsulation performance in a fast and realistic
way without requiring in vivo experiments. It allows the reproduction of in vivo
damage modes which had not been observed before in aging experiments. However,
while promising, the method is still not able to fully emulate in vivo aging. As an
example, the chemical analysis showed a difference in both the bulk and the surface
composition between in vivo and in vitro aged arrays, hinting that further improve-
ments to the accelerated aging methods may be necessary.

1.4 Wireless Technologies for Use with the UEA

The challenge faced when using a multichannel microelectrode array is how to

connect the implanted array to an external signal generator (for stimulation) or
data evaluation systems for post processing (in case of recording). Most of the
existing neural recording devices use wired connections for data communication.
Despite the pioneering and ground-breaking results that tethered studies have pro-
duced in the past [31, 47, 48], the wired recording configuration faces limitations
with regard to mobility of the subject and a significant inherent risk of infection due
to connected mechanical stress induced by the constraining cabling [58]. A straight-
forward solution to overcome these challenges is the application of wireless tech-
nology. This, however, comes with its own obstacles: due to the size and power
dissipation limitations of the application, the fabrication of neural recording devices
with a high-speed wireless link for reliable wideband neural data transmission at
very low power budget is extremely difficult. In device design, a tradeoff between
power consumption, noise performance, signal quality, battery life, and size will
have to be made. To achieve small device sizes, constraints consist of either limiting
the number of recording channels [42] or conceding signal fidelity for increased
channel count [99].
Over the years, these issues have been addressed one after another, leading to the
development of headstage wireless connectors that are not fully implantable but
mark an important milestone as they are already used in animals as well as inves-
tigative clinical applications.
Such wireless devices have been developed and tested for various animal species
applications, such as insects [44, 94], rodents (Seung Bae [42, 68, 110]), sheep [89],
and nonhuman primates (NHPs) [16, 24, 37, 53, 74, 99]. Some of these devices were
even advanced beyond the simple headstage concept by employing an enclosure fit
for subcutaneous implantation [16, 141]. Additionally, there are efforts to create and
utilize application-specific integrated circuits (ASICs) to further advance miniatur-
ization and power efficiency in these kind of devices [12, 33, 75, 136].
In order to advance the general idea of wireless implantable neurodevices even
further, efforts have been made in parallel to create fully integrated neural interface
(INI) devices. In corresponding developments that were directly tailored for the
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 29

Fig. 1.18 Image of an

integrated neural interface
based on a modified Utah
slant electrode array with
flip chip bonded ASIC
(in this case an INI-R6 chip)
for amplification and spike
thresholding with AuPd
insulated wire coil and SMD
capacitors. The device is
inductively powered and
transmits data wirelessly.
(Reprinted, with permission,
from Xie et al. [132])

UEA, an ASIC INI chip designed to fit the architecture and 400 μm interelectrode
pitch of the UEA was directly flip-chip bonded to a modified array. The modification
of the array allowed to use the backplane as a PCB substrate for direct chip mounting
of active and passive components such as the ASIC INI chip, a power antenna, and
SMD components (Fig. 1.18) [58, 105, 107, 132].
Due to the technical challenges of system miniaturization, power dissipation
limits, and, most importantly, establishing a reliable wireless data link with high
throughput in situations with a highly active and mobile patient, wireless recording
systems have just recently been introduced on the market. Present examples for such
systems are the W-Series recording system by Triangle BioSystems International
[113]; the Cube2 system by NeuraLynx, Inc. [84]; the Wireless-Systems by
Multichannel Systems [76]; and a series of wireless systems by Blackrock
Microsystems LLC. The latter include the Brown Wireless Device (BWD), CerePlex
W [14], and CerePlex Exilis [13], which are all headstage-based devices. Figure 1.19
depicts the roadmap of the wireless system development from Blackrock
Microsystems LLC, with the first device in the series being the BWD.
Although the roadmap in Fig. 1.19 only covers a short time span of 2 years, the
requirements and trends for the development are clearly visible. Preferable proper-
ties for wireless headstage-type devices are:
1. Small and lightweight (less than 10 g has been demonstrated for the CerePlex
Exilis)
2. High-channel count
3. High data transmission rate
4. Range of few meters for reliable wireless transmission
5. Prolonged battery life-time
6. Offering the possibility for practical sterilization.
Reduced weight is ensured by using polymer-based encapsulation materials
which not only allow protection of the electronics from mechanical and electrical
impact but are also easily sterilizable. A common way to connect the wireless
30 M. Leber et al.

Fig. 1.19 The Roadmap of the 96-channel wireless headstage development at Blackrock
Microsystems LLC: from BWD to CerePlex Exilis

transmitter to an implanted microelectrode array is via a “screw-on” attachment to a

pedestal connector port on the patient’s/subject’s skull that is in turn connected to the
neural implant. This is illustrated for the BWD from Blackrock Microsystems LLC
in Fig. 1.20. The BWD has been extensively tested and is currently actively used in
both animal and investigative clinical research.
Figure 1.21 gives an overview of the used wired and wireless components and the
recorded neural signals in an experiment with a chair constraint monkey. The compar-
ison clearly shows that the BWD wireless system produced a similar output signal as a
typical commercial wired recording system. Other experiments indicated likewise
performance. This validates the currently used setup for wireless recording which can
be the basis for further developments toward a miniaturized and fully implantable chip.
Although the BWD’s performance was promising, there were also drawbacks
with regard to design and device performance which needed to be addressed. One of
the concerns was that the size and weight of the device was still too big even for large
animals and certainly not feasible for small animals such as rodents and marmosets.
Motivated by that, the other two devices from the wireless series, CerePlex W and
Exilis, were developed and commercialized by Blackrock Microsystems LLC to
address these challenges. Compared to the BWD, the CerePlex W and Exilis are
much smaller in size and lighter in weight, while providing 1.5 times higher
sampling rates and lower noise levels. Furthermore, they both offer the possibility
of electrode impedance measurement as an advancement.
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 31

Fig. 1.20 Architecture of the Brown Wireless Device. (a) 3D computer-added design (CAD)
model showing the complete assembly of the wireless neurosensor. (b) 3D CAD model showing
the attachment of the wireless neurosensor to a head-mounted pedestal (side view). (Reprinted, with
permission, from Yin et al. [137])

1.5 Vision and Concept for Integrated Systems

and Implantable Electronics

The ultimate vision for advanced and fully implantable wireless neural interfaces are
solutions that integrate the electrode and those electronics necessary to carry out
rudimentary signal processing functions, e.g., for amplification and data compression
and to facilitate wireless telemetry of data and power. While there is a multitude of
options, a potential roadmap, applied to a penetrating electrode such as the UEA, is
shown in Fig. 1.22. It has to be noted here that the developmental work on the
engineering side is only one side of the medal and that on the other side, regulatory
guidelines and constraints have to be met for implantable devices (especially for
human use). The stepwise approach depicted in Fig. 1.22 is advantageous to address
both parts by successive technological and regulatory iterations of the devices.
32 M. Leber et al.

Fig. 1.21 Comparison of recorded neural data from a wired and a wireless setup in an experiment
with a monkey. (a) Experimental setup and sketch of brain location where the recording took place
(primary motor cortex, arm area). The red arrow marks the device location. (b) Images of the used
devices: the commercial wired system (Cerebus) and the BWD system. (c) Recorded neural data for
the wired and wireless device. (Reprinted, with permission, from Yin et al. [137])

The neural information system 1 (NIS-1) stage constitutes a conventional passive

electrode with percutaneous connector, connected to an external headstage and
cable. NIS-2 transfers the headstage function into an implantable device which
then generates an amplified, digital signal and allows reduction of the number of
wires, independent of the number of electrode channels. An example for this stage is
the CerePlex-I system by Blackrock Microsystems LLC. NIS-3 constitutes a passive
device with percutaneous connector, attached to a wireless headstage, allowing
rigorous testing, market release, and regulatory clearance for a wireless module for
high-resolution, high-channel neural data. This is the current development stage for
the Utah electrode array by Blackrock Microsystems LLC. NIS-4 provides power
through a percutaneous connection but allows data to be transmitted wirelessly from
the implant. NIS-5 allows wireless power and data transfer (to a receiver hub in its
direct proximity). The NIS-6 stage finally allows wireless power and longer distance
transfer directly to, e.g., a beltpack receiver.
The abovementioned CerePlex-I system (Fig. 1.23) is an intermediate step toward
a wireless fully integrated system, which transfers part of the headstage functionality
to a digitization board which is implanted under the skin on the skull.
The device consists of three main parts: (1) a neural interface such as the UEA for
stimulation/recording of neural tissue, (2) a microsystem composed of an
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 33

Fig. 1.22 Potential roadmap and concepts for implantable wired and wireless neural interfaces.
(Image copyright and courtesy of Blackrock Microsystems LLC)

encapsulated printed circuit board (PCB) containing a 128 channel ASIC implanted
under the skin, and (3) an electrode tail connector referred to as the pigtail,
connecting the implanted part to the neural signal processing system. A combination
of the CerePlex I and multiport approach allows further scaling of channel count.
The vision for the future is the complete miniaturization and integration of all
components on the device. A first demonstration toward that goal was given by
Sharma et al. [105]. Such a system would allow to fully implant the recording
functionality by a microelectrode array in combination with an onboard first data
analysis and wireless power and data transmission. Further iterations of the device
designed for high-channel count would feature a modular design that amplifies and
multiplexes neural signals through an ASIC mounted on the chip but connected with
a wired connection to a pedestal connector or a chest-mounted pacemaker style
implantable module with rechargeable batteries and wireless data link. With
34 M. Leber et al.

Fig. 1.23 Image of the Blackrock Microsystems LLC CerePlex-I system connected to a UEA.
(Image copyright and courtesy of Blackrock Microsystems LLC)

constantly advancing microfabrication leading to decreasing footprint and ultralow

power consumption of the necessary electronic components, big steps toward a
realization of this vision are expected over the next years.

References

1. Abaya TVF et al (2012) 3D waveguide penetrating arrays for optical neural stimulation. In:
2012 international conference on optical MEMS and nanophotonics. IEEE, pp 216–217
2. Abaya TVF et al (2014) Deep-tissue light delivery via optrode arrays. J Biomed Opt 19
(1):15006
3. Abdo A et al (2011) Floating light-activated microelectrical stimulators tested in the rat spinal
cord. J Neural Eng 8(5):56012
4. Abdulagatov AI et al (2011) Al2O3 and TiO2 atomic layer deposition on copper for water
corrosion resistance. ACS Appl Mater Interfaces 3(12):4593–4601
5. Adamantidis AR et al (2007) Neural substrates of awakening probed with optogenetic control
of hypocretin neurons. Nature 450(7168):420–424
6. Aravanis AM et al (2007) An optical neural interface: in vivo control of rodent motor cortex
with integrated fiberoptic and optogenetic technology. J Neural Eng 4(3):S143–S156
7. Badwey JA, Karnovsky ML (1980) Active oxygen species and the functions of phagocytic
leucocytes. Annu Rev Biochem 49:695–726
8. Bajaj P et al (2007) Ultrananocrystalline diamond film as an optimal cell interface for
biomedical applications. Biomed Microdevices 9(6):787–794
9. Barrese JC et al (2013) Failure mode analysis of silicon-based intracortical microelectrode
arrays in non-human primates. J Neural Eng 10(6):66014
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 35

10. Bhandari R et al (2008) A novel method of fabricating convoluted shaped electrode arrays for
neural and retinal prostheses. Sensors Actuators A Phys 145–146:123–130
11. Bhandari R et al (2009) Wafer-scale processed, low impedance, neural arrays with varying
length microelectrodes. In: TRANSDUCERS 2009–2009 international solid-state sensors,
actuators and microsystems conference. IEEE, pp 1210–1213
12. Biederman W et al (2013) A fully-integrated, miniaturized (0.125 mm2) 10.5 μW wireless
neural sensor. IEEE J Solid State Circuits 48(4):960–970
13. Blackrock Microsystems (2018a) CerePlex Exilis. Available at: http://blackrockmicro.com/
cereplex-exilis/. Accessed 27 June 2018
14. Blackrock Microsystems (2018b) Cereplex W. Available at: http://blackrockmicro.com/
cereplex-wireless-headstage/. Accessed 27 June 2018
15. Bolzan AE et al (1988) Smooth and rough platinum deposits resulting from the
electroreduction of hydrous oxide platinum overlayers—a mechanistic approach. Electrochim
Acta 33(12):1743–1751
16. Borton DA et al (2013) An implantable wireless neural interface for recording cortical circuit
dynamics in moving primates. J Neural Eng 10(2):26010
17. Boutte RW, Blair S (2016) Maskless wafer-level microfabrication of optical penetrating neural
arrays out of soda-lime glass: Utah optrode array. Biomed Microdevices 18(6):115
18. Branner A, Stein RB, Normann RA (2001) Selective stimulation of cat sciatic nerve using an
Array of varying-length microelectrodes. J Neurophysiol 85(4):1585–1594
19. Buzsáki G (2004) Large-scale recording of neuronal ensembles. Nat Neurosci 7(5):446–451
20. Caldwell RB (2017) Strategies towards the mitigation of shunting in implanted neural arrays to
improve device stability for chronic applications (Order No. 10688653). Available from
ProQuest Dissertations & Theses Global. (2189849447)
21. Caldwell R et al (2017) Analysis of Al2O3 —parylene C bilayer coatings and impact of
microelectrode topography on long term stability of implantable neural arrays. J Neural Eng 14
(4):46011
22. Campbell PK et al (1991) A silicon-based, three-dimensional neural interface: manufacturing
processes for an intracortical electrode array. IEEE Trans Biomed Eng 38(8):758–768
23. Castagnola E et al (2014) Smaller, softer, lower-impedance electrodes for human
neuroprosthesis: a pragmatic approach. Front Neuroeng 7:8
24. Chestek CA et al (2009) HermesC: low-power wireless neural recording system for freely
moving Primates. IEEE Trans Neural Syst Rehabil Eng 17(4):330–338
25. Cheung K (2007) Implantable microscale neural interfaces. Biomed Microdevices 9
(6):923–938
26. Cho I-J, Baac HW, Yoon E (2010) A 16-site neural probe integrated with a waveguide for
optical stimulation. In: 2010 IEEE 23rd international conference on micro electro mechanical
systems (MEMS). IEEE, pp 995–998
27. Clark GA et al Selective, high-optrode-count, artifact-free stimulation with infrared light via
intrafascicular utah slanted optrode arrays. In Photonic Therapeutics and Diagnostics VIII
(Vol. 8207, p. 82075I). International Society for Optics and Photonics
28. Cogan SF (2008) Neural stimulation and recording electrodes. Annu Rev Biomed Eng
10:275–309
29. Cogan SF et al (2003) Plasma-enhanced chemical vapor deposited silicon carbide as an
implantable dielectric coating. J Biomed Mater Res 67A(3):856–867
30. Cogan SF et al (2016) Tissue damage thresholds during therapeutic electrical stimulation. J
Neural Eng 13(2):21001
31. Collinger JL et al (2013) High-performance neuroprosthetic control by an individual with
tetraplegia. Lancet 381(9866):557–564
32. Davis TS et al (2016) Restoring motor control and sensory feedback in people with upper
extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar
nerves. J Neural Eng 13(3):36001
36 M. Leber et al.

33. Denison T et al (2007) A 2 uW 100 nV/rtHz chopper-stabilized instrumentation amplifier for

chronic measurement of neural field potentials. IEEE J Solid State Circuits 42(12):2934–2945
34. Drake KL et al (1988) Performance of planar multisite microprobes in recording extracellular
single-unit intracortical activity. IEEE Trans Biomed Eng 35(9):719–732
35. Ersen A et al (2015) Chronic tissue response to untethered microelectrode implants in the rat
brain and spinal cord. J Neural Eng 12(1):16019
36. Fortin JB, Lu T-M (2004) Chemical vapor deposition polymerization: the growth and prop-
erties of parylene thin films. Kluwer Academic Publishers, Boston
37. Foster DJ et al (2014) A freely-moving monkey treadmill model. J Neural Eng 11(4):46020
38. Franks W et al (2005) Impedance characterization and modeling of electrodes for biomedical
applications. IEEE Trans Biomed Eng 52(7):1295–1302
39. Gradinaru V et al (2007) Targeting and readout strategies for fast optical neural control in vitro
and in vivo. J Neurosci 27(52):14231–14238
40. Green RA et al (2012) Variation in performance of platinum electrodes with size and surface
roughness. Sens Mater 24(4):165–180
41. Green RA et al (2014) Laser patterning of platinum electrodes for safe neurostimulation. J
Neural Eng 11(5):56017
42. Greenwald E et al (2011) A VLSI neural monitoring system with ultra-wideband telemetry for
awake behaving subjects. IEEE Trans Biomed Circuits Syst 5(2):112–119
43. Hämmerle H et al (2002) Biostability of micro-photodiode arrays for subretinal implantation.
Biomaterials 23(3):797–804
44. Harrison RR et al (2011) Wireless neural/EMG telemetry Systems for Small Freely Moving
Animals. IEEE Trans Biomed Circuits Syst 5(2):103–111
45. Hassler C et al (2010) Characterization of parylene C as an encapsulation material for
implanted neural prostheses. J Biomed Mater Res B Appl Biomater 9999B(1):266–274
46. Hemmerich KJ (1998) General aging theory and simplified protocol for accelerated aging of
medical devices. Med Plast Biomaterials 5:16–23
47. Hochberg LR et al (2006) Neuronal ensemble control of prosthetic devices by a human with
tetraplegia. Nature 442(7099):164–171
48. Hochberg LR et al (2012) Reach and grasp by people with tetraplegia using a neurally
controlled robotic arm. Nature 485(7398):372–375
49. Hsu J-M et al (2007) Characterization of a-SiCx:H thin films as an encapsulation material for
integrated silicon based neural interface devices. Thin Solid Films 516(1):34–41
50. Hsu JM et al (2008) Effect of thermal and deposition processes on surface morphology,
crystallinity, and adhesion of Parylene-C. Sens Mater 20(2):87–102
51. Hsu J-MJ et al (2009) Encapsulation of an integrated neural interface device with Parylene
C. IEEE Trans Biomed Eng 56(1):23–29
52. Hukins DWL, Mahomed A, Kukureka SN (2008) Accelerated aging for testing polymeric
biomaterials and medical devices. Med Eng Phys 30(10):1270–1274
53. Jackson A, Mavoori J, Fetz EE (2007) Correlations between the same motor cortex cells and
arm muscles during a trained task, free behavior, and natural sleep in the macaque monkey. J
Neurophysiol 97(1):360–374
54. Janting J, Branebjerg J, Rombach P (2001) Conformal coatings for 3D multichip microsystem
encapsulation. Sensors Actuators A Phys 92(1–3):229–234
55. Jiang G, Zhou DD (2010) Technology advances and challenges in hermetic packaging for
implantable medical devices. In: Zhou D, Greenbaum E (eds) Implantable neural prostheses 2:
techniques and engineering approaches. Springer, New York, pp 27–61
56. Jun JJ et al (2017) Fully integrated silicon probes for high-density recording of neural activity.
Nature 551(7679):232–236
57. Keuning W et al (2012) Cathode encapsulation of organic light emitting diodes by atomic
layer deposited Al2O3 films and Al2O3/a-SiNx:H stacks. J Vac Sci Technol A Vac Surf Films
30(1):01A131
58. Kim S et al (2009) Integrated wireless neural interface based on the Utah electrode array.
Biomed Microdevices 11(2):453–466
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 37

59. Klein JD, Clauson SL, Cogan SF (1989) Morphology and charge capacity of sputtered iridium
oxide films. J Vac Sci Technol A 7(5):3043–3047
60. Kozai TDY et al (2014) Chronic tissue response to carboxymethyl cellulose based dissolvable
insertion needle for ultra-small neural probes. Biomaterials 35(34):9255–9268
61. Kuppusami S, Oskouei RH (2015) Parylene coatings in medical devices and implants: a
review. Univ J Biomed Eng 3(2):9–14
62. Leach J, Achyuta AKH, Murthy SK (2010) Bridging the divide between neuroprosthetic
design, tissue engineering and neurobiology. Front Neuroeng 2:18
63. Leber M (2018) Unpublished (in progress). unpublished
64. Leber M et al (2016) Different methods to alter surface morphology of high aspect ratio
structures. Appl Surf Sci 365:180
65. Leber M et al (2017) Novel method of fabricating self-dissolvable and freely floating neural
array. In 2017 19th international conference on solid-state sensors, actuators and microsystems
(TRANSDUCERS). IEEE, pp 1726–1729
66. Ledermann N et al (2000) Sputtered silicon carbide thin films as protective coating for MEMS
applications. Surf Coat Technol 125(1–3):246–250
67. Lee H et al (2005) Biomechanical analysis of silicon microelectrode-induced strain in the
brain. J Neural Eng 2(4):81–89
68. Lee SB et al (2013) A wideband dual-antenna receiver for wireless recording from animals
behaving in large arenas. IEEE Trans Biomed Eng 60(7):1993–2004
69. Limb SJ et al (1998) Growth of fluorocarbon polymer thin films with high CF2 fractions and
low dangling bond concentrations by thermal chemical vapor deposition. Appl Phys Lett 68
(20):2810
70. Loeb GE et al (1977) Parylene as a chronically stable, reproducible microelectrode insulator.
IEEE Trans Biomed Eng BME-24(2):121–128
71. Marrese CA (1987) Preparation of strongly adherent platinum black coatings. Anal Chem 59
(1):217–218
72. Meyer RD et al (2001) Electrodeposited iridium oxide for neural stimulation and recording
electrodes. IEEE Trans Neural Syst Rehabil Eng 9(1):2–11
73. Minnikanti S et al (2014) Lifetime assessment of atomic-layer-deposited Al2O3–Parylene C
bilayer coating for neural interfaces using accelerated age testing and electrochemical charac-
terization. Acta Biomater 10(2):960–967
74. Miranda H et al (2010) HermesD: a high-rate long-range wireless transmission system for
simultaneous multichannel neural recording applications. IEEE Trans Biomed Circuits Syst 4
(3):181–191
75. Muller R et al (2014) A miniaturized 64-channel 225μW wireless electrocorticographic neural
sensor. In: 2014 IEEE international solid-state circuits conference digest of technical papers
(ISSCC). IEEE, pp 412–413
76. Multi Channel Systems (2018) Wireless-systems. Available at: https://www.
multichannelsystems.com/products/wireless-systems. Accessed 27 June 2018
77. Musallam S et al (2007) A floating metal microelectrode array for chronic implantation. J
Neurosci Methods 160(1):122–127
78. Najafi K, Wise KD, Mochizuki T (1985) A high-yield IC-compatible multichannel recording
array. IEEE Trans Electron Devices 32(7):1206–1211
79. Nature (2011) Method of the year 2010. Nat Methods 8(1):1–1
80. Negi S et al (2009) Effect of sputtering pressure on pulsed-DC sputtered iridium oxide films.
Sensors Actuators B Chem 137(1):370–378
81. Negi S, Bhandari R, Rieth L, Solzbacher F (2010a) In vitro comparison of sputtered iridium
oxide and platinum-coated neural implantable microelectrode arrays. Biomed Mater (Bristol,
England) 5(1):15007
82. Negi S, Bhandari R, Rieth L, Van Wagenen R et al (2010b) Neural electrode degradation from
continuous electrical stimulation: comparison of sputtered and activated iridium oxide. J
Neurosci Methods 186(1):8–17
38 M. Leber et al.

83. Negi S, Bhandari R, Solzbacher F (2012) Morphology and electrochemical properties of

activated and sputtered iridium oxide films for functional electrostimulation. J Sensor Technol
2(3):138–147
84. NeuraLynx (2018) Cube2. Available at: https://neuralynx.com/hardware/cube2. Accessed
27 June 2018
85. Park S et al (2010) Nanoporous Pt microelectrode for neural stimulation and recording: in vitro
characterization. J Phys Chem C 114(19):8721–8726
86. Pashaie R et al (2014) Optogenetic brain interfaces. IEEE Rev Biomed Eng 7:3–30
87. Prochazka A, Mushahwar VK, McCreery DB (2001) Neural prostheses. J Physiol 533
(1):99–109
88. Richardson RR, Miller JA, Reichert WM (1993) Polyimides as biomaterials: preliminary
biocompatibility testing. Biomaterials 14(8):627–635
89. Rizk M et al (2009) A fully implantable 96-channel neural data acquisition system. J Neural
Eng 6(2):26002
90. Robblee LS et al (1983a) Electrical stimulation with Pt electrodes. VII. Dissolution of Pt
electrodes during electrical stimulation of the cat cerebral cortex. J Neurosci Methods 9
(4):301–308
91. Robblee LS, Lefko JL, Brummer SB (1983b) Activated Ir: an electrode suitable for reversible
charge injection in saline solution. J Electrochem Soc 130(3):731
92. Robblee LS et al (1985) Charge injection properties of thermally-prepared iridium oxide films.
MRS Proc 55:303
93. Roy RK, Lee K-R (2007) Biomedical applications of diamond-like carbon coatings: a review.
J Biomed Mater Res B Appl Biomater 83B(1):72–84
94. Sato H et al (2009) Remote radio control of insect flight. Front Integr Neurosci 3:24
95. Scharf R et al (2016) Depth-specific optogenetic control in vivo with a scalable, high-density
μLED neural probe. Sci Rep 6(1):28381
96. Scharf R, Reiche CF, McAlinden N, Cheng Y, Xie E, Sharma R, Tathireddy P, Rieth L,
Mathieson K, Blair S (2018) A compact integrated device for spatially selective optogenetic
neural stimulation based on the Utah Optrode Array. In Optogenetics and Optical Manipula-
tion 2018 (Vol. 10482, p. 104820M). International Society for Optics and Photonics
97. Schmitt G et al (1999) Passivation and corrosion of microelectrode arrays. Electrochim Acta
44(21–22):3865–3883
98. Schwartz AB et al (2006) Brain-controlled interfaces: movement restoration with neural
prosthetics. Neuron 52(1):205–220
99. Schwarz DA et al (2014) Chronic, wireless recordings of large-scale brain activity in freely
moving rhesus monkeys. Nat Methods 11(6):670–676
100. Seidl K et al (2011) CMOS-based high-density silicon microprobe arrays for electronic depth
control in Intracortical neural recording. J Microelectromech Syst 20(6):1439–1448
101. Selbmann F et al (2016) Deposition of Parylene C and characterization of its hermeticity for
the encapsulation of MEMS and medical devices. In: 2016 IEEE 11th annual international
conference on nano/micro engineered and molecular systems (NEMS). IEEE, pp 427–432
102. Seymour JP et al (2009) The insulation performance of reactive parylene films in implantable
electronic devices. Biomaterials 30(31):6158–6167
103. Shandhi MMH et al (2015) A novel method of fabricating high channel density neural array
for large neuronal mapping. In: 2015 Transducers – 2015 18th international conference on
solid-state sensors, actuators and microsystems (TRANSDUCERS). IEEE, pp 1759–1762
104. Shandhi MMH et al (2017) Reusable high aspect ratio 3-D nickel shadow mask. J
Microelectromech Syst 26(2):376–384
105. Sharma A et al (2010) Long term in vitro stability of fully integrated wireless neural interfaces
based on Utah slant electrode array. Appl Phys Lett 96(7):73702
106. Sharma R et al (2011) Application-specific customizable architectures of Utah neural inter-
faces. Procedia Eng 25:1016–1019
1 Advances in Penetrating Multichannel Microelectrodes Based on the. . . 39

107. Sharma A et al (2012) Evaluation of the packaging and encapsulation reliability in fully
integrated, fully wireless 100 channel Utah slant electrode Array (USEA): implications for
long term functionality. Sensors Actuators A Phys 188:167–172
108. Stadler S, Ajmera PK (2008) Sensors and materials: an international journal on sensor
technology. Scientific Publ. Division of MYU, Tokyo
109. Subbaroyan J, Martin DC, Kipke DR (2005) A finite-element model of the mechanical effects
of implantable microelectrodes in the cerebral cortex. J Neural Eng 2(4):103–113
110. Szuts TA et al (2011) A wireless multi-channel neural amplifier for freely moving animals. Nat
Neurosci 14(2):263–269
111. Takmakov P et al (2015) Rapid evaluation of the durability of cortical neural implants using
accelerated aging with reactive oxygen species. J Neural Eng 12(2):26003
112. Tan X et al (2018) Auditory neural activity in congenitally deaf mice induced by infrared
neural stimulation. Sci Rep 8(1):388
113. Triangle Biosystems International (2018) W-Series. Available at: http://www.
trianglebiosystems.com/w-series-systems.html. Accessed 27 June 2018
114. Troyk PR, Cogan SF (2005) Sensory neural prostheses. In: Neural engineering. Springer,
Boston, pp 1–48
115. Tykocinski M et al (2001) Chronic electrical stimulation of the auditory nerve using high
surface area (HiQ) platinum electrodes. Hear Res 159:53–68
116. Vanhoestenberghe A, Donaldson N (2013) Corrosion of silicon integrated circuits and lifetime
predictions in implantable electronic devices. J Neural Eng 10(3):31002
117. Wang J et al (2012) Integrated device for combined optical neuromodulation and electrical
recording for chronic in vivo applications. J Neural Eng 9(1):16001
118. Wark HAC et al (2013) A new high-density (25 electrodes/mm2) penetrating microelectrode
array for recording and stimulating sub-millimeter neuroanatomical structures. J Neural Eng
10(4):45003
119. Wellman SM et al (2017) A materials roadmap to functional neural Interface design. Adv
Funct Mater 28(12):1701269
120. Wells J et al (2005) Optical stimulation of neural tissue in vivo. Opt Lett 30(5):504
121. Weremfo A et al (2015) Investigating the interfacial properties of electrochemically roughened
platinum electrodes for neural stimulation. Langmuir 31(8):2593–2599
122. Wessling B, Mokwa W, Schnakenberg U (2006) RF-sputtering of iridium oxide to be used as
stimulation material in functional medical implants. J Micromech Microeng 16(6):S142–S148
123. Whalen JJ et al (2006) Electrochemical characterization of charge injection at electrodeposited
platinum electrodes in phosphate buffered saline. J Electrochem Soc 153(12):C834
124. Wise KD (2005) Silicon microsystems for neuroscience and neural prostheses. IEEE Eng Med
Biol Mag 24(5):22–29
125. Wise KD, Angell JB, Starr A (1970) An integrated-circuit approach to extracellular micro-
electrodes. IEEE Trans Biomed Eng BME-17(3):238–247
126. Wu F et al (2015) Monolithically integrated μLEDs on silicon neural probes for high-
resolution Optogenetic studies in behaving animals. Neuron 88(6):1136–1148
127. Xiang Z et al (2014) Ultra-thin flexible polyimide neural probe embedded in a dissolvable
maltose-coated microneedle. J Micromech Microeng 24(6):65015
128. Xiao X et al (2006) In vitro andin vivo evaluation of ultrananocrystalline diamond for coating
of implantable retinal microchips. J Biomed Mater Res B Appl Biomater 77B(2):273–281
129. Xie X (2013) Atomic layer deposited aluminum oxide and parylene C bi-layer encapsulation
for biomedical implantable devices (Order No. 3607382). Available from ProQuest Disserta-
tions & Theses Global. (1492997360)
130. Xie X et al (2012) Plasma-assisted atomic layer deposition of Al2O3 and parylene C bi-layer
encapsulation for chronic implantable electronics. Appl Phys Lett 101(9):93702
131. Xie X et al (2013) Long-term bilayer encapsulation performance of atomic layer deposited
Al2O3 and parylene c for biomedical implantable devices. IEEE Trans Biomed Eng 60
(10):2943–2951
40 M. Leber et al.

132. Xie X, Rieth L, Williams L et al (2014a) Long-term reliability of Al2O3 and Parylene C
bilayer encapsulated Utah electrode array based neural interfaces for chronic implantation. J
Neural Eng 11(2):26016
133. Xie X, Rieth L, Negi S et al (2014b) Self aligned tip Deinsulation of atomic layer deposited
Al2O3 and Parylene C coated Utah electrode Array based neural interfaces. J Micromech
Microeng Struct Devices Syst 24(3):35003
134. Xie W, Kothari V, Terry BS (2015) A bio-inspired attachment mechanism for long-term
adhesion to the small intestine. Biomed Microdevices 17(4):68
135. Yao Q et al (1999) Adhesion enhancement of underfill materials by silane additives. In:
Proceedings – international symposium on advanced packaging materials: processes, proper-
ties and interfaces. IMAPS – International microelectronics assembly and packaging society,
pp 27–30
136. Yin M et al (2013) A 100-channel hermetically sealed implantable device for chronic wireless
neurosensing applications. IEEE Trans Biomed Circuits Syst 7(2):115–128
137. Yin M et al (2014) Wireless neurosensor for full-spectrum electrophysiology recordings
during free behavior. Neuron 84(6):1170–1182
138. Yoo J-M et al (2012) Hybrid laser and reactive ion etching of Parylene-C for deinsulation of a
Utah electrode array. J Micromech Microeng 22(10):105036
139. Yoo J-M et al (2013) Excimer laser deinsulation of Parylene-C on iridium for use in an
activated iridium oxide film-coated Utah electrode array. J Neurosci Methods 215(1):78–87
140. Yu Q, Deffeyes J, Yasuda H (2001) Engineering the surface and interface of Parylene C
coatings by low-temperature plasmas. Prog Org Coat 41(4):247–253
141. Zhang J et al (2009) Integrated device for optical stimulation and spatiotemporal electrical
recording of neural activity in light-sensitized brain tissue. J Neural Eng 6(5):055007
142. Zhou D (2007) Platinum electrode surface coating and method for manufacturing the same
Chapter 2
EEG-Based Brain-Computer Interfaces

Yijun Wang, Masaki Nakanishi, and Dan Zhang

Abstract Brain-computer interfaces (BCIs) provide a direct communication chan-

nel between human brain and output devices. Due to advantages such as
non-invasiveness, ease of use, and low cost, electroencephalography (EEG) is the
most popular method for current BCIs. This chapter gives an overview of the current
EEG-based BCIs for the main purpose of communication and control. This chapter
first provides a taxonomy of the EEG-based BCI systems by categorizing them into
three major groups: (1) BCIs based on event-related potentials (ERPs), (2) BCIs
based on sensorimotor rhythms, and (3) hybrid BCIs. Next, this chapter describes
challenges and potential solutions in developing practical BCI systems toward high
communication speed, convenient system use, and low user variation. Then this
chapter briefly reviews both medical and non-medical applications of current BCIs.
Finally, this chapter concludes with a summary of current stage and future perspec-
tives of the EEG-based BCI technology.

Keywords Brain-computer interfaces · EEG-based BCI · hybrid BCI · ERP ·

SSVEP

2.1 Introduction

Since Vidal first proposed the term “brain-computer interface (BCI)” in early 1970s
[1], BCIs have received increasing attention in the fields of neuroscience, neural
engineering, and clinical rehabilitation [2–4]. Especially during the past two

Y. Wang (*)
Institute of Semiconductors, Chinese Academy of Sciences, Beijing, China
e-mail: [email protected]
M. Nakanishi
Institute for Neural Computation, University of California San Diego, San Diego, CA, USA
D. Zhang
Department of Psychology, Tsinghua University, Beijing, China

© Springer Nature Singapore Pte Ltd. 2019 41

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_2
42 Y. Wang et al.

Fig. 2.1 System diagram of a typical EEG-based BCI system for communication and control

decades, BCI has rapidly become a hot topic in both research and practice. The
primary goal of BCI research is to provide a non-muscular communication channel
to help people with severe motor disabilities to communicate with their environ-
ments. Among different brain imaging techniques that have been applied to BCIs,
electroencephalography (EEG) has become the most popular method due to its
advantages such as non-invasiveness, ease of use, and low cost. Figure 2.1 shows
the system diagram of a typical EEG-based BCI system for communication and
control, which consists of three major components: signal acquisition, data
processing, and device control. Although the scope of application of EEG-based
BCIs has been significantly extended in recent years, this chapter mainly focuses on
the traditional EEG-based BCIs for the purpose of communication and control.
In the twentieth century, early works by research pioneers laid important ground-
work for the BCI field. In the 1970s, Vidal first proposed the term of BCI for the
purpose of controlling external apparatus with brain signals and developed a BCI
system based on visual event-related potentials (ERPs) [5]. The system converted
ERPs corresponding to four retinotopic maps of a checkerboard stimulus into
movement control commands to move a mobile through a maze. In 1988, Farwell
and Donchin described a mental prosthesis system using the P300 component of
ERPs [6]. The system consisting of a six-by-six matrix served as a keyboard and was
operated by flash elicited P300 at a speed of ~2 characters per minute. In the early
1990s, BCI systems based on sensorimotor rhythms emerged. Woplaw et al. trained
subjects to learn self-control of amplitude of mu rhythm [7]. The system realized
one-dimensional cursor control by translating mu rhythm amplitudes into cursor
movements. Pfurtscheller et al. performed a BCI system based on event-related
desynchronization (ERD) of sensorimotor rhythms [8]. The system was operated
by discriminating the spatio-temporal patterns of ERD during imagination of left and
right hand movements. During this period, the ERP-based BCI paradigms were
2 EEG-Based Brain-Computer Interfaces 43

further extended. Sutter developed a brain response interface using pseudo-random

sequence modulated visual evoked potentials (VEPs) in 1992 [9]. The eight-by-eight
visual keyboard recognized the user’s eye gaze direction using VEPs and obtained
high communication rates. In the late 1990s, Birbaumer et al. developed a spelling
device for the completely paralyzed patients based on slow cortical potentials (SCPs)
[10]. The system allowed the subjects to select letters by driving a cursor with their
SCPs using an imagery strategy.
Almost two decades into the twenty-first century, BCI research has achieved
significant progress and shown promising prospects and applications in various
fields. The progress mainly includes the following aspects: signal processing
[11, 12], machine learning [13, 14], system paradigm [15, 16], and applications
[17, 18]. Advanced EEG signal processing and machine learning algorithms have
been applied to significantly improve the performance of the early developed BCI
paradigms (e.g., BCIs based on P300 and sensorimotor rhythms). Meanwhile, new
BCI paradigms have been proposed and achieved rapid development. Various types
of visual and auditory BCIs based on ERPs have been developed and proven highly
efficient in terms of information transfer rate (ITR) [19]. A hybrid BCI system that
combines a traditional BCI with other physiological signals has shown advantages in
BCI performance and system flexibility [20]. Over the past decade, a wide range of
BCI applications have emerged, which considerably expanded the original goals.
Assistive BCIs and rehabilitative BCIs, which are designed for communication and
motor rehabilitation, have been widely investigated in experimental and clinical
studies [21]. In addition, the potential applications of BCI go far beyond the medical
applications. For example, passive BCIs have been developed for assessing and
monitoring mental states in real time [22].
Although the performance of BCI systems has improved significantly in recent
years, researchers still face a lot of challenges in promoting wide use of the BCI
technology. On one hand, compared to traditional input technologies, the current
levels of BCI performance are still very low and do not support fast and complex
communication and control. The performance bottleneck is mainly caused by the
lack of knowledge of the underlying neural mechanisms of information encoding in
EEG signals. Besides, decoding EEG signals requires more efficient computational
modeling approaches based on signal processing and machine learning techniques.
On the other hand, transitioning BCI systems from a well-controlled laboratory
setting to a real-life environment remains challenging due to the complications of
EEG measurement [23]. A practical BCI system must meet the requirements of good
user experience and robust system performance [24]. At present, there is a great need
for a mobile EEG platform with user-friendly electrode technology.
This chapter gives a comprehensive overview of the current EEG-based BCIs for
the purpose of communication and control. This chapter first provides a taxonomy of
the EEG-based BCI systems by categorizing them into three major groups: (1) BCIs
based on ERPs, (2) BCIs based on sensorimotor rhythms, and (3) hybrid BCIs. And
then, this chapter describes challenges and potential solutions in developing practical
BCI systems for real-life applications toward high communication speed, convenient
system use, and low user variation. Next, this chapter briefly reviews both medical
44 Y. Wang et al.

and non-medical applications of current BCIs. Finally, this chapter concludes with a
summary of current stage and future perspectives of the EEG-based BCI technology.

2.2 Taxonomy of Current BCIs

Since many brain signals and system paradigms have been applied to BCIs, a
taxonomy is useful for providing a summary of the current status of BCIs. A
taxonomy not only provides a systematic description of brain signals and modulation
methods used in system design but also aids building general frameworks in system
implementation. In general, BCI systems can be classified by the brain signals
employed in the system (e.g., P300, VEP, or sensorimotor rhythms) [2]. Alterna-
tively, BCIs can be categorized by operation paradigms like dependent/independent
BCIs or synchronous/asynchronous BCIs [15]. Recently, a taxonomy of visual and
auditory BCIs was proposed according to the multiple target access methods used in
information encoding [19]. To give a meaningful overview of the current BCI
systems, this chapter provides a simple taxonomy of BCI based on the types of
EEG control signals. By jointly considering brain signals summarized in recent
reviews [19, 20, 25], this chapter categorizes the current BCI systems into three
major groups: (1) BCIs based on ERPs, (2) BCIs based on sensorimotor rhythms,
and (3) hybrid BCIs. Figure 2.2 illustrates the proportions of journal papers in ISI
Web of Science for these three BCI groups in the last five years (2012–2016). The
numbers of papers for the ERP-based BCIs and the sensorimotor BCIs are

Fig. 2.2 Distribution of major BCI studies including ERP BCI, sensorimotor BCI, and hybrid BCI
in recent years.
Notes: Data were obtained from ISI Web of Science using the corresponding topics of journal
papers during the last five years (2012–2016). The analysis includes a total of 2658 papers (1234
papers for the ERP BCI, 1213 papers for the sensorimotor BCI, and 211 papers for the hybrid BCI).
2 EEG-Based Brain-Computer Interfaces 45

comparable (1234 versus 1213, percentage: 46.4% versus 45.6%), which are signif-
icantly larger than that of the hybrid BCIs (211, percentage: 7.9%). Note that the
hybrid BCIs also include other physiological signals such as electromyogram
(EMG) and electrooculogram (EOG) [20]. Without loss of generality, this chapter
focuses on the most popular brain signals and system paradigms used in recent BCI
studies.

2.2.1 BCIs Based on ERPs

2.2.1.1 ERP Signals for BCI

ERPs, which were originally called evoked potentials (EPs), are measured electrical
brain potentials that are evoked by stimuli, which display stable time relationships to
a reference event (as opposed to the spontaneous EEG rhythms) [26]. Typically, an
ERP waveform consists of multiple peaks and troughs, which are called ERP
components. Major ERP components include visual and auditory responses, N2,
P3, response-related components, and language-related components. Among these
ERP components, two types of signals have been widely used in BCI systems since
very early studies in the 1970s and 1980s [5, 6]. The first type is the VEP family,
which includes subtypes of transient VEP (TVEP), steady-state VEP (SSVEP),
motion VEP (mVEP), and code modulated VEP (cVEP). VEPs are ERPs elicited
by visual stimuli, which can be measured with maximum amplitude over the
occipital region [27]. Event-related P300 component is another type of commonly
used ERP signal. P300, a positive deflection around 300–400 ms after a relevant
stimulus, which can be measured over the central and parietal areas, reflects stimulus
evaluation, selective attention, and conscious discrimination in an oddball task
[28]. In addition to VEP and P300 signals, other types of ERP signals including
auditory evoked potential, N2pc, error-related negativity, and N400 have also been
investigated in a few BCI studies.
ERP signals are modulated by exogenous stimuli or endogenous mental activities.
The exogenous stimuli in BCI systems can be visual or auditory stimuli (e.g., VEPs
are elicited by exogenous visual stimuli), while endogenous activities include users’
attention or mental tasks (e.g., P300 is an endogenous brain response to an oddball
stimulus). In a BCI system, the ERP signals of sensation, perception, or cognition
can be modulated by voluntary control (e.g., attention) of the subject. The informa-
tion encoded in the modulated ERP signals can be decoded by signal processing and
classification algorithms. In contrast to spontaneous EEG rhythms that do not require
external stimuli, the ERP signals exhibit good characteristics including high signal-
to-noise ratio (SNR), reliable spatio-temporal features, and low user variation. With
rapid advances of signal modulation and demodulation techniques, ERP-based BCI
systems have achieved very high communication speed and therefore become more
popular in recent BCI research [19].
46 Y. Wang et al.

2.2.1.2 Modulation of ERP Signals

Signal modulation plays an important role in the ERP-based BCIs. First, to convey
the user’s intent, ERP signals must be modulated by the user’s mental activities in an
efficient way so that the modulated ERP signals can be reliably demodulated into the
original messages. Second, toward high communication rate, a large bandwidth is
required so that the modulated ERP signals corresponding to different intents can be
easily discriminated. By introducing the multiple access (MA) techniques in tele-
communications [29], the signal modulation approaches in ERP-based BCI systems
can be classified by the major MA techniques: time division multiple access
(TDMA), frequency division multiple access (FDMA), code division multiple
access (CDMA), and space division multiple access (SDMA). This classification
can facilitate the understanding of various ERP-based BCI paradigms, which can be
considered under a unified signal modulation framework.
First, TDMA allows multiple users to share the same communication channel by
dividing the signal into different time slots. In ERP-based BCIs, TDMA is the most
popular method for multiple target coding. In these BCI systems, multiple targets
appear at different time slots. For example, TDMA has been widely used in the
visual P300-based BCI systems [30]. In the standard ‘row/column’ paradigm, in
which each row and column are flashed in a random order, flashing of the row and
column of the desired character will elicit P300 signals. Detection of P300 in
different time slots can identify the target character. The other single character,
checkerboard, and region-based P300 paradigms all follow the principle of
TDMA. In addition to P300, the VEP-based BCIs using mVEP [31] and TVEP
[32] also employ the TDMA method. Second, FDMA divides the entire bandwidth
into multiple individual bands, each for a single user. The SSVEP-based BCI [33] is
a typical FDMA system. In a typical frequency coding-based SSVEP BCI, all stimuli
flash concurrently at different frequencies, and a target stimulus can be identified by
analyzing the frequency of the elicited SSVEP. In ERP-based BCIs, FDMA has
higher capacity than TDMA. The SSVEP-based BCIs can discriminate 40 different
frequencies within a short stimulus duration (e.g., 500 ms) and thereby achieve very
high ITRs [34, 35]. Third, CDMA assigns separate codes to modulate users’ signals.
The BCIs based on pseudorandom cVEP [9, 36] are typical CDMA systems. The
CDMA BCI systems show high performance comparable to FDMA. Last, SDMA
divides the geographical space into smaller spaces for multiple users. The SDMA
BCI systems use retinotopic mappings of different stimulus locations in the visual
field to elicit different spatial patterns of TVEP or SSVEP signals [5, 37]. The
number of targets in SDMA is generally limited by the low spatial resolution of
EEG-based retinotopic mapping.

2.2.1.3 Demodulation of ERP Signals

In an ERP-based BCI, information embedded in the modulated ERP signals needs to

be demodulated and converted into control commands. Due to high trial-to-trial
2 EEG-Based Brain-Computer Interfaces 47

variability of ERP signals, advanced data analysis methods are required in the
demodulation of ERP, especially in signal-trial analysis and classification of ERP
[38]. ERP demodulation can be considered an EEG pattern recognition problem,
which includes data pre-processing, feature extraction, and classification. Signal
processing [11] and machine learning [13] are two major data analysis techniques.
The signal processing algorithms have been widely used in the pre-processing and
feature extraction procedures. The aim of data pre-processing is to improve the SNR
of the ERP signals by removing task-irrelevant noises. Bandpass filtering and spatial
filtering are two commonly used pre-processing procedures. Bandpass filtering only
allows the frequency range of ERP signals to remain. Spatial filtering further
enhances ERP signals by calculating a weighted linear combination of multiple
EEG channels [39]. Besides, trial averaging is another efficient way to enhance the
SNR of ERPs. After pre-processing, features that reflect the characteristic of ERP
signals can be extracted. The features for ERPs generally exploit the spatio-temporal
information of the signals (e.g., concatenation of ERP amplitude for all time points
and all channels) [38]. In addition to time and space domains, features in the
frequency domain (e.g., power spectral density) have also been used in the detection
of SSVEP signals [40]. After feature extraction, the machine learning algorithms are
applied to classification (e.g., discriminating target and non-target ERPs). Classifiers
such as linear discriminant analysis (LDA) and support vector machine (SVM) have
been widely used in the classification of ERPs with the extracted spatio-temporal
features. In online BCI systems, classifiers are obtained from labelled training data
collected in calibration sessions and then applied to predict the label of testing data
during the system operation.

2.2.1.4 Dependent and Independent ERP-Based BCIs

EEG-based BCIs are categorized as dependent and independent systems [2]. An

independent BCI does not depend on the brain’s normal output pathways (i.e.,
peripheral nerves and muscles) to generate the brain activity that carries the user’s
intent. In contrast, a dependent BCI requires some activity from peripheral nerves
and muscles to modulate the brain activity. The ERP-based BCIs include both
dependent and independent systems. In BCIs, ERP signals are generally modulated
by eye fixation or attention. BCIs based on visual ERPs majorly depend on the
control of the muscles of the eye to gaze at desired targets and fall into the dependent
BCI class. Most of VEP and visual P300-based BCIs use gaze-dependent paradigms.
Different from direct eye gaze (i.e., overt attention), covert attention mechanisms
such as spatial selective attention and feature selective attention have been investi-
gated to develop gaze-independent BCI paradigms. There are three types of inde-
pendent BCIs based on ERP signals: auditory BCIs, tactile BCIs, and independent
visual BCIs [41]. For example, using standard oddball tasks, P300 signals can be
elicited by auditory, tactile, or visual stimuli, which do not require gaze control to
select a target. In addition, SSVEP BCIs that use spatial or feature selective attention
to modulate the amplitude of attended/unattended SSVEPs are independent
[42, 43]. Due to a larger number of classes and a higher level of signal modulation,
48 Y. Wang et al.

the dependent BCIs can achieve much higher communication speed than the inde-
pendent BCIs. However, for totally locked-in patients who even lose extraocular
muscle control, only independent BCIs are useful.

2.2.2 BCIs Based on Sensorimotor Rhythms

2.2.2.1 Sensorimotor Rhythms for BCI

BCI systems based on sensorimotor mu/beta rhythms have been developed rapidly
in recent years [25]. The neurophysiological studies of the sensorimotor cortex
reveal that mu (8–12 Hz) and beta (18–26 Hz) rhythms are modulated by actual
movement or preparation of movement. The event-related power change of brain
rhythms in a specific frequency band is known as event-related desynchronization
and synchronization (ERD/ERS) [44]. Specifically, in the sensorimotor cortex, ERD
indicates power decrease of mu/beta rhythms during movement. In contrast, ERS
indicates power increase that typically occurs after movement. ERD/ERS of senso-
rimotor rhythms occur not only with movement, but also with preparation for
movement or motor imagery (i.e., imagined movement). During motor imagery,
ERD/ERS of mu/beta rhythms display characteristic spatial patterns corresponding
to different body parts (e.g., left hand versus right hand). Source imaging studies
further show that the ERD sources are located in corresponding cortical areas
somatotopically during motor imagery of different body parts. These findings
suggest that motor imagery can provide a good way to implement an independent
BCI based on self-regulation of the sensorimotor rhythms.
Motor imagery is the basic strategy to operate a sensorimotor BCI. Many studies
have demonstrated that people can learn to control (increase or decrease) the
amplitude of the sensorimotor rhythms using motor imagery. For example, with
the Wadsworth BCI developed by Wolpaw and his colleagues, subjects were trained
to learn self-control of mu/beta rhythm amplitude to realized 1-D, 2-D, or 3-D cursor
control by translating mu/beta rhythm amplitudes into cursor movements [7, 45,
46]. In the initial training sessions, subjects mostly employed motor imagery (e.g.,
imagination of hand movements) to control the cursor movement. Instead of con-
tinuous movement control, another approach for sensorimotor BCIs is to classify the
motor imagery states (e.g., left hand, right hand, feet) using pattern classification
techniques. Spatial patterns of ERD/ERS of the mu/beta rhythms are used as features
for classification. The Graz BCI developed by Pfurtscheller and his colleagues
employed spatial filters and classifiers to train subjects to generate discriminable
motor imagery states and therefore built a co-adaptive learning system, which
allowed brain learning and machine learning at the same time [8, 47]. The system
can discriminate more complex spatio-temporal patterns of ERD/ERS during imag-
ination of movements of different body parts including left and right hands, feet, and
tongue [48].
2 EEG-Based Brain-Computer Interfaces 49

2.2.2.2 Feature Extraction and Classification

The most commonly used features for indexing ERD/ERS of the mu/beta rhythms
include band power values, amplitude envelope by the Hilbert transform, and
autoregressive model [47]. Time-frequency analysis can provide more detailed
features by jointly considering EEG dynamics in frequency and time domains and
thereby improve the classification performance [49, 50]. In addition to temporal and
spectral features, since ERD/ERS during motor imagery exhibit characteristic spatial
patterns, the extraction of spatial features is especially important for the discrimina-
tion of motor imagery states. In the space domain, various approaches have been
proposed to enhance the extraction of spatial features. First, spatial filtering tech-
niques can significantly enhance the SNR of the mu/beta rhythms by removing the
task-irrelevant brain activities. The supervised common spatial pattern (CSP) algo-
rithm and its extensions such as multi-class CSP and filter bank CSP algorithms have
been proved highly efficient for extracting discriminative information cross-motor
imagery states [51–53]. Unsupervised methods such as independent component
analysis (ICA) have also been applied to extract independent sensorimotor compo-
nents toward a calibration-free system [54]. Second, EEG source imaging methods
have been proposed to extract spatial features in the source space, which show
improved spatial resolution and specificity than scalp EEG, for classifying motor
imagery states in BCIs [55]. Source imaging approaches with a high spatial resolu-
tion were able to enhance BCI performance of decoding complex right-hand motor
imagery tasks (i.e., flexion, extension, supination, and pronation) [56]. Third, inde-
pendent from spectral features, amplitude and phase coupling measures such as
non-linear regressive coefficients and phase locking value have been applied to
extract brain synchrony features that reflect connectivity between multiple cortical
regions involved in motor imagery [57]. By enhancing the extraction of spatial
features from the sensorimotor areas, these signal processing approaches signifi-
cantly facilitate the classification of motor imagery states. After feature extraction,
the widely used classifiers such as LDA and SVM convert the features to continuous
or discrete control commands to operate output devices [14].

2.2.2.3 User Training

User training plays a key role in the sensorimotor BCI. In the BCI system that
requires subject to learn self-control of mu/beta rhythm amplitude, subjects usually
take multiple training sessions for weeks and months to realize multi-dimensional
control of cursor movement [45]. For the system that discriminates motor imagery
states, the selection of the kind of motor imagery affects the BCI performance. For
example, kinesthetic motor imagery was found to be superior to visual-motor
imagery with respect to distinct spatial patterns [58]. Therefore, user training should
emphasize kinesthetic experiences instead of visual representations of movements to
obtain reliable motor-imagery-based BCI control. In the BCI systems that employ
50 Y. Wang et al.

machine learning approaches, the brain and the machine can adapt to each other
simultaneously, leading to a co-adaptive learning paradigm, which can significantly
reduce the user training time [59, 60].
In the sensorimotor BCI, “BCI Illiteracy” is a big challenge for user training and
system applications [61]. An estimation of 10% of people was found not able to use
the motor imagery BCI after a short training session around 10 min [47]. To avoid
costly training of participants who might not obtain BCI control, a neurophysiolog-
ical predictor of BCI performance based on resting state EEG was proposed and
achieved a high correlation between the predictor and the BCI performance after
online training [62]. Currently, a general concern of researchers is how a naïve user
can learn to operate the sensorimotor BCI. There are few BCI studies focusing on the
training of naïve users. By adopting advanced signal processing and machine
learning algorithms, a motor imagery BCI system achieved accurate performance
in naïve subjects even in the first session [63]. In addition, the co-adaptive learning
approach was demonstrated helpful for subjects who suffer from the BCI illiteracy
problem to successfully gain control of the system [59].

2.2.3 Hybrid BCIs

2.2.3.1 Hybrid Modalities

A hybrid BCI combines a BCI with other physiological or technical signals

[20, 64]. The goal of hybrid BCIs is to integrate multiple input signals to improve
the performance of normal BCIs, and enlarge the population of end-users. In a
hybrid BCI, at least one brain signal should be employed to provide intentional
input to the BCI for real-time communication or control. According to the type of the
second signal modality, hybrid BCIs can be categorized into pure hybrid BCIs and
mixed hybrid BCIs [65]. A pure hybrid BCI typically combines two EEG patterns
together toward enhanced BCI performance. The widely used hybrid EEG patterns
are the combination of ERPs and sensorimotor rhythms (e.g., SSVEP-motor imag-
ery, P300-motor imagery), and the combination of two ERP signals (e.g., P300-
SSVEP, N2pc-SSVEP). In addition to hybrid EEG signals, other types of brain
signals can also be employed to build a pure hybrid BCI. For example, a hybrid near-
infrared spectroscopy (NIRS)-EEG BCI was developed and significantly improved
the classification accuracy of motor imagery [66]. A mixed hybrid BCI combines
EEG with other non-neuronal control signals toward more robust and reliable
control. The control signals can be physiological signals (e.g., EMG, EOG, or
heart rate) or signals from other existing input devices like an eye tracking system.
These control signals typically are generated by residual muscle functions. For
example, EMG signals, which can be modulated by residual muscle activities of
patients, have been used to implement hybrid BCIs with various EEG signals and
achieved significantly improved accuracy and ITR [20, 65].
2 EEG-Based Brain-Computer Interfaces 51

The hybrid BCI processes the multiple input signals simultaneously or sequen-
tially. Accordingly, hybrid BCIs can be divided into these two types (i.e., simulta-
neous and sequential hybrid BCIs) [64]. Simultaneous hybrid BCIs fuse information
from different input signals, which carry the same desired goal for intentional
control, to improve the accuracy. For example, in visual ERP-based BCIs, SSVEP
and P300 signals have been widely used to detect gaze direction, therefore, a hybrid
SSVEP-P300 BCI can significantly enhance the performance of gaze-dependent
target identification by combining the two independent EEG signals [67]. Data
fusion plays an important role in a simultaneous hybrid BCI [68]. Information fusion
of multiple input signals is required in the procedures of feature extraction, feature
combination, and classification decision. Sequential hybrid BCIs operate two sys-
tems sequentially, typically with one system as a switch or selector and another
system for regular BCI control. The input signal for a switch or selector generally
requires high accuracy and reliability, but a small number of discriminable condi-
tions. For example, a brain switch based on motor imagery has been widely used in
sequential hybrid BCIs [64]. In a sequential hybrid BCI, multiple control signals are
processed separately toward different outputs, and therefore data fusion is not
required.

2.2.3.2 Hybrid EEG Signals

In hybrid BCIs, the simultaneous pure hybrid BCIs that only employ multiple EEG
patterns are of great interest to BCI researchers due to the great challenges in system
design and data fusion. There are two different ways to form hybrid EEG signals.
The first approach is to design hybrid BCI paradigms with multiple mental tasks,
which can be used to elicit different EEG signals for single or multiple dimensional
controls. An example is hybrid BCIs based on SSVEP and P300 signals, which have
been widely used in visual BCIs [19]. Frequency-coded SSVEP stimuli can be
superimposed onto P300 stimuli coded with a standard row/column paradigm. The
attended target stimuli can simultaneously evoke P300 and SSVEP, which can
facilitate the discrimination between target and non-target stimuli [67]. Another
diagram used the SSVEP blocking approach to evoke P300 and block SSVEP at
the same time, which also significantly improved target discrimination [69]. The
combination of motor imagery task and visual attention tasks was proposed to
develop a hybrid BCI, which can be applicable for more users [70]. The concurrent
motor imagery (i.e., imagination of left or right hand movements) and visual
attention (gazing left or right flickering stimuli) tasks simultaneously produced
independent ERD and SSVEP features for enhanced classification performance. In
another study, motor imagery and P300 tasks were performed simultaneously and
independently to control 2D cursor movement [71]. Subjects obtained independent
mu/beta rhythms and P300 for horizontal and vertical movement controls, respec-
tively. The second approach to form hybrid EEG signals only employs a single type
of mental tasks, which can elicit multiple EEG signals. The combination of ERPs
and oscillatory brain rhythms can improve the discriminability of EEG patterns. For
52 Y. Wang et al.

example, the combination of ERD/ERS and the lateralized readiness potential

facilities the classification of motor imagery tasks [72]. In the gaze-independent
SSVEP BCI based on visual spatial attention, the combination of SSVEP and the
parieto-occipital alpha rhythm resulted in improved classification accuracy [73]. In
addition, with appropriate design, multiple ERP signals can be modulated by a
common task. For example, hybrid N2pc and SSVEP features were combined to
realize fast detection of covert visuospatial attention in a recent study [74].

2.3 Challenges in Developing BCIs for Practical

Applications

Although the feasibility of various BCI paradigms has been well demonstrated in the
past decade, translating BCI systems from well-controlled laboratories to compli-
cated living environments still poses great challenges [19, 24]. First of all, the BCI
performance in terms of ITR is a key factor in building practical applications for
communication and control. Due to limits of the EEG technology, the bottleneck
problem in communication speed remains unsolved. Especially in the noisy living
conditions, the low ITR is still the key obstacle to BCI applications. Second, the
usability of the system plays another key role in practical applications for daily
routine use. The two major issues regarding user experience include system calibra-
tion and asynchronous system design. The third factor is the practicality of the
hardware and software platform. Specifically, practical BCI applications require a
convenient mobile EEG system with electrodes that are easy to apply. Besides,
analysis of mobile EEG signals, which are more easily contaminated by various
noises, requires further development on algorithms. These technical issues need to
be addressed to develop BCIs that can satisfy requirements for widespread usage in
daily life.

2.3.1 Improving Information Transfer Rate

2.3.1.1 ITR in EEG-Based BCIs

Various metrics (e.g., classification accuracy, ITR, sensitivity and specificity, effi-
ciency and utility, and the like) have been proposed for evaluating the performance
of BCIs [75]. Among them, the ITR has been the most widely used metric in BCIs
for the purpose of communication and control. Currently, the most popular method
for calculating ITR based on Shannon channel theory is defined as follows [2]:
    
1P 60
ITR ¼ log2 M þ Plog2 P þ ð1  PÞlog2  ð2:1Þ
M1 T
2 EEG-Based Brain-Computer Interfaces 53

where N is the number of possible choices, P is the classification accuracy, and

T (in seconds/selection) is the average time for each selection. Due to its capability to
quantify communication speed, ITR in bits/minute (bpm) has been widely used to
evaluate the performance of online BCI systems.
According to temporal control paradigm, BCI systems can be divided into
synchronous and asynchronous BCIs [15]. Most current BCIs use synchronous
control protocols where the system controls are periodically available and the timing
of the operation is determined by the system. In general, ITR calculation in Eq. (2.1)
can be directly used for synchronous BCIs. In contrast, the system controls are
continuously available in the asynchronous control protocols. By discriminating
non-control state and intentional control state, the asynchronous BCIs allow users
to make self-paced decisions on when to start or stop the operation. Since the timing
of system operation may vary dramatically depending on the user’s control state,
asynchronous BCI systems usually do not report ITR [76].

2.3.1.2 BCIs with High ITRs

In Wolpaw et al.’s review in 2002 [2], the BCIs showed maximum ITRs up to
10–25 bpm [2]. Over the past 15 years, advances in BCI research have dramatically
increased the ITRs for BCIs in all categories. First, for the sensorimotor BCIs, the
ITRs have been improved by applying efficient signal processing and machine
learning algorithms. However, due to the small number of possible choices and
the endogenous control property, ITRs reported for the motor imagery BCIs were
generally lower than 15 bpm [77]. Second, for the ERP-based BCIs, the ITRs have
achieved a number of breakthroughs. With recent advances in information coding
and EEG decoding, the current ERP-based BCI systems have achieved multifold
increases of ITR compared with the early systems. Notably in recent studies on
visual BCI spellers, the ITRs above 100 bpm have been reported in BCIs using P300
[78], cVEP [36], and SSVEP [34, 35, 79–81]. Townsend et al. used temporal
constraints to develop an asynchronous presentation paradigm for the P300 BCI
and reported an ITR of 120 bpm with a subject using a 72-target matrix [78]. Bin
et al. developed a 32-target BCI using cVEP and achieved an average ITR of
108 bpm [36]. In the past several years, the BCIs based on SSVEPs have demon-
strated a series of systems with high ITRs. Chen et al. developed a 45-target BCI,
which was calibration free, with frequency-based stimulus coding and EEG
decoding methods and obtained an ITR of 105 bpm [79]. The ITR was further
improved to 151 bpm by adopting a filter bank analysis algorithm [80]. More
recently, the combination of advanced SSVEP detection and multi-target coding
methods has been proved highly efficient for the SSVEP-based BCIs. Nakanishi
et al. proposed to incorporate individual calibration data in SSVEP detection and
reported an ITR of 166 bpm using a mixed frequency and phase coding method
[81]. Chen et al. reached a spelling speed at 1 character per second by proposing a
joint frequency and phase modulation method and pushed the ITR up to 270 bpm
[34]. In a recent study, by adopting a new spatial filtering algorithm, Nakanishi et al.
achieved an ITR of 325 bpm [35], which was the maximum ITR reported in current
54 Y. Wang et al.

BCIs. Note that, since the spelling speed of 1 character per second is close to the
speed limit of human gaze control, there might be little room for improving the
practical ITR of BCI spellers [34]. Third, for the simultaneous hybrid BCIs, the ITRs
have been significantly increased compared with the single modalities. The ITRs of
hybrid BCIs highly depend on the signals employed in system design. For example,
the hybrid P300-SSVEP BCIs reported ITRs around 50–60 bpm [67, 82], while the
hybrid SSVEP-EMG BCI reached an ITR above 90 bpm [65]. Compared with the
ERP-based BCIs, there is still large room for improvement of ITRs in the
hybrid BCIs.

2.3.1.3 Approaches for Improving ITR

The performance bottleneck of current BCIs is mostly attributed to the poor SNR of
EEG signals. Therefore, approaches for improving ITR generally aim to improve the
SNR of EEG by adopting advanced signal processing and machine learning algo-
rithms [11, 13]. However, according to Eq. (2.1), the methods for improving ITR can
be considered with respect to number of choices (N ), classification accuracy (P), and
target selection time (T ) separately. Note that, these parameters cannot be considered
independently because they always interact with each other in BCI systems. For
example, there is always a tradeoff between classification accuracy and target
selection time toward an optimal ITR.
Classification accuracy is crucial in ITR estimation. The methods to improve
classification accuracy can be summarized into two directions [19]. The first direc-
tion is to enhance the SNR of task-related EEG signals. In this direction, advanced
signal processing algorithms have been applied to artifact removal, spatial filtering,
and feature extraction [11]. For example, in the sensorimotor BCIs, the CSP-based
spatial filtering algorithms have been widely used to extract the task-related power
change of mu/beta rhythms during motor imagery [52, 53]. In the SSVEP BCIs, the
canonical correlation analysis-based spatial filtering algorithms significantly
improve the SNR of SSVEPs [80, 81, 83]. Besides, the amplitude and SNR of
task-related EEG signal can be enhanced by user training. In the sensorimotor BCIs,
users learn to control the amplitude of sensorimotor rhythms through online training
with feedbacks [45]. Alpha neurofeedback training has been shown efficient for
improving the SNR of SSVEPs [84]. The second direction for improving classifica-
tion accuracy is to maximize the separability of multiple classes. To this end,
machine learning-based classification techniques have been widely used in feature
selection, feature combination, and classification [13, 14]. For example, linear
classifiers like LDA have shown good performance in the sensorimotor and
ERP-based BCIs [38]. In addition, separability of multiple classes can be improved
by encoding more informative features in EEG signals. For example, the hybrid
frequency and phase coding methods have been applied in the SSVEP-based BCIs
[34, 81]. The hybrid EEG signals used in the simultaneous hybrid BCIs can
significantly improve the separability of different classes by combining multiple
EEG signals.
2 EEG-Based Brain-Computer Interfaces 55

The number of choices also plays an important role in ITR estimation. In the
sensorimotor BCIs, the number of choices are generally very limited due to the low
spatial resolution of EEG patterns. The discrete classification of motor imagery tasks
typically includes imagination of hands, feet, and tongue movements [48]. The
number of choices can be improved by enabling the continuous control of 2D and
3D movements [45, 46]. The ERP-based BCIs show good capability in
implementing a large number of choices. For example, a P300 BCI using the method
of flashing quasi-random groups of characters realized a 712 matrix speller
[85]. The BCI system using cVEP could discriminate 64 classes [9]. In these
systems, the MA methods from telecommunications facilitate the implementation
of a large number of classes [19].
The target selection time requires to be optimized toward high ITRs, especially in
the ERP-based BCIs. Due to the non-stationarity of EEG signals, the number of trial
repetitions required for target identification should be able to vary over time. In the
P300 BCI, the dynamic stopping method has been proposed to reduce the target
selection time by adaptively adjusting the number of trial repetitions [86]. In addi-
tion, the target selection time can be largely reduced by employing more efficient
target coding methods. For example, in the VEP-based BCIs, the FDMA and CDMA
methods require a much shorter time duration to code multiple targets than the
TDMA method.

2.3.2 Reducing System Calibration Time

The non-stationarity and individual variance of EEG signals pose great challenges to
the implementation of BCIs using training-based machine learning algorithms
[19]. A calibration session is typically required to collect training data for building
a classification model before the operation of the BCI system. In practice, the
repetitive calibration procedures could be boring and time-consuming. To facilitate
system calibration, different methods have been developed to reduce the calibration
time or even realize zero-training paradigms. To reduce the calibration time, the
machine learning algorithms need to have good generalization ability and robustness
even when the training dataset is small. In recent years, the zero-training methods
have obtained increasing attention. These methods aim to transfer information across
multiple sessions, subjects, or tasks to obtain generalizable models that can be
directly used for operating BCIs. For example, for each individual user, the cross-
session transfer of spatial filters and EEG templates was developed for detecting
SSVEPs without new calibration [87]. Besides, the cross-subject transfer of models
has been demonstrated feasible for SSVEP [88] and P300 [89] detection without
individual calibration data. In addition, the cross-task transfer of spatial filters
derived from resting EEG data was proposed to implement zero-training classifica-
tion of EEG in motor imagery tasks [90]. Furthermore, to better address the
non-stationarity issues, adaptive classification methods have been developed to
automatically update the classifiers during online BCI operations [88, 91]. The
56 Y. Wang et al.

combination of information transfer and adaptive classification methods provides a

practical solution to facilitate the calibration procedure of current BCIs.

2.3.3 Asynchronous System

Asynchronous BCIs, in which users make self-paced operations, are more flexible
and natural for routine use than synchronous BCIs [15]. In asynchronous BCIs,
continuous detection of idle state is required for discriminating the intentional
control state and the non-control state. The BCI system only sends control command
when an intentional control state is detected. Accurate detection of the idle states is
crucial for practical applications; however, it remains a challenging task due to the
large variability of idle states, which might involve different mental activities. One
way to address the control state detection issue is to design a switch, which can turn
on/off the BCI control mode. This method has been applied in SSVEP-based BCIs.
For example, the idle state and the control state were switched by selecting an on/off
stimulus, which turned on/off the visual stimuli of the SSVEP BCI [92]. To imple-
ment a user-initiated asynchronous BCI system, the discrimination of EEG signals
between the idle state and the intentional control state is important and of great
interest. For example, in a sensorimotor BCI-based brain switch, EEGs
corresponding to idle and imaginary movement states were classified sample-by-
sample and the averaged results over time were used to control the switch [93]. In the
ERP-based BCIs, the method for detecting idle states is generally based on the
computational modeling of EEG signals under the idle and control states. In an
SSVEP BCI, spontaneous EEG signals under different types of idle states
(on-screen, off-screen, eye-closed) and SSVEPs were classified by SVM classifiers
[94]. In an N200 BCI, a computational model of the mVEP response patterns, which
integrated the spatial profile of the speller matrix, was used to detect the non-control
state effectively [95]. In a P300 BCI, computational models for target P300,
non-target P300, and non-control EEG signals were proposed for detecting control
and non-control states based on likelihood [96].

2.3.4 Developing Mobile BCI System

Despite the significant performance improvement of EEG-based BCIs, real-life BCI

applications have still been hindered by the lack of portability and cumbersome
system settings such as skin preparation and gel application. Developing a mobile
BCI platform enables the movement of BCI systems from well-controlled laboratory
settings to real-world environments. To design a mobile BCI system, the following
three major challenges need to be addressed.
First, a mobile BCI system requires advanced sensor technologies to facilitate the
skin preparation and gel application, which may take around an hour for
2 EEG-Based Brain-Computer Interfaces 57

conventional high-density EEG systems. Recently, researchers have developed

various types of EEG electrodes. For example, dry- and non-contact electrodes
showed good performance in an online BCI experiment based on SSVEPs recorded
from the occipital area without conductive gels [97]. In another study, a soft,
foldable, and ultra-thin electrode constructing fractal mesh geometry was introduced
for long-term recording of high-fidelity EEG signals for a persistent BCI
[98]. Besides, placing electrodes over non-hair-bearing areas is another way to
avoid the interference from hairs. For instance, several studies demonstrated that
behind/around-the-ear electrodes can be an alternative solution to measure EEG for
BCIs [99, 100]. In addition, the platform for ear-EEG, which can record EEG signals
from electrodes placed in-the-ear, has been developed for brain monitoring
[101]. Second, a portable platform using mobile hardware for stimulus presentation,
signal acquisition, and data processing is required to expand the capability of BCIs.
Recent rapid advances in mobile devices like smartphones made it possible to
implement a truly portable BCI. For example, a cellphone-based phone-dialing
BCI system was demonstrated using SSVEPs [102]. A head-mounted virtual reality
(VR) platform also showed great potentials for implementing a mobile BCI system.
Nakanishi et al. proposed a portable BCI, which integrated wireless and wearable
EEG, dry EEG and EOG sensors, and a head-mounted VR display, for assessing
visual field loss in glaucoma patients based on multi-focal SSVEPs [103]. Third,
artifact removal techniques are necessary for a robust mobile BCI. In general,
subjects are instructed to remain stationary and focus only on their tasks in labora-
tory experiments, resulting in clean EEG recordings. However, in real-world situa-
tions, motion artifacts and ambient noises can be a severe problem since users’
behaviors are not restricted. To simulate realistic motion artifacts, dual-task exper-
imental design, in which subjects needed to perform a primary motor task and a
secondary cognitive task at the same time, has been proposed. For example,
Gramann et al. recorded EEG from subjects standing or walking on a treadmill
while performing a visual oddball response task [104]. Although the scalp record-
ings were contaminated with motion artifacts, source ERP activities decomposed by
ICA did not differ across conditions [104]. Lin et al. performed an online SSVEP
BCI using a wearable EEG system in human subjects under walking conditions
[105]. These results suggest that subjects can use ERP-based BCIs even while
moving by combining mobile EEG with efficient de-noising algorithms.

2.4 Applications of BCIs

Facilitated by the rapid development of modern information technology, BCIs are

moving beyond laboratories into real-world applications. The majority of the avail-
able BCI applications has been focusing on decoding and outputting users’ intention,
in which the users actively perform certain mental tasks for the purpose of control-
ling external devices. The devices can be assistive or rehabilitative, used in both
medical and non-medical scenarios. BCIs can also be applied to monitor users’
58 Y. Wang et al.

mental states in real time without their active participation. Applications in this
direction mainly focus on healthy people in non-medical scenarios. Below we briefly
review the state-of-the-art BCI applications and organize them by their design
purposes (i.e., medical or non-medical).

2.4.1 BCIs for Medical Use

ERP-based BCIs are probably the most popular BCI systems serving as an assistive
tool for severely motor disabled people, for example, patients with amyotrophic
lateral sclerosis (ALS) [106, 107]. Compared to sensorimotor BCIs, the major
advantage for ERP-based BCIs is its relative ease of acquiring reliable signals with
little training required [17]. Using the classical P300 speller, ALS patients have
demonstrated the capability of operating necessary software such as word
processing, painting, e-mailing, and controlling home appliances, in their home
environment for a considerably long time (>2.5 years in [108] and 14 weeks in
[109]). Nevertheless, challenges still remain for patients with complete locked-in
syndrome, who are speculated to lack the contingency between a voluntary intention
and its consequence [110].
Sensorimotor BCIs, by contrast, have been suggested as an effective approach for
rehabilitative purposes (see [10, 45] for attempts to implement assistive BCIs). The
most widely studied population is the stroke patients, in which rehabilitative BCIs
are expected to help them regain the lost motor functions. Sensorimotor BCIs detect
and translate patients’ movement intentions into actual limb movements, with the
help of a prosthesis or a direct functional electrical stimulation. Such rehabilitation
training is believed to induce reorganization of neural circuits, thus facilitating motor
function recovery [111].

2.4.2 BCIs for Non-medical Use

Both ERP-based and sensorimotor BCIs can be used in non-medical scenarios, as a

novel human-computer interaction (HCI) approach, especially in gaming industry
[112, 113]. BCIs are considered as an additional input modality (along with key-
board, speech, gestures, and so on) to control a game or a general HCI application
(e.g., in VR environments), providing a unique and possibly useful way for
interaction.
An emerging but more important BCI application direction for healthy users is
real-time monitoring of critical mental states, such as attention, stress, cognitive
load, emotion, and the like. Most of these BCIs, recently termed as ‘passive BCIs’
2 EEG-Based Brain-Computer Interfaces 59

[22], usually take the form of hybrid BCIs, in which other physiological or environ-
mental signals are also included when necessary, with the aim of having a more
comprehensive overview of the users’ states. Passive BCIs provide an objective
evaluation in the fields of driving safety [114, 115], education [116], user experi-
ence, or neuro-marketing [117]. Although still at an early stage, BCI development in
this direction is believed to benefit a much larger population, beyond the scope of
clinical population.

2.5 Summary

The past 20 years have witnessed unprecedented progress in the BCI technology. In
addition to the original purpose to help the patients with motor disabilities to
communicate with their environments, current BCI applications have been extended
to various fields including brain state monitoring, neuro-rehabilitation, and human
performance improvement in the last few years. More generally, BCI can be defined
as a real-time platform that uses online brain signal analysis to influence human
interactions with their environments. In the field of communication and control,
beyond the initial definition of interface, the current BCI technology places more
emphasis on the interaction between human brain and the output devices. The
practical BCI applications will allow researchers to study the mutual interference
between brain and devices in long-term interaction.
Despite significant progress in performance improvement, low communication
rates remain key obstacles to BCI-based communication and control. The improve-
ment of communication speed will continue to be a main challenge in BCI research.
The following efforts can be made in this direction. First, by increasing our under-
standing of how sensation, perception, and cognition are encoded in brain signals,
new BCI paradigms with wider communication bandwidths can be developed.
Second, by considering BCI as a communication system, advanced technologies
from telecommunications can be employed to improve information modulation and
demodulation in BCIs. Third, more efficient signal processing and machine learning
algorithms can facilitate the decoding of EEG. Fourth, brain stimulation methods
such as electrical and magnetic stimulation can be employed to enhance the brain
signals. Furthermore, the future performance breakthroughs of BCI require the
collaboration between neuroscience, artificial intelligence (AI), and engineering. In
the near future, the combination of BCI with AI can lead to new hybrid intelligence
systems that can effectively fuse human and machine intelligence.

Acknowledgments This work is supported by the National Natural Science Foundation of China
(61671424, 61335010, and 61634006), the Key project of Chinese Academy of Science (KJZD-
EW-L11-01), Beijing S&T planning task (Z161100002616019), and the Recruitment Program of
Young Professionals.
60 Y. Wang et al.

References

1. Vidal JJ (1973) Toward direct brain-computer communication. Annu Rev Riophys Bioeng
2:157–180
2. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain
computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791
3. Birbaumer N (2006) Brain-computer-interface research: coming of age. Clin Neurophysiol
117(3):479–483
4. Lebedev MA, Nicolelis MAL (2017) Brain-machine interfaces: from basic science to
neuroprostheses and neurorehabilitation. Physiol Rev 97(2):767–837
5. Vidal JJ (1977) Real-time detection of brain events in EEG. Proc IEEE 65(5):633–664
6. Farwell LA, Donchin E (1988) Talking off the top of your head – toward a mental prosthesis
utilizing event-related brain potentials. Electroencephalogr Clin Neurophysiol 70(6):510–523
7. Wolpaw JR, McFarland DJ, Neat GW, Forneris CA (1991) An EEG-based brain-computer
interface for cursor control. Electroencephalogr Clin Neurophysiol 78(3):252–259
8. Pfurtscheller G, Flotzinger D, Kalcher J (1993) Brain-computer interface – a new communi-
cation device for handicapped persons. J Microcomput Appl 16(3):293–299
9. Sutter EE (1992) The brain response interface: communication through visually-induced
electrical brain responses. J Microcomput Appl 15(1):31–45
10. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kubler A,
Perelmouter J, Taub E, Flor H (1999) A spelling device for the paralyzed. Nature 398:297–298
11. Bashashati A, Fatourechi M, Ward RK, Birch GE (2007) A survey of signal processing
algorithms in brain-computer interfaces based on electrical brain signals. J Neural Eng 4(2):
R32–R57
12. Makeig S, Kothe C, Mullen T, Bigdely-Shamlo N, Zhang Z, Kreutz-Delgado K (2012)
Evolving signal processing for brain–computer interfaces. Proc IEEE 100:1567–1584
13. Müller KR, Krauledat M, Dornhege G, Curio G, Blankertz B (2004) Machine learning
techniques for brain-computer interfaces. Biomed Tech 49(1):11–22
14. Lotte F, Congedo M, Lecuyer A, Lamarche F, Arnaldi B (2007) A review of classification
algorithms for EEG-based brain-computer interfaces. J Neural Eng 4(2):R1–R13
15. Mason SG, Bashashati A, Fatourechi M, Navarro KF, Birch GE (2007) A comprehensive
survey of brain interface technology designs. Ann Biomed Eng 35(2):137–169
16. Lance BJ, Kerick SE, Ries AJ, Oie KS, McDowell K (2012) Brain computer interface
technologies in the coming decades. Proc IEEE 100:1585–1599
17. Mak JN, Wolpaw JR (2009) Clinical applications of brain-computer interfaces: current state
and future prospects. IEEE Rev Biomed Eng 2:187–199
18. Van Erp J, Lotte F, Tangermann M (2012) Brain-computer interfaces: beyond medical
applications. Computer 45(4):26–34
19. Gao S, Wang Y, Gao X, Hong B (2014) Visual and auditory brain-computer interfaces. IEEE
Trans Biomed Eng 61(5):1436–1447
20. Müller-Putz G, Leeb R, Tangermann M, Höhne J, Kübler A, Cincotti F, Mattia D, Rupp R,
Müller KR, Millan JR (2015) Towards noninvasive hybrid brain–computer interfaces: frame-
work, practice, clinical application, and beyond. Proc IEEE 103(6):926–943
21. Chaudhary U, Birbaumer N, Ramos-Murguialday A (2016) Brain–computer interfaces for
communication and rehabilitation. Nat Rev Neurol 12(9):513–525
22. Zander TO, Kothe C (2011) Towards passive brain–computer interfaces: applying brain–
computer interface technology to human–machine systems in general. J Neural Eng 8
(2):025005
23. Millán JR, Rupp R, Müller-Putz GR, Murray-Smith R, Giugliemma C, Tangermann M,
Vidaurre C, Cincotti F, Kübler A, Leeb R, Neuper C, Müller KR, Mattia D (2010) Combining
brain–computer interfaces and assistive technologies: state-of-the-art and challenges. Front
Neurosci 4:161
2 EEG-Based Brain-Computer Interfaces 61

24. Wang Y, Gao X, Hong B, Gao S (2010) Practical designs of brain–computer interfaces based
on the modulation of EEG rhythms. In: Graimann B, Allison B, Pfurtscheller G (eds) Brain–
computer interfaces. Springer, Heidelberg, pp 137–154
25. Yuan H, He B (2014) Brain–computer interfaces using sensorimotor rhythms: current state and
future perspectives. IEEE Trans Biomed Eng 61(5):1425–1435
26. Luck SJ (2005) An introduction to the event-related potential technique. MIT Press,
Cambridge
27. Regan D (1989) Human brain electrophysiology: evoked potentials and evoked magnetic
fields in science and medicine. Elsevier, New York
28. Patel SH, Azzam PN (2005) Characterization of N200 and P300: selected studies of the event-
related potential. Int J Med Sci 2(4):47–154
29. Rappaport TS (2001) Wireless communication, principle and practice, 2nd edn. Prentice-Hall,
Englewood Cliffs
30. Fazel-Rezai R, Allison BZ, Guger C, Sellers EW, Kleih SC, Kübler A (2012) P300 brain
computer interface: current challenges and emerging trends. Front Neuroeng 5:14
31. Guo F, Hong B, Gao X, Gao S (2008) A brain–computer interface using motion-onset visual
evoked potential. J Neural Eng 5(4):477–485
32. Lee PL, Hsieh JC, Wu CH, Shyu KK, Chen SS, Yeh TC, Wu YT (2006) The brain computer
interface using flash visual evoked potential and independent component analysis. Ann
Biomed Eng 34(10):1641–1654
33. Vialatte FB, Maurice M, Dauwels J, Cichocki A (2010) Steady-state visually evoked poten-
tials: focus on essential paradigms and future perspectives. Prog Neurobiol 90(4):418–438
34. Chen X, Wang Y, Nakanishi M, Gao X, Jung TP, Gao S (2015) High-speed spelling with a
noninvasive brain-computer interface. Proc Natl Acad Sci U S A 112(44):E6058–E6067
35. Nakanishi M, Wang Y, Chen X, Wang YW, Gao X, Jung TP (2017) Enhancing detection of
SSVEPs for a high-speed brain speller using task-related component analysis. IEEE Trans
Biomed Eng. (in press)
36. Bin G, Gao X, Wang Y, Li Y, Hong B, Gao S (2011) A high-speed BCI based on code
modulation VEP. J Neural Eng 8(2):025015
37. Maye A, Zhang D, Engel AK (2017) Utilizing retinotopic mapping for a multi-target SSVEP
BCI with a single flicker frequency. IEEE Trans Neural Syst Rehabil Eng. (in press)
38. Blankertz B, Lemm S, Treder M, Haufe S, Müller KR (2011) Single trial analysis and
classification of ERP components—a tutorial. NeuroImage 56(2):814–825
39. Parra LC, Spence CD, Gerson AD, Sajda P (2005) Recipes for the linear analysis of EEG.
NeuroImage 28(2):326–341
40. Wang Y, Gao X, Hong B, Jia C, Gao S (2008) Brain–computer interfaces based on visual
evoked potentials-feasibility of practical system designs. IEEE Eng Med Biol Mag 27
(5):64–71
41. Riccio A, Mattia D, Simione L, Olivetti M, Cincotti F (2012) Eye-gaze independent
EEG-based brain–computer interfaces for communication. J Neural Eng 9(4):045001
42. Kelly SP, Lalor EC, Finucane C, McDarby G, Reilly RB (2005) Visual spatial attention
control in an independent brain–computer interface. IEEE Trans Biomed Eng 52
(9):1588–1596
43. Zhang D, Maye A, Gao X, Hong B, Engel AK, Gao S (2010) An independent brain–computer
interface using covert non-spatial visual selective attention. J Neural Eng 7(1):016010
44. Pfurtscheller G, Lopes da Silva FH (1999) Event-related EEG/MEG synchronization and
desynchronization: basic principles. Clin Neurophysiol 110(11):1842–1857
45. Wolpaw JR, McFarland DJ (2004) Control of a two-dimensional movement signal by a
noninvasive brain-computer interface in humans. Proc Natl Acad Sci U S A 101
(51):17849–17854
46. McFarland DJ, Sarnacki WA, Wolpaw JR (2010) Electroencephalographic (EEG) control of
three-dimensional movement. J Neural Eng 7(3):036007
62 Y. Wang et al.

47. Pfurtscheller G, Neuper C (2001) Motor imagery and direct brain-computer communication.
Proc IEEE 89(7):1123–1134
48. Pfurtscheller G, Brunner C, Schlogl A, Lopes da Silva FH (2006) Mu rhythm (de)synchroni-
zation and EEG single-trial classification of different motor imagery tasks. NeuroImage 31
(1):153–159
49. Wang T, Deng J, He B (2004) Classifying EEG-based motor imagery tasks by means of time-
frequency synthesized spatial patterns. Clin Neurophysiol 115(12):2744–2753
50. Li J, Wang Y, Zhang L, Cichocki A, Jung TP (2016) Decoding EEG in cognitive tasks with
time-frequency and connectivity masks. IEEE Trans Cogn Dev Syst 8(4):298–308
51. Ramoser H, Müller-Gerking J, Pfurtscheller G (2000) Optimal spatial filtering of single trial
EEG during imagined hand movement. IEEE Trans Neural Syst Rehabil Eng 8(4):441–446
52. Blankertz B, Tomioka R, Lemm S, Kawanabe M, Müller KR (2008) Optimizing spatial filters
for robust EEG single-trial analysis. IEEE Signal Process Mag 25(1):41–56
53. Ang KK, Chin ZY, Wang C, Guan C, Zhang H (2012) Filter bank common spatial pattern
algorithm on BCI competition IV datasets 2a and 2b. Front Neurosci 6:39
54. Wang Y, Jung TP (2013) Improving brain-computer interfaces using independent component
analysis. In: Allison B, Dunne S, Leeb R, Millan JR, Nijholt A (eds) Towards practical brain-
computer interfaces: bridging the gap from research to real-world applications. Springer,
Heidelberg, pp 67–83
55. Qin L, Ding L, He B (2005) Motor imagery classification by means of source analysis for
brain-computer interface applications. J Neural Eng 2(4):65–72
56. Edelman BJ, Baxter B, He B (2016) EEG source imaging enhances the decoding of complex
right-hand motor imagery tasks. IEEE Trans Biomed Eng 63(1):4–14
57. Wei Q, Wang Y, Gao X, Gao S (2007) Amplitude and phase coupling measures for feature
extraction in an EEG-based brain-computer interface. J Neural Eng 4(2):120–129
58. Neuper C, Scherer R, Reiner M, Pfurtscheller G (2005) Imagery of motor actions: differential
effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Brain Res Cogn
Brain Res 25(3):668–677
59. Vidaurre C, Sannelli C, Müller KR, Blankertz B (2011) Machine-learning-based coadaptive
calibration for brain-computer interfaces. Neural Comput 23(3):791–816
60. Acqualagna L, Botrel L, Vidaurre C, Kübler A, Blankertz B (2016) Large-scale assessment of
a fully automatic co-adaptive motor imagery-based brain computer interface. PLoS ONE 11
(2):e0148886
61. Kübler A, Neumann N, Wilhelm B, Hinterberger T, Birbaumer N (2004) Predictability of
brain-computer communication. J Psychophysiol 18:121–129
62. Blankertz B, Sannelli C, Halder S, Hammer EM, Kübler A, Müller KR, Curio G, Dickhaus T
(2010) Neurophysiological predictor of SMR-based BCI performance. NeuroImage 51
(4):1303–1309
63. Blankertz B, Losch F, Krauledat M, Dornhege G, Curio G (2008) The Berlin brain-computer
interface: accurate performance from first-session in BCI-naïve subjects. IEEE Trans Biomed
Eng 55(10):2452–2462
64. Pfurtscheller G, Allison BZ, Bauernfeind G, Brunner C, Solis Escalante T, Scherer R, Zander
TO, Müller-Putz G, Neuper C, Birbaumer N (2010) The hybrid BCI. Front Neurosci 4:30
65. Lin K, Cinetto A, Wang Y, Chen X, Gao S, Gao X (2016) An online hybrid BCI system based
on SSVEP and EMG. J Neural Eng 13(2):026020
66. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller KR, Blankertz B (2012)
Enhanced performance by a hybrid NIRS–EEG brain computer interface. NeuroImage 59
(1):519–529
67. Yin E, Zhou Z, Jiang J, Chen F, Liu Y, Hu D (2013) A novel hybrid BCI speller based on the
incorporation of SSVEP into the P300 paradigm. J Neural Eng 10(2):026012
68. Li Y, Pan J, Long J, Yu T, Wang F, Yu Z, Wu W (2016) Multimodal bcis: target detection,
multidimensional control, and awareness evaluation in patients with disorder of conscious-
ness. Proc IEEE 104(2):332–352
2 EEG-Based Brain-Computer Interfaces 63

69. Xu M, Qi H, Wan B, Yin T, Liu Z, Ming D (2013) A hybrid BCI speller paradigm combining
P300 potential and the SSVEP blocking feature. J Neural Eng 10(2):026001
70. Allison BZ, Brunner C, Kaiser V, Müller-Putz GR, Neuper C, Pfurtscheller G (2010) Toward a
hybrid brain–computer interface based on imagined movement and visualattention. J Neural
Eng 7(2):26007
71. Li Y, Long J, Yu T, Yu Z, Wang C, Zhang H, Guan C (2010) An EEG-based BCI system for
2-D cursor control by combining Mu/Beta rhythm and P300 potential. IEEE Trans Biomed
Eng 57(10):2495–2505
72. Höhne J, Holz E, Staiger-Sälzer P, Müller KR, Kübler A, Tangermann M (2014) Motor
imagery for severely motor-impaired patients: evidence for brain-computer interfacing as
superior control solution. PLoS ONE 9(8):e104854
73. Kelly SP, Lalor EC, Reilly RB, Foxe JJ (2005) Visual spatial attention tracking using high-
density SSVEP data for independent brain-computer communication. IEEE Trans Neural Syst
Rehabil Eng 13(2):172–178
74. Xu M, Wang Y, Nakanishi M, Wang YT, Qi H, Jung TP, Ming D (2016) Fast detection of
covert visuospatial attention using hybrid N2pc and SSVEP features. J Neural Eng 13
(6):066003
75. Billinger M, Daly I, Kaiser V, Jin J, Allison BZ, Müller-Putz GR, Brunner R (2013) Is it
significant? Guidelines for reporting BCI performance. In: Allison B, Dunne S, Leeb R, Millan
JR, Nijholt A (eds) Towards practical brain-computer interfaces: bridging the gap from
research to real-world applications. Springer, Heidelberg, pp 333–354
76. Yuan P, Gao X, Allison B, Wang Y, Bin G, Gao S (2013) A study of the existing problems of
estimating the information transfer rate in online brain–computer interfaces. J Neural Eng 10
(2):026014
77. Müller KR, Tangermann M, Dornhege G, Krauledat M, Curio G, Blankertz B (2008) Machine
learning for real-time single-trial EEG-analysis: from brain – computer interfacing to mental
state monitoring. J Neurosci Methods 167(1):82–90
78. Townsend G, Platsko V (2016) Pushing the P300-based brain-computer interface beyond
100 bpm: extending performance guided constraints into the temporal domain. J Neural Eng
13(2):026024
79. Chen X, Chen Z, Gao S, Gao X (2014) A high-ITR SSVEP-based BCI speller. Brain-Comp
Interfaces 1(3–4):181–191
80. Chen X, Wang Y, Gao S, Jung TP, Gao X (2015) Filter bank canonical correlation analysis for
implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng 12
(4):046008
81. Nakanishi M, Wang Y, Wang YT, Mitsukura Y, Jung TP (2014) A high-speed brain speller
using steady-state visual evoked potentials. Int J Neural Syst 24(6):1450019
82. Xu M, Chen L, Zhang L, Qi H, Ma L, Tang J, Wan B, Ming D (2014) A visual parallel-BCI
speller based on the time–frequency coding strategy. J Neural Eng 11(2):026014
83. Lin Z, Zhang C, Wu W, Gao X (2007) Frequency recognition based on canonical correlation
analysis for SSVEP-based BCIs. IEEE Trans Biomed Eng 54(6):1172–1176
84. Wan F, Da Cruz JN, Nan W, Wong CM, Vai MI, Rosa A (2016) Alpha neurofeedback training
improves SSVEP-based BCI performance. J Neural Eng 13(3):036019
85. Jin J, Allison BZ, Sellers EW, Brunner C, Horki P, Wang X, Neuper C (2011) Optimized
stimulus presentation patterns for an event-related potential EEG-based brain-computer inter-
face. Med Biol Eng Comput 49(2):181–191
86. Schreuder M, Höhne J, Blankertz B, Haufe S, Dickhaus T, Tangermann M (2013) Optimizing
event-related potential based brain–computer interfaces: a systematic evaluation of dynamic
stopping methods. J Neural Eng 10(3):036025
87. Nakanishi M, Wang Y, Jung TP (2016) Session-to-session transfer in detecting SSVEPs with
individual calibration data. In: Schmorrow DD, Fidopiastis CM (eds) Foundations of aug-
mented cognition: neuroergonomics and operational neuroscience. Springer International
Publishing, Cham, pp 253–260
64 Y. Wang et al.

88. Yuan P, Chen X, Wang Y, Gao X, Gao S (2015) Enhancing performances of SSVEP-based
brain-computer interfaces via exploiting inter-subject information. J Neural Eng 12(4):046006
89. Xu M, Liu J, Chen L, Qi H, He F, Zhou P, Wan B, Ming D (2016) Incorporation of inter-
subject information to improve the accuracy of subject-specific P300 classifiers. Int J Neural
Syst 26(3):1650010
90. Wang Y, Wang YT, Jung TP (2012) Translation of EEG spatial filters from resting to motor
imagery using independent component analysis. PLoS ONE 7(5):e37665
91. Vidaurre C, Sannelli C, Müller KR, Blankertz B (2010) Machine-learning-based coadaptive
calibration for brain-computer interfaces. Neural Comput 23(3):791–816
92. Cheng M, Gao X, Gao S, Xu D (2002) Design and implementation of a brain-computer
interface with high transfer rates. IEEE Trans Biomed Eng 49(10):1181–1186
93. Mason SG, Birch GE (2000) A brain-controlled switch for asynchronous control applications.
IEEE Trans Biomed Eng 47(10):1297–1307
94. Zhang D, Huang B, Wu W, Li S (2015) An idle-state detection algorithm for SSVEP-based
brain-computer interfaces using a maximum evoked response spatial filter. Int J Neural Syst 25
(7):1550030
95. Zhang D, Song H, Xu H, Wu W, Gao S, Hong B (2012) An N200 speller integrating the spatial
profile for the detection of the non-control state. J Neural Eng 9(2):026016
96. Zhang H, Guan C, Wang C (2008) Asynchronous P300-based brain-computer interfaces: a
computational approach with statistical models. IEEE Trans Biomed Eng 55(6):1754–1763
97. Chi YW, Wang YT, Wang Y, Maier C, Jung TP, Cauwenberghs G (2012) Dry and non-contact
EEG sensors for mobile brain-computer interfaces. IEEE Trans Neural Syst Rehabil Eng 20
(2):228–235
98. Norton JJ, Lee DS, Lee JW, Lee W, Kwon O, Won P, Jung SY, Cheng H, Jeong JW, Akce A,
Umunna S, Na I, Kwon YH, Wang XQ, Liu Z, Paik U, Huang Y, Bretl T, Yeo WH, Rogers JA
(2015) Soft, curved electrode systems capable of integration on the auricle as a persistent
brain-computer interface. Proc Natl Acad Sci U S A 112(13):3920–3925
99. Wang YT, Nakanishi M, Wang Y, Wei CS, Cheng CK, Jung TP (2017) An online brain-
computer interface based on SSVEPs measured from non-hair-bearing areas. IEEE Trans
Neural Syst Rehabil Eng 25(1):11–18
100. Pacharra M, Debener S, Wascher E (2017) Concealed around-the-ear EEG captures cognitive
processing in a visual Simon task. Front Hum Neurosci 11:290
101. Looney D, Kidmose P, Park C, Ungstrup M, Rank ML, Rosenkranz K, Mandic DP (2012) The
in-the-ear recording concept: use-centered and wearable brain monitoring. IEEE Pulse 3
(6):32–42
102. Wang YT, Wang Y, Jung TP (2011) A cell-phone based brain-computer interface for
com-munication in daily life. J Neural Eng 8(2):025018
103. Nakanishi M, Wang YT, Jung TP, Zao JK, Chien YY, Diniz-Filho A, Daga FB, Lin YP,
Wang Y, Medeiros FA (2017) Detecting glaucoma with a portable brain-computer interface
for objective assessment of visual function loss. JAMA Ophthalmol 135(6):550–557
104. Gramann K, Gwin JT, Bigdely-Shamlo N, Ferris DP, Makeig S (2010) Visual evoked
re-sponses during standing and walking. Front Hum Neurosci 4:202
105. Lin YP, Wang Y, Jung TP (2014) Assessing the feasibility of online SSVEP decoding for
moving humans using a consumer EEG headset. J Neuroeng Rehabil 11:119
106. Sellers EW, Donchin E (2006) A P300-based brain-computer interface: initial tests by ALS
patients. Clin Neurophysiol 117(3):538–548
107. Nijboer F, Sellers EW, Mellinger J, Jordan MA, Matuz T, Furdea A, Kübler A (2008) A P300-
based brain–computer interface for people with amyotrophic lateral sclerosis. Clin
Neurophysiol 119(8):1909–1916
108. Sellers EW, Vaughan TM, Wolpaw JR (2010) A brain-computer interface for long-term
independent home use. Amyotroph Lateral Scler 11(5):449–455
2 EEG-Based Brain-Computer Interfaces 65

109. Holz EM, Botrel L, Kaufmann T, Kübler A (2015) Long-term independent brain-computer
interface home use improves quality of life of a patient in the locked-in state: a case study.
Arch Phys Med Rehabil 96(3):S16–S26
110. Birbaumer N, Piccione F, Silvoni S, Wildgruber M (2012) Ideomotor silence: the case of
complete paralysis and brain-computer interfaces (BCI). Psychol Res 76(2):183–191
111. Van Dokkum L, Ward T, Laffont I (2015) Brain computer interfaces for neurorehabilitation-its
current status as a rehabilitation strategy post-stroke. Ann Phys Rehabil Med 58(1):3–8
112. Bos DPO, Reuderink B, van de Laar B, Gürkök H, Mühl C, Poel M, Heylen D (2010) Brain-
computer interfacing and games. In: Tan DS, Nijholt A (eds) Brain-computer interfaces.
Springer, London, pp 149–178
113. Ahn M, Lee M, Choi J, Jun SC (2014) A review of brain-computer interface games and an
opinion survey from researchers, developers and users. Sensors 14(8):14601–14633
114. Lal SK, Craig A (2002) Driver fatigue: electroencephalography and psychological assessment.
Psychophysiology 39(3):313–321
115. Healey J, Picard RW (2005) Detecting stress during real-world driving tasks using physio-
logical sensors. IEEE Trans Intell Transp Syst 6(2):156–166
116. Poulsen AT, Kamronn S, Dmochowski J, Parra LC, Hansen LK (2017) EEG in the classroom:
synchronised neural recordings during video presentation. Sci Rep 7:43916
117. Khushaba RN, Wise C, Kodagoda S, Louviere J, Kahn BE, Townsend C (2013) Consumer
neuroscience: assessing the brain response to marketing stimuli using electroencephalogram
(EEG) and eye tracking. Expert Syst Appl 40(9):3803–3812
Chapter 3
Invasive Brain Machine Interface System

Yile Jin, Junjun Chen, Shaomin Zhang, Weidong Chen,

and Xiaoxiang Zheng

Abstract Because of high spatial-temporal resolution of neural signals obtained by

invasive recording, the invasive brain-machine interfaces (BMI) have achieved great
progress in the past two decades. With success in animal research, BMI technology
is transferring to clinical trials for helping paralyzed people to restore their lost motor
functions. This chapter gives a brief review of BMI development from animal
experiments to human clinical studies in the following aspects: (1) BMIs based on
rodent animals; (2) BMI based on non-human primates; and (3) pilot BMIs studies in
clinical trials. In the end, the chapter concludes with a summary of potential
opportunities and future challenges in BMI technology.

Keywords Brain-machine interfaces · Neural ensemble recording · Neural

decoding · Neural stimulation · Electrocorticogram

3.1 Introduction

Spinal cord and limb nerve injury, amyotrophic lateral sclerosis, and other neuro-
muscular degeneration are common neurological disorders that cause hemiplegia
and paraplegia. There are 130,000 new paralyzed patients due to spinal cord injury
(SCI) per year in the world [1]. About half of these patients are injured above the
sixth section of the cervical vertebra. Thus, the movement of the limbs will be
affected. Except for spinal cord injury, some other diseases can cause a loss of motor

Y. Jin · J. Chen · S. Zhang (*) · X. Zheng

Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, China
Department of Biomedical Engineering, Zhejiang University, Hangzhou, China
e-mail: [email protected]
W. Chen
Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, China
College of Computer Science, Zhejiang University, Hangzhou, China

© Springer Nature Singapore Pte Ltd. 2019 67

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_3
68 Y. Jin et al.

function, when patients deeply appreciate how important the motor function is. Most
of these patients are unable to take care of themselves and have a poor quality of life.
Meanwhile, the drug is not effective in their cases.
Brain-Machine interfaces (BMIs) establish a direct pathway for communication
and control between the brain and external devices, which means utilizing neural
signals to control external devices to accomplish the desired action [2–4]. This
pathway enables direct interaction between the brain and external devices indepen-
dent of spinal cord/peripheral neuromuscular system. Not only does it assist and
enhance the interaction between the human and outside world, but also repair the
sensory, motor, and other cognitive dysfunction, providing a new means of rehabil-
itation for the disabled.
BMI is mainly composed of signal acquisition, signal processing, signal
decoding, and external device control module [3, 4]. Signal acquisition module
utilizes various sensor brain signal acquisition to collect electroencephalogram
(EEG), magnetic signals, image signals, and others which can reflect neural activ-
ities in the brain. Signal processing refers to amplifying and filtering the original
signal, consequently enhancing the effective signals and inhibiting the invalid
signals like noise. Signal decoding is a process mapping the neural activities to
motion parameters, which can be further converted to commands to control external
devices. External devices aim at assisting or enhancing the motor function of
subjects. Some common ones are a computer mouse, electric wheelchair, robotic
arm, and so on.
BMIs are divided into non-invasive BMIs and invasive BMIs according to
whether the signal acquisition equipment invades the organism of subjects [2]. Inva-
sive BMIs based on cortical neural signals can directly obtain spike and field
potentials with the features of being an informative and high temporal-spatial
resolution, so as to predict the subjects’ behavior intention and achieve real-time
and precise control of external devices in multiple degrees of freedom. Since this
century, scientific journals such as Nature and Science have reported a number of
significant research achievements about invasive BMIs. The related research also
promotes people’s awareness of the nervous system and establishes a large number
of methods to process complex information, which greatly helps computer improve
comprehension to complex perceptual information and increase the efficiency of
processing mass heterogeneous information.
From the point of view of the development of BMIs studies, the current trend of
development of BMI is gradually transformed from laboratory to clinical applica-
tion. Experimental study of BMI is more and more advanced, from rodent animals
gradually shifted to non-human primates, and has carried out preliminary clinical
studies; The paradigms in studies are getting more complex, from the
two-dimensional (2-D) cursor control to the development of multi-degree of freedom
prosthesis for performing reach-to-grasp movements. The clinical study of these
invasive BMI will provide a solid foundation for its clinical applications.
3 Invasive Brain Machine Interface System 69

3.2 BMIs on Rodents

With the deepening understanding of the encoding rules of the cerebral motor cortex
in the rodent and the continuous improvement of the accuracy of the off-line
decoding algorithms, researchers try to bypass the natural neural pathway and
establish the artificial information transmission pathways, which means direct
extraction of neural signals of cerebral motor cortex and real-time control of external
device. According to the difference of the direction of information transmission, the
rodent BMI can be classified into one-way BMI and two-way BMI.

3.2.1 Neural Decoding Studies in Rodents

One-way BMI refers to the extraction of neural signal from the cerebral cortex of
animals, the translation of neural signals into motion commands according to the
mapping between them and controlling external device to complete the task. In
one-way BMI, information is transmitted from the nervous system to the external
device on the artificial pathway. The animal can only receive the feedback of
external device through its natural sensory system, such as vision, audition, touch,
etc. One-way BMI is the earliest brain-machine interface system.
According to whether the animal gets the feedback of external device in real time,
one-way BMI is classified into open-loop BMI and closed-loop BMI. The world’s
first open-loop BMI system was built by Chapin in 1999, in which the rat pressed the
lever by its neural signals [5]. As shown in Fig. 3.1, the rat was first trained to learn to
press the lever with its forelimb (Fig. 3.1b). The displacement of the lever was
proportional to that of a robot arm which delivers water to rats (Fig. 3.1c). Once the
displacement exceeded the threshold, the rat could get the water reward. During the
experiment, multi-channel neural signals of the primary motor cortex (MI) and the
ventrolateral (VL) thalamus were recorded simultaneously. In behavioral control
mode, the rat drank water by pressing the lever. In the BMI mode, the connection
between the lever and the robot was cut off, and the displacement of the robot was
controlled by control commands decoded from the 32-channel neural signals. Then,
the control mode switches to BMI mode. In the early stage of BMI mode, the rat still
tried to press the lever to get the reward. As time went on, the rat completed the task
only by modulating its neural activity, and the association between the forelimb’s
displacement and neural activity was weaker and weaker. Finally, the rat drank water
only by modulating neural signals without moving its forearm. The experiment first
proved the feasibility of an online BMI system and opened the door to the BMI field.
In 2009, Lee et al. built a rat BMI system to control turntables [6]. In behavioral
training, the rat was trained to control the direction and magnitude of a turntable to
obtain water. The motion set Cq consisted of seven elements {
3,
2,
1, 0, 1,
2, 3}, where the sign of the element represented the rotation direction and the
70 Y. Jin et al.

Fig. 3.1 The first BMI study in rodents [5]

Fig. 3.2 A turning table paradigm in rodent BMI study [6]

magnitude represented the rotation amplitude. After behavioral training, multi-

electrodes were implanted in the SI vibrissae area. As shown in Fig. 3.2, multi-
channel spike train was sent into feature extraction and channel selection units which
selected two channels sj1, sj2 as decoding channels according to correlation. The
3 Invasive Brain Machine Interface System 71

coding unit obtained the mapping between the relative deviation of sj1, sj2 and the
motion data Cq ¼f(sj1, sj2). According to the mapping obtained above, the decoder
generates the control command every 200 ms to make the turntable rotation.
In 2012, Manohar et al. built the BMI system of the rat hindlimb pressing lever
under three different conditions [7]. The rat was trained to press the lever to get the
reward in the behavioral control mode, which was followed by three neural control
modes (NC). In the first mode, the mechanical connection between the lever and the
reward delivery was cut off, but the rat could still press the lever. In the second mode,
the lever was removed and the rat got the reward purely by modulating its neural
signals. In the third mode, the spinal cord of the rat was completely transected and
the rat relearned to modulate the BMI system. They compared the performance and
neural firing pattern in the behavior control mode and three neural control modes.
When the control mode was switched from the behavior control mode to the neural
control mode, the response of the direct population was significantly enhanced, and
the decoded information was significantly increased. When the lever was removed,
the control performance was similar but the neural response changed significantly.
When the spinal cord completely transected, the decoded information decreased by
40%. With a period of learning, the control performance gradually recovered, but
still worse than before. The experiment demonstrated that the animals’ state, such as
the limitation of movement and the integrity of neural pathways, had a large
influence on the BMI performance and the firing pattern of a neural population.
For some paralyzed patients, such as those with spinal cord injury, their cerebral
motor cortex and effector (such as the arm) are intact and the lesion is located on the
neural pathway, which breaks the transmission of information. For these patients, the
best way to repair the motor function is to build an artificial neural pathway between
the cerebral motor cortex and the effector. So researchers tried to combine traditional
BMI with muscle electrical stimulation technique and put forward a new repairing
technique of neural pathway. In 2014, Alam et al. reconstructed the neural pathway
between the primary motor cortex and the hind limb of the rat with spinal cord
transection [8]. They recorded the neural activity from the primary motor cortex of
running rats before and after spinal cord transection in rats running tasks. The results
showed that the association between neural activity and motion states decreased
significantly after the transaction, but the stationary state and moving state could be
still distinguished by decoding. After transection of the spinal cord, action potentials
of the motor cortex were used to stimulate the muscles of the hindlimb. When the
smoothed firing rate exceeds the threshold for the five consecutive time windows,
the neural stimulator stimulated the hind limb muscle with a bidirectional pulse
(biphasic pulse), which resulted in the synchronization movement between the hind
legs and forelimb
The traditional BMI has to acquire a section of neural signal and movement data
to train the decoder before translating the neural signals into control commands.
However, for paralyzed patients, it is difficult to get the training data. A solution is
based on motor imagination in which the paralyzed patients observe the automatic
movement of external device while imagining themselves controlling the device.
Then, the neural signals and the movement data are used to train decoder. With the
72 Y. Jin et al.

in-depth study of brain plasticity, researchers try to set the mapping between the
neural signal and motion states arbitrarily without a training set. Relying on the
neural plasticity, the brain modulates its firing pattern to adapt to the artificial
mapping and finally control the external device.
In 2005, without the advanced training and neural data for animals, Kipke et al.
made use of the brain plasticity and adaptive decoder to build a rat BMI system to
control an auditory cursor [9]. In each trial, the rat was given a target tone (10 Hz)
and then required to control the auditory cursor to match the target tone. A Kalman
decoder was used to translate the multi-channel neural signals into the pitch of an
auditory cursor which was fed back to the rat in real time. The initial parameters of
the decoder were randomly generated. A block estimation adaptive strategy was
used to update the decoder parameters. Each block contained 10 trials. After each
block, the pool of the neural signals and pitch of the auditory cursor during the
previous 10 trials served as the training set to retrain the decoder. The new decoder
was then used in the next block of trials. As the training went on, the success rate
increased gradually from the random level.
The experiment carried by Kipke et al. harnessed the plasticity of the brain and
adaptive strategy of the decoder to converge the encoding rules of both natural neural
system and decoder. Thus, the final neural mapping was based on natural mapping,
but not exactly the same. Considering the great role of neural plasticity in the BMI
system, researchers have tried to build some BMI systems entirely relying on the
plasticity of the neural system. In 2012, Carnema et al. proposed the operative
conditioning experiment based on neurons, which artificially set new neuronal
encoding rules which are completely different from natural ones. Then, the rat was
trained to modulate the activities of neurons to control an auditory cursor [10]. The
researchers successfully achieved the two-way control of the auditory cursor with
this method and studied the functional coherence and plasticity of cortex and
striatum. Specifically, two groups of primary motor cortex (MI) neurons were
selected as conditioned neurons, and the rat was trained to control the auditory
cursor to reach the high or low threshold by modulating neural firing patterns. The
relationship between the pitch of the auditory cursor and the response of the two
groups of neurons was opposite: the increased response of the first neural population
increased the pitch of the auditory cursor while that of the second neural population
reduced the pitch. Therefore, to reach the high pitch target, the rat had to activate the
first neural population while suppressing the second one; to reach the low pitch
target, the modulation was opposite. The results indicated that the firing patterns of
striatal neurons changed as the rat learned to control the auditory cursor and
gradually matched the preset encoding rules. Furthermore, the firing pattern of the
motor cortex and striatum become more and more similar. The experiment further
extended our understanding of neural plasticity and proved that through training the
animal could master a new encoding rule which was completely different from
natural ones and achieved control of the external device.
3 Invasive Brain Machine Interface System 73

In 2014, Widge et al. made a similar operant conditioning experiment to control

an auditory cursor [11]. Different from the Carnema’s experiment, they only
extracted one neuron from the prefrontal cortex (PFC) as the conditioned neuron,
and its firing rate was positively related to the pitch of the auditory cursor. The
experiment trained the rat to move the auditory cursor to the target within a period of
time to and maintain for 500 ms. After training, all rats were able to complete the
BMI tasks successfully. The results demonstrated the feasibility of a single neuron in
the operant conditioning control of BMI. In 2013, Arduin et al. achieved to train rat
to self-drink by modulating single neuron in the motor cortex to control the one-way
movement of the water bottle [12]. In the same year, he further improved his
experiment and achieved two-way continuous brain control [13].

3.2.2 Bidirectional Recording and Stimulating System

for Free Moving Rats

Just as described before, the information pathway of the one-way BMI is from the
brain to the external device whose feedback is transmitted back by natural sense such
as vision, audition, or touch. The development of intracranial microstimulation
(ICMS) provides a new approach to transmit information directly to the brain
independent of the natural neural feedback pathways. Recently, the technique of
guiding the animal to conduct a specific task by ICMS has become more and more
mature. Therefore, researchers tried to build the two-way information interaction
between the neural system and the external device, e.g., bidirectional BMI.
The first bidirectional BMI is a hybrid neuro-robotic system connecting the
brainstem tissue of lamprey in vitro and a moving robot, which was built in
Northwest University in 2003 [14]. After that, the bidirectional BMIs based on
animal tissue in vitro have been widely studied.
In 2012, Vato et al. built a bidirectional BMI in vivo on an aesthetic rat.
16-channel microwire arrays were implanted into the primary motor cortex
(M1) and primary sensory cortex (S1), respectively [15]. On the “read-out” pathway,
the first two principal components of the M1 neural response were used to form a
two-dimensional force vector (Fig. 3.3b, c), which drive a simulated point-mass
moving in a viscous medium (Fig. 3.3d). On the “write-in” pathway, the position of
the point-mass was transformed into one of four stimulus pattern by “Sensory
Interface,” and each stimulus pattern was produced by a pair of channels of S1
microwire array (Fig. 3.3a). The rat was trained to modulate its M1 neurons firing
according to the stimulus feedback to move the point-mass from an arbitrary starting
location to the selected equilibrium point.
In 2010, Kipke et al. built a bidirectional BMI based on the awake rat [16]. An
abstract task was designed that required the rat to modulate its neural activity and
gave the feedback by ICMS. The motor cortex and the visual cortex of the untrained
74 Y. Jin et al.

Fig. 3.3 A bidirectional BMI in vivo on an aesthetic rat [15]

rat was implanted with multi-channel electrodes. In each trial, the rat maintained its
motor cortex ensemble baseline firing rates for 450 ms to begin a trial. Then, the “go”
cue was given by ICMS and the rat had to modulate its motor cortex for 450 ms
within the 4 s response period in order to receive the food reward. The rat was given
continuous feedback via visual cortex ICMS during the response periods that was
representative of the motor cortex ensemble dynamics. The results indicated that the
rat could master an open-loop control task and build the association between the
stimulus and the firing pattern of the motor cortex through a period of training.
Gage et al. designed a more systematic BMI to transform neurons in rats into
sounds of different frequencies [9]. For a successful trial, rats need to regulate
neuronal activity to reach the target frequency and maintain a certain time.
DiGiovanna et al. proposed another BMI adaptation paradigm based on enhanced
learning [17]. The goal of the intelligent system was to maximize the final reward
after making the action in the intelligent system. The brain cortex uses the intrinsic
reinforcement learning mechanism to regulate the relevant neurons, and the machine
needs to simulate the corresponding reinforcement learning process. Mahmoudi
et al. proposed the concept of symbiotic BMI, which obtained the reward informa-
tion directly from the NAcc region of the rat brain as feedback to machine enhanced
learning [18].
3 Invasive Brain Machine Interface System 75

3.2.3 New Exploration of BMIs Researches in Rodent

Animals

The BMI technique based on the neural signal recording, encoding, decoding, and
ICMS could not only build the artificial open-loop information pathway between the
brain and the external device and rebuild or repair some damaged motor function but
also achieve the direct communication between the brain and external devices. In
2013, Nicolelis’s team in Duke University built a brain-to-brain interface (BTBI)
between two rats, which transmitted information from the brain of encoder rat to that
of decoder rat [19]. Specifically, the two rats were placed in two separated boxes, and
the encoder rat was trained to press one of two levels according to the visual stimulus
to receive the water reward. Meanwhile, M1 neural activity was recorded from the
encoder rat and transmitted to the decoder rat by ICMS. The decoder rat had to select
the same lever pressed by the encoder to get the water reward. Once the decoder rat
completed the trial successfully, the encoder rat got an extra reward. This suggested
that BTBIs could enable networks of animal’s brains to exchange, process, and store
information and, hence, serve as the basis for studies of novel types of social
interaction and for biological computing devices (Fig. 3.4).
Traditional BMI technique is based on electrophysiology, i.e., neural signal
collecting and ICMS. Apart from this, researchers have tried to make use of other
techniques to build BMIs. In 2014, Carnema’s team in UC Berkley used two-photon
imaging to perform a conditional operant experiment on awake rats and control an
auditory cursor [20]. The fluorescence value was binned by 200 ms time windows
and transmitted to the decoder to get the pitch of auditory cursor. The increase of the
first neural ensemble (E1) increased the pitch while the increase of the second neural
ensemble (E2) decreased the pitch. The rat was trained to modulate the activity of
both E1 and E2 population to maintain the cursor within a baseline range to start a

Fig. 3.4 A brain-to-brain interface (BTBI) system demonstrated on animal studies [19]
76 Y. Jin et al.

trial and then moved it to a high threshold within 30 s to complete a trial. This
experiment expanded the technique of BMI further.

3.3 BMIs on Non-human Primates

3.3.1 Neural Signal Processing and Neural Decoding

• Neural signal processing

Carmena et al. started studying invasive BMIs in non-human primates (macaques) in
2003, which promoted the rapid development of this technology [21]. Researchers
parsed a variety of movement-related parameters from recorded neural signals to
control a robotic arm and accomplish reaching and grasp with the visual feedback.
In 2008, Schwartz and his colleagues implemented that the monkey could
perform self-feeding in three-dimensional space with its arm fixed [22]. Researchers
sent the food to a monkey by equipment during the experiment. While the neural
signals of the monkey were decoded to control a robotic arm, the monkey could learn
to grip the food and feed itself. The research was of great significance in that it
reconstructed the motor function of arms in three-dimensional space.
The recording quality of neural activities is very important for neural signal
analysis. The collecting and recording of neural signals are becoming more accurate
with the development of the microelectrodes or probes. Nowadays, there exist
commercial electrodes that were approved by the Food and Drug Administration
(FDA) to be utilized even on human beings. The microelectrodes can be implanted
into several brain cortex to collect multi-channel neural activities. One of the key
questions in BMI is to decode the multi-channel neural activities efficiently and
accurately.
Neural signals contain some noises that are irrelative to the task, which raises the
difficulty of analyzing the model parameters, the complex of information learning
models, and real-time decoding. So it is essential to decrease the scale of neural
activities and extract the valid part of neural activities.
Chapin et al. utilized the principal component analysis (PCA) to reduce the
dimensionality of the neural activities to one. They established an analytical model
to predict the trajectory of the forearm [5]. The PCA reduces the dimensionality by
eliminating the correlation between the signals, and the dimensionality reduction
process might miss some useful information. Wessberg et al. used the Neuron-
Dropping Analysis to achieve information reduction [23]. This method is target
oriented with high computational complexity.
• Neural signal decoding
The studies on the neural population decoding in BMI can be divided into two
categories. The first method assumes that the relationship between neural activities
and movement conforms to the linear model, such as population vector and related
3 Invasive Brain Machine Interface System 77

algorithm [22, 24, 25], Wiener filter-based analysis method [21, 26, 27], Kalman
filter-based analysis method [28–30], Fisher linear discriminant based method, and
maximum likelihood estimation method. Although the linear analytic model has
achieved good results in relatively simple decoding applications, recent studies have
found that the nonlinear method has a more reasonable analogy to the real nervous
system. In recent years, many attempts have been made to use non-linear methods
for neural decodings, such as particle filter [31, 32] and unscented Kalman filters
[33, 34].
The above researches have established a good foundation for neural population
decoding. However, on account of the complexity of the brain system, it is necessary
to achieve efficient and accurate neural population decoding. First, the brain cortex is
a plastic system, so the time-varying neural activities raise great challenges to the
traditional static information processing method; second, the neural activities show
similar features in both time domain and frequency domain. Neural activities from
limited brain cortex and frequency band can hardly represent the whole brain system.
So the exploration of efficient, dynamic, and joint analytic models and algorithms is
still an open question for BMI.

3.3.2 Closed-Loop Decoding for Online Control

The ideal BMI aims to achieve bi-directional information interaction between the
brain and the external devices. In recent years, with the development of neural signal
analysis technology and neural stimulation technology, researchers have focused
their attention on bi-directional closed-loop BMI system research.
Novellino et al. combined in vitro cultured rat neurons and embedded signal
processing platform to build a bi-directional BMI system prototype. The system used
the signals of the neurons to control the movement of a car. There is stimulus
feedback to the neurons when the car hits the obstacle. So the bi-directional
interaction between neurons and the external environment was established
[35]. Doherty et al. gave monkey direction instruction by electrical stimulation on
the sensory cortex. Then, the direction selection task was achieved by real-time
decoding of the motor cortex. The results showed that the monkey could obtain
external information and output control sequences to the external devices [36]. In
addition, Ye, Rolston, and Venkatraman et al. developed a number of stimulated
neural signal acquisition systems that combine both neural stimulation and recording
techniques to achieve a preliminary bi-directional communication system. Above
studies showed that the bi-directional BMI has great application potential in devel-
oping disease treatment and understanding brain network mechanism [37–39].
Intracortical electrical stimulation, as a somatosensory information input, not
only establishes a new information transfer path between brain cortex and external
environment but may also improve the performance of the vision-based feedback
BMI by information integration. Preliminary studies have shown that the electrical
78 Y. Jin et al.

stimulation can not only instruct the movement of animals but also feedback the
movement information to the brain cortex.
O’Doherty et al. trained monkeys to finish the movement direction selection task
with tactile feedback and electrical stimulation [36, 40]. After two weeks, the result
showed that the natural tactile feedback in the experiment and cortical electrical
stimulation provide almost the same amount of information. At the same time, a
considerable number of studies have confirmed that multi-modal feedback can
provide more information and improve the performance of BMI. Nicoliles et al.
found that visual feedback is very important for the motor control BMI and visual
feedback coupled with tactile feedback can further improve the overall performance
of the system; [4] Suminski et al. studied the natural somatosensory feedback and the
mechanism of visual and found that somatosensory feedback is more likely to affect
the neural activity of the motor cortex than visual feedback, and more importantly,
the combination of two different modes of feedback can provide more information
[41]. Further results showed that the dual-mode feedback could significantly
improve the control performance of the BMI when the feedback of the two modes
is consistent.
Several studies have shown that the plasticity of the brain cortex is closely related
to BMI. In the process of neural decoding, neural activity is a dynamic learning
process, and the mapping between the output signal (usually the neural signals) and
the input signal (usually the kinematic data of the limb) is changing over time. In
previous BMI studies, results have shown that the activity of a single neuron may
change with the decoding algorithm. Ganguly et al. conducted a preliminary explo-
ration of the adaptability and plasticity of the neural population in the context of
BMI. Their findings suggest that the cerebral cortex has a very prominent learning
ability and can quickly adapt to mutations in the decoding model [42].
Other researchers also found that cortical electrical stimulation can also affect the
plasticity of the cortex. Moritz et al. have shown that functional connectivity can be
established between the two neurons by cortical electrical stimulation, which pro-
vides a new idea for nerve repair [43]. Rebesco et al. have shown that cortical
electrical stimulation can not only serve as an information input, but also can
effectively enhance the functional connection of nerve, and eventually lead to
behavioral changes, and this plasticity may be associated with natural stimulus
caused by Hebby learning mechanism [44].
The study of the changes of the neuron population under the condition of invasive
BMI would help us understand the plastic mechanism of the brain and construct the
bi-directional closed-loop interaction. The use of the plasticity of the brain, com-
bined with machine learning, can improve the performance of BMI. Taylor et al.
used a mutual adaptation algorithm on the rhesus monkey’s three-dimensional
cursor control task and updated the mapping of neuronal activity to the movement
of the cursor by iterating the changes in neuron during the learning process [24].
3 Invasive Brain Machine Interface System 79

3.4 Pilot BMIs Studies in Clinic Researches

3.4.1 Invasive BMI System Based on Spike Signals from

Human Brain

The ultimate goal of brain-machine interface development is to realize the practical

application of clinical patients. Currently, there are mainly two types of invasive
BMIs clinical studies. One type of BMIs employed fully implantable microelectrode
arrays to collect the spike signals of neuron populations. This method has the
advantage of acquiring high spatial and temporal resolution signals that contain
rich information of movement and can be used to realize the high-precision motion
control. However, due to the need of microelectrode array implanted directly into the
cerebral cortex, the damage induced by implantation makes it difficult to achieve
long-term application; the other type is a semi-invasive BMIs, which mainly used the
electrodes placed on the surface of the cortex, collecting field potential signals from
the cerebral cortex. Its damage to the brain is relatively small, but the spatial
resolution of this type of signal is relatively lower than that of the invasive micro-
electrode array. Each method has its advantages and disadvantages.
In the past two decades, many BMI system has also been developed for clinical
transformations. In 1998, Philip Kennedy and Roy Bakay from Emory University
started their researcher invasive BMIs [45]. Their study implanted a neurotrophic
electrode into a patient who had severe amyotrophic lateral sclerosis (ALS) for 2.5
years prior to the implantation and was ventilator-dependent for 1.5 years. The
electrode consists of a hollow glass conical tip about 1.5 mm in length which
contains two gold recording wires. They recorded action potentials in the patient’s
brain over several months. By decoding the action potential, the patient ultimately
was able to control the clicking of a computer mouse. This study was the first clinical
study of invasive BMI which achieve the high-quality neural signal recording and
movement output decoding, which laid the foundation for the development of
invasive BMI.
With the development of the electrode technology, many researchers began to
study the clinical BMI experiments based on the microelectrode arrays. In 2006, the
group of Donoghue achieved a BMI for movement intent decoding for a tetraplegic
patient [29]. The patient in this study was a 25-year-old male who had sustained
knife wound, the spinal cord between cervical vertebrae C3–C4 of the patient was
transected, which results in complete tetraplegia. During the research, a 96-ch
microelectrode array was implanted in the arm area of the primary motor cortex of
the patient for neural signal recording. The BMI system decoded the different
direction movement imagination and was finally used to control a computer mouse
in a two-dimensional plane. By making use of this BMI system, the patient was able
to open the e-mail, draw figures using a paint program, play video games, and even
adjusted the volume, channel, and power to his television. This study was the first
attempt to make use of microelectrode arrays in clinical BMI use, which provides
valuable research experience for the clinical transformation of invasive BMI. Then,
80 Y. Jin et al.

the group further study the long-term stability performance of the BMI system in
chronic application [46]. They found that the BMI system still kept a high system
performance even 1000 days after the microelectrode arrays implantation. Though
there were still many problems such as electrode aging and signal attenuation which
may affect the long-term use of BMI system, this study proved that the BMI based on
microelectrode array could keep a stable system performance for 2–3 years. In 2008,
other two subjects participated in the BMI experiment. One participant was a
54-year-old woman who had thrombosis of the basilar artery and extensive pontine
infarction 9 years prior to trial recruitment while another participant was a 37-year-
old man who had been diagnosed with ALS for about 6 years. The two participants
successfully achieved the online control of a computer mouse by using the BMI
system after some training sessions. This result proved that the BMI system might be
a benefit to the patients who suffer long-term motor dysfunction. Then, the further
study of robot hand control by BMI was carried out in 2012 [29]. A stroke patient
and the 54-year-old woman described above participated in this study. The system
successfully decoded the three-dimensional trajectory of the hand movement. The
brain activity of M1 was extracted to control the movement and grasp of a robot
hand. The 54-year-old woman successfully used the robot hand to grasp a coffee cup
and move to her own mouth to drink the coffee by absolutely brain control. This
study shows us the prospects of invasive BMI system in daily life application.
The group of Andrew B Schwartz also had some research on clinical invasive
BMI. They set up a clinical BMI system based on the microelectrode array in
2013 [46]. A 52-year-old woman who was tetraplegic participant in this study.
Two 96-ch microelectrode arrays were implanted in her arm area of premotor cortex
(PM), and the participant had to control a prosthesis using her brain signals by the
BMI system after a 13-weeks’ training. The participant had to perform a seven-
dimensional sequence task including three-dimensional position, rotating the palm
in one of six directions (possible orientations included pronation–supination 45 ,
ulnar-radial deviation 20 , and flexion-extension 45 of the wrist), or a grasp
target (open or closed hand). The participant used the prosthesis under full brain
control to do nine tasks (selected from 19 possible tasks) on the action research arm
test (ARAT), which is an assessment of the unilateral upper limb function used
commonly in patients who have had a stroke. This study showed a high-performance
BMI system with multi-degree of freedom, which was difficult to achieve by
non-invasive BMI. Then in 2015, the group of Schwartz further achieved the
ten-dimensional anthropomorphic arm control in human BMI [47]. The ten dimen-
sions contain translation (x,y,z), orientation (θx,θy,θz), pinch (p), scoop (s), finger
ab/adduction (f), and thumb opposition/extension (t). This study found that individ-
ual motor cortical neurons encode many parameters of movement and that high-
dimensional operation of prosthetic devices can be achieved with simple decoding
algorithms, which shows the invasive BMI has the potential to control high-degree
freedom prosthetic devices.
In recent years, more and more fully invasive BMI system has been studied.
Richard A. Andersen et al. set up a BMI system based on the signals in posterior
parietal cortex (PCC) [48]. PCC is regarded as being responsible for the process of
3 Invasive Brain Machine Interface System 81

high-level movement planning. This study implanted a microelectrode array to the

PCC and successfully decoded the different object and trajectory of motor imagina-
tion. Bouton et al. study the hand motor function rehabilitation based on invasive
BMI [49]. The study participant was a 24-year-old male with quadriplegia from
cervical spinal cord injury sustained in a diving accident. The researcher decoded the
neuronal activity and control activation of the participant’s forearm muscles through
a high-resolution neuromuscular electrical stimulation system. Through the system,
the participant complete a Grasp-pour-and-stir functional movement task, which
need the participant to opening his hand (a), grasping the glass bottle (b), pouring its
contents (dice) into a jar (c), grasping a stir stick from another jar (d), transferring the
stir stick without dropping it (e), and using it to stir the dice in the jar (f). These
results have significant implications in advancing neuroprosthetic technology for
people worldwide living with the effects of paralysis, which extended the application
of BMI.

3.4.2 Invasive BMI System Based on Clinic ECoG Signals

We can obtain high temporal-spatial resolution brain signals by implanting micro-

electrode array to decode higher degrees of freedom and finer arm motion and
achieve more natural and complex arm movement, which is the goal and direction
of BMI. However, the fully implantable microelectrode arrays may cause damage to
the blood-brain barrier, inflammatory response, and immune reaction, which may
produce a series of biological and non-biological effects, leading to the attenuation
and loss of the brain signals. It limits the application of implantable microelectrode
array in the clinical BMI studies. On the contrary, the electrocorticogram (ECoG)
electrode is located under the subdural and on the surface of cortical, which does not
cause damage to the cerebral cortex. The neuronal damage can be completely
avoided. Although the spatial resolution of the ECoG signal is slightly lower than
that of the microelectrode array, the ECoG has a fewer surgical risk and better long-
term stability of the signal. So the ECoG is relatively more meaningful for the
clinical application of BMI. In the past few years, the study of clinical BMI based
on ECoG signals had made great progress.
As early as in 2004, Gerwin Schalk, Eric C Leuthardt, Daniel W Moran ,and
Jeffrey G Ojemann started the study of ECoG-based clinical BMI [49]. All the
participants in the study were clinical epilepsy patients in whom subdural electrode
arrays were implanted for three to eight days in preparation for surgery to remove an
epileptic focus. The ECoG arrays located in sensorimotor cortex. Over brief training,
the four participants then used these signals to master closed-loop control and to
achieve success rates of 74–100% in a one-dimensional (1-D) binary task. Further
analysis showed that ECoG also has a correlation with the direction of the
two-dimensional joystick movement. Then in 2006, the same group set up a
closed-loop BMI in paralyzed patients [50]. There are four participants, and the
electrode covered their sensorimotor cortex and some speech cortex areas. In order
82 Y. Jin et al.

to identify brain signals used for BCI control, the subjects were asked to perform
paired blocks of repetitive hand/tongue, foot/shoulder movements, or repetitive
speaking of the word “move” (screening procedure). From the spectral analysis,
the researchers identified the locations and frequency bands in which amplitude was
different between the task and rest, which will be selected in the subsequent closed-
loop BMI. The participants received online feedback that consisted of the 1-D cursor
movement that had shown correlation with tasks during the screening procedure.
Their study showed that visual cursor feedback could effectively improve the
success rate of the task. The work of Schalk was the first attempt to set up ECoG-
based clinical BMI, which proved that ECoG was an available signal source in
clinical BMI use.
The group of Schalk then used the ECoG signal to decode the trajectory of the
cursor in two-dimensional plane [51]. The study included five patients with intrac-
table epilepsy. The subjects were asked to use a joystick to move a white cursor in
two dimensions to track a green target. By decoding the kinematic parameters from
ECoG, the study showed that the correlation coefficients (r) between the actual and
decoded kinematic parameters were as high as 0.7, which proved that ECoG-based
BMI was feasible in two-dimensional trajectory decoding. Carsten Mehring et al. did
similar work in 2008 [56]. Six patients suffering from intractable pharmaco-resistant
epilepsy were included in their study, and the electrical stimulation through the
electrode grid was performed to identify the functional mapping of the cortex. A
Kalman filter was applied to decode the two-dimensional trajectory of the arm, and
their result showed that the low-frequency component of ECoG has better perfor-
mance for the two-dimensional trajectory decoding. In 2013, Yasuhiko Nakanishi
and Takufumi Yanagisawa from Japan successfully decoded the three-dimensional
trajectory of arm movement using ECoG signals [52]. There were three participants
in this study; two of them had spastic paresis and weakness due to stroke, and the
other was diagnosed with intractable epilepsy. The participants were asked to
reposition three blocks one by one and clockwise at the corners of a square table,
and an optical motion capture system was used to record the arm movement. Their
work successfully predicted three-dimensional arm trajectories in time series from
ECoG signals, which proved that we could extract the motion parameters of three-
dimensional arm movement from ECoG signals.
In addition to continuous arm movement trajectory decoding, some teams have
used ECoG signals to decode different finger movement and gestures. In 2009,
Schalk et al. proved that it is possible to decode the flexion of individual fingers
using ECoG signals [53]. A data glove (5DT) was used to record the flexion of each
finger in this study. Their result showed that there was a different spatial pattern of
ECoG signals for different finger movement and the LMP, and high gamma com-
ponent of ECoG could decode the flexion of each finger with best decoding
performance. Then in 2010, Nitish V. Thakor et al. found that ECoG signals
correlated with finger positions [54]. Their study indicated that when the subjects
engaged in slow and deliberate grasping motions, the amplitudes of the low-pass
ECoG (LMP) over specific electrodes were correlated with the position of individual
fingers. The work of Mehring and Shenoy both showed that ECoG signals could be
3 Invasive Brain Machine Interface System 83

used to decode different gestures [55–58]. Mehring’s experiment requires subjects to

grasp different objects with different gestures; and in Shenoy’s experiment, the
subject was asked to perform several specific gestures, which require different
fingers to bend and stretch. Both experiments proved that ECoG signal was able to
decode different gestures.
For that ECoG signals could be used to decode different motion parameters, some
studies have implemented prosthesis control based on ECoG signals. In 2013,
Matthew S. Fifer et al. trained two human subjects to achieve simultaneous neural
control of reaching and grasping movements with a dexterous robotic prosthetic arm
[59]. The group of Toshiki Yoshimine from Japan also used human ECoG signal to
real-time control of a prosthetic hand [60]. Two decoders were trained to control the
prosthetic hand. The first decoder was trained to classify the movement state R or M,
and the second was trained to predict the types of performed movement with the state
M. Based on the two decoders, the subject could asynchronously control of the
prosthetic hand to perform three types of hand movement. In a further study, the
subject could control the prosthetic hand to perform more tasks such as grasping,
thumb flexion, elbow flexion, pinching, hand-opening, elbow flexion, and elbow
extension [61]. This study tested the clinical BMI in a large number of subjects
(12 subjects included) and achieved prosthetic hand control for multiple types of
gestures. Wei Wang et al. used ECoG signals to control of 3-D cursor movement and
3-D prosthetic arm movement [62]. The participant in this study was a 30-year-old
man with tetraplegia; he was asked to observe hand and arm movements of a virtual
character on an LCD screen and simultaneously attempted the same movement. The
participant successfully controlled the prosthetic arm hitting physical targets. This
study indicated that we could extract the information of motion intent from ECoG
signals. Another application of clinical BMI was a brain-control typing system.
Ramsey et al. set up a BMI-based computer typing program for a locked-in patient
with ALS [63]. The ALS patient was a 58 years old woman, and four subdural
electrode strips were implanted in her brain motor cortex. The signal recording,
conversion, and transmission devices of the typing system were fully implanted,
which make the BMI system more simple and convenient for use. Portability and
simplification have gradually become a key factor in the clinical transformation of
the BMI system.
The key factor in invasive BMI is to improve the decoding performance of the
system as much as possible. ECoG arrays usually cover a wide range, and ECoG
signals have a wide frequency range, so it’s meaningful to set up a BMI of ECoG
based on multi-frequency band and multi-brain area. Ramsey et al. set up a BMI to
decode gesture and compared the decoding performance of M1, S1, and whole
sensorimotor cortex (M1 + S1) [63]. The result showed that decoding performance
of M1 and S1 was similar, while the decoding performance of the whole sensori-
motor cortex was significantly better, which means that the BMI of ECoG based on
the multi-brain area may improve the system performance. Hotson et al. study the
decoding performance of different frequency component of ECoG [64]. They
divided the ECoG signal to seven frequency band, 0–4 Hz, 4–8 Hz, 7–13 Hz,
14–30 Hz, 30–50 Hz, 70–110 Hz, and 130–200 Hz, and used the signals of the
84 Y. Jin et al.

seven single frequency band and multi-frequency band to decode the arm trajectory,
respectively. Their result showed that multi-frequency band has the best decoding
performance for all subjects. The above studies showed that take advantage of the
ECoG signals of different brain area and different frequency band was of great
benefit to improve the decoding performance.

3.5 Opportunities and Challenges

3.5.1 Opportunities

• Restoration of motor function in the paralyzed

One of the most important applications of BMIs is the restoration of motor function
in the paralyzed. Two main reasons that lead to paralysis are descending
corticospinal tract injury caused by spinal cord injury and neuromuscular dysfunc-
tion. The paralyzed suffer great physical and psychological trauma. One of the
reasons for the rise of the technology of BMIs exactly is to help the paralyzed restore
motor function. Researchers have conducted a series of experiments on non-human
primates and human and have made some progress, laying a good foundation for the
clinical practice of BMIs.
Capogross and his colleagues alleviated gait deficits after a spinal cord injury in
non-human primates in 2016 [65]. First, unilateral spinal cord injury model was
established by the side cutting the middle section of the monkey’s spinal cord.
Second, the monkeys were implanted with a microelectrode array into the leg area
of the left motor cortex and with electrical stimulators into the lower spinal cord
segment and the spinal cord segment associated with motion control of lower limbs.
During the movement of monkeys, neural signals were used to decode the movement
of lower limbs. Decoding results were then sent to control electrical stimulators to
simulate spinal cord, consequently alleviating gait deficits of lower limbs.
• BMIs as a potential therapy for nerve recovery
Since the 1990s, BMIs have been mainly focused on two aspects: (1) The field of
neuroscience, in which the physiological characteristics of the neural circuits are
studied by designing new experimental paradigms. (2) The field of neural engineer-
ing, in which technologies and means of restoring motor function for the paralyzed
are studied. Recent clinical studies have found that the BMIs used as one of the
potential tools for neurorehabilitation. The results have shown that long-term BMIs
training combined with physical therapy has a significant effect on the recovery of
motor function. So does BMIs training combined with virtual reality system.
In 2016, Donati conducted 12 months training with a multi-stage BMI-based gait
neurorehabilitation paradigm aimed at restoring locomotion in eight paraplegic
patients [66]. This paradigm combined intense, immersive virtual reality training,
enriched visual-tactile feedback, and walking with two EEG-controlled robotic
3 Invasive Brain Machine Interface System 85

actuators, including a custom-designed lower limb exoskeleton capable of delivering

tactile feedback to subjects. Following 12 months of training with this paradigm, all
eight patients experienced neurological improvements in somatic sensation. Patients
also regained voluntary motor control in key muscles below the SCI level. As a
result, 50% of these patients were upgraded to an incomplete paraplegia classifica-
tion. Donati and his colleagues hypothesized that this unprecedented neurological
recovery results from both cortical and spinal cord plasticity triggered by long-term
BMI usage.

3.5.2 Challenges

• Biocompatibility and long-term stability of electrodes

Biocompatibility and long-term stability of electrodes is the key that determines
whether BMIs technology can be widely applied to the clinic. Biocompatibility of
electrodes is closely related to the long-term stability of recording. Currently, the
biocompatibility of implanted microelectrodes is poor. And with the increase of
implant time, it’s easy to induce the rejection of biological tissue. The hyperplastic
tissue covers the surface of the electrode, resulting in a reduction in the quality of the
signal recorded by the electrodes. Simultaneously, the impedance variations caused
by the corrosion and degradation of the electrode materials will also influence the
long-term stability of recording.
Researchers have proposed some relevant strategies to enhance the stability of
electrode recording, including electrode improvement in size, geometry, shape,
coating, and materials. The results of the study confirm that smaller electrode size
can achieve a relatively long signal recording time. At present, researchers have
focused on the feasibility of extracellular or intracellular neural recording with
carbon nanotube electrodes in that they have low impedance, and are capable of
recording the activities of a single unit. Another focus is a bioactive coating, which
has the effect of slowing the rejection and enhancing the stability of the signal
recording. In addition, the bioactive coating can reduce the response of glial cells and
increase the survival rate of peripheral nerve cells.
• The portability of BMIs system
In real life, a BMI operator may not carry a large number of signal cables and sig-
nal acquisition equipment. It is important to study how to simplify the BMI system or
make it more portable. Ramsey et al. developed a highly integrated BMI
system which could be fully implanted into a patient with late-stage amyotrophic
lateral sclerosis (ALS) [63]. The patient could accurately and independently con-
trolled a computer typing program with the assistance from this fully implanted BMI
system. The future BMI system could be further integrated wireless communication
and power. Another critical factor in the clinical application of invasive BMI is the
decoding performance of the system. In practical applications, there might be no
86 Y. Jin et al.

fault tolerant space so that we have to improve the decoding performance of a BMI
system and minimize the decoding error rate as much as possible. The studies of
Ramsey and Hotson et al. have proved that combining the neural signals from
different brain regions and frequency components is of great significance to improve
the decoding performance of a BMI system [63, 64]. In addition, looking for a better
feature, exploring the neural mechanism underlying movement control, and devel-
oping decoding algorithm more in line with the neural characteristics will play a key
role to improve the performance of the BMI system.
Clinical translation of invasive BMI is an essential process before a BMI system
could be brought into clinical application and even real-life application. We still
need to solve some critical issues when translating the BMI system from the lab to
the bed of patients and form a stable and mature application system. When the BMI
system could be applied to reallife application, the brain control as Avatar will no
longer be fantasy.

References

1. Andersen RA et al (2014) Toward more versatile and intuitive cortical brain–machine inter-
faces. Curr Biol 24(18):R885–R897
2. Donoghue JP (2008) Bridging the brain to the world: a perspective on neural interface systems.
Neuron 60(3):511–521
3. Hatsopoulos NG, Donoghue JP (2009) The science of neural interface systems. Annu Rev
Neurosci 32:249–266
4. Lebedev MA, Nicolelis MAL (2006) Brain-machine interfaces: past, present and future. Trends
Neurosci 29(9):536–546
5. Chapin JK et al (1999) Real-time control of a robot arm using simultaneously recorded neurons
in the motor cortex. Nat Neurosci 2(7):664–670
6. Lee Y et al (2009) Classification of BMI control commands from rat’s neural signals using
extreme learning machine. Biomed Eng Online 8:29
7. Manohar A et al (2012) Decoding hindlimb movement for a brain machine interface after a
complete spinal transection. PLoS One 7(12):e52173
8. Alam M et al (2014) A brain-machine-muscle interface for restoring hindlimb locomotion after
complete spinal transection in rats. PLoS One 9(8):e103764
9. Gage GJ et al (2005) Naive coadaptive cortical control. J Neural Eng 2(2):52–63
10. Koralek AC et al (2012) Corticostriatal plasticity is necessary for learning intentional
neuroprosthetic skills. Nature 483(7389):331–335
11. Widge AS, Moritz CT (2014) Pre-frontal control of closed-loop limbic neurostimulation by
rodents using a brain-computer interface. J Neural Eng 11(2):024001
12. Arduin PJ et al (2013) "Master" neurons induced by operant conditioning in rat motor cortex
during a brain-machine interface task. J Neurosci 33(19):8308–8320
13. Arduin PJ et al (2014) Bidirectional control of a one-dimensional robotic actuator by operant
conditioning of a single unit in rat motor cortex. Front Neurosci 8:206
14. Kositsky M et al (2003) Dynamical dimension of a hybrid neurorobotic system. IEEE Trans
Neural Syst Rehabil Eng 11(2):155–159
15. Vato A et al (2012) Shaping the dynamics of a bidirectional neural interface. PLoS Comput Biol
8(7):e1002578
3 Invasive Brain Machine Interface System 87

16. Marzullo TC et al (2010) Development of closed-loop neural interface technology in a rat

model: combining motor cortex operant conditioning with visual cortex microstimulation. IEEE
Trans Neural Syst Rehabil Eng 18(2):117–126
17. DiGiovanna J et al (2009) Coadaptive brain-machine interface via reinforcement learning. IEEE
Trans Biomed Eng 56(1):54–64
18. Mahmoudi B, Sanchez JC (2011) A symbiotic brain-machine interface through value-based
decision making. PLoS One 6(3):e14760
19. Pais-Vieira M et al (2013) A brain-to-brain interface for real-time sharing of sensorimotor
information. Sci Rep 3:1319
20. Clancy KB et al (2014) Volitional modulation of optically recorded calcium signals during
neuroprosthetic learning. Nat Neurosci 17(6):807–809
21. Carmena JM et al (2003) Learning to control a brain-machine interface for reaching and
grasping by primates. PLoS Biol 1(2):E42
22. Velliste M et al (2008) Cortical control of a prosthetic arm for self-feeding. Nature 453
(7198):1098–1101
23. Wessberg J et al (2000) Real-time prediction of hand trajectory by ensembles of cortical neurons
in primates. Nature 408(6810):361–365
24. Taylor DM, Tillery SIH, Schwartz AB (2002) Direct cortical control of 3D neuroprosthetic
devices. Science 296(5574):1829–1832
25. Wahnoun R, He J, Helms TS (2006) Selection and parameterization of cortical neurons for
neuroprosthetic control. J Neural Eng 3(2):162–171
26. Kim S et al (2008) Neural control of computer cursor velocity by decoding motor cortical
spiking activity in humans with tetraplegia. J Neural Eng 5(4):455–476
27. Hochberg LR et al (2006) Neuronal ensemble control of prosthetic devices by a human with
tetraplegia. Nature 442(7099):164–171
28. Wu W et al (2003) Neural decoding of cursor motion using a Kalman filter
29. Hochberg LR et al (2012) Reach and grasp by people with tetraplegia using a neurally
controlled robotic arm. Nature 485(7398):372–375
30. Gilja V et al (2012) A high-performance neural prosthesis enabled by control algorithm design.
Nat Neurosci 15(12):1752–1757
31. Brockwell AE, Rojas AL, Kass RE (2004) Recursive bayesian decoding of motor cortical
signals by particle filtering. J Neurophysiol 91(4):1899–1907
32. Shoham S et al (2005) Statistical encoding model for a primary motor cortical brain-machine
interface. IEEE Trans Biomed Eng 52(7):1312–1322
33. Li S, Li J, Li Z (2016) An improved unscented Kalman filter based decoder for cortical brain-
machine interfaces. Front Neurosci 10:587
34. Li Z et al (2009) Unscented Kalman filter for brain-machine interfaces. PLoS One 4(7):e6243
35. Novellino A et al (2007) Connecting neurons to a mobile robot: an in vitro bidirectional neural
interface. Comput Intell Neurosci 2007:12725
36. O’Doherty JE et al (2011) Active tactile exploration using a brain-machine-brain interface.
Nature 479(7372):228–U106
37. Ye X et al (2008) A portable telemetry system for brain stimulation and neuronal activity
recording in freely behaving small animals. J Neurosci Methods 174(2):186–193
38. Rolston JD, Gross RE and Potter SM (2009) NeuroRighter: closed-loop multielectrode stimu-
lation and recording for freely moving animals and cell cultures. Conf Proc IEEE Eng Med Biol
Soc, 2009:6489–6492
39. Venkatraman S and Carmena JM (2011) Active sensing of target location encoded by cortical
microstimulation. IEEE Trans Neural Syst Rehabil Eng, 19(3):317–324
40. O’Doherty JE et al (2011) Active tactile exploration using a brain-machine-brain interface.
Nature, 479(7372):228–231
41. Suminski AJ et al (2010) Incorporating Feedback from Multiple Sensory Modalities Enhances
Brain-Machine Interface Control. Journal of Neuroscience 30 (50):16777–16787
88 Y. Jin et al.

42. Karunesh Ganguly, Jose M. Carmena and James Ashe (2009) Emergence of a Stable Cortical
Map for Neuroprosthetic Control. PLoS Biology 7(7):e1000153
43. Chet T Moritz and Eberhard E Fetz (2011) Volitional control of single cortical neurons in a
brain–machine interface. Journal of Neural Engineering 8(2):025017
44. James M Rebesco and Lee E Miller (2011) Enhanced detection threshold for cortical stimula-
tion produced by Hebbian conditioning . Journal of Neural Engineering 8(1):016011
45. P.R. Kennedy et al (2000) Direct control of a computer from the human central nervous system.
IEEE Transactions on Rehabilitation Engineering 8(2):198–202
46. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, McMorland
AJC, Velliste M, Boninger ML, Schwartz AB (2013) High-performance neuroprosthetic control
by an individual with tetraplegia. Lancet 381(9866):557–564
47. Wodlinger B, Downey JE, Tyler-Kabara EC, Schwartz AB, Boninger ML, Collinger JL (2015)
Ten-dimensional anthropomorphic arm control in a human brain
machine interface: difficul-
ties, solutions, and limitations. J Neural Eng 12(1):016011
48. T. Aflalo et al (2015) Decoding motor imagery from the posterior parietal cortex of a tetraplegic
human. Science 348(6237):906–910
49. Eric C Leuthardt et al (2004) A brain–computer interface using electrocorticographic signals in
humans. Journal of Neural Engineering 1(2):63–71
50. E.C. Leuthardt et al (2006) Electrocorticography-Based Brain Computer Interface—The
Seattle Experience. IEEE Transactions on Neural Systems and Rehabilitation Engineering
14(2):194–198
51. G Schalk et al (2007) Decoding two-dimensional movement trajectories using
electrocorticographic signals in humans. Journal of Neural Engineering 4(3):264–275
52. Yasuhiko Nakanishi et al (2013) Prediction of Three-Dimensional Arm Trajectories Based on
ECoG Signals Recorded from Human Sensorimotor Cortex. PLoS ONE 8(8):e72085
53. J Kubánek et al (2009) Decoding flexion of individual fingers using electrocorticographic
signals in humans. Journal of Neural Engineering 6(6):066001
54. Acharya S et al (2010) Electrocorticographic amplitude predicts finger positions during slow
grasping motions of the hand. J Neural Eng 7(4):046002
55. Pistohl T et al (2008) Prediction of arm movement trajectories from ECoG-recordings in
humans. J Neurosci Methods 167(1):105–114
56. Pistohl T et al (2012) Decoding natural grasp types from human ECoG. NeuroImage
59(1):248–260
57. Pistohl T et al (2013) Grasp detection from human ECoG during natural reach-to-grasp
movements. PLoS One 8(1):e54658
58. Chestek CA et al (2013) Hand posture classification using electrocorticography signals in the
gamma band over human sensorimotor brain areas. J Neural Eng 10(2):026002
59. Fifer MS et al (2014) Simultaneous neural control of simple reaching and grasping with
the modular prosthetic limb using intracranial EEG. IEEE Trans Neural Syst Rehabil Eng
22(3):695–705
60. Yanagisawa T et al (2011) Real-time control of a prosthetic hand using human
electrocorticography signals. J Neurosurg 114(6):1715–1722
61. Yanagisawa T et al (2012) Electrocorticographic control of a prosthetic arm in paralyzed
patients. Ann Neurol 71(3):353–361
62. Wang W et al (2013) An Electrocorticographic brain Interface in an individual with tetraplegia.
PLoS One 8(2):e55344
63. Vansteensel MJ et al (2016) Fully implanted brain–computer Interface in a locked-in patient
with ALS. N Engl J Med 375(21):2060–2066
64. Hotson G, Fifer MS, Acharya S, Benz HL, Anderson WS, Thakor NV, Crone NE, Robin DA
(2014) Coarse Electrocorticographic decoding of Ipsilateral reach in patients with brain lesions.
PLoS One 9(12):e115236
65. Capogrosso M, Milekovic T, Borton D, Wagner F, Moraud EM, Mignardot J-B, Buse N,
Gandar J, Barraud Q, Xing D, Rey E, Duis S, Jianzhong Y, Ko WKD, Li Q, Detemple P,
3 Invasive Brain Machine Interface System 89

Denison T, Micera S, Bezard E, Bloch J, Courtine G (2016) A brain–spine interface alleviating

gait deficits after spinal cord injury in primates. Nature 539(7628):284–288
66. Donati AR et al (2016) Long-term training with a brain-machine Interface-based gait protocol
induces partial neurological recovery in paraplegic patients. Sci Rep 6:30383
Chapter 4
Peripheral Neural Interface

Peng Zhang, Xiao Li, Dingyin Hu, Qiuxia Lai, Yuanyuan Wang, Xuan Ma,
Qi Xu, Wei Li, Jian Huang, and Jiping He

Abstract Peripheral nervous system, widely spread in the whole body, is the
important bridge for the transmission of neural signals. Signals from the central
nervous system (brain and spinal cord) are transmitted to different parts of the body
by the peripheral nerves, while along the way they also feedback all kinds of sensory
information. Certain level of information integration and processing also occurs in
the system. It has been shown that neural signals could be extracted from the distal
end of the stump, indicating that the bridge is still effective after limb damage or
amputation, which is the neurophysiological basis for the research and development
of peripheral nerve interface for the prosthetic system.

Keywords Peripheral nervous · Motor information · Sensory information · Nerve

regeneration and repair

4.1 Introduction

The physical disability can be caused by many factors such as accidents, wars, and
natural disasters besides the primary malignant tumors in the limb, such as
thromboangiitis obliterans, diabetes, and other diseases caused by tissue infection

P. Zhang · X. Li · D. Hu · Q. Lai · Y. Wang · X. Ma · Q. Xu · W. Li · J. Huang

Neural Interface and Rehabilitation Technology Research Center, School of Automation,
Huazhong University of Science and Technology, Wuhan, China
J. He (*)
Neural Interface and Rehabilitation Technology Research Center, School of Automation,
Huazhong University of Science and Technology, Wuhan, China
Beijing Advanced Innovation Center for Intelligent Robot and System, School of Mechatronical
Engineering, Beijing Institute of Technology, Beijing, China
Center for Neural Interface Design, School of Biological and Health Systems Engineering,
Arizona State University, Tempe, AZ, USA
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2019 91

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_4
92 P. Zhang et al.

necrosis, which may also lead to amputation. According to the data from the
National Bureau of Statistics in 2006, there are 24.12 million people with various
physical disabilities in China, accounting for 29.07% of the total number of disabled
people. For these persons with physical disability, the movement function restora-
tion of hand and upper limb is particularly important. Developing novel intelligent
prostheses and corresponding control methods can help them fetch food, grasp
different objects, and restore important motor functions in their daily life, thus
improving their life quality and even helping them return to work, which is of
great significance to social development and stability.
Many amputees are not willing to wear and use traditional prostheses because of
their limited functions and degrees of freedom (DoF). In recent years, some intelli-
gent prostheses with excellent and reliable performance and high DoF have been
developed, such as DLR/HIT prosthetic manipulator developed by the German
Aerospace Center and Harbin Institute of Technology [1]; the i-limb prosthetic
hand with multiple DoF made by British Touch Bionics; the SmartHand, dexterity
prosthetic hand, developed in Italy; and so on. However, the outstanding perfor-
mance of these prostheses will not be fully realized until the methods enabling the
amputees to control the prosthesis with multiple DoF accurately and easily have
been developed.
Extracting motor commands from neural signals to control the prosthesis by
amputees’ intention is a key breakthrough for improving the prosthesis performance
and is also an interdisciplinary research focus. Electromyogram (EMG), electroen-
cephalogram (EEG), cortical neural signal (CNS), or peripheral neural signal (PNS)
all can be used as the source of prosthetic control instructions. However, they have
different characteristics in signal acquisition technique and quality, and actual
control performance while constructing interfaces with different degrees of inva-
siveness and feasibility, details are described in Table 4.1.
EMG can be recorded easily and noninvasively, and there are commercial EMG
prosthetic products. However, limited control instructions can be extracted from
EMG signals while a sufficient number of muscles without atrophy in the residual
limb are required, which is not applicable for patients with high amputation and
muscular atrophy. The target muscle rehabilitation technology (target muscle recon-
struction, TMR), created by the American Chicago Rehabilitation Center, can
effectively solve the problem of controlling in high limb amputation. To a certain
extent, the DoF and flexibility of controlling can be improved by TMR, but its
clinical application is limited for the complexity in implementation [2].
EEG can also be recorded noninvasively and has high temporal resolution. Brain
computer interface (BCI) based on EEG has been widely applied in rehabilitation
training and prostheses control. However, prosthesis control based on EEG requires
users to concentrate on movement imagery without other actions (like talking or
other movements and so on); otherwise, correct movement intentions cannot be
extracted. In daily life, people always performed more than one action at the same
time. So, this method is unnatural and is not consistent with user habits while having
bottlenecks in rate of information transmission, control accuracy, and control com-
mands decoding [3].
4 Peripheral Neural Interface 93

Table 4.1 Characteristic comparison among different prosthetic control interfaces

Interface type Advantages Disadvantages
EMG Surface Normal Noninvasive, simple procedure and Low DOF, unnatural
EMG sEMG convenient adjustment
(sEMG) High real-time in controlling Sufficient normal muscles
in residual limb
Existing commercial products, Not suitable for high level
mature technology amputation
Electrode movements,
channel crosstalk, Low
long term stability
TMR Suitable for high level Not suitable for distal
amputation amputation
Higher flexibility of control
Some sensory feedback can be Complexity and a large
restored by functional electrical number of electrodes are
stimulation required
Invasive EMG High signal quality A lot of electrodes are
(iEMG) Small channel crosstalk required
High selectivity
EEG High temporal resolution Electrode placement is
complex
Noninvasive Unnatural
Have been widely used in disease Low information transfer
detection and rehabilitation rate
treatment Low control command
type
Susceptible to
interference
CNS (Local field potential Directly decode the high-level Large invasive
and spike) control command
PNS Natural Special electrode is
required
More commands can be decoded Lack of decoding method
from one electrode channel
High reliability Immature technology

A large amount of movement and sensory information can be accurately extracted

from cortical neural signals in particular cortex areas for controlling assistive devices
precisely. However, electrode implantation in the cortex has to be performed via
craniotomy, and this surgery has relatively larger risk. Moreover, long-term reliabil-
ity and biocompatibility of the electrodes are also a problem because the patient can
only undergo such electrode implantation surgery once [4].
Comprehensive consideration of invasiveness, expected effects, and ease of
implementation, the best way for constructing interface for prosthesis control, is
based on PNS. G. S. Dhillon et al. in the University of Utah have demonstrated that
residual peripheral nerve keeps the original function and can transmit control
commands for the amputated limb even after amputation for a long time [5]. Making
94 P. Zhang et al.

full use of the signals from the residual peripheral nerve, amputees may use the
prosthesis freely as with their own arms. PNS can be acquired from special elec-
trodes implanted into the peripheral nerve with relatively simple surgery, while CNS
recording requires craniotomy and is more complex with greater risk. In addition,
both movement commands and sensory information are transmitted in the peripheral
nerve and make closed-loop neural interface with sensory feedback possible.
There are three important research topics about PNS which will be presented in
this chapter. Sections 4.2 and 4.3 present the research about PNS decoding and
sensory feedback, respectively. Movement signals decoding and sensory signal
feedback construct a closed-loop system, while nerve regeneration and repair in
Sect. 4.4 provide the foundation for this closed-loop system. How to acquire periph-
eral neural signals, which were transmitted from the central nervous system, from
residual peripheral nerve will be introduced in Sect. 4.2; how to provide sensory
feedback by peripheral neural interface will be introduced in Sect. 4.3; some findings
in neural cell culture and artificial stretch will be presented in Sect. 4.4.

4.2 Peripheral Neural Signal Acquisition

PNS has captured wide attention from international research community. As early as
2007, Chen et al. of Zhongshan Hospital affiliated to Fudan University conducted an
experimental study on the control of electronic prostheses by the amputee for the first
time in China [6]. They inserted the intrafascicular electrodes directly into the three
main nerve bundles of the upper arm, and the electrodes were connected to a
prosthetic device at the distal end. It was found that the signal derived from the radial
nerve (simple motor nerve bundle) could trigger the prosthetic simulation device
reaching out the fingers when the patient imagined the movement of the hand, but the
fingers could not be closed. On the other hand, signals from other nerves (mixed
nerve bundles) couldn’t effectively drive the prosthetic device. This study validates
the possibility of using PNS to control prostheses in a further step; however, the
neural signals are only used as a control switch for prosthetics other than being fully
analyzed. In 2010, PM Rossini and S. Micera et al. implanted four intrafascicular
electrodes into the ulnar nerve and the median nerve of an amputee to perform a long
track recording [7]. Slight hold, tight hold, and flexion and extension of little thumb
are extracted from the recording signals to develop commands for the robotic arm’s
performing online. However, there is still a lack of research on the use of PNS to
control prosthesis to complete more complex reaching and grasping tasks. For
peripheral neural electrode, S. Micera et al. developed a variety of recording or
stimulation electrodes, such as cuff electrode, the flat interface nerve electrode
(FINE), longitudinal intrafascicular electrode (LIFE), etc. As to the PNS processing,
they applied wavelet denoising method for preprocessing and the support vector
machine (SVM) or artificial neural network (ANN) for classification corresponding to
different tasks. Clinical trials have been conducted to verify the feasibility of the PNS
on controlling the external machines. S. Micera pointed out that their future PNS
research would focus on decoding more control instructions from the electrode
4 Peripheral Neural Interface 95

recordings to achieve a higher degree of freedom for prosthesis control; higher user
autonomy; developing bio-PNS interface by providing the users with the sense of
feedback; and artificial limb control algorithm with higher robustness and
adaptability.
Overall, the earlier carried-out research on EMG and EEG identification and
analysis comes out with many practical applications. The modeling research on
neural signals in the cerebral cortex and the corresponding limb motor function
gained much progress, which provides lots of references for the study of PNS. In
recent years, PNS shows the unique advantages and great promotion potential in the
field of brain machine interface, thus attracting great interest from international
research groups. However, great breakthroughs are expected in the PNS identifica-
tion and other issues, as with the specialized analysis software and practical algo-
rithms. Although the use of PNS to control robots is already realized by some teams,
there is still a long walk to achieve flexible movement patterns with more degrees of
freedom by accurate control. Before applying the neural interface constructed by
PNS to clinical practice, a large number of animal studies are needed. Primates (such
as macaques) become ideal animal test subjects since the anatomical and physiolog-
ical functions of the nervous system as well as the motor system are the closest to
humans. Moreover, PNS animal experiments are related to some key techniques
such as the implantation of multichannel nerve electrodes, neural signal acquisition,
and monkeys’ training, which are challenges for the research institutions.

4.2.1 Surgery and Data Collection

There are mainly three peripheral nerves involved in the transmission of control
commands and sensory signals for upper limb, including the ulnar nerve, the radial
nerve, and the median nerve. In order to acquire meaningful neural information and
to perform electrical stimulation for sensory feedback, we implanted special
designed electrodes into these nerves of the monkey. All surgery and data recording
procedures were strictly in compliance with the guidelines for the care and use of
laboratory animals and were approved by the Institutional Animal Care and Use
Committee in Huazhong University of Science and Technology.

4.2.1.1 Electrode Implantation

The electrodes we used in this experiment included floating microelectrode arrays

(FMAs), which can be inserted into the nerve fasciculus and cuff electrode, which
wrapped the nerves. One of the key concerns for electrode implantation is to meet the
requirements for stable recording while allowing monkeys to perform various behavior
tasks. To prevent the monkey scratching the connectors or the wires, the connectors
must be mounted on its skull firmly. Since the electrodes were implanted into the nerves
in the arm, long wires had to be used for linking the electrodes and the connector. Here
96 P. Zhang et al.

Fig. 4.1 The CT image of the monkey with peripheral neural electrodes implantation. The
electrodes were implanted into the median nerve and ulnar nerve of the monkey and the electrodes
were connected to the connector with wire through a subcutaneous tunnel. The connector was fixed
on the skull with a custom-made pedestal

the wires we adopted in this experiment had a length of 30 cm, as shown in Fig. 4.1,
which were connected with the connector and electrode through a subcutaneous tunnel.
The surgery procedures are shown in Fig. 4.2. The first step was to incise the
muscles and expose the nerves. According to the relative positions of the nerves,
different nerves can be identified clearly, mainly were median nerve, ulnar nerve and
radial nerve. For FMA implantation, the array was held by an inserter, which can
generate negative pressure, and then inserted fast into the target nerve. For firm
fixing, a silicone film was used to wrap the array and sewed up with medical sutures.
The surgery was finished after sewing the cuttings, and the total surgery time
expended was about 3–5 h.
One FMA electrode and one cuff electrode were implanted in the median nerve
and ulnar nerve of a monkey. The two electrodes implantation was achieved through
a single surgical site at the elbow. The FMA electrode was inserted into the median
nerve with its pins and the cuff electrode wrapped on the ulnar nerve. Moreover, the
FMA electrode and that part of the nerve were wrapped by a silicone film.

4.2.1.2 Acute Electrode Implantation test

Movements like hand grasping can be generated by current pulse stimulation

on given electrodes after peripheral neural electrode implantation, as shown in
4 Peripheral Neural Interface 97

Fig. 4.2 Procedure of the surgery for peripheral electrode implantation. (A) is the flow chart of the
surgery. (B) shows the surgical stie and these three exposed nerves, median nerve, ulnar nerve and
radial nerve. (C) shows the implantation of the FMA electrode. (D) shows the implantation of the
cuff electrode

Fig. 4.3 [8]. The hand shape at static state, when no current pulses were delivered to
the electrodes, was shown in Fig. 4.3a. At the beginning phase of electrical stimu-
lation, clear movement tendency could be observed, as indicated in Fig. 4.3b, c. The
termination of a movement induced by current pulse was shown in Fig. 4.3d. The
experiments described here demonstrate the feasibility of recruiting motor units to
generate certain movements by electrical stimulation. However, the details about the
patterns of stimulation parameters require further researches.

4.2.2 Experiment

4.2.2.1 Experimental Protocols

When performing the tasks, the animal was comfortably seated in a primate chair
with its left arm restricted. The monkey was required to reach and grasp one of the
three target objects indicated by the target LED in each trial, as shown in Fig. 4.4.
For investigating the neuronal activity corresponding to different grasping gestures,
three types of target objects were used, including cylinder, plate, and ring. It is
obvious in Fig. 4.4b that the monkey adopted three kinds of hand shapes with large
discrepancy with regard to the three types of target objects.
98 P. Zhang et al.

Fig. 4.3 Stimulation of median nerve recruits specific digit and wrist movements. A microelec-
trode array was implanted into the median nerve of the monkey and pulse-trian was delivered by a
electrical stimulator (Plexstim, Plexon, Inc.), meanwhile sEMGs of the forearm were recorded by
the Delsys surface EMG system (Delsys, Inc.) to evaluate the electrical stimulation perfor-
mance. Hand grasping can be generated with specific electrical stimulation parameters and different
parameters can evoke different movements

The procedures of the task in Fig. 4.4c show a typical successful trial. One
successful experimental trial began with the center LED lighting (center on); then
the monkey rapidly put its hand on the center pad and kept for 500 ms at least. Then,
one of the three target LEDs was on (target on), cueing the monkey to release from
the center pad (center release) and reach to the target indicated by that LED. The
monkey should maintain a proper hand gesture according to the shape of the object
and made a powerful grasp so as to trigger the FSRs (target hit). After keeping a firm
grasping for about 200 ms, it was allowed to release the hand (target release) and
then received several drops of water as reward. After an inter-trial interval of about
5–10 s, the monkey returned its hand to the center pad and waited for the beginning
of the next trial.
The definition about several important periods during the tasks, are the baseline,
the reaction period, and the moving period. During the reaction period, the monkey
was in preparation for the future movements, but no movement occurred. During the
early stage of the moving period (0–200 ms), hand grasping gestures have not been
fully formed.
4 Peripheral Neural Interface 99

Fig. 4.4 Paradigm design of grasping multiple targets. (A) is the schematic diagram of monkey
behavioral experiment. The monkey was guided to reach and grasp three different target
objects. (B) is the pictures that the monkey was grasping these three target objects, cylinder,
plate, and ring. (C) shows the sequence of the behavioral task. Each trial began with a cueing of
center light on, instructing the monkey to fixate on the center pad. After a center holding time of
500ms, the center light went out meanwhile one arbitrary target light went on, cueing the monkey to
reach for the corresponding target and make a whole-hand grasp movement. After a target holding
time (THT), the target light went out. The monkey would return the hand and a reward of a few
drops of water would be delivered

4.2.2.2 Peripheral Neural Data

The acquisition and recordings of neural data were finished by the OmniPlex system
(Plexon, USA). The field potentials acquired by the cuff and the FMA on the median
nerve were shown in Figs. 4.5 and 4.6 correspondingly.

4.2.3 Neural Decoding Algorithms

Over the decades, various practical decoding algorithms have been developed to
analyze neural signals, with continuous improvement in their applicability to neural
experimental data (see [9, 10] for recent reviews). A typical Bayesian framework of
recursive estimation is introduced here [11], known as a Bayes filter, where the EMG
signals from each muscle and the limb kinematics from the lower limb movement
were regarded as hidden states of a dynamic network system, while neuronal spiking
signals were recorded from motor cortical cortex using the implanted microelectrode
arrays as the measurements of the dynamic system, in which the value updates
constantly (see [12] for more details about the experiment).
In our experimental data analysis, the xn denoted the hidden state x value at the n-
th time instant. In EMG signal decoding, xn is given by xn ¼ [u1(n), u2(n), . . .,
uM(n)]T, where um(n), m ¼ 1, . . ., M is the value of EMG signal from m-th muscle
100 P. Zhang et al.

0.5

-0.5

-1
0 2000 4000 6000 8000 10000 12000 14000 16000 18000

0.5

-0.5

-1
0 2000 4000 6000 8000 10000 12000 14000 16000

0.5

-0.5

-1
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
4
x 10

Fig. 4.5 The field potentials acquired by the cuff electrode

Fig. 4.6 The field potentials acquired by the FMA

4 Peripheral Neural Interface 101

activity at the time n and M denotes the total number of muscles where we implanted
EMG recording electrodes. When decoding lower limb kinematics, the xn given by
xn ¼ [Z(n), V(n), A(n)]T, where the Z(n), V(n), A(n) separately respect the position,
the velocity, and the acceleration of the marker at the time n. At the same time, the yn
denotes the constantly updated measurement values of the dynamic system at the
time n, which is a vector data containing the spike counts of neurons in the n-th
time bin.
The estimation process consists of two steps: (i) prediction and (ii) update. In the
prediction step, the prior distribution of xn is estimated from the previous states, then
use a Markov model to generate xn , so we have:
Z
pðxn jy1:n1 Þ ¼ pðxn jxn1 Þpðxn1 jyn1 Þdxn1 ð4:1Þ

where p(xn| xn  1) represents the state of dynamic system changed from current time
instant to the next moment and p(xn  1| yn  1) corresponds to the posterior
probability value for the previous states.
The update step, the posterior prediction p(xn| y1 : n) is calculated by updating the
prior prediction with the new measurement values yn:

pðyn jxn Þpðxn jy1:n1 Þ

pðxn jy1:n Þ ¼ ¼ αpðyn jxn Þpðxn jy1:n1 Þ ð4:2Þ
pðyn jy1:n1 Þ

where p(yn| xn) represents the mapping relation between the dynamic system and the
spike measurements and the normalization constant α in the formula can usually be
ignored. Utilizing Eqs. (4.1) and (4.2), we can calculate the p(xn| y1 : n) recursively.
For practical data process, there are two widely used algorithms, the Kalman filter
method [13] and the unscented Kalman filter method [14], that could be used to
estimate the EMG signals and kinematics recursively from the neuronal signals
recorded from motor cortex; for more detailed information, refer to [15].

4.3 Sensory Signal Feedback

For people who lost a limb after amputation, the concomitant loss of motor and
sensory functions results in enormous inconvenience in activities of daily living and
deterioration in quality of life [16–19]. To regain ability for grasping and manipu-
lating objects requires appropriate tactile information in addition to motor-driven
grippers [20–22]. To better restore amputees’ limb function and enable them to
better manipulate delicate objects in daily life, adequate sensory feedback is neces-
sary, especially tactile information.
Open-loop myoelectric control via surface EMG once was the most sophisticated
bio-control strategy clinically available since powered limb prostheses were
102 P. Zhang et al.

developed to functionally restore arm/hand function in 1960s. However, this kind of

control approach did not improve significantly grasping/manipulation due to lack of
intuitive user interaction and/or sensory feedback involved in task control [23]. In
order to achieve closed-loop control in grasping tasks, several feedback approaches
have been proposed in the development of neural interface. For example, force
feedback is often used in the hierarchical manner of shared-control for the manip-
ulation tasks involving grasp [24]. Visual and/or auditory feedback has frequently
been introduced to assist users in the control of prosthetic limb for reaching and
grasping [25]. For sensory substitution, vibrotactile or electrotactile stimulations
play a crucial role in providing sensory perception as feedback [26–29]. More
recently direct stimulation of the peripheral nerves becomes more focused research
corresponding to signals generated by the artificial sensor systems on the prosthetic
limb [30–32].
As the complexity of the prosthetic hardware advanced significantly over the last
decade, effective control strategies utilizing such high degrees of freedom in more
dexterous prosthetic limbs become notably inadequate. For example, the sophisti-
cated prosthesis, SmartHand [33–35], contains so many degrees of freedom, yet it is
difficult for a patient to control it naturally. Furthermore, versatile sensors on new
prostheses provide much valuable feedback information that substitute naturally the
missing sensory information, and if transmitted to patient in a more natural or
intuitive way [36], such feedback can be perceived as originating from the missing
limb while without overwhelming the concentration of the user [37].
Recently, a study [38] has presented an evident improvement in the quality of
sensation and showed that experience of stable touch perceptions can lead to more
precise and reliable control of the prosthetic limb. While too many afferents are
synchronously stimulated through these electrodes, it will be feasible to achieve
more selective stimulation in order to generate more naturalistic tactile sensations.
This is important to make the conversion of the prosthetic sensor outputs into
patterns of electrical stimulation on peripheral sensory nerves [39].
In this section, we describe a neural prosthesis based on electrical stimulation to
provide sensory feedback for closed-loop control. A Schunk Dexterous Hand (SDH)
with array sensors on every segment of its fingers provides sensory feedback through
a fully implantable electrical stimulation device, which can generate time-variant
stimulation parameters according to the sensor measurements and is promising for
the generation of natural, graded intensity of grasp force perception [38]. Particularly,
the implanted part of this stimulation device is wirelessly powered via magnetically
resonant coupling so that there’s no need for an internal battery [40].

4.3.1 Materials and Method

The prosthetic part of the experimental neural prosthetic system consists of a

6-degree-of-freedom (DoF) Schunk Powerball Lightweight Arm 4P (LWA 4P),
with a 7-DoF three-fingered SDH attached as its end effector. Each finger is
4 Peripheral Neural Interface 103

Fig. 4.7 The prosthesis is a combination of a LWA 4P and a SDH, which can provide sensory
feedback information as well as moving according to the commands delivered by the computer. The
stimulation device consists of the external controller, the implantable pulse generator (IPG), the lead
extension and electrode. (Adapted from Qiuxia Lai; Dingyin Hu; Ang Ke; Jiping He, Providing
Sensory Feedback Using Electrical Stimulation for Neural Prosthesis, Neuroscience and Biomed-
ical Engineering, 2014. Spring, 2: 99~104)

equipped with two tactile array sensors to provide primitive contact pressure distri-
bution as well as the synthetic force information for the calculation of the parameters
needed for electrical stimulation, namely, the pulse width, the pulse amplitude, and
the stimulation frequency. The robotic arm and hand communicate with a computer
through CAN (controller area network) bus, and the tactile sensors are connected
through serial port.
The stimulation system is based on wireless power transfer technology and
contains the external controller, the implantable pulse generator (IPG), the lead
extension, and electrode, as shown in Fig. 4.7. The controller receives stimulation
parameters from the computer via a USB port after the real-time processing of the
sensor data and then delivers the parameters to the IPG via wireless communication.
The IPG generates time-variant stimulation pulses and delivers them to the bipolar
electrode by the lead extension to elicit continuous “natural” sensations on the
human body according to the stimulation patterns. Due to its tiny size and wireless
power transfer this stimulation system is easily suited for implant operation.
We intend to provide sensory feedback signals that mimic the body’s original
biological sensors according to the tactile sensor data via the electrical stimulation
104 P. Zhang et al.

device. The improvement in quality of sensation and the closed-loop control strategy
is promising to lead to more precise, reliable, and natural control of the prosthetic
limbs in daily life tasks.

4.3.2 Experiment

At current preliminary stage, we conducted two separate experiments. One aimed at

collecting sensor data to be analyzed later for the generation of electrical stimulation
patterns, and the other focused on the functional testing of the stimulation device. In
the near future, the integrated complete experiment combining these two parts will
be conducted, first in animal models for safety and reliability verification before into
clinical trial having patients’ participation.

4.3.2.1 Sensor Data Acquisition

In daily life, drinking water from a bottle is a simple but frequent behavior, yet most
current prostheses cannot provide satisfactory assistance due to their innate shortage
in utilizing natural sensory information.
In order to study the sensory information and regulation in this daily life activity
as well as to find out a general research approach for similar issues, we conducted
the experiment to collect the outputs of these array sensors during the task
execution process when the prosthesis took up a bottle of water, moved to another
place, poured water out, held the bottle upright again, and then put down the
emptied bottle.
The recorded contact sensor data were visualized using pseudo-coloring scheme
to present intuitive information of the pressure distribution and its dynamic variation
during the experiment, as shown in Fig. 4.8. Detailed explanation is presented next.
In Fig. 4.8 four phases of the water bottle grasp and take-up, tilting for water-
pouring procedure in the experiment task, are shown on the top panel, and the
corresponding visualized contact pressure presentations are placed below. The
water level is indicated by dash line on the bottle. The meaning of pseudo-color
picture used to visualize the acquired contact pressure data is illustrated by the color
bar and the unit of sensor readouts is millivolt. Each of the three fingers contains two
tactile array sensors consisting of 78 and 86 tactile cells, respectively, and each cell
can perceive the pressure applied on it.
All sensor data were delivered to the computer and recorded for offline analyses.
In the pseudo-color picture, each figure under the representing bottle orientation
shows three dynamic pressure distributions for the three fingers: the left color
column represents the right-lower finger seen in the real object picture, the middle
column represents the left (opposing) finger, and the right represents the right-upper
finger. The primitive readouts are the voltages measured on each cell.
4 Peripheral Neural Interface 105

Fig. 4.8 Several typical time phases during the water-pouring procedure in the experiment task.
The top panel shows the real scenes and the bottom panel shows the corresponding pseudo-color
picture drawn according to the sensor readouts. The dash line indicates the water level inside the
bottle, while the water outside isn’t shown. The color bar demonstrates the meaning of the pseudo-
color picture. Each of the color square in the pseudo-color picture represents a cell in the array
sensor and its color demonstrates the voltage value which reflects the force strength exerted on it at
that time point. Qiuxia Lai; Dingyin Hu; Ang Ke; Jiping He, Providing Sensory Feedback Using
Electrical Stimulation for Neural Prosthesis, Neuroscience and Biomedical Engineering, 2014.
Spring, 2: 99~104

4.3.2.2 Stimulation Pattern Generation

The stimulation device was designed to be capable of altering stimulation parameters

in real time during its working procedure. The external controller receives com-
mands from a computer via a USB port connection [41]. Then the external controller
transmits the stimulation parameters to the IPG by radio frequency telemetry. The
IPG generates pulses through the electrode to elicit certain neural responses [42].
In order to prevent corrosion and irreversible electrolyte reactions, the waveforms
used in clinical stimulation should be charge-balanced, while compared with biphasic
charge-balanced waveforms, asymmetric charge-balanced waveforms can provide
better control of electrochemical reactions and may suppress undesired reactions [43].
To ensure charge balance at the electrode site, we monitored the raw output
signals applied to a pair of electrode contacts under different specified stimulation
parameters with the active electrodes immersed in 0.9% saline. Two waveforms
from each electrode contact form together and generate the final stimulation pulse, as
shown in Fig. 4.9.
We also altered the stimulation parameters during voltage observing procedure in
order to test the capability of our stimulation device to generate different stimulation
waveforms real time according to the computer commands. The voltage waveforms
were recorded via the oscilloscope, as well. Some examples are demonstrated in
Fig. 4.9. The stimulation parameters tested in the experiment were 50 Hz, 2.64 V,
and 1200 μs and 10 Hz, 1.32 V, and 400 μs, respectively.
106 P. Zhang et al.

Fig. 4.9 Voltage

waveforms observed on the
oscilloscope. The active
electrodes were immersed in
0.9% saline. The specified
stimulation parameters are
(a) 50 Hz, 2.64 V, 1200 μs
and (b) 10 Hz, 1.32 V,
400 μs, respectively

4.3.3 Results and Discussions

4.3.3.1 Sensor Data Analysis

In the sensor data acquisition experiment, we observed an evident variation of the

measured voltage. The pressure remained zero before the SDH touched the water
bottle, and as can be seen in Fig. 4.8, the pressure later reached its desired maximum
value when the fingers firmly held the bottle without squeezing hard to break it, and
4 Peripheral Neural Interface 107

Fig. 4.10 Average forces of the 6 array sensors averaged from 12 experiments. In the “finger xy,” x
with the value 0–2 represents the three fingers from left to right in the bottom panel of Fig. 4.8, and y
with value 1 represents the proximal part, and 2 represents the distal part of the finger

then decreased when water flowed out of the bottle. Finally, it returned to zero after
putting down the bottle. We can also find out that each array sensor experienced
different amounts of pressure, and this pressure distribution may result from the
relative position and contact area between the bottle and the SDH, i.e., the holding
posture affected the sensor output value. However, the variation rules of the pressure
values on all six array sensors are similar during the experiment.
The array sensor data are initially in the form of voltage, and we calibrated the
voltage into pressure using the formula:

p ¼ v P =V  : ð4:3Þ

where the P ¼ 0.000473 N/(mm  mm) and the V ¼ 592.1 mV according to the
documentation of SDH.
After acquiring the pressure value on each cell, the average force on every array
sensor can be calculated. The calculated force of the 6 array sensors averaged from
12 experiments is displayed in Fig. 4.10, and the time points corresponding to the
given phases in Fig. 4.8 are marked by the dash lines. In Fig. 4.10, the “finger xy”
indicates the six tactile array sensors embedded in the SDH, in which x ranging from
0 to 2 indicates the finger and y with values 1 and 2 indicates whether the sensor is
proximal or distal.
We can see that there exists high-frequency jiggering in the force curves, indi-
cating that the bottle slid a little during the whole experiment procedure. The distal
108 P. Zhang et al.

part of the right-lower finger and the proximal part of the right-upper finger,
i.e. “finger 01” and “finger 22,” have the smallest average force during the 12 exper-
iments, and as can be seen from the original data, the pressure of each cell stays 0 in
most of the 12 experiments, resulting in the equal-zero average values. The distal
parts of the left finger and the right-upper finger, i.e., “finger 11” and “finger 21,”
have the second largest average force, and the sum of the two values seems to remain
the same. The remaining two finger patches, i.e., “finger 02” and “finger 12,” have
the largest average force, which then diminished to a smaller value when water
started to flow out of the bottle. From the analysis above we can conclude that as for
the holding posture in the experiments, only four finger patches played the main role,
which form two interaction force pairs and keep the bottle balanced. When water
was flowing out of the bottle, the average force measured on “finger 02” and “finger
21” increased a little, which may be due to that the water at bottom of the bottle
flowed to the bottle neck. So there was more water need to be supported by these two
fingers at that moment.
However, the analysis above is only for the averaged results. As for the data
acquired in each single experiment, we discover that there often exist certain finger
patches that didn’t experience any pressure during the whole procedure. This
phenomenon may not be ideal for shared load of fingers, but it is not unusually
seen in real life because of the uneven surface or shape when grasping an object. This
observation suggests that, for larger, heavier or irregular shaped/structured objects,
grasping postures may need pre-planning to distribute pressure more evenly among
all fingers of a robotic hand to prevent slippery or damage of robot hand or weak
parts of the objects.
In further research, we plan to extract more information from these sensor data
to generate proper electrical stimulation parameters and elicit sensations in human
body real time. The type of stimulation represents the objects’ contact properties
detected by the array sensors on the robotic hand and is promising to contribute a
lot in the control of neural prosthesis because the sensory feedback information
provided can help to reach a closed-loop control effect and will greatly improve the
performance of the neural prosthesis when carrying out daily life tasks. What we
need to do next is to propose more effective algorithms and make better use of
these force data to acquire motor-related information and apply it to the controller
designing.

4.3.3.2 Stimulation Pattern Generation

In the stimulation pattern generation experiment, we observed a charge-balanced

waveform between the bipolar electrodes from oscilloscope. The active electrodes
were immersed in 0.9% saline and the raw output signals were monitored under
different specified stimulation parameters. As illustrated in Fig. 4.9, stimulation
parameters are 50 Hz, 2.64 V, and 1200 μs (a) and 10 Hz, 1.32 V, and 400 μs (b),
respectively. One channel of the electrode contact outputs a square pulse OUT1 and
the other electrode contact outputs a delayed square pulse OUT2 whose amplitude is
4 Peripheral Neural Interface 109

one-half and the duration is double of the OUT1 waveform. The charge-balanced
biphasic stimulation waveforms Vsim are formed by these two square pulses and
have a negative phase followed by a positive phase with an inter-pulse delay. We can
observe a significant difference when stimulation parameters changed.
We conclude that both changes in the pulse width and the stimulation frequency
can result in different stimulation waveforms, demonstrating the capability of the
stimulation device to provide different stimulation patterns for eliciting variable
sensations.
In future research, we will utilize the calculated force information to generate
corresponding stimulation parameters (pulse width, amplitude, and frequency) that
are promising to elicit natural sensory responses. The sensation elicited on sensory
nerves can provide sensory feedback different from other kinds of feedback such as
visual and vibratory feedback, and can induce users’ intuitive response in the control
of neural prosthesis, which is destined to realize a more natural control performance.
Currently the exact relationship between the stimulation parameters and the elicited
sensations is still unclear; we need to do further carefully designed research to
investigate and acquire this kind of information and find out the proper way to
provide users with natural sensations.
In this section, we report a novel experiment to measure the tactile sensor
information during the process of taking up a bottle or cup of water, moving to a
designated location while holding the bottle/cup upright, pouring the water out,
returning the bottle back to upright position again and then putting down the empty
bottle, and monitored the output stimulation waveforms of the stimulation device
under specific stimulation parameters. These are preliminary experiments to explore
approaches to restore the motor function for paralyzed people or amputees by
introducing sensory feedback via electrical stimulation on peripheral nerves and
controlling the motion of prostheses according to the motor intention decoded from
user consciousness.
We now have successful access to acquire the data used for sensory feedback as
well as the approach to generate various stimulation patterns; what we need to do in
successive stages include conducting more experiments combining these two sub-
systems and developing an effective algorithm to convert the output of tactile array
sensors on the prosthesis into patterns of electrical stimulation of sensory nerve.
Moreover, we will also do investigative work to use decoded motor intention into
real-time control commands to directly manipulate the prosthetic limbs as the natural
sensory feedback are elicited via electrical stimulation synchronically, thus reaching
a natural closed-loop control performance.
There are many scientific and technical challenges to be conquered before we can
achieve the ultimate research goal. Along the way, the development effort should
bring significant medical benefits to the patients and society. If the natural sensory
feedback is achieved, the amputees will be able to perform naturally functional
control on the prostheses as if they were just extension parts of their body.
110 P. Zhang et al.

4.4 Neural Regeneration and Repair

Autologous nerve grafts remain the “gold standard” for repair of peripheral nerve
lesions. However, this approach is plagued by permanent loss of harvested nerve
function and the potential formation of painful neuromas. Furthermore, there are
obvious limitations in the supply of donor nerves, making autografts can only be
used to repair short nerve lesions. Neural repair technology based on tissue engi-
neering has shown promising prospects in the treatment of nerve injury. However,
this treatment needs autologous nerve to stimulate axonal regeneration and extension
into target tissues. Therefore, how to produce plenty of living nerve constructs in a
short time is full of significance.
Fine-motor control of an artificial arm will ultimately require a different kind of
link between living tissue and the prosthesis. Except for muscles, nerve fibers can
also be the intermediary between the severed axons in a stump and the electrical
wiring of a prosthetic device. D. Kacy Cullen and Douglas H. Smith believed this
kind of neural interface has some advantages, such as low invasion, high sensitivity,
convenient for signal separation, and short rehabilitation period [44]. To create such
a neural bridge, we firstly have to figure out how to grow nerve fibers that are long
enough to span the gap between the host axons and the electronics.
Although it has been well known that axon growth can be guided by chemical
cues and electrical stimulation, both of these two methods couldn’t produce regular
neural tissues. In nature, the actual growth of the axon also appears to be dependent
on mechanical stimuli. By the proper stretch speed, we can achieve regular nerve
constructs. The nervous tissue constructs consisting of living axons, which provides
an enticing target for host axon ingrowth and synaptic integration. Compared with
original neural interface, our method shows some advantages. Firstly, living axonal
constructs can provide regenerative support for the proximal stump where target is
cut off. Secondly, the array and neurons can make stable connection in vitro.
Thirdly, most host nerves will never connect directly with the electrical devices
which usually cause damage to the respective tissue. Finally, the process of mechan-
ical stretch can produce more regular nerve fascicles (details in Table 4.2).

Table 4.2 Comparison of different methods for the directed nerve growth
Methods Advantages Disadvantages
Chemical The inherent characteristics of the organisms Long period
The growth speed and direction are controllable Unquantifiable
Both direction: Attraction and rejection Irregular
Kinds of neurotrophic factors for varied nerves
Electrical The growth speed and direction are controllable Unquantifiable
Noninvasive Irregular
Neural regulation and functional reconstruction
Mechanical The inherent characteristics of the organisms Easily cause nerve damage
The growth speed and direction are controllable
Regular nerve fibers Few studies
Quantifiable
4 Peripheral Neural Interface 111

4.4.1 Axon Stretch Growth

In the past, a few reported studies have attempted gradually lengthening the axons by
towing growth cone. In 1984, Bray successfully stretched the growth cones of
individual chick sensory axons more than 100 μm within a few hours by a micro-
electrode [45]. In 1993, Heidemann et al. performed a series of studies. The growth
cones of single axons were lengthened at precise increments via stretching glass
pipettes using an external device. The rate of towed growth showed a linear
dependence of growth rate with applied tensions between 25 and 560 micro dynes.
He found the axons could be towed at a maximal limit of 1 mm every day without
thinning down [46]. In 1977, S. Chada found the elongation rate and magnitude of
applied tension have linear relationship in both central and peripheral neurons [47].
When the stretch time is too long or the speed too fast, the axon will be
disconnected. Therefore, we can’t produce long neural tissues by stretch growth
cone. Actually, integrated axon tracts without growth cones could also undergo a
form of more rapid and sustained growth under stretch. The majority of cell types in
nature increase their volumes by division during development. However, neurons
are non-mitotic, so the expansion of nervous tissue depends on another process. For
example, during development of blue whale, the spinal axons need to undergo
continuous mechanical tension as their spinal cord grows in length. Although
without growth cones, these integrated axons could reach 30 m in the end.
However, the elongation of integrated axon tracts was not experimentally dem-
onstrated until 2001 [47]. Smith et al. found integrated axon tracts without growth
cones could be stretched at a faster rate. Two large populations of neurons were
seeded on adjacent membranes within a bioreactor, and then the top membrane was
slided across the bottom membrane at set rates by a programmable microstepper
motor system. When the microstepper motor moved the top membrane, the axonal
bundles crossing between the two membranes were stretched to a new length. In the
end, the axons had grown up to 1 cm after 10 days of stretch. In the following years,
they optimized the stretch growth paradigm, and dorsal root ganglion (DRG) axon
tracts initially only 100 μm in length could be extended up to 10 cm by stretch for
2 weeks [48]. Despite this rapid stretching, sodium channel activation, inactivation,
and recovery or potassium channel activation were demonstrated no change
according to the whole cell patch clamp technique.

4.4.1.1 Principles of Axon Stretch Growth

In order to stretch axon bundles, as depicted in Fig. 4.11, one stationary substrate
membrane as the bottom of the culture chamber. Another towing membrane made by
the same material was positioned over the bottom membrane. Both of the two
membranes would need to be thin and transparent enough so that adherent cells on
their surfaces could still be within the working distance of a high magnification
microscope objective. The towing membrane was fixed to a machined plastic block
112 P. Zhang et al.

Fig. 4.11 Schematic diagram of axon stretch-growth

so that it could be moved over the bottom membrane by a microstepper motor

system. First, the neurons were placed on the two adjoining substrates and formed
new synaptic connections between each other over 5 days. Then, the axon bundles
across the border between the top and bottom membranes were stretched in a
stepwise fashion. After several days’ stretch, the axon tracts could reach several
centimeters in length.

4.4.1.2 Fabrication of Bioreactor

In order to stretch neural cells, axon expansion chamber will not only need to culture
cells like a common culture vessel, but also need to stretch axons. The design needs
to meet the following requirements:
1. Conditions for culture of neural cells in vitro. A. It can store and replace culture
medium. B. It can be sterilized and maintained in a sterile environment. C. To
exchange gas and maintain appropriate pH value medium. D. The material used in
the incubator has no toxic gas under special temperature and humidity conditions.
E. The bottom and top are transparent to facilitate observation during culture.
4 Peripheral Neural Interface 113

Fig. 4.12 Control structure diagram of axon stretch system

Fig. 4.13 Axon stretch growth bioreactor system. (Adapted from Li, Xiao, et al. “Development of a
new miniaturized bioreactor for axon stretch growth.” Journal of integrative neuroscience 15.03
(2016): 365–380)

2. Conditions for axon stretch growth. A. The materials used under special temper-
ature and humidity conditions of the incubator will not deform and ensure the
stability and accuracy during stretch growth. B. Stepper motor needs to work well
in the incubator; each displacement distance is small enough to not affect the cell
growth. C. The wire passes through the incubator, but does not destroy the air
pressure and the sterile environment in the incubator.
A miniaturized bioreactor (Fig. 4.12) was engineered to gradually apply tension
to axon bundles spanning two separate membranes. As showed in Fig. 4.13, the axon
114 P. Zhang et al.

Fig. 4.14 Axon expansion chamber

stretch growth bioreactor system was composed of axon expansion chamber, linear
motion table, microstepper motor, and controller. Aside from the motor indexer, the
bioreactor was located in a CO2 incubator during neural culture and axon stretch
growth experiments. The axon expansion chamber was seated within the chassis
underlying the linear motion table, meanwhile the towing rod extending from the
axon expansion chamber was fastened to the table using an acrylonitrile-butadiene-
styrene (ABS) adaptor.
As is shown in Fig. 4.14, the axon expansion chamber is composed of the
stretching frame, towing block, towing leg, and so on.

4.4.2 Axon Stretch Growth

When the axons sprouted from their cell body and formed synaptic connections with
target cells, the axon stretch growth will begin. As the distance between the neuronal
somata and the target cells increased, the axons will undergo the continuous tension.
These forces increased axons’ growth rate by stimulating axons to add cytoskeleton,
axolemma, and other cellular building materials somewhere along their central
length to minimize strain.

4.4.2.1 Key Factors in Axon Stretch Growth

1. Traction Mode.
Small and frequent stretches are important to the survival of axons undergoing
stretch growth, particularly at the start of stretching. As shown in Fig. 4.15, if high
elongation rates are applied early, growth cannot be sustained, and axons will
rupture within the first 24 h. But if the rate increases slowly, the cell can
accommodate the rate and grow healthily. Moreover, if the stretch rate increased
4 Peripheral Neural Interface 115

Fig. 4.15 Graphic

representation of stretching
conditions that define the
boundaries of axon growth
or disconnection

faster than the neuron, the tension will accumulate and cause the axons rupture.
Therefore, two sequential factors that defined the boundaries of long-term stretch
growth of integrated axons: strain and acclimation.
2. Substrate Attachment.
Another key factor is the neuronal adhesion with the substrate. The elongator sub-
strates coated by both PDL and rattail collagen had been proved it was the best
coating method. PLL had been widely used in the cell culture experiment. However,
it could be taken in through adsorptive endocytosis by neurons which may provoke
inflammatory responses either directly or indirectly through its necrosis-inducing
abilities, most of axons had not grown out from the DRGs. Although collagen could
provide an optimal condition for cell adhesion, it will rehydrate in the culture
medium, which would often cause axon stretch growth experiments to fail. The
axons grew very well after 3 days’ culture, while they were seemed to be ill at the
sixth day. High-molecular-weight PDL could provide enough adhesive force during
axon stretch growth, but some residual PDL could not be digested by lysosome
which would cause inflammatory responses when the nervous tissue constructs
comprised of stretch-grown axons were used to repair nerve damage. By combining
soluble PDL and collagen, we can get the most ideal result.
116 P. Zhang et al.

3. Slope Substrate.
In 1911, Harrison cultured cells on a spider web and found the cells grew along the
fibers of the web [49]. Since then, with the development of micro- and nano-
fabrication techniques, a large number of studies have shown that cells react strongly
to topography. Rajnicek et al. reported that central nervous system neurites could be
guided by shallow grooves with 14 nm deep and 1 mm wide [50]. Stepien grew
chicken DRG neurons on single scratches. He found that direction often increases
with increasing depth, but decreases with increasing groove width. If the grooves are
>20 mm, most cell types will lose guide [51]. Goldner et al. describe an unusual
capability of a subpopulation of DRG neurons could span across the grooves with
200 mm width, even with no underlying solid support [52]. Fricke et al. designed a
variety of gradient patterns with slight changes in slope to control axon directional-
ity. They found that reduction in the slope of structure from 0.04 (0.3 mm/7.5 mm) to
0.01 (0.1 mm/7.5 mm) strongly influence neurite growth. Actin and microtubules
align along walls and edges, the microtubules being the first element to be aligned,
followed by actin. Grooved surfaces also induce changes in transcription and the up
and down regulation of several genes, but the explicit mechanism for cell guidance
has yet to be clarified [53].
In the axon stretch growth system, there is a topography between the top
membrane and the bottom membrane, which will affect the growth direction of the
axons. The top membrane need to be polished on one edge to create gradual slope to
the border of the exposed underlying membrane.

4.4.2.2 Experimental Design of Axon Stretch Growth

1. Modification of the Membranes.

DRG were isolated from 1 day-old infant Sprague–Dawley rats. The culture surfaces
were respectively treated with high-molecular-weight PDL and collagen.
2. The Process of Axon Stretch Growth.
The neurons are placed on the two adjoining substrates and formed new synaptic
connections between each other over 5 days. When the axon bundles are across the
border between the top and bottom membranes, we can begin to stretch. The
following axon stretch growth began at a rate of 0.864 mm/day by taking 1-μm
steps every 100 s over 864 iterations. Alternatively, in order to achieve several
centimeters’ axon tracts rapidly, the stretch rate could begin slowly and gradually
accelerate to the desired rate. As is shown in the Fig. 4.16, the important steps are as
follows:
4 Peripheral Neural Interface 117

Fig. 4.16 Flowchart of

axon stretch growth
118 P. Zhang et al.

1. Traction condition determination: a large number of axon bundles are across the
border between the top and bottom membranes.
2. Install axon expansion chamber: fasten the culture chamber within the chassis
underlying the linear motion table, whereas the towing rods extending from the
culture chambers were fastened to the table using an ABS adaptor.
3. Set parameters: first of all, in the PC machine, set the traction parameters such as:
step length and waiting time, and then downloaded to the controller, you can use
the keyboard to achieve the stepper motor to start, pause, positive rotation,
negative rotation and other operations.
4. Medium replacement: after 1 week’s culture, replace the medium. When the
liquid is replaced, the utility model is firstly suspended by the keyboard, and
the towing rod is fixed by the nut to prevent the towing rod from moving. After
that, loosen the nut between the towing rod and the connecting block, and move
axon expansion chamber into the super clean table. Finally, the axon expansion
chamber is moved back to the incubator, and the towing rod is fixed with nuts.
Connect the block, and then loosen the nut on the pull link, you can restart
the pull.

4.4.2.3 Axon Stretch Growth

In order to increase the density of elongating axons in each culture, the elongator
substrates were first treated by PDL and then followed by rattail collagen coating.
After DRG explants were plated, cultures were maintained for 10 days in the
incubator so that axons could grow across the interface of the two adjoining sub-
strates. Before stretching, the total diameter of the axon bundles was about 100 μm,
with about 100 axons (Fig. 4.17b). According to the above test, 1 μm step was the
better choice. In addition, the axons need sufficient acclimation time to decrease the
residual stress after stretching. Therefore, the stretch growth rate was programmed
into the motion control system by implementing a displacement step and a resting
time in a circular manner, about 1 μm every 100 s. These bridging axons adapted to
the stretch by increasing their length from 500 μm to become 5.94 mm long over
7 days of stretch growth (Fig. 4.17h). Because the neighboring axon bundles
preferred to join together during stretch-induced growth, the number of axon bundles
in the end looked fewer. The neural tracts were stained with β-tubulin antibodies to
confirm which were real axons. The entire length of stretch-growing axons labeled
positive for β-tubulin protein (Fig. 4.18).
Our work has obtained some successful experience in verifying that mechanical
stretch could promote the directional growth of axons, and demonstrating the
possibility of repairing nerve injury. Some recent reports highlight the necessary
of synergistic training for axons and myelin. This will be our next research goal.
4 Peripheral Neural Interface 119

Fig. 4.17 Comparison of DRGs static culture group (left) and stretch-grown group (right) in the
axon stretch growth bioreactor. (a) DRGs were cultured 3 days. (b) DRGs were cultured 10 days.
(c) DRGs were cultured 10 days. (d) DRGs were stretched growth 3 days. (e) DRGs were cultured
13 days. (f) DRGs were stretched growth 5 days. (g) DRGs were cultured 17 days. (h) DRGs were
stretched growth 7 days. Scale bars: (a)–(c), (e), and (g) 200 μm; (d), (f), and (h) 500 μm. (Adapted
from Li, Xiao, et al. “Development of a new miniaturized bioreactor for axon stretch growth.”
Journal of integrative neuroscience 15.03 (2016): 365–380)
120 P. Zhang et al.

Fig. 4.18 Immunohistochemistry analysis of axon. (a) Antibodies against β-tubulin (red) were
utilized to determine that elongating processes were axons. (b) Antibodies against myelin basic
protein (green) were utilized to determine whether there was myelin around the axons. (c) Cell
nuclei were labelled with DAPI (blue). (e) overlay. Scale bars: 100 μm. (Adapted from Li, Xiao,
et al. “Development of a new miniaturized bioreactor for axon stretch growth.” Journal of integra-
tive neuroscience 15.03 (2016): 365–380)

References

1. Liu H et al (2008) Multisensory five-finger dexterous hand: the DLR/HIT hand II. IEEE/RSJ
international conference on intelligent robots and systems, 22–26 September 2008, Nice, France
2. Micera S, Carpaneto J, Raspopovic S (2010) Control of hand prostheses using peripheral
information. IEEE Rev Biomed Eng 3:48–68
3. Reuderink B, Poel M, Nijholt A (2011) The impact of loss of control on movement BCIs. IEEE
Trans Neural Syst Rehabil Eng 19(6):628–637
4. McFarland DJ, Wolpaw JR (2008) Brain-computer Interface operation of robotic and prosthetic
devices. Computer 41(10):52–56
5. Dhillon GS et al (2005) Effects of short-term training on sensory and motor function in severed
nerves of long-term human amputees. J Neurophysiol 93(5):2625–2633
6. Xiaofeng Jia, Matthew A. Koenig, Xiaowen Zhang, Jian Zhang, Tongyi Chen, Zhongwei Chen
(2007) Residual motor signal in long-term human severed peripheral nerves and feasibility of
neural signal-controlled artificial limb. J Hand Surg Am 32(5):657–666
7. Rossini PM et al (2010) Double nerve intraneural interface implant on a human amputee for
robotic hand control. Clin Neurophysiol 121(5):777–783
8. Ledbetter NM et al (2013) Intrafascicular stimulation of monkey arm nerves evokes coordinated
grasp and sensory responses. J Neurophysiol 109(2):580–590
9. Quian Quiroga R, Panzeri S (2009) Extracting information from neuronal populations: infor-
mation theory and decoding approaches. Nat Rev Neurosci 10(3):173–185
4 Peripheral Neural Interface 121

10. Rolls ET, Treves A (2011) The neuronal encoding of information in the brain. Prog Neurobiol
95(3):448–490
11. Bergman N (1999) Recursive Bayesian estimation, Doctoral dissertation, Department of Elec-
trical Engineering, Linköping University
12. Chaolin Ma XM, Zhang H, Jiang X, He J (2015) Neuronal representation of stand and squat in
the primary motor cortex of monkeys. Behav Brain Funct 11:15
13. Grewal MS (2011) Kalman filtering. Springer, Berlin
14. Julier SJ, Uhlmann JK, Durrant-Whyte HF (1995) A new approach for filtering nonlinear
systems. In: Proceedings of the American control conference, 21–23 June 1995, Seattle, WA,
USA
15. Ma X et al (2017) Decoding lower limb muscle activity and kinematics from cortical neural
spike trains during monkey performing stand and squat movements. Front Neurosci 11:44
16. Antfolk C et al (2013) Sensory feedback in upper limb prosthetics. Expert Rev Med Devices 10
(1):45–54
17. Deans SA, McFadyen AK, Rowe PJ (2008) Physical activity and quality of life: a study of a
lower-limb amputee population. Prosthetics Orthot Int 32(2):186–200
18. Whitehouse F, Jurgensen C, Block MA (1968) The later life of the diabetic amputee: another
look at fate of the second leg. Diabetes 17(8):520–521
19. Østlie K, Magnus P, Skjeldal OH, Garfelt B, Tambs K (2011) Mental health and satisfaction
with life among upper limb amputees: a Norwegian population-based survey comparing adult
acquired major upper limb amputees with a control group. Disabil Rehabil 33:17–18, 1594–
1607. https://doi.org/10.3109/09638288.2010.540293
20. Johansson RS, Flanagan JR (2009) Coding and use of tactile signals from the fingertips in object
manipulation tasks. Nat Rev Neurosci 10(5):345–359
21. Campbell CS et al (1999) What you feel must be what you see: adding tactile feedback to the
trackpoint. In: Proceedings of IFIP INTERACT’99, p 383–390
22. Mugge W et al (2009) Sensory weighting of force and position feedback in human motor
control tasks. J Neurosci 29(17):5476–5482
23. Ortiz-Catalan M, Hakansson B, Branemark R (2014) An osseointegrated human-machine
gateway for long-term sensory feedback and motor control of artificial limbs. Sci Transl Med
6(257):257re6
24. Cipriani C et al (2008) On the shared control of an EMG-controlled prosthetic hand: analysis of
user-prosthesis interaction. IEEE Trans Robot 24(1):170–184
25. Childress DS (1980) Closed-loop control in prosthetic systems: historical perspective. Ann
Biomed Eng 8(4–6):293–303
26. Pylatiuk C et al (2004) Progress in the development of a multifunctional hand prosthesis.
Conference on engineering in medicine and biology society, 1–5 September 2004, San
Francisco, CA, USA
27. Sasaki Y, Nakayama Y, Yoshida M (2002) Sensory feedback system using interferential current
for EMG prosthetic hand. Conference proceedings of second joint Embs-Bmes conference,
23–26 October 2002, Houston, TX, USA
28. Kaczmarek KA et al (1991) Electrotactile and vibrotactile displays for sensory substitution
systems. IEEE Trans Biomed Eng 38(1):1–16
29. Riso RR, Ignagni AR, Keith MW (1991) Cognitive feedback for use with FES upper extremity
neuroprostheses. IEEE Trans Biomed Eng 38(1):29–38
30. Navarro X et al (2005) A critical review of interfaces with the peripheral nervous system for the
control of neuroprostheses and hybrid bionic systems. J Peripher Nerv Syst 10(3):229–258
31. Micera S et al (2006) Hybrid bionic systems for the replacement of hand function. Proc IEEE 94
(9):1752–1762
32. Dhillon GS, Horch KW (2005) Direct neural sensory feedback and control of a prosthetic arm.
IEEE Trans Neural Syst Rehabil Eng 13(4):468–472
33. Cipriani C, Controzzi M, Carrozza MC (2011) The SmartHand transradial prosthesis. J
Neuroeng Rehabil 8(1):29
122 P. Zhang et al.

34. Cipriani C, Controzzi M, Carrozza MC (2010) Objectives, criteria and methods for the design of
the SmartHand transradial prosthesis. Robotica 28(06):919–927
35. Antfolk C et al (2010) SmartHand tactile display: a new concept for providing sensory feedback
in hand prostheses. J Plast Surg Hand Surg 44(1):50–53
36. Micera S, Carpaneto J, Raspopovic S (2010) Control of hand prostheses using peripheral
information. IEEE Rev Biomed Eng 3:48–68
37. Almström C, Herberts P, Körner L (1981) Experience with Swedish multifunctional prosthetic
hands controlled by pattern recognition of multiple myoelectric signals. Int Orthop 5(1):15–21
38. Tan DW et al (2014) A neural interface provides long-term stable natural touch perception. Sci
Transl Med 6(257):257ra138
39. Talan J (2014) Neural Interface activates a sense of touch in prosthetic limbs. Neurol Today 14
(21):10–11
40. Xu Q et al (2013) A novel mat-based system for position-varying wireless power transfer to
biomedical implants. IEEE Trans Magn 49(8):4774–4779
41. Zhou H et al (2012) A fully implanted programmable stimulator based on wireless communi-
cation for epidural spinal cord stimulation in rats. J Neurosci Methods 204(2):341–348
42. Xu Q et al (2011) A fully implantable stimulator with wireless power and data transmission for
experimental use in epidural spinal cord stimulation. Annual international conference of the Ieee
engineering in medicine and biology society, 30 August to 3 September 2011, Boston, MA,
USA
43. Qi X et al (2005) A versatile microprocessor-based multichannel stimulator for experimental
use in epidural spinal cord stimulation. In: IEEE proceedings of first international conference on
neural Interface and control, 28–26 May 2005, Wuhan, China
44. Cullen DK, Smith DH (2013) Bionic connections. Sci Am 308(1):52–57
45. Bray D (1984) Axonal growth in response to experimentally applied mechanical tension. Dev
Biol 102(2):379–389
46. Heidemann SR, Buxbaum RE (1994) Mechanical tension as a regulator of axonal development.
Neurotoxicology 13(1):95–107
47. Smith DH, Smith DH, Wolf JA, Meaney DF (2004) A new strategy to produce sustained growth
of central nervous system axons: continuous mechanical tension. Tissue Eng 7(2):131–139
48. Pfister BJ et al (2004) Extreme stretch growth of integrated axons. J Neurosci 24
(36):7978–7983
49. Harrison RG (1911) On the stereotropism of embryonic cells. Science 34(870):279–281
50. Rajnicek A, Britland S, McCaig C (1997) Contact guidance of CNS neurites on grooved quartz:
influence of groove dimensions, neuronal age and cell type. J Cell Sci 110(23):2905–2913
51. StEpieŃ E, Stanisz J, Korohoda W (1999) Contact guidance of chick embryo neurons on single
scratches in glass and on underlying aligned human skin fibroblasts. Cell Biol Int 23
(2):105–116
52. Goldner JS et al (2006) Neurite bridging across micropatterned grooves. Biomaterials 27
(3):460–472
53. Fricke R et al (2011) Axon guidance of rat cortical neurons by microcontact printed gradients.
Biomaterials 32(8):2070–2076
Chapter 5
Brain-Machine Interface-Based Rat-Robot
Behavior Control

Jiacheng Zhang, Kedi Xu, Shaomin Zhang, Yueming Wang,

Nenggan Zheng, Gang Pan, Weidong Chen, Zhaohui Wu,
and Xiaoxiang Zheng

Abstract Brain-machine interface (BMI) provides a bidirectional pathway between

the brain and external facilities. The machine-to-brain pathway makes it possible to
send artificial information back into the biological brain, interfering neural activities
and generating sensations. The idea of the BMI-assisted bio-robotic animal system is
accomplished by stimulations on specific sites of the nervous system. With the
technology of BMI, animals’ locomotion behavior can be precisely controlled as
robots, which made the animal turning into bio-robot. In this chapter, we reviewed
our lab works focused on rat-robot navigation. The principles of rat-robot system

J. Zhang · K. Xu (*) · S. Zhang · X. Zheng

Qiushi Academy for Advanced Studies (QAAS), Zhejiang University, Hangzhou,
People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education Ministry,
Zhejiang University, Hangzhou, People’s Republic of China
Zhejiang Provincial Key Laboratory of Cardio-Cerebral Vascular Detection Technology and
Medicinal Effectiveness Appraisal, Zhejiang University, Hangzhou, People’s Republic of China
e-mail: [email protected]
Y. Wang · N. Zheng
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University, Hangzhou,
People’s Republic of China
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China
G. Pan · Z. Wu
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China
W. Chen
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University, Hangzhou,
People’s Republic of China
Department of Biomedical Engineering, Key Laboratory of Ministry of Education Ministry,
Zhejiang University, Hangzhou, People’s Republic of China
College of Computer Science and Technology, Zhejiang University, Hangzhou,
People’s Republic of China

© Springer Nature Singapore Pte Ltd. 2019 123

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_5
124 J. Zhang et al.

have been briefly described first, including the target brain sites chosen for locomo-
tion control and the design of remote control system. Some methodological advances
made by optogenetic technologies for better modulation control have then been
introduced. Besides, we also introduced our implementation of “mind-controlled”
rat navigation system. Moreover, we have presented our efforts made on combining
biological intelligence with artificial intelligence, with developments of automatic
control and training system assisted with images or voices inputs. We concluded this
chapter by discussing further developments to acquire environmental information as
well as promising applications with write-in BMIs.

Keywords Brain-machine interface · Rat-robot · Behavior control · Brain

stimulation

5.1 Introduction

Brain-machine interface (BMI) is a bidirectional pathway between the brain and

external facilities. Besides its ability to read out information from the brain, provid-
ing an available communication method for the disabled, the BMI also has further
applications in “writing” information into the brain. With encoded stimulations,
researchers can send artificial information into targeted brain area, interfering neural
activities and even generating sensations.
The using of physical stimulation to affect the brain function, mainly by applying
electrical current into the brain, has been applied for clinical treatments for more than
a century. Since electrical stimulation targeted at motor cortex can elicit limb
movements [1], it was well adopted for improving neurosurgical procedures from
early times [2]. More recently, deep brain stimulation (DBS) has become a common
therapy to alleviate symptoms of some brain disorders such as chronic pain [3, 4] and
the Parkinson’s disease [5, 6]. Besides, electrical stimulation was also introduced as
a tool for regenerating sensory feedback. Cochlear implants [7, 8] and retinal
prostheses [9, 10] are two of such applications, replacing the impaired peripheral
nerves with artificial devices. Moreover, recent works have shown that intracortical
micro-stimulation in primary somatosensory system can generate discriminable
tactile percepts from lab animals [11, 12] to human beings [13]. These approaches
draw bright prospects of the “write-in” BMI applications.
However, it is not until the mid-1990s that this therapeutic stimulation application
had found a new potential in the field of BMIs, when the concept of “bio-robot” first
appeared. The idea of the BMI-assisted robotic animal system is accomplished by
stimulations on specific sites of the nervous system. Studies in functional mecha-
nisms of the nervous system gave rise to the potential of this animal locomotion
control system with focal brain stimulations. Taking advantages of the artificial-
evoked locomotion changes, researchers attempted to control animals with virtual
perception cues elicited by electrical stimulations. Compared to traditional robot or
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 125

biomimetic robot, robotic animals have shown superior performances due to their
flexibility, self-sufficient energy supply, and adaptability to complex environments.
One of the earliest attempts on direct animal controlling was made by a Japanese
group. In their experiment, an American cockroach was successfully controlled to
follow the forward and/or turning commands with electrical stimulations on the
antenna [14]. Since the advent of the remarkable work, the researches on robotic
animals, or bio-robots, have been developing rapidly in the following decades.
Winged insects were one of the most interesting subjects. Artificial stimulation-
controlled insects were thought to be an alternative to the insect-mimetic micro-air
vehicles (MAVs) for their fascinating flight performances. An expanded variety of
insects have been engaged in this cyborg insects study, including months [15, 16],
beetles [17], and honeybees [18]. By implanting microelectrodes into antenna and
optic lobes, the insects can be controlled to fly according to preset routes or even
switch the wing oscillations.
Another hot spot in the field of bio-robot is behavior control of vertebrates [19–
22]. Compared to insects, vertebrates have much more complicated brain structural
and functional divisions and thus can be controlled to finish more complex tasks.
Among various animals, rodents are always elusive with high agility, indicating the
possibility of utilizing them for investigations and rescues. Furthermore, the rodents
have an acute sense of smell, which exceeds most of the man-made sensors in both
sensitivity and accuracy, making it a promising subject for explosive or drug
detections. However, these small mammals are hard to be tamed with traditional
training protocols. With the help of electrical stimulations, locomotion commands
can be directly “written” into the animal’s brain, saving training time and simplify-
ing training paradigms at the same time. The initial idea of rat-robot control was
accomplished by evoking virtual reward stimulation and whisker touching feelings
to indicate moving forward or provide turning cues, respectively. Encouraged by this
work, some other groups have made advances in control improvements.
In this chapter, we firstly introduced how rat-robot be controlled with write-in
BMI. We then described some methodological advances developed for better loco-
motion modulation, such as optogenetic modulation methods. We also introduced
our implementation of combining “write-in” BMI with “readout” BMI, as mind-
controlled rat navigation system. Besides, we presented our efforts made on opti-
mizing control strategies for automatic control and combining machine intelligence
with biological intelligence for cognition enhancements. We concluded this chapter
by discussing further developments to acquire environment information as well as
promising applications with write-in BMIs.

5.2 Basis of Rat Navigation System

In 2002, Dr. Chapin and his colleagues managed to navigate rats through a complex
field with electrical stimuli instead of external turning cues and rewards used in
operant learning paradigms [23]. To accomplish the locomotion control, the rats
126 J. Zhang et al.

Fig. 5.1 Overview of the rat navigation system. The whole system is consisted of three parts:
animal preparation, stimulator backpack, and remote control system. (a) Target brain sites for rat
locomotion control, including turning cues (whisker barrel field/ventral posterior medial nucleus of
thalamus, indicated in black dots and red circles, respectively), reward stimulation (medial forebrain
bundle), and force holding (dorsolateral periaqueductal gray). (b) The stimulation backpack
equipped with a video camera, which turning the received commands into electrical stimulations
into corresponding brain sites and relaying captured videos back to operators at the same time. (c)
GUI for operators to adjust stimulation parameters and deliver locomotion commands. The rat-robot
can be controlled with a wireless mouse, and the information is exchanged wirelessly with
Bluetooth

were firstly implanted with bipolar microelectrodes. After a period of training for the
rats to correlate the electrical stimulation commands and corresponding behaviors,
the rat’s locomotor behavior can be controlled by the operators. In this model,
electrical stimulations in different sites of the brain evoked different behavioral
responses in rats, enabling accurate movement control in real time. Besides stimu-
lation electrodes implantation, a backpack containing microprocessor and micro-
stimulator needs to be mounted onto the rat for electrical stimuli delivering.
The commands are then transferred wirelessly between computer and backpack.
Figure 5.1 shows an overview of the whole rat-robot navigation system.

5.2.1 Principles of Rat-Robot Control

The BMI-assisted rat-robot locomotion control is mainly managed by evoking

sensations and perceptions that lead to behavior changes caused by habits or
reflections rather than directly activating targeted muscles. For movement control
strategy design, the most classical method is to use electrical stimulation-evoked
reward mechanism as an alternative for operant conditioning training. Other
methods based on fear or punishment were also brought up as training-free solutions
for rat-robot system improvement.
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 127

5.2.1.1 Virtual Reward-Based Training Method

To navigate a robot, at least two commands are needed for basic movement control:
moving forward and turning left or right. Similarly, for typical rat-robot navigation
control system, electrical stimulations are delivered into medial forebrain bundle
(MFB) and somatosensory cortex (barrel field, S1BF) as virtual reward and turning
cues, respectively, for movement control.
Medial forebrain bundle (MFB) has been proved to be involved in study and
reward mechanisms. Electrical stimulation of MFB not only reinforces forward
locomotion but also motivates further locomotion. Thus, MFB stimulation can be
served as a virtual reward cue for rats to move forwards [24]. The effects of MFB
stimulation can be tested and evaluated by behavioral tasks. Two typical behavioral
designs are free-roaming task and lever-pressing task. The free-roaming task takes
place in circular field or eight-arm maze. Rats receiving electrode implantation in
MFB are allowed free roaming in the field. MFB stimulation with appropriate
intensity will increase long-term locomotor activity of rats, inducing the rats to
approach toward centered regions more often. In lever-pressing task, the rats are
trained for a self-stimulation reward task. Once the lever is pressed by rat, a train of
MFB stimulation will be delivered immediately to the rat as virtual reward. In the
beginning of the first training session, the rats may touch the lever accidentally. After
several tries, the rats learn the rule of self-reward stimulation and start pressing the
lever frequently for reward seeking. After several days of reward training, the rats
will show active movements with MFB reward stimulation.
The turning cues are induced by stimulating bilateral somatosensory cortex. Rat’s
somatosensory cortex barrel field (S1BF) is related to whisker sensing process.
Stimulations in S1BF produce virtual whisker “touch” perceptions, inducing head
movements and turning behaviors [25]. With this strategy, the rats need to receive a
reinforcement training section of about 5–7 days. During this section, the animals are
trained to link electrical cues with appropriate turning directions. At the beginning of
each training session, the rats are motivated to keep running forwards in an eight-arm
maze with MFB stimulations. Either left or right turning cues will be given when the
rats entering the central area of the maze. Once the animal turns correctly after
turning cue delivers, the rat will receive a train of MFB stimulation as reward. By
associating MFB rewards to each correct turn, the rats can learn to perform highly
accurate turning behavior following electrical stimulations. Generally, the initial
turning directions are not uniform across untrained rats. While some animals
would turn ipsilateral to the stimulated site, others show contralateral turning
behavior. As the training procedure associated turning cues with rewards, the rats
could finally turn to consistent direction according to turning instructions. Particu-
larly, the turning direction is largely depended on the association of MFB rewards,
especially in those initial trainings. When MFB rewards are given upon ipsilateral
turns, the rat could learn to turn ipsilaterally to the S1BF stimulation, and vice
versa [26].
128 J. Zhang et al.

5.2.1.2 Optimization of the Turning Behavior

With the combination of MFB reward and S1BF turning cues, the rats can be
navigated over three-dimensional structures by operators. Motivated by MFB
rewards, the animals are able to run forwards and even climb or descend steps.
However, the control of turning behavior of the reported rat-robot exhibits a large
variability in the results produced. Although the training procedure ensures a high
success rate of turning behavior in rats after corresponding stimulation cue, the
trained rats are observed to have no response to the electrical stimuli occasionally.
Recent studies suggested that deep brain stimulation in thalamus might elicit more
stable sensations in animals [27]. In rat’s whisker sensing pathway, deflections of
whiskers first activate neural responses in the brainstem. The information is then
transferred into ventral posteromedial (VPM) nuclei of thalamus before it finally
projected to the somatosensory cortex (S1BF).
We have tested the effects of VPM stimulation for turning behavior control
[28]. The behavioral study showed robust turning reactions following VPM stimu-
lation without training (see Fig. 5.2a). Parameter modulation tests display a near-
linear relationship between the turning angle of the rat and stimulation intensity,
suggesting the possibility of quantitative control of turning behavior [29]. A later
micro-PET imaging study has shown that stimulation in VPM induces significant
glucose uptake changes in S1 [30]. Further optical imaging studies display a larger
activated area in S1BF with stimulation in VPM thalamus rather than directly in
S1BF area (see Fig. 5.3), suggesting an amplifying effect through thalamocortical
pathway. These behavioral and imaging results indicated that VPM stimulation may
provide a more efficient alternative method for turning control in rat-robot system.

Fig. 5.2 Performances of turning behavior with VPM stimulations. (a) Turning accuracies calcu-
lated from both S1BF stimulation group and VPM stimulation group. VPM stimulation resulted in
highly successful turning behavior without training. (b) The effect of stimulation strength on rat’s
turning angle. With the increase of stimulation voltage, the turning angle of rat showed near-linear
trends. (a and b are modified from [28], Table 1 and Fig. 2A, respectively.)
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 129

Fig. 5.3 Optical imaging responses to electrical stimulations. (a and b) Intrinsic optical imaging
responses to S1BF stimulation and VPM stimulation, respectively. The darkened area indicated
cortical activation spread. After a 500 ms stimulation (between 0.5 s and 1.0 s illustrated), the
cortical area imaged over time responded differently to the two stimulation methods. The
red-colored areas in the vessel maps illustrated significance (p < 0.01, t-test) between stimulation
trials and control trials. P posterior, L lateral. (c and d) The corresponding response magnitudes as
function of distance from the arrowed ROIs (regions of interest). S1BF stimulation resulted in a
quicker drop of response magnitudes at distant spots. (Student’s t-test, , p < 0.05; , p < 0.01).
(b and d are modified from [29], Fig. 2.)

5.2.1.3 Punishment-Driven Behavior Control

With a combination of forward and turning commands, the rat can be controlled to
move in a complex environment. Unlikely to traditional mechanical robots, one of
the limitations of this intelligent robot is that the rats can hardly stay still without
commands. Therefore, it is important to find a way to keep the rat immobile. One of
the solutions is to utilize the distinct feel of fear and defense. Researches have shown
that electrical stimulation in dorsolateral periaqueductal gray (dlPAG) induces
defensive behaviors [31]. With the increase of stimulation strength, the animals
show series of behavioral changes from alertness to freezing and finally escape
[32]. In order to maintain immobility, stimuli should be carefully controlled in a
certain range. We have tested the impacts of stimulation parameters and found that
variation of both stimulation frequency and pulse train duration is efficient in
130 J. Zhang et al.

Fig. 5.4 Freezing behavior evoked by dlPAG stimulations. (a) Rat’s freezing time changes resulted
from different stimulation inter-pulse intervals (IPI). Rats showed longer freezing time with shorter
pulse intervals and even start escape at the inter-pulse interval of 15 ms. (b) Reward stimulation
effects on freezing behavior evoked by dlPAG stimulations. The MFB reward can remarkably
reduce the freezing time durations at varying combination of inter-pulse interval (IPI) and burst
width (BW). , P < 0.01 different from dlPAG stimulation and dlPAG +MFB stimulation; +,
P < 0.01 different from dlPAG+5 s MFB stimulation and dlPAG +8 s MFB stimulation. (Modified
from [29], Figs. 1 and 3, respectively)

modulation of freezing time of the animal [33]. Furthermore, since MFB stimulation
has reward effects, an MFB stimulation following dlPAG stimulation can remark-
ably reduce the freezing time, bringing the animal back from the holding state more
quickly (as illustrated in Fig. 5.4b). Hence, by stimulating dlPAG or MFB, the rat’s
behavior can be switched between hold still and move on, like a real robot.
This “fear-caused” freezing reaction is more instinctive and can be evoked
directly without any training. Similarly, a new method was proposed as a “virtual
punishment”-driven solution to rat-robot navigation control [34]. This study inves-
tigated the responses of rats with stimulations in amygdale nucleus (AMY) and
thalamic ventral posterolateral (VPL) as the alternatives to forward and turning cues,
respectively. These two sites are involved in nociception or fear/anxiety information
transmitting, providing a potential for learning-free rat-robot developments. More
lately, another team also reported a direct turning behavior control method accom-
plished by stimulating nigrostriatal pathway [35]. This newly adopted brain site
exhibits an immediate contralateral turning behavior in response to a single
stimulation.

5.2.2 The Remote Control System

During locomotion control of rat-robot, cable connections for stimulation delivery

may be a major limitation for animal’s movements while roaming in large area.
Thus, telemetry systems have been designed to overcome this problem. Feng et al.
have designed a remote control stimulation system for controlling free-roaming
animals [26]. The control system includes a remote-controlled micro-stimulator, a
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 131

control program, and the Bluetooth modules for wireless communication between
stimulator and control program. The control program firstly initiates the hardware
and opens a protocol file for a specific rat. When user orders an instruction such as
move forward or turn left/right, the program sends commands to the transmitter with
preset stimulation parameters. The commands are finally translated by the receiver,
and corresponding stimulations are delivered into the corresponding brain sites
through electrodes implanted.
The stimulator is designed to be mounted on the rats for commands receiving
and stimulation delivering. To fit small animals, the remote-controlled micro-
stimulator should be kept small and light. At this point, one of the main concerns
is the power consumption of the micro-stimulator. Thanks to the development of
microelectronics, microprogrammed control unit (MCU) with high speed and low
power consumption can be adopted as the main processor for control of receiving
commands and delivering stimulations. The whole backpack mounted on con-
trolled rat contains a receiver and a stimulator and weights only 20 g, with a total
size of 36 mm  22 mm  15 mm. The stimulation commands received are
translated by the MCU and then electrical pulses are delivered into corresponding
sites of the rat’s brain. Generally, the electrical stimuli are trains of biphasic,
charge-balanced pulse trains. The stimuli delivered are effective with either con-
stant voltage or constant current.
The control program is designed for navigation orders delivering as well as
stimulation parameter setting. A basic instruction set for rat-robot navigation
includes “Forward”, “Left”, and “Right”, corresponding to the virtual reward and
turning cues. For some cases, the “Stop” command can be added for the holding
behavior control. With preset communication protocols, the instructions made by
operators can then be transmitted to the stimulator and be finally translated into
electrical stimulation pulses. For convenience, a wireless mouse or other portable
devices can be employed as remote controller, providing a real mobile solution to
working in complicated circumstances. As to optimize the animal locomotion
control, the stimulation parameters are adjusted individually and can be saved and
retrieved for reuse or edition in file management panel.
A Bluetooth module is applied for wireless communication in this system. The
module can work properly within a range of 100 m with communication speed up to
70 kb/s, which is even sufficient for multi-channel bidirectional communication.
Therefore, an improved version has been developed realizing a bidirectional telem-
etry system with both electrical stimulation and signal recording for free-roaming
rats [36]. In this version, besides producing four-channel stimulus, the system can
also acquire two-channel neural signals in real time. The recording module is
capable of recording both neural action potentials (APs) and local field potentials
(LFPs). This bidirectional communication system takes step towards the final goal of
developing a closed-loop BMI. By analyzing neural signals in real time, stimulations
can be triggered when necessary, forming a closed-loop brain-machine interface.
Such closed-loop system is flexible in stimulation time, providing better treatment
effects in clinical therapy with deep brain stimulation (DBS).
132 J. Zhang et al.

5.2.3 Rat Navigation Guided by Remote Control

After a few days of adaptation and training, the rats can learn to link the S1BF
turning cues with MFB stimulation rewards, showing the feasibility of electrical
stimulation-evoked “virtual” learning. With proper combinations of these two stim-
ulation commands, the rats can be well steered towards targeted locations by
operators. Notably, during navigation control, the MFB stimulation not simply
acts as a reward feedback whenever the rat responding to turning cue correctly but
also serves as a driving force to keep the rats moving continuously. In fact, frequent
MFB stimulations during rats’ locomotion reinforce their motivation of moving
forward. Free-roaming rats with MFB stimulations show an increased locomotion
activity and take more approaches toward the central area of the field. Therefore,
with the help of this “virtual reward” stimulation, the rats can be induced to climb or
descend from ladders/stairs and steep ramps or even jump off platforms.
Together with turning cues, well-trained rat-robots can then be adapted to com-
plex three-dimensional obstacle courses quickly. As shown in Fig. 5.5, with the

Fig. 5.5 An example of guided rat navigation using brain micro-stimulation. Well-trained rat-robot
can be guided to pass through slalom courses, climb steep ramps, and even hold still for 3–5 s at
target spots. (From [33], Fig. 4.)
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 133

“holding” command of dPAG stimulation, the rats can even be controlled to stay still
for several seconds at any locations that be instructed to. Moreover, we have
managed to guide the rats in some real conditions, such as passing through offices
and corridors, crossing narrow pipelines and clumps of grass, and circumventing
obstacles. While configuring the rat’s backpack with a wireless micro-camera, the
environmental images can be captured in real time, enabling the operators to
navigate the rat-robots at beyond-visual ranges.

5.3 Novel Rat-Robot Control Method Based

on Optogenetics

Optogenetics is one of the newly developed neural modulation techniques in recent

years. By genetically expressing light-sensitive channel protein on specific type of
neural population, targeted opsin-transduced neurons can be activated via optical
stimulations with certain wavelengths. Compared with traditional electrical stimu-
lation, optogenetic technique has the advantage of activating specific types of
neurons, thus narrowing the activated range to desired areas. Besides, it also pro-
vides an opportunity to record neural electrophysiological activities during stimula-
tion. For precise locomotion control, we developed a rat-robot system based on
optical modulation.

5.3.1 Model of Optogenetics-Based Rat-Robot

The basic idea of optogenetic technique is to utilize the light-sensitive membrane ion
channels to selectively activate neural populations. Different opsin channels can be
activated with stimulation lights at different wavelengths, enabling either excitation
or inhibition of the selected neural populations. Neural modulation by optogenetic
technique requires transduction of light-sensitive opsin, optical stimulus with spe-
cific wavelengths, and proper stimulation intensities. In our current optical modu-
lated rat-robot model, opsin gene of channelrhodopsin-2 (ChR2) is chosen for
exciting neural populations. This opsin carried by adeno-associated virus (AAV) is
transduced into excitatory neurons under the guidance of calcium-calmodulin-
dependent kinase II type-FC; (CaMKIIFC;) promoters. After 3–4 weeks of recovery
and ChR2 expression, the targeted areas in the rat’s brain can then be activated by
delivering blue light at a wavelength of 473 nm.
Similar to electrical stimulation control, optogenetic modulation of locomotion is
also based on evoking virtual reward and defense behavior. For the “holding”
behavioral modulation, optical stimulation is targeted at dPAG, the same brain
area as used in electrical stimulation model. However, as to reward delivery, MFB
134 J. Zhang et al.

Fig. 5.6 Setup of the optogenetic model. (a) A typical view of the ChR2-mCherry expression on
the neurons expressing CaMKIIα in and around the region of VTA (ventral tegmental area). An
enlarged region of interest (ROI, squared in a) is illustrated in (b). (c) ChR2-transduced rat with
implanted devices for optogenetic modulation. (d) The implanted device (“optrodes”) with both
optogenetic stimulations and electrophysiological recordings. (e) Enlargement of the electrode tips.
The centered guide cannula is prepared for virus injection and fiber implantation, and the
surrounded electrodes are for neural signal recording, with two different depths. (a and b are
modified from [37], and Fig. 1. (c–e) is modified from [38], Figs. 1 and 3.)

is not an ideal target for optogenetic model since this bypassing axon bundle is
lacking of cell bodies to be transduced with opsin genes and activated by light.
Actually, MFB is one of the projections from the ventral tegmental area (VTA) with
ventral striatum. VTA is believed to be involved in natural reward circuitry and
motor activity, as well as drug addictions, providing an ideal site for opsin gene
expression.
Successful expression of opsin genes can be verified by histological studies. The
ChR2-mCherry expression traces can be observed on frozen brain slices with
fluorescent images taken under a microscope. Figure 5.6a and b shows a typical
view of the ChR2-mCherry expression image captured with fluorescent microscope,
revealing a high density of transduced neural populations in the targeted brain
regions with normal cell morphologies [37].
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 135

5.3.2 Optical Stimulation and Electrophysiology

Similar to electrical stimulations, during optical stimulations, the light intensity is of

crucial importance. Previous researches showed that light intensity lower than the
threshold of 1 mW
mm 2 can hardly activate ChR2-transduced neurons [39, 40],
while overlarge stimulation intensity may damage the brain tissue around the
stimulation site. Therefore, to activate enough range of brain tissue safely and
efficiently, the appropriate stimulation intensity needs to be carefully calculated.
Besides, due to diffusion and scattering, the light transmission distance in brain
tissue is limited. According to the Kubelka-Munk model for diffuse scattering media
[39], the light transmission fraction of an ideal model expresses relationships with
both light penetration in the brain and scatter coefficient of penetration distance. For
rats we have used in rat-robot control model, the appropriate distance for light
delivery over the targeted brain area is 0.5 mm [38].
Besides the main advantage of activating neural populations selectively and
precisely, the optical stimulation also enjoys the benefit of being free from stimula-
tion artifacts. We have designed a multi-electrode array coupled with fiberoptic for
simultaneous optical stimulation and electrophysiological recording [38]. As shown
in Fig. 5.6c–e, this integrated optrode array device is consisted of a centered fiber
guide cannula and 16 surrounding recording electrodes with 2 layers for different
recording depths. In vivo electrophysiological recording demonstrates that optical
stimulation can successfully evoke an increased neuronal activity with time speci-
ficity. Figure 5.7a and b illustrates the electrophysiological responses to optical
stimulation [37]. During optical stimulation, neural spike firing activities show
obvious increases in recorded area. At the end of optical stimulation, neural activities
drop back to normal background level. Similar changes are also observed in local
field potential signals, where the magnitude recorded increases during optical stim-
ulation period. With the help of simultaneous electrophysiological recording, we
have further tested the effects of stimulation parameters. As shown in Fig. 5.7c,
optical stimulation with low frequency (5 Hz) can hardly induce significant increase
in neuronal activities. While stimulated with higher frequencies, the spike firing rates
recorded display significant increases from background level. The spike firing rate
changes can be modulated by both frequency and stimulation intensity, showing
increasing firing activities with higher frequency or intensity.

5.3.3 Behavioral Modulation of Optogenetic Rat-Robots

Optical stimulation with proper parameters also exhibits stable behavioral modula-
tion effects on optogenetic rats. Optical stimulation in dPAG area can induce typical
freezing and escape responses in rats, similar to that observed on electrically
induced behavior change of rats. Track navigation tests showed precise freezing
136 J. Zhang et al.

Fig. 5.7 Optogenetic modulation effects on rat-robots. (a and b) Typical in vivo spike and LFP
signals recorded from the implanted optrode device, respectively. The blue bars indicate the onset of
stimulation light, a train of 1.0 s pulses, with 15 ms pulse width and 50 Hz frequency. (c) The optical
modulation effects on electrophysiological recordings. Both stimulation intensity and frequency
have modulation effects on spiking firing rates (, p < 0.05; , p < 0.01; one-way ANOVA,
between stimulation and control group). (d and e) The modulation effects on rat holding behaviors.
The optical simulations have obvious and stable modulation impacts on rat’s behaviors at different
mark points. (a and b are modified from [37], Fig. 1c is modified from [41], Fig. 4, and (d) and (f)
are modified from [42], Fig. 1.)

behavior modulation both temporally and spatially. The optogenetic rats can be
modulated to hold on any location along the trace, with short response delays
(to start freezing) as well as recovery delays (to end freezing after stimulation)
[42]. In accordance with electrophysiological recordings, stimulations with higher
frequency display better modulation performances on both success rate and response
delays (shown in Fig. 5.7d and e). Compared with electrical stimulation, this optical
stimulation in dPAG induces more stable and moderate defensive responses,
exhibiting shorter recovery periods after stimulation [41]. These results show the
superiority of optogenetics over the traditional electrical stimulations.
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 137

As to stimulation in VTA, the optogenetic rats exhibit conditionings on reward

learning. With optical stimulation, the optogenetic rats exhibit an increased, long-
term locomotion activity in a free-roaming task. Further self-reward lever-pressing
task shows an ascending tendency of lever presses over training sessions, demon-
strating the “virtual reward” effect induced by optical stimulation in VTA neurons
[37]. These results indicate that optical stimulation in VTA is an alternative way to
induce reward-seeking behavior in rats rather than electrical stimulation in MFB.
Taken together, the technique of optogenetics provides a novel method to regulate
rat’s locomotion switching between move and stop more precisely than traditional
electrical stimulations both spatially and temporally.

5.4 Rat-Robot Navigation System Controlled by Human

Brain

As mentioned above, the BMIs are bidirectional communication pathways which

are able to modulate brain activities as well as to interpret thoughts in living brains
through computers. Therefore, by combining the two directional BMI pathways, a
“Brain-to-Brain Interface” (BBI) can be established where neural activities of two
or more individual brains can be linked by external devices [43]. Experiments have
proved that BBIs have the ability to translate locomotor intentions in real-time
among brains [44–48] and even across species [49]. The application of BBI may
augment the mutual coupling of brains, coupling neural processes of one brain to
another, and thus may have a positive impact on human social behavior. Another
potential application of BBI system is to implement locomotor control of cyborgs
directly with human thoughts. Zhang’s group has demonstrated a cyborg cock-
roach navigation system controlled by steady-state visual evoked potential
(SSVEP) with a noninvasive BMI device. Since the SSVEP-based BMI decoding
relies on visual stimulations, the operator’s attention may be distracted from
cyborg’s location feedback by real-time images. To overcome this shortage, we
have explored the utilization of motor imagery as an alternative solution. Here, we
will briefly introduce our efforts made on motor imagery-based “mind control” of
our rat-robots system [50].

5.4.1 Configuration of the BBI System

The BBI system is comprised of an electroencephalography (EEG)-based noninva-

sive BMI device, a host computer, the rat-robot navigation control system, and a
video camera-based visual feedback system. The EEG-based BMI device is applied
for motor imagery signal acquisition from human operators. The signals are sent to
the host computer through Bluetooth, where motor intentions are recognized and
138 J. Zhang et al.

Fig. 5.8 The framework of mind-controlled rat-robot system. (From [50], Fig.1)

classified. The decoding results of motor imagery signals are then translated into
locomotion control commands of the rat-robots. With the help of the remote control
system, the navigation commands from the operators can finally be applied on rats
through stimulator backpack.
To form a closed-loop control system, we have chosen video information as
feedback signals. The visual feedback information is provided by a bird’s-eye
camera, with which the entire experimental field is supervised. The recorded images
are sent back to an LCD screen and presented in front of the operator; thus the rat’s
location can be traced by an operator in real time. Figure 5.8 shows the framework of
a complete BBI system.

5.4.2 Implementation of Motor Imagery-Based BMI

Different from typical manual control model, the mind control signals need to be
detected and deciphered before controlling stimulations delivered to rat-robot. The
operator’s “thoughts” are recorded by noninvasive EEG devices. In our current
BBI system, the Emotiv EPOC Neuroheadset (Emotiv Company) [51] is employed
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 139

for EEG signal recording, providing up to 14 recording channels and wireless

information transmission in real time. The neural signal obtained is transmitted to
the host computer for further processing. As the SSVEP signals are liable to
distract the operator’s attention from the visual feedback of rat-robot’s locomotion
states, we tried a combination of nictation signals and motor imagery signals as
mind control signal sources, where the nictation signal is used for reward control
and motor imagery as turning cues. The eye blinks are detected by amplitude
changes from raw data recorded with the frontopolar channel of the Emotiv device.
And the turning cues are decoded from motor intent of left and/or right arm
movement of the operator. This imagination signal requires the operators to
modulate their oscillation rhythms of sensorimotor cortex into the upper mu
frequency band (10–14 Hz). The power spectrum of the imagination raw data is
calculated as the main character for motor intent decoding. After several sessions
of signal decoding training, motor intents can be detected and evaluated by
classification algorithms.
To eliminate the noises from real motor intent signals, thresholds are set as
criteria for corresponding control stimulation delivering. Decoding results are
detected as real intentions only when the resulting values are above the thresholds,
while subliminal ones are considered as noises. However, the threshold needs to be
determined carefully, since higher thresholds can efficiently reduce false alarm rate
while increasing the miss rate at the same time. During rat-robot navigation
control, either the increasing of fake control stimulations or the missing of key
turning commands may cause failures in rat’s locomotion control, resulting in
lowered control efficiency. One of the feasible alternatives is to use changing
gradients as the evaluation criteria. Recorded data shows that compared to back-
ground noises, the real intention signals display a larger gradient at the beginning.
By calculating the changing gradients of the values decoded, the control efficiency
can be significantly improved while ensuring a low false alarm rate as well.
Open-field tasks show that the system performs a mean delay of 158 ms
between the detection of motor intents and the delivering of electrical stimulations,
which is sufficient for stable control of rat’s locomotion. Further eight-armed maze
tasks also show an overall success rate above 80%. These control results suggest
that our BBI system realizes a high-performance mind-controlled rat-robot
navigation model.

5.5 Optimized Solutions for Control Strategy

In previous sections we have demonstrated a navigation system manipulated by

human operators with hand or even by the mind. However, for mass preparations,
human training seems to be inefficient, since experiences and concentrations are
required for the training sessions. Besides, lack of information feedback is another
main drawback of current rat-robot model. These restrictions hold the rat-robot
technique back in demonstration stage rather than real practical applications. One
140 J. Zhang et al.

of the solutions is to employ artificial intelligence as controlling assistant. In this

section, we will demonstrate our attempts made on automatic control systems and
other artificial equipment assisted control strategies.

5.5.1 Automatic Control System Designed for Rat Navigation

Machine learning gives computers artificial intelligence in many aspects. With the
help of this technology, robots can be controlled automatically by computers. Thus,
it is natural to expect an automatic controlled rat-robot system. However, unlike
traditional mechanical robots, there are several challenges of controlling these living
creatures. Self-consciousness and biological intelligence are the major differences
between bio-robots and traditional robots. To rat-robots, stimulations are more like
“cues” to induce proper movements rather than force driven, resulting in uncer-
tainties of reactions to the same command. It makes the decision logic hard to be
modeled with explicitness and preciseness. In addition, the body of rat-robot is
variable among individuals and not rigid while moving, making it difficult to extract
locomotion information of the animal. Without precise locomotion parameters, it is
impossible to evaluate the feedback of control command and thus difficult to make
the following decisions.
To solve the problem of locomotion parameters, an extra bird’s-eye camera is
employed to supervise the entire experimental scene for real-time movement capture
of the rat. The rat’s locomotion information including body position and head
orientation is tracked and extracted with imaging processing techniques [52]. A
decision algorithm based on state machines can then be used to choose proper
command to induce rat-robot to accomplish a preset navigation task. The schematics
of the navigation system are illustrated in Fig. 5.9.
As to individual variations, we borrow experiences from human navigating
operations and propose a new method to learn and imitate this process [53]. Different
from the traditional methods that try to describe controlling logic explicitly, the new
model regards the navigation as a whole procedure and learns to make controlling
decision based on the operations made by human operators. In this model, a General
Regression Neural Network (GRNN) is applied as the mathematical model for
control decision learning. A total of two steps are included in this model. During
the first training stage, the control commands made by human operators and the
corresponding locomotion information of rat-robot (including body position and
head orientation) are extracted and synchronized as the labels and inputs to the
model. During this short training session, the GRNN configures the network with the
instances. After the training sessions, the GRNN model is able to generate control-
ling instructions automatically and thereby takes place of human operators to
navigate the rat-robot. This newly proposed method displays comparable
performance to human operators with high accuracy and reasonable stimulation
instructions quantities, indicating that the basic locomotion as position and
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 141

Fig. 5.9 The framework of automatic navigation system for rat-robot. (From [54], Fig. 5)

orientation can provide enough evidences for controlling decision-making [54]. This
conclusion is essential for further studies of bio-robots.

5.5.2 A Reward-Driven Control System

The control strategy mentioned above mainly focuses on simplifying the decision
logic by mimic the controlling principles conducted by human operators as a whole.
In this process, three directional commands are needed to be classified from loco-
motion information. In fact, thanks to its biological intelligence, the rat-robots have
shown the capabilities of reward seeking during daily training. It is well known that
rats have remarkable ability in navigation [55]. In spatial learning studies, the rats are
trained to seek for rewards and correct their behaviors whenever the rewards are
deprived [56]. Therefore, it is feasible to use single reward command to guide the
rat-robot, further reducing the decision-making procedure of the automatic control-
ling strategy.
According to this idea, we have designed a new automatic controlling strategy by
applying reward-only commands [57]. This method simplifies the automatic navi-
gation algorithm largely by taking full advantage of the rats’ learning capabilities. A
schematic diagram of control strategy is illustrated in Fig. 5.10. Before the naviga-
tion test, the destination of the task is preset, and the corresponding navigation route
is yielded by path planning algorithm. During navigation task, the rat-robot receives
rewards frequently when it performs ideal behaviors as walking along the right route.
If it walks into incorrect direction, the rewards are cut off as a cue. The rat-robot
142 J. Zhang et al.

Fig. 5.10 Schematic of the reward-only automatic control model for rat-robot navigation. The
green grids indicate current correct locations of the rat, the yellow grids denote the ideal route
toward destinations, and the red ones represent the wrong route. (a) The rat-robot enters the crossing
area of the maze following reward stimulation without turning cues. (b) The rat turns into one of the
arms at its own will. As entering a wrong grid, no reward is given, and the ideal route is
re-computed. (c) The rat turns back seeking for reward and enters the correct direction. As the
route is reset, the rat gets reward again. (d) The rat moves forward following the reward stimulation
toward the targeted end. (From [57], Fig. 1)

would realize its error and correct to proper direction for rewards. After several days
of training with the automatic control system, the rat-robots show high performances
in simple T-maze tasks, with their turning directional preferences dropping along
with the training session. These results indicate a successful attempt at combining
biological intelligence with artificial intelligence, providing new ideas for further
studies on bio-robot controlling and intelligence hybrid.
Furthermore, to enhance the capability of exploring un-configured environment,
we propose another reinforcement learning process-involved graded reward-
generating algorithm [58]. Q-learning procedure, one of the classical reinforcement
learning methods, is introduced in this method, to generate an incremental sequence
of electrical reward according to the distance from the preset destination. During
navigation task, the rat-robot receives higher reward stimulations when it approaches
closer to the given destination. After the initial trial-and-error test, the animal learns
to seek the maximum reward spots in an unknown environment. Since the rat has
excellent spatial recognition, the rat-robot and the reinforcement learning algorithm
can convergent to an optimal route by co-learning. This work has significant
inspiration for the practical development of bio-robot navigation with hybrid
intelligence.
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 143

5.5.3 Cognition Enhancement of Rat-Robotics with Visual/

Audio Cue

Another advantage of hybrid intelligence is that we can balance biological intelli-

gence with artificial intelligence. Utilizing of animals instead of mechanics over-
comes difficulties faced by traditional robotics such as motion flexibility and
stability, and meanwhile, external devices can be a compensation for rat’s own
limitations. For example, it is broadly known that rats communicate with ultrasound
band (higher than 20 kHz) [55], which is far beyond human speech wavelength
range (300–3400 Hz). It seems impossible for rats to understand or even hear voice
commands from human beings. Nevertheless, with the help of speech recognition
module, computers can directly translate vocal navigation commands into electrical
stimulations without hand operations [59].
In addition, when the rats are equipped with sensory devices, environmental
information can be acquired and analyzed by computers remotely. This equipment
upgrade may encourage the rat-robot to move further into larger areas. Considering
the poor vision of rats, one of the most intuitive approaches is taking images. With
miniature cameras mounted on backpack, environmental imaging information can
be captured and transferred back to a host computer. Operators can then guide the
rat-robot from far beyond visual range. Furthermore, with the technology of image
recognition, objects of interest can be detected efficiently by algorithms. The object
detection results can in turn be taken as guidance on adjusting the moving directions
of rat-robot toward the object. Incorporated with object detection methods, this
closed-loop model can accomplish beyond visual range control of rat robotics
automatically, which is of great significance for remote searching tasks [60].

5.6 Conclusions and Future Perspectives

For decades, great progress has been made in locomotion control of rat-robots.
Commands “written” into the rat’s brain have become more and more efficient.
The main idea of locomotion control is to use electrical stimulation to mimic natural
perceptions and feelings that result in corresponding behaviors of rats. By carefully
selecting stimulation sites and parameters, rats can be guided in complex environ-
ments. Integrated with artificial intelligence, rats can even be controlled automati-
cally by computers to reach a preset destination. On the contrary, there are also many
limitations to what has so far been accomplished with rat navigation system. The
major problem is the lack of feedback information of the environment surroundings.
Without efficient information feedback, operators can hardly control the rats to avoid
obstacles or adjust heading direction towards targets. Video cameras with image
stabilization technique mentioned in Sect. 5.2 are one of the solutions. However, the
angle of view restricts the ranges of information obtainment. Besides, weights of
bear load also limit quantity and dimension of equipment that can be carried by rats.
144 J. Zhang et al.

In actual situations, odor information is always of great importance. In fact, rodents

have excellent sense of smells. Given their small sizes and light body weights,
rodents are more suitable for demining than traditional canine. APOPO (Anti-
Personnel Landmines Detection Product Development), a non-governmental
organization, has been developing technologies for landmine detection using Afri-
can Giant Pouched rats (Cricetomys gambianus) since 1996. Although the trained
rats showed a relatively high average indication rate of 76%, the score was varied
among individuals while taking a long period of training as well. When taking
advantage of BMI technology, the training paradigms are simplified. The electrical
stimulation can provide more stable control of rat’s locomotion, and the virtual
reward stimulation may accelerate combination learning between target odor and
reward. Moreover, with a bidirectional BMI, odor information might be read out
straightly from the rat’s brain, and more information could then be decoded includ-
ing odorant components and concentrations. These readouts might largely enrich
operator’s knowledge of environmental information, leading to more precise navi-
gation control of rat-robots.
Clinically, BMIs are designed for movement assistance and rehabilitation therapy
of paralyzed patients. An ideal BMI system requires bidirectional, closed-loop
information communication. Besides reading out motion intentions through brain-
to-machine pathway to drive prosthetics, sensory information feedback is of equal
importance. Compared to neural signal decoding, this recoding process through
machine-to-brain pathway still needs a long way to go deep into. To write in sensory
information properly, stimulation modes, target sites, and stimulation parameters are
all needed to be taken into consideration carefully. As our rat-robot model is based
on virtual sensation generation, studies on locomotion control also shed lights on
stimulation targets and parameter determinations. Through behavioral and imaging
tests, stimulation parameters can be optimized for more natural sensation regenera-
tion. Inspiringly, researchers from Pittsburgh have recently succeeded in restoring
tactile sensation through intracortical micro-stimulation upon human being [13]. By
delivering electrical micro-stimulation into corresponding representation locations
in somatosensory cortex, stable sensations can be evoked naturally. Similar to the
relationship between somatosensory cortex and thalamus in rodents, some study
suggested more stable encoding in thalamus than in cortex with nonhuman primates
[61]. As to stimulation methods, for clinical consideration, researchers are making
great efforts on seeking for possible noninvasive stimulation approaches.
Transcranial direct current stimulation (tDCS) and transcranial magnetic stimulation
(TMS) are two of the most commonly used noninvasive stimulation methods for
neural activity modulation. Both tDCS and TMS are considered safe and painless for
human use; nonetheless, lacking spatial specificity, these two stimulation methods
are not suitable for localized stimulations. Ultrasound has recently been shown to be
capable of inducing reversible changes of neural activity noninvasively. Unlike
tDCS and TMS, focused ultrasound can be applied to deep brain structures and hit
the target specifically with greater spatial precision [62]. Though it still needs to be
assessed before clinical trials, this technique and other new ones will have impacts
on neural modulations.
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 145

References

1. Fritsch G, Hitzig E (1870) Uber die elektrische Erregbarkeit des Grossihirns. Arch Anat Physiol
Wissen Med 37:300–332
2. Gildenberg PL (2005) Evolution of neuromodulation. Stereotact Funct Neurosurg 83
(2–3):71–79
3. Owen SL, Green AL, Stein JF et al (2006) Deep brain stimulation for the alleviation of post-
stroke neuropathic pain. Pain 120(1–2):202–206
4. Marchand S, Kupers RC, Bushnell MC et al (2003) Analgesic and placebo effects of thalamic
stimulation. Pain 105(3):481–488
5. Krack P, Batir A, Van Blercom N et al (2003) Five-year follow-up of bilateral stimulation of the
subthalamic nucleus in advanced Parkinson's disease. N Engl J Med 349(20):1925–1934
6. Bittar RG, Burn SC, Bain PG et al (2005) Deep brain stimulation for movement disorders and
pain. J Clin Neurosci 12(4):457–463
7. Wilson BS, Dorman MF (2008) Cochlear implants: a remarkable past and a brilliant future.
Hear Res 242(1–2):3–21
8. Shannon RV (2012) Advances in auditory prostheses. Curr Opin Neurol 25(1):61–66
9. Chow AY, Chow VY, Packo KH et al (2004) The artificial silicon retina microchip for the
treatment of vision loss from retinitis pigmentosa. Arch Ophthalmol 122(4):460–469
10. Yanai D, Weiland JD, Mahadevappa M et al (2007) Visual performance using a retinal
prosthesis in three subjects with retinitis pigmentosa. Am J Ophthalmol 143(5):820–827
11. Romo R, Hernandez A, Zainos A et al (2000) Sensing without touching: psychophysical
performance based on cortical microstimulation. Neuron 26(1):273–278
12. Houweling AR, Brecht M (2007) Behavioural report of single neuron stimulation in somato-
sensory cortex. Nature 451(7174):65–68
13. Flesher SN, Collinger JL, Foldes ST et al (2016) Intracortical microstimulation of human
somatosensory cortex. Sci Transl Med 8(361):141r–361r
14. Holzer R, Shimoyama I (1997) Locomotion control of a bio-robotic system via electric
stimulation. In: Intelligent robots and systems, 1997. IROS’97., Proceedings of the 1997
IEEE/RSJ international conference on IEEE, pp 1514–1519
15. Bozkurt A, Gilmour R, Stern D et al (2008) MEMS based bioelectronic neuromuscular
interfaces for insect cyborg flight control. In: Proceedings IEEE micro electro mechanical
systems IEEE, p 160
16. Bozkurt A, Gilmour RF Jr, Lal A (2009) Balloon-assisted flight of radio-controlled insect
biobots. IEEE Trans Biomed Eng 56(9):2304–2307
17. Sato H, Berry CW, Peeri Y et al (2009) Remote radio control of insect flight. Front Integr
Neurosci 3:24
18. Bao L, Zheng N, Zhao H et al (2011) Flight control of tethered honeybees using neural electrical
stimulation. In: International IEEE EMBS conference on neural engineering IEEE, pp 558–561
19. Talwar SK, Xu S, Hawley ES et al (2002) Direct brain stimulation enables training of complex
behaviors in freely roaming animals. In: Proceedings of annual international conference of the
IEEE engineering in medicine and biology society IEEE, pp 2076–2077
20. Gomes W, Perez D, Catipovic J (2006) Autonomous shark tag with neural reading and
stimulation capability for open-ocean experiments. EOS Trans Am Geophys Union 87:36
21. Brown S (2006) Stealth sharks to patrol the high seas. New Sci 189(2541):30–31
22. Wang W, Guo C, Sun J et al (2009) Locomotion elicited by electrical stimulation in the
midbrain of the Lizard Gekko gecko. In: Budiyono A, Riyanto B, Joelianto E (eds) Studies in
computational intelligence. Springer, Berlin, p 145
23. Talwar SK, Xu S, Hawley ES et al (2002) Rat navigation guided by remote control. Nature 417
(6884):37–38
24. Hermer-Vazquez L, Hermer-Vazquez R, Rybinnik I et al (2005) Rapid learning and flexible
memory in "habit" tasks in rats trained with brain stimulation reward. Physiol Behav 84
(5):753–759
146 J. Zhang et al.

25. Doty RW (1969) Electrical stimulation of the brain in behavioral context. Annu Rev Psychol
20:289–320
26. Feng Z, Chen W, Ye X et al (2007) A remote control training system for rat navigation in
complicated environment. J Zheijang Univ Sci A 8(2):323–330
27. Francis JT, Xu S, Chapin JK (2008) Proprioceptive and cutaneous representations in the rat
ventral Posterolateral thalamus. J Neurophysiol 99(5):2291–2304
28. Xu K, Zhang J, Zhou H et al (2016) A novel turning behavior control method for rat-robot
through the stimulation of ventral posteromedial thalamic nucleus. Behav Brain Res 298
(Pt B):150–157
29. Zhang J, Zhang S, Yu C et al (2018) Intrinsic optical imaging study on cortical responses to
electrical stimulation in ventral posterior medial nucleus of thalamus. Brain Res 1684:40–49
30. Zhu Y, Xu K, Xu C et al (2016) PET mapping for brain-computer Interface stimulation of the
Ventroposterior medial nucleus of the thalamus in rats with implanted electrodes. J Nucl Med
57(7):1141–1145
31. Keay KA, Bandler R (2001) Parallel circuits mediating distinct emotional coping reactions to
different types of stress. Neurosci Biobehav Rev 25(7):669–678
32. Brandao ML, Troncoso AC, Silva M et al (2003) The relevance of neuronal substrates of
defense in the midbrain tectum to anxiety and stress: empirical and conceptual considerations.
Eur J Pharmacol 463(1–3):225–233
33. Lin J, Yu C, Jia J et al (2010) Using dlPAG-evoked immobile behavior in animal-robotics
navigation. 2010 5th international conference on computer science & education, (iccse 2010) -
hefei, china, 1295–1298
34. Huai R, Yang J and Wang H et al (2009) A new robo-animals navigation method guided by the
remote control. In: Shi R, Fu WJ, Wang YQ, Wang HB(eds) International conference on
biomedical engineering and informatics. IEEE, pp 1170–1173
35. Koo B, Koh CS, Park HY et al (2017) Manipulation of rat movement via Nigrostriatal
stimulation controlled by human visually evoked potentials. Sci Rep 7(1):2340
36. Ye X, Wang P, Liu J et al (2008) A portable telemetry system for brain stimulation and neuronal
activity recording in freely behaving small animals. J Neurosci Methods 174(2):186–193
37. Guo S, Chen S, Zhang Q et al (2014) Optogenetic activation of the excitatory neurons
expressing CaMKIIFC; in the ventral tegmental area upregulates the locomotor activity of
free behaving rats. Biomed Res Int 2014:687469
38. Guo S, Zhou H, Zhang J et al (2013) A multi-electrode array coupled with fiberoptic for deep-
brain optical neuromodulation and electrical recording. In: Engineering in medicine and biology
society (EMBC), 2013 35th annual international conference of the IEEE, pp 2752–2755
39. Aravanis AM, Wang LP, Zhang F et al (2007) An optical neural interface: in vivo control of
rodent motor cortex with integrated fiberoptic and optogenetic technology. J Neural Eng 4(3):
S143–S156
40. Wang J, Wagner F, Borton DA et al (2012) Integrated device for combined optical
neuromodulation and electrical recording for chronic in vivo applications. J Neural Eng 9
(1):16001
41. Chen S, Zhou H, Guo S et al (2015) Optogenetics based rat-robot control: optical stimulation
encodes “stop” and “escape” commands. Ann Biomed Eng 43(8):1851–1864
42. Guo S, Zhou H, Wang Y et al (2014) A rat-robot control system based on optogenetics. In:
Ren L, Wang H, Dai Z (eds) Applied mechanics and materials, pp 848–852
43. Min BK, Marzelli MJ, Yoo SS (2010) Neuroimaging-based approaches in the brain-computer
interface. Trends Biotechnol 28(11):552–560
44. Pais-Vieira M, Lebedev M, Kunicki C et al (2013) A brain-to-brain interface for real-time
sharing of sensorimotor information. Sci Rep 3:1319
45. Deadwyler SA, Berger TW, Sweatt AJ et al (2013) Donor/recipient enhancement of memory in
rat hippocampus. Front Syst Neurosci 7:120
46. Pais-Vieira M, Chiuffa G, Lebedev M et al (2015) Building an organic computing device with
multiple interconnected brains. Sci Rep 5:11869
5 Brain-Machine Interface-Based Rat-Robot Behavior Control 147

47. Grau C, Ginhoux R, Riera A et al (2014) Conscious brain-to-brain communication in humans

using non-invasive technologies. PLoS One 9(8):e105225
48. Rao RP, Stocco A, Bryan M et al (2014) A direct brain-to-brain interface in humans. PLoS One
9(11):e111332
49. Yoo SS, Kim H, Filandrianos E et al (2013) Non-invasive brain-to-brain interface (BBI):
establishing functional links between two brains. PLoS One 8(4):e60410
50. Zhang S, Yuan S, Huang L, Zheng X, Wu W, Xu K, Pan G (2019) Human mind control of rat
cyborg’s continuous locomotion with wireless brain-to-brain interface. Sci Rep 9:1321. https://
doi.org/10.1038/s41598-018-36885-0
51. Martinez-Leon J, Cano-Izquierdo J, Ibarrola J (2016) Are low cost brain computer Interface
headsets ready for motor imagery applications? Expert Syst Appl 49:136–144
52. Zhang Y, Sun C, Zheng N, et al (2010) An Automatic Control System for Ratbot Navigation
[C]// 2010 IEEE/ACM International Conference on Green Computing and Communications &
2010 IEEE/ACM International Conference on Cyber, Physical and Social Computing. IEEE
Computer Society, 2010, Hangzhou, China
53. Sun C, Zheng N, Zhang X et al (2011) An automatic control model for rat-robot. In: Engineer-
ing in medicine and biology society, EMBC, 2011 Annual international conference of the IEEE,
Boston, USA
54. Sun C, Zheng N, Zhang X et al (2013) Automatic navigation for rat-robots with Modeling of the
human guidance. J Bionic Eng 10(1):46–56
55. Whishaw IQ, Kolb B (2004) The behavior of the laboratory rat: a handbook with tests. Oxford
University Press, Oxford
56. Hull CL, Spence KW (1938) “Correction” vs “non-correction” method of trial-and-error
learning in rats. J Comp Psychol 25(1):127–145
57. Sun C, Zhang X, Zheng N et al (2012) Bio-robots automatic navigation with electrical reward
stimulation. Engineering in medicine and biology society (EMBC), 2012 annual international
conference of the IEEE,San Diego, California, USA
58. Zhang C, Sun C, Gao L et al (2013) Bio-robots automatic navigation with graded electric
reward stimulation based on reinforcement learning. Conf Proc IEEE Eng Med Biol Soc
2013:6901–6904
59. Wu Z, Yang Y, Xia B et al (2014) Speech interaction with a rat. Chin Sci Bull 59
(28):3579–3584
60. Wang Y, Lu M, Wu Z et al (2015) Visual Cue-guided rat cyborg for automatic navigation. IEEE
Comput Intel Mag 10(2):42–52
61. Song W, Semework M (2015) Tactile representation in somatosensory thalamus (VPL) and
cortex (S1) of awake primate and the plasticity induced by VPL neuroprosthetic stimulation.
Brain Res 1625:301–313
62. Yoo SS, Bystritsky A, Lee JH et al (2011) Focused ultrasound modulates region-specific brain
activity. NeuroImage 56(3):1267–1275
Chapter 6
Realizing Efficient EMG-Based Prosthetic
Control Strategy

Guanglin Li, Oluwarotimi Williams Samuel, Chuang Lin,

Mojisola Grace Asogbon, Peng Fang, and Paul Oluwagbengba Idowu

Abstract As an integral part of the body, the limb poses dexterous and fine motor
grasping and sensing capabilities that enable humans to effectively communicate
with their environment during activities of daily living (ADL). Hence, limb
loss severely limits individuals’ ability especially when they need to perform tasks
requiring their limb functions during ADL, thus leading to decreased quality of life.
To effectively restore limb functions in amputees, the advanced prostheses that are
controlled by electromyography (EMG) signal have been widely investigated and
used. Since EMG signals reflect neural activity, they would contain information on
the muscle activation related to limb motions. Pattern recognition-based myoelectric
control is an important branch of the EMG-based prosthetic control. And the
EMG-based prosthetic control theoretically supports multiple degrees of freedom
movements that allows amputees to intuitively manipulate the device. This chapter
focuses on EMG-based prosthetic control strategy that involves utilizing intelligent
computational technique to decode upper limb movement intentions from which
control commands are derived. Additionally, different techniques/methods for
improving the overall performance of EMG-based prostheses control strategy were
introduced and discussed in this chapter.

G. Li (*) · O. W. Samuel · C. Lin · P. Fang

CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes
of Advanced Technology (SIAT), Chinese Academy of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
e-mail: [email protected]
M. G. Asogbon · P. O. Idowu
CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes
of Advanced Technology (SIAT), Chinese Academy of Science (CAS), Shenzhen, China
Institute of Advanced Integration Technology, SIAT, Shenzhen, China
Shenzhen College of Advanced Technology, University of Chinese Academy of Sciences,
Shenzhen, China

© Springer Nature Singapore Pte Ltd. 2019 149

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_6
150 G. Li et al.

Keywords Myoelectric signal · Pattern recognition · Limb motion · Upper limb

amputee · Prosthesis · Rehabilitation robot

6.1 Introduction

Human upper limb represent an integral part of the body that provides incredible array
of functionalities during activities of daily living (ADL). In addition to possessing
dexterous and fine motor grasping/sensing capabilities, the upper limb enables indi-
viduals to freely and effectively communicate with their natural environments through
artistic expressions, sign languages, and greetings (handshake/hand gesture), among
others. Therefore, the loss of one or both arms severely limits individuals’ ability
especially when they need to perform tasks requiring their arm functions during ADL,
thus leading to decreased quality of life [1]. Limb loss also has profound negative
impact on amputees including stump pain or phantom limb pain, or perhaps even both,
depression, changes in body image, and psychological burden [1–5]. Under certain
circumstance, upper limb loss may also affect amputees’ stability (dis-balance) during
mobility, thus making them prone to falls or collisions with other individuals/objects.
To this end, limb loss causes a devastating experience necessitating the need for proper
psychological and physical rehabilitation strategy for arm amputees.
A large number of the commercially available upper limb prostheses are either
mechanically (body-powered) or electrically (motorized) driven. Meanwhile, body-
powered upper limb prosthetic devices were primarily developed as alternative to aid
amputees in carrying out ADL requiring their arm functions [6]. Fundamentally, these
prostheses operate by using cables to link the movement of the body to the device and
thus actuating simple hand tasks such as opening or closing of the hook/gripper. To
achieve simple hand grasp tasks, the amputee would need to possess a significant
amount control over his/her shoulder, chest, and even the residual limb which should
be long enough to support the motion. Although body-powered prostheses have fine
cosmetic appearances that are comparable to the human natural arm [7], they are
nonintuitive and require significant amount of efforts (burdensome) to perform simple
hand tasks and as well support only single degree of freedom (DOF) movement per
time [7]. With these limitations, motorized upper limb prostheses are considered as
viable alternative for intuitive restoration of multiple DOF arm functions in amputees.
At the forefront of this technology are the prostheses that utilize electromyography
(EMG) recordings to characterize upper limb movement intentions of the amputees
[8–12]. Thus, the properties of the EMG signals are encoded in form of control
commands to the prosthetic controller which then actuates the device accordingly. It
is noteworthy that in the last two to three decades, the motorized upper limb prosthetic
technology has progressively advanced leading to the development of better control
methods as well as designs. Hence, this article focuses on the EMG-based prosthetic
control with emphases on methods/technologies that could help improve the overall
performance and acceptability of the prostheses system.
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 151

The remaining part of the chapter is organized as follows: Section 6.2 discusses
different EMG-based prosthetic control methods; Section 6.3 introduces some poten-
tial techniques for improving the performance of EMG-based prosthetic control;
Section 6.4 presents the concluding remarks.

6.2 EMG-Based Prosthetic Control Methods

6.2.1 Background

Fundamentally, EMG signals represent electrical manifestation of neuromuscular

activation associated with a contracting muscle. Such signals have been proven to
contain rich set of neural information from which different types of upper limb
movements could be accurately decoded, thus making them useful control input to
prosthetic devices as well as other forms of rehabilitation robots [8, 13]. Additionally,
EMG signals carry significant information for diagnosis of neuromuscular disorders
and other muscle-related diseases. It has been established that EMG signal patterns
for a specific limb movement are repeatable over time and distinct from the patterns
of other limb movements. By exploring this phenomenon, researchers have been
able to decode a range of arm movements from EMG recordings using intelligent
computational methods. To this end, dozens of previous studies have reiterated the
potential of EMG-based method in the control of multifunctional prostheses [14–
16]. Thus, the following section will focus on discussing the currently applied
EMG-based prosthetic control schemes including: pattern recognition (PR) control
method and the simultaneous and proportional control (SPC) method.

6.2.2 Pattern Recognition-Based Prosthetic Control

Although the concept of EMG pattern recognition (EMG-PR) is by no means new, it

has been proven over time to be promising especially for improving the dexterity of
control in upper limb prosthetic devices in comparison with the conventional
amplitude-based methods. It is noteworthy that the first generation of pattern
recognition-driven prosthetic controllers was built in the late 1960s/early 1970s
[17–19]. Since then, advancement in signal processing, multichannel recording,
and microprocessor technology has expedited implementation of EMG-PR method
in embedded control systems. Also, with a lot of synergistic efforts from the
academia and industries, EMG-PR-based prosthetic arms have been recently made
available for commercial use.
EMG-PR prosthetic control method is primarily based on the assumption that
EMG signal patterns corresponding to a particular class of limb movement are
consistently repeatable across trials and as well distinct from EMG signal patterns
of the other classes. Due to the promising nature of this control method, it has been
152 G. Li et al.

Classifier Training Prosthetic

control
EMG
Recordings

Control Command
EMG Preprocessing

Feature Extraction

Classification
Motion
Electrode
Speed
placement EMG Amplitude

Fig. 6.1 Schematic diagram of EMG-PR-based prosthetic control system

widely explored in the last two to three decades with significant progress reported in
terms of its robustness. The control schemes as a whole consist of a sequence of steps
beginning with: EMG signal acquisition, preprocessing of the recorded signals,
extraction of feature set, classification of the extracted features, and issuing of
control commands to drive prosthesis. Figure 6.1 shows the schematic diagram of
a typical EMG-PR-based prosthetic control system.
In the above represented control scheme, myoelectric signals are collected via
electrodes placed on the skin surface underlying the arm muscles. Often times, the
electrodes are integrated with miniaturized amplifier to improve the quality of the
recorded signals through standard amplification and filtering procedures. After-
wards, the acquired EMG recordings are subjected to preprocessing procedure, so
as to attenuate inherent interferences. In this regard, the 50/60 Hz notch filter is
usually applied to the raw acquired signals to minimize interferences resulting from
power line noise. In addition to the notch filter, a band-pass filter with cut of
frequency between 10/20 Hz and 500 Hz is often applied to the EMG signals to
obtain the most useful spectrum with high motor information content.
EMG signals recorded during targeted limb movements are continuous in nature,
thus for a specific length of recording, a decision corresponding to a subset of the
observed limb movements may be required to actuate the prosthesis. Therefore, the
use of data segmentation technique in handling the preprocessed data prior to feature
extraction is necessary. This step helps increase the probability of the prosthetic
control system achieving high accuracy as well as quick response time. Ideally in
real life applications, a response time of no more than 300 ms is required [20]. Note
that the response time (which is a function of the segment length and the processing
time) is the time required to produce the control commands that drive the prostheses.
In a study conducted by Englehart and Hudgins [21], it was shown that by adopting
continuous segmentation on a steady-state EMG signal, the segment length can be
reduced to about 128 ms without a significant reduction in classification accuracy.
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 153

Also, it has been shown previously that segmented steady-state EMG signals
achieved better accuracy compared to segmented transient EMG signals, and less
degradation were recorded with shorter segments of the steady-state EMG data.
After considering the segment length and the state of the data, the third important
data segmentation factor is the windowing technique adopted. Data segmentation is
often achieved via either of the following windowing techniques: sliding analyses
windowing and adjacent windowing method [20]. The adjacent windowing method
uses disjoint segments of predefined length for its feature extraction; and the
intended limb movement is classified after a certain processing delay. Because the
processing time often denotes a small fraction of segment length, the processor is
found to be mostly idle during the remaining time of the segment length. Meanwhile,
in the sliding windowing method, the new window segment slides over the current
window segment, with an increment time less than the segment length. In compar-
ison, this sliding windowing method takes advantage of the processor’s idle time in
adjacent windowing method to produce more classified outputs. Thus, the sliding
windowing technique has been widely applied to steady-state EMG signals for
enhanced feature extraction process in most recent studies on EMG-PR-based
prosthetic control.
Aside from the feature extraction stage, the choice of classification algorithm has
been reported to be another important factor in EMG-PR-based prosthetic control
systems. Thus, choosing an efficient classifier becomes necessary to ensure accurate
and consistent classification performance. In this regard, a number of classification
schemes have been examined with respect to their performance in identifying
different upper limb movements. For instance, classification schemes based on
Bayesian statistics [22], adaptive neural networks (ANN) [23], fuzzy logic [24],
support vector machine [25], and linear discriminant analysis (LDA) [11–26] were
previously studied. The ANN-based classifier and the LDA classifier were found to
be more accurate for the prediction of limb movement intention. However, the LDA
has been widely applied because it is relatively less complex in terms of implemen-
tation and performs well in comparison with the ANN and other complex classifi-
cation Schemes [26].
After reaching a decision on the classification scheme to adopt, a classifier is
developed by utilizing part of extracted EMG feature vector. Afterwards, the per-
formance of the built classifier in identifying multiple classes of limb movement
intent is evaluated using the testing set as obtained from the remaining part of the
feature vector. The performance of the EMG-PR-driven prostheses in terms of its
controllability is mostly examined in both off-line and real-time metrics. In general,
metrics including recognition rate (classification accuracy) of limb movements/sub-
jects, the time taken to complete a motion (motion completion time), number of
successfully completed motion (motion completion rate), and the time taken to select
the target motion (motion selection time) are often considered. Also, the perfor-
mance of the device could be assessed based on the number success recording
reaching and grasping tasks.
154 G. Li et al.

6.2.3 Simultaneous and Proportional EMG-Based Prosthetic

Control

Multiple DOF pattern recognition-based prostheses control schemes have recently

been implemented to enable upper limb amputees perform the usual arm function
tasks during ADL. Even with this remarkable achievement, the available EMG-PR-
based controlled prostheses do not support simultaneous control of multiple DOFs
[27, 28]. This limitation prevents upper limb prostheses users from having the
natural feel of the coordinated joint control often associated with an intact limb.
Although advanced prosthetic arms [29, 30], including multiple DOF wrists, offer
the mechanical means to restore arm movements, there is still a significant need for
systems that enable simultaneous control of such devices.
To develop a simultaneous prosthetic control scheme which offers multiple DOF
movements, different approaches have been proposed in the recent years. These
approaches include the use of artificial neural networks for joint kinematics pre-
dictions during upper limb movements [31], analysis of the muscle synergies
underlying a range of upper limb movements [32], and the utilization of joint
kinetics [33]. While these methods have been reported to be promising toward
clinical realization of upper limb prostheses, they however mostly focus on provid-
ing control for limited classes of limb movements such as wrist without hand or have
imposed identical velocities on all active degrees of freedom movements and thus do
not provide independent proportional control of individual degrees of freedom.
Additionally, toward achieving simultaneous and proportional control scheme,
some investigators developed a model of muscle synergy driven by nonnegative
matrix factorization (NMF) method to resolve myoelectric signal factorization issue
that is capable of hindering continuous control [32, 34]. The NMF technique often
utilizes a DOF-wise training approach, based on which simultaneous control of
multi-DOF is achieved via linearly combined single degree of freedom. This kind
of strategy supports simultaneous and proportional myoelectric-based control
[32, 34]. Despite the promising results of this scheme, there are still some short-
comings that have hindered its real life implementation. The most critical disadvan-
tage of the NMF-driven EMG-based prostheses is that it needs a DOF-wise
calibration during which requires a user to activate precisely single DOF in a specific
manner, e.g., firstly, trigger DOF1 for several trials followed by DOF2. Besides the
inconvenience and the time needed for the execution of this phase, more notably, if
the prostheses users are unable to trigger an individual DOF (i.e., the activation
patterns during the training session are not exactly of single DOF), the DOF-wise-
driven NMF approach will produce inadequate muscle synergy matrix that may
result in poor control outcome. This behavior could be attributed to the fact that the
classical NMF-based scheme lacks the ability to produce a factorization that is sparse
enough for efficient control of multiple DOF [35, 36]. Hence, we proposed an
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 155

alternative approach to the classical NMF that identifies the factorization which
separates the generated basis functions robustly and concurrently in a quasi-
unsupervised way to simultaneously and proportionally control the prostheses.
In the proposed approach, the subject does not need to follow a predefined
sequence to activate single DOF in the calibration phase, and can even activate
more than one DOF simultaneously in the training stage, which is not allowed in the
classical NMF method. Nevertheless, the idea of the synergy matrix could be realized
in one step, which is contrary to the conventional NMF approach that attempts to
sequentially extract the bases for each DOF. For this reason, constraints were included
in the factorization method and the method is at maximizing the sparseness of the
resulting control outcome. The constraints associated with the sparseness limits the
space of possible NMF solutions. Precisely, the solution with bases functions associ-
ated to single DOF, which is the target solution, represents the sparsest of the other
solutions. In this regard, the factorization with constraints usually does not need prior
calibrations/activations of single DOF and could be utilized to any set of target tasks
carried out by the user with no any form of strict labeling. The proposed sparse
nonnegative matrix factorization (SNMF)-based simultaneous and proportional
myo-control scheme is presented mathematically as follows.
In myoelectric control, multiple DOF limb movements could be re-represented as a
linearly combined individual DOFs, and the synergy basis and control input are
nonnegative [34]. Suppose we represent the root mean square (RMS) values of an
N-channel with T-length surface myoelectric signal as Z, in which the t-th column is
the myoelectric signal at a given time t and m represents the number of DOFs. Thus,
the multichannel observation can be approximated by the product of a N
2m latent
nonnegative synergy matrix W and a 2m
T latent nonnegative control signal matrix.
F is expressed in the following equation:

ZN
T  WN
2m F2m
T ð6:1Þ

While the sparseness of the solution is obtained by imposing a specific single

DOF activation sequence for calibration, the above factorization problem can be
solved in a semi-supervised way with the summation property of DOFs in the muscle
signal domain.
A mathematical way to generate a sparse solution without the need for the specific
set of calibration data for individual DOF activation is utilized. Meanwhile, this is
achieved by imposing a sparseness constraint on the control signals, which leads to
the proposed SNMF Scheme [37].
Mathematically, the extent (degree) of sparseness could be controlled by utilizing
either 11-norm or l0-norm. To build a computationally efficient system, we chose the
l1-norm-based sparse NMF scheme, SNMF [37]. Meanwhile, the objective function
of the SNMF method is expressed mathematically as follows:
156 G. Li et al.

1 X T
min kZ  WFk2Fro þ λ kFð:; t Þk21 s:t: W,F  0,
W, F 2
 t¼1
2 þ 32 2 3
 F1   F ð:; t Þþ 2
   1 
 6 F  7 T  6 F ð:; t Þ 7
1  þ  6 1 7 X  6 1 7
, min  Z  W 1 ; W 1 ;   W þ  6
m ; W m 6 ⋮ 7
7 þ λ 6 ⋮ 7
W , F 2 4 F þ 5
6
 4 þ
7
5
 m 
t¼1
 F ð:; t Þm 
  
F m Fro   
F ð:; t Þm 1
s:t: W ij ,F ij  0
ð6:2Þ

where F (:, t) is the t-th column vector of F [Eq. (6.1)], Fro is the Frobenius norm,
and λ > 0 is a regularization parameter to balance the accuracy of the factorization
and the degree of sparseness of F. In the experiment, we choose the optimalλ through
cross-validation with the value of m ¼ 2. The superscripts “+” and “–” denote the
positive and negative direction of each DOF, respectively. The above formulation
can be rewritten as:
   2
 W Z 
min  p ffiffi
ffi F  s:t: W,F  0, ð6:3Þ
W, F λe1
2m 01
T Fro

where e1
2mrepresent a row vector that contains values equal to 1 and 01
Trepresent
entries with values equal to 0. Equation (6.3) could be adequately solved via the
alternating nonnegative least squares (ANLS) approach. Meanwhile, for the ANLS
method, F and W are iteratively updated by adjusting the other one:
   2
 Wð k Þ Z 
Fðkþ1Þ 
¼ argmin  p ffiffi
ffi F  ð6:4Þ
λ e 0 1
T

F 1
2m Fro
 
Wðkþ1Þ ¼ argmin WFðkþ1Þ  ZFro
2
ð6:5Þ
F

It could be seen that Eqs. (6.4) and (6.5) represent the classical least square
problem, and either of the equation has a closed-form solution. Meanwhile, the
derivation in [37] reveals that the SNMF method could converge to a stationary
point. The SNMF method can equally extract all the basis functions in a single step
from the EMG recordings produced via arbitrary combinations of DOFs per user.
Only the ordering of the control input signals (basis functions) would be
undetermined using this method, but this issue could be simply addressed. In
practice, the labels of the first two single DOFs need to be recorded in order to
decide the sequence of the basis in the matrix (F) from which control signal is
obtained, thus making the method quasi-unsupervised instead of completely
unsupervised.
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 157

Having computed the synergy basis matrix W from the recorded myoelectric
signals, it is assumed to be a constant (at least over the period of the test experi-
ments). To estimate the control input signal associated with the intended DOF
activation level, a pseudo inverse of W (represented as W+) is applied,
 and it was
multiplied with the newly obtained surface myoelectric signals f Z . Thus, the
~ is defined as follows:
estimated control input signal (F

~ ¼ Wþ Z
F ~ ð6:6Þ

where
 þ  þ 
~ ¼ F
F ~1 ; F
~1 ; F
~2 ; F
~2 ; ð6:7Þ

Toward ensuring that no components are surpassed by the others, the individual
component of F~ in (6.7) is subject to a normalization process based on its maximum
value. An estimate of the control input signal in (6.7) is thus scaled by the scalar
correction factorsτij that are utilized to account for the indetermination of the signal
power (range of control signals) during the factorization process:

F~ 1 ¼ τ11 F~ þ ~
1  τ12 F 1 ð6:8Þ
F 2 ¼ τ21 F 2  τ22 F~ 
~ ~ þ
2

where we name F~ 1 and F~ 2the control signals for DOF1 and DOF2, respectively. The
multiplicative factors  τijare determined for each subject so that the final control
signals, F~ ¼ F~ 1 ; F~ 2 match the range of joint angles in the respective DOFs. The
tau-s parameters are manually set before operating online tasks according to different
subjects. The obtained control signals, low-pass filtered at 6 Hz (kinematic band-
width), are then used by the subjects for controlling the DOFs.
It is noteworthy that the proposed SNMF scheme is aimed at robust and concur-
rent extraction of basis functions for myo-control in upper limb prostheses with a
quasi-unsupervised scheme. This factorization scheme shown in our study has merits
compared to the classical NMF because it selects the sparsest solution among infinite
feasible solutions. To that end, there is no need for pre-calibration of the constraint
tasks by users but rather the constraint on the solution. The method has been
compared in similar conditions to the NMF and linear regression with results
proving it to be superior over these methods even when a single DOF activation
was used for the factorization which is an ideal condition for the NMF. Furthermore,
the approach could be used by factorizing the myoelectric signals from arbitrary
tasks by combining individual DOFs. The proposed factorization algorithm allows
robust simultaneous and proportional control and is superior to previous supervised
algorithms, and its minimal supervision characteristic paves the way to online
adaptation in myoelectric control.
158 G. Li et al.

6.3 Neural Interfaces for Improvement of EMG-Based

Prosthetic Control

6.3.1 Background

The previously described sections show that EMG-based prostheses control method
could potentially allow amputees to intuitively manipulate the prosthetic arm with
high level of dexterity and as well achieve multiple DOF controls. Developing
EMG-based prostheses with these characteristics mainly depends on whether the
users have enough residual muscles which act as source of EMG control signals. For
instance, individuals with below the elbow amputation (transradial amputees)
generally have enough residual muscles for the control of wrist functions such as
wrist flexion/extension, wrist pronation/supination, and so on. Also, these category
of amputees could provide enough EMG signals from their residual forearm muscles
to control hand and even finger movements as reported in a number of previous
studies [7]. On the other hand, individuals with high-level amputations
(transhumeral or shoulder disarticulation) usually do not have enough residual
muscles to provide myoelectric signals for the successful control of the wrist and
hand movements although they can produce enough EMG signals for the control of
elbow functions (elbow flexion/extension). Thus, intuitively controlling multiple
DOF hand/wrist movements becomes a major challenge for these categories of
amputees. One possible means of addressing the issue of insufficient generation of
myoelectric control signals for these users would be to consider neural machine
interfaces approaches such as targeted muscle reinnervation (TMR) and targeted
nerve function replacement (TNFR). The theoretical background is that muscles can
be considered as the bio-amplifier of the neural signals.

6.3.2 Targeted Muscle Reinnervation Technology

Toward enabling high-level upper limb amputees producing sufficient myoelectric

control signals again, a neural machine interface technology known as TMR was
proposed by Todd Kuiken et al. [28]. TMR is a surgical procedure that involves the
transfer of residual nerves from an amputated limb unto alternative muscle group
usually located around the chest region. During the surgery, brachial plexus nerves
that provide motor control and sensory feedback prior to amputation are transferred
to the arm or chest muscles that are biomechanically nonfunctional. Afterward, a
functional link is established between the implanted nerves and the targeted muscles,
thereby inducing the contractility of targeted muscles and enabling the generation of
additional myoelectric signals. Importantly, this surgical technique is aimed at
restoring the peripheral nerve functions in amputees thus enabling the provision of
sufficient myoelectric control signals again. A schematic diagram of the TMR
surgical procedure is conceptualized in Fig. 6.2a. For the subject shown in
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 159

Fig. 6.2 The schematic representation of the TMR technology. (a) TMR surgical procedure for
shoulder disarticulation amputee. (b) A bilateral amputee with successful TMR surgery controlling
a prosthetic arm [28]

Fig. 6.2b, the surgery was performed on his right chest region, and it involves
transferring the median, ulnar, radial, and musculocutaneous nerves to the subject’s
pectoralis major muscle segmented into four parts, respectively.
The above described neural machine interface technology for upper limb pros-
theses control has been implemented and tested on a number of human subjects. The
test sessions conducted with a bilateral amputee (Fig. 6.2b) shows that the high-level
amputee could efficiently perform additional hand function tasks such as drinking
and eating which could not be achieved prior to the TMR surgery. Importantly, the
success of performing additional tasks was objectively linked to the realization of
more EMG control signals after the TMR surgery.

6.3.3 Targeted Nerve Function Reconstruction Technology

Despite the success recorded by the TMR technology, it still has some limitations
which have slowed down its widespread adoption. For instance, TMR surgery
involves establishing functional links between the nerves and targeted muscles
leading to two major challenges: First, the targeted muscle would atrophy with
time because it has been denied of the required nutrients provided by the original
nerve which has been replaced with the implanted nerve. Second, intramuscular
nerve distribution (a key indicator of the motor function recovery) will also degen-
erate after a while due to the lack of nutrients from the surrounding cell body.
Peripheral nerves which play an important role after TMR surgery have been
reported to have a slow growth rate. That is, it takes about 3~6 months or even more
after TMR for proper redistribution of the implanted nerve into the targeted muscle,
thus leading to atrophy of the targeted muscle. Aside the slow growth rate, the
absence of nerve supplements is another factor responsible for atrophy of the
160 G. Li et al.

Fig. 6.3 A representation of the TMR/TNFR technology. (a) TMR/TNFR surgical procedure for a
transhumeral amputee, (b) EMG recordings after undergoing a TNFR surgery when the subject was
doing different arm movements

targeted muscle [38–40]. In such a situation, high-level upper limb amputees would
need training and treatment over an extended period for new functional connections
to be established between the targeted muscle and the implanted nerves. It is worth
noting that end-to-end anastomosis has been proven to be the best approach for
restoring the functions of injured nerves in patients [41]. Thus based on this
principle, targeted nerve function reconstruction (TNFR) technology is considered
to address the abovementioned problems associated with TMR. This is because
TNFR has the potential to better restore the motor functions associated with multiple
degrees of freedom upper limb movements. Aiming at improving intramuscular
nerve distribution as well as preventing muscle atrophy, TNFR establishes func-
tional links between the nerves of the targeted muscles and nerves from the ampu-
tated region/arm of the individual. A schematic representation of the TNFR
technology is shown in Fig. 6.3.
Shown in Fig. 6.3 is a male transhumeral amputee that underwent a TNFR
surgery. He actually got his left limb amputated in 2012 as a result of injury
sustained from a rolling machine. At the time of the surgery (2015), the amputee
was approximately 48 years old. Prior to the TNFR surgery, his residual limb was
subjected to a number of assessments to ensure that he meets up with the require-
ment. From the assessments, we found that the residual limb has a length of 22 cm
and has not been infected. In addition, the results of an MRI scan show that the
musculus biceps brachii and musculus triceps brachii of the residual limb were intact
with neuroma tissue formed at median nerve, ulnar nerve, and musculospiral nerve
terminals. Afterward, the TNFR surgery was performed on the subject by transecting
the MCN, MN, RN, and UN from the residual limb and implanting them into the
corresponding MCN, MN, RN, and UN on the amputee’s chest region.
About 7 days after the surgery, the operation regions were observed twice a week
for 8 consecutive weeks. Indexes, such as status of the incision (healing rate), skin
color and temperature, and limb peripheral circumference, were closely monitored.
The pre-surgery and post-surgery experiences of the subject were also utilized to
assess the limb status. These experiences include the subject’s feelings of missing
fingers, hand, and arm as well as his experience during observing different
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 161

movements (finger, wrist, hand, and elbow). Subsequently, a TNFR testing envi-
ronment was setup to demonstrate the feasibility of our proposed technology. In the
testing environment shown in Fig. 6.3b, the amputee was able to control the virtual
prostheses reliably across a range of upper limb movements compared to his
pre-TNFR experience. It is noteworthy that the TNFR technology is proposed by
our research group and we hope it will revolutionize prostheses control technology
and other related EMG-based rehabilitation device technologies in the future.
We further built a number of animal models recently to compare the performance
of proposed TNFR technology with that of TMR and found the TNFR model
obviously demonstrates better intramuscular nerve distribution compared to the
TMR model. Additionally, we found out that bicep brachii muscles on the opera-
tional sides of the TNFR model achieved better wet mass compared to that of the
TMR model.

6.3.4 Restoration of Sensory Feedback

Tactile sensation is an essential element for a dexterous hand manipulation. In an

ideal bidirectional hand prostheses, the user’s intentions are first decoded through
the efferent pathway, and then a nearly “natural” sensory feedback is delivered
through the remnant afferent pathways, simultaneously in real time, as illustrated
in Fig. 6.4.
The lack of proper sensory feedback mechanism limits prostheses users in
grasping and manipulating objects, which makes the use of prostheses cumbersome
and uncomfortable, leading to prosthetic abandonment. In fact, lack of sensory
feedback schemes in the currently available multiple-DOF upper-limb prostheses
have slowed down its clinical and commercial success [42]. Improper feedback of
tactile perception may also lead to “over-grasp” causing damage to an object or
“under-grasp” causing slip of object. Report from previous survey indicated that
about 95% of male amputees desire to have force feedback in their prostheses and
88% placed more emphasis on grip force and proprioceptive information of pros-
thesis position and movement. With the rejection rate of upper limb prostheses
reaching 39% and the increasing rate of people living with the loss of a limb
projected to be doubled by the year 2050 [43], the need for integrating efficient
sensory feedback mechanism into the prostheses becomes more necessary.
To restore the lost sensory feedback, Raspopovic et al. [44] were able to
stimulate the median and ulnar nerve fascicles using transversal multi-channel
intrafascicular electrodes (TIMEs) connected to the artificial hand sensors. The
subject was able to control the prosthesis through information provided by the
artificial sensors. This feedback enabled users to effectively control the grasping
force of the prosthesis with no visual or auditory feedback. To restore a sense of
touch that is close to the natural perception, the artificial receptors known as
sensors need to first register the characteristics of the sensed object/environment
[45], which works in a similar way to the receptor organs in humans. These sensory
162 G. Li et al.

Fig. 6.4 Closed loop architecture for restoring sensory feedback

receptors are broadly classified into pain receptors, cold receptors, warm receptors,
and four mechanoreceptors. Then, the receptors encode information into trains of
action potentials and also possess characteristic innocuous temperature variation
evoked by different afferents between ~5 and 48  C [46]. The mechanoreceptors
are grouped into slow-adapting receptors (SA-I and SA-II) and fast-adapting
receptors (FA-I and FA-II) having different receptive fields to which they respond
to mechanical stimulus, as well as different rates of adaptation. Also, tactile stimuli
are transferred from the artificial receptors to the intact skin through the actuators.
At this stage, signals are transported from receptors through nerve fibers to the
brain where complex information are interpreted. The time taken for the signal to
get to the brain is less than tens of milliseconds [47]. And lastly, a learning process
is initiated by the central nervous system to adapt to the new type of afferent
signals. A closed loop architecture for restoring a sensory feedback is described in
Fig. 6.4. This figure shows the transduction and interfacing system which transmit
the electronic signals and the control system which performs signal processing and
interpretation.
To provide a sensory feedback mechanism, data are firstly acquired through the
use of tactile sensors that come in contact with the target object (Fig. 6.4). Secondly,
the analog-to-digital converter digitizes the signal at the signal transduction stage
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 163

prior to further analysis. Before further signal processing, unwanted signal distur-
bances called artifact are removed by applying a wavelet transform and empirical
mode decomposition strategies to reduce the noise in the signal [48]. At the last
stage, the transmitted signal which mimics the real signal (action potential) is then
sent to the brain for interpretation.
With the invasive mechanism, action potentials traveling in the form of electrical
signals through the nervous systems could possibly regain all somatic sensory
information with modality matching by implanting neural electrodes in the periph-
eral nerves [49]. This approach is able to restore the sense of touch by means of
implantable nerve interfaces but always associated with risk of surgical procedure,
and their long-term usability needs to be fully investigated [50]. There are three
major approaches for invasive sensory feedback which are peripheral nervous
system (PNS) stimulation, targeted sensory reinnervation (TSR), and central nervous
system (CNS) stimulation. On the other hand, the promising noninvasive mechanism
could be realized through techniques such as mechanotactile, vibrotactile, or
electrotactile stimulations. To achieve any of these stimulation techniques, the
characteristics of the actuator (weight, size, and power consumption) would need
to be considered. Importantly, more research focus has been shifted toward the
noninvasive sensory feedback mechanism in the recent years with a lot of work
currently ongoing in this area.
In summary, integrating an efficient noninvasive sensory feedback mechanism
into the currently available multifunctional upper limb prostheses will not only
improve the control performance of the prostheses but also give users a feel of
their true natural environment during ADL.

6.4 Conclusion

The human upper limb is an important part of the body that enables the accomplish-
ment of a range of tasks in workplace, at home, and in social gathering during ADL.
The loss of this part of the body severely limits the capability of amputees while
undergoing ADL. In this regard, prosthetic device driven by different control
methods have been built to restore the lost limb functions in amputees. At the
forefront of these prostheses, control method is the EMG-based control which has
attracted considerable attention in both the academia and industry in the recent years.
Hence, this chapter focuses on EMG-based prostheses control methods with empha-
ses on pattern recognition-based control as well as simultaneous and proportional
control techniques, with details on the fundamental procedure of decoding limb
movement intents. Also, a number of techniques for improving the overall perfor-
mance of EMG-based prosthetic system were discussed with emphases on the
advantages and limitations.
164 G. Li et al.

Acknowledgments This work was supported in part by the National Natural Science Foundation
of China (#U1613222, #81850410557), the Shenzhen Basic Research Grant
(#JCYJ20160331185848286), and the Outstanding Youth Innovation Research Fund of Shenzhen
Institutes of Advanced Technology, Chinese Academy of Sciences (#Y7G016).

References

1. Holzer LA, Sevelda F, Fraberger G, Bluder O, Kickinger W, Holzer G (2014) Body image and
self-esteem in lower-limb amputees. PLoS One 9(3):e92943
2. Geertzen J et al (2015) Dutch evidence-based guidelines for amputation and prosthetics of the
lower extremity: amputation surgery and postoperative management Part 1. Prosthetics Orthot
Int 39(5):351–360
3. Mckechnie PS, John A (2014) Anxiety and depression following traumatic limb amputation: a
systematic review. Injury 45(12):1859–1866
4. Physical and Emotional effects of limb amputation: http://www.seriousinjurylaw.co.uk/other-
serious-claims/amputation/effects-of-amputation/Date. Last accessed 26 Sept 2017
5. Đurović A, Ilić D, Brdareski Z, Plavšić A, Đurđević S (2007) Pain, functional status, social
function and conditions of habitation in elderly unilaterally lower limb amputees. Vojnosanit
Pregl 64(12):837–843
6. Muilenburg AL, LeBlanc MA (1989) Body-powered upper-limb components. In: Comprehen-
sive management of the upper-limb amputee. Springer, New York, NY, pp 28–38
7. Li G (2011) Chapter 6: Electromyography pattern-recognition-based control of powered
multifunctional upper-limb prostheses. In: Mizrahi J (ed) Advances in applied electromyogra-
phy, IntechOpen (IntechOpen Limited), London, pp 99–117. ISBN: 978-953-307-382-8
8. Samuel OW, Li X, Geng Y, Asogbon MG, Fang P, Huang Z, Li G (2017) Resolving the adverse
impact of mobility on myoelectric pattern recognition in upper-limb multifunctional prostheses.
Comput Biol Med 90:76–87
9. Scheme E, Englehart K (2011) Electromyogram pattern recognition for control of powered
upper-limb prostheses: state of the art and challenges for clinical use. J Rehabil Res Dev
48:643–660
10. Smith LH, Hargrove LJ, Lock BA, Kuiken TA (2011) Determining the optimal window length
for pattern recognition-based myoelectric control: balancing the competing effects of classifi-
cation error and controller delay. IEEE Trans Neural Syst Rehabil Eng 19(2):186–192
11. Li G, Schultz AE, Kuiken TA (2010) Quantifying pattern recognition-based myoelectric control
of multifunctional transradial prostheses. IEEE Trans Neural Syst Rehabil Eng 18(2):185–192
12. Li G, Li Y, Yu L, Geng Y (2011) Conditioning and sampling issues of EMG signals in motion
recognition of multifunctional myoelectric prostheses. Ann Biomed Eng 39(6):1779–1787
13. Samuel OW, Zhou H, Li X, Wang H, Zhang H, Sangaiah AK, Li G (2017) Pattern recognition
of electromyography signals based on novel time domain features for amputees’ limb motion
classification. Comput Electr Eng 2017:1–10
14. Adewuyi AA, Hargrove LJ, Kuiken TA (2016) An analysis of intrinsic and extrinsic hand
muscle EMG for improved pattern recognition control. IEEE Trans Neural Syst Rehabil Eng 24
(4):485–494
15. Li X, Samuel OW, Zhang X, Wang H, Fang P, Li G (2017) A motion-classification strategy
based on sEMG-EEG signal combination for upper-limb amputees. J Neuroeng Rehabil 14(1):2
16. Geng Y, Samuel OW, Wei Y, Li G (2017) Improving the robustness of real-time myoelectric
pattern recognition against arm position changes in Transradial amputees. Biomed Res Int
2017:5090454
17. Finley FR, Wirta RW (1967) Myocoder studies of multiple myopotential response. Arch Phys
Med Rehabil 48(11):598–601. [PMID: 6060789]
6 Realizing Efficient EMG-Based Prosthetic Control Strategy 165

18. Lawrence P, Herberts P, Kadefors R (1973) Experiences with a multifunctional hand prosthesis
controlled by myoelectric patterns. In: Gav rilovic MM, Wilson AB Jr (eds) Advances in
external control of human extremities. Etan, Belgrade, pp 47–65
19. Lyman JH, Freedy A, Prior R (1976) Fundamental and applied research related to the design
and development of upper-limb externally powered prostheses. Bull Prosthet Res 13:184–195
20. Oskoei MA, Hu H (2007) Myoelectric control systems—a survey. Biomed Signal Process
Control 2(4):275–294
21. Englehart K, Hudgins B (2003) A robust, real-time control scheme for multifunction myoelec-
tric control. IEEE Trans Biomed Eng 50(7):848–854
21. Khushaba RN, Al-Timemy A, Kodagoda S, Nazarpour K (2016) Combined influence of
forearm orientation and muscular contraction on EMG pattern recognition. Expert Syst Appl
61:154–161
22. Huang YH, Englehart K, Hudgins B, Chan AD (2005) A Gaussian mixture model based
classification scheme for myoelectric control of powered upper limb prostheses. IEEE Trans
Biomed Eng 52:1801–1811
23. Hudgins B, Parker P, Scott RN (1993) A new strategy for multifunction myoelectric control.
IEEE Trans Biomed Eng 40(1):82–94
24. Ajiboye AB, Weir RF (2005) A heuristic fuzzy logic approach to EMG pattern recognition for
multifunctional prosthesis control. IEEE Trans Neural Syst Rehabil Eng 13:280–291
25. Oskoei MA, Hu H (2008) Support vector machine-based classification scheme for myoelectric
control applied to upper limb. IEEE Trans Biomed Eng 55(8):1956–1965
26. Hargrove L, Englehart K, Hudgins B (2007) A comparison of surface and intramuscular
myoelectric signal classification. IEEE Trans Biomed Eng 54:847–853
27. Coapt (2013) Available: http://coaptengineering.com/
28. Kuiken TA, Li G, Lock BA, Lipschutz RD, Miller LA, Stubblefield KA, Englehart KB (2009)
Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms.
JAMA 301:619–628
29. Harris A, Katyal K, Para M, Thomas J (2011) Revolutionizing prosthetics software technology.
In: Systems, man, and cybernetics (SMC), 2011 IEEE international conference on, October,
IEEE, pp 2877–2884
30. Samuel OW, Asogbon MG, Geng Y, Chen S, Fang P, Lin C, Wang L, Li G (2018) A novel
time-domain descriptor for improved prediction of upper limb movement intent in EMG-PR
system. In: Engineering in medicine and biology society (EMBC), 40th annual international
conference of the IEEE, 2018, July 17–21, 2018, Honolulu, Hawaii, USA
31. Muceli S, Farina D (2012) Simultaneous and proportional estimation of hand kinematics from
EMG during mirrored movements at multiple degrees-of-freedom. IEEE Trans Neural Syst
Rehabil Eng 20:371–378
32. Jiang N, Englehart KB, Parker PA (2009) Extracting simultaneous and proportional neural
control information for multiple-DOF prostheses from the surface Electromyographic signal.
IEEE Trans Biomed Eng 56:1070–1080
33. Samuel OW et al (2016) Examining the effect of subjects’ mobility on upper-limb motion
identification based on EMG-pattern recognition. In: Intelligent robot systems (ACIRS), Asia-
Pacific conference, IEEE, 137–14
34. Jiang N, Rehbaum H, Vujaklija I, Graimann B, Farina D (2014) Intuitive, online, simultaneous,
and proportional myoelectric control over two degrees-of-freedom in upper limb amputees.
IEEE Trans Neural Syst Rehabil Eng 22(3):501–510
35. Lee DD, Seung HS (2001) Algorithms for non-negative matrix factorization. In: Advances in
neural information processing systems. MIT Press, Cambridge, MA, pp 556–562
36. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization.
Nature 401(6755):788–791
37. Kim H, Park H (2007) Sparse non-negative matrix factorizations via alternating non-negativity-
constrained least squares for microarray data analysis. Bioinformatics 23(12):1495–1502
38. Kuiken TA, Childress DS, Rymer WZ (1995) The hyper-reinnervation of rat skeletal muscle.
Brain Res 676:113–123
166 G. Li et al.

39. Williams HB (1996) The value of continuous electrical muscle stimulation using a completely
implantable system in the preservation of muscle function following motor nerve injury and
repair: an experimental study. Microsurgery 17:589–596
40. Nicolaidis SC, Williams HB (2001) Muscle preservation using an implantable electrical system
after nerve injury and repair. Microsurgery 21:241–247
41. Grinsell D, Keating CP (2014) Peripheral nerve reconstruction after injury: a review of clinical
and experimental therapies. J Biomed Res Int 2014:698256
42. Samuel OW, Asogbon MG, Geng Y, Al-Timemy AH, Pirbhulal S, Ji N et al (2019) Intelligent
EMG pattern recognition control method for upper-limb multifunctional prostheses: advances,
current challenges, and future prospects. IEEE Access 7:10150–10165
43. Ziegler-Graham K, MacKenzie J, Ephraim P, Travison T, Brookmeyer R (2008) Estimating
the prevalence of limb loss in the United States: 2005 to 2050. Arch Phys Med Rehabil
89(3):422–429
44. Raspopovic S, Capogrosso M, Petrini FM, Bonizzato M, Rigosa J, DiPino G, Carpaneto J,
Controzzi M, Boretius T, Fernandez E, Granata G, Oddo CM, Citi L, Ciancio AL, Cipriani C,
Carrozza MC, Jensen W, Guglielmelli E, Stieglitz T, Rossini PM, Micera S (2014) Restoring
natural sensory feedback in real-time bidirectional hand prostheses. Sci Transl Med 6:222ra19
45. Christian A, Anders B, Sven-Olof F, Fredik S, Goran L, Birgitta R (2012) Sensory feedback
from a prosthetic hand based on air-mediated pressure from the hand to the forearm skin. J
Rehabil Med 44:702–707
46. Chortos A, Liu J, Bao Z (2016) Pursuing prosthetic electronic skin. Nat Mater 15(9):937–950
47. Zou L, Ge C, Wang ZJ, Cretu E, Li X (2017) Novel tactile sensor technology and smart tactile
sensing systems: a review. Sensors 17(11):2653
48. Saal HP, Bensmaia SJ (2015) Biomimetic approaches to bionic touch through a peripheral nerve
interface. Neuropsychologia 79:344–353
49. D’anna E, Petrini FM, Artoni F, Popovic I, Simanić I, Raspopovic S, Micera S (2017) A
somatotopic bidirectional hand prosthesis with transcutaneous electrical nerve stimulation
based sensory feedback. Sci Rep 7(1):10930
50. Micera S (2016) Staying in touch: toward the restoration of sensory feedback in hand prostheses
using peripheral neural stimulation. IEEE Pulse 7(3):16–19
Chapter 7
Neural Interface: Frontiers
and Applications
Cochlear Implants

Xiaoan Sun, Sui Huang, and Ningyuan Wang

Abstract The theory and implementation of modern cochlear implant are presented
in this chapter. Major signal processing strategies of cochlear implants are discussed
in detail. Hardware implementation including wireless signal transmission circuit,
integrated circuit design of implant circuit, and neural response measurement circuit
are provided in the latter part of the chapter. Finally, new technologies that are likely
to improve the performance of current cochlear implants are introduced.

Keywords Cochlear implant · Signal processing strategy · Neural response

measurement

7.1 Introduction of Cochlear Implant

A cochlear implant is a surgically implanted electronic device that provides func-

tional hearing to a person with severe to profound sensorineural hearing loss. It
bypasses a non-functional inner ear and stimulates the auditory nerve directly with
patterns of electrical currents derived from incoming sounds. A cochlear implant
system usually consists of an external sound processor and an implanted current
stimulator. Cochlear implant is widely accepted as the only effective way to restore
functional hearing for deaf people in daily clinical practice and is the most successful
neural prosthesis in current world. Up to date, about 450,000 people worldwide have
received cochlear implants; half of them are pediatric users who were able to develop
near normal language skills and merged seamlessly into mainstream schools and
society. As the technology of cochlear implant improves, and the risk associated
with the device and the surgery decrease over the years, more and more people

X. Sun (*) · S. Huang · N. Wang

Nurotron Biotechnology Inc., Irvine, CA, USA
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2019 167

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_7
168 X. Sun et al.

benefit from the device each year. The clinical threshold to be considered as a
cochlear implant candidate has been dropped from 100 dB hearing loss in 1980s
and 1990s to 85 dB hearing loss in current years. Therefore, the boundary between
hearing aids user and cochlear implant candidate becomes obscure and interleaved.
According to census by World Health Organization in 2010, there are about
360 million hearing-impaired people worldwide, representing the second largest
group of handicapped people. This number grows annually as a result of 0.1–0.2%
rate of congenital deaf among newborns. In addition, consequential deaf from
disease, medicine intoxication, trauma, environmental noise, and genetic disorders
also contribute to the increase of deaf population. Traditionally, large amount of
resource is required to set up rehabilitation facilities for deaf people, yet the
effectiveness of these rehabilitations is very limited. They still live within the deaf
society and find it hard to communicate with the outside world. The appearance of
cochlear implants changes the situation fundamentally. Most modern cochlear
implant recipients can carry a normal conversation in a quite environment, although
they still find it challenging to communicate in a noisy environment.
The development of modern cochlear implant is a long and interesting journey,
including joint efforts from physiologists, physicians, and engineers. In 1957,
French physician Djourno and his colleagues induced successful hearing with
electrical stimulation in two totally deaf patients. The news spurred an intensive
level of research activities worldwide to restore hearing to deaf people. In 1961,
William House in Los Angeles, California, implanted a single-channel cochlear
implant in the scalar tympani of two deaf people. Both subjects report useful hearing
from electrical stimulation although the initial devices only lasted 2 weeks. In 1978,
Graeme Clark in Australia developed a multi-channel cochlear implant and
implanted in two deaf people. Although single-channel cochlear implant won the
first round as the first FDA-approved device for clinical use, multi-channel device is
the standard for modern cochlear implants because of its superior performance.
A cochlear implant usually includes an internal implant device and an external
sound processor, as shown in Fig. 7.1. The external sound processor (1) powered by
battery pack (2) picks up sound through microphone and processes and encodes sound
signal, which is then transmitted wirelessly by transmission coil (3) to the receiver coil
(4) of internal device. The current stimulator (5) of internal device converts the
received signals into electrical impulses and sends the impulses through electrode
body (6) to an electrode array (7). The electrical impulses from the electrode array
(7) stimulate the auditory nerves (8), allowing the brain to perceive sound.
Now there are four major cochlear implant providers in the world: Cochlear
Corporation in Australia (www.cochlear.com), Med-El Corporation in Austria
(www.medel.com), Advanced Bionics Corporation in the United States (www.
advancedbionics.com), and Nurotron Biotechnology Inc. in China (www.nurotron.
com). There are a couple of smaller cochlear implant companies with no significant
market shares. They all provide multi-channel devices with 12–24 electrodes, while
Nurotron device has the most intra-cochlear electrodes of 24. Functions like virtual
channel, electrode impedance measurement, and neural response telemetry also
become standard for modern cochlear implant devices. The recognition rate of short
sentences in quiet is around 80% for the products of all four major manufactures.
7 Neural Interface: Frontiers and Applications 169

Fig. 7.1 External device of cochlear implant system includes (1) sound processor, (2) battery pack,
(3) transmission coil. Internal device of cochlear implant system includes (4) receiver coil, (5) cur-
rent stimulator, (6) body of electrode, (7) electrode array, which stimulator auditory nerves (8) with
current pulses

7.2 System Specification

The ultimate goal of cochlear implant is to restore effective hearing for severe to
profound hearing-impaired people through safe electrical stimulation of auditory
nerve. To this end, the design and manufacture of a cochlear implant system should
satisfy requirements in three aspects. First, the material of the implant device should
be safe for long-term implantation. Second, the circuit of the implant should be able
to deliver safe, reliable, and accurate current stimulation. And last, the structure of
the implant should consider surgical requirement and provide enough protection for
electronic components. The architecture of a modern cochlear implant system is
shown in Fig. 7.2, consisting of a speech processor, a RF unit, an internal unit, an
electrode array, a remote controller, and a fitting unit [76].

7.2.1 Speech Processor

The speech processor shown in Fig. 7.3 is the core of the external part of the cochlear
implant system. The entire speech processor is powered by a battery block. The
clock block provides clock signal to all blocks except the power amplifier and the
reverse telemetry decoder. The audio signal from natural environment or other
170 X. Sun et al.

Skin

Fitting
Unit

RF
Unit

Audio Speech Internal Electrode Auditory

Signal Processor Unit Array Nerve

Remote
Controller

Fig. 7.2 Typical architecture of a modern cochlear implant system

Fitting
Unit

Speech Clock IO I2C

Processor Controller Module

Audio Signal Digital Reverse

Audio Sampling Pre-Process Signal Telemetry
Signal Module Module Processor Decoder

Wireless Battery Data Power RF

Tranceiver Modulator Amplifier Unit

Remote
Controller

Fig. 7.3 Block diagram of speech processor

7 Neural Interface: Frontiers and Applications 171

electrical devices (TV, radio, tele-coil, etc.) is first sampled, digitized, and synchro-
nized in audio sampling module. Processed by signal pre-processing module and
digital signal processor (DSP) based on certain encoding strategies, the output of
audio sampling module is converted to data that contains stimulation information.
Then the data are modulated at data modulation block by radio frequency
(RF) carrier signal and sent to power amplifier (PA). The amplified signal is
eventually sent to RF unit. The reverse telemetry decoder detects and modulates
the signal from the RF unit when the system is in the status of reverse telemetry. The
decoded signal goes into the DSP for further operation. The I2C module and wireless
transceiver, which are both controlled by the DSP, are used to communicate with the
fitting unit and remote controller, respectively. The IO controller controls indication
lights and buttons on the speech processor.

7.2.2 RF Unit

The RF unit is composed of a transmitter coil and a receiver coil connected with their
resonant circuits, which are on the outside and inside of human body, respectively.
Through this inductive link, the stimulating information as well as power is trans-
mitted into the internal unit to control the unit to stimulate the auditory nerve. More
interestingly, using the reverse telemetry technique, the data from the implant can be
sent back to the speech processor with the same link.

7.2.3 Remote Controller

The remote controller is used to execute some regular operation, for example,
turning on and off the system, adjusting the sound volume, changing fitting maps,
and so on. In some applications, it could also be used to check the status of the
speech processor and the internal unit. Most of the recent cochlear implant products
favor this wireless control unit because it is much more convenient for users or
guardians of users to manipulate the device instead of pushing buttons on the speech
processors.

7.2.4 Fitting Unit

The fitting unit, which is not worn by users, is only used by clinicians for clinical
uses, e.g., impedance test, neural response measurement, fitting program, etc. The
fitting unit can bi-directionally communicate with the speech processor. On one
hand, when clinicians want to optimize performance of devices, different stimulating
172 X. Sun et al.

information can be sent to speech processors from fitting programs. On the other
hand, when they need to detect the condition of implants, the internal data from
implants can also be sent back to the unit.

7.2.5 Internal Unit

The block diagram of the internal unit is shown in Fig. 7.4. The RF signal from the
RF unit is divided into two paths. One goes into the power generator to provide a
high voltage for the stimulating circuit and power supply for all other internal
circuits. The other path goes to RF signal demodulator. If the RF signal is a
command signal, the demodulator sends the signal to the command decoder,
which stores the decoded information in the setting register. If the RF signal is a
data signal, the data decoder is activated, and the stimulating data is sent to the data
distributor. The output of the demodulator is also the source of the clock generator,
which is basically a clock recovery circuit to provide a stable clock signal for other
blocks. The stimulating circuit is controlled by data distributor that decides cur-
rent amplitude and electrode address and by the timing controller that controls the
stimulating time. The stimulating current finally flows from the stimulating circuit to
the electrode array. The recording circuit is used to record the voltage signal in the
electrode, which is converted to the digital signal and transmitted back to RF unit
through the reverse telemetry modulator.

7.2.6 Electrode Array

The electrode array is a major part of the implant, which is inserted into the cochlea.
The stimulating current pulse goes out from the stimulating electrode to activate the
neurons and flows back to the return electrode in the end. To avoid the DC voltage
shift accumulation, normally the current pulse has two phases with positive and
negative current as shown in Fig. 7.5.

7.3 Signal Processing Strategies

7.3.1 Introduction

Speech processor can be described as the brain of a cochlear implant. Acoustic

signals are processed and encoded into specific parameters of electric pulses using
particular signal processing methods, termed as “signal coding strategy”.
In the CI system, the signal coding strategy plays an extremely important role in
generating auditory perception of CI recipients (e.g., [72, 74]), which determines
7 Neural Interface: Frontiers and Applications 173

Fig. 7.4 Block diagram of internal unit

Fig. 7.5 A typical biphasic stimulating pulse using the return electrode as reference
174 X. Sun et al.

which electrodes should be activated in each cycle according to the input acoustic
signals. Many stimulating strategies have been developed over the past two decades.
An ideal stimulating strategy is one that allows a CI user to perceive sounds clearly
and accurately.
All modern multi-electrode CIs have been developed to take advantage of the
so-called tonotopic organization in the cochlea that the apical part and the basal part
of the cochlea encode low-frequency and high-frequency sounds, respectively. In
these types of CIs, a bank of filters is employed to divide speech into different
frequency bands, but they differ in processing and encoding acoustic signals into
electric pulses.

7.3.2 Formant-Based Strategies

One group of coding strategies was based on the idea of speech production and
perception, in which spectral peaks or formants are extracted and delivered to
different electrodes. The F0/F2 strategy was first developed and implemented in
the Nucleus Wearable Speech Processor [16]. In this strategy, the fundamental
frequency (F0) and the second formant (F2) are extracted from the speech signal
in each frame. One zero-crossing detector is used to estimate F0 from the output of a
low-pass filter, which has a cutoff frequency of 270 Hz, and another zero-crossing
detector is used to estimate F2 from the output of a 1000–4000 Hz band-pass filter.
The amplitude of F2 is then estimated by rectifying and low-pass filtering the output
signal. The schematic diagram is shown in Fig. 7.6.
The addition of the first formant (F1) information was found to improve the
speech recognition performance of patients in speech perception tests. In fact, both
F1 and F2 were reported important in vowel recognition for normal-hearing listeners
[24]. Dowell et al. [24] found that the average scores on word recognition increased
from 30% to 63% after switching from F0/F2 to F0/F1/F2 strategy.
Further improvements to the F0/F1/F2 strategy were made in the late 1980s,
including the involvement of high-frequency information. MULTIPEAK
(or MPEAK), a new coding strategy developed upon these changes, was used to
extract both formants and high-frequency information from the speech signal
(as shown in Fig. 7.7). The MPEAK strategy stimulates four electrodes at a rate of
F0 pulses/sec (pps) for voiced sounds, and at an average rate of 250 pps for unvoiced
sounds [57].
Although the MPEAK strategy has proved to be an efficient strategy for
extracting important information from the speech signal, the performance is still
limited. The MPEAK strategy and F0/F2 and F0/F1/F2 strategies tend to make errors
in formant extraction, especially when the speech signal is embedded in noise.
However, all the attempts made in the improvements of formant extraction methods
motivated the development of the next generation coding strategies for CIs.
7 Neural Interface: Frontiers and Applications 175

Fig. 7.6 Block diagram of F0/F2 strategy

Fig. 7.7 Block diagram of MPEAK strategy

7.3.3 Temporal Envelope-Based Strategies

Another thought of spectrum regeneration with electric stimulation was based on the
implicit encoding of temporal envelope cues and could be traced back to earlier work
in speech vocoder [25]. Most of algorithms implemented in current commercial
devices origin from this idea.

7.3.3.1 Continuous Interleaved Sampling (CIS) Strategy

For modern multi-channel cochlear implants, which have M (e.g., 12–24) implanted
electrodes, CIS is a strategy used in the speech processors of all major cochlear
implant manufacturers [72]. CIS is a multi-band coding strategy, in which the
temporal envelope of signal in each band is extracted at a cutoff frequency of
176 X. Sun et al.

Fig. 7.8 Block diagram of CIS strategy

160–400 Hz. For each frame of sound, the signal is applied through M band-pass
filters, and the envelopes of these frequency bands are extracted. The envelope also
has to be compressed, usually logarithmically, to match the narrow dynamic range in
electric stimulation. Then, all M frequency bands are stimulated in a
non-overlapping sequence at a constant rate. In this pattern, the stimulation rate of
each channel is relatively high and the interaction across electrodes can also be
reduced. Meanwhile, the original spectrum of received sound is regenerated by M
channels and one channel of the signal is produced by the stimulation of one
electrode. The schematic diagram of CIS is shown in Fig. 7.8. According to Shannon
et al. [56], four spectral bands in CIS processing can deliver acceptable speech
intelligibility in quiet.

7.3.3.2 n-of-m Strategies

A group of coding strategies based on channel selection or “spectral peak picking”

scheme is represented by “n-of-m strategy” (Fig. 7.9), which is designed in part to
reduce the density of stimulation while still reserving the most important aspects of
speech signals [38]. In each cycle, only N channels with largest amplitudes are
selected from all M channels and stimulated. The deletion of stimulation from
low-energy channels for each frame of sound may reduce the interference across
electrodes and restrict the area of stimulated region in the cochlea. Taking the fact
into consideration that low-level channels usually do not contain significant infor-
mation, such “unmasking” may improve the perception of sounds by patients. In
addition, selection of the channels containing higher energy in each frame may
enhance the signal-to-noise ratio in signal domain situations.
7 Neural Interface: Frontiers and Applications 177

Fig. 7.9 Schematic diagram of n-of-m strategy

The ACE and SPEAK strategy (differ in stimulation rate), used in the Nucleus
implant, consists of 22 fixed channels. The input signal is divided into 22 frequency
bands for each frame just as CIS strategy. After the envelope information for each
frequency band is extracted by rectifying and low-pass filtering, 6–10 frequency
bands with the largest amplitudes are stimulated. The number of stimulated channels
could be set by audiologists. Electrodes corresponding to the selected channels are
then activated, and the original spectrum is reproduced by 6–10 fixed channels [49].
Advanced peak selection (APS) is a coding strategy employed by the Nurotron
device [76]. The APS strategy can also be defined as an “n-of-m” strategy, with the
number of frequency bands being 24. The number of channels being stimulated in
each frame is typically set to 6–8, dependent on the number of available electrodes or
the feedback from patients. In the APS strategy, stimulation rate can be adjusted to
improve the representation of temporal fine structure. The typical stimulating rate of
each selected electrode is normally set to 680 pps or 890 pps.

7.3.4 Coding Strategies with Advanced Cues

As described above, both n-of-m and CIS strategies only employ a certain number of
channels to reproduce the original sound spectrum, which means only 12–24
electrodes can currently be implanted into a CI user’s ear to generate 12–24 fixed
channels. However, the population of auditory nerve fibers in a human cochlear is
around 30,000. Due to the limitation of the electrode number and design, the spectral
resolution of a CI user is normally worse than that of a normal-hearing listener,
which severely weakens their pitch and tone perception. Therefore, a school of new
coding strategies designed to improve pitch or tone perception of CI users has been
proposed in recent years.
178 X. Sun et al.

7.3.4.1 HiRes120 Strategy

Another alternative method is virtual channel strategy, in which the electrical stimu-
lation pattern is shaped by current steering technique [22, 41]. To implement the
virtual channel technique in a cochlear implant system, each electrode must have an
independent power source to allow more than one electrode to be stimulated simulta-
neously. Theoretically, with simultaneous stimulation of neighboring electrodes, the
locus of stimulation is controlled between fixed channels, creating virtual channels.
The HiRes120 strategy, used in the Advanced Bionics implant, is the first
commercial virtual channel strategy [40]. Since the Advanced Bionics implant has
16 implanted electrodes, there are 15 electrode pairs that can be used to steer the
electrical stimulation and 8 sub-channels within each electrode pair, generating
120 channels in total. Figure 7.10 is a diagrammatic representation of the
HiRes120 strategy. Similar to CIS strategy, in each frame of sound, the signal is
divided into 15 frequency bands and the envelope of each band is extracted. Next,
15 spectral peaks are also derived, which indicate the most important frequency
component within each channel. These spectral peaks are then steered by
corresponding electrode pairs based on the virtual channel technique. However,
according to results from clinical tests, HiRes120 strategy did not achieve expecta-
tion of researchers.

7.3.4.2 C-Tone Strategy

CI signal processing significantly affects lexical tone recognition due to the restricted
spectral resolution of CI users, which is to the disadvantage of overall speech
understanding of tonal languages (e.g., [30]). One way to improve tone recognition
rate is to deliver information other than temporal envelope. Since F0 has been found
to carry most of the lexically meaningful information for tonal languages, Luo and
Fu [47] evaluated a novel method to improve Chinese tone recognition in which
amplitude contour was modified to better resemble the F0 contour. In this study,
results showed significantly better tone recognition with the amplitude contour
modification. An amplitude contour manipulation algorithm (C-tone) that originate
from this idea was implemented in Nurotron implant system to challenge the
difficulties in lexical tone recognition of CI users [51].
C-tone was implemented as a tone-enhanced version of APS strategy as shown in
Fig. 7.11. A voice activity detector (VAD) is applied at the frontend of the algorithm
process. In each frame, the first step of C-tone is to estimate the F0 with the average
magnitude difference function (AMDF) algorithm [54]. Second, the input signal is
passed through a similar process as APS strategy. Finally, the amplitude envelope of
each selected stimulation channel is adjusted based on the relative change of F0
comparing to that estimated from the previous frame. An increment in F0 results in
an enhancement in the amplitude envelope and vice versa. The preliminary data
suggest that real-time enhancement of the amplitude envelope contour in C-tone
may benefit Chinese CI users’ tone recognition.
7 Neural Interface: Frontiers and Applications 179

Fig. 7.10 Block diagram of HiRes120 strategy

Fig. 7.11 Block diagram of C-tone

7.3.4.3 Fine Structure Processing (FSP) Strategy

New processing options have also been introduced to increase the transmission of
fine structure information compared with the CIS and other n-of-m strategies in
current use. In one approach, the timing of positive zero crossings in the output of the
four band-pass filters with the lowest center frequency dominates the presentation of
a short group of pulses for the corresponding channels of stimulation as opposed to
180 X. Sun et al.

the continuous presentation of pulses for standard CIS channels. This strategy is
called the “fine structure processing” (FSP) strategy [55], applied in MED-EL’s
implant. In FSP, signals in low-frequency channels and in high-frequency channels
are treated and processed differently. In low-frequency channels, the amplitude of
output signal is determined by the total energy in this band. In high-frequency
channels, however, signal is processed similar as that in CIS strategy.
The FSP and related approaches may provide an advantage compared with CIS
and other envelope-based strategies with the delivery of fine structure information in
the lower channels. With the help of additional fine structure information, FSP has
been reported to enhance music or speech perception of CI users [2].

7.3.5 Pre-processing Techniques

Despite the good performance of CI in quiet, there are still significant gaps in
performance between normal-hearing people and CI users. For example, the perfor-
mance of CI users in speech perception is extremely poor with additive noise. At
least a 15-dB loss in functional signal-to-noise ratio (SNR) could be produced in a
steady-state noise background [75]. In a study by Firszt et al. [28], speech recogni-
tion was assessed using the hearing in noise test (HINT) sentences. Results revealed
that CI recipients’ performance on sentence recognition tasks was significantly
poorer in noise compared with when listening at speech stimuli in quiet at a soft
level. In Spahr and Dorman [61]‘s study, it was reported that the average speech
intelligibility performance of CI recipients is around 40% in tasks involving con-
versational speech presented at 10 dB SNR. The average recognition of conversa-
tional speech dropped to around 20% when the SNR level was lowered to 5 dB.
Fetterman and Domico [27] had similar finding in their study, in which CI users were
asked to identify speech in noise. Average sentence recognition scores of CI users
decreased from 82% in quiet to 73% at 10 dB SNR and to 47% at 5 dB SNR.
The poor frequency selectivity of hearing-impaired listeners has been reported as
a significant reason why they cannot distinguish speech signal from the noise as
normal-hearing listeners [64]. Recent research efforts have been focusing on state-
of-the-art noise reduction solutions much more to improve speech intelligibility in
noisy environments. Since CIs deliver electrical pulses to stimulate auditory nerves
to help restore hearing sensation, multiple signal processing algorithms have been
applied to convert acoustic signals into electrical stimuli (e.g., [46, 74]). As we
mentioned above, the majority of CI users can score high in open-set speech
recognition tasks regardless of the device or speech coding strategy used (e.g.,
[58, 61]). However, few of them could overcome the problems of additive noise.
Many of current noise reduction algorithms can improve output SNR, but few of
them can really enhance the speech intelligibility (e.g., [44]). The generated speech
distortion has been considered as one of the reasons. A traditional method to reduce
the effect of noise is to apply a gain calculated from estimated SNR level to suppress
the noise. However, since the power of noise cannot be accurately estimated, the
7 Neural Interface: Frontiers and Applications 181

speech signal can be either amplified or attenuated due to the underestimation or

overestimation to the noise power, respectively. Considerable speech distortion is
then introduced [39], resulting in no or even negative benefits in speech
intelligibility [33].
In general, noise reduction algorithms designed for CIs could be divided into two
classes: single-microphone and multi-microphone methods. Single-microphone
approaches rely mostly on statistical models of speech and noise, and therefore
can only remove noise with different temporal and spectral features as speech
signals. In recent years, there has been a growing tendency toward the use of noise
reduction methods with multiple microphones in CI devices (e.g., [42, 73]). Large
improvements in SNR and therefore considerable benefits in speech intelligibility
can be obtained, only if the target speech and the noise sources are located at
different locations in space. In the following session, both single-channel and
multi-channel noise reduction techniques are reviewed.

7.3.5.1 Single-Channel Noise Reduction Methods

One of the first proposed single-channel noise reduction algorithms is spectral

subtraction, based on a maximum likelihood (ML) estimator, and has been
implemented in numerous applications. Its efficiency and low computational com-
plexity have resulted in its widespread use. The general idea of spectral subtraction is
to estimate the spectrum of noise during gaps in the speech, and then to remove it
from the noisy signal [66]. The performance of this class of algorithms depends
critically on the accuracy of noise estimation. A conventional method to estimate the
noise is to update the noise power cumulatively during speech gaps and hold it
unchanged during speech frames. To determine when speech is present or absent, a
voice activity detector (VAD) is required [4].
Despite its attractive simplicity, spectral subtraction has a number of shortcom-
ings, which limit its use in applications such as hearing aids and CIs. One commonly
audible artifact produced by the technique is termed “musical noise” and is generated
primarily by inaccurate estimation of the noise spectrum. Since only the average
power of the noise can be estimated over time, short-term, potentially important
details, such as momentary spectral peaks and valleys, are ignored by the algorithm.
As a result, after the estimated average noise is removed, those residual components
can produce annoying tonal sounds, which can worsen, rather than improve, speech
intelligibility and perceived quality.
Over the past three decades, much effort has gone into developing methods that
remove or reduce musical noise [17, 52]. A common technique is to set up a noise
floor, that when the signal level is under a certain threshold, it will be left
unprocessed [4]. A factor could be used to control the strength of this technique,
that how much the signal should be removed when its level is under the threshold.
Moreover, it has been noticed that the influence of noise on speech signal may not be
unified; it is rational to apply different control factors at different frequencies.
182 X. Sun et al.

Wiener filtering is another well-studied technique in speech enhancement, based

on optimal minimum mean square error (MMSE) estimator of each speech spectral
component [13, 45]. Martin [48] developed an algorithm based on the combination
of Wiener filtering and spectral subtraction to overcome the limitation of VAD, so
that in a speech frame, noise information could still be updated by a Wiener filter.
Extended Wiener filters have been proposed to further enhance the performance of
noise reduction algorithms. Multi-channel Wiener filtering, for example, has been
tested [68, 69]. In addition, the speech distortion weighted Wiener filter was
developed to reduce speech distortion, in order to maintain speech intelligibility
while cleaning speech [21, 62].
Most commercial CI devices employ single-channel noise reduction algorithm
due to its simplicity and efficiency. A recent proposed technique named as eVoice
applied in Nurotron speech processor, which can adapt to the external SNR condi-
tions so as to generate stable auditory perception, was evaluated with speech
perception test of CI users. Results are exhibited in Fig. 7.12. The percent of correct
rates of subjects using APS strategy were compared with the performance of the

Fig. 7.12 Results of speech perception test in evaluating the effects of eVoice. Notes: APS
represents using APS strategy along and eVoice indicates using eVoice as the pre-processing
technique prior to the application of APS strategy. SSN and Babble represent speech-shaped
noise and babble noise, respectively
7 Neural Interface: Frontiers and Applications 183

same group of subjects when using APS and eVoice together. In summary, eVoice
improves speech intelligibility of CI users in 5- and 10-dB SNR conditions in the
presence of both speech-shaped noise and babble noise. Meanwhile, most subjects
reported the preference of using eVoice in noisy environments.

7.3.5.2 Multi-Channel Noise Reduction Methods

Cocktail party problem [14] as one of the most challenging tasks for hearing-
impaired listeners has been attempted to be solved via multi-channel algorithms.
The crucial ability of completing this task is to separate the mixture of target speech
and interfering noise or speech. One group of methods is blind source separation
(BSS), which does not require the prior knowledge of microphone array and spatial
information of target speech (e.g., [70, 71]). Information-based criteria are usually
employed in such methods. A representative work was done by Douglas and Sun
[23], using convolutive blind source separation with natural gradient algorithm to
automatically resolve mixed recordings from the real environment. In this study, it
was found that the modified BSS technique with linear prediction filters was able to
complete the task of separating the mixture of two-channel speech signals.
Beamforming is another group of multi-microphone spatial enhancement tech-
nique via source separation and works by exploiting phase differences in the signals
arriving at two spatially separated microphones from a particular direction.
Beamformers can maximally attenuate noise at a null point whereas reserve the
target speech in front (Fig. 7.13). This pattern of directionality can either be fixed to
generate a static pattern of directionality or be adaptively updated to suppress
moving noise sources. In order for beamforming to work, a number of assumptions
typically have to be achieved: the target should always be located within a narrow
range on angles in front of the listener and the noise source should be spatially
separated from the target, meanwhile, the two microphones have to be well matched
and calibrated.
Over the past two decades, a few beamforming techniques have been used to
enhance speech perception of hearing-impaired listeners in noise. In a CI or hearing
aids, the combination of two microphones can detect the differences in amplitude
and phase of received signals and then generate the directionality of beam pattern to
block noise coming from directions other than that of the target.
In 2005, BEAM strategy, a modified dual-microphone technique was
implemented in the behind the ear (BTE) speech processor in Cochlear’s Nucleus
Freedom CI system. In BEAM, the first step is implemented through a spatially
adaptive system that integrates the front directional microphone (which is changed to
an omnidirectional microphone later in Nucleus 5 processor) and a rear omnidirec-
tional microphone to separate speech from noise. The output from the rear omnidi-
rectional microphone is firstly filtered through a fixed finite impulse response (FIR)
filter and then subtracted from the delayed output from the front directional
184 X. Sun et al.

Fig. 7.13 A typical beam

pattern of beamforming
technique. Note: 0
and
180
represent front and
back, respectively

microphone to create the noise reference [73]. The filtered signal from the omnidi-
rectional microphone is then added to the delayed signal from the directional
microphone to create the speech reference. The reference signals are then delivered
to an adaptive filter to remove additional noise, which can be updated with a least-
mean-squares (LMS) or a normalized least-mean-squares (NLMS) method. This
bilateral noise reduction system was tested and showed an average SRT improve-
ment of 11 dB over a directional microphone.

7.4 Wireless Signal Transmission

In this section, we mainly focus on the analog circuit of cochlear implant system, and
several critical analog blocks of wireless signal and power transmission are
discussed in detail in the following parts.
As mentioned in the previous section, the internal unit of the cochlear implant
system is powered by the inductive link for several considerations [76]. First, the
skin thickness of the implanting location is about 3–10 mm, where the wireless
transfer efficiency is still acceptable. Second, the expected implanting duration for
the cochlear implant is more than 10 years, and it is difficult to use the implanted
battery to sustain such a long period, because unlike pacemakers, the cochlear
implant has to stimulate the nerve all the time so that its power consumption is
relatively large. Third, both surgeon and patients request a small implanted device
for easy operation in the surgery and gentle appearance from outside to see.
7 Neural Interface: Frontiers and Applications 185

VDD Skin

L1
C2 D1
Node 1
M1
C1 L2 L3 C3 C4 R1

GND1 GND2

Fig. 7.14 A typical class E power amplifier with inductive link

Figure 7.14 shows the basic schematic of a power amplifier with an inductive
link. The class E power amplifier drives the primary RF coil L2 that sends power
inductively to the secondary RF coil L3. C2 and C3 are used to be resonant in series
and in parallel with L2 and L3 at the working frequency, respectively. The RF signal
on L3-C3 tank then is rectified to provide the power for the internal unit and
stimulating electrodes. M1 is the main power transistor for the class E amplifier,
which is usually controlled by the modulated signal. L1 is the choke inductor to
provide supply current, and C1 is used to adjust the waveform of node 1 to make sure
that M1 is on right at the moment when the voltage of node 1 is close to ground
shown in Fig. 7.15a [60].
However, due to the variability of patient skin-flap-thickness and stimulating
current, the equivalent load of the power amplifier varies. Therefore, in many cases
as shown in Fig. 7.15b and c, the class E amplifier does not operate perfectly and
the entire power transfer efficiency reduced. To solve this issue, many researches
have been done. In Fig. 7.16a, the peak detector is used to find the zero crossing of
current going through L2, and M1 is trigger by adding a delay tdelay, which can be
expressed as:



T 1 idd
tdelay ¼ π  sin ð7:1Þ
2π iL2, P

where T is the period of the input signal; idd is the supply current from L1; iL2, P is the
peak current going through L2 [37]. Figure 7.16b shows another method that tries to
achieve zero-voltage switching condition [3]. In this technique, when Vd is lower
than the reference voltage, M1 is on and the switch losses are minimized. Both
techniques show good results with different distances between the primary and
secondary coils. However, these proposed circuits could consume additional
power, especially because they are running at a high speed. In some cases, the
saved power could be less than the power these circuits dissipate.
186 X. Sun et al.

(a)

(b)

(c)

Fig. 7.15 Waveforms of node 1 when the coupling factor of the inductive link varies

VDD

L1
C2

M1
Pulse C1 L2
Generator

GND1
Delay Peak
Line Detector

(a)

Comparator VDD
VREF
+
- L1
C2
Vd

One Shot Fast Latch

SET C1 L2
Circuit
CLR
M1

CLK GND1

(b)

Fig. 7.16 (a) Using peak detector to find the zero crossing of current going through L2. (b) Using
comparator to guarantee Vd is near ground when M1 is on
7 Neural Interface: Frontiers and Applications 187

The efficiency of the inductive link is determined by the coupling coefficient k,

the quality factor of L2 Q2, the quality factor of L3 Q3, and the effective quality
factor of load network QL and can be simplified as:
!

k2 Q2 Q03 Q3
ηLink ¼ ð7:2Þ
1 þ k 2 Q2 Q03 Q3 þ QL

where Q03 ¼ Q3 QL =ðQ3 þ QL Þ: From (7.2), we can clearly see that with a higher
coupling coefficient, the transfer efficiency of the inductive link increases. It means
that the efficiency increases with thinner skin thickness. Also, Q2 should be as large
as possible to increase the efficiency. Normally k and QL varies in different circum-
stances so that it is difficult to find a determined Q3 to maximize the efficiency.
Moreover, the system usually uses on-off keying modulation, which means that the
input signal of the amplifier is not a continuous signal. The complexity of the design
needs to be more carefully considered.

7.5 ASIC Design

Stimulating auditory nerves and recording neural response are two main functions of
the internal unit in a cochlear system. To reduce dimension and power consumption
of implant, the internal circuit is implemented with application specific integrated
circuit (ASIC). Usually, the function blocks inside the dashed border of Fig. 7.4 is
achieved by ASIC.

7.5.1 Current Source

In Fig. 7.17, a current mode stimulating circuit is shown [77]. Twenty-four stimu-
lating electrodes are connected to node E1, E2, . . . E24, respectively. MP1 and MP2
are two reference electrodes. Two types of current sources, P-source and N-source,
can be selected by controlling switch S5 and S6. The reason why current mode is
chosen for stimulating circuit is because the charge injected into tissue could be
precisely controlled. To clearly describe the mechanism of this circuit, choose
electrode 1 as the stimulating electrode and MP1 as the reference electrode. Mean-
while, make S5 closed and S6 open to select P-type current source. In the cathodic
phase, S2[1] and S3[1] are closed, and the current flows from MP1 to E1. In the
anodic phase, S1[1] and S4[1] are closed and the current flows from E1 to MP1.
Stimulating charge balance is one of the most important factors to be considered
in the circuit design. The tissue damage does not occur in chronic stimulation of
cochlear implants if residual DC current is lower than 100 nA [34]. In this biphasic
stimulation as explained above, the positive and negative charge going through the
188 X. Sun et al.

Fig. 7.17 Current mode

stimulating circuit
S6 P-Source
S1[24:1]

S2[2:1]
C1 C2 C24
Tissue MP1
E1 E2 E24 MP2

S4[2:1]

S3[24:1]
S5 N-Source

tissue should be balanced ideally, since both phases utilize the same current source.
To further suppress imbalanced DC charge, a DC blocking capacitor, Cblock, in series
with each electrode is usually inserted as shown in Fig. 7.18. Moreover, when
stimulation is completed, switches S7 and S8 are closed in Fig. 7.18 to guarantee
that the stimulating and reference electrodes are on the same electrical potential [15].

7.5.2 Recording Circuit

A robust recording circuit is the key to successfully capture the electrically evoked
compound action potential (ECAP) signal. As shown in Fig. 7.19, the recording
circuit normally is composed of four blocks [12]. The targeted signal is received by
the sensing electrode, and then transmitted to the band-pass filter to remove
undesired noise. The variable gain amplifier (VGA), whose gain is set according
to different circumstances, is used to amplify this signal, because the amplitude of
this type of signals is only several millivolt or even smaller. At last, an analog-to-
digital converter (ADC) is employed to measure this signal and send the data out.
Mismatch compensation is a critical technique for VGA design, since the ampli-
tude of the input signal is comparable with the input offset of the operational
amplifier. In Fig. 7.20, a typical VGA circuit with mismatch compensation is
shown [53]. In this circuit, the first stage amplifier consists of Gm1, Gm2, and R,
and A1 is the second stage amplifier. Switches S7, S8, S9, and S10 are used to
change the gain of amplifier A1. The rest of switches are used for mismatch
compensation of the first stage. In detail, S1, S2, S5, and S6 are turned on first. At
7 Neural Interface: Frontiers and Applications 189

Fig. 7.18 Stimulating VDD

circuit with charge
balancing techniques
S6 P-Source

S1 S2
Tissue
S7 Cblock S8
Stimulating Return
Electrode Electrode
VSHORT VSHORT
S3 S4

S5 N-Source

GND

ECAP
Signal
Sensing Bandpass SAR N-bit Digital
Electrode Filter VGA Output
ADC

Fig. 7.19 A block diagram of neural response recording circuit

this moment, only the input offset VOS is amplified by the first stage. Then S3 and S4
are also turned on, and the output of the first stage VOUT becomes:

gm1
V OUT ¼ V OS ð7:3Þ
gm2

where gm1 and gm2 are the trans-conductance of Gm1 and Gm2, respectively. The
input offset information is stored in C3 and C4 by turning S1, S2, S3, S4, S5, and S6
off. Therefore, the offset of the first stage is cancelled by the output of Gm2.
Low power, small area, and high acquisition precision are three major factors for
ADC design in the cochlear implant, which should be considered carefully. Since the
neural response is relatively slow (~KHz), the speed of this circuit is not needed to be
very high. Based on the discussion above, the successive approximation (SAR)
ADC is one of the most suitable candidates for ADC design in the internal unit
[12]. Figure 7.21a shows a 12-bit fully differential SAR ADC circuit. The basic
theory is using the output of a 12-bit DAC to compare with the input signal
[19]. After comparing 12 times, the ADC finally gets all the digital output. Unlike
flash ADC and pipeline ADC, SAR ADC only needs one comparator so that both
 VINN
VCOM
Vin-

VINP
Vin+
190

VCOM

30
30
PH_SMP PH_SMP

1
1
PH_HOLD PH_HOLD

C2
SP<0> SN<0>
C1

S1
VREFM VREFM
SN<0> SP<0>

1
1
VREFP VREFP
PH_SMP PH_SMP

PH_HOLD

V
PH_HOLD
SH<0> SH<0>

S2
SP<1> SN<1>
VREFM VREFM
SN<1> SP<1>

2
2
VREFP VREFP
PH_SMP PH_SMP

Vcm1
Gm1

SH<1> SH<1>

SP<2> SN<2>
VREFM VREFM
SN<2> SP<2>

4
4
VREFP VREFP
PH_SMP PH_SMP

SH<2> SH<2>

LSB_5
SP<3> SN<3>
VREFM VREFM
SN<3> SP<3>

8
8
VREFP VREFP
R

PH_SMP PH_SMP

SH<3> SH<3>

DAC_P

DAC_N
SP<4> SN<4>
VREFM VREFM
Gm2

12
SN<4> SP<4>

16
16
VREFP VREFP
PH_SMP PH_SMP
C3

SH<4> SH<4>

PH_AZ PH_AZ
VCOM VCOM

2
2

SAR Logic
PH_AZ PH_AZ

(a)

(b)
VCOM VCOM
SP<5> SN<5>
VREFM VREFM

1
SN<5> SP<5>
C4

1
VREFP VREFP
S4
S3

PH_SMP PH_SMP

Digital Output
Control Logic
SH<5> SH<5>

SP<6> SN<6>
VREFM VREFM
SN<6> SP<6>

2
2
VREFP VREFP
C6
C5

PH_SMP PH_SMP
S5

SH<6> SH<6>
Vcm2

SP<7> SN<7>
VREFM VREFM
SN<7> SP<7>

4
4
VREFP VREFP
S6

PH_SMP PH_SMP
MSB_6

SH<7> SH<7>

SP<8> SN<8>
VREFM VREFM
SN<8> SP<8>
R2
R1

8
8
VREFP VREFP
PH_SMP PH_SMP
S8
S7

Fig. 7.20 Variable gain amplifier circuit with mismatch compensation

SH<8> SH<8>
PH_AZ
PH_SMP
PH_HOLD

SP<9> SN<9>
VREFM VREFM

t
SN<9> SP<9>
16
16
R4
R3

VREFP VREFP
PH_SMP PH_SMP
S9

S10

SH<9> SH<9>

CLK
SP<10> SN<10>
VREFM VREFM
SN<10> SP<10>
R6
R5

32
32

+
VREFP VREFP

Comparator
PH_SMP PH_SMP
A1

SP<N>:SW VREFP <N>

SN<N>:SW VREFN<N>
SH<N> SW HOLD<N>

SH<10> SH<10>
DAC_P

DAC_N

Fig. 7.21 (a) A 12-bit SAR ADC circuit. (b) Simulation waveforms of DAC_P and DAC_N
Vout-
X. Sun et al.

Vout+
7 Neural Interface: Frontiers and Applications 191

power and area are saved. Figure 7.21b shows the simulation results of DAC_P and
DAC_N. We can clearly see that the comparator compares two input signals one
stage by one stage in sequential way and the input of next stage is the subtraction of
the output of this stage and the output of the DAC.

7.5.3 Reverse Telemetry Circuit

In a cochlear system, the implant in some cases needs to reversely transmit data to
the speech processor, for example, neural impedance test data, neural response
measurement data, or indication of operation status. Load-shift keying (LSK),
which was referred as “reflectance modulation,” is a commonly used technique for
reverse telemetry [65]. The essence of LSK is the variation of the equivalent
impedance in the primary coil, which could be influenced by changing the load of
secondary coil. Therefore, this telemetry can be achieved using the same inductive
link with forward transmission. Fig. 7.22a shows a modulation circuit that is based
on a circuit configuration modulator [65]. When switch S1 is open, the circuit on the
implant side is a full-wave rectifier. When S1 is closed, the circuit becomes a voltage
clamp circuit. Using this method, the load of the secondary coil is successfully
modulated by controlling S1. In Fig. 7.22b, LSK is realized by changing the resonant
capacitor of the secondary coil [77]. This technique is especially effective for half-
wave rectifier circuit.
The demodulator circuit is used to sense and digitize the variation of the signal
amplitude on the external coil, which is modulated by the implant. Unlike other regular
amplitude-shift keying (ASK) demodulator, this circuit is suffered from variable signal
amplitudes, a shifting DC level, and an undetermined digitizing threshold due to the
variation of patient’s skin depth and the misalignment of the coils. Figure 7.23 shows
that a demodulation architecture, which is based on a compact CMOS integrated
circuit solution, is proposed to greatly enhance the transmission robustness, increase
the maximum bit rate, reduce the bit error rate, and save the board area [35].
The entire architecture is composed of four blocks: sensor, rectifier, amplifier, and
digitizer. An additional small coil (Fig. 7.24a) is introduced to pick up the modulated
signal to separate the forward and reverse telemetry and reduce the interaction
between each other. A half-wave rectification (Fig. 7.24 b) is implemented to filter
high-frequency components and remove the negative part of the signal. In this block,
the RC filter should be carefully designed to minimize the load of the additional coil
to alleviate its effect on the main coil and save power consumption. The amplifier
(Fig. 7.24c) is used to amplify the output of the rectifier. It has two modes: a normal
mode and a sleep mode. The amplifier is in the sleep mode while the processor is
transmitting regular data containing sound information to the implant. In this mode,
a switch that is connected between the input of the amplifier and a constant DC
voltage source is on and the DC level of the input is determined. Once the amplifier
goes into the normal mode, the switch becomes off, and the common mode voltage
of the input is still well defined. A high-pass feedback network is employed in the
192 X. Sun et al.

Fig. 7.22 Reverse Skin

telemetry circuit based on C1
(a) half-wave rectifier and
(b) full-wave rectifier D1 D2

V1 L1 L2 D4
C2 D3 S1 C3 R1

Digital
Output
Demodulator

(a)
Skin
C1 D1

V1 C3 C4 C5 R1
L1 L2

Digital
Output
Demodulator

(b)

Fig. 7.23 A block diagram of the demodulator

amplifier so that the voltage on the capacitor in the network is the integral of the
output signal. This voltage is used as a reference for the digitization block to
differentiate digital “1” and “0”. A Schmitt trigger (Fig. 7.24d) is implemented to
convert the amplified analog signal to a clean digital signal. To further suppress
random and deterministic noise, a hysteresis characteristic is utilized to avoid
7 Neural Interface: Frontiers and Applications 193

Fig. 7.24 (a) Sensor circuit. (b) Rectifier circuit. (c) Amplifier circuit. (d) Digitizer Circuit

undesirable data transition caused by a small disturbance near the threshold voltage.
Figure 7.25 shows the outputs of the sensor, rectifier, amplifier, and digitizer. We can
clearly see how the modulated signal is transformed to be digital data.
194 X. Sun et al.

Fig. 7.25 Waveforms of the outputs of four main blocks: blue, purple, green, and yellow wave-
forms are the output of the sensor, the rectifier, the amplifier, and the digitizer, respectively

7.6 Neural Response Measurement

7.6.1 History

Neural response measurement (NRM) in cochlear implants is to detect and measure

the electrically evoked compound action potential (ECAP) from the auditory nerve
in cochlear implant (CI) recipients. With the development of the multi-electrode
cochlear implant, there is a strong demand to request an easy and fast tool to give
objective information directly from devices and patients themselves to surgeons and
audiologists, because most of CI users are profoundly hearing-impaired people who
have difficulty to communicate with other people. The ECAP has been demonstrated
to be the best candidate, since the amplitude of the EAP could be proportional to the
number of auditory nerve fibers activated.
In 1970s and 1980s, many researches have been done to test the ECAP in
different animals [5, 11, 50, 63]. Compared with the acoustically evoked compound
action potential (CAP), which has been studied much earlier than ECAP, ECAP is
very difficult to be recorded, because the stimulating signal is always hundred or
even thousand times larger than the response signal. In 1990, Brown et al. firstly
used forward-masking subtraction method, which was originally proposed by
Charlet de Sauvage et al., to measure the neural response in the human beings
[6]. The results showed that the contamination of stimulus artifact on the recorded
response was greatly reduced and ECAPs were successfully detected and measured.
7 Neural Interface: Frontiers and Applications 195

Fig. 7.26 Neural response measurement (NRM) tool

Based on this technique, the first commercial ECAP measurement tool, whose
name was called “neural response telemetry” (NRT), was available in 1998. The
initial results showed that based on a carefully designed protocol, the ECAP can be
reliably recorded in most patients and most electrodes. After that, the similar tools
with different names (“neural response imaging,” “auditory nerve response teleme-
try,” “neural response measurement”) were issued by different brands of CI pro-
ducers. Nowadays, ECAP measurement tools (Fig. 7.26) become one of the most
useful tools to provide objective information of CI patients for clinical evaluation
from surgeons and audiologists.

7.6.2 Electrically Evoked Compound Action Potential

Essentially, the ECAP is the Wave I of the auditory brainstem response (ABR).
Once the auditory nerve is stimulated by an electrical signal, which is larger than a
specific threshold, the ECAP is generated. Figure 7.27 shows two types of ECAPs,
which were recorded with intra-cochlear electrodes in Nurotron Venus users. In
most cases, the ECAP, as shown in Fig. 7.27a, is composed of a negative peak
(N1) occurring at approximately 0.2–0.4 ms following stimulus onset and a much
smaller positive peak (P2) at approximately 0.6–0.8 ms [1, 6, 7, 18]. But two-peak
ECAPs as shown in Fig. 7.27b can also be occasionally recorded, whose incidence
is around 10–20% [43, 67]. In this case, the N1 occurs at similar time as the
one-peak case, but the P1 and the P2 typically occur at around 0.4–0.5 ms and
0.6–0.7 ms, respectively [43, 67].
196 X. Sun et al.

Fig. 7.27 The recorded ECAPs with (a) one positive peak and (b) two positive peaks

The amplitude of the ECAP varies from about 20 μV to 2 mV, and its maximum
magnitude usually is larger than the electrically evoked ABR [8]. Because of its large
amplitude, myogenic activity has little influence on the ECAP [32]. Moreover,
maturation of the central auditory system also is not a main factor that will cause
any changes in the ECAP. Therefore, the difference of ECAPs between adult and
pediatric CI users is very small [6, 26, 31], and the recorded waveforms do not
change with the increase of the duration of CI use. However, many factors also lead
to the variation of the latency and amplitude of the ECAP recorded, for example, the
stimulation level, the polarity of the stimulation, the surviving neural population, the
intra-cochlear test electrode location, and so on. The amplitude of the ECAP is the
voltage difference between N1 and P2 peaks. As shown in Fig. 7.28, ECAP
amplitude increases as the stimulation level increases.
The ECAP threshold and slope of the ECAP amplitude growth, as shown in
Fig. 7.29, are two major points in the neural response measurement, in which a
number of researchers are interested. The ECAP threshold is the stimulation current
level, at which the neural response measurement is able to capture the ECAP
waveform. Many studies indicate that the ECAP threshold falls between the behav-
ioral threshold (T-level) and comfort levels (C-level) of CI users [7, 9, 10, 18, 29,
59]. The slope of the ECAP amplitude growth is also widely used to objectively
predict the fitting program of CI users [10, 29].
7 Neural Interface: Frontiers and Applications 197

Fig. 7.28 Recorded ECAP waveform with different amplitudes of stimulation current

Fig. 7.29 The relationship between the stimulation current level and the amplitude of ECAP

7.6.3 Hardware Implementation

Hardware implementation is one of the key factors to successfully record the

neural response. Figure 7.30 shows how the cochlear implant system realizes the
NRM function. Generally, at the beginning the computer, where the NRM tool is
installed and all the parameters are set, controls the speech processor to send a
command to the implant with the information about the electrode and stimulation
current level. The modulated signal, which is encoded by the speech processor
according to this command, is transmitted through the inductive link between the
external coil and the internal coil. The stimulator decodes the signal from the
internal coil and sends a biphasic current pulse to the stimulating electrode (e.g.,
electrode 3 in Fig. 7.31). Once the electrode finishes the stimulation, the forward
transmission is completed. After a determined delay, the recording electrode (e.g.,
electrode 5 in Fig. 7.31) starts to record the voltage signal it receives. This voltage
signal firstly is amplified by an amplifier with a very large gain (usually more than
60 dB). Then the signal is digitized by a high-resolution ADC (usually more than
10 bits). Later, all the data are reversely transmitted to the speech processor by the
198 X. Sun et al.

Reverse Telemetry
Forward Transmission
External Internal
Coil Coil
Speech
Processor

Stimulator Implant
Computer
Electrode
Array

Skin

Fig. 7.30 The diagram of hardware implementation of neural response measurement

Fig. 7.31 Electrode array. Note: In this case, electrode 3 is the stimulating electrode, and electrode
5 is the recording electrode

reverse telemetry technique. In the end, the computer completes the analysis and
provides the report based on the data which are sent from the speech processor.
The entire system is very flexible so that the tool could change a number of
parameters. For example, the gain of the amplifier in the stimulator should be
variable. In most cases, a large gain (e.g., 60 dB) is required to amplify very weak
neural signal. By contrast, in some other cases, a relatively small gain (e.g., 40 dB) is
necessary to avoid a severe saturation problem. In addition, one electrode should be
able to consecutively generate two stimulating current pulses within a period, which
varies from 50 μs to several milliseconds. Moreover, the turn-on time of the analog-
to-digital converter (ADC) in the stimulator should also be well controlled, and the
requirement of the ADC’s turn-on duration time is at least 1 ms. Furthermore, the
7 Neural Interface: Frontiers and Applications 199

roles between recording and stimulating electrodes can be easily changed or

switched. In the real application, every electrode could be either a stimulating
electrode or a recording electrode.

7.6.4 Measurement Techniques

The forward-masking subtraction method, which is briefly introduced above, now

becomes the routine technique for most commercial neural response measurement
tools [1, 6, 7, 9, 20]. The main purpose of this tool is to cancel the artifact caused by
the probe stimulus, which is substantially larger than the neural response. The details
of this method shown in Fig. 7.32 are described below.
Four types of stimulation are used each time in the measurement. For type A, the
probe is stimulated alone at t1. For type B, the masker is stimulated in the same
electrode at t0, and then the probe is stimulated at t1. For type C, the masker alone is
stimulated at t0. For type D, zero stimuli for both masker and probe are presented.
Each type of stimulation is measured separately. The main theory behind this
technique is that the neurons in the auditory nerve will be in the refractory state
for a short while after electrical stimulation. Therefore, the measurement results of
type A, type B, type C, and type D, A, B, C, and D, are presented below,
respectively:

A ¼ SA þ PA þ PN ð7:4Þ
B ¼ SA þ MA þ MN þ PA ð7:5Þ
C ¼ SA þ MA þ MN ð7:6Þ
D ¼ SA ð7:7Þ

where SA is the system artifact; PA is the probe artifact; PN is the neural response
caused by the probe; MA is the masker artifact; and MN is the neural response
caused by the masker. In (7.5), the reason why PN is missed is because in this case,
the auditory nerve is in the refractory state and has no response for the probe
stimulus. Combining (7.4), (7.5), (7.6) and (7.7), the final measurement result of
the ECAP (FMRE) is:

FMRE ¼ A  B þ C  D ¼ PN: ð7:8Þ

Not surprisingly, the recorded signal is equal to the neural response caused by the
probe. According to (7.8), it is known that the amplitude of the probe stimulus is
variable and directly related to the amplitude of the ECAP. For the masker, the
stimulating current should be at a high level to make sure enough neurons are
stimulated by the masker and go into the refractory state [1].
200 X. Sun et al.

Fig. 7.32 The forward- t0 t1

masking subtraction IPI
method. Note: For each
Probe
case, the upper one is the
electrical pulse generated by SE
the stimulating electrode A PA+PN
(SE), and the lower one is RE
the measured signal
recorded by the recording Masker Probe
electrode (RE) SE
B MA+MN PA
RE
Masker
SE
C MA+MN
RE

SE
D
RE

In practice, the results of this method are affected by several non-ideal factors.
The saturation issue is one of the most serious factors, which could lead to the
failure of neural response measurement. In Fig. 7.32, it shows that the signal the
recording circuit received is comprised of the stimulus artifact and the neural
response. However, the amplitude of the stimulus artifact is much larger than the
neural response. Accordingly, if the amplitude of the stimulus artifact is too large
so that the amplifier is saturated at its largest output, the ECAP could not be
captured after measurement. In this case, reducing amplifier’s gain and increasing
the recording delay after stimulation could improve the final results of ECAP
measurement.
In addition, changing the inter-pulse interval (IPI), which is the time difference
between t0 and t1 in type B stimulation, could also change the amplitude of
recorded ECAP. On one hand, the number of neurons in the refractory state after
the masker stimulus in type B stimulation gradually decreases with increasing the
IPI. Thus more and more neurons in the auditory nerve have responses to the probe
stimulus in type B stimulation, which may reduce the amplitude of the ECAP based
on (7.8). On the other hand, if the IPI is too short, the integration of probe and
masker stimulation current may reduce the amplitude of the ECAP as well [1]. Fig-
ure 7.33 shows a group of ECAP signals, which were captured by NRM tool. It
clearly shows that the amplitude of ECAP of this patient reaches the maximum
value when the IPI is 510 μs.
7 Neural Interface: Frontiers and Applications 201

Fig. 7.33 The ECAP signal of patient P1 using NRM tool (stimulation of 130 CL was applied at
electrode 12)

7.6.5 Clinical Uses

As mentioned in the previous section, the invention of the reliable ECAP measure-
ment meets great demands from surgeons and audiologists for their clinical work.
The neural response measurement now is clinically used in the following
applications [36].
First, the ECAP measurement is used to evaluate and verify the neural response
from the auditory nerve. In many cases, this objective information is very important
for surgeons and audiologists. For example, surgeons can use it to monitor whether
the electrode arrays are well placed in the cochlea in surgery. After surgery, the
recovering condition of CI recipients varies significantly, and the audiologists can
use this information to judge the effectiveness of the electrical stimulation in the
auditory nerve fibers.
Second, the electrode and device can be functionally verified. Even though the
impedance test also can offer useful information about the device in some degree, the
capture of the ECAP signal is the direct proof that the device works properly inside
the cochlea.
Third, the ECAP measurement creates a great convenience for programming the
speech processor. As we all know, the first and very critical step to use CIs is to find a
map with corresponding T and C levels, which is fitted with every user’s own
circumstance. Many studies prove that the NRM threshold fall between T- and C-
levels [7, 9, 10, 18, 29, 59]. In Figs. 7.34, 21 adult subjects are tested using NRM
tool, and the preliminary results also support this conclusion. Thus the ECAP
threshold can guide audiologists how to start to test T and C levels individually.
More importantly, for the people who poorly communicate with audiologists, for
example, very young children, the results greatly assist audiologists to build rela-
tively accurate maps for those people.
202 X. Sun et al.

Fig. 7.34 Preliminary results of the relationship between NRM threshold and T- and C- levels
using NRM tool

Compared with other measurement techniques, e.g., electrically auditory

brainstem response (EABR), ECAP measurement is easy and convenient to operate,
because it does not require special equipment, extra recording electrodes, and
entirely different software. All the measurement is achieved based on the standard
CI product itself and the tool is embedded into the programming software. Addi-
tionally, there is no strict requirement to limit patient’s status so that it is easy to
cooperate with patient to complete test, which is essentially useful for pediatric CI
users. Moreover, the ECAP does not have an obvious difference between adults and
children, a typical protocol with determined parameters can be used for most
patients. Thus the operation of audiologists is much simplified. Because of the
advantages discussed above, neural response measurement becomes a very promis-
ing technique for more clinical applications.

7.7 Emerging Technologies

Three pioneers of cochlear implant Graeme Clark, Ingeborg Hochmair, and Blake
Wilson received the 2013 Lasker~DeBakey Clinical Medical Research Award for
their outstanding contributions to the development of the modern cochlear implant.
Meanwhile, active and diverse research is currently taking place in the area of cochlear
implant. Compared with normal-hearing people, cochlear implant recipients find it
hard to listen in noisy environment or to listen to music or tonal languages. Studies on
conveying fine structure information, combined acoustic and electric hearing, and
7 Neural Interface: Frontiers and Applications 203

bilateral implantation have been proposed to improve the performance of cochlear

implant listeners in above mentioned difficult tasks. Recently, optogenetic stimulation
of auditory nerve has been proposed to develop optical cochlear implant, which can
significantly reduce the channel interference of electrical stimulation. Cochlear
implant is on its way to confer more benefit to mankind.

References

1. Abbas PJ, Brown CJ, Shallop JK, Firszt JB, Hughes ML, Hong SH, Staller SJ (1999) Summary
of results using the nucleus CI24M implant to record the electrically evoked compound action
potential. Ear Hear 20:45–59
2. Arnoldner C, Riss D, Brunner M, Durisin M, Baumgartner WD, Hamzavi JS (2007) Speech and
music perception with the new fine structure speech coding strategy: preliminary results. Acta
Otolaryngol 127:1298–1303
3. Baker M, Sarpeshkar R (2007) Feedback analysis and design of RF power links for low-power
bionic systems. IEEE Trans Biomed Circuits Syst 1:28–38
4. Boll SF (1979) Suppression of acoustic noise in speech using spectral subtraction. IEEE Trans
Acoust Speech Signal Process 27:113–120
5. Brown CJ, Abbas PJ (1990) Electrically evoked whole-nerve action potentials: parametric data
from the cat. J Acoust Soc Am 88:2205–2210
6. Brown CJ, Abbas PJ, Gantz B (1990) Electrically evoked whole-nerve action potentials: data
from human cochlear implant user. J Acoust Soc Am 88(3):1385–1391
7. Brown CJ, Abbas PJ, Gantz B (1998) Preliminary experience with neural response telemetry in
the nucleus CI24M cochlear implant. Am J Otol 19:320–327
8. Brown CJ, Hughes ML, Lopez SM, Abbas PJ (1999) Relationship between EABR thresholds
and levels used to program the Clarion speech processor. Ann Otol Rhinol Laryngol 108(Suppl.
177):50–57
9. Brown CJ, Hughes ML, Luk B, Abbas PJ, Wolaver A, Gervais J (2000) The relationship
between EAP and EABR thresholds and levels used to program the nucleus 24 speech proces-
sor: data from adults. Ear Hear 21:151–163
10. Cafarelli Dees D, Dillier N, Lai WK et al (2005) Normative findings of electrically evoked
compound action potential measurements using the neural response telemetry of the nucleus
CI24M cochlear implant system. Audiol Neurotol 10:105–116
11. Charlet de Sauvage R, Cazals Y, Erre JP, Aran JM (1983) Acoustically derived auditory nerve
action potential evoked by electrical stimulation: an estimation of the waveform of single unit
contribution. J Acoust Soc Am 73:616–627
12. Chaturvedi V, Anand T, Amrutur B (2013) An 8-to-1 bit 1-MS/s SAR ADC with VGA and
integrated data compression for neural recording. IEEE Trans Very Large Scale Integr VLSI
Syst 21(11):2034–2044
13. Chen JD, Benesty J, Huang Y, Doclo S (2006) New insights into the noise reduction Wiener
filter. IEEE Trans Audio Speech Lang Process 14:1218–1234
14. Cherry C (1953) Some experiments on the recognition of speech with one and two ears. J
Acoust Soc Am 25:975–981
15. Chun H, Yang Y, Lehmann T (2014) Safety ensuring retinal prosthesis with precise charge
balance and low power consumption. IEEE Trans Biomed Circuits Syst 8(1):108–118
16. Clark GM, Tong YC, Dowell RC (1984) Comparison of two cochlear implant speech-
processing strategies. Ann Otol Rhinol Laryngol 93:127–131
17. Crozier PM, Cheetham BMG, Holt C, Munday E (1993) Speech enhancement employing
spectral subtraction and linear predictive analysis. Electron Lett 29:1094–1095
18. Cullington H (2000) Preliminary neural response telemetry results. Br J Audiol 34:131–140
204 X. Sun et al.

19. Culurcielllo E, Andreou A (2003) An 8-bit, 1mW successive approximation ADC in SOI
CMOS. In: Proceedings of IEEE international symposium on circuit and system, IEEE,
Bangkok, Thailand pp 301–304
20. Dillier N, Lai WK, Almqvist B, Frohne C, Müller-Deile J, Stecker M, von Wallenberg E (2002)
Measurement of the electrically evoked compound action potential via a neural response
telemetry system. Ann Otol Rhinol Laryngol 111(5):407–414
21. Doclo S, Spriet A, Wouters J, Moonen M (2007) Frequency-domain criterion for the speech
distortion weighted multichannel Wiener filter for robust noise reduction. Speech Comm
49:636–656
22. Donaldson GS, Kreft HA, Litvak L (2005) Place-pitch discrimination of single- versus dual-
electrode stimuli by cochlear implant users (L). J Acoust Soc Am 118(2):623–626
23. Douglas SC, Sun XA (2003) Convolutive blind separation of speech mixtures using the natural
gradient. Speech Comm 39:65–78
24. Dowell R, Seligman P, Blamey P, Clark G (1987) Evaluation of a two-formant-estimating
speech processor for a multiple-channel cochlear prosthesis. Ann Otol Rhinol Laryngol 96
(Suppl. 128):132–134
25. Dudley H (1939) The vocoder. Bell labs record, 18(4), pp 122–126
26. Eisen MD, Franck KH (2004) Electrically evoked compound action potential amplitude growth
functions and HiResolution programming levels in pediatric CII implant users. Ear Hear
25:528–538
27. Fetterman BL, Domico EH (2002) Speech recognition in background noise of cochlear implant
patients. Otolaryngol Head Neck Surg 126:257–263
28. Firszt JB, Holden LK, Skinner MW, Tobey EA, Peterson A, Gaggl W, Runge-Samuelson CL,
Wackym PA (2004) Recognition of speech presented at soft to loud levels by adult cochlear
implant recipients of three cochlear implant systems. Ear Hear 25:375–387
29. Franck KH, Norton SJ (2001) Estimation of psychophyical levels using the electrically evoked
compound action potential measured with the neural response telemetry capabilities of Cochlear
Corporation’s CI24M device. Ear Hear 22:289–299
30. Fu QJ, Zeng FG, Shannon RV, Soli SD (1998) Importance of tonal envelope cues in Chinese
speech recognition. J Acoust Soc Am 104:505–510
31. Gordon KA, Papsin BC, Harrison RV (2004) Toward a battery of behavioral and objective
measures to achieve optimal cochlear implant stimulation levels in children. Ear Hear
25:447–463
32. He S, Teagle HFB, Buchman CA (2017) The electrically evoked compound action potential:
from laboratory to clinic. Front Neurosci 11(Article 339):1–20
33. Hu Y, Loizou PC (2007) A comparative intelligibility study of single-microphone noise
reduction algorithms. J Acoust Soc Am 122:1777
34. Huang C, Shepherd R, Center P, Seligman P, Tabor B (1999) Electrical stimulation of the
auditory nerve: direct current measurement in vivo. IEEE Trans Biomed Eng 46(4):461–469
35. Huang S, Xia B, Wang S, Sun X (2015) A novel demodulation technique in reverse telemetry
for cochlear device. In: Conference on implantable auditory prostheses, p 119
36. Hughes ML (2010) Fundamentals of clinical ECAP measures in cochlear implants, Part 1: use
of the ECAP in speech processor programming, 2nd edn, Audiology Online, November
8, 2010, Article 2347
37. Kendir GA, Liu W, Bashirullah R, Wang G, Humayun MS, Weiland J (2005) An optimal
design methodology for inductive power link with class-E amplifier. IEEE Trans Circuits Syst
Regul Pap 52(5):857–866
38. Kiefer J, Hohl S, Stürzebecher E, Pfennigdorff T, Gstöett-ner W (2001) Comparison of speech
recognition with different speech coding strategies (SPEAK, CIS, and ACE) and their relation-
ship to telemetric measures of compound action potentials in the nucleus CI 24M cochlear
implant system. Audiology 40(1):32–42
39. Kim G, Loizou PC (2011) Gain-induced speech distortions and the absence of intelligibility
benefit with existing noise-reduction algorithms. J Acoust Soc Am 130:1581–1596
7 Neural Interface: Frontiers and Applications 205

40. Koch DB, Osberger MJ, Segal P, Kessler D (2004) HiResolution and conventional sound
processing in the HiResolution bionic ear: using appropriate outcome measures to assess speech
recognition ability. Audiol Neurootol 9(4):214–223
41. Koch DB, Downing M, Osberger MJ, Litvak L (2007) Using current steering to increase
spectral resolution in CII and HiRes 90K users. Ear Hear 28(2 Suppl):39S–41S
42. Kokkinakis K, Loizou PC (2008) Using blind source separation techniques to improve speech
recognition in bilateral cochlear implant patients. J Acoust Soc Am 123:2379–2390
43. Lai WK, Dillier N (2000) A simple two-component model of the electrically evoked compound
action potential in the human cochlea. Audiol Neurotol 5:333–345
44. Li N, Loizou PC (2008) Factors influencing intelligibility of ideal binary-masked speech:
implications for noise reduction. J Acoust Soc Am 123:1673–1682
45. Lim JS, Oppenheim AV (1979) Enhancement and bandwidth compression of Noisy speech.
Proc IEEE 67:1586–1604
46. Loizou PC, Poroy O, Dorman M (2000) The effect of parametric variations of cochlear implant
processors on speech understanding. J Acoust Soc Am 108:790–802
47. Luo X, Fu QJ (2004) Enhancing Chinese tone recognition by manipulating amplitude envelope:
implications for cochlear implants. J Acoust Soc Am 116:3659–3667
48. Martin R (1994) Spectral subtraction based on minimum statistics. In: Proceedings European
signal process, Edingburgh, Scotland, U.K. pp 1182–1185
49. McDermott HJ, McKay CM, Vandali AE (1992) A new portable sound processor for the
University of Melbourne/Nucleus Limited multielectrode cochlear implant. J Acoust Soc Am
91(3367–3371):1992
50. Nagal D (1974) Compound action potential of the cochlear nerve evoked electrically. Arch
Otorhinolaryngol 206:293–298
51. Ping L, Wang N, Tang G, Lu T, Yin L, Tu W, Fu QJ (2017) Implementation and preliminary
evaluation of ‘C-tone’: a novel algorithm to improve lexical tone recognition in Mandarin-
speaking cochlear implant users. Cochlear Implants Int 18(5):240–249
52. Plapous C, Marro C, Scalart P (2006) Improved signal-to-noise ratio estimation for speech
enhancement. IEEE Trans Audio Speech Lang Process 14:2098–2108
53. Razavi B (2001) Design of analog CMOS integrated circuits. McGraw-Hill, New York, pp
471–479
54. Ross M, Shaffer H, Cohen A, Freudberg R, Manley H (1974) Average magnitude difference
function pitch extractor. IEEE Trans Acoust Speech Signal Process 22:10
55. Schatzer R, Krenmayr A, Au DK, Zierhofer C (2010) Temporal fine structure in Cochlear
implants: preliminary speech perception results in Cantonese speaking implant users. Acta
Otolaryngol 130:1031–1039
56. Shannon RV, Zeng FG, Kamath V (1995) Speech recognition with primarily temporal cues.
Science 270:303–304
57. Skinner MW, Holden LK, Holden TA (1991) Performance of postlinguistically deaf adults with
the wearable speech processor (WSP III) and mini speech processor (MSP) of the nucleus multi-
electrode Cochlear implant. Ear Hear 12:3–22
58. Skinner MW, Holden LK, Whitford LA, Plant KL, Psarros C, Holden TA (2002) Speech
recognition with the nucleus 24 SPEAK, ACE, and CIS speech coding strategies in newly
implanted adults. Ear Hear 23:207–223
59. Smoorenburg GF, Willeboer C, van Dijk JE (2002) Speech perception in nucleus CI24M
cochlear implant users with processor settings based on electrically evoked compound action
potential thresholds. Audiol Neurotol 7:335–347
60. Sokal NO, Sokal AD (1975) Class-E- a new class of high-efficiency tuned single-ended
switching power amplifiers. IEEE J Solid State Circuits 10(3):168–176
61. Spahr AJ, Dorman MF (2004) Performance of subjects fit with the advanced bionics CII and
nucleus 3G cochlear implant devices. Arch Otolaryngol Head Neck Surg 130:624–628
62. Spriet A, Moonen M, Wouters J (2004) Spatially pre-processed speech distortion weighted
multi-channel Wiener filtering for noise reduction. Signal Process 84:2367–2387
206 X. Sun et al.

63. Stypulkowski PH, van den Honert C (1984) Physiological properties of the electrically stimu-
lated auditory nerve. I. Compound action potential recordings. Hear Res 14:205–223
64. Summers IR, Al-Dabbagh AD (1982) Simulated loss of frequency selectivity and its effects on
speech perception. Acoust Lett 5:129–132
65. Tang Z, Smith B, Schild JH, Peckham PH (1995) Data transmission from an implantable
biotelemeter by load-shift keying using circuit configuration modulator. IEEE Trans Biomed
Eng 42(5):524–528
66. Vary P (1985) Noise suppression by spectral magnitude estimation – mechanism and theoretical
limits. Signal Process 8:387–400
67. van de Heyning P, Arauz SL, Atlas M, Baumgartner WD, Caversaccio M, Chester-Browne R
et al (2016) Electrically evoked compound action potentials are different depending on the site
of cochlear stimulation. Cochlear Implants Int 17:251–262
68. Van den Bogaert T, Doclo S, Wouters J, Moonen M (2009) Speech enhancement with
multichannel wiener filter techniques in multimicrophone binaural hearing aids. J Acoust Soc
Am 125:360–371
69. Van Dun B, Wouters J, Moonen M (2007) Multi-channel wiener filtering based auditory steady-
state response detection. In: IEEE international conference on acoustics, speech and signal
processing, Honolulu, Hawaii, USA pp 929–932
70. Van Gerven S, Van Compernolle D (1995) Signal separation by symmetric adaptive
decorrelation: stability, convergence, and uniqueness. IEEE Trans Signal Process
43:1602–1612
71. Weinstein E, Feder M, Oppenheim AV (1993) Multi-channel signal separation by
decorrelation. IEEE Trans Signal Process 1:405–413
72. Wilson BS, Finley CC, Lawson DT, Wolford RD, Eddington DK, Rabinowitz WM (1991)
Better speech recognition with cochlear implants. Nature 352:236–238
73. Wouters J, Vanden Berghe J (2001) Speech recognition in noise for cochlear implantees with a
two-microphone monaural adaptive noise reduction system. Ear Hear 22:420–430
74. Zeng FG (2004) Trends in cochlear implants. Trends Amplif 8:1–34
75. Zeng FG, Nie K, Stickney GS, Kong YY, Vongphoe M, Bhargave A, Wei C, Cao K (2005)
Speech recognition with amplitude and frequency modulations. Proc Natl Acad Sci U S A
102:2293–2298
76. Zeng FG, Rebscher SJ, Fu QJ, Chen H, Sun X, Yin L, Ping L, Feng H, Yang S, Gong S,
Yang B, Kang HY, Gao N, Chi F (2015) Development and evaluation of the Nurotron
26-electrode cochlear implant system. Hear Res 322:188–199
77. Zeng F, Rebscher S, Harrison W, Sun X, Feng H (2008) Cochlear implants: system design,
integration, and evaluation. IEEE Rev Biomed Eng 1:115–142
Chapter 8
Neuromodulation for Pain Management

Jing Wang and Zhe Chen

Abstract Pain is a salient and complex sensory experience with important affective
and cognitive dimensions. The current definition of pain relies on subjective reports
in both humans and experimental animals. Such definition lacks basic mechanistic
insights and can lead to a high degree of variability. Research on biomarkers for pain
has previously focused on genetic analysis. However, recent advances in human
neuroimaging and research in animal models have begun to show the promise of a
circuit-based neural signature for pain. At the treatment level, pharmacological
therapy for pain remains limited. Neuromodulation has emerged as a specific form
of treatment without the systemic side effects of pharmacotherapies. In this review,
we will discuss some of the current neuromodulatory modalities for pain, research on
newer targets, as well as emerging possibility for an integrated brain-computer
interface approach for pain management.

Keywords Pain management · Neuromodulation · Brain-computer interfaces ·

Circuit-based neural signature

8.1 Introduction

Pain is a complex human experience which contains sensory, affective, and cogni-
tive dimensions. The sensory component of pain provides information about the
exact location, timing, and character of a noxious stimulus – the stimulus that
provokes pain. The affective component includes feelings of annoyance, anger,

J. Wang (*)
Department of Anesthesiology, Perioperative Care and Pain Medicine, Department of
Neuroscience and Physiology, New York University School of Medicine, New York, NY, USA
e-mail: [email protected]
Z. Chen
Department of Psychiatry, Department of Neuroscience and Physiology, New York University
School of Medicine, New York, NY, USA

© Springer Nature Singapore Pte Ltd. 2019 207

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_8
208 J. Wang and Z. Chen

sadness, anxiety, and depression in response to that noxious stimulus. Finally, the
cognitive component of pain dictates behavioral responses to this stimulus. During a
painful episode, these three components are integrated and contribute to the full
experience of pain.
The Webster dictionary definition of pain is “physical suffering or discomfort
caused by illness or injury.” However, this definition does not provide any anatom-
ical or mechanistic dimensions. Nor does it necessarily delineate the contribution of
the three components of pain. Compatible with this vague definition, the clinical
diagnosis of pain is established purely by verbal report and thus remains highly
subjective. Therefore, an objective diagnosis of pain can greatly improve both our
scientific understanding of pain and its clinical management.
At the treatment level, pharmacological options are limited. Opioids remain the
most efficacious analgesics, and these drugs have been utilized for pain management
for over 1000 years. Opioids, however, carry significant side effects, including
constipation, sedation, respiratory depression, immunosuppression, neuro-hormonal
dysregulation, dependence, and addiction. There is currently a great debate regard-
ing the use of opioids in the management of chronic non-malignant pain. Despite
enormous research over the last century, most of which focuses on spinal and
peripheral nervous systems, there have been relatively few new discoveries of
pharmacological therapy for pain. Neuromodulation, on the other hand, has emerged
as a potentially more specific form of treatment without the systemic side effects of
pharmacotherapies. In this review, we will discuss some of the current
neuromodulatory treatment modalities for pain, as well as ongoing research to
identify newer targets. We will also discuss the possibility for an integrated brain-
computer interface approach for pain management that involves both decoding pain
signals and neuromodulation of pain.

8.2 Spinal Neuromodulation for Pain

8.2.1 Theory for Spinal Cord Modulation

One of the key concepts in the scientific understanding of pain comes from a classic
paper by Melzack and Wall [1]. In this paper, Melzack and Wall argued that there are
possible gates of control for nociceptive and non-nociceptive signals at the spinal
cord level. More specifically, pyramidal or excitatory neurons in the spinal dorsal
horn (SDH) receive sensory inputs – both nociceptive and non-nociceptive – from
the peripheral primary sensory neurons, and then in turn transmit this sensory
information by projection to the thalamus and brain stem. The SDH, however, also
contains interneurons, which are GABAergic and inhibitory. These interneurons also
receive peripheral sensory information, but instead of projecting to the brain, they
project to other pyramidal neurons in the SDH to inhibit these neurons from
providing ascending signals. Thus, non-nociceptive sensory inputs to the SDH can
activate interneurons to inhibit the pain signal – in a sense shutting the gate for pain.
8 Neuromodulation for Pain Management 209

Based on this theory for gate control, spinal cord stimulation (SCS) was devel-
oped as a way to treat pain. It was thought that external electric stimulator can
simulate the firing of interneurons to inhibit ascending pyramidal neurons in the
SDH. As a result, SCS can inhibit the transmission of the pain signal from the
periphery to the brain. Surgically, SCS includes a trial phase and a permanent
implantation phase. During the trial phase, a trial stimulator is implanted into the
epidural space under fluoroscopy guidance. The stimulator is then turned on, and the
frequency of the stimulation and the location of the stimulator are adjusted over a
period of hours to ensure patient comfort and analgesic effects. As the goal of the
SCS therapy is to replace the actual pain sensation with paresthesia, the SCS leads
need to cover the body area where the patient’s pain is generated [2]. If the trial is
successful and the patient is able to tolerate the side effects, a permanent implant is
then done in the operating room. The permanent implantation is procedurally similar
to the trial lead placement, but the leads and battery are implanted in a pocket inside
the patient. The battery can be remotely charged. In some cases, permanent lead
implantation is done by laminectomy and the leads are sutured to the dura to ensure
stable placement [2].

8.2.2 Current Clinical Practice

The frequency of spinal cord stimulation is typically 40–60 Hz [2]. Newer advances,
however, include the use of very high-frequency stimulation (10,000 Hz) and burst
stimulation. Through mechanisms that are not entirely understood, such higher
frequency stimulations have been shown in a limited number of preliminary studies
to confer superior pain relief compared with traditional stimulation protocols [3–7].
Numerous studies have confirmed the analgesic efficacy for SCS. In two recent
meta-analyses of SCS treatment, it was found that >50% of the patients experienced
pain relief with an average follow-up of 24 months [8, 9]. The pain syndromes most
responsive to therapy include lumbosacral radiculitis that is refractory to pharmaco-
logical and interventional treatments, failed back surgery syndrome, diabetic periph-
eral neuropathy, and other distal peripheral neuropathic syndromes [2]. In Europe,
SCS has also been utilized for ischemic chest pain [2].

8.2.3 Limitations of Spinal Modulation for Pain

Despite its efficacy in pain treatment, there are several distinct disadvantages for SCS.
First, the SCS is implanted in the epidural space, and with a constant movement of the
spine, there is a high risk for lead displacement, with rates reported to be as high as
30% in some studies [10]. Another key disadvantage of SCS is that it requires constant
stimulation. This is because while SCS was developed as a neuromodulation tech-
nique, it was not developed as a demand-based neuromodulation technique. As a
210 J. Wang and Z. Chen

result, many patients develop tolerance to SCS, and in a significant subset of patients,
SCS loses its analgesic efficacy over time. In addition, continuous stimulation makes
side effects more problematic. The main side effect of SCS is paresthesia. Over time,
many patients elect to turn off the SCS due to the continuous bothersome paresthesia.
Theoretically, a demand-based approach to neuromodulation can alleviate many
of the side effects associated with SCS. SCS would be turned on only when pain is
sensed. This way, the patient will not experience a long period of paresthesia when
there is no pain signal. The design for a demand-based neuromodulation approach,
however, requires real-time decoding of pain signals, which has not been achieved at
the level of SDH and DRG neurons.

8.3 Brain Neuromodulation for Pain

After it first emerged 70 years ago, brain neuromodulation has been studied to be a
treatment modality for refractory pain conditions. Some of the earlier studies are in
rodents. A number of brain targets have been sought, with the overall goal of
reducing the activities in these pain-producing or processing regions with high-
frequency stimulations in a protocol that is similar to deep brain stimulation
(DBS) utilized for Parkinson’s disease [11]. In some cases, direct stimulation with
lower frequency has been attempted to activate the region of interest. These targets
include the anterior cingulate cortex (ACC), motor cortex, sensory thalamus,
intralaminar parafascicular complex (CMP), periacqueductal gray (PAG), and
nucleus accumbens (NAc). A number of studies targeting these areas have shown
a varying degree of success in pain relief.

8.3.1 Current Practice of Brain Modulation for Pain

Cortical Stimulations
The ACC is well known to be a critical hub for the processing for the affective
component of pain [12–22]. Not surprisingly, it was one of the first regions to be
targeted for pain relief. The first case series of electrical stimulation of the ACC as a
treatment for chronic pain was published in 1960 [23]. A large number of these cases
involved pain from spinal cord injuries [24, 25]. In these studies, bilateral stimula-
tion using a DBS protocol achieved temporary pain relief as measured by the
patient’s pain reporting and medication dosages. In particular, patients who received
ACC stimulation reported a decrease in the affective component of pain. It was
reported that these patients still felt pain, but “it didn’t bother” them as much.
However, pain relief was not found to be enduring. In a more recent study of
15 patients with chronic pain, chronic DBS of the ACC using implanted electrodes
produced an analgesic effect in five patients [26]. The side effects have in general
been mild in most studies.
8 Neuromodulation for Pain Management 211

Although not a region known for pain processing, the motor cortex has also been
targeted for pain relief. Motor cortex stimulation (MCS) has been evaluated partic-
ularly in central pain syndromes such as post-stroke pain or refractory trigeminal
neuralgia. The rationale for MCS comes from the observation that motor cortex
stimulation achieved sensory responses, and treatment as early as in 1971 has shown
partial analgesic effects [27]. Modern approach to MCS began with a study in 1991,
where eight of 12 patients were able to experience some degree of pain relief after
1 year of MCS with implanted electrodes [28]. A number of studies since then have
demonstrated efficacy for MCS, particularly for post-stroke pain syndrome, where
up to 70% of patients responded positively, and facial pain, with up to 80% efficacy
[29–34]. In these studies, a chronic implant with stimulation of 5–130 Hz has been
investigated to be efficacious.
Subcortical Stimulations
The role of the sensory thalamus in pain regulation is well established, as studies
have shown that thalamic injury such as in the case of stroke can lead to persistent
central pain or post-stroke pain. The thalamus is a highly heterogeneous structure,
and the regions involved in pain processing include the ventroposterior lateral (VPL)
and ventroposterior medial (VPM) nuclei. Whereas the lateral thalamus is thought to
project to the somatosensory cortex (S1) and provide accurate sensory information
related to pain, the VPM has been shown to project to a variety of cortical and
subcortical structures and is thought to be involved in the affective processing of
pain. Hyperactivity of the thalamus has been observed in neuropathic pain condi-
tions [35–37]. In an early and well-cited study, four of five facial pain patients who
received chronic VPM stimulation demonstrated pain relief [38]. Subsequently,
stimulation of both VPM and VPL regions have been carried in a number of studies.
Studies have shown that contralateral VPM stimulation can result in persistent
analgesia as well as paresthesia, whereas study with neuropathic pain patients
showed that thalamic stimulation can result in improvement in pain scores [39–
41]. The technical difficulty associated with thalamic stimulation, however, is
considerable, due to its position in the brain and its numerous subdivisions with
highly divergent functions. Perhaps as a result of these anatomic and structural
concerns, more recent studies have shown more conflicting results [42, 43].
The PAG projects to the rostroventral medial medulla (RVM), which in turn
projects to the spinal dorsal horn neurons, and depending on the exact neuronal
subgroups and circuit dynamics, this projection can provide either descending
inhibition or facilitation of ascending pain signals [44]. Given its well-established
involvement in descending pain regulation, the PAG has been pursued as a prom-
ising target in a number of studies [45–47]. Recent data, however, did not demon-
strate consistent and enduring pain relief [24, 43, 48–51]. Some of these studies
showed that PAG stimulation results in pain relief, whereas other studies did not
demonstrate persistent analgesic effect. These divergent results may be due to the
fact that the PAG can project to the RVM, but it projects to two distinct populations
of RVM cells. Whereas the off cells in the RVM produce pain relief, the on cells
actually give rise to pain intensification. As a result, a constant stimulation approach
212 J. Wang and Z. Chen

without differentiating on- or off-cells in the PAG may not be highly effective.
Furthermore, PAG stimulation appears to cause significant side effects including
nausea, nystagmus, vertigo, and nausea [45].
Additional targets in the brain include the NAc, CMP, hypothalamus (HT), and
the internal capsule (IC). The NAc forms the ventral striatum and is known for its
role in reward-based behaviors [52]. However, the NAc has also been shown in
recent studies to play a crucial role in pain and analgesia [53–55]. Specifically,
studies have shown that the projection from the medial prefrontal cortex (PFC) to the
NAc can produce relief of both sensory and affective components of pain in acute as
well as chronic pain conditions [56]. There are in fact case studies of post-stroke
(central) pain where stimulation of the NAc when combined with PAG stimulation
can achieve significant pain relief [57]. Meanwhile, the CMP has high levels of
expression of opioid receptors and is anatomically connected to the thalamus, ACC,
and NAc, and thus it has also been postulated to function as a target for pain
treatment [58]. Surgical stimulation of the CMP, however, has achieved variable
results. Some studies have shown CMP stimulation to have modest analgesic effects
in central or neuropathic pain syndromes, whereas other studies have failed to
reproduce such results with any significant time frame [59–61]. Finally, a recent
number of studies have shown that DBS of the posterior HT demonstrated relief for
cluster headache as well as for cephalgias [62–64]. The rationale for this approach is
that cluster headache has a circadian component, which in turn is regulated in large
part by the posterior HT [65]. These studies, however, are relatively small in sample
size. Meanwhile, the earliest study on IC stimulation in 1974 showed analgesic
efficacy [66, 67]. Subsequent studies in this area have also shown that DBS in this
area may have a role in reducing neuropathic pain [68–70]. The mechanisms for the
analgesic effects of IC stimulation, however, are not well understood.

8.3.2 Technical Considerations for Brain Modulation

for Pain

Electrical leads for MCS or DBS are typically two side by side four-contact
electrodes. MCS is achieved with an amplitude of 0.5–10 volts, with frequency
ranging from 5 to 130 Hz, with 40–100 Hz being the most common, and pulse
widths spanning from 60 to 450 ms. Meanwhile, standard DBS electrodes can be
implanted for pain similar to implants for Parkinson’s patients. Unfortunately, due to
the small number of studies, the difficulty in enrollment, and the nature of the
implantation surgery, MCS and DBS protocols have not been well investigated in
studies using large sample sizes. Important factors for consideration for cortical and
subcortical stimulation include the duration of stimulation, frequency of stimulation,
and the possibility of low-frequency stimulation to activate neurons vs high-
frequency stimulation to inhibit neuronal activities. The desired outcome would be
for the device to perform stimulation or inhibition of the neuronal networks during
8 Neuromodulation for Pain Management 213

the occurrence of pain episodes. The ultimate goal, meanwhile, is to achieve

demand-based automated stimulation. The accomplishment of this goal, however,
requires accurate decoding of the pain signals in the brain.

8.4 Decoding Pain in the Brain

8.4.1 Electrophysiological Recordings in Rodents and Non-

human Primates

From a neural coding perspective, the goal of pain encoding is to understand how
neurons represent and transmit the information of pain. Current decoding of pain
relies on neuroimaging techniques such as fMRI or PET. These neuroimaging
studies have a few critical limitations. First, at the anatomical level, fMRI imaging
experiments cannot provide desirable single-cell resolution. Second, fMRI measure-
ments reflect the blood-flow changes, and such changes indirectly indicate brain
activity. Therefore, it only measures the correlation between blood flow and neural
activity instead of direct neurophysiological measurements. Nociceptive information
is distributed so widely that blood-flow changes sometimes cannot accurately depict
primary and secondary neural responses. Third, imaging studies cannot identify the
causal relationship between circuit change and pain behaviors. This lack of causality
limits the mechanistic insight that neuronal codes can provide. Fourth, the limited
temporal resolution of imaging studies precludes the analysis of ultrafast circuit and
synaptic events. It is the last constraint that makes various imaging techniques less
ideal tools for real-time decoding in the brain. Due to these limitations, circuit
mechanisms of pain, particularly the affective and cognitive dimensions of pain,
remain poorly defined. On the other hand, in vivo electrophysiological recordings in
animal studies have provided insight on pain mechanism at cellular and circuit levels
and at a millisecond timescale. Below we will review a few important electrophys-
iological studies in rodents and non-human primates. We focus on the primary
neural circuits that modulate the pain stimuli, including thermal, mechanical, inflam-
matory, and neuropathic pain.
Recent animal research on pain has focused on spinal and peripheral systems.
Spinal cord and peripheral systems mostly inform nociception. Pain, however, is a
multi-dimensional experience and includes sensory, affective, and cognitive
dimensions. Therefore, attempts to decode pain should ideally include circuitry
in the brain. Meanwhile, there is no “pain cortex,” as pain is perceived and
processed in a distributed network [71]. Among the many brain areas, the S1 and
ACC are two of the most studied areas for pain-related responses. The S1 has long
been thought to encode the sensory component of pain, whereas the ACC has been
known as a key anatomic region for understanding the aversive and other affective
manifestations of pain [72, 73]. Other brain regions that are engaged in pain
processing include the medial prefrontal cortex (mPFC) [74], insular cortex [75],
214 J. Wang and Z. Chen

NAc [76], thalamus including the thalamic reticular nucleus (TRN), medial dorsal
(MD) nucleus, and other thalamic nuclei [77, 78].
In rodent experiments, neurophysiological recordings in pain studies were
reported in both anesthetized and freely behaving animals. For instance, it was
found that neurons in the TRN and ventroposterior lateral nucleus responded to
CO2 laser in anesthetized rats [79]. These neurons have shown specificity in their
responses and hence have been thought to be nociceptive thalamic neurons. How-
ever, in another independent report, it was found that none of the mechanical
nociceptive neurons showed a clear relationship between discharge and thermal
pain temperature [80]. The differences in these studies may be attributed to the
fact that animals were under anesthesia, which can make the magnitude and duration
of the sensory information transfer variable.
Physiological recordings in conscious, behaving rats have been collected in S1
and ACC [81–83], as well as in VPL and MD [83]. Specifically, S1 and VPL are
within the lateral sensory pain pathway, whereas ACC and MD are within the medial
pathway more involved in the affective and cognitive processing of pain. Kuo and
Yen [81] compared noxious responses in S1 and ACC and observed that ACC
neurons were less frequently activated and had a longer latency of onset in response
to thermal pain induced by a CO2 laser [81]. Additional experiments showed that
neuronal responses in both the S1 and ACC increased with the intensity of the laser
heat; and the stimulus-response curve of S1 ensemble activity had a steeper slope
than that of the ACC. Furthermore, S1 neurons were better at coding laser-heat
intensity than ACC neurons, whereas more ACC neurons were involved in condi-
tioned fear associated with an electric shock than S1 neurons. These results suggest
differential contributions of the S1 and ACC to sensory and affective dimensions of
pain. With the simultaneous recording of multiple brain areas in lateral and medial
pain pathways, Zhang et al. [83] reported significant correlations between laser
intensity and the number of responsive neurons, the firing rates, as well as the
mass spike counts. In addition, their results also showed a comparable coding
capacity of medial and lateral pain pathway neurons [83]. Furthermore, Zhang
et al. used a linear discriminant analysis for pain intensities based on the mass
spike counts of neuronal recordings from multiple brain areas. Although several
electrophysiological investigations in rats similar to the above studies have
established that activities in the S1 and ACC are altered by pain stimuli, very little
effort has been devoted specifically to decoding analysis, especially for the onset
detection of pain signals.
In non-human primate experiments, Chudler et al. [84] have found that nocicep-
tive neurons in the S1 of anesthetized monkeys were responsive to noxious thermal
stimulation applied to the face [84]. One type of nociceptive S1 neurons showed
increased firing rate with increasing stimulus intensity, whereas another type of
nociceptive S1 neurons showed saturated response once reaching a plateau above
a specific temperature. At the behavioral level, the intensity of pain sensation or pain
rating in some ways parallels the neuronal responses of nociceptive S1 neurons. In
another study, Iwata et al. [85] recorded ACC neurons from awake behaving
monkeys [85]. Positively and negatively modulated ACC neurons were reported in
8 Neuromodulation for Pain Management 215

response to a heat stimulus on the face. Their results further suggested that ACC
neurons are involved in the attention to pain and escape from pain but not in the
sensory-discriminative aspect of pain.

8.5 Potential for a Demand-Based Neuromodulation

Approach

From a treatment perspective, a neuromodulation is a potential option for refractory

pain. As discussed above, both spinal and brain stimulation has the potential for
modulating the pain experience [86]. In addition to therapeutic benefits, the use of
brain stimulation also allows mechanistic understanding of brain circuits that under-
lie the pain experience. However, current techniques such as deep brain stimulation
or transcranial magnetic stimulation (TMS) lack optimal targets in the brain and
require constant stimulation resulting in unacceptable side effects.
Demand-based neuromodulation, or closed-loop control for pain, provides an
appealing alternative strategy. This strategy can be combined with either spinal or
brain modulation. The realization of a demand-based neuromodulatory approach
depends critically on two factors: first, accurate and timely identification of the pain
signal, and second, low-latency neuromodulatory control. In other words, timing
matters in closed-loop neuromodulation therapy [87].

8.5.1 Challenges for Demanded-Based Pain Modulation

In neuroscience and clinical experiments, a closed-loop brain-machine interface

(BMI) offers a powerful platform to investigate the causal link between brain circuits
and behavior [88]. A basic closed-loop BMI system for pain modulation (Fig. 8.1)
would consist of a detection arm (which is aimed for detecting the onset of pain
signals) and a treatment arm (which is aimed for delivering neuromodulation).
For the detection arm, the central challenge in change-point detection is the speed
and accuracy of detection. In real-time processing, the objective of quickest detection
is to detect changes as quickly as possible in an online fashion, and the requirement
of time-delay for closed-loop feedback usually lies within tens or hundreds of
milliseconds.
For the treatment arm, the first step is to identify effective neural circuits for
modulating pain. Some candidate targets have been discussed above, including
the nucleus accumbens [89], ACC [90], and thalamus [79, 91]. The second step is
to apply neuromodulation methods. There are also a few strategies based on invasive
or noninvasive stimulations. In animal studies, the emergence of optogenetics
[56, 92–96] and chemogenetics [96] offer intriguing options. While not yet appli-
cable in humans due to the nature of viral transfer, these techniques allow quick and
216 J. Wang and Z. Chen

Fig. 8.1 A schematic of the rodent closed-loop BMI system. A closed-loop system consists of
the detection arm (detection of the onset of pain signals via online neural decoding) and treatment
arm (neuromodulation via electrical or optogenetic stimulation)

targeted methods to alter brain circuits. Such techniques can be used to perform
feasibility studies for therapeutics as well as to probe circuits in basic mechanistic
investigations.

8.5.2 Pilot Rodent Studies

In our investigations, we have used a rat model to study mechanisms of acute pain,
including acute thermal pain and acute mechanical pain. In thermal pain experi-
ments, we used a blue diode-pumped solid-state laser applied 1 mm from the plantar
surface of the hind paw contralateral to the brain recording site to generate pain in
freely moving rats. The laser was turned on by a transistor-to-transistor pulse
generator until paw withdrawal was observed. The fiber output power was calibrated
by compact power and energy at the beginning of every recording day [90, 97]. We
used custom-designed microdrives (32 channels) to simultaneously record neuronal
ensemble spike activity from the rat S1 and ACC areas. Both spike (>300 Hz) and
local field potential (<300 Hz) data were obtained.
For detection of the onset of pain signals, we have designed model-based and
model-free algorithms based on the input of (off-line or on-line) sorted neuronal
ensemble spike activity [97–99].
8 Neuromodulation for Pain Management 217

A Model-Based Approach
Notations: Given C neuronal recordings, let yk ¼ [y1, k,. . .,yC, k] denote a
C-dimensional population vector, with each element consisting of the neuronal
spike count at the kth time bin (bin size Δ); let exp.(ηk) denote the Poisson firing
rate vector for C neurons. We assumed that the data follows a latent-state Poisson
linear dynamical system (PLDS), where the latent univariate variable zk denotes an
unobserved common input that drives neuronal ensemble spiking activity yk as
follows [97, 98]:

zk ¼ azk1 þ Ek ð8:1Þ
ηk ¼ czk þ d, ð8:2Þ
yk ePoissonðexpðηk Þ△Þ, ð8:3Þ

where the state Eq. (8.1) is a first-order autoregressive (AR) model (0 < |a| < 1)
driven by a Gaussian noise process with zero mean and variance N(0, σ 2). We used
an iterative expectation-maximization (EM) algorithm to estimate the latent state
(E-step) and the unknown parameters {a, c, d, σ 2} (M-step). Alternatively, we
transformed the Poisson population vector (nonnegative) into a Gaussian random
vector and obtained a transformed linear dynamical system (TLDS). In this case, the
inference procedure was simpler due to the approximation methods. In on-line
applications, once the model parameters are identified, we used a recursive (forward)
filter to estimate the latent state variable [97, 99].
From the estimated latent state ^z k , we further computed the Z-score related to the
ðzbaseline Þ
baseline: Z ¼ zmean
SDðzbaseline Þ . Under the assumption that the Z-score is standard
normally distributed, we converted it to the one-tailed P-value. The criterion of
Z-score change was determined by a critical threshold for reaching statistical
significance.
A Model-Free Approach
We designed a greedy CUSUM (cumulative sum) algorithm. The CUSUM method
is aimed to detect a change point based on the instantaneous log-likelihood ratio
(LLR):

λ1
LLRf 1 kf 0 ¼ ylog  ðλ 1  λ 0 Þ ð8:4Þ
λ0

where λ0 and λ1are the rate parameters of two Poisson distributions f0and f1. We
assume that the baseline rate λ0 is known or can be estimated directly from data. We
sig
pffiffiffiffiffi an empirical criterion for a significant change in λ1 such that λ1 ¼ λ0  2
define
λ0 in either a positive or negative direction. The CUSUM algorithm computes the
instantaneous LLR sc, k for the cth neuron and update the cumulative sum as follows:

Sk ¼ maxc fSc, k g ¼ maxc fmaxf0; Sc,k1 þ sc, k gg ð8:5Þ

218 J. Wang and Z. Chen

Fig. 8.2 Online detection of the onset of thermal acute pain. Top: Spike counts of simulta-
neously recorded, online-sorted 17 units (seven S1 units and ten ACC units) from a freely behaving
rat. Bottom: Continuous tracking and detection of pain signals derived from the rat S1 and ACC
activity. Shaded area denotes the CI of Z-score. Vertical lines denote the moments of change-point
detection. In this example, the detection results of the PLDS were better than those derived from the
TLDS and model-free method (not shown) [100]

where S0 ¼ 0. The technical details are referred to [99]. Once the cumulative sum Sk
is above a predetermined threshold θ0 and the trend continuous more than two
consecutive steps, then a decision is made and Sk is reset. The threshold θ0 in the
CUSUM algorithm controls the false alarm rate. An empirical choice is to use the
test statistic (twofold log-likelihood) being a chi-square distribution with 1 degree of
freedom: χ 21, ð1αÞ . If we set α ¼ 0.01, we have θ0 ¼ 0.5  6.64 ¼ 3.38.

To illustrate these two methods, we apply these two methods to simultaneous

recordings of rat S1 and ACC neuronal populations. Two representative examples of
online detection of pain signals are shown in Fig. 8.2.
Therefore, as these pilot studies demonstrate, the S1 and ACC neuronal ensemble
activity encode important information about pain signals. Even more importantly,
these experiments demonstrate that integrating neural signals from multiple brain
areas will further improve the decoding accuracy for the onset of pain. Developing
online statistical methods for detecting the onset of pain signals in closed-loop neural
interfaces can open new research opportunities for both acute and chronic pain. Such
methods, when combined with spinal or brain modulation, can have the potential to
achieve pain control with limited side effects.

8.6 Summary

Pain is a complex experience. In a subset of patients who suffer from chronic pain, it
is a debilitating experience. The current state-of-the-art neuromodulation therapy is
spinal cord stimulation. While this treatment modality is well validated, it continues
to have side effects. Neuromodulation in the brain, meanwhile, has also been
8 Neuromodulation for Pain Management 219

explored, but optimal targets remain lacking. Recent advances in neuro-engineering

and computational neuroscience, however, when combined with an improved under-
standing of neurocircuitry of pain, allow for the possibility of demand-based
neuromodulation of pain. The possibility of a brain-computer interface approach
can allow better mechanistic understanding of basic pain mechanisms, and at the
same time, it can provide much-needed treatment for refractory pain syndromes.

References

1. Melzack R, Wall PD (1965) Pain mechanisms: a new theory. Science 150(3699):971–979

2. Deer TR et al (2014) The appropriate use of neurostimulation of the spinal cord and peripheral
nervous system for the treatment of chronic pain and ischemic diseases: the Neuromodulation
Appropriateness Consensus Committee. Neuromodulation 17(6):515–550. discussion 550
3. Liem L et al (2015) One-year outcomes of spinal cord stimulation of the dorsal root ganglion in
the treatment of chronic neuropathic pain. Neuromodulation 18(1):41–48. discussion 48-9
4. De Ridder D et al (2013) Burst spinal cord stimulation for limb and back pain. World
Neurosurg 80(5):642–649 e1
5. De Ridder D et al (2010) Burst spinal cord stimulation: toward paresthesia-free pain
suppression. Neurosurgery 66(5):986–990
6. Van Buyten JP et al (2013) High-frequency spinal cord stimulation for the treatment of chronic
back pain patients: results of a prospective multicenter European clinical study.
Neuromodulation 16(1):59–65. discussion 65-6
7. Tiede J et al (2013) Novel spinal cord stimulation parameters in patients with predominant
back pain. Neuromodulation 16(4):370–375
8. Taylor RS et al (2014) Predictors of pain relief following spinal cord stimulation in chronic
back and leg pain and failed back surgery syndrome: a systematic review and meta-regression
analysis. Pain Pract 14(6):489–505
9. Grider JS et al (2016) Effectiveness of spinal cord stimulation in chronic spinal pain:
a systematic review. Pain Physician 19(1):E33–E54
10. Deer TR et al (2014) The appropriate use of neurostimulation: avoidance and treatment of
complications of neurostimulation therapies for the treatment of chronic pain.
Neuromodulation 17(6):571–598
11. Levy RM, Lamb S, Adams JE (1987) Treatment of chronic pain by deep brain stimulation:
long term follow-up and review of the literature. Neurosurgery 21(6):885–893
12. Turnbull IM (1972) Bilateral cingulumotomy combined with thalamotomy or mesencephalic
tractotomy for pain. Surg Gynecol Obstet 134(6):958–962
13. Craig AD et al (1996) Functional imaging of an illusion of pain. Nature 384(6606):258–260
14. Rainville P et al (1997) Pain affect encoded in human anterior cingulate but not somatosensory
cortex. Science 277(5328):968–971
15. Foltz EL, White LE (1968) The role of rostral cingulumotomy in “pain” relief. Int J Neurol
6(3–4):353–373
16. Talbot JD et al (1995) Evaluation of pain perception after anterior capsulotomy: a case report.
Somatosens Mot Res 12(2):115–126
17. Koyama T, Kato K, Mikami A (2000) During pain-avoidance neurons activated in the
macaque anterior cingulate and caudate. Neurosci Lett 283(1):17–20
18. Koyama T et al (2001) Anterior cingulate activity during pain-avoidance and reward tasks in
monkeys. Neurosci Res 39(4):421–430
19. Qu C et al (2011) Lesion of the rostral anterior cingulate cortex eliminates the aversiveness of
spontaneous neuropathic pain following partial or complete axotomy. Pain 152(7):1641–1648
220 J. Wang and Z. Chen

20. Johansen JP, Fields HL, Manning BH (2001) The affective component of pain in rodents:
direct evidence for a contribution of the anterior cingulate cortex. Proc Natl Acad Sci U S A
98(14):8077–8082
21. LaGraize SC et al (2006) Selective regulation of pain affect following activation of the opioid
anterior cingulate cortex system. Exp Neurol 197(1):22–30
22. Lubar JF (1964) Effect of medial cortical lesions on the avoidance behavior of the cat. J Comp
Physiol Psychol 58:38–46
23. Lewin W, Whitty CW (1960) Effects of anterior cingulate stimulation in conscious human
subjects. J Neurophysiol 23:445–447
24. Boccard SG et al (2013) Long-term outcomes of deep brain stimulation for neuropathic pain.
Neurosurgery 72(2):221–230. discussion 231
25. Spooner J et al (2007) Neuromodulation of the cingulum for neuropathic pain after spinal cord
injury. Case report. J Neurosurg 107(1):169–172
26. Boccard SG et al (2014) Deep brain stimulation of the anterior cingulate cortex: targeting the
affective component of chronic pain. Neuroreport 25(2):83–88
27. Lende RA, Kirsch WM, Druckman R (1971) Relief of facial pain after combined removal of
precentral and postcentral cortex. J Neurosurg 34(4):537–543
28. Tsubokawa T et al (1991) Chronic motor cortex stimulation for the treatment of central pain.
Acta Neurochir Suppl (Wien) 52:137–139
29. Katayama Y, Tsubokawa T, Yamamoto T (1994) Chronic motor cortex stimulation for central
deafferentation pain: experience with bulbar pain secondary to Wallenberg syndrome.
Stereotact Funct Neurosurg 62(1–4):295–299
30. Katayama Y, Fukaya C, Yamamoto T (1998) Poststroke pain control by chronic motor cortex
stimulation: neurological characteristics predicting a favorable response. J Neurosurg
89(4):585–591
31. Herregodts P et al (1995) Cortical stimulation for central neuropathic pain: 3-D surface MRI
for easy determination of the motor cortex. Acta Neurochir Suppl 64:132–135
32. Ebel H et al (1996) Chronic precentral stimulation in trigeminal neuropathic pain. Acta
Neurochir 138(11):1300–1306
33. Nguyen JP et al (2000) Treatment of central and neuropathic facial pain by chronic stimulation
of the motor cortex: value of neuronavigation guidance systems for the localization of the
motor cortex. Neurochirurgie 46(5):483–491
34. Garcia-Larrea L et al (1999) Electrical stimulation of motor cortex for pain control: a combined
PET-scan and electrophysiological study. Pain 83(2):259–273
35. Lenz FA et al (1989) Characteristics of the bursting pattern of action potentials that occurs in
the thalamus of patients with central pain. Brain Res 496(1–2):357–360
36. Tasker RR et al (1987) Thalamic microelectrode recording and microstimulation in central and
deafferentation pain. Appl Neurophysiol 50(1–6):414–417
37. Lenz FA et al (1987) Abnormal single-unit activity recorded in the somatosensory thalamus of
a quadriplegic patient with central pain. Pain 31(2):225–236
38. Hosobuchi Y, Adams JE, Rutkin B (1973) Chronic thalamic stimulation for the control of
facial anesthesia dolorosa. Arch Neurol 29(3):158–161
39. Mazars G, Merienne L, Cioloca C (1974) Treatment of certain types of pain with implantable
thalamic stimulators. Neurochirurgie 20(2):117–124
40. Mazars G, Merienne L, Ciolocca C (1973) Intermittent analgesic thalamic stimulation.
Preliminary note. Rev Neurol (Paris) 128(4):273–279
41. Turnbull IM, Shulman R, Woodhurst WB (1980) Thalamic stimulation for neuropathic pain.
J Neurosurg 52(4):486–493
42. Marchand S et al (2003) Analgesic and placebo effects of thalamic stimulation. Pain
105(3):481–488
43. Coffey RJ (2001) Deep brain stimulation for chronic pain: results of two multicenter trials and
a structured review. Pain Med 2(3):183–192
8 Neuromodulation for Pain Management 221

44. Heinricher MM, Cheng ZF, Fields HL (1987) Evidence for two classes of nociceptive
modulating neurons in the periaqueductal gray. J Neurosci 7(1):271–278
45. Richardson DE, Akil H (1977) Pain reduction by electrical brain stimulation in man. Part 1:
acute administration in periaqueductal and periventricular sites. J Neurosurg 47(2):178–183
46. Richardson DE, Akil H (1977) Pain reduction by electrical brain stimulation in man. Part 2:
chronic self-administration in the periventricular gray matter. J Neurosurg 47(2):184–194
47. Hosobuchi Y, Adams JE, Linchitz R (1977) Pain relief by electrical stimulation of the central
gray matter in humans and its reversal by naloxone. Science 197(4299):183–186
48. Nandi D et al (2003) Thalamic field potentials in chronic central pain treated by periventricular
gray stimulation -- a series of eight cases. Pain 101(1–2):97–107
49. Owen SL et al (2007) Deep brain stimulation for neuropathic pain. Acta Neurochir Suppl
97(Pt 2):111–116
50. Green AL et al (2004) N-of-1 trials for assessing the efficacy of deep brain stimulation in
neuropathic pain. Neuromodulation 7(2):76–81
51. Bittar RG et al (2005) Deep brain stimulation for pain relief: a meta-analysis. J Clin Neurosci
12(5):515–519
52. Nestler EJ, Carlezon WA Jr (2006) The mesolimbic dopamine reward circuit in depression.
Biol Psychiatry 59(12):1151–1159
53. Goffer Y et al (2013) Calcium-permeable AMPA receptors in the nucleus accumbens
regulate depression-like behaviors in the chronic neuropathic pain state. J Neurosci
33(48):19034–19044
54. Su C et al (2015) Persistent pain alters AMPA receptor subunit levels in the nucleus
accumbens. Mol Brain 8(1):46
55. Baliki MN et al (2010) Predicting value of pain and analgesia: nucleus accumbens response
to noxious stimuli changes in the presence of chronic pain. Neuron 66(1):149–160
56. Lee M et al (2015) Activation of corticostriatal circuitry relieves chronic neuropathic pain.
J Neurosci 35(13):5247–5259
57. Mallory GW, Abulseoud O, Hwang SC, Gorman DA, Stead SM, Klassen BT, Sandroni P,
Watson JC, Lee KH (2012) The nucleus accumbens as a potential target for central poststroke
pain. Mayo Clin Proc 87(10):1025–1031
58. Weigel R, Krauss JK (2004) Center median-parafascicular complex and pain control. Review
from a neurosurgical perspective. Stereotact Funct Neurosurg 82(2–3):115–126
59. Andy OJ (1980) Parafascicular-center median nuclei stimulation for intractable pain and
dyskinesia (painful-dyskinesia). Appl Neurophysiol 43(3–5):133–144
60. Krauss JK et al (2002) Deep brain stimulation of the centre median-parafascicular complex in
patients with movement disorders. J Neurol Neurosurg Psychiatry 72(4):546–548
61. Ray CD, Burton CV (1980) Deep brain stimulation for severe, chronic pain. Acta Neurochir
Suppl (Wien) 30:289–293
62. Franzini A et al (2003) Stimulation of the posterior hypothalamus for treatment of chronic
intractable cluster headaches: first reported series. Neurosurgery 52(5):1095–1099. discussion
1099-101
63. Schoenen J et al (2005) Hypothalamic stimulation in chronic cluster headache: a pilot study of
efficacy and mode of action. Brain 128.(Pt 4:940–947
64. Green AL et al (2006) Deep brain stimulation for neuropathic cephalalgia. Cephalalgia
26(5):561–567
65. Magis D, Schoenen J (2012) Advances and challenges in neurostimulation for headaches.
Lancet Neurol 11(8):708–719
66. Fields HL, Adams JE (1974) Pain after cortical injury relieved by electrical stimulation of the
internal capsule. Brain 97(1):169–178
67. Adams JE, Hosobuchi Y, Fields HL (1974) Stimulation of internal capsule for relief of chronic
pain. J Neurosurg 41(6):740–744
68. Hosobuchi Y, Adams JE, Rutkin B (1975) Chronic thalamic and internal capsule stimulation
for the control of central pain. Surg Neurol 4(1):91–92
222 J. Wang and Z. Chen

69. Namba S et al (1984) Electrical stimulation of the posterior limb of the internal capsule for
treatment of thalamic pain. Appl Neurophysiol 47(3):137–148
70. Namba S et al (1985) Sensory and motor responses to deep brain stimulation. Correlation with
anatomical structures. J Neurosurg 63(2):224–234
71. Mano H, Seymour B (2015) Pain: a distributed brain information network? PLoS Biol 13(1):
e1002037
72. Sewards TV, Sewards MA (2002) The medial pain system: neural representations of the
motivational aspect of pain. Brain Res Bull 59(3):163–180
73. Bushnell MC, Ceko M, Low LA (2013) Cognitive and emotional control of pain and its
disruption in chronic pain. Nat Rev Neurosci 14(7):502–511
74. Apkarian AV et al (2004) Chronic back pain is associated with decreased prefrontal and
thalamic gray matter density. J Neurosci 24(46):10410–10415
75. Segerdahl AR et al (2015) The dorsal posterior insula subserves a fundamental role in human
pain. Nat Neurosci 18(4):499–500
76. Baliki MN et al (2012) Corticostriatal functional connectivity predicts transition to chronic
back pain. Nat Neurosci 15(8):1117–1119
77. Lenz FA et al (2004) The role of the thalamus in pain. Suppl Clin Neurophysiol 57:50–61
78. Dostrovsky JO (2000) Role of thalamus in pain. Prog Brain Res 129:245–257
79. Yen CT, Shaw FZ (2003) Reticular thalamic responses to nociceptive inputs in anesthetized
rats. Brain Res 968(2):179–191
80. Peschanski M, Guilbaud G, Gautron M (1981) Posterior intralaminar region in rat: neuronal
responses to noxious and nonnoxious cutaneous stimuli. Exp Neurol 72(1):226–238
81. Kuo CC, Yen CT (2005) Comparison of anterior cingulate and primary somatosensory
neuronal responses to noxious laser-heat stimuli in conscious, behaving rats. J Neurophysiol
94(3):1825–1836
82. Kuo CC et al (2009) Differential involvement of the anterior cingulate and primary sensori-
motor cortices in sensory and affective functions of pain. J Neurophysiol 101(3):1201–1210
83. Zhang Y et al (2011) Ensemble encoding of nociceptive stimulus intensity in the rat medial
and lateral pain systems. Mol Pain 7:64
84. Chudler EH et al (1990) Responses of nociceptive SI neurons in monkeys and pain sensation
in humans elicited by noxious thermal stimulation: effect of interstimulus interval. J
Neurophysiol 63(3):559–569
85. Iwata K et al (2005) Anterior cingulate cortical neuronal activity during perception of noxious
thermal stimuli in monkeys. J Neurophysiol 94(3):1980–1991
86. Riva-Posse P et al (2013) Practical considerations in the development and refinement of
subcallosal cingulate white matter deep brain stimulation for treatment-resistant depression.
World Neurosurg 80(3–4):S27.e25–S27.e34
87. Sun FT, Morrell MJ (2014) Closed-loop neurostimulation: the clinical experience.
Neurotherapeutics 11(3):553–563
88. Thakor NV (2013) Translating the brain-machine interface. Sci Transl Med 5(210):210ps17
89. Rauschecker JP et al (2015) Frontostriatal gating of tinnitus and chronic pain. Trends Cogn Sci
19(10):567–578
90. Zhang Q et al (2017) Chronic pain induces generalized enhancement of aversion. elife 6:
e25302
91. Wilson HD, Uhelski ML, Fuchs PN (2008) Examining the role of the medial thalamus in
modulating the affective dimension of pain. Brain Res 1229:90–99
92. Daou I et al (2013) Remote optogenetic activation and sensitization of pain pathways in freely
moving mice. J Neurosci 33(47):18631–18640
93. Iyer SM et al (2014) Virally mediated optogenetic excitation and inhibition of pain in freely
moving nontransgenic mice. Nat Biotechnol 32(3):274–278
94. Gu L et al (2015) Pain inhibition by optogenetic activation of specific anterior cingulate
cortical neurons. PLoS One 10(2):e0117746
8 Neuromodulation for Pain Management 223

95. Zhang Z et al (2015) Role of Prelimbic GABAergic circuits in sensory and emotional aspects
of neuropathic pain. Cell Rep 12(5):752–759
96. Iyer SM et al (2016) Optogenetic and chemogenetic strategies for sustained inhibition of pain.
Sci Rep 6:30570
97. Chen Z et al (2017) Deciphering neuronal population codes for acute thermal pain. J Neural
Eng 14(3):036023
98. Chen Z, Wang J (2016) Statistical analysis of neuronal population codes for encoding acute
pain. In: Proceedings of IEEE ICASSP. IEEE Press, New York, pp 829–833
99. Chen Z, Hu S, Zhang Q, Wang J (2017) Quickest detection for abrupt changes in neuronal
ensemble spiking activity using model-based and model-free approaches. In: Proceedings of
the IEEE conference neural engineering. IEEE Press, New York, pp 481–484
100. Hu S, Zhang Q, Wang J, Chen Z (2017) Real-time particle filtering and smoothing
algorithms for detecting abrupt changes in neural ensemble spike activity. J Neurophys
119(4):1394–1410
Chapter 9
Future of Neural Interfaces

Farah Laiwalla and Arto Nurmikko

Abstract The technological ability to capture electrophysiological activity of

populations of cortical neurons through chronic implantable devices has led to
significant advancements in the field of brain-computer interfaces. Recent progress
in the field has been driven by developments in integrated microelectronics, wireless
communications, materials science, and computational neuroscience. Here, we
review major device development landmarks in the arena of neural interfaces from
FDA-approved clinical systems to prototype head-mounted and fully implantable
wireless systems for multi-channel neural recording. Additionally, we provide an
outlook toward next-generation, highly miniaturized technologies for minimally
invasive, vastly parallel neural interfaces for naturalistic, closed-loop
neuroprostheses.

Keywords Brain-computer interfaces · Wireless neural interfaces · Spatially

distributed neural sensors · Neuroprostheses

9.1 Overview: Where We Are Now

This volume contains many illustrative examples of the early development and
specific applications, across a spectrum of contemporary techniques, whereby direct
electronic access to brain circuits, primarily from/to the neocortex, offers opportu-
nities for basic brain science, emergence of medical devices for assisting and/or
correcting neurological deficits, as well as discovering potential device-based ther-
apies as an alternative to pharmaceuticals. The emergence of electronic neural
interfaces, in particular, reflects outcomes of worldwide multidisciplinary work
over the past two decades whereby device engineering, neuroscience, and clinical
needs have met at the intersection of fundamental and applied brain science. Here,

F. Laiwalla · A. Nurmikko (*)

School of Engineering, Brown University, Providence, RI, USA
e-mail: [email protected]

© Springer Nature Singapore Pte Ltd. 2019 225

X. Zheng (ed.), Neural Interface: Frontiers and Applications,
Advances in Experimental Medicine and Biology 1101,
https://doi.org/10.1007/978-981-13-2050-7_9
226 F. Laiwalla and A. Nurmikko

we consider progress—and challenges ahead—for the development of brain-

computer interfaces for neural prostheses. We narrow the focus to cortical brain-
computer interfaces (BCI) in offering a subjective view of future assistive technol-
ogies, particularly those applicable to severe neurological injury to the motor system
such as in case of paralysis from neuromotor diseases, stroke, or spinal cord injury.
To gain a broader perspective, the reader is referred elsewhere for reviews of neural
interfaces including those which engage deeper brain structures such as deep brain
stimulation (DBS) which has reached medical device maturity in widespread treat-
ment of, e.g., Parkinsonian disorder [33]. The DBS technology is one example of
neuromodulation application of electrical stimulation; many other approved electri-
cal stimulation-based medical devices are also in widespread clinical use, such as
spinal cord treatment for chronic pain.
When compared to maturity of the DBS-like neuromodulation technologies,
and leaving questions about their therapeutic efficacy to clinicians, cortical BCIs
are still at relative infancy, limited at this writing to perhaps up to a dozen human
pilot trials worldwide. The fundamental challenges are, in our opinion, consider-
ably more difficult than devices which deploy electrical stimulation from a handful
of electrodes near/at the anatomical central or peripheral nervous system target. In
particular, for BCIs, one must be able to record cortical circuit activity at sufficient
level of detail and then decode typically many parallel channels of electrophysio-
logical activity (action potentials, field potentials, etc.). The decoding, in turn, for
essentially reconstructing cortical network activity in a predictive context requires
sophisticated computational models and tools to decipher, e.g., a subject’s inten-
tion to move a limb.
One set of illustrative examples of progress is from research where intracortical
microelectrode arrays (MEA) are used to record population dynamics from (<1 cm2
area) cortical patches at single neuron-level and subsequent application of stochastic
state-space dynamical models to decode, e.g., movement planning intention of a
subject. The genesis and progress have been well documented in a number of review
articles and book chapters. Suffice it to say here that from early discoveries in
non-human primates [9, 21, 30, 51], the adaptation of these methods has enabled
tetraplegic subjects to control communication devices and electromechanical
devices such as robotic arms and hands [3, 13, 22, 23, 46, 52]. The cortical locations
where such recordings are made have typically involved primary and/or premotor
and parietal areas where, e.g., direction and velocity tuning is distinctly encoded in
subsets of neurons. While specific tuning properties of neurons can be helpful across
a population which is part of an interconnected network with manifolds of internal
state representation (e.g., motor and visual cortices), new decoding techniques are
emerging using, e.g., deep learning methods, where presence of such tuning prop-
erties is not necessary for successful decoding.
While it follows from biophysical fundamentals that intracortical probes with
their inherently single-neuron space-time resolution offer the most direct means to
communicate with neuronal populations and their associate networks, they are by no
means the only, and not always necessary, requirement for building electronic neural
interfaces for prosthetic and other assistive use. As shown elsewhere in this volume,
9 Future of Neural Interfaces 227

whether scalp-based (EEG) electrode arrays [36], epidural or subdural

electrocorticography (ECoG) [34, 50], peripheral nerves, or elsewhere, there are
multiple means to acquire meaningful control signals for assistive use. Broadly, for
any assistive medical technology, the overall cost-to-benefit calculations must weigh
in many factors: engineering and technological complexity, required medical pro-
cedures (such as surgery), safety and reliability—and above all, the level of required
performance of the neural interface from the user’s point of view.
This chapter examines electrophysiological BCIs only. While the brain is an
electrical (electrochemical) biological machine with remarkable level of perfor-
mance (e.g., per Watt of metabolic power) and lends itself readily to electrophysi-
ology, active research is being pursued to search for neural interfaces relying on
recording/stimulation modalities which range from optical to acoustic (ultrasound),
to magnetic to chemical modalities, and combinations thereof. Time will tell if these
alternative biophysical agents/modalities can compete or augment the direct electri-
cal/electronic approaches.
We summarize in Sect. 2 some recent progress where innovation of wireless
neural interfaces can now be envisioned to enable mobile BCIs, for untethered
movement of subjects. In Sect. 3, we peek into the early ambitious efforts to develop
very large-scale neural recording and microstimulation systems where scaling the
electrophysiological access to, say, thousands of cortical points requires entirely new
ways of approaching the problem, from developing new types of neural probes to
re-examining systems level neuroengineering concepts.

9.2 Examples of State-of-the-Art Electronic Brain

Interfaces

From a systems level perspective, a brain-computer interface may be broadly

partitioned into (i) the physical neural probe (electrodes), (ii) an electronic custom
integrated circuit (IC) core for signal acquistion, (iii) the (generally noisy) signal
conditioning and telemetry, and (iv) the computing modules with dedicated back-
end hardware embedding computational algorithms specific to the neural decoding
task. The last decade has seen significant advances in each of these areas, even if a
challenging road still lies ahead to meet the overarching goal of developing versatile
BCI systems that can adaptively engage with the underlying non-stationary neural
circuits in a closed-loop fashion, while compatible with long-term chronic implan-
tation of neural probes in a (future) mobile subject without eliciting significant
immune response.
The fundamental challenges of chronic intracortical or deep-brain neural probe
design are manyfold. First, it is arguably desirable that the spatial scale and density of
microscale probes reach close to single neuron resolution (say, ~20–100 μm electrode
proximity for a layer V pyramidal neural cell body). Second, the probe material in
direct electrical contact with tissue must enable high signal to noise ratio (SNR)
228 F. Laiwalla and A. Nurmikko

recordings and high-charge-density biphasic microstimulation (the latter being critical

for eventual fully closed-loop BCIs) while maintaining designed form factor (rigidity,
etc.) and microscale surface characteristics that hopefully limit unavoidable immuno-
logic response. There have been multiple recent reports of successful refinement of
electrode microfabrication methodologies to yield, for example, high-density surface
ECoG electrodes [26, 49], as well as penetrating 3-D probes [40] approaching an
electrode pitch of tens of microns for high-density neural sampling. In contrast to
traditional epi and intracortical approaches, Oxley et al. [37] have described a mini-
mally invasive intravascular approach for the placement of a “stentrode” in cortical
veins to access neural information at high spatial resolution (where the latter approach
is currently being commercialized by Synchron med). Meanwhile, material science
research has provided several promising candidates for the integration of next-
generation BCI probes in addition to established work horses of Pt and IrOx, from
silicon-carbide [15, 16], carbon nanotubes [12], nanoelectronic threads [32] to hybrids
of conductive polymers like poly(3,4-ethylenedioxythiophene) (PEDOT) [10, 11,
27]. Success of optical techniques such as optogenetics has led to the development
of hybrid probes integrating, for example, optical stimulation and electronic recording
onto the same intracortical platform [22, 28, 45], where the latter target closed-loop
hybrid optoelectronic BCIs in rodents and non-human primates. Successful
transitioning of these novel experimental probe technologies for a chronic implantable
BCI is the next major challenge and goal.
The electronic core of a BCI system invariably includes signal amplifiers, condi-
tioning (filters) and dedicated analog-to-digital neural signal conversion. A number
of groups worldwide have been developing custom IC solutions for low-power and
low-noise neural signal acquisition, particularly focusing on channel scalability
(from a handful to >100) and ease of physical integration with the typically mono-
lithic multi-element neural probe. While it is not possible to review these develop-
ments exhaustively here, we highlight one recent work, namely, the “Neuropixel”
probe system where up to 1356 recording channels were microfabricated within one
silicon shank with integrated CMOS signal preconditioning [25, 39]. Systems such
as these which allow high-resolution, sub-acute (up to a few months) of robust,
wired access to the brain are instrumental in driving neuroscientific knowledge
forward, yet are limited for translational chronic use by the complex percutaneous
connectors in terms of full subcutaneous implantation. In general, adding a wireless
telecommunication capability to neural probes defines one crucial need to extend the
utility of these methods to freely moving animals and clinical applications.

9.2.1 BCI Going Wireless: General Considerations

In designing wireless neural interfaces, it is important to consider the requirements

and constraints in the full ecosystem context. Relevant questions to ask up front
might include, e.g., how a given intracortical MEA or subdural ECoG array may be
interfaced with the electronic payload, the latter now located within the hermetically
9 Future of Neural Interfaces 229

sealed envelope of the wireless device system. A key question is where within or on
the body will the electronics package locate. We start with the design where the
system electronics are configured as a head-mounted compact package whereby
percutaneous/transcranial connectorization from, e.g., an MEA, is required. One
rationale for this configuration as a starting choice on the wireless path is that
silicon-based commercial intracortical MEAs are now widely available with a
variety of channel counts compatible with small-animal (rodent) as well as
non-human and human primate models, including reasonable chronic robustness
demonstrated in clinical trials.
Given that contemporary intracortical (or subdual ECoG) sensor arrays typically
sample on the order of 100 nodes (single neurons for MEAs, summated field
potentials for ECoGs), it is remarkable how such pronounced under-sampling of
neuronal population can be successful in enabling subjects to control, e.g., a robotic
hand/arm with multiple degrees of freedom of motion. The good news is that
this limited amount of neural data sets not unreasonable requirements for wireless
telemetry. As also noted elsewhere in this volume, much credit to BCI development
must be assigned to (scalable) sophisticated statistical models of neuronal dynamics
which have produced powerful algorithms for effective decoding of, e.g., patient’s
movement intention, from this under-sampled pool of “noisy” brain signals, for real-
time assistive device use. Since the “noise” typically hides valuable neural informa-
tion, it is important that the signal fidelity of a wireless link be maintained (use of
data compression techniques must thus be approached judiciously). The wire-
less neural population data provides direct inputs to the neuro-computational
decoding models. There are numerous approaches to neural decoding; one
representative example is state-space dynamical modeling, which distils
multidimensional neural data (spikes and field potentials) to lower dimensional
cortical state representation, suitable for interpretation both by visualization
(graphics algorithms), as well as for prosthetic use through forwarding the outputs
to assistive electronic devices (e.g., direct cortically operated laptops). The role of
machine learning applied to neural decoding and encoding is a rapidly evolving new
area of computational neuroscience, with very promising preliminary results sug-
gestive of significant performance and efficiency benefits for real-time BCIs. We
bypass the field of neural decoding/encoding in this article and refer the reader to the
rich literature, which by now exists for this topic.
From a generic system level view, a fully wired BCI instrumentation collects the
raw neural (multi-channel) data via multi-wire cables into electronic neural signal
processors (composed of combinations of analog and digital electronics). The
digitized data is fed to computers where even today’s desktop machines have enough
on-board processing power to carry out much of neural decoding to run a simple set
of assistive devices. Figure 9.1 shows a block diagram of the basic electronic
ecosystem which is more or less common to most (non-human and human) primate
researchers in the field.
The next level of significance in advancing this type of electronic brain interface
neurotechnology (even if still in early days) is to “untether the patient” by engineer-
ing direct wireless access to neural probes such as depicted in Fig. 9.2. While for
external body wearable biosensors (whether EEG, EKG, etc.), the transition to a
230 F. Laiwalla and A. Nurmikko

Fig. 9.1 A block diagram view of a multi-channel neural recording system for brain-computer
interface applications. Multi-channel implantable microelectrode arrays are connectorized using
wire bundles, and external cabled electronics and computers are used for signal acquisition and
processing

Fig. 9.2 Block diagram and representative device view of a head-mounted wireless neural record-
ing system [55]. Integrated microelectronics are placed within a compact package powered by a
replaceable Li-ion battery
9 Future of Neural Interfaces 231

wireless system for enhancing subject’s mobility is already possible by adapting any
number of wireless electronics developed in the past decade for consumer and
industrial use, the situation for implanted wireless BCIs poses a number of chal-
lenges. These include the following which must be designed and ultimately inte-
grated seamlessly into one viable medical grade implant device system:
• Very low-power and high-fidelity analog and digital (custom) application specific
integrated circuits (ASICs) for broadband (spikes, field potentials) neural signal
acquisition and electrical microstimulation.
• Very low-power radio frequency transmitter capable of high data rate transmis-
sion (~100 MBits/s and beyond), custom designed as an integrated circuit (“RF
ASIC”).
• Connectorization of the multi-element neural probe front-end (such as one or
more MEAs) to and integration with the on-board ASICs.
• Packaging the electronics within a compact housing which is hermetically viable
for long-term chronic use.
• Designing external RF receivers which capture the wirelessly emitted, digitally-
encoded signals for inputs to computational devices for decoding.
• Strategy for powering the implant electronics (internal battery or RF inductive
coupling by external coils).
These engineering challenges are further bound by the first priority in any medical
device candidate—safety—which overrides even the main functional claim of the
device, here the neurotechnological performance. In the wireless device system
which we use here as an example (from our laboratories), safety has multiple
components such as (and revisited further below):
• Electrical safety: the electrical energy required by the active ASICs in the implant
must have zero accidental probability of discharging into the body/brain tissue.
• Mechanical safety: if the implant unit locates, for example, in the subject’s head,
the packaging of the electronics must be correspondingly impact-proof.
• Chemical safety: the hermetic sealing of the electronics package must ensure that
there is no leakage of toxic or other chemically harmful materials into tissue.
• Thermal safety: ensure that maximum temperature, e.g., in the subcutaneous
vicinity of the implant during operation (or recharging batteries), does not exceed
ΔT < 2.0  C and that corresponding temperature rise in the cortical space are kept
at ΔT < 0.1  C.
• Electromagnetic (EM) safety: RF (or similar wireless) signal and power trans-
mission of the device are secure, and are not susceptible to external EM interfer-
ence by ambient RF traffic and vice versa.
Most of these requirements follow the footsteps of established protocols, such as
those for pacemakers or cochlear implants for human use, thereby mirroring
established regulatory guidelines determined by FDA in the USA. We note, how-
ever, that at the frontiers of neurotechnology such as discussed in this volume, the
device systems involve considerably complex electronics and require orders of
magnitude larger wireless data rates so that early discussions by the technology
developers with regulatory agencies is very important.
232 F. Laiwalla and A. Nurmikko

A number of neurotechnology research groups have developed compact, exter-

nally head-mounted wireless neural recording systems as the first step, whereby this
approach has allowed leveraging commercially available microelectronics for the
development of proof-of-concept wireless BCI systems. Among some of the earliest
innovation in this arena, a set of contributions came from the Shenoy group at
Stanford. This team has built a series of incrementally sophisticated “Hermes” series
of wireless neural recording platforms [12, 19, 30, 38] beginning from a two-channel
Hermes-B [41] to a 96-channel battery-operated Hermes-E [17]. In the laboratories
of the authors of this chapter, we have similarly developed an external battery-
operated, 96-channel broadband neural recording system, featuring custom ICs for
neural signal capture and conditioning as well as a dedicated RF IC for 3.5 GHz
short-range (~4 m), very-low power wireless transmission with a net power con-
sumption of 61.2 mW [55]. The latter is now a licensed technology which is
commercially available, allowing animal researchers to conduct studies on freely
moving animals, for example in the context of foraging and naturalistic locomotion.
In addition to the systems level achievements highlighted above, new research
is particularly focused at pushing the limits of microelectronic technology to attain
improved performance for wireless neural recording with ever-shrinking power
budgets. This research extends from subsystems utilizing commercial technologies
such as UWB radios, WiFi, and Zigbee to highly optimized low-power integrated
approaches. For example, a 4096 channel multiplexed ECoG recording chip with
64-channel amplifiers and 5.12 Mbps data rata, transmitted at 7.8 GHz using UWB
radio has been described by Ando et al. [2] with a net power consumption of
1400 mW. Schwarz et al. [42] similarly describe a 128-channel head-mounted
system working with microwire array implants, but now integrating the capability
for on-board neural signal processing and bidirectional communications with a
power consumption of 264 mW, or ~2 mW per channel. Several other groups have
described multi-channel ASICs with sub-milliwatt power consumptions per chan-
nel while representing various trade-offs between signal compression and power
budgets [5, 8]. Finally, multi-modal sensing and stimulation, for example, inte-
grating dopamine sensing and electrical stimulation alongside electrical neural
recording [7, 22], and approaches incorporating optogenetic approaches, are
increasingly building a niche in the BCI component level landscape which is
anticipated to grow significantly in future.

9.2.2 Fully Implanted Wireless Devices—A Case Example

The development of head-mounted and other outside-body wireless neural record-

ing systems has provided a new BCI tool for neuroscience research by enabling
more complex and naturalistic animal experiments. However, the need for a
percutaneous connection still poses a limitation for BCI use of these systems, in
particular, limiting their translation into the human/clinical BCI domain. In con-
sidering the bridge from a head-mounted to a fully implanted wireless neural
9 Future of Neural Interfaces 233

interface device system, we reiterate key challenges which have to be addressed,

typically through extensive animal testing and performance validation:
• Implantation of the system’s electronic components within the body in a hermet-
ically encapsulated unit, and choice for the anatomical location of this payload
relative to the actual neural probes, where the electronic package has a form factor
compatible with subcutaneous placement, for example, in the epicranial space.
• ASIC design and optimization for ultra-low power, consistent with thermal safety
guidelines defined by FDA for active brain implants (maximum of 0.5  C incre-
ment in tissue temperature). Extensive thermal simulations and metrology of
implants in animal models are typically required.
• Implant power management (rechargeable batteries or continuous inductive cou-
pling) and transcutaneous wireless communication, mitigating the impact of
tissue RF absorption, and keeping within the specific absorption rate (SAR) limits
prescribed by IEEE Std C95.1–2005.
The basic microelectronic building blocks of a fully implantable neural interface
system are nearly identical to those described in previous sections; the caveat is that
now the spatial scale and performance metrics and requirements (such as those listed
above) for a fully implantable device are far more stringent. As the case example for
this chapter, the authors’ labs have developed a 100-channel hermetically-sealed, fully
implantable broadband wireless neural recording system utilizing a 100-channel
neural amplifier ASIC, 12-bit SAR ADCs, and a custom RF transmitter IC. The
components mirror those developed for the head-mounted system described previ-
ously, but now focus on a system architecture with a titanium (Ti)-based hermetic
enclosure. The Ti-enclosure has been equipped with a sapphire window to provide full
electromagnetic transparency for wireless charging and telecommunications. Further,
a custom planar interconnect interface has been built to feed the 100 microwires from
the intracortical MEA to the active electronics via multi-channel custom high-density
hermetic feedthroughs as shown in Fig. 9.3 [6]. In keeping with the need for further

Fig. 9.3 A fully implantable wireless device for neural recording using titanium hermetic enclo-
sure, ceramic feedthroughs, and a sapphire window for broadband wireless communication. Device
is powered using a rechargeable Li-ion battery, with multi-hour operation between recharges
234 F. Laiwalla and A. Nurmikko

electronic integration, there have been several recent efforts to develop system on chip
electronics (SoC), where a single chip solution provides neural recording in conjunc-
tion with wireless charging, compressive sensing or spike detection, electrical stimu-
lation and bidirectional telecommunications [4, 31, 36]. These advancements are
important elements for the further development of fully-implantable systems, but the
most critical current bottle-neck for these devices is the availability of scalable
hermetic packaging solutions. Hermetic sealing approaches being further studied
include ceramic, metallic, or thin-film materials, among others. In addition to the
materials choice, a major engineering challenge is to microfabricate the large number
of electrical “feedthroughs” without compromising the hermeticity of the implant.

9.3 New Horizons: Large-Scale Neural Interfaces to 10,000

Nodes and Beyond

There has been significant progress in enhancing and maintaining neural access
through dense-electrode arrays and biocompatible surface modifications as
described in previous sections, yet the scalability of practical neural interfaces to
orders of magnitude higher numbers of recording and/or stimulating nodes (thou-
sands and tens of thousands) remains a persistent challenge. As we move from the
realm of highly controlled, experimental BCIs to more naturalistic, deployable
systems, neural task complexity (NTC) is expected to grow. To gauge the expected
number of microelectrode sites with increasing NTC, Gao et al. [18] have postulated
that neural decoding would be critically dependent on access to significantly higher
numbers of neurons at high densities, ideally from an anatomically diverse neural
population. Figure 9.4 represents the outcome from a high-dimensionality neural
state-space dynamical theoretical model. Current monolithic multi-electrode con-
structs are, in our view, inherently incompatible with large-scale implementations of
BCIs due to a variety of anatomical and fabrication constraints. For example,

Fig. 9.4 A dimensionality frontier in motor cortical data. Allowed possibilities of dimensionality D
and neural task complexity, NTC, exhibit three distinct regimes: (i) the number of recorded neurons
M but not NTC restricts dimensionality, (ii) NTC but not M restricts D, and (iii) D is far less than
both M and NT C, reflecting an unexplained circuit constraint beyond smoothness and task
simplicity. (Courtesy of [18])
9 Future of Neural Interfaces 235

traditional hermetic sealing approaches require bulky Ti or ceramic cases, which add
significant volume overhead to device size. The intricacy of interconnect design,
particularly in the case of a multi-areal implant, is an added complication. While the
short tethers between intracortical arrays and the wireless electronics described in the
previous section represent a significant improvement over percutaneous tethers, they
are still relying on the traditional monolithic intra- or epicortical electrode arrays.
Looking forward, one needs to consider paradigm shifts in approaching future
BCI-compatible neural interface designs to mitigate these issues in the design of
vastly scalable systems. One such approach is to split individual recording and/or
neurostimulation electrodes into separate autonomous microscale active electronic
chiplets, each with its own internal electronics and (wireless) means to communicate
with a central information processing hub. The idea takes advantage of the remark-
able progress made by silicon microelectronics in the past 20 years which has pushed
component and transistor sizes deep into the sub-100 nm regime. For neural and BCI
applications, having access to such CMOS technologies is very attractive provided
that commensurate expertise is available for custom design of “mixed-signal”
CMOS chips (analog-digital-RF).
Among contemporary work is the concept of free-floating individual “neural
dust” sensors in conjunction with a sub-cranial interrogator which was proposed
by a number of neuroengineering researchers, most notably Seo et al. [43] (we show
concrete examples from our own work below to illustrate a related approach for
wireless microsensors and stimulators). The proposed system by Seo et al. [43]
involves ultrasonic power and telemetry to circumvent the challenges of efficient
electromagnetic energy coupling at millimeter-scale and has been demonstrated at
the level of a single 1 mm3 recording node in the peripheral nervous system
[44]. While the concept is presented as an ultra-miniaturizable sensor system, the
authors identify several fabrication and materials challenges requiring further
advances in CMOS die post-processing completion of microscale implantable
microdevices. Elsewhere, a purely electromagnetic-based distributed neural sensor
has been demonstrated by Yeon et al. [53], where a three-coil resonant near field
inductively coupled system is used to improve the efficiency of wireless power
transfer to a millimeter-sized free-floating wireless implantable neural recording
system (FF-WINeR). The approach to assembling the FF-WINeR sensor node by
manual techniques is also described by Yeon et al. [54] including integration of
microwire electrodes, ASICs with through-silicon vias (TSVs), discrete microcoils,
and surface mount components. Finally, hermetic sealing with polymeric film
deposition (parylene in this case) was applied to form a single “push-pin” recording
node measuring 1.05  1.05  0.3 mm3. Here too, the authors outline a number of
challenges ahead in scaling to many devices given the number of heterogeneous
materials and the complexity plus fabrication requirements comprising each
sensor node.
With these snapshots of recent early work, we support the view that spatially
distributed sensor systems, once developed as full-fledged medical devices can
contribute to future breakthroughs in the field of BCIs while providing new tools
for brain science. There are, however, several caveats for the development of a truly
236 F. Laiwalla and A. Nurmikko

scalable distributed sensor system. First, it is imperative that it be possible to pursue

the fabrication of sensors in a uniform, high throughput batch process. Second,
sophisticated and ultra-miniaturized sensor node electronics are needed to support
semi-autonomous operation including critical functions such as memory and net-
working. Third, the full neural interface system needs to be defined in the context of
a highly-efficient multi-node telecom network. This includes the design and imple-
mentation of a compact wearable telecommunications hub that would drive and tune
the performance specifications of individual sensor nodes in an adaptive fashion,
depending on the specific neural interface application. Fourth, the next frontier in
BCI development beyond neural recording for decoding must include neural
encoding implemented as patterned cortical microstimulation by a spatially distrib-
uted ensemble. This capability needs to be integrated into the individual nodes to
form an independent, bidirectional neural interfacing element. Fifth, for chronic
implants, it is critical to develop the means for (thin film) hermetic seals which
provide a chemically impermeable envelope for the microdevices over decades in
the body environment. Finally, the implementation of any neural interface system for
use in a human subject requires a real-time neural encoding/decoding using a
wearable neurocomputational processor with wireless telemetry to supporting com-
putational platforms, including cloud computing. The concept image of Fig. 9.5
shows schematically the general approach (with some specifics to the approach
adopted in the authors’ laboratories).
Designing a system to meet the above-mentioned caveats is a challenging prop-
osition on multiple levels. Recent work in our laboratories has focused on develop-
ing a spatially distributed implantable wireless “cortical communication” system
toward these aspirations, which is built around 500  500 μm individually address-
able custom microelectronic “neurograins” chiplets (see Fig. 9.6, which also

Fig. 9.5 Overview diagram of the spatially distributed implanted wireless neurograin read-out-
write-in network—in full deployable system context. Here, two cortical areas are targeted. The
external telecom hub in this representation is synonymous with neural computation engine
9 Future of Neural Interfaces 237

Fig. 9.6 Comparison of some mm-scale neural microdevices using electrical and/or ultrasound

compares several other recent microdevices). Each chiplet integrates RF-energy

harvesting, broadband neural recording or cortical microstimulation. Ensembles of
neurograins operate as a synchronized, networked bidirectional RF telecommunica-
tion network, scalable in principle across populations of nodes up to tens of
thousands. We have chosen to utilize near-field electromagnetic coupling at
~1 GHz to mitigate RF tissue absorption for SAR compliance and designed a single
RF-channel simultaneous power-data link for network level simplicity. An external
RF telecommunications hub, implemented as a wearable module, wirelessly man-
ages implant performance, and is envisioned also to integrate real-time data
processing for neural decoding [19, 20]. Further, we have addressed the microscale
packaging challenge by utilizing a batch-process stacked multi-layer thin-film depo-
sition process which is able to yield packaged individual devices that are <0.01 mm3
in physical volume. This thin film hermetic sealing technique using atomic layer
epitaxy, has been demonstrated to provide implant impermeability in accelerated
lifetime tests exceeding 10 years [24]. These features have been validated in the
context of ex vivo rodent brain slices as well as acute in vivo experiments [29], and
now mature for transition to a specific application context.
The above-described system from our labs is one example of an approach for a
completely untethered neural interface system that is minimally invasive from the
perspective of implant volume burden. Alternative approaches involve flexible,
floating epicortical probes with very high channel counts [48] versus integration of
genetic approaches to modify neural responsiveness (optogenetics, etc.) toward the
realization of next-generation highly efficient, integrated neural interface systems.
238 F. Laiwalla and A. Nurmikko

9.4 Summary: Outlook for High-Performance BCIs

In this chapter, we have described approaches to high data rate neural interfaces via
examples of technical solutions to intracortical/intracranial recording systems. While
early clinical trials are under way, and mainly use well-established, but bulky cabled
systems, there is a need and opportunity to invest resources to pursue innovative
approaches across the entire ecosystem from micro- and optoelectronic probe arrays
to chipscale integration of tailored on-chip signal processing and storage functions,
to name two areas.
The microminiaturization approaches reviewed here offer a particular opportunity
to enhance the functionality of the wireless interfaces by adding neuromodulation/
stimulation capabilities. Among such functions is the implementation of patterned
electrical (e.g., [14, 47]) or/and optical microstimulation for enabling truly bidirec-
tional communication opportunities with the brain. Efforts to develop such multi-
node targeted microstimulation tools are being pursued in several academic and
commercial laboratories and are likely to reach primates in the near term. These
efforts must link closely with neural decoding/encoding models using combination
of theoretical neuroscience and machine learning tools.
We re-emphasize the importance of close synergy between closed-loop sense/
stimulate-based algorithms which have recently made progress in deep brain stim-
ulation [1]; however, the challenges encountered herein, for bidirectional interfaces
comprising hundreds of channels (nodes) and beyond require that large amounts of
real-time data, recorded from the nervous system, must be processed and decoded
(e.g., [38, 52]). Among the many challenges to this aspiration are the inherent
variability and statistical entropy-driven fluctuations in neural circuits, the latter
requiring approaches which can adapt to such “non-stationarities.”

Acknowledgments The authors are very grateful to many members, past and present, in their
laboratory. These include Jihun Lee, Joonsoo Jeong, David Borton, Ming Yin, Y.-K. Song, William
R. Patterson III, Naubahar Agha, and Chris Heelan, among others. At Brown University, the authors
are part of a synergistic brain science and neurotechnology effort with many colleagues including
John Donoghue, Leigh Hochberg, David Rosler, John Simeral, and Carlos Vargas-Irwin whom we
thank for their continuing input and expertise. Special thanks are also extended to Krishna Shenoy
and his colleagues at Stanford for providing early leadership in the field.
All animal procedures referred in this chapter were conducted according to protocols approved
by Institutional Animal Care and Use Committee (IACUC) at each institution. Research in the
authors’ laboratory was supported by US National Institutes of Health, Defense Advanced Projects
Agency and the National Science Foundation.

References

1. Afshar P, Khambhati A, Stanslaski S, Carlson D, Jensen R, Dani S, . . ., Denison T (2013) A

translational platform for prototyping closed-loop neuromodulation systems. Front Neural
Circuits 6:117
9 Future of Neural Interfaces 239

2. Ando H, Takizawa K, Yoshida T, Matsushita K, Hirata M, Suzuki T (2016) Wireless

multichannel neural recording with a 128-mbps UWB transmitter for an implantable brain-
machine interfaces. IEEE Trans Biomed Circuits Syst 10(6):1068–1078
3. Aflalo T, Kellis S, Klaes C, Lee B, Shi Y, Pejsa K, . . ., Liu C (2015) Decoding motor imagery
from the posterior parietal cortex of a tetraplegic human. Science 348(6237):906–910
4. Biederman W, Yeager DJ, Narevsky N, Leverett J, Neely R, Carmena JM, . . ., Rabaey JM
(2015) A 4.78 mm 2 fully-integrated neuromodulation SoC combining 64 acquisition channels
with digital compression and simultaneous dual stimulation. IEEE J Solid State Circuits
50(4):1038–1047
5. Bonfanti A, Ceravolo M, Zambra G, Gusmeroli R, Borghi T, Spinelli AS, Lacaita AL (2010) A
multi-channel low-power IC for neural spike recording with data compression and narrowband
400-MHz MC-FSK wireless transmission. In: 2010 Proceedings of ESSCIRC, September.
IEEE, pp 330–333
6. Borton DA, Yin M, Aceros J, Nurmikko A (2013) An implantable wireless neural interface for
recording cortical circuit dynamics in moving primates. J Neural Eng 10(2):026010
7. Bozorgzadeh B, Schuweiler DR, Bobak MJ, Garris PA, Mohseni P (2016) Neurochemostat:
a neural interface SoC with integrated chemometrics for closed-loop regulation of brain
dopamine. IEEE Trans Biomed Circuits Syst 10(3):654–667
8. Brenna S, Padovan F, Neviani A, Bevilacqua A, Bonfanti A, Lacaita AL (2016) A 64-channel
965-$\mu\text {W} $ neural recording SoC with UWB wireless transmission in 130-nm
CMOS. IEEE Trans Circuits Syst Express Briefs 63(6):528–532
9. Carmena JM, Lebedev MA, Crist RE, O’Doherty JE, Santucci DM, Dimitrov DF, . . ., Nicolelis
MA (2003) Learning to control a brain–machine interface for reaching and grasping by
primates. PLoS Biol 1(2):e42
10. Castagnola E, Maiolo L, Maggiolini E, Minotti A, Marrani M, Maita F, . . ., Fadiga L (2015a)
PEDOT-CNT-coated low-impedance, ultra-flexible, and brain-conformable micro-ECoG
arrays. IEEE Trans Neural Syst Rehabil Eng 23(3):342–350
11. Castagnola V, Descamps E, Lecestre A, Dahan L, Remaud J, Nowak LG, Bergaud C (2015b)
Parylene-based flexible neural probes with PEDOT coated surface for brain stimulation and
recording. Biosens Bioelectron 67:450–457
12. Chen G, Dodson B, Hedges DM, Steffensen SC, Harb JN, Puleo C, . . ., Davis RC (2018)
Fabrication of high aspect ratio millimeter-tall free-standing carbon nanotube-based microelec-
trode arrays. ACS Biomater Sci Eng 4(5):1900–1907
13. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, . . ., Schwartz
AB (2013) High-performance neuroprosthetic control by an individual with tetraplegia. Lancet
381(9866):557–564
14. Dadarlat MC, O’doherty JE, Sabes PN (2015) A learning-based approach to artificial sensory
feedback leads to optimal integration. Nat Neurosci 18(1):138
15. Deku F, Cohen Y, Joshi-Imre A, Kanneganti A, Gardner TJ, Cogan SF (2018) Amorphous
silicon carbide ultramicroelectrode arrays for neural stimulation and recording. J Neural Eng
15(1):016007
16. Diaz-Botia CA, Luna LE, Neely RM, Chamanzar M, Carraro C, Carmena JM, . . ., Maharbiz
MM (2017) A silicon carbide array for electrocorticography and peripheral nerve recording.
J Neural Eng 14(5):056006
17. Gao H, Walker RM, Nuyujukian P, Makinwa KA, Shenoy KV, Murmann B, Meng TH (2012)
HermesE: a 96-channel full data rate direct neural Interface in 0.13μm CMOS. IEEE J Solid
State Circuits 47(4):1043–1055
18. Gao P, Trautmann E, Byron MY, Santhanam G, Ryu S, Shenoy K, Ganguli S (2017) A theory
of multineuronal dimensionality, dynamics and measurement. bioRxiv 1:214262
19. Heelan C, Komar J, Vargas-Irwin CE, Simeral JD, Nurmikko AV (2015) A mobile embedded
platform for high performance neural signal computation and communication. In: 2015 IEEE
biomedical circuits and systems conference (BioCAS), October. IEEE, pp 1–4
240 F. Laiwalla and A. Nurmikko

20. Heelan C, Nurmikko AV, Truccolo W (2018) FPGA implementation of deep-learning recurrent
neural networks with sub-millisecond real-time latency for BCI-decoding of large-scale neural
sensors (104 nodes). In: 2018 40th annual international conference of the ieee engineering in
medicine and biology society (EMBC), July. IEEE, pp 1070–1073
21. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, . . ., Donoghue JP
(2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature
442(7099):164
22. Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, . . ., Donoghue JP
(2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm.
Nature 485(7398):372
23. Jarosiewicz B, Sarma AA, Bacher D, Masse NY, Simeral JD, Sorice B, . . ., Cash SS (2015)
Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer
interface. Sci Trans Med 7(313):313ra179
24. Jeong J, Laiwalla F, Lee J, Ritasalo R, Pudas M, Larson L, . . ., Nurmikko A (2019) Conformal
hermetic sealing of wireless microelectronic implantable Chiplets by multilayered atomic layer
deposition (ALD). Adv Funct Mater 29(5):1806440
25. Jun JJ, Steinmetz NA, Siegle JH, Denman DJ, Bauza M, Barbarits B et al (2017) Fully
integrated silicon probes for high-density recording of neural activity. Nature 551(7679):232
26. Khodagholy D, Gelinas JN, Thesen T, Doyle W, Devinsky O, Malliaras GG, Buzsáki G (2015)
NeuroGrid: recording action potentials from the surface of the brain. Nat Neurosci 18(2):310
27. Kozai TD, Catt K, Du Z, Na K, Srivannavit O, Razi-ul MH, . . ., Cui XT (2016) Chronic
in vivo evaluation of PEDOT/CNT for stable neural recordings. IEEE Trans Biomed Eng
63(1):111–119
28. Kwon KY, Lee HM, Ghovanloo M, Weber A, Li W (2015) Design, fabrication, and packaging
of an integrated, wirelessly-powered optrode array for optogenetics application. Front Syst
Neurosci 9:69
29. Lee J, Mok E, Huang J, Cui L, Lee AH, Leung VW, Mercier P, Shellhammer S, Larson L,
Asbeck A, Song YK, Nurmikko A, Laiwalla F (2019) An implantable wireless network of
distributed microscale sensors for neural applications. IEEE EMBS conference on neural
engineering 2019
30. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW (2004) A brain–computer
interface using electrocorticographic signals in humans. J Neural Eng 1(2):63
31. Liu X, Zhang M, Xiong T, Richardson AG, Lucas TH, Chin PS, . . ., Van der Spiegel J (2016) A
fully integrated wireless compressed sensing neural signal acquisition system for chronic
recording and brain machine interface. IEEE Trans Biomed Circuits Syst 10(4):874–883
32. Luan L, Wei X, Zhao Z, Siegel JJ, Potnis O, Tuppen CA, . . ., Dunn AK (2017) Ultraflexible
nanoelectronic probes form reliable, glial scar–free neural integration. Sci Adv 3(2):e1601966
33. McDermott H (2016) Neurobionics: treatments for disorders of the central nervous system. In:
Neurobionics: the biomedical engineering of neural prostheses. Wiley, Hoboken, pp 213–230
34. Mestais CS, Charvet G, Sauter-Starace F, Foerster M, Ratel D, Benabid AL (2015)
WIMAGINE: wireless 64-channel ECoG recording implant for long term clinical applications.
IEEE Trans Neural Syst Rehabil Eng 23(1):10–21
35. Mirbozorgi SA, Bahrami H, Sawan M, Rusch LA, Gosselin B (2016) A single-chip full-duplex
high speed transceiver for multi-site stimulating and recording neural implants. IEEE Trans
Biomed Circuits Syst 10(3):643–653
36. Müller-Putz GR, Schwarz A, Pereira J, Ofner P (2016) From classic motor imagery to complex
movement intention decoding: the noninvasive Graz-BCI approach. In: Progress in brain
research, vol 228. Elsevier, pp 39–70
37. Oxley TJ, Opie NL, John SE, Rind GS, Ronayne SM, Wheeler TL, . . ., Steward C (2016)
Minimally invasive endovascular stent-electrode array for high-fidelity, chronic recordings of
cortical neural activity. Nat Biotechnol 34(3):320
9 Future of Neural Interfaces 241

38. Park YS, Hochberg LR, Eskandar EN, Cash SS, Truccolo W (2013) Early detection of human
epileptic seizures based on intracortical local field potentials. In: 2013 6th international IEEE/
EMBS conference on neural engineering (NER), November. IEEE, pp 323–326
39. Raducanu BC, Yazicioglu RF, Lopez CM, Ballini M, Putzeys J, Wang S et al (2016) Time
multiplexed active neural probe with 678 parallel recording sites. In: 2016 46th European solid-
state device research conference (ESSDERC). IEEE, pp 385–388
40. Rios G, Lubenov EV, Chi D, Roukes ML, Siapas AG (2016) Nanofabricated neural probes for
dense 3-D recordings of brain activity. Nano Lett 16(11):6857–6862
41. Santhanam G, Linderman MD, Gilja V, Afshar A, Ryu SI, Meng TH, Shenoy KV (2007)
HermesB: a continuous neural recording system for freely behaving primates. IEEE Trans
Biomed Eng 54(11):2037–2050
42. Schwarz DA, Lebedev MA, Hanson TL, Dimitrov DF, Lehew G, Meloy J, . . ., Ramakrishnan A
(2014) Chronic, wireless recordings of large-scale brain activity in freely moving rhesus
monkeys. Nat Methods 11(6):670
43. Seo D, Carmena JM, Rabaey JM, Alon E, Maharbiz MM (2013) Neural dust: an ultrasonic, low
power solution for chronic brain-machine interfaces. arXiv preprint arXiv:1307.2196
44. Seo D, Neely RM, Shen K, Singhal U, Alon E, Rabaey JM, . . ., Maharbiz MM (2016) Wireless
recording in the peripheral nervous system with ultrasonic neural dust. Neuron 91(3):529–539
45. Seymour EÇ, Freedman DS, Gökkavas M, Özbay E, Sahin M, Ünlü MS (2014) Improved
selectivity from a wavelength addressable device for wireless stimulation of neural tissue. Front
Neuroeng 7:5
46. Simeral JD, Kim SP, Black MJ, Donoghue JP, Hochberg LR (2011) Neural control of cursor
trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical
microelectrode array. J Neural Eng 8(2):025027
47. Tabot GA, Dammann JF, Berg JA, Tenore FV, Boback JL, Vogelstein RJ, Bensmaia SJ (2013)
Restoring the sense of touch with a prosthetic hand through a brain interface. Proc Natl Acad Sci
110(45):18279–18284
48. Tsai D, Sawyer D, Bradd A, Yuste R, Shepard KL (2017) A very large-scale microelectrode
array for cellular-resolution electrophysiology. Nat Commun 8(1):1802
49. Viventi J, Kim DH, Vigeland L, Frechette ES, Blanco JA, Kim YS, . . ., Wulsin DF (2011)
Flexible, foldable, actively multiplexed, high-density electrode array for mapping brain activity
in vivo. Nat Neurosci 14(12):1599
50. Wang W, Collinger JL, Degenhart AD, Tyler-Kabara EC, Schwartz AB, Moran DW, . . ., Kelly
JW (2013) An electrocorticographic brain interface in an individual with tetraplegia. PLoS One
8(2):e55344
51. Wessberg J, Stambaugh CR, Kralik JD, Beck PD, Laubach M, Chapin JK, . . ., Nicolelis MA
(2000) Real-time prediction of hand trajectory by ensembles of cortical neurons in primates.
Nature 408(6810):361
52. Wodlinger B, Downey JE, Tyler-Kabara EC, Schwartz AB, Boninger ML, Collinger JL (2014)
Ten-dimensional anthropomorphic arm control in a human brain machine interface: difficul-
ties, solutions, and limitations. J Neural Eng 12(1):016011
53. Yeon P, Mirbozorgi S, Ash B, Eckhardt H, Ghovanloo M (2016) Fabrication and
microassembly of a mm-sized floating probe for a distributed wireless neural interface.
Micromachines 7(9):154
54. Yeon P, Gonzalez JL, Zia M, Rajan SK, May GS, Bakir MS, Ghovanloo M (2017)
Microfabrication, assembly, and hermetic packaging of mm-sized free-floating neural probes.
In 2017 IEEE biomedical circuits and systems conference (BioCAS), October. IEEE, pp 1–4
55. Yin M, Borton DA, Komar J, Agha N, Lu Y, Li H, . . ., Larson L (2014) Wireless neurosensor
for full-spectrum electrophysiology recordings during free behavior. Neuron 84(6):1170–1182