Wolf et al.

Genome Biology (2018) 19:15

https://doi.org/10.1186/s13059-017-1382-0

S O FT W A R E Open Access

SCANPY: large-scale single-cell gene

expression data analysis
F. Alexander Wolf1* , Philipp Angerer1 and Fabian J. Theis1,2*

Abstract
SCANPY is a scalable toolkit for analyzing single-cell gene expression data. It includes methods for preprocessing,
visualization, clustering, pseudotime and trajectory inference, differential expression testing, and simulation of gene
regulatory networks. Its Python-based implementation efficiently deals with data sets of more than one million cells
(https://github.com/theislab/Scanpy). Along with SCANPY, we present ANNDATA, a generic class for handling
annotated data matrices (https://github.com/theislab/anndata).
Keywords: Single-cell transcriptomics, Machine learning, Scalability, Graph analysis, Clustering, Pseudotemporal
ordering, Trajectory inference, Differential expression testing, Visualization, Bioinformatics

Background to PHENOGRAPH [18–20], identification of marker genes

Simple integrated analysis work flows for single-cell tran- for clusters via differential expression tests and pseu-
scriptomic data [1] have been enabled by frameworks such dotemporal ordering via diffusion pseudotime [21], which
as SEURAT [2], MONOCLE [3], SCDE/PAGODA [4], MAST compares favorably [22] with MONOCLE 2 [22], and
[5], C ELL R ANGER [6], S CATER [7], and S CRAN [8]. How- W ISHBONE [23] (Fig. 1a).
ever, these frameworks do not scale to the increasingly
available large data sets with up to and more than one mil- SCANPY is benchmarked in comparisons with established
lion cells. Here, we present a framework that overcomes packages
this limitation and provides similar analysis possibilities. In a detailed clustering tutorial of 2700 peripheral
Moreover, in contrast to the existing R-based frameworks, blood mononuclear cells (PBMCs), adapted from one of
S CANPY’s Python-based implementation is easy to inter- SEURAT’s tutorials (http://satijalab.org/seurat/pbmc3k_
face with advanced machine-learning packages, such as tutorial.html) [2], all steps starting from raw count data to
TENSORFLOW [9]. the identification of cell types are carried out, providing
speedups between 5 and 90 times in each step (https://
Results github.com/theislab/scanpy_usage/tree/master/170505_
SCANPY integrates canonical analysis methods in a scalable seurat). Benchmarking against the more run-time opti-
way mized C ELL R ANGER R kit [6], we demonstrate a speedup
S CANPY integrates the analysis possibilities of established of 5 to 16 times for a data set of 68,579 PBMCs (Fig. 1a,b,
R-based frameworks and provides them in a scalable and https://github.com/theislab/scanpy_usage/tree/master/
modular form. Specifically, S CANPY provides preprocess- 170503_zheng17) [6]. Moreover, we demonstrate the
ing comparable to SEURAT [10] and C ELL R ANGER [6], feasibility of analyzing 1.3 million cells without subsam-
visualization through TSNE [11, 12], graph-drawing [13– pling in a few hours of computing time on eight cores of
15] and diffusion maps [11, 16, 17], clustering similar a small computing server (Fig. 1c, https://github.com/
theislab/scanpy_usage/tree/master/170522_visualizing_
*Correspondence: [email protected]; one_million_cells). Thus, S CANPY provides tools with
[email protected] speedups that enable an analysis of data sets with more
1
Helmholtz Zentrum München – German Research Center for Environmental
Health, Institute of Computational Biology, Neuherberg, Munich, Germany than one million cells and an interactive analysis with run
2
Department of Mathematics, Technische Universität München, Munich, times of the order of seconds for about 100,000 cells.
Germany

© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the
Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Wolf et al. Genome Biology (2018) 19:15 Page 2 of 5

a b

Fig. 1 a SCANPY’s analysis features. We use the example of 68,579 peripheral blood mononuclear cells of [6]. We regress out confounding variables,
normalize, and identify highly variable genes. TSNE and graph-drawing (Fruchterman–Reingold) visualizations show cell-type annotations obtained
by comparisons with bulk expression. Cells are clustered using the Louvain algorithm. Ranking differentially expressed genes in clusters identifies
the MS4A1 marker gene for B cells in cluster 7, which agrees with the bulk labels. We use pseudotemporal ordering from a root cell in the CD34+
cluster and detect a branching trajectory, visualized with TSNE and diffusion maps. b Speedup over CELL RANGER R kit. We consider representative
steps of the analysis [6]. c Visualizing and clustering 1.3 million cells. The data, brain cells from E18 mice, are publicly available from 10x Genomics.
PCA = principal component analysis, DC = diffusion component

In addition to the mentioned standard clustering-based disk, a format independent of platform, framework, and
analyses approaches, we demonstrate the reconstruc- language. This allows operating on an ANNDATA object
tion of branching developmental processes via diffusion without fully loading it into memory—the functionality
pseudotime [21] as in the original paper (https:// is offered via ANNDATA’s backed mode as opposed to its
github.com/theislab/scanpy_usage/tree/master/170502_ memory mode. To simplify memory-efficient pipelines,
haghverdi16), the simulation of single cells using S CANPY’s functions operate in-place by default but allow
literature-curated gene regulatory networks based on the optional non-destructive transformation of objects.
the ideas of [24] (https://github.com/theislab/scanpy_ Pipelines written this way can then also be run in
usage/tree/master/170430_krumsiek11), and the analysis backed mode to exploit online-learning formulations of
of deep-learning results for single-cell imaging data [25] algorithms. Almost all of S CANPY’s tools are parallelized.
(https://github.com/theislab/scanpy_usage/tree/master/ S CANPY introduces a class for representing a graph
170529_images). of neighborhood relations among data points. The com-
putation of neighborhood relations is much faster than
SCANPY introduces efficient modular implementation in the popular reference package [27]. This is achieved
choices by aggregating rows (observations) in a data matrix to
With S CANPY, we introduce the class ANNDATA— submatrices and computing distances for each submatrix
with a corresponding package ANNDATA—which stores using fast parallelized matrix multiplication. Moreover,
a data matrix with the most general annotations pos- the class provides several functions to compute random-
sible: annotations of observations (samples, cells) and walk-based metrics that are not available in other graph
variables (features, genes), and unstructured annotations. software [14, 28, 29]. Typically, S CANPY’s tools reuse a
As S CANPY is built around that class, it is easy to add new once-computed, single graph representation of data and
functionality to the toolkit. All statistics and machine- hence, avoid the use of different, potentially inconsistent,
learning tools extract information from a data matrix, and computationally expensive representations of data.
which can be added to an A NNDATA object while leaving
the structure of A NNDATA unaffected. ANNDATA is sim- Conclusions
ilar to R’s E XPRESSION SET [26], but supports sparse data S CANPY’s scalability directly addresses the strongly
and allows HDF5-based backing of A NNDATA objects on increasing need for aggregating larger and larger data
Wolf et al. Genome Biology (2018) 19:15 Page 3 of 5

sets [30] across different experimental setups, for preprocessing—the trajectory inference tool SCIMITAR
example within challenges such as the Human Cell (https://github.com/dimenwarper/scimitar), the cluster-
Atlas [31]. Moreover, being implemented in a highly ing tool PHENO GRAPH (https://github.com/jacoblevine/
modular fashion, S CANPY can be easily developed fur- PhenoGraph) [19], the single-cell experiment design tool
ther and maintained by a community. The transfer of MIMOSCA (https://github.com/asncd/MIMOSCA)[50],
the results obtained with different tools used within UMIS (https://github.com/vals/umis) for handling raw
the community is simple, as S CANPY’s data storage read data [51], the tree-inference tool ECLAIR (https://
formats and objects are language independent and github.com/GGiecold/ECLAIR) [52], and the framework
cross-platform. S CANPY integrates well into the exist- FLOTILLA (https://github.com/yeolab/flotilla), which
ing Python ecosystem, in which no comparable toolkit comes with modules for simple visualization, simple clus-
yet exists. tering, and differential expression testing. Hence, only
During the revision of this article, the loom file for- the latter provides a data analysis framework that solves
mat (https://github.com/linnarsson-lab/loompy) was pro- more than one specific task. In contrast to S CANPY,
posed for HDF5-based storage of annotated data. Within however, FLOTILLA is neither targeted at single-cell nor
a joint effort of facilitating data exchange across different at large-scale data and does not provide any graph-based
labs, ANNDATA now supports importing and exporting to methods, which are the core of S CANPY. Also, FLOTILLA
loom (https://github.com/linnarsson-lab/loompy). In this is built around a complicated class STUDY, which contains
context, we acknowledge the discussions with S. Linnar- data, tools, and plotting functions. S CANPY, by contrast,
son, which motivated us to extend ANNDATA’s previously is built around a simple HDF5-backed class ANNDATA,
static to a dynamic HDF5 backing. Just before submis- which makes S CANPY both scalable and extendable
sion of this manuscript, a C++ library that provides sim- (law of Demeter).
ple interfacing of HDF5-backed matrices in R was made
available as a preprint [32]. Availability and requirements
S CANPY’s and ANNDATA’s open-source code are main-
Methods tained on GITHUB (https://github.com/theislab/scanpy,
SCANPY’s technological foundations https://github.com/theislab/anndata) and published
S CANPY’s core relies on NUMPY [33], S CIPY [34], M AT- under the BSD3 license.
PLOTLIB [35], PANDAS [36], and H 5 PY [37]. Parts of the
S CANPY and ANNDATA are released via the Python
toolkit rely on SCIKIT-LEARN [27], STATSMODELS [38], packaging index: https://pypi.python.org/pypi/scanpy
SEABORN [39], NETWORK X [28], IGRAPH [14], the TSNE and https://pypi.python.org/pypi/anndata.
package of [40], and the Louvain clustering package of Demonstrations and benchmarks discussed in the main
[41]. The ANNDATA class—available within the package text are all stored at https://github.com/theislab/scanpy_
ANNDATA —relies only on NUMPY , S CIPY , PANDAS , and
usage and summarized here:
H 5 PY .
S CANPY’s Python-based implementation allows easy • Analyzing 68,579 PBMCs (Fig. 1) in a comparison
interfacing to advanced machine-learning packages such with the CELL RANGER R kit [6]: https://github.com/
as TENSORFLOW [9] for deep learning [42], L IMIX for theislab/scanpy_usage/tree/master/170503_zheng17.
linear mixed models [43], and GP Y/GPFLOW for Gaus- • Clustering and identifying cell types, adapted from
sian processes [44, 45]. However, we note that the Python and benchmarked with http://satijalab.org/seurat/
ecosystem comes with less possibilities for classical statis- pbmc3k_tutorial.html and one of SEURAT’s tutorials
tical analyses compared to R. [2]: https://github.com/theislab/scanpy_usage/tree/
master/170505_seurat.
Comparison with existing Python packages for single-cell • Visualizing and clustering 1.3 million cells (Fig. 1c):
analysis https://github.com/theislab/scanpy_usage/tree/
Aside from the highly popular SC LVM (https://github. master/170522_visualizing_one_million_cells.
com/PMBio/scLVM) [46, 47], which uses Gaussian • Reconstructing branching processes via diffusion
process latent variable models for inferring hidden pseudotime [21]: https://github.com/theislab/
sources of variation, there are, among others, the visu- scanpy_usage/tree/master/170502_haghverdi16.
alization frameworks FASTPROJECT (https://github. • Simulating single cells using gene regulatory
com/YosefLab/FastProject) [48], ACCENSE (http:// networks [24]: https://github.com/theislab/scanpy_
www.cellaccense.com/) [49], and SPRING (https:// usage/tree/master/170430_krumsiek11.
github.com/AllonKleinLab/SPRING) [15]—the lat- • Analyzing deep-learning results for single-cell images
ter uses the JavaScript package (http://d3js.org D3.js [25]: https://github.com/theislab/scanpy_usage/tree/
for the actual visualization and Python only for master/170529_images.
Wolf et al. Genome Biology (2018) 19:15 Page 4 of 5

The data sets used in demonstrations and benchmarks 13. Reingold EM. Graph drawing by force-directed placement. Softw Pract
are three data sets from 10x Genomics. Exp. 1991;21:1129–64.
14. Csardi G, Nepusz T. The IGRAPH SOFTWARE PACKAGE FOR COMPLEX
Programming language: Python NETWORK RESEARCH . InterJournal Compl Syst. 2006;2006:1695.
Operating system: Linux, Mac OS and Windows 15. Weinreb C, Wolock S, Klein A. SPRING: a kinetic interface for visualizing
high dimensional single-cell expression data. bioRxiv. 2017. https://doi.
Acknowledgements org/10.1093/bioinformatics/btx792.
We thank the authors of SEURAT, CELL RANGER, and SPRING for sharing their 16. Buettner F, Theis FJ. Diffusion maps for high-dimensional single-cell
great tutorials. We are grateful to Sten Linnarson for discussions on analysis of differentiation data. Bioinformatics. 2015;31:2989–98.
HDF5-backing of data on disk. We thank S. Tritschler, L. Simon, D. S. Fischer, 17. Angerer P, et al. DESTINY: diffusion maps for large-scale single-cell data in
and M. Büttner for commenting on the software package. We thank M. R. Bioinformatics. 2015;32:1241.
Lotfollahi for clustering the 1.3-million-cell data set and N. K. Chlis for setting 18. Blondel VD, Guillaume J-L, Lambiotte R, Lefebvre E. Fast unfolding of
up installation instructions for Windows. communities in large networks. J Stat Mech. 2008;2008:P10008.
19. Levine JH, et al. Data-driven phenotypic dissection of AML reveals
Funding progenitor-like cells that correlate with prognosis. Cell. 2015;162:184–97.
FAW acknowledges the support of the Helmholtz Postdoc Programme, 20. Xu C, Su Z. Identification of cell types from single-cell transcriptomes
Initiative and Networking Fund of the Helmholtz Association. using a novel clustering method. Bioinformatics. 2015;31:1974–80.
FJT acknowledges support from the German Research Foundation (DFG) 21. Haghverdi L, Bü, Wolf FA, Buettner F, Theis FJ. Diffusion pseudotime
within the Collaborative Research Centre 1243, Subproject A17. robustly reconstructs branching cellular lineages. Nat Methods. 2016;13:
845–8.
Authors’ contributions 22. Qiu X, et al. Reversed graph embedding resolves complex single-cell
FAW conceived the project and developed the software. PA co-developed the trajectories. Nat Methods. 2017;14:979–82.
software, mainly in regard to architecture and maintainability. FJT supervised 23. Setty, M, et al. WISHBONE identifies bifurcating developmental trajectories
the project and helped interpret and present the results. FAW wrote the from single-cell data. Nat Biotechnol. 2016;34:637–45.
manuscript with the help of PA and FJT. All authors read and approved the 24. Wittmann, DM, et al. Transforming Boolean models to continuous
final manuscript. models: methodology and application to T-cell receptor signaling. BMC
Syst Biol. 2009;3:98.
Ethics approval and consent to participate 25. Eulenberg P, et al. Reconstructing cell cycle and disease progression
Ethics approval was not applicable for this study. using deep learning. Nat Commun. 2017;8:463.
26. Huber, W, et al. Orchestrating high-throughput genomic analysis with
Competing interests
BIOCONDUCTOR. Nat Methods. 2015;12:115–21.
None of the authors declare competing interests.
27. Pedregosa F, et al. SCIKIT-LEARN: machine learning in Python. J Mach Learn
Res. 2011;12:2825–30.
Publisher’s Note 28. Hagberg AA, Schult DA, Swart PJ. Exploring network structure, dynamics,
Springer Nature remains neutral with regard to jurisdictional claims in and function using NETWORKX. In: Proceedings of the 7th Python in
published maps and institutional affiliations. Science Conference (SciPy2008). Pasadena; 2008. p. 11–15.
29. Bastian M, Heymann S, Jacomy M. GEPHI: an open source software for
Received: 16 August 2017 Accepted: 20 December 2017 exploring and manipulating networks. International AAAI Conference on
Weblogs and Social Media. 2009.
30. Angerer, P, et al. Single cells make big data: new challenges and
References opportunities in transcriptomics. Curr Opin Syst Biol. 2017;4:85–91.
1. Wagner A, Regev A, Yosef N. Revealing the vectors of cellular identity 31. Regev A, et al. Science forum: the human cell atlas. eLife. 2017;6:e27041.
with single-cell genomics. Nat Biotechnol. 2016;34:1145–60. 32. Lun ATL, Pagè H, Smith ML. BEACHMAT: a BIOCONDUCTOR C++ API for
2. Satija R, Farrell JA, Gennert D, Schier AF, Regev A. Spatial reconstruction accessing single-cell genomics data from a variety of R matrix types.
of single-cell gene expression data. Nat Biotechnol. 2015;33:495–502. bioRxiv. 2017. https://doi.org/10.1101/167445.
3. Trapnell C, et al. The dynamics and regulators of cell fate decisions are 33. van der Walt S, Colbert SC, Varoquaux G. The NUMPY array: a structure for
revealed by pseudotemporal ordering of single cells. Nat Biotechnol. efficient numerical computation. Comput Sci Eng. 2011;13:22–30.
2014;32:381–6. 34. Jones E, Oliphant T, Peterson P, et al. SCIPY: open source scientific tools
4. Kharchenko PV, Silberstein L, Scadden DT, Bayesian approach to for Python. 2001. https://www.scipy.org/citing.html.
single-cell differential expression analysis. Nat Methods. 2014;11:740–2. 35. Hunter JD. MATPLOTLIB: a 2D graphics environment. Comput Sci Eng.
5. Finak, G, et al. MAST: a flexible statistical framework for assessing 2007;9:90–5.
transcriptional changes and characterizing heterogeneity in single-cell 36. McKinney W. Data structures for statistical computing in Python. In: van
RNA sequencing data. Genome Biol. 2015;16:278. der Walt S, Millman J, editors. Proceedings of the 9th Python in Science
6. Zheng GXY, et al. Massively parallel digital transcriptional profiling of Conference; 2010. p. 51–6.
single cells. Nat Commun. 2017;8:14049. 37. Collette A. Python and HDF5. Sebasto pol: O’Reilly; 2013.
7. McCarthy D, Wills Q, Campbell K. SCATER: single-cell analysis toolkit for 38. Seabold S, Perktold J. STATSMODELS: econometric and statistical
gene expression data in R. Bioinformatics. 2017;33:1179. modeling with Python. 9th Python in Science Conference. 2010.
8. Lun A, McCarthy D, Marioni J. A step-by-step workflow for low-level 39. Waskom, M, et al. In: Varoquaux G, Vaught T, Millman J, editors. SEABORN;
analysis of single-cell RNA-seq data with BIOCONDUCTOR. F1000Research. 2016. http://doi.org/10.5281/zenodo.12710, https://networkx.github.io/
2016;5:2122. documentation/networkx-1.10/reference/citing.html.
9. Abadi M, et al. TENSORFLOW: large-scale machine learning on 40. Ulyanov D. MULTICORE-TSNE. 2016. https://github.com/DmitryUlyanov/
heterogeneous systems. 2015. https://www.tensorflow.org/about/bib. Multicore-TSNE.
10. Macosko EZ, et al. Highly parallel genome-wide expression profiling of 41. Traag V, Louvain. GITHUB. 2017. https://doi.org/10.5281/zenodo.595481.
individual cells using nanoliter droplets. Cell. 2015;161:1202–14. 42. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:
11. Coifman RR, et al. Geometric diffusions as a tool for harmonic analysis and 436–44.
structure definition of data: diffusion maps. Proc Natl Acad Sci. 2005;102: 43. Lippert C, Casale FP, Rakitsch B, Stegle O. In: van der Walt S, Millman
7426–31. J, editors. LIMIX: genetic analysis of multiple traits; 2014. https://doi.org/
12. Amir EAD, Davis KL, Tadmor MD, Simonds EF, Levine JH, Bendall SC, et 10.1101/003905, http://conference.scipy.org/proceedings/scipy2010/
al. VISNE enables visualization of high dimensional single-cell data and mckinney.html. bioRxiv.
reveals phenotypic heterogeneity of leukemia. Nat Biotechnol. 2013;31: 44. Matthews AGdeG, van der Wilk M, Nickson T, Fujii K, Boukouvalas A,
545–52. Le’on-Villagr’a P, Ghahramani Z, Hensman J. GPflow: A Gaussian process
Wolf et al. Genome Biology (2018) 19:15 Page 5 of 5

library using TensorFlow. J Mach Learn Res. 2017;18(40):1–6. http://jmlr.

org/papers/v18/16-537.html.
45. Matthews de, G, Alexander G, et al. GPFLOW: a Gaussian process library
using TENSORFLOW. J Mach Learn Res. 2017;18:1–6. https://github.com/
SheffieldML/GPy.
46. Buettner F, et al. Computational analysis of cell-to-cell heterogeneity in
single-cell RNA-sequencing data reveals hidden subpopulations of cells.
Nat Biotechnol. 2015;33:155.
47. Buettner F, Pratanwanich N, McCarthy DJ, Marioni JC, Stegle O.
F - SC LVM: scalable and versatile factor analysis for single-cell RNA-seq.
Genome Biol. 2017;18:212.
48. DeTomaso D, Yosef N. FASTPROJECT: a tool for low-dimensional analysis of
single-cell RNA-seq data. BMC Bioinform. 2016;17:315.
49. Shekhar K, Brodin P, Davis MM, Chakraborty AK. Automatic classification
of cellular expression by nonlinear stochastic embedding (ACCENSE);
2013. p 202–7.
50. Dixit A, et al. PERTURB-SEQ: dissecting molecular circuits with scalable
single-cell RNA profiling of pooled genetic screens. Cell. 2016;167:
1853–66.e17.
51. Svensson V, et al. Power analysis of single cell RNA-sequencing
experiments. Nat Methods. 2017;14:381.
52. Giecold G, Marco E, Garcia SP, Trippa L, Yuan G-C. Robust lineage
reconstruction from high-dimensional single-cell data. Nucleic Acids Res.
2016;44:e122.

Submit your next manuscript to BioMed Central

and we will help you at every step:
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research

Submit your manuscript at

www.biomedcentral.com/submit