pandas: powerful Python data analysis toolkit

Release 0.7.1

Wes McKinney

February 29, 2012

CONTENTS

Whats New 1.1 v.0.7.1 (February 29, 2012) . . . . . . . . . . . . . . . 1.2 v.0.7.0 (February 9, 2012) . . . . . . . . . . . . . . . . 1.3 v.0.6.1 (December 13, 2011) . . . . . . . . . . . . . . . 1.4 v.0.6.0 (November 25, 2011) . . . . . . . . . . . . . . . 1.5 v.0.5.0 (October 24, 2011) . . . . . . . . . . . . . . . . 1.6 v.0.4.3 through v0.4.1 (September 25 - October 9, 2011) Installation 2.1 Python version support 2.2 Binary installers . . . 2.3 Dependencies . . . . . 2.4 Optional dependencies 2.5 Installing from source 2.6 Running the test suite

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

3 3 3 9 9 11 12 15 15 15 15 15 16 16 17 19 19 20 20 20 20 20 23 23 27 40 43 43 43 45 48 52 55 61 63 i

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

3 4

Frequently Asked Questions (FAQ) Package overview 4.1 Data structures at a glance . . . 4.2 Mutability and copying of data . 4.3 Getting Support . . . . . . . . 4.4 Credits . . . . . . . . . . . . . 4.5 Development Team . . . . . . . 4.6 License . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

Intro to Data Structures 5.1 Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 DataFrame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Essential basic functionality 6.1 Head and Tail . . . . . . . . . . . 6.2 Attributes and the raw ndarray(s) 6.3 Flexible binary operations . . . . 6.4 Descriptive statistics . . . . . . . 6.5 Function application . . . . . . . 6.6 Reindexing and altering labels . . 6.7 Iteration . . . . . . . . . . . . . . 6.8 Sorting by index and value . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

. . . . . . . .

6.9 Copying, type casting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.10 Pickling and serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.11 Console Output Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Indexing and selecting data 7.1 Basics . . . . . . . . . . . . . . . . . . . . 7.2 Advanced indexing with labels . . . . . . . 7.3 Index objects . . . . . . . . . . . . . . . . 7.4 Hierarchical indexing (MultiIndex) . . . . 7.5 Adding an index to an existing DataFrame 7.6 Indexing internal details . . . . . . . . . . Computational tools 8.1 Statistical functions . . . . . . . . . . . . 8.2 Moving (rolling) statistics / moments . . . 8.3 Exponentially weighted moment functions 8.4 Linear and panel regression . . . . . . . . Working with missing data 9.1 Missing data basics . . . . . . . . . . . 9.2 Calculations with missing data . . . . . 9.3 Cleaning / lling missing data . . . . . 9.4 Missing data casting rules and indexing

64 65 66 69 69 76 80 81 90 92

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

95 . 95 . 97 . 101 . 102 109 109 111 112 115 117 117 121 122 124 125 126 127

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

10 Group By: split-apply-combine 10.1 Splitting an object into groups . 10.2 Iterating through groups . . . . 10.3 Aggregation . . . . . . . . . . 10.4 Transformation . . . . . . . . . 10.5 Dispatching to instance methods 10.6 Flexible apply . . . . . . . . 10.7 Other useful features . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

11 Merge, join, and concatenate 129 11.1 Concatenating objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 11.2 Database-style DataFrame joining/merging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 12 Reshaping and Pivot Tables 12.1 Reshaping by pivoting DataFrame objects 12.2 Reshaping by stacking and unstacking . . 12.3 Reshaping by Melt . . . . . . . . . . . . 12.4 Combining with stats and GroupBy . . . 12.5 Pivot tables and cross-tabulations . . . . 13 Time Series / Date functionality 13.1 DateOffset objects . . . . . . . . . . 13.2 Generating date ranges (DateRange) . 13.3 Time series-related instance methods 13.4 Up- and downsampling . . . . . . . . 147 147 148 151 152 153 157 157 159 161 162

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

14 Plotting with matplotlib 165 14.1 Basic plotting: plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 14.2 Other plotting features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 15 IO Tools (Text, CSV, HDF5, ...) 175

ii

15.1 15.2 15.3 15.4

Clipboard . . . . CSV & Text les Excel les . . . HDF5 (PyTables)

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

175 175 181 181

16 Sparse data structures 183 16.1 SparseArray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 16.2 SparseList . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 16.3 SparseIndex objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 17 Caveats and Gotchas 17.1 NaN, Integer NA values and NA type promotions 17.2 Integer indexing . . . . . . . . . . . . . . . . . 17.3 Label-based slicing conventions . . . . . . . . . 17.4 Miscellaneous indexing gotchas . . . . . . . . . 187 187 189 189 189 193 193 194 194

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

18 rpy2 / R interface 18.1 Transferring R data sets into Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Calling R functions with pandas objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3 High-level interface to R estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19 Related Python libraries 195 19.1 la (larry) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 19.2 scikits.statsmodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 19.3 scikits.timeseries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 20 Comparison with R / R libraries 20.1 data.frame . . . . . . . . . 20.2 zoo . . . . . . . . . . . . . 20.3 xts . . . . . . . . . . . . . 20.4 plyr . . . . . . . . . . . . . 20.5 reshape / reshape2 . . . . . 21 API Reference 21.1 General functions . 21.2 Series . . . . . . . 21.3 DataFrame . . . . 21.4 Panel . . . . . . . Python Module Index Python Module Index Index 197 197 197 197 197 197 199 199 214 236 270 271 273 275

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

iii

iv

pandas: powerful Python data analysis toolkit, Release 0.7.1

PDF Version Date: February 29, 2012 Version: 0.7.1 Binary Installers: http://pypi.python.org/pypi/pandas Source Repository: http://github.com/pydata/pandas Issues & Ideas: https://github.com/pydata/pandas/issues Q&A Support: http://stackoverow.com/questions/tagged/pandas Developer Mailing List: http://groups.google.com/group/pystatsmodels pandas is a Python package providing fast, exible, and expressive data structures designed to make working with relational or labeled data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and exible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal. pandas is well suited for many different kinds of data: Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet Ordered and unordered (not necessarily xed-frequency) time series data. Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the vast majority of typical use cases in nance, statistics, social science, and many areas of engineering. For R users, DataFrame provides everything that Rs data.frame provides and much more. pandas is built on top of NumPy and is intended to integrate well within a scientic computing environment with many other 3rd party libraries. Here are just a few of the things that pandas does well: Easy handling of missing data (represented as NaN) in oating point as well as non-oating point data Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations Powerful, exible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects Intelligent label-based slicing, fancy indexing, and subsetting of large data sets Intuitive merging and joining data sets Flexible reshaping and pivoting of data sets Hierarchical labeling of axes (possible to have multiple labels per tick) Robust IO tools for loading data from at les (CSV and delimited), Excel les, databases, and saving / loading data from the ultrafast HDF5 format Time series-specic functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc. Many of these principles are here to address the shortcomings frequently experienced using other languages / scientic research environments. For data scientists, working with data is typically divided into multiple stages: munging and

CONTENTS

pandas: powerful Python data analysis toolkit, Release 0.7.1

cleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting or tabular display. pandas is the ideal tool for all of these tasks. Some other notes pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However, as with anything else generalization usually sacrices performance. So if you focus on one feature for your application you may be able to create a faster specialized tool. pandas will soon become a dependency of statsmodels, making it an important part of the statistical computing ecosystem in Python. pandas has been used extensively in production in nancial applications. Note: This documentation assumes general familiarity with NumPy. If you havent used NumPy much or at all, do invest some time in learning about NumPy rst. See the package overview for more detail about whats in the library.

CONTENTS

CHAPTER

ONE

WHATS NEW
These are new features and improvements of note in each release.

1.1 v.0.7.1 (February 29, 2012)

This release includes a few new features and addresses over a dozen bugs in 0.7.0.

1.1.1 New features

Add to_clipboard function to pandas namespace for writing objects to the system clipboard (#774) Add itertuples method to DataFrame for iterating through the rows of a dataframe as tuples (#818) Add ability to pass ll_value and method to DataFrame and Series align method (#806, #807) Add ll_value option to reindex, align methods (#784) Enable concat to produce DataFrame from Series (#787) Add between method to Series (#802) Add HTML representation hook to DataFrame for the IPython HTML notebook (#773) Support for reading Excel 2007 XML documents using openpyxl

1.1.2 Performance improvements

Improve performance and memory usage of llna on DataFrame Can concatenate a list of Series along axis=1 to obtain a DataFrame (#787)

1.2 v.0.7.0 (February 9, 2012)

1.2.1 New features
New unied merge function for efciently performing full gamut of database / relational-algebra operations. Refactored existing join methods to use the new infrastructure, resulting in substantial performance gains (GH220, GH249, GH267)

pandas: powerful Python data analysis toolkit, Release 0.7.1

New unied concatenation function for concatenating Series, DataFrame or Panel objects along an axis. Can form union or intersection of the other axes. Improves performance of Series.append and DataFrame.append (GH468, GH479, GH273) Can pass multiple DataFrames to DataFrame.append to concatenate (stack) and multiple Series to Series.append too Can pass list of dicts (e.g., a list of JSON objects) to DataFrame constructor (GH526) You can now set multiple columns in a DataFrame via __getitem__, useful for transformation (GH342) Handle differently-indexed output values in DataFrame.apply (GH498)
In [902]: df = DataFrame(randn(10, 4)) In [903]: df.apply(lambda x: Out[903]: 0 1 count 10.000000 10.000000 mean -0.556258 -0.268695 std 0.775352 1.072879 min -1.696646 -2.251671 25% -1.234969 -0.734632 50% -0.340199 -0.311720 75% -0.037106 0.451343 max 0.401255 1.199708 x.describe()) 2 10.000000 -0.215066 1.431537 -1.739074 -0.901618 -0.555855 -0.072265 2.997692 3 10.000000 0.073787 0.962624 -1.545595 -0.384952 0.001073 0.867466 1.357252

Add reorder_levels method to Series and DataFrame (PR534) Add dict-like get function to DataFrame and Panel (PR521) Add DataFrame.iterrows method for efciently iterating through the rows of a DataFrame Add DataFrame.to_panel with code adapted from LongPanel.to_long Add reindex_axis method added to DataFrame Add level option to binary arithmetic functions on DataFrame and Series Add level option to the reindex and align methods on Series and DataFrame for broadcasting values across a level (GH542, PR552, others) Add attribute-based item access to Panel and add IPython completion (PR563) Add logy option to Series.plot for log-scaling on the Y axis Add index and header options to DataFrame.to_string Can pass multiple DataFrames to DataFrame.join to join on index (GH115) Can pass multiple Panels to Panel.join (GH115) Added justify argument to DataFrame.to_string to allow different alignment of column headers Add sort option to GroupBy to allow disabling sorting of the group keys for potential speedups (GH595) Can pass MaskedArray to Series constructor (PR563) Add Panel item access via attributes and IPython completion (GH554) Implement DataFrame.lookup, fancy-indexing analogue for retrieving values given a sequence of row and column labels (GH338) Can pass a list of functions to aggregate with groupby on a DataFrame, yielding an aggregated result with hierarchical columns (GH166)

Chapter 1. Whats New

pandas: powerful Python data analysis toolkit, Release 0.7.1

Can call cummin and cummax on Series and DataFrame to get cumulative minimum and maximum, respectively (GH647) value_range added as utility function to get min and max of a dataframe (GH288) Added encoding argument to read_csv, read_table, to_csv and from_csv for non-ascii text (GH717) Added abs method to pandas objects Added crosstab function for easily computing frequency tables Added isin method to index objects Added level argument to xs method of DataFrame.

1.2.2 API Changes to integer indexing

One of the potentially riskiest API changes in 0.7.0, but also one of the most important, was a complete review of how integer indexes are handled with regard to label-based indexing. Here is an example:
In [904]: s = Series(randn(10), index=range(0, 20, 2)) In [905]: s Out[905]: 0 0.324022 2 -0.983597 4 0.856254 6 1.396004 8 -0.374833 10 -0.883614 12 0.251235 14 0.280914 16 -1.374563 18 0.066904 In [906]: s[0] Out[906]: 0.32402202262891427 In [907]: s[2] Out[907]: -0.98359726644878798 In [908]: s[4] Out[908]: 0.85625373985124742

This is all exactly identical to the behavior before. However, if you ask for a key not contained in the Series, in versions 0.6.1 and prior, Series would fall back on a location-based lookup. This now raises a KeyError:
In [2]: s[1] KeyError: 1

This change also has the same impact on DataFrame:

In [3]: df = DataFrame(randn(8, 4), index=range(0, 16, 2)) In [4]: df 0 1 2 0 0.88427 0.3363 -0.1787 2 0.14451 -0.1415 0.2504 4 -1.44779 -0.9186 -1.4996

3 0.03162 0.58374 0.27163

1.2. v.0.7.0 (February 9, 2012)

pandas: powerful Python data analysis toolkit, Release 0.7.1

6 8 10 12 14

-0.26598 -2.4184 -0.2658 0.11503 -0.58776 0.3144 -0.8566 0.61941 0.10940 -0.7175 -1.0108 0.47990 -1.16919 -0.3087 -0.6049 -0.43544 -0.07337 0.3410 0.0424 -0.16037

In [5]: df.ix[3] KeyError: 3

In order to support purely integer-based indexing, the following methods have been added: Method Series.iget_value(i) Series.iget(i) DataFrame.irow(i) DataFrame.icol(j) DataFrame.iget_value(i, j) Description Retrieve value stored at location i Alias for iget_value Retrieve the i-th row Retrieve the j-th column Retrieve the value at row i and column j

1.2.3 API tweaks regarding label-based slicing

Label-based slicing using ix now requires that the index be sorted (monotonic) unless both the start and endpoint are contained in the index:
In [909]: s = Series(randn(6), index=list(gmkaec)) In [910]: s Out[910]: g 1.109413 m 0.346131 k -0.359977 a -1.018111 e 0.790509 c 0.787672

Then this is OK:

In [911]: s.ix[k:e] Out[911]: k -0.359977 a -1.018111 e 0.790509

But this is not:

In [12]: s.ix[b:h] KeyError b

If the index had been sorted, the range selection would have been possible:
In [912]: s2 = s.sort_index() In [913]: s2 Out[913]: a -1.018111 c 0.787672 e 0.790509 g 1.109413 k -0.359977

Chapter 1. Whats New

pandas: powerful Python data analysis toolkit, Release 0.7.1

0.346131

In [914]: s2.ix[b:h] Out[914]: c 0.787672 e 0.790509 g 1.109413

1.2.4 Changes to Series [] operator

As as notational convenience, you can pass a sequence of labels or a label slice to a Series when getting and setting values via [] (i.e. the __getitem__ and __setitem__ methods). The behavior will be the same as passing similar input to ix except in the case of integer indexing:
In [915]: s = Series(randn(6), index=list(acegkm)) In [916]: s Out[916]: a -0.393513 c -1.147209 e 1.757948 g -0.608897 k -0.231560 m -1.098559 In [917]: s[[m, a, c, e]] Out[917]: m -1.098559 a -0.393513 c -1.147209 e 1.757948 In [918]: s[b:l] Out[918]: c -1.147209 e 1.757948 g -0.608897 k -0.231560 In [919]: s[c:k] Out[919]: c -1.147209 e 1.757948 g -0.608897 k -0.231560

In the case of integer indexes, the behavior will be exactly as before (shadowing ndarray):
In [920]: s = Series(randn(6), index=range(0, 12, 2)) In [921]: s[[4, 0, 2]] Out[921]: 4 -0.553988 0 0.394646 2 -2.230633 In [922]: s[1:5]

1.2. v.0.7.0 (February 9, 2012)

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[922]: 2 -2.230633 4 -0.553988 6 -0.884280 8 -0.022224

If you wish to do indexing with sequences and slicing on an integer index with label semantics, use ix.

1.2.5 Other API Changes

The deprecated LongPanel class has been completely removed If Series.sort is called on a column of a DataFrame, an exception will now be raised. Before it was possible to accidentally mutate a DataFrames column by doing df[col].sort() instead of the side-effect free method df[col].order() (GH316) Miscellaneous renames and deprecations which will (harmlessly) raise FutureWarning drop added as an optional parameter to DataFrame.reset_index (GH699)

1.2.6 Performance improvements

Cythonized GroupBy aggregations no longer presort the data, thus achieving a signicant speedup (GH93). GroupBy aggregations with Python functions signicantly sped up by clever manipulation of the ndarray data type in Cython (GH496). Better error message in DataFrame constructor when passed column labels dont match data (GH497) Substantially improve performance of multi-GroupBy aggregation when a Python function is passed, reuse ndarray object in Cython (GH496) Can store objects indexed by tuples and oats in HDFStore (GH492) Dont print length by default in Series.to_string, add length option (GH489) Improve Cython code for multi-groupby to aggregate without having to sort the data (GH93) Improve MultiIndex reindexing speed by storing tuples in the MultiIndex, test for backwards unpickling compatibility Improve column reindexing performance by using specialized Cython take function Further performance tweaking of Series.__getitem__ for standard use cases Avoid Index dict creation in some cases (i.e. when getting slices, etc.), regression from prior versions Friendlier error message in setup.py if NumPy not installed Use common set of NA-handling operations (sum, mean, etc.) in Panel class also (GH536) Default name assignment when calling reset_index on DataFrame with a regular (non-hierarchical) index (GH476) Use Cythonized groupers when possible in Series/DataFrame stat ops with level parameter passed (GH545) Ported skiplist data structure to C to speed up rolling_median by about 5-10x in most typical use cases (GH374)

Chapter 1. Whats New

pandas: powerful Python data analysis toolkit, Release 0.7.1

1.3 v.0.6.1 (December 13, 2011)

1.3.1 New features
Can append single rows (as Series) to a DataFrame Add Spearman and Kendall rank correlation options to Series.corr and DataFrame.corr (GH428) Added get_value and set_value methods to Series, DataFrame, and Panel for very low-overhead access (>2x faster in many cases) to scalar elements (GH437, GH438). set_value is capable of producing an enlarged object. Add PyQt table widget to sandbox (PR435) DataFrame.align can accept Series arguments and an axis option (GH461) Implement new SparseArray and SparseList data structures. SparseSeries now derives from SparseArray (GH463) Better console printing options (PR453) Implement fast data ranking for Series and DataFrame, fast versions of scipy.stats.rankdata (GH428) Implement DataFrame.from_items alternate constructor (GH444) DataFrame.convert_objects method for inferring better dtypes for object columns (GH302) Add rolling_corr_pairwise function for computing Panel of correlation matrices (GH189) Add margins option to pivot_table for computing subgroup aggregates (GH114) Add Series.from_csv function (PR482) Can pass DataFrame/DataFrame and DataFrame/Series to rolling_corr/rolling_cov (GH #462) MultiIndex.get_level_values can accept the level name

1.3.2 Performance improvements

Improve memory usage of DataFrame.describe (do not copy data unnecessarily) (PR #425) Optimize scalar value lookups in the general case by 25% or more in Series and DataFrame Fix performance regression in cross-sectional count in DataFrame, affecting DataFrame.dropna speed Column deletion in DataFrame copies no data (computes views on blocks) (GH #158)

1.4 v.0.6.0 (November 25, 2011)

1.4.1 New Features
Added melt function to pandas.core.reshape Added level parameter to group by level in Series and DataFrame descriptive statistics (PR313) Added head and tail methods to Series, analogous to to DataFrame (PR296) Added Series.isin function which checks if each value is contained in a passed sequence (GH289) Added float_format option to Series.to_string

1.3. v.0.6.1 (December 13, 2011)

pandas: powerful Python data analysis toolkit, Release 0.7.1

Added skip_footer (GH291) and converters (GH343) options to read_csv and read_table Added drop_duplicates and duplicated functions for removing duplicate DataFrame rows and checking for duplicate rows, respectively (GH319) Implemented operators &, |, ^, - on DataFrame (GH347) Added Series.mad, mean absolute deviation Added QuarterEnd DateOffset (PR321) Added dot to DataFrame (GH65) Added orient option to Panel.from_dict (GH359, GH301) Added orient option to DataFrame.from_dict Added passing list of tuples or list of lists to DataFrame.from_records (GH357) Added multiple levels to groupby (GH103) Allow multiple columns in by argument of DataFrame.sort_index (GH92, PR362) Added fast get_value and put_value methods to DataFrame (GH360) Added cov instance methods to Series and DataFrame (GH194, PR362) Added kind=bar option to DataFrame.plot (PR348) Added idxmin and idxmax to Series and DataFrame (PR286) Added read_clipboard function to parse DataFrame from clipboard (GH300) Added nunique function to Series for counting unique elements (GH297) Made DataFrame constructor use Series name if no columns passed (GH373) Support regular expressions in read_table/read_csv (GH364) Added DataFrame.to_html for writing DataFrame to HTML (PR387) Added support for MaskedArray data in DataFrame, masked values converted to NaN (PR396) Added DataFrame.boxplot function (GH368) Can pass extra args, kwds to DataFrame.apply (GH376) Implement DataFrame.join with vector on argument (GH312) Added legend boolean ag to DataFrame.plot (GH324) Can pass multiple levels to stack and unstack (GH370) Can pass multiple values columns to pivot_table (GH381) Use Series name in GroupBy for result index (GH363) Added raw option to DataFrame.apply for performance if only need ndarray (GH309) Added proper, tested weighted least squares to standard and panel OLS (GH303)

1.4.2 Performance Enhancements

VBENCH Cythonized cache_readonly, resulting in substantial micro-performance enhancements throughout the codebase (GH361) VBENCH Special Cython matrix iterator for applying arbitrary reduction operations with 3-5x better performance than np.apply_along_axis (GH309)

10

Chapter 1. Whats New

pandas: powerful Python data analysis toolkit, Release 0.7.1

VBENCH Improved performance of MultiIndex.from_tuples VBENCH Special Cython matrix iterator for applying arbitrary reduction operations VBENCH + DOCUMENT Add raw option to DataFrame.apply for getting better performance when VBENCH Faster cythonized count by level in Series and DataFrame (GH341) VBENCH? Signicant GroupBy performance enhancement with multiple keys with many empty combinations VBENCH New Cython vectorized function map_infer speeds up Series.apply and Series.map signicantly when passed elementwise Python function, motivated by (PR355) VBENCH Signicantly improved performance of Series.order, which also makes np.unique called on a Series faster (GH327) VBENCH Vastly improved performance of GroupBy on axes with a MultiIndex (GH299)

1.5 v.0.5.0 (October 24, 2011)

1.5.1 New Features
Added DataFrame.align method with standard join options Added parse_dates option to read_csv and read_table methods to optionally try to parse dates in the index columns Added nrows, chunksize, and iterator arguments to read_csv and read_table. The last two return a new TextParser class capable of lazily iterating through chunks of a at le (GH242) Added ability to join on multiple columns in DataFrame.join (GH214) Added private _get_duplicates function to Index for identifying duplicate values more easily (ENH5c) Added column attribute access to DataFrame. Added Python tab completion hook for DataFrame columns. (PR233, GH230) Implemented Series.describe for Series containing objects (PR241) Added inner join option to DataFrame.join when joining on key(s) (GH248) Implemented selecting DataFrame columns by passing a list to __getitem__ (GH253) Implemented & and | to intersect / union Index objects, respectively (GH261) Added pivot_table convenience function to pandas namespace (GH234) Implemented Panel.rename_axis function (GH243) DataFrame will show index level names in console output (PR334) Implemented Panel.take Added set_eng_float_format for alternate DataFrame oating point string formatting (ENH61) Added convenience set_index function for creating a DataFrame index from its existing columns Implemented groupby hierarchical index level name (GH223) Added support for different delimiters in DataFrame.to_csv (PR244) TODO: DOCS ABOUT TAKE METHODS

1.5. v.0.5.0 (October 24, 2011)

11

pandas: powerful Python data analysis toolkit, Release 0.7.1

1.5.2 Performance Enhancements

VBENCH Major performance improvements in le parsing functions read_csv and read_table VBENCH Added Cython function for converting tuples to ndarray very fast. Speeds up many MultiIndex-related operations VBENCH Refactored merging / joining code into a tidy class and disabled unnecessary computations in the oat/object case, thus getting about 10% better performance (GH211) VBENCH Improved speed of DataFrame.xs on mixed-type DataFrame objects by about 5x, regression from 0.3.0 (GH215) VBENCH With new DataFrame.align method, speeding up binary operations between differently-indexed DataFrame objects by 10-25%. VBENCH Signicantly sped up conversion of nested dict into DataFrame (GH212) VBENCH Signicantly speed up DataFrame __repr__ and count on large mixed-type DataFrame objects

1.6 v.0.4.3 through v0.4.1 (September 25 - October 9, 2011)

1.6.1 New Features
Added Python 3 support using 2to3 (PR200) Added name attribute to Series, now prints as part of Series.__repr__ Added instance methods isnull and notnull to Series (PR209, GH203) Added Series.align method for aligning two series with choice of join method (ENH56) Added method get_level_values to MultiIndex (IS188) Set values in mixed-type DataFrame objects via .ix indexing attribute (GH135) Added new DataFrame methods get_dtype_counts and property dtypes (ENHdc) Added ignore_index option to DataFrame.append to stack DataFrames (ENH1b) read_csv tries to sniff delimiters using csv.Sniffer (PR146) read_csv can read multiple columns into a MultiIndex; DataFrames to_csv method writes out a corresponding MultiIndex (PR151) DataFrame.rename has a new copy parameter to rename a DataFrame in place (ENHed) Enable unstacking by name (PR142) Enable sortlevel to work by level (PR141)

1.6.2 Performance Enhancements

Altered binary operations on differently-indexed SparseSeries objects to use the integer-based (dense) alignment logic which is faster with a larger number of blocks (GH205) Wrote faster Cython data alignment / merging routines resulting in substantial speed increases Improved performance of isnull and notnull, a regression from v0.3.0 (GH187) Refactored code related to DataFrame.join so that intermediate aligned copies of the data in each DataFrame argument do not need to be created. Substantial performance increases result (GH176) 12 Chapter 1. Whats New

pandas: powerful Python data analysis toolkit, Release 0.7.1

Substantially improved performance of generic Index.intersection and Index.union Implemented BlockManager.take resulting in signicantly faster take performance on mixed-type DataFrame objects (GH104) Improved performance of Series.sort_index Signicant groupby performance enhancement: removed unnecessary integrity checks in DataFrame internals that were slowing down slicing operations to retrieve groups Optimized _ensure_index function resulting in performance savings in type-checking Index objects Wrote fast time series merging / joining methods in Cython. Will be integrated later into DataFrame.join and related functions

1.6. v.0.4.3 through v0.4.1 (September 25 - October 9, 2011)

13

pandas: powerful Python data analysis toolkit, Release 0.7.1

14

Chapter 1. Whats New

CHAPTER

TWO

INSTALLATION
You have the option to install an ofcial release or to build the development version. If you choose to install from source and are running Windows, you will have to ensure that you have a compatible C compiler (MinGW or Visual Studio) installed. How-to install MinGW on Windows

2.1 Python version support

Ofcially Python 2.5 to 2.7 and Python 3.1+, although Python 3 support is less well tested. Python 2.4 support is being phased out since the userbase has shrunk signicantly. Continuing Python 2.4 support will require either monetary development support or someone contributing to the project to maintain compatibility.

2.2 Binary installers

Available on PyPI

2.3 Dependencies
NumPy: 1.4.0 or higher. Recommend 1.5.1 or higher python-dateutil 1.5

2.4 Optional dependencies

SciPy: miscellaneous statistical functions PyTables: necessary for HDF5-based storage matplotlib: for plotting scikits.statsmodels Needed for parts of pandas.stats pytz Needed for time zone support with DateRange

15

pandas: powerful Python data analysis toolkit, Release 0.7.1

Note: Without the optional dependencies, many useful features will not work. Hence, it is highly recommended that you install these. A packaged distribution like the Enthought Python Distribution may be worth considering.

2.5 Installing from source

Note: Installing from the git repository requires a recent installation of Cython as the cythonized C sources are no longer checked into source control. Released source distributions will contain the built C les. I recommend installing the latest Cython via easy_install -U Cython The source code is hosted at http://github.com/pydata/pandas, it can be checked out using git and compiled / installed like so:
git clone git://github.com/pydata/pandas.git cd pandas python setup.py install

On Windows, I suggest installing the MinGW compiler suite following the directions linked to above. Once congured property, run the following on the command line:
python setup.py build --compiler=mingw32 python setup.py install

Note that you will not be able to import pandas if you open an interpreter in the source directory unless you build the C extensions in place:
python setup.py build_ext --inplace

2.6 Running the test suite

pandas is equipped with an exhaustive set of unit tests covering about 97% of the codebase as of this writing. To run it on your machine to verify that everything is working (and you have all of the dependencies, soft and hard, installed), make sure you have nose and run:
$ nosetests pandas .......................................................................... .......................S.................................................. .......................................................................... .......................................................................... .......................................................................... .......................................................................... .......................................................................... .......................................................................... .......................................................................... .......................................................................... .................S........................................................ .... ---------------------------------------------------------------------Ran 818 tests in 21.631s OK (SKIP=2)

16

Chapter 2. Installation

CHAPTER

THREE

FREQUENTLY ASKED QUESTIONS (FAQ)

17

pandas: powerful Python data analysis toolkit, Release 0.7.1

18

Chapter 3. Frequently Asked Questions (FAQ)

CHAPTER

FOUR

PACKAGE OVERVIEW
pandas consists of the following things A set of labeled array data structures, the primary of which are Series/TimeSeries and DataFrame Index objects enabling both simple axis indexing and multi-level / hierarchical axis indexing An integrated group by engine for aggregating and transforming data sets Date range generation (DateRange) and custom date offsets enabling the implementation of customized frequencies Input/Output tools: loading tabular data from at les (CSV, delimited, Excel 2003), and saving and loading pandas objects from the fast and efcient PyTables/HDF5 format. Memory-efcent sparse versions of the standard data structures for storing data that is mostly missing or mostly constant (some xed value) Moving window statistics (rolling mean, rolling standard deviation, etc.) Static and moving window linear and panel regression

4.1 Data structures at a glance

Dimensions 1 1 2 3 Name Description

Series 1D labeled homogeneously-typed array TimeSeries with index containing datetimes Series DataFrame General 2D labeled, size-mutable tabular structure with potentially heterogeneously-typed columns Panel General 3D labeled, also size-mutable array

4.1.1 Why more than 1 data structure?

The best way to think about the pandas data structures is as exible containers for lower dimensional data. For example, DataFrame is a container for Series, and Panel is a container for DataFrame objects. We would like to be able to insert and remove objects from these containers in a dictionary-like fashion. Also, we would like sensible default behaviors for the common API functions which take into account the typical orientation of time series and cross-sectional data sets. When using ndarrays to store 2- and 3-dimensional data, a burden is placed on the user to consider the orientation of the data set when writing functions; axes are considered more or less equivalent (except when C- or Fortran-contiguousness matters for performance). In pandas, the axes are 19

pandas: powerful Python data analysis toolkit, Release 0.7.1

intended to lend more semantic meaning to the data; i.e., for a particular data set there is likely to be a right way to orient the data. The goal, then, is to reduce the amount of mental effort required to code up data transformations in downstream functions. For example, with tabular data (DataFrame) it is more semantically helpful to think of the index (the rows) and the columns rather than axis 0 and axis 1. And iterating through the columns of the DataFrame thus results in more readable code:
for col in df.columns: series = df[col] # do something with series

4.2 Mutability and copying of data

All pandas data structures are value-mutable (the values they contain can be altered) but not always size-mutable. The length of a Series cannot be changed, but, for example, columns can be inserted into a DataFrame. However, the vast majority of methods produce new objects and leave the input data untouched. In general, though, we like to favor immutability where sensible.

4.3 Getting Support

The rst stop for pandas issues and ideas is the Github Issue Tracker. If you have a general question, pandas community experts can answer through Stack Overow. Longer discussions occur on the developer mailing list, and commercial support inquiries for Lambda Foundry should be sent to: [email protected]

4.4 Credits
pandas development began at AQR Capital Management in April 2008. It was open-sourced at the end of 2009. AQR continued to provide resources for development through the end of 2011, and continues to contribute bug reports today. Since January 2012, Lambda Foundry, has been providing development resources, as well as commercial support, training, and consulting for pandas. pandas is only made possible by a group of people around the world like you who have contributed new code, bug reports, xes, comments and ideas. A complete list can be found on Github.

4.5 Development Team

pandas is a part of the PyData project. The PyData Development Team is a collection of developers focused on the improvement of Pythons data libraries. The core team that coordinates development can be found on Github. If youre interested in contributing, please visit the project website.

4.6 License

20

Chapter 4. Package overview

pandas: powerful Python data analysis toolkit, Release 0.7.1

====================== PANDAS LICENSING TERMS ====================== pandas is licensed under the BSD 3-Clause (also known as "BSD New" or "BSD Simplified"), as follows: Copyright (c) 2011-2012, Lambda Foundry, Inc. and PyData Development Team All rights reserved. Copyright (c) 2008-2011 AQR Capital Management, LLC All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the copyright holder nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. About the Copyright Holders =========================== AQR Capital Management began pandas development in 2008. Development was led by Wes McKinney. AQR released the source under this license in 2009. Wes is now an employee of Lambda Foundry, and remains the pandas project lead. The PyData Development Team is the collection of developers of the PyData project. This includes all of the PyData sub-projects, including pandas. The core team that coordinates development on GitHub can be found here: http://github.com/pydata. Full credits for pandas contributors can be found in the documentation. Our Copyright Policy ====================

4.6. License

21

pandas: powerful Python data analysis toolkit, Release 0.7.1

PyData uses a shared copyright model. Each contributor maintains copyright over their contributions to PyData. However, it is important to note that these contributions are typically only changes to the repositories. Thus, the PyData source code, in its entirety, is not the copyright of any single person or institution. Instead, it is the collective copyright of the entire PyData Development Team. If individual contributors want to maintain a record of what changes/contributions they have specific copyright on, they should indicate their copyright in the commit message of the change when they commit the change to one of the PyData repositories. With this in mind, the following banner should be used in any source code file to indicate the copyright and license terms: #----------------------------------------------------------------------------# Copyright (c) 2012, PyData Development Team # All rights reserved. # # Distributed under the terms of the BSD Simplified License. # # The full license is in the LICENSE file, distributed with this software. #-----------------------------------------------------------------------------

22

Chapter 4. Package overview

CHAPTER

FIVE

INTRO TO DATA STRUCTURES

Well start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started. The fundamental behavior about data types, indexing, and axis labeling / alignment apply across all of the objects. To get started, import numpy and load pandas into your namespace:
In [225]: import numpy as np # will use a lot in examples In [226]: randn = np.random.randn In [227]: from pandas import *

Here is a basic tenet to keep in mind: data alignment is intrinsic. Link between labels and data will not be broken unless done so explicitly by you. Well give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.

5.1 Series
Series is a one-dimensional labeled array (technically a subclass of ndarray) capable of holding any data type (integers, strings, oating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:
>>> s = Series(data, index=index)

Here, data can be many different things: a Python dict an ndarray a scalar value (like 5) The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is: From ndarray If data is an ndarray, index must be the same length as data. If no index is passed, one will be created having values [0, ..., len(data) - 1].
In [228]: s = Series(randn(5), index=[a, b, c, d, e]) In [229]: s Out[229]:

23

pandas: powerful Python data analysis toolkit, Release 0.7.1

a b c d e

-0.284 -1.537 0.163 -0.648 -1.703

In [230]: s.index Out[230]: Index([a, b, c, d, e], dtype=object) In [231]: Series(randn(5)) Out[231]: 0 0.654 1 -1.146 2 1.144 3 0.167 4 0.148

Note: The values in the index must be unique. If they are not, an exception will not be raised immediately, but attempting any operation involving the index will later result in an exception. In other words, the Index object containing the labels lazily checks whether the values are unique. The reason for being lazy is nearly all performance-based (there are many instances in computations, like parts of GroupBy, where the index is not used). From dict If data is a dict, if index is passed the values in data corresponding to the labels in the index will be pulled out. Otherwise, an index will be constructed from the sorted keys of the dict, if possible.
In [232]: d = {a : 0., b : 1., c : 2.} In [233]: Series(d) Out[233]: a 0 b 1 c 2 In [234]: Series(d, index=[b, c, d, a]) Out[234]: b 1 c 2 d NaN a 0

Note: NaN (not a number) is the standard missing data marker used in pandas From scalar value If data is a scalar value, an index must be provided. The value will be repeated to match the length of index
In [235]: Series(5., index=[a, b, c, d, e]) Out[235]: a 5 b 5 c 5 d 5 e 5

24

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.1.1 Series is ndarray-like

As a subclass of ndarray, Series is a valid argument to most NumPy functions and behaves similarly to a NumPy array. However, things like slicing also slice the index.
In [236]: s[0] Out[236]: -0.28367872471747613 In [237]: s[:3] Out[237]: a -0.284 b -1.537 c 0.163 In [238]: s[s > s.median()] Out[238]: a -0.284 c 0.163 In [239]: s[[4, 3, 1]] Out[239]: e -1.703 d -0.648 b -1.537 In [240]: np.exp(s) Out[240]: a 0.753 b 0.215 c 1.177 d 0.523 e 0.182

We will address array-based indexing in a separate section.

5.1.2 Series is dict-like

A Series is alike a xed-size dict in that you can get and set values by index label:
In [241]: s[a] Out[241]: -0.28367872471747613 In [242]: s[e] = 12. In [243]: s Out[243]: a -0.284 b -1.537 c 0.163 d -0.648 e 12.000 In [244]: e in s Out[244]: True In [245]: f in s Out[245]: False

5.1. Series

25

pandas: powerful Python data analysis toolkit, Release 0.7.1

If a label is not contained, an exception

>>> s[f] KeyError: f >>> s.get(f) nan

5.1.3 Vectorized operations and label alignment with Series

When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary. Series can be also be passed into most NumPy methods expecting an ndarray.
In [246]: s + s Out[246]: a -0.567 b -3.074 c 0.326 d -1.296 e 24.000 In [247]: s * 2 Out[247]: a -0.567 b -3.074 c 0.326 d -1.296 e 24.000 In [248]: np.exp(s) Out[248]: a 0.753 b 0.215 c 1.177 d 0.523 e 162754.791

A key difference between Series and ndarray is that operations between Series automatically align the data based on label. Thus, you can write computations without giving consideration to whether the Series involved have the same labels.
In [249]: s[1:] + s[:-1] Out[249]: a NaN b -3.074 c 0.326 d -1.296 e NaN

The result of an operation between unaligned Series will have the union of the indexes involved. If a label is not found in one Series or the other, the result will be marked as missing (NaN). Being able to write code without doing any explicit data alignment grants immense freedom and exibility in interactive data analysis and research. The integrated data alignment features of the pandas data structures set pandas apart from the majority of related tools for working with labeled data. Note: In general, we chose to make the default result of operations between differently indexed objects yield the union of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is

26

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

typically important information as part of a computation. You of course have the option of dropping labels with missing data via the dropna function.

5.1.4 Name attribute

Series can also have a name attribute:
In [250]: s = Series(np.random.randn(5), name=something) In [251]: s Out[251]: 0 -1.334 1 -0.171 2 0.050 3 -0.650 4 -1.084 Name: something In [252]: s.name Out[252]: something

The Series name will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as you will see below.

5.2 DataFrame
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input: Dict of 1D ndarrays, lists, dicts, or Series 2-D numpy.ndarray Structured or record ndarray A Series Another DataFrame Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specic index will discard all data not matching up to the passed index. If axis labels are not passed, they will be constructed from the input data based on common sense rules.

5.2.1 From dict of Series or dicts

The result index will be the union of the indexes of the various Series. If there are any nested dicts, these will be rst converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.
In [253]: d = {one : Series([1., 2., 3.], index=[a, b, c]), .....: two : Series([1., 2., 3., 4.], index=[a, b, c, d])} In [254]: df = DataFrame(d)

5.2. DataFrame

27

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [255]: df Out[255]: one two a 1 1 b 2 2 c 3 3 d NaN 4 In [256]: DataFrame(d, index=[d, b, a]) Out[256]: one two d NaN 4 b 2 2 a 1 1 In [257]: DataFrame(d, index=[d, b, a], columns=[two, three]) Out[257]: two three d 4 NaN b 2 NaN a 1 NaN

The row and column labels can be accessed respectively by accessing the index and columns attributes: Note: When a particular set of columns is passed along with a dict of data, the passed columns override the keys in the dict.
In [258]: df.index Out[258]: Index([a, b, c, d], dtype=object) In [259]: df.columns Out[259]: Index([one, two], dtype=object)

5.2.2 From dict of ndarrays / lists

The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays. If no index is passed, the result will be range(n), where n is the array length.
In [260]: d = {one : [1., 2., 3., 4.], .....: two : [4., 3., 2., 1.]} In [261]: DataFrame(d) Out[261]: one two 0 1 4 1 2 3 2 3 2 3 4 1 In [262]: DataFrame(d, index=[a, b, c, d]) Out[262]: one two a 1 4 b 2 3

28

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

c d

3 4

2 1

5.2.3 From structured or record array

This case is handled identically to a dict of arrays.
In [263]: data = np.zeros((2,),dtype=[(A, i4),(B, f4),(C, a10)]) In [264]: data[:] = [(1,2.,Hello),(2,3.,"World")] In [265]: DataFrame(data) Out[265]: A B C 0 1 2 Hello 1 2 3 World In [266]: DataFrame(data, index=[first, second]) Out[266]: A B C first 1 2 Hello second 2 3 World In [267]: Out[267]: C 0 Hello 1 World DataFrame(data, columns=[C, A, B]) A 1 2 B 2 3

Note: DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.

5.2.4 From a list of dicts

In [268]: data2 = [{a: 1, b: 2}, {a: 5, b: 10, c: 20}] In [269]: DataFrame(data2) Out[269]: a b c 0 1 2 NaN 1 5 10 20 In [270]: DataFrame(data2, index=[first, second]) Out[270]: a b c first 1 2 NaN second 5 10 20 In [271]: DataFrame(data2, columns=[a, b]) Out[271]: a b 0 1 2 1 5 10

5.2. DataFrame

29

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.2.5 From a Series

The result will be a DataFrame with the same index as the input Series, and with one column whose name is the original name of the Series (only if no other column name provided). Missing Data Much more will be said on this topic in the Missing data section. To construct a DataFrame with missing data, use np.nan for those values which are missing. Alternatively, you may pass a numpy.MaskedArray as the data argument to the DataFrame constructor, and its masked entries will be considered missing.

5.2.6 Alternate Constructors

DataFrame.from_dict DataFrame.from_dict takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates like the DataFrame constructor except for the orient parameter which is columns by default, but which can be set to index in order to use the dict keys as row labels. DataFrame.from_records DataFrame.from_records takes a list of tuples or an ndarray with structured dtype. Works analogously to the normal DataFrame constructor, except that index maybe be a specic eld of the structured dtype to use as the index. For example:
In [272]: data Out[272]: array([(1, 2.0, Hello), (2, 3.0, World)], dtype=[(A, <i4), (B, <f4), (C, |S10)]) In [273]: Out[273]: A Hello 1 World 2 DataFrame.from_records(data, index=C) B 2 3

DataFrame.from_items DataFrame.from_items works analogously to the form of the dict constructor that takes a sequence of (key, value) pairs, where the keys are column (or row, in the case of orient=index) names, and the value are the column values (or row values). This can be useful for constructing a DataFrame with the columns in a particular order without having to pass an explicit list of columns:
In [274]: DataFrame.from_items([(A, [1, 2, 3]), (B, [4, 5, 6])]) Out[274]: A B 0 1 4 1 2 5 2 3 6

If you pass orient=index, the keys will be the row labels. But in this case you must also pass the desired column names:
In [275]: DataFrame.from_items([(A, [1, 2, 3]), (B, [4, 5, 6])], .....: orient=index, columns=[one, two, three]) Out[275]: one two three A 1 2 3 B 4 5 6

30

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.2.7 Column selection, addition, deletion

You can treat a DataFrame semantically like a dict of like-indexed Series objects. Getting, setting, and deleting columns works with the same syntax as the analogous dict operations:
In [276]: df[one] Out[276]: a 1 b 2 c 3 d NaN Name: one In [277]: df[three] = df[one] * df[two] In [278]: df[flag] = df[one] > 2 In [279]: df Out[279]: one two three a 1 1 1 b 2 2 4 c 3 3 9 d NaN 4 NaN

flag False False True False

Columns can be deleted or popped like with a dict:

In [280]: del df[two] In [281]: three = df.pop(three) In [282]: df Out[282]: one flag a 1 False b 2 False c 3 True d NaN False

When inserting a scalar value, it will naturally be propagated to ll the column:

In [283]: df[foo] = bar In [284]: df Out[284]: one flag a 1 False b 2 False c 3 True d NaN False

foo bar bar bar bar

When inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrames index:
In [285]: df[one_trunc] = df[one][:2] In [286]: df Out[286]: one flag a 1 False

foo bar

one_trunc 1

5.2. DataFrame

31

pandas: powerful Python data analysis toolkit, Release 0.7.1

b c d

2 3 NaN

False True False

bar bar bar

2 NaN NaN

You can insert raw ndarrays but their length must match the length of the DataFrames index. By default, columns get inserted at the end. The insert function is available to insert at a particular location in the columns:
In [287]: df.insert(1, bar, df[one]) In [288]: df Out[288]: one bar flag a 1 1 False b 2 2 False c 3 3 True d NaN NaN False

foo bar bar bar bar

one_trunc 1 2 NaN NaN

5.2.8 Indexing / Selection

The basics of indexing are as follows: Operation Select column Select row by label Select row by location (int) Slice rows Select rows by boolean vector Syntax df[col] df.xs(label) or df.ix[label] df.ix[loc] df[5:10] df[bool_vec] Result Series Series Series DataFrame DataFrame

Row selection, for example, returns a Series whose index is the columns of the DataFrame:
In [289]: df.xs(b) Out[289]: one 2 bar 2 flag False foo bar one_trunc 2 Name: b In [290]: df.ix[2] Out[290]: one 3 bar 3 flag True foo bar one_trunc NaN Name: c

Note if a DataFrame contains columns of multiple dtypes, the dtype of the row will be chosen to accommodate all of the data types (dtype=object is the most general). For a more exhaustive treatment of more sophisticated label-based indexing and slicing, see the section on indexing. We will address the fundamentals of reindexing / conforming to new sets of lables in the section on reindexing.

32

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.2.9 Data alignment and arithmetic

Data alignment between DataFrame objects automatically align on both the columns and the index (row labels). Again, the resulting object will have the union of the column and row labels.
In [291]: df = DataFrame(randn(10, 4), columns=[A, B, C, D]) In [292]: df2 = DataFrame(randn(7, 3), columns=[A, B, C]) In [293]: df + df2 Out[293]: A B C 0 0.002 0.831 -1.171 1 0.269 3.238 0.268 2 -0.513 0.090 3.120 3 0.684 1.042 2.282 4 1.188 -1.805 1.166 5 -1.972 0.407 -3.513 6 -0.850 0.306 0.237 7 NaN NaN NaN 8 NaN NaN NaN 9 NaN NaN NaN

D NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

When doing an operation between DataFrame and Series, the default behavior is to align the Series index on the DataFrame columns, thus broadcasting row-wise. For example:
In [294]: df - df.ix[0] Out[294]: A B C D 0 0.000 0.000 0.000 0.000 1 0.808 1.358 2.420 -1.339 2 -0.070 -0.814 4.027 -0.293 3 0.073 -0.519 2.195 -0.002 4 1.342 0.372 2.510 -1.543 5 -0.523 0.665 0.942 0.018 6 0.101 -0.066 1.943 -0.817 7 0.744 0.834 3.473 -0.665 8 0.045 0.772 1.406 -0.965 9 0.860 1.677 1.462 -1.165

In the special case of working with time series data, if the Series is a TimeSeries (which it will be automatically if the index contains datetime objects), and the DataFrame index also contains dates, the broadcasting will be column-wise:
In [295]: index = DateRange(1/1/2000, periods=8) In [296]: df = DataFrame(randn(8, 3), index=index, .....: columns=[A, B, C]) In [297]: df Out[297]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A 0.361 -0.646 0.613 0.624 1.559 -1.381 -0.673 -0.583 B -0.192 -1.051 0.501 0.790 0.335 0.365 -1.968 -0.999 C -0.058 -0.716 -1.380 0.818 0.919 -1.811 -0.401 -0.629

5.2. DataFrame

33

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [298]: type(df[A]) Out[298]: pandas.core.series.TimeSeries In [299]: df - df[A] Out[299]: A B C 2000-01-03 0 -0.553 -0.419 2000-01-04 0 -0.405 -0.070 2000-01-05 0 -0.112 -1.993 2000-01-06 0 0.166 0.195 2000-01-07 0 -1.224 -0.640 2000-01-10 0 1.746 -0.429 2000-01-11 0 -1.294 0.272 2000-01-12 0 -0.416 -0.046

Technical purity aside, this case is so common in practice that supporting the special case is preferable to the alternative of forcing the user to transpose and do column-based alignment like so:
In [300]: (df.T - df[A]).T Out[300]: A B C 2000-01-03 0 -0.553 -0.419 2000-01-04 0 -0.405 -0.070 2000-01-05 0 -0.112 -1.993 2000-01-06 0 0.166 0.195 2000-01-07 0 -1.224 -0.640 2000-01-10 0 1.746 -0.429 2000-01-11 0 -1.294 0.272 2000-01-12 0 -0.416 -0.046

For explicit control over the matching and broadcasting behavior, see the section on exible binary operations. Operations with scalars are just as you would expect:
In [301]: df * 5 + 2 Out[301]: A B 2000-01-03 3.805 1.040 2000-01-04 -1.230 -3.257 2000-01-05 5.064 4.504 2000-01-06 5.118 5.948 2000-01-07 9.795 3.676 2000-01-10 -4.906 3.826 2000-01-11 -1.367 -7.838 2000-01-12 -0.915 -2.993 In [302]: 1 / df Out[302]: A 2000-01-03 2.770 2000-01-04 -1.548 2000-01-05 1.632 2000-01-06 1.604 2000-01-07 0.641 2000-01-10 -0.724 2000-01-11 -1.485 2000-01-12 -1.715 In [303]: df ** 4

C 1.708 -1.580 -4.902 6.091 6.597 -7.053 -0.006 -1.146

B C -5.211 -17.148 -0.951 -1.397 1.997 -0.724 1.266 1.222 2.983 1.088 2.738 -0.552 -0.508 -2.493 -1.001 -1.589

34

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[303]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A 0.017 0.174 0.141 0.151 5.908 3.639 0.206 0.116 B 0.001 1.222 0.063 0.389 0.013 0.018 14.988 0.995 C 0.000 0.263 3.631 0.448 0.715 10.748 0.026 0.157

Boolean operators work as well:

In [304]: df1 = DataFrame({a : [1, 0, 1], b : [0, 1, 1] }, dtype=bool) In [305]: df2 = DataFrame({a : [0, 1, 1], b : [1, 1, 0] }, dtype=bool) In [306]: df1 & df2 Out[306]: a b 0 False False 1 False True 2 True False In [307]: df1 | df2 Out[307]: a b 0 True True 1 True True 2 True True In [308]: df1 ^ df2 Out[308]: a b 0 True True 1 True False 2 False True In [309]: -df1 Out[309]: a b 0 False True 1 True False 2 False False

5.2.10 Transposing
To transpose, access the T attribute (also the transpose function), similar to an ndarray:
# only show the first 5 rows In [310]: df[:5].T Out[310]: 2000-01-03 2000-01-04 2000-01-05 A 0.361 -0.646 0.613 B -0.192 -1.051 0.501 C -0.058 -0.716 -1.380

2000-01-06 0.624 0.790 0.818

2000-01-07 1.559 0.335 0.919

5.2. DataFrame

35

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.2.11 DataFrame interoperability with NumPy functions

Elementwise NumPy ufuncs (log, exp, sqrt, ...) and various other NumPy functions can be used with no issues on DataFrame, assuming the data within are numeric:
In [311]: np.exp(df) Out[311]: A B 2000-01-03 1.435 0.825 2000-01-04 0.524 0.349 2000-01-05 1.846 1.650 2000-01-06 1.866 2.203 2000-01-07 4.754 1.398 2000-01-10 0.251 1.441 2000-01-11 0.510 0.140 2000-01-12 0.558 0.368 In [312]: np.asarray(df) Out[312]: array([[ 0.361 , -0.1919, [-0.646 , -1.0514, [ 0.6128, 0.5007, [ 0.6236, 0.7896, [ 1.5591, 0.3352, [-1.3812, 0.3652, [-0.6734, -1.9676, [-0.583 , -0.9986,

C 0.943 0.489 0.251 2.267 2.508 0.164 0.670 0.533

-0.0583], -0.716 ], -1.3804], 0.8183], 0.9194], -1.8106], -0.4012], -0.6293]])

The dot method on DataFrame implements matrix multiplication:

In [313]: Out[313]: A A 6.444 B 3.334 C 4.677 df.T.dot(df) B 3.334 7.131 1.784 C 4.677 1.784 7.772

Similarly, the dot method on Series implements dot product:

In [314]: s1 = Series(np.arange(5,10)) In [315]: s1.dot(s1) Out[315]: 255

DataFrame is not intended to be a drop-in replacement for ndarray as its indexing semantics are quite different in places from a matrix.

5.2.12 Console display

For very large DataFrame objects, only a summary will be printed to the console (here I am reading a CSV version of the baseball dataset from the plyr R package):
In [316]: baseball = read_csv(data/baseball.csv) In [317]: print baseball <class pandas.core.frame.DataFrame> Int64Index: 100 entries, 88641 to 89534 Data columns:

36

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

id 100 non-null values year 100 non-null values stint 100 non-null values team 100 non-null values lg 100 non-null values g 100 non-null values ab 100 non-null values r 100 non-null values h 100 non-null values X2b 100 non-null values X3b 100 non-null values hr 100 non-null values rbi 100 non-null values sb 100 non-null values cs 100 non-null values bb 100 non-null values so 100 non-null values ibb 100 non-null values hbp 100 non-null values sh 100 non-null values sf 100 non-null values gidp 100 non-null values dtypes: float64(9), int64(10), object(3)

However, using to_string will return a string representation of the DataFrame in tabular form, though it wont always t the console width:
In [318]: print baseball.ix[-20:, :12].to_string() id year stint team lg g ab 88641 womacto01 2006 2 CHN NL 19 50 88643 schilcu01 2006 1 BOS AL 31 2 88645 myersmi01 2006 1 NYA AL 62 0 88649 helliri01 2006 1 MIL NL 20 3 88650 johnsra05 2006 1 NYA AL 33 6 88652 finlest01 2006 1 SFN NL 139 426 88653 gonzalu01 2006 1 ARI NL 153 586 88662 seleaa01 2006 1 LAN NL 28 26 89177 francju01 2007 2 ATL NL 15 40 89178 francju01 2007 1 NYN NL 40 50 89330 zaungr01 2007 1 TOR AL 110 331 89333 witasja01 2007 1 TBA AL 3 0 89334 williwo02 2007 1 HOU NL 33 59 89335 wickmbo01 2007 2 ARI NL 8 0 89336 wickmbo01 2007 1 ATL NL 47 0 89337 whitero02 2007 1 MIN AL 38 109 89338 whiteri01 2007 1 HOU NL 20 1 89339 wellsda01 2007 2 LAN NL 7 15 89340 wellsda01 2007 1 SDN NL 22 38 89341 weathda01 2007 1 CIN NL 67 0 89343 walketo04 2007 1 OAK AL 18 48 89345 wakefti01 2007 1 BOS AL 1 2 89347 vizquom01 2007 1 SFN NL 145 513 89348 villoro01 2007 1 NYA AL 6 0 89352 valenjo03 2007 1 NYN NL 51 166 89354 trachst01 2007 2 CHN NL 4 7 89355 trachst01 2007 1 BAL AL 3 5 89359 timlimi01 2007 1 BOS AL 4 0 89360 thomeji01 2007 1 CHA AL 130 432 r 6 0 0 0 0 66 93 2 1 7 43 0 3 0 0 8 0 2 1 0 5 0 54 0 18 0 0 0 79 h 14 1 0 0 1 105 159 5 10 10 80 0 6 0 0 19 0 4 4 0 13 0 126 0 40 1 0 0 119 X2b 1 0 0 0 0 21 52 1 3 0 24 0 0 0 0 4 0 1 0 0 1 0 18 0 11 0 0 0 19 X3b 0 0 0 0 0 12 2 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 3 0 1 0 0 0 0 hr 1 0 0 0 0 6 15 0 0 1 10 0 1 0 0 4 0 0 0 0 0 0 4 0 3 0 0 0 35

5.2. DataFrame

37

pandas: powerful Python data analysis toolkit, Release 0.7.1

89361 89363 89365 89366 89367 89368 89370 89371 89372 89374 89375 89378 89381 89382 89383 89384 89385 89388 89389 89396 89398 89400 89402 89406 89410 89411 89412 89420 89421 89425 89426 89429 89430 89431 89438 89439 89442 89445 89450 89451 89452 89460 89462 89463 89464 89465 89466 89467 89468 89469 89473 89474 89480 89481 89482 89489 89493 89494

thomafr04 tavarju01 sweenma01 sweenma01 suppaje01 stinnke01 stantmi02 stairma01 sprinru01 sosasa01 smoltjo01 sheffga01 seleaa01 seaneru01 schmija01 schilcu01 sandere02 rogerke01 rodriiv01 ramirma02 piazzmi01 perezne01 parkch01 oliveda02 myersmi01 mussimi01 moyerja01 mesajo01 martipe02 maddugr01 mabryjo01 loftoke01 loftoke01 loaizes01 kleskry01 kentje01 jonesto02 johnsra05 hoffmtr01 hernaro01 hernaro01 guarded01 griffke02 greensh01 graffto01 gordoto01 gonzalu01 gomezch02 gomezch02 glavito02 floydcl01 finlest01 embreal01 edmonji01 easleda01 delgaca01 cormirh01 coninje01

2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007

1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 2

TOR BOS LAN SFN MIL SLN CIN TOR SLN TEX ATL DET NYN LAN LAN BOS KCA DET DET BOS OAK DET NYN LAA NYA NYA PHI PHI NYN SDN COL CLE TEX LAN SFN LAN DET ARI SDN LAN CLE CIN CIN NYN MIL PHI LAN CLE BAL NYN CHN COL OAK SLN NYN NYN CIN NYN

AL AL NL NL NL NL NL AL NL AL NL AL NL NL NL AL AL AL AL AL AL AL NL AL AL AL NL NL NL NL NL AL AL NL NL NL AL NL NL NL AL NL NL NL NL NL NL AL AL NL NL NL AL NL NL NL NL NL

155 2 30 76 33 26 67 125 72 114 30 133 31 68 6 1 24 1 129 133 83 33 1 5 6 2 33 38 5 33 28 52 84 5 116 136 5 10 60 22 2 15 144 130 86 44 139 19 73 33 108 43 4 117 76 139 6 21

531 4 33 90 61 82 2 357 1 412 54 494 4 1 7 2 73 2 502 483 309 64 1 0 1 2 73 0 9 62 34 173 317 7 362 494 0 15 0 0 0 0 528 446 231 0 464 53 169 56 282 94 0 365 193 538 0 41

63 0 2 18 4 7 0 58 0 53 1 107 0 0 1 0 12 0 50 84 33 5 0 0 0 0 4 0 1 2 4 24 62 0 51 78 0 0 0 0 0 0 78 62 34 0 70 4 17 3 40 9 0 39 24 71 0 2

147 1 9 23 8 13 0 103 0 104 5 131 0 0 1 1 23 0 141 143 85 11 0 0 0 0 9 0 1 9 4 49 96 1 94 149 0 1 0 0 0 0 146 130 55 0 129 15 51 12 80 17 0 92 54 139 0 8

30 0 1 8 0 3 0 28 0 24 1 20 0 0 0 0 7 0 31 33 17 3 0 0 0 0 2 0 1 2 1 9 16 0 27 36 0 0 0 0 0 0 24 30 8 0 23 2 10 1 10 3 0 15 6 30 0 2

0 0 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 0 3 1 1 0 0 0 0 0 0 0 0 0 0 3 3 0 3 1 0 0 0 0 0 0 1 1 0 0 2 0 1 0 1 0 0 2 0 0 0 0

26 0 0 2 0 1 0 21 0 21 0 25 0 0 1 0 2 0 11 20 8 1 0 0 0 0 0 0 0 0 1 0 7 0 6 20 0 0 0 0 0 0 30 10 9 0 15 0 1 0 9 1 0 12 10 24 0 0

38

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

89495 89497 89498 89499 89501 89502 89521 89523 89525 89526 89530 89533 89534

coninje01 clemero02 claytro01 claytro01 cirilje01 cirilje01 bondsba01 biggicr01 benitar01 benitar01 ausmubr01 aloumo01 alomasa02

2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007 2007

1 1 2 1 2 1 1 1 2 1 1 1 1

CIN NYA BOS TOR ARI MIN SFN HOU FLO SFN HOU NYN NYN

NL AL AL AL NL AL NL NL NL NL NL NL NL

80 2 8 69 28 50 126 141 34 19 117 87 8

215 2 6 189 40 153 340 517 0 0 349 328 22

23 0 1 23 6 18 75 68 0 0 38 51 1

57 1 0 48 8 40 94 130 0 0 82 112 3

11 0 0 14 4 9 14 31 0 0 16 19 1

1 0 0 0 0 2 0 3 0 0 3 1 0

6 0 0 1 0 2 28 10 0 0 3 13 0

5.2.13 DataFrame column types

The four main types stored in pandas objects are oat, int, boolean, and object. A convenient dtypes attribute return a Series with the data type of each column:
In [319]: baseball.dtypes Out[319]: id object year int64 stint int64 team object lg object g int64 ab int64 r int64 h int64 X2b int64 X3b int64 hr int64 rbi float64 sb float64 cs float64 bb int64 so float64 ibb float64 hbp float64 sh float64 sf float64 gidp float64

The related method get_dtype_counts will return the number of columns of each type:
In [320]: baseball.get_dtype_counts() Out[320]: float64 9 int64 10 object 3

5.2.14 DataFrame column attribute access and IPython completion

If a DataFrame column label is a valid Python variable name, the column can be accessed like attributes:

5.2. DataFrame

39

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [321]: df = DataFrame({foo1 : np.random.randn(5), .....: foo2 : np.random.randn(5)}) In [322]: df Out[322]: foo1 foo2 0 -0.548001 -0.966162 1 -0.852612 -0.332601 2 -0.126250 -1.327330 3 1.765997 1.225847 4 -1.593297 -0.348395 In [323]: df.foo1 Out[323]: 0 -0.548001 1 -0.852612 2 -0.126250 3 1.765997 4 -1.593297 Name: foo1

The columns are also connected to the IPython completion mechanism so they can be tab-completed:
In [5]: df.fo<TAB> df.foo1 df.foo2

5.3 Panel
Panel is a somewhat less-used, but still important container for 3-dimensional data. The term panel data is derived from econometrics and is partially responsible for the name pandas: pan(el)-da(ta)-s. The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data and, in particular, econometric analysis of panel data. However, for the strict purposes of slicing and dicing a collection of DataFrame objects, you may nd the axis names slightly arbitrary: items: axis 0, each item corresponds to a DataFrame contained inside major_axis: axis 1, it is the index (rows) of each of the DataFrames minor_axis: axis 2, it is the columns of each of the DataFrames Construction of Panels works about like you would expect:

5.3.1 From 3D ndarray with optional axis labels

In [324]: wp = Panel(randn(2, 5, 4), items=[Item1, Item2], .....: major_axis=DateRange(1/1/2000, periods=5), .....: minor_axis=[A, B, C, D]) In [325]: wp Out[325]: <class pandas.core.panel.Panel> Dimensions: 2 (items) x 5 (major) x 4 (minor) Items: Item1 to Item2 Major axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor axis: A to D

40

Chapter 5. Intro to Data Structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.3.2 From dict of DataFrame objects

In [326]: data = {Item1 : DataFrame(randn(4, 3)), .....: Item2 : DataFrame(randn(4, 2))} In [327]: Panel(data) Out[327]: <class pandas.core.panel.Panel> Dimensions: 2 (items) x 4 (major) x 3 (minor) Items: Item1 to Item2 Major axis: 0 to 3 Minor axis: 0 to 2

Note that the values in the dict need only be convertible to DataFrame. Thus, they can be any of the other valid inputs to DataFrame as per above. One helpful factory method is Panel.from_dict, which takes a dictionary of DataFrames as above, and the following named parameters: Parameter intersect orient Default False items Description drops elements whose indices do not align use minor to use DataFrames columns as panel items

For example, compare to the construction above:

In [328]: Panel.from_dict(data, orient=minor) Out[328]: <class pandas.core.panel.Panel> Dimensions: 3 (items) x 4 (major) x 2 (minor) Items: 0 to 2 Major axis: 0 to 3 Minor axis: Item1 to Item2

Orient is especially useful for mixed-type DataFrames. Note: Unfortunately Panel, being less commonly used than Series and DataFrame, has been slightly neglected featurewise. A number of methods and options available in DataFrame are not available in Panel. This will get worked on, of course, in future releases. And faster if you join me in working on the codebase.

5.3.3 From DataFrame using to_panel method

This method was introduced in v0.7 to replace LongPanel.to_long, and converts a DataFrame with a two-level index to a Panel.
In [329]: midx = MultiIndex(levels=[[one, two], [x,y]], labels=[[1,1,0,0],[1,0,1,0]]) In [330]: df = DataFrame({A : [1, 2, 3, 4], B: [5, 6, 7, 8]}, index=midx) In [331]: df.to_panel() Out[331]: <class pandas.core.panel.Panel> Dimensions: 2 (items) x 2 (major) x 2 (minor) Items: A to B Major axis: one to two Minor axis: x to y

5.3. Panel

41

pandas: powerful Python data analysis toolkit, Release 0.7.1

5.3.4 Item selection / addition / deletion

Similar to DataFrame functioning as a dict of Series, Panel is like a dict of DataFrames:
In [332]: wp[Item1] Out[332]: A B C D 2000-01-03 1.424840 -0.879202 0.544803 0.627214 2000-01-04 0.696649 0.107734 -0.776466 -1.257592 2000-01-05 -0.600682 -1.504853 1.052050 0.586573 2000-01-06 -2.637236 -0.014948 -1.208531 -1.257020 2000-01-07 -0.499747 0.429787 -0.242210 -0.723848 In [333]: wp[Item3] = wp[Item1] / wp[Item2]

The API for insertion and deletion is the same as for DataFrame. And as with DataFrame, if the item is a valid python identier, you can access it as an attribute and tab-complete it in IPython.

5.3.5 Indexing / Selection

Operation Select item Get slice at major_axis label Get slice at minor_axis label Syntax wp[item] wp.major_xs(val) wp.minor_xs(val) Result DataFrame DataFrame DataFrame

For example, using the earlier example data, we could do:

In [334]: wp[Item1] Out[334]: A B C D 2000-01-03 1.424840 -0.879202 0.544803 0.627214 2000-01-04 0.696649 0.107734 -0.776466 -1.257592 2000-01-05 -0.600682 -1.504853 1.052050 0.586573 2000-01-06 -2.637236 -0.014948 -1.208531 -1.257020 2000-01-07 -0.499747 0.429787 -0.242210 -0.723848 In [335]: wp.major_xs(wp.major_axis[2]) Out[335]: Item1 Item2 Item3 A -0.600682 -2.138612 0.280875 B -1.504853 -0.592654 2.539177 C 1.052050 -1.059136 -0.993309 D 0.586573 0.118816 4.936815 In [336]: wp.minor_axis Out[336]: Index([A, B, C, D], dtype=object) In [337]: wp.minor_xs(C) Out[337]: Item1 Item2 Item3 2000-01-03 0.544803 -0.543730 -1.001973 2000-01-04 -0.776466 -1.566259 0.495746 2000-01-05 1.052050 -1.059136 -0.993309 2000-01-06 -1.208531 -0.129101 9.361112 2000-01-07 -0.242210 0.759091 -0.319079

42

Chapter 5. Intro to Data Structures

CHAPTER

SIX

ESSENTIAL BASIC FUNCTIONALITY

Here we discuss a lot of the essential functionality common to the pandas data structures. Heres how to create some of the objects used in the examples from the previous section:
In [1]: index = DateRange(1/1/2000, periods=8) In [2]: s = Series(randn(5), index=[a, b, c, d, e]) In [3]: df = DataFrame(randn(8, 3), index=index, ...: columns=[A, B, C]) In [4]: wp = Panel(randn(2, 5, 4), items=[Item1, Item2], ...: major_axis=DateRange(1/1/2000, periods=5), ...: minor_axis=[A, B, C, D])

6.1 Head and Tail

To view a small sample of a Series or DataFrame object, use the head and tail methods. The default number of elements to display is ve, but you may pass a custom number.
In [5]: long_series = Series(randn(1000)) In [6]: long_series.head() Out[6]: 0 0.996243 1 -0.240173 2 -0.788816 3 1.164099 4 -0.099182 In [7]: long_series.tail(3) Out[7]: 997 0.273410 998 -0.811665 999 1.872824

6.2 Attributes and the raw ndarray(s)

pandas objects have a number of attributes enabling you to access the metadata

43

pandas: powerful Python data analysis toolkit, Release 0.7.1

shape: gives the axis dimensions of the object, consistent with ndarray Axis labels Series: index (only axis) DataFrame: index (rows) and columns Panel: items, major_axis, and minor_axis Note, these attributes can be safely assigned to!
In [8]: df[:2] Out[8]: A B C 2000-01-03 -0.397375 -1.964789 -0.170366 2000-01-04 0.316897 -0.090626 -1.399808 In [9]: df.columns = [x.lower() for x in df.columns] In [10]: df Out[10]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 a -0.397375 0.316897 -0.543386 -0.019389 -0.246211 -0.616430 0.745438 -0.182547 b c -1.964789 -0.170366 -0.090626 -1.399808 -1.065025 0.424420 0.330196 0.637809 0.773164 0.019443 -0.305674 0.844000 -0.280596 -0.796469 -0.499865 1.541597

To get the actual data inside a data structure, one need only access the values property:
In [11]: s.values Out[11]: array([ 2.3971, 0.0065, 1.2489, -0.8971, 0.7268])

In [12]: df.values Out[12]: array([[-0.3974, -1.9648, -0.1704], [ 0.3169, -0.0906, -1.3998], [-0.5434, -1.065 , 0.4244], [-0.0194, 0.3302, 0.6378], [-0.2462, 0.7732, 0.0194], [-0.6164, -0.3057, 0.844 ], [ 0.7454, -0.2806, -0.7965], [-0.1825, -0.4999, 1.5416]]) In [13]: wp.values Out[13]: array([[[-0.7461, 0.6748, [-1.3674, -0.1887, [ 0.2275, -0.1511, [-0.1162, 1.9625, [-0.2509, 0.5349, [[ 1.6887, 0.272 , [ 1.1353, -0.6189, [ 0.9363, 0.4164, [ 0.7311, 2.0043, [ 0.1582, 0.7534,

-0.5713, -0.9046, 1.4874, -0.8264, -2.3785, 1.0161, -0.7009, -0.2274, -1.2222, -1.5557,

0.1542], -0.8528], 0.2663], 0.7547], 1.3824]], -1.7139], 0.4606], 1.4496], -0.8807], -0.522 ]]])

44

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

If a DataFrame or Panel contains homogeneously-typed data, the ndarray can actually be modied in-place, and the changes will be reected in the data structure. For heterogeneous data (e.g. some of the DataFrames columns are not all the same dtype), this will not be the case. The values attribute itself, unlike the axis labels, cannot be assigned to. Note: When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all of the data involved. For example, if strings are involved, the result will be of object dtype. If there are only oats and integers, the resulting array will be of oat dtype.

6.3 Flexible binary operations

With binary operations between pandas data structures, there are two key points of interest: Broadcasting behavior between higher- (e.g. DataFrame) and lower-dimensional (e.g. Series) objects. Missing data in computations We will demonstrate how to manage these issues independently, though they can be handled simultaneously.

6.3.1 Matching / broadcasting behavior

DataFrame has the methods add, sub, mul, div and related functions radd, rsub, ... for carrying out binary operations. For broadcasting behavior, Series input is of primary interest. Using these functions, you can use to either match on the index or columns via the axis keyword:
In [14]: df Out[14]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661 In [15]: row = df.ix[1] In [16]: column = df[two] In [17]: df.sub(row, axis=columns) Out[17]: one three two a 0.333410 NaN 1.455920 b 0.000000 0.000000 0.000000 c -0.886445 -0.400125 0.614319 d NaN -0.326204 0.685712 In [18]: df.sub(row, axis=1) Out[18]: one three two a 0.333410 NaN 1.455920 b 0.000000 0.000000 0.000000 c -0.886445 -0.400125 0.614319 d NaN -0.326204 0.685712 In [19]: df.sub(column, axis=index) Out[19]:

two 1.848993 0.393073 1.007392 1.078785

6.3. Flexible binary operations

45

pandas: powerful Python data analysis toolkit, Release 0.7.1

one three a -1.069035 NaN b 0.053475 -0.500531 c -1.447289 -1.514975 d NaN -1.512446

two 0 0 0 0

In [20]: df.sub(column, axis=0) Out[20]: one three two a -1.069035 NaN 0 b 0.053475 -0.500531 0 c -1.447289 -1.514975 0 d NaN -1.512446 0

With Panel, describing the matching behavior is a bit more difcult, so the arithmetic methods instead (and perhaps confusingly?) give you the option to specify the broadcast axis. For example, suppose we wished to demean the data over a particular axis. This can be accomplished by taking the mean over an axis and broadcasting over the same axis:
In [21]: major_mean = wp.mean(axis=major) In [22]: major_mean Out[22]: Item1 Item2 A -0.450606 0.929929 B 0.566477 0.565418 C -0.638662 -0.538023 D 0.340967 -0.241284 In [23]: wp.sub(major_mean, axis=major) Out[23]: <class pandas.core.panel.Panel> Dimensions: 2 (items) x 5 (major) x 4 (minor) Items: Item1 to Item2 Major axis: 2000-01-03 00:00:00 to 2000-01-07 00:00:00 Minor axis: A to D

And similarly for axis=items and axis=minor. Note: I could be convinced to make the axis argument in the DataFrame methods match the broadcasting behavior of Panel. Though it would require a transition period so users can change their code...

6.3.2 Missing data / operations with ll values

In Series and DataFrame (though not yet in Panel), the arithmetic functions have the option of inputting a ll_value, namely a value to substitute when at most one of the values at a location are missing. For example, when adding two DataFrame objects, you may wish to treat NaN as 0 unless both DataFrames are missing that value, in which case the result will be NaN (you can later replace NaN with some other value using fillna if you wish).
In [24]: df Out[24]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661

two 1.848993 0.393073 1.007392 1.078785

46

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [25]: df2 Out[25]: one three a 0.779958 1.000000 b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661 In [26]: df Out[26]: one a 1.559916 b 0.893097 c -0.879794 d NaN + df2 three NaN -0.214915 -1.015165 -0.867323

two 1.848993 0.393073 1.007392 1.078785

two 3.697985 0.786146 2.014785 2.157570

In [27]: df.add(df2, fill_value=0) Out[27]: one three two a 1.559916 1.000000 3.697985 b 0.893097 -0.214915 0.786146 c -0.879794 -1.015165 2.014785 d NaN -0.867323 2.157570

6.3.3 Combining overlapping data sets

A problem occasionally arising is the combination of two similar data sets where values in one are preferred over the other. An example would be two data series representing a particular economic indicator where one is considered to be of higher quality. However, the lower quality series might extend further back in history or have more complete data coverage. As such, we would like to combine two DataFrame objects where missing values in one DataFrame are conditionally lled with like-labeled values from the other DataFrame. The function implementing this operation is combine_first, which we illustrate:
In [28]: df1 = DataFrame({A : [1., np.nan, 3., 5., np.nan], ....: B : [np.nan, 2., 3., np.nan, 6.]}) In [29]: df2 = DataFrame({A : [5., 2., 4., np.nan, 3., 7.], ....: B : [np.nan, np.nan, 3., 4., 6., 8.]}) In [30]: df1 Out[30]: A B 0 1 NaN 1 NaN 2 2 3 3 3 5 NaN 4 NaN 6 In [31]: df2 Out[31]: A B 0 5 NaN 1 2 NaN 2 4 3 3 NaN 4 4 3 6 5 7 8

6.3. Flexible binary operations

47

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [32]: df1.combine_first(df2) Out[32]: A B 0 1 NaN 1 2 2 2 3 3 3 5 4 4 3 6 5 7 8

6.3.4 General DataFrame Combine

The combine_first method above calls the more general DataFrame method combine. This method takes another DataFrame and a combiner function, aligns the input DataFrame and then passes the combiner function pairs of Series (ie, columns whose names are the same). So, for instance, to reproduce combine_first as above:
In [33]: combiner = lambda x, y: np.where(isnull(x), y, x) In [34]: df1.combine(df2, combiner) Out[34]: A B 0 1 NaN 1 2 2 2 3 3 3 5 4 4 3 6 5 7 8

6.4 Descriptive statistics

A large number of methods for computing descriptive statistics and other related operations on Series, DataFrame, and Panel. Most of these are aggregations (hence producing a lower-dimensional result) like sum, mean, and quantile, but some of them, like cumsum and cumprod, produce an object of the same size. Generally speaking, these methods take an axis argument, just like ndarray.{sum, std, ...}, but the axis can be specied by name or integer: Series: no axis argument needed DataFrame: index (axis=0, default), columns (axis=1) Panel: items (axis=0), major (axis=1, default), minor (axis=2) For example:
In [35]: df Out[35]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661 In [36]: df.mean(0) Out[36]: one 0.262203

two 1.848993 0.393073 1.007392 1.078785

48

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

three two

-0.349567 1.082061

In [37]: df.mean(1) Out[37]: a 1.314475 b 0.244055 c 0.019971 d 0.322562

All such methods have a skipna option signaling whether to exclude missing data (True by default):
In [38]: df.sum(0, skipna=False) Out[38]: one NaN three NaN two 4.328243 In [39]: df.sum(axis=1, skipna=True) Out[39]: a 2.628951 b 0.732164 c 0.059913 d 0.645124

Combined with the broadcasting / arithmetic behavior, one can describe various statistical procedures, like standardization (rendering data zero mean and standard deviation 1), very concisely:
In [40]: ts_stand = (df - df.mean()) / df.std() In [41]: Out[41]: one three two ts_stand.std() 1 1 1

In [42]: xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0) In [43]: xs_stand.std(1) Out[43]: a 1 b 1 c 1 d 1

Note that methods like cumsum and cumprod preserve the location of NA values:
In [44]: df.cumsum() Out[44]: one three a 0.779958 NaN b 1.226506 -0.107458 c 0.786610 -0.615040 d NaN -1.048702

two 1.848993 2.242066 3.249458 4.328243

Here is a quick reference summary table of common functions. Each also takes an optional level parameter which applies only if the object has a hierarchical index.

6.4. Descriptive statistics

49

pandas: powerful Python data analysis toolkit, Release 0.7.1

Function count sum mean mad median min max abs prod std var skew kurt quantile cumsum cumprod cummax cummin

Description Number of non-null observations Sum of values Mean of values Mean absolute deviation Arithmetic median of values Minimum Maximum Absolute Value Product of values Unbiased standard deviation Unbiased variance Unbiased skewness (3rd moment) Unbiased kurtosis (4th moment) Sample quantile (value at %) Cumulative sum Cumulative product Cumulative maximum Cumulative minimum

Note that by chance some NumPy methods, like mean, std, and sum, will exclude NAs on Series input by default:
In [45]: np.mean(df[one]) Out[45]: 0.2622032058481874 In [46]: np.mean(df[one].values) Out[46]: nan

Series also has a method nunique which will return the number of unique non-null values:
In [47]: series = Series(randn(500)) In [48]: series[20:500] = np.nan In [49]: series[10:20] = 5

In [50]: series.nunique() Out[50]: 11

6.4.1 Summarizing data: describe

There is a convenient describe function which computes a variety of summary statistics about a Series or the columns of a DataFrame (excluding NAs of course):
In [51]: series = Series(randn(1000)) In [52]: series[::2] = np.nan In [53]: series.describe() Out[53]: count 500.000000 mean 0.015500 std 1.011408 min -3.163112 25% -0.691748 50% 0.036114 75% 0.678996

50

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

max

4.199996

In [54]: frame = DataFrame(randn(1000, 5), columns=[a, b, c, d, e]) In [55]: frame.ix[::2] = np.nan In [56]: frame.describe() Out[56]: a b count 500.000000 500.000000 mean 0.008892 -0.019903 std 0.976105 0.995283 min -2.991973 -3.186717 25% -0.640447 -0.713719 50% 0.014034 0.008756 75% 0.704305 0.641368 max 2.658383 2.933960

c 500.000000 -0.047967 1.028454 -3.130757 -0.697681 -0.020365 0.617965 2.830546

d 500.000000 -0.038778 0.970528 -3.006165 -0.683452 -0.071134 0.600726 3.239495

e 500.000000 0.036809 1.037206 -2.775444 -0.700587 0.091943 0.725512 3.391848

For a non-numerical Series object, describe will give a simple summary of the number of unique values and most frequently occurring values:
In [57]: s = Series([a, a, b, b, a, a, np.nan, c, d, a]) In [58]: s.describe() Out[58]: count 9 unique 4 top a freq 5

There also is a utility function, value_range which takes a DataFrame and returns a series with the minimum/maximum values in the DataFrame.

6.4.2 Index of Min/Max Values

The idxmin and idxmax functions on Series and DataFrame compute the index labels with the minimum and maximum corresponding values:
In [59]: s1 = Series(randn(5)) In [60]: s1 Out[60]: 0 1.110029 1 0.830766 2 -1.281703 3 -0.425454 4 0.710819 In [61]: s1.idxmin(), s1.idxmax() Out[61]: (2, 0) In [62]: df1 = DataFrame(randn(5,3), columns=[A,B,C]) In [63]: df1 Out[63]: A B 0 -0.084098 -1.076266

C 0.094150

6.4. Descriptive statistics

51

pandas: powerful Python data analysis toolkit, Release 0.7.1

1 -0.123818 2 -1.938757 3 0.318179 4 -1.530668

2.412418 0.588569 0.586176 -0.008005 0.024081 0.597530 0.956943 -0.608121

In [64]: df1.idxmin(axis=0) Out[64]: A 2 B 0 C 4 In [65]: df1.idxmax(axis=1) Out[65]: 0 C 1 B 2 B 3 C 4 B

6.5 Function application

Arbitrary functions can be applied along the axes of a DataFrame or Panel using the apply method, which, like the descriptive statistics methods, take an optional axis argument:
In [66]: df.apply(np.mean) Out[66]: one 0.262203 three -0.349567 two 1.082061 In [67]: df.apply(np.mean, axis=1) Out[67]: a 1.314475 b 0.244055 c 0.019971 d 0.322562 In [68]: Out[68]: one three two df.apply(lambda x: x.max() - x.min()) 1.219855 0.400125 1.455920

In [69]: df.apply(np.cumsum) Out[69]: one three two a 0.779958 NaN 1.848993 b 1.226506 -0.107458 2.242066 c 0.786610 -0.615040 3.249458 d NaN -1.048702 4.328243 In [70]: df.apply(np.exp) Out[70]: one three two a 2.181381 NaN 6.353417 b 1.562908 0.898115 1.481526

52

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

c d

0.644103 NaN

0.601949 0.648132

2.738451 2.941104

Depending on the return type of the function passed to apply, the result will either be of lower dimension or the same dimension. apply combined with some cleverness can be used to answer many questions about a data set. For example, suppose we wanted to extract the date where the maximum value for each column occurred:
In [71]: tsdf = DataFrame(randn(1000, 3), columns=[A, B, C], ....: index=DateRange(1/1/2000, periods=1000)) In [72]: tsdf.apply(lambda x: x.index[x.dropna().argmax()]) Out[72]: A 2003-06-09 00:00:00 B 2001-02-05 00:00:00 C 2001-06-21 00:00:00

You may also pass additional arguments and keyword arguments to the apply method. For instance, consider the following function you would like to apply:
def subtract_and_divide(x, sub, divide=1): return (x - sub) / divide

You may then apply this function as follows:

df.apply(subtract_and_divide, args=(5,), divide=3)

Another useful feature is the ability to pass Series methods to carry out some Series operation on each column or row:
In [73]: tsdf Out[73]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 2000-01-13 2000-01-14 A -2.569072 -2.177425 1.863567 NaN NaN NaN NaN 2.532888 -1.857103 -0.616887 B 0.915718 -0.028034 1.072148 NaN NaN NaN NaN -1.265961 -0.500810 -1.097486 C -0.470533 -0.758511 0.737064 NaN NaN NaN NaN -0.586280 1.055360 -1.673839

In [74]: tsdf.apply(Series.interpolate) Out[74]: A B C 2000-01-03 -2.569072 0.915718 -0.470533 2000-01-04 -2.177425 -0.028034 -0.758511 2000-01-05 1.863567 1.072148 0.737064 2000-01-06 1.997431 0.604526 0.472395 2000-01-07 2.131295 0.136905 0.207726 2000-01-10 2.265160 -0.330717 -0.056943 2000-01-11 2.399024 -0.798339 -0.321612 2000-01-12 2.532888 -1.265961 -0.586280 2000-01-13 -1.857103 -0.500810 1.055360 2000-01-14 -0.616887 -1.097486 -1.673839

Finally, apply takes an argument raw which is False by default, which converts each row or column into a Series before applying the function. When set to True, the passed function will instead receive an ndarray object, which has 6.5. Function application 53

pandas: powerful Python data analysis toolkit, Release 0.7.1

positive performance implications if you do not need the indexing functionality. See Also: The section on GroupBy demonstrates related, exible functionality for grouping by some criterion, applying, and combining the results into a Series, DataFrame, etc.

6.5.1 Applying elementwise Python functions

Since not all functions can be vectorized (accept NumPy arrays and return another array or value), the methods applymap on DataFrame and analogously map on Series accept any Python function taking a single value and returning a single value. For example:
In [75]: f = lambda x: len(str(x)) In [76]: df[one].map(f) Out[76]: a 14 b 14 c 15 d 3 Name: one In [77]: df.applymap(f) Out[77]: one three two a 14 3 13 b 14 15 14 c 15 15 13 d 3 15 13

Series.map has an additional feature which is that it can be used to easily link or map values dened by a secondary series. This is closely related to merging/joining functionality:
In [78]: s = Series([six, seven, six, seven, six], ....: index=[a, b, c, d, e]) In [79]: t = Series({six : 6., seven : 7.}) In [80]: s Out[80]: a six b seven c six d seven e six In [81]: s.map(t) Out[81]: a 6 b 7 c 6 d 7 e 6

54

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

6.6 Reindexing and altering labels

reindex is the fundamental data alignment method in pandas. It is used to implement nearly all other features relying on label-alignment functionality. To reindex means to conform the data to match a given set of labels along a particular axis. This accomplishes several things: Reorders the existing data to match a new set of labels Inserts missing value (NA) markers in label locations where no data for that label existed If specied, ll data for missing labels using logic (highly relevant to working with time series data) Here is a simple example:
In [82]: s = Series(randn(5), index=[a, b, c, d, e]) In [83]: s Out[83]: a -0.558130 b 0.232643 c 1.083226 d -0.071506 e -1.303966 In [84]: s.reindex([e, b, f, d]) Out[84]: e -1.303966 b 0.232643 f NaN d -0.071506

Here, the f label was not contained in the Series and hence appears as NaN in the result. With a DataFrame, you can simultaneously reindex the index and columns:
In [85]: df Out[85]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661

two 1.848993 0.393073 1.007392 1.078785

In [86]: df.reindex(index=[c, f, b], columns=[three, two, one]) Out[86]: three two one c -0.507583 1.007392 -0.439897 f NaN NaN NaN b -0.107458 0.393073 0.446548

For convenience, you may utilize the reindex_axis method, which takes the labels and a keyword axis paramater. Note that the Index objects containing the actual axis labels can be shared between objects. So if we have a Series and a DataFrame, the following can be done:
In [87]: rs = s.reindex(df.index) In [88]: rs Out[88]: a -0.558130

6.6. Reindexing and altering labels

55

pandas: powerful Python data analysis toolkit, Release 0.7.1

b c d

0.232643 1.083226 -0.071506

In [89]: rs.index is df.index Out[89]: True

This means that the reindexed Seriess index is the same Python object as the DataFrames index. See Also: Advanced indexing is an even more concise way of doing reindexing. Note: When writing performance-sensitive code, there is a good reason to spend some time becoming a reindexing ninja: many operations are faster on pre-aligned data. Adding two unaligned DataFrames internally triggers a reindexing step. For exploratory analysis you will hardly notice the difference (because reindex has been heavily optimized), but when CPU cycles matter sprinking a few explicit reindex calls here and there can have an impact.

6.6.1 Reindexing to align with another object

You may wish to take an object and reindex its axes to be labeled the same as another object. While the syntax for this is straightforward albeit verbose, it is a common enough operation that the reindex_like method is available to make this simpler:
In [90]: df Out[90]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661 In [91]: df2 Out[91]: one two a 0.517755 0.76584 b 0.184345 -0.69008 c -0.702100 -0.07576 In [92]: df.reindex_like(df2) Out[92]: one two a 0.779958 1.848993 b 0.446548 0.393073 c -0.439897 1.007392

two 1.848993 0.393073 1.007392 1.078785

6.6.2 Reindexing with reindex_axis 6.6.3 Aligning objects with each other with align
The align method is the fastest way to simultaneously align two objects. It supports a join argument (related to joining and merging): join=outer: take the union of the indexes

56

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

join=left: use the calling objects index join=right: use the passed objects index join=inner: intersect the indexes It returns a tuple with both of the reindexed Series:
In [93]: s = Series(randn(5), index=[a, b, c, d, e]) In [94]: s1 = s[:4] In [95]: s2 = s[1:] In [96]: s1.align(s2) Out[96]: (a 1.007125 b 0.467322 c -0.514897 d 1.051824 e NaN, a NaN b 0.467322 c -0.514897 d 1.051824 e -0.654226) In [97]: s1.align(s2, join=inner) Out[97]: (b 0.467322 c -0.514897 d 1.051824, b 0.467322 c -0.514897 d 1.051824) In [98]: s1.align(s2, join=left) Out[98]: (a 1.007125 b 0.467322 c -0.514897 d 1.051824, a NaN b 0.467322 c -0.514897 d 1.051824)

For DataFrames, the join method will be applied to both the index and the columns by default:
In [99]: df.align(df2, join=inner) Out[99]: ( one two a 0.779958 1.848993 b 0.446548 0.393073 c -0.439897 1.007392, one two a 0.517755 0.76584 b 0.184345 -0.69008 c -0.702100 -0.07576)

You can also pass an axis option to only align on the specied axis: 6.6. Reindexing and altering labels 57

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [100]: df.align(df2, join=inner, axis=0) Out[100]: ( one three two a 0.779958 NaN 1.848993 b 0.446548 -0.107458 0.393073 c -0.439897 -0.507583 1.007392, one two a 0.517755 0.76584 b 0.184345 -0.69008 c -0.702100 -0.07576)

If you pass a Series to DataFrame.align, you can choose to align both objects either on the DataFrames index or columns using the axis argument:
In [101]: df.align(df2.ix[0], axis=1) Out[101]: ( one three two a 0.779958 NaN 1.848993 b 0.446548 -0.107458 0.393073 c -0.439897 -0.507583 1.007392 d NaN -0.433661 1.078785, one 0.517755 three NaN two 0.765840 Name: a)

6.6.4 Filling while reindexing

reindex takes an optional parameter method which is a lling method chosen from the following table: Method pad / fll bll / backll Action Fill values forward Fill values backward

Other ll methods could be added, of course, but these are the two most commonly used for time series data. In a way they only make sense for time series or otherwise ordered data, but you may have an application on non-time series data where this sort of interpolation logic is the correct thing to do. More sophisticated interpolation of missing values would be an obvious extension. We illustrate these ll methods on a simple TimeSeries:
In [102]: rng = DateRange(1/3/2000, periods=8) In [103]: ts = Series(randn(8), index=rng) In [104]: ts2 = ts[[0, 3, 6]] In [105]: ts Out[105]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12

0.377650 -1.061610 -0.307404 -0.337795 0.502206 0.149962 -0.719562 -0.615624

58

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [106]: ts2 Out[106]: 2000-01-03 0.377650 2000-01-06 -0.337795 2000-01-11 -0.719562 In [107]: ts2.reindex(ts.index) Out[107]: 2000-01-03 0.377650 2000-01-04 NaN 2000-01-05 NaN 2000-01-06 -0.337795 2000-01-07 NaN 2000-01-10 NaN 2000-01-11 -0.719562 2000-01-12 NaN In [108]: ts2.reindex(ts.index, method=ffill) Out[108]: 2000-01-03 0.377650 2000-01-04 0.377650 2000-01-05 0.377650 2000-01-06 -0.337795 2000-01-07 -0.337795 2000-01-10 -0.337795 2000-01-11 -0.719562 2000-01-12 -0.719562 In [109]: ts2.reindex(ts.index, method=bfill) Out[109]: 2000-01-03 0.377650 2000-01-04 -0.337795 2000-01-05 -0.337795 2000-01-06 -0.337795 2000-01-07 -0.719562 2000-01-10 -0.719562 2000-01-11 -0.719562 2000-01-12 NaN

Note the same result could have been achieved using llna:
In [110]: ts2.reindex(ts.index).fillna(method=ffill) Out[110]: 2000-01-03 0.377650 2000-01-04 0.377650 2000-01-05 0.377650 2000-01-06 -0.337795 2000-01-07 -0.337795 2000-01-10 -0.337795 2000-01-11 -0.719562 2000-01-12 -0.719562

Note these methods generally assume that the indexes are sorted. They may be modied in the future to be a bit more exible but as time series data is ordered most of the time anyway, this has not been a major priority.

6.6.5 Dropping labels from an axis

A method closely related to reindex is the drop function. It removes a set of labels from an axis: 6.6. Reindexing and altering labels 59

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [111]: df Out[111]: one three a 0.779958 NaN b 0.446548 -0.107458 c -0.439897 -0.507583 d NaN -0.433661

two 1.848993 0.393073 1.007392 1.078785

In [112]: df.drop([a, d], axis=0) Out[112]: one three two b 0.446548 -0.107458 0.393073 c -0.439897 -0.507583 1.007392 In [113]: df.drop([one], axis=1) Out[113]: three two a NaN 1.848993 b -0.107458 0.393073 c -0.507583 1.007392 d -0.433661 1.078785

Note that the following also works, but a bit less obvious / clean:
In [114]: df.reindex(df.index - [a, d]) Out[114]: one three two b 0.446548 -0.107458 0.393073 c -0.439897 -0.507583 1.007392

6.6.6 Renaming / mapping labels

The rename method allows you to relabel an axis based on some mapping (a dict or Series) or an arbitrary function.
In [115]: s Out[115]: a 1.007125 b 0.467322 c -0.514897 d 1.051824 e -0.654226 In [116]: s.rename(str.upper) Out[116]: A 1.007125 B 0.467322 C -0.514897 D 1.051824 E -0.654226

If you pass a function, it must return a value when called with any of the labels (and must produce a set of unique values). But if you pass a dict or Series, it need only contain a subset of the labels as keys:
In [117]: df.rename(columns={one : foo, two : bar}, .....: index={a : apple, b : banana, d : durian}) Out[117]: foo three bar apple 0.779958 NaN 1.848993

60

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

banana 0.446548 -0.107458 c -0.439897 -0.507583 durian NaN -0.433661

0.393073 1.007392 1.078785

The rename method also provides a copy named parameter that is by default True and copies the underlying data. Pass copy=False to rename the data in place. The Panel class has an a related rename_axis class which can rename any of its three axes.

6.7 Iteration
Considering the pandas as somewhat dict-like structure, basic iteration produces the keys of the objects, namely: Series: the index label DataFrame: the column labels Panel: the item labels Thus, for example:
In [118]: for col in df: .....: print col .....: one three two

6.7.1 iteritems
Consistent with the dict-like interface, iteritems iterates through key-value pairs: Series: (index, scalar value) pairs DataFrame: (column, Series) pairs Panel: (item, DataFrame) pairs For example:
In [119]: for item, frame in wp.iteritems(): .....: print item .....: print frame .....: Item1 A B C D 2000-01-03 -0.746076 0.674788 -0.571265 0.154209 2000-01-04 -1.367352 -0.188750 -0.904606 -0.852790 2000-01-05 0.227502 -0.151056 1.487440 0.266303 2000-01-06 -0.116207 1.962528 -0.826398 0.754717 2000-01-07 -0.250897 0.534875 -2.378483 1.382397 Item2 A B C D 2000-01-03 1.688713 0.271984 1.016070 -1.713926 2000-01-04 1.135326 -0.618932 -0.700908 0.460558 2000-01-05 0.936316 0.416414 -0.227416 1.449636 2000-01-06 0.731070 2.004273 -1.222206 -0.880667 2000-01-07 0.158218 0.753351 -1.555652 -0.522021

6.7. Iteration

61

pandas: powerful Python data analysis toolkit, Release 0.7.1

6.7.2 iterrows
New in v0.7 is the ability to iterate efciently through rows of a DataFrame. It returns an iterator yielding each index value along with a Series containing the data in each row:
In [120]: for row_index, row in df2.iterrows(): .....: print %s\n%s % (row_index, row) .....: a one 0.517755 two 0.765840 Name: a b one 0.184345 two -0.690080 Name: b c one -0.70210 two -0.07576 Name: c

For instance, a contrived way to transpose the dataframe would be:

In [121]: df2 = DataFrame({x: [1, 2, 3], y: [4, 5, 6]}) In [122]: print df2 x y 0 1 4 1 2 5 2 3 6 In [123]: print df2.T 0 1 2 x 1 2 3 y 4 5 6 In [124]: df2_t = DataFrame(dict((idx,values) for idx, values in df2.iterrows())) In [125]: print df2_t 0 1 2 x 1 2 3 y 4 5 6

6.7.3 itertuples
This method will return an iterator yielding a tuple for each row in the DataFrame. The rst element of the tuple will be the rows corresponding index value, while the remaining values are the row values proper. For instance,
In [126]: for r in df2.itertuples(): print r (0, 1, 4) (1, 2, 5) (2, 3, 6)

62

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

6.8 Sorting by index and value

There are two obvious kinds of sorting that you may be interested in: sorting by label and sorting by actual values. The primary method for sorting axis labels (indexes) across data structures is the sort_index method.
In [127]: unsorted_df = df.reindex(index=[a, d, c, b], .....: columns=[three, two, one]) In [128]: unsorted_df.sort_index() Out[128]: three two one a NaN 1.848993 0.779958 b -0.107458 0.393073 0.446548 c -0.507583 1.007392 -0.439897 d -0.433661 1.078785 NaN In [129]: unsorted_df.sort_index(ascending=False) Out[129]: three two one d -0.433661 1.078785 NaN c -0.507583 1.007392 -0.439897 b -0.107458 0.393073 0.446548 a NaN 1.848993 0.779958 In [130]: unsorted_df.sort_index(axis=1) Out[130]: one three two a 0.779958 NaN 1.848993 d NaN -0.433661 1.078785 c -0.439897 -0.507583 1.007392 b 0.446548 -0.107458 0.393073

DataFrame.sort_index can accept an optional by argument for axis=0 which will use an arbitrary vector or a column name of the DataFrame to determine the sort order:
In [131]: df.sort_index(by=two) Out[131]: one three two b 0.446548 -0.107458 0.393073 c -0.439897 -0.507583 1.007392 d NaN -0.433661 1.078785 a 0.779958 NaN 1.848993

The by argument can take a list of column names, e.g.:

In [132]: df = DataFrame({one:[2,1,1,1],two:[1,3,2,4],three:[5,4,3,2]}) In [133]: df[[one, two, three]].sort_index(by=[one,two]) Out[133]: one two three 2 1 2 3 1 1 3 4 3 1 4 2 0 2 1 5

Series has the method order (analogous to Rs order function) which sorts by value, with special treatment of NA values via the na_last argument:

6.8. Sorting by index and value

63

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [134]: s[2] = np.nan In [135]: s.order() Out[135]: e -0.654226 b 0.467322 a 1.007125 d 1.051824 c NaN In [136]: s.order(na_last=False) Out[136]: c NaN e -0.654226 b 0.467322 a 1.007125 d 1.051824

Some other sorting notes / nuances: Series.sort sorts a Series by value in-place. This is to provide compatibility with NumPy methods which expect the ndarray.sort behavior. DataFrame.sort takes a column argument instead of by. This method will likely be deprecated in a future release in favor of just using sort_index.

6.9 Copying, type casting

The copy method on pandas objects copies the underlying data (though not the axis indexes, since they are immutable) and returns a new object. Note that it is seldom necessary to copy objects. For example, there are only a handful of ways to alter a DataFrame in-place: Inserting, deleting, or modifying a column Assigning to the index or columns attributes For homogeneous data, directly modifying the values via the values attribute or advanced indexing To be clear, no pandas methods have the side effect of modifying your data; almost all methods return new objects, leaving the original object untouched. If data is modied, it is because you did so explicitly. Data can be explicitly cast to a NumPy dtype by using the astype method or alternately passing the dtype keyword argument to the object constructor.
In [137]: df = DataFrame(np.arange(12).reshape((4, 3))) In [138]: df[0].dtype Out[138]: dtype(int64) In [139]: df.astype(float)[0].dtype Out[139]: dtype(float64) In [140]: df = DataFrame(np.arange(12).reshape((4, 3)), dtype=float) In [141]: df[0].dtype Out[141]: dtype(float64)

64

Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

6.9.1 Inferring better types for object columns

The convert_objects DataFrame method will attempt to convert dtype=object columns to a better NumPy dtype. Occasionally (after transposing multiple times, for example), a mixed-type DataFrame will end up with everything as dtype=object. This method attempts to x that:
In [142]: df = DataFrame(randn(6, 3), columns=[a, b, c]) In [143]: df[d] = foo In [144]: df Out[144]: a b c 0 1.249928 1.275456 0.929073 1 1.371313 2.535220 -0.631121 2 -0.328488 -0.969865 -0.170630 3 0.014908 -0.473429 -0.700453 4 -0.328677 1.794254 1.708850 5 1.157524 0.111890 1.165142 In [145]: df = df.T.T In [146]: df.dtypes Out[146]: a object b object c object d object In [147]: converted = df.convert_objects() In [148]: converted.dtypes Out[148]: a float64 b float64 c float64 d object

d foo foo foo foo foo foo

6.10 Pickling and serialization

All pandas objects are equipped with save methods which use Pythons cPickle module to save data structures to disk using the pickle format.
In [149]: df Out[149]: 0 1 2 3 4 5 a b c 1.249928 1.275456 0.9290728 1.371313 2.53522 -0.6311205 -0.3284879 -0.9698651 -0.1706295 0.01490762 -0.4734291 -0.7004532 -0.3286771 1.794254 1.70885 1.157524 0.1118896 1.165142 d foo foo foo foo foo foo

In [150]: df.save(foo.pickle)

The load function in the pandas namespace can be used to load any pickled pandas object (or any other pickled

6.10. Pickling and serialization

65

pandas: powerful Python data analysis toolkit, Release 0.7.1

object) from le:

In [151]: load(foo.pickle) Out[151]: a b c 0 1.249928 1.275456 0.9290728 1 1.371313 2.53522 -0.6311205 2 -0.3284879 -0.9698651 -0.1706295 3 0.01490762 -0.4734291 -0.7004532 4 -0.3286771 1.794254 1.70885 5 1.157524 0.1118896 1.165142

d foo foo foo foo foo foo

There is also a save function which takes any object as its rst argument:
In [152]: save(df, foo.pickle) In [153]: load(foo.pickle) Out[153]: a b c 0 1.249928 1.275456 0.9290728 1 1.371313 2.53522 -0.6311205 2 -0.3284879 -0.9698651 -0.1706295 3 0.01490762 -0.4734291 -0.7004532 4 -0.3286771 1.794254 1.70885 5 1.157524 0.1118896 1.165142

d foo foo foo foo foo foo

6.11 Console Output Formatting

Use the set_eng_float_format function in the pandas.core.common module to alter the oating-point formatting of pandas objects to produce a particular format. For instance:
In [154]: set_eng_float_format(accuracy=3, use_eng_prefix=True) In [155]: df[a]/1.e3 Out[155]: 0 1.250m 1 1.371m 2 -328.488u 3 14.908u 4 -328.677u 5 1.158m Name: a In [156]: df[a]/1.e6 Out[156]: 0 1.250u 1 1.371u 2 -328.488n 3 14.908n 4 -328.677n 5 1.158u Name: a

The set_printoptions function has a number of options for controlling how oating point numbers are formatted (using hte precision argument) in the console and . The max_rows and max_columns control how many rows and columns of DataFrame objects are shown by default. If max_columns is set to 0 (the default, in fact), the library 66 Chapter 6. Essential basic functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

will attempt to t the DataFrames string representation into the current terminal width, and defaulting to the summary view otherwise.

6.11. Console Output Formatting

67

pandas: powerful Python data analysis toolkit, Release 0.7.1

68

Chapter 6. Essential basic functionality

CHAPTER

SEVEN

INDEXING AND SELECTING DATA

The axis labeling information in pandas objects serves many purposes: Identies data (i.e. provides metadata) using known indicators, important for for analysis, visualization, and interactive console display Enables automatic and explicit data alignment Allows intuitive getting and setting of subsets of the data set In this section / chapter, we will focus on the nal point: namely, how to slice, dice, and generally get and set subsets of pandas objects. The primary focus will be on Series and DataFrame as they have received more development attention in this area. Expect more work to be invested higher-dimensional data structures (including Panel) in the future, especially in label-based advanced indexing.

7.1 Basics
As mentioned when introducing the data structures in the last section, the primary function of indexing with [] (a.k.a. __getitem__ for those familiar with implementing class behavior in Python) is selecting out lower-dimensional slices. Thus, Series: series[label] returns a scalar value DataFrame: frame[colname] returns a Series corresponding to the passed column name Panel: panel[itemname] returns a DataFrame corresponding to the passed item name Here we construct a simple time series data set to use for illustrating the indexing functionality:
In [423]: dates = np.asarray(DateRange(1/1/2000, periods=8)) In [424]: df = DataFrame(randn(8, 4), index=dates, columns=[A, B, C, D]) In [425]: df Out[425]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A 0.469112 1.212112 -0.861849 0.721555 -0.424972 -0.673690 0.404705 -0.370647 B -0.282863 -0.173215 -2.104569 -0.706771 0.567020 0.113648 0.577046 -1.157892 C -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427 -1.715002 -1.344312 D -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885

69

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [426]: panel = Panel({one : df, two : df - df.mean()}) In [427]: panel Out[427]: <class pandas.core.panel.Panel> Dimensions: 2 (items) x 8 (major) x 4 (minor) Items: one to two Major axis: 2000-01-03 00:00:00 to 2000-01-12 00:00:00 Minor axis: A to D

Note: None of the indexing functionality is time series specic unless specically stated. Thus, as per above, we have the most basic indexing using []:
In [428]: s = df[A] In [429]: s[dates[5]] Out[429]: -0.67368970808837059 In [430]: panel[two] Out[430]: A B 2000-01-03 0.409571 0.113086 2000-01-04 1.152571 0.222735 2000-01-05 -0.921390 -1.708620 2000-01-06 0.662014 -0.310822 2000-01-07 -0.484513 0.962970 2000-01-10 -0.733231 0.509598 2000-01-11 0.345164 0.972995 2000-01-12 -0.430188 -0.761943

C -0.610826 1.017442 0.403304 -0.141342 1.174465 -0.580194 -0.816769 -0.446079

D -0.936507 -0.845111 1.270929 0.470985 -0.888276 0.724113 -0.840143 1.044010

7.1.1 Fast scalar value getting and setting

Since indexing with [] must handle a lot of cases (single-label access, slicing, boolean indexing, etc.), it has a bit of overhead in order to gure out what youre asking for. If you only want to access a scalar value, the fastest way is to use the get_value method, which is implemented on all of the data structures:
In [431]: s.get_value(dates[5]) Out[431]: -0.67368970808837059 In [432]: df.get_value(dates[5], A) Out[432]: -0.67368970808837059

There is an analogous set_value method which has the additional capability of enlarging an object. This method always returns a reference to the object it modied, which in the fast of enlargement, will be a new object:
In [433]: df.set_value(dates[5], E, 7) Out[433]: A B C D E 2000-01-03 0.469112 -0.282863 -1.509059 -1.135632 NaN 2000-01-04 1.212112 -0.173215 0.119209 -1.044236 NaN 2000-01-05 -0.861849 -2.104569 -0.494929 1.071804 NaN 2000-01-06 0.721555 -0.706771 -1.039575 0.271860 NaN 2000-01-07 -0.424972 0.567020 0.276232 -1.087401 NaN 2000-01-10 -0.673690 0.113648 -1.478427 0.524988 7

70

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

2000-01-11 0.404705 0.577046 -1.715002 -1.039268 NaN 2000-01-12 -0.370647 -1.157892 -1.344312 0.844885 NaN

7.1.2 Additional Column Access

You may access a column on a dataframe directly as an attribute:
In [434]: df.A Out[434]: 2000-01-03 0.469112 2000-01-04 1.212112 2000-01-05 -0.861849 2000-01-06 0.721555 2000-01-07 -0.424972 2000-01-10 -0.673690 2000-01-11 0.404705 2000-01-12 -0.370647 Name: A

If you are using the IPython environment, you may also use tab-completion to see the accessible columns of a DataFrame. You can pass a list of columns to [] to select columns in that order: If a column is not contained in the DataFrame, an exception will be raised. Multiple columns can also be set in this manner:
In [435]: df Out[435]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A 0.469112 1.212112 -0.861849 0.721555 -0.424972 -0.673690 0.404705 -0.370647 B -0.282863 -0.173215 -2.104569 -0.706771 0.567020 0.113648 0.577046 -1.157892 C -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427 -1.715002 -1.344312 D -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885

In [436]: df[[B, A]] = df[[A, B]] In [437]: df Out[437]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A -0.282863 -0.173215 -2.104569 -0.706771 0.567020 0.113648 0.577046 -1.157892 B 0.469112 1.212112 -0.861849 0.721555 -0.424972 -0.673690 0.404705 -0.370647 C -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427 -1.715002 -1.344312 D -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885

You may nd this useful for applying a transform (in-place) to a subset of the columns.

7.1.3 Data slices on other axes

Its certainly possible to retrieve data slices along the other axes of a DataFrame or Panel. We tend to refer to these slices as cross-sections. DataFrame has the xs function for retrieving rows as Series and Panel has the analogous 7.1. Basics 71

pandas: powerful Python data analysis toolkit, Release 0.7.1

major_xs and minor_xs functions for retrieving slices as DataFrames for a given major_axis or minor_axis label, respectively.
In [438]: date = dates[5] In [439]: df.xs(date) Out[439]: A 0.113648 B -0.673690 C -1.478427 D 0.524988 Name: 2000-01-10 00:00:00 In [440]: panel.major_xs(date) Out[440]: one two A -0.673690 -0.733231 B 0.113648 0.509598 C -1.478427 -0.580194 D 0.524988 0.724113 In [441]: panel.minor_xs(A) Out[441]: one two 2000-01-03 0.469112 0.409571 2000-01-04 1.212112 1.152571 2000-01-05 -0.861849 -0.921390 2000-01-06 0.721555 0.662014 2000-01-07 -0.424972 -0.484513 2000-01-10 -0.673690 -0.733231 2000-01-11 0.404705 0.345164 2000-01-12 -0.370647 -0.430188

7.1.4 Slicing ranges

The most robust and consistent way of slicing ranges along arbitrary axes is described in the Advanced indexing section detailing the .ix method. For now, we explain the semantics of slicing using the [] operator. With Series, the syntax works exactly as with an ndarray, returning a slice of the values and the corresponding labels:
In [442]: s[:5] Out[442]: 2000-01-03 -0.282863 2000-01-04 -0.173215 2000-01-05 -2.104569 2000-01-06 -0.706771 2000-01-07 0.567020 Name: A In [443]: s[::2] Out[443]: 2000-01-03 -0.282863 2000-01-05 -2.104569 2000-01-07 0.567020 2000-01-11 0.577046 Name: A In [444]: s[::-1]

72

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[444]: 2000-01-12 2000-01-11 2000-01-10 2000-01-07 2000-01-06 2000-01-05 2000-01-04 2000-01-03 Name: A

-1.157892 0.577046 0.113648 0.567020 -0.706771 -2.104569 -0.173215 -0.282863

Note that setting works as well:

In [445]: s2 = s.copy() In [446]: s2[:5] = 0 In [447]: s2 Out[447]: 2000-01-03 0.000000 2000-01-04 0.000000 2000-01-05 0.000000 2000-01-06 0.000000 2000-01-07 0.000000 2000-01-10 0.113648 2000-01-11 0.577046 2000-01-12 -1.157892 Name: A

With DataFrame, slicing inside of [] slices the rows. This is provided largely as a convenience since it is such a common operation.
In [448]: df[:3] Out[448]: A B C D 2000-01-03 -0.282863 0.469112 -1.509059 -1.135632 2000-01-04 -0.173215 1.212112 0.119209 -1.044236 2000-01-05 -2.104569 -0.861849 -0.494929 1.071804 In [449]: df[::-1] Out[449]: 2000-01-12 2000-01-11 2000-01-10 2000-01-07 2000-01-06 2000-01-05 2000-01-04 2000-01-03 A -1.157892 0.577046 0.113648 0.567020 -0.706771 -2.104569 -0.173215 -0.282863 B -0.370647 0.404705 -0.673690 -0.424972 0.721555 -0.861849 1.212112 0.469112 C -1.344312 -1.715002 -1.478427 0.276232 -1.039575 -0.494929 0.119209 -1.509059 D 0.844885 -1.039268 0.524988 -1.087401 0.271860 1.071804 -1.044236 -1.135632

7.1.5 Boolean indexing

Using a boolean vector to index a Series works exactly as in a numpy ndarray:
In [450]: s[s > 0] Out[450]: 2000-01-07 0.567020

7.1. Basics

73

pandas: powerful Python data analysis toolkit, Release 0.7.1

2000-01-10 2000-01-11 Name: A

0.113648 0.577046

In [451]: s[(s < 0) & (s > -0.5)] Out[451]: 2000-01-03 -0.282863 2000-01-04 -0.173215 Name: A

You may select rows from a DataFrame using a boolean vector the same length as the DataFrames index (for example, something derived from one of the columns of the DataFrame):
In [452]: df[df[A] > 0] Out[452]: A B C D 2000-01-07 0.567020 -0.424972 0.276232 -1.087401 2000-01-10 0.113648 -0.673690 -1.478427 0.524988 2000-01-11 0.577046 0.404705 -1.715002 -1.039268

Consider the isin method of Series, which returns a boolean vector that is true wherever the Series elements exist in the passed list. This allows you to select out rows where one or more columns have values you want:
In [453]: df2 = DataFrame({a : [one, one, two, three, two, one, six], .....: b : [x, y, y, x, y, x, x], .....: c : np.random.randn(7)}) In [454]: df2[df2[a].isin([one, two])] Out[454]: a b c 0 one x 1.075770 1 one y -0.109050 2 two y 1.643563 4 two y 0.357021 5 one x -0.674600

Note, with the advanced indexing ix method, you may select along more than one axis using boolean vectors combined with other indexing expressions.

7.1.6 Indexing a DataFrame with a boolean DataFrame

You may wish to set values on a DataFrame based on some boolean criteria derived from itself or another DataFrame or set of DataFrames. This can be done intuitively like so:
In [455]: df2 = df.copy() In [456]: df2 < 0 Out[456]: A 2000-01-03 True 2000-01-04 True 2000-01-05 True 2000-01-06 True 2000-01-07 False 2000-01-10 False 2000-01-11 False 2000-01-12 True

B False False True False True True False True

C True False True True False True True True

D True True False False True False True False

74

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [457]: df2[df2 < 0] = 0 In [458]: df2 Out[458]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A 0.000000 0.000000 0.000000 0.000000 0.567020 0.113648 0.577046 0.000000 B 0.469112 1.212112 0.000000 0.721555 0.000000 0.000000 0.404705 0.000000 C 0.000000 0.119209 0.000000 0.000000 0.276232 0.000000 0.000000 0.000000 D 0.000000 0.000000 1.071804 0.271860 0.000000 0.524988 0.000000 0.844885

Note that such an operation requires that the boolean DataFrame is indexed exactly the same.

7.1.7 Take Methods

TODO: Fill Me In

7.1.8 Duplicate Data

If you want to indentify and remove duplicate rows in a DataFrame, there are two methods that will help: duplicated and drop_duplicates. Each takes as an argument the columns to use to identify duplicated rows. duplicated returns a boolean vector whose length is the number of rows, and which indicates whether a row is duplicated. drop_duplicates removes duplicate rows. By default, the rst observed row of a duplicate set is considered unique, but each method has a take_last parameter that indicates the last observed row should be taken instead.
In [459]: df2 = DataFrame({a : [one, one, two, three, two, one, six], .....: b : [x, y, y, x, y, x, x], .....: c : np.random.randn(7)}) In [460]: df2.duplicated([a,b]) Out[460]: 0 False 1 False 2 False 3 False 4 True 5 True 6 False In [461]: Out[461]: a 0 one 1 one 2 two 3 three 6 six df2.drop_duplicates([a,b]) b c x -0.968914 y -1.294524 y 0.413738 x 0.276662 x -0.362543

In [462]: df2.drop_duplicates([a,b], take_last=True)

7.1. Basics

75

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[462]: a 1 one 3 three 4 two 5 one 6 six

b y x y x x

c -1.294524 0.276662 -0.472035 -0.013960 -0.362543

7.1.9 Dictionary-like get method

Each of Series, DataFrame, and Panel have a get method which can return a default value.
In [463]: s = Series([1,2,3], index=[a,b,c]) In [464]: s.get(a) Out[464]: 1 In [465]: s.get(x, default=-1) Out[465]: -1 # equivalent to s[a]

7.2 Advanced indexing with labels

We have avoided excessively overloading the [] / __getitem__ operator to keep the basic functionality of the pandas objects straightforward and simple. However, there are often times when you may wish get a subset (or analogously set a subset) of the data in a way that is not straightforward using the combination of reindex and []. Complicated setting operations are actually quite difcult because reindex usually returns a copy. By advanced indexing we are referring to a special .ix attribute on pandas objects which enable you to do getting/setting operations on a DataFrame, for example, with matrix/ndarray-like semantics. Thus you can combine the following kinds of indexing: An integer or single label, e.g. 5 or a A list or array of labels [a, b, c] or integers [4, 3, 0] A slice object with ints 1:7 or labels a:f A boolean array Well illustrate all of these methods. First, note that this provides a concise way of reindexing on multiple axes at once:
In [466]: subindex = dates[[3,4,5]] In [467]: df.reindex(index=subindex, columns=[C, B]) Out[467]: C B 2000-01-06 -1.039575 0.721555 2000-01-07 0.276232 -0.424972 2000-01-10 -1.478427 -0.673690 In [468]: df.ix[subindex, [C, B]] Out[468]: C B 2000-01-06 -1.039575 0.721555 2000-01-07 0.276232 -0.424972 2000-01-10 -1.478427 -0.673690

76

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

Assignment / setting values is possible when using ix:

In [469]: df2 = df.copy() In [470]: df2.ix[subindex, [C, B]] = 0 In [471]: df2 Out[471]: 2000-01-03 2000-01-04 2000-01-05 2000-01-06 2000-01-07 2000-01-10 2000-01-11 2000-01-12 A B C -0.282863 0.469112 -1.509059 -0.173215 1.212112 0.119209 -2.104569 -0.861849 -0.494929 -0.706771 0.000000 0.000000 0.567020 0.000000 0.000000 0.113648 0.000000 0.000000 0.577046 0.404705 -1.715002 -1.157892 -0.370647 -1.344312 D -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885

Indexing with an array of integers can also be done:

In [472]: df.ix[[4,3,1]] Out[472]: A B C D 2000-01-07 0.567020 -0.424972 0.276232 -1.087401 2000-01-06 -0.706771 0.721555 -1.039575 0.271860 2000-01-04 -0.173215 1.212112 0.119209 -1.044236 In [473]: df.ix[dates[[4,3,1]]] Out[473]: A B C D 2000-01-07 0.567020 -0.424972 0.276232 -1.087401 2000-01-06 -0.706771 0.721555 -1.039575 0.271860 2000-01-04 -0.173215 1.212112 0.119209 -1.044236

Slicing has standard Python semantics for integer slices:

In [474]: df.ix[1:7, Out[474]: A 2000-01-04 -0.173215 2000-01-05 -2.104569 2000-01-06 -0.706771 2000-01-07 0.567020 2000-01-10 0.113648 2000-01-11 0.577046 :2] B 1.212112 -0.861849 0.721555 -0.424972 -0.673690 0.404705

Slicing with labels is semantically slightly different because the slice start and stop are inclusive in the label-based case:
In [475]: date1, date2 = dates[[2, 4]] In [476]: print date1, date2 2000-01-05 00:00:00 2000-01-07 00:00:00 In [477]: df.ix[date1:date2] Out[477]: A B C D 2000-01-05 -2.104569 -0.861849 -0.494929 1.071804 2000-01-06 -0.706771 0.721555 -1.039575 0.271860 2000-01-07 0.567020 -0.424972 0.276232 -1.087401

7.2. Advanced indexing with labels

77

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [478]: df[A].ix[date1:date2] Out[478]: 2000-01-05 -2.104569 2000-01-06 -0.706771 2000-01-07 0.567020 Name: A

Getting and setting rows in a DataFrame, especially by their location, is much easier:
In [479]: df2 = df[:5].copy() In [480]: df2.ix[3] Out[480]: A -0.706771 B 0.721555 C -1.039575 D 0.271860 Name: 2000-01-06 00:00:00 In [481]: df2.ix[3] = np.arange(len(df2.columns)) In [482]: df2 Out[482]: A B C D 2000-01-03 -0.282863 0.469112 -1.509059 -1.135632 2000-01-04 -0.173215 1.212112 0.119209 -1.044236 2000-01-05 -2.104569 -0.861849 -0.494929 1.071804 2000-01-06 0.000000 1.000000 2.000000 3.000000 2000-01-07 0.567020 -0.424972 0.276232 -1.087401

Column or row selection can be combined as you would expect with arrays of labels or even boolean vectors:
In [483]: df.ix[df[A] > 0, B] Out[483]: 2000-01-07 -0.424972 2000-01-10 -0.673690 2000-01-11 0.404705 Name: B In [484]: df.ix[date1:date2, B] Out[484]: 2000-01-05 -0.861849 2000-01-06 0.721555 2000-01-07 -0.424972 Name: B In [485]: df.ix[date1, B] Out[485]: -0.86184896334779992

Slicing with labels is closely related to the truncate method which does precisely .ix[start:stop] but returns a copy (for legacy reasons).

7.2.1 Returning a view versus a copy

The rules about when a view on the data is returned are entirely dependent on NumPy. Whenever an array of labels or a boolean vector are involved in the indexing operation, the result will be a copy. With single label / scalar indexing and slicing, e.g. df.ix[3:6] or df.ix[:, A], a view will be returned.

78

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

7.2.2 The select method

Another way to extract slices from an object is with the select method of Series, DataFrame, and Panel. This method should be used only when there is no more direct way. select takes a function which operates on labels along axis and returns a boolean. For instance:
In [486]: df.select(lambda x: x == A, axis=1) Out[486]: A 2000-01-03 -0.282863 2000-01-04 -0.173215 2000-01-05 -2.104569 2000-01-06 -0.706771 2000-01-07 0.567020 2000-01-10 0.113648 2000-01-11 0.577046 2000-01-12 -1.157892

7.2.3 The lookup method

Sometimes you want to extract a set of values given a sequence of row labels and column labels, and the lookup method allows for this and returns a numpy array. For instance,
In [487]: dflookup = DataFrame(np.random.rand(20,4), columns = [A,B,C,D]) In [488]: dflookup.lookup(xrange(0,10,2), [B,C,A,B,D]) Out[488]: array([ 0.4973, 0.9423, 0.8626, 0.6341, 0.5629])

7.2.4 Advanced indexing with integer labels

Label-based indexing with integer axis labels is a thorny topic. It has been discussed heavily on mailing lists and among various members of the scientic Python community. In pandas, our general viewpoint is that labels matter more than integer locations. Therefore, advanced indexing with .ix will always attempt label-based indexing, before falling back on integer-based indexing.

7.2.5 Setting values in mixed-type DataFrame

Setting values on a mixed-type DataFrame or Panel is supported when using scalar values, though setting arbitrary vectors is not yet supported:
In [489]: df2 = df[:4] In [490]: df2[foo] = bar In [491]: print df2 2000-01-03 2000-01-04 2000-01-05 2000-01-06 A B C D -0.282863 0.469112 -1.509059 -1.135632 -0.173215 1.212112 0.119209 -1.044236 -2.104569 -0.861849 -0.494929 1.071804 -0.706771 0.721555 -1.039575 0.271860 foo bar bar bar bar

In [492]: df2.ix[2] = np.nan In [493]: print df2

7.2. Advanced indexing with labels

79

pandas: powerful Python data analysis toolkit, Release 0.7.1

A 2000-01-03 -0.282863 2000-01-04 -0.173215 2000-01-05 NaN 2000-01-06 -0.706771

B C D 0.469112 -1.509059 -1.135632 1.212112 0.119209 -1.044236 NaN NaN NaN 0.721555 -1.039575 0.271860

foo bar bar NaN bar

In [494]: print df2.dtypes A float64 B float64 C float64 D float64 foo object

7.3 Index objects

The pandas Index class and its subclasses can be viewed as implementing an ordered set in addition to providing the support infrastructure necessary for lookups, data alignment, and reindexing. The easiest way to create one directly is to pass a list or other sequence to Index:
In [495]: index = Index([e, d, a, b]) In [496]: index Out[496]: Index([e, d, a, b], dtype=object) In [497]: d in index Out[497]: True

You can also pass a name to be stored in the index:

In [498]: index = Index([e, d, a, b], name=something) In [499]: index.name Out[499]: something

Starting with pandas 0.5, the name, if set, will be shown in the console display:
In [500]: index = Index(range(5), name=rows) In [501]: columns = Index([A, B, C], name=cols) In [502]: df = DataFrame(np.random.randn(5, 3), index=index, columns=columns) In [503]: df Out[503]: cols A B C rows 0 3.357427 -0.317441 -1.236269 1 0.896171 -0.487602 -0.082240 2 -2.182937 0.380396 0.084844 3 0.432390 1.519970 -0.493662 4 0.600178 0.274230 0.132885 In [504]: df[A] Out[504]: rows 0 3.357427

80

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

1 0.896171 2 -2.182937 3 0.432390 4 0.600178 Name: A

7.3.1 Set operations on Index objects

The three main operations are union (|), intersection (&), and diff (-). These can be directly called as instance methods or used via overloaded operators:
In [505]: a = Index([c, b, a]) In [506]: b = Index([c, e, d]) In [507]: a.union(b) Out[507]: Index([a, b, c, d, e], dtype=object) In [508]: a | b Out[508]: Index([a, b, c, d, e], dtype=object) In [509]: a & b Out[509]: Index([c], dtype=object) In [510]: a - b Out[510]: Index([a, b], dtype=object)

7.3.2 isin method of Index objects

One additional operation is the isin method that works analogously to the Series.isin method found here.

7.4 Hierarchical indexing (MultiIndex)

Hierarchical indexing (also referred to as multi-level indexing) is brand new in the pandas 0.4 release. It is very exciting as it opens the door to some quite sophisticated data analysis and manipulation, especially for working with higher dimensional data. In essence, it enables you to effectively store and manipulate arbitrarily high dimension data in a 2-dimensional tabular structure (DataFrame), for example. It is not limited to DataFrame In this section, we will show what exactly we mean by hierarchical indexing and how it integrates with the all of the pandas indexing functionality described above and in prior sections. Later, when discussing group by and pivoting and reshaping data, well show non-trivial applications to illustrate how it aids in structuring data for analysis. Note: Given that hierarchical indexing is so new to the library, it is denitely bleeding-edge functionality but is certainly suitable for production. But, there may inevitably be some minor API changes as more use cases are explored and any weaknesses in the design / implementation are identied. pandas aims to be eminently usable so any feedback about new functionality like this is extremely helpful.

7.4. Hierarchical indexing (MultiIndex)

81

pandas: powerful Python data analysis toolkit, Release 0.7.1

7.4.1 Creating a MultiIndex (hierarchical index) object

The MultiIndex object is the hierarchical analogue of the standard Index object which typically stores the axis labels in pandas objects. You can think of MultiIndex an array of tuples where each tuple is unique. A MultiIndex can be created from a list of arrays (using MultiIndex.from_arrays) or an array of tuples (using MultiIndex.from_tuples).
In [511]: arrays = [[bar, bar, baz, baz, foo, foo, qux, qux], .....: [one, two, one, two, one, two, one, two]] In [512]: tuples = zip(*arrays) In [513]: tuples Out[513]: [(bar, one), (bar, two), (baz, one), (baz, two), (foo, one), (foo, two), (qux, one), (qux, two)] In [514]: index = MultiIndex.from_tuples(tuples, names=[first, second]) In [515]: s = Series(randn(8), index=index) In [516]: s Out[516]: first second bar one two baz one two foo one two qux one two

-0.023688 2.410179 1.450520 0.206053 -0.251905 -2.213588 1.063327 1.266143

All of the MultiIndex constructors accept a names argument which stores string names for the levels themselves. If no names are provided, some arbitrary ones will be assigned:
In [517]: index.names Out[517]: [first, second]

This index can back any axis of a pandas object, and the number of levels of the index is up to you:
In [518]: df = DataFrame(randn(3, 8), index=[A, B, C], columns=index) In [519]: df Out[519]: first bar baz foo qux second one two one two one two one two A 0.299368 -0.863838 0.408204 -1.048089 -0.025747 -0.988387 0.094055 1.262731 B 1.289997 0.082423 -0.055758 0.536580 -0.489682 0.369374 -0.034571 -2.484478 C -0.281461 0.030711 0.109121 1.126203 -0.977349 1.474071 -0.064034 -1.282782 In [520]: DataFrame(randn(6, 6), index=index[:6], columns=index[:6]) Out[520]:

82

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

first bar baz foo second one two one two one two first second bar one 0.781836 -1.071357 0.441153 2.353925 0.583787 0.221471 two -0.744471 0.758527 1.729689 -0.964980 -0.845696 -1.340896 baz one 1.846883 -1.328865 1.682706 -1.717693 0.888782 0.228440 two 0.901805 1.171216 0.520260 -1.197071 -1.066969 -0.303421 foo one -0.858447 0.306996 -0.028665 0.384316 1.574159 1.588931 two 0.476720 0.473424 -0.242861 -0.014805 -0.284319 0.650776

Weve sparsied the higher levels of the indexes to make the console output a bit easier on the eyes. Its worth keeping in mind that theres nothing preventing you from using tuples as atomic labels on an axis:
In [521]: Series(randn(8), index=tuples) Out[521]: (bar, one) -1.461665 (bar, two) -1.137707 (baz, one) -0.891060 (baz, two) -0.693921 (foo, one) 1.613616 (foo, two) 0.464000 (qux, one) 0.227371 (qux, two) -0.496922

The reason that the MultiIndex matters is that it can allow you to do grouping, selection, and reshaping operations as we will describe below and in subsequent areas of the documentation. As you will see in later sections, you can nd yourself working with hierarchically-indexed data without creating a MultiIndex explicitly yourself. However, when loading data from a le, you may wish to generate your own MultiIndex when preparing the data set.

7.4.2 Reconstructing the level labels

The method get_level_values will return a vector of the labels for each location at a particular level:
In [522]: index.get_level_values(0) Out[522]: array([bar, bar, baz, baz, foo, foo, qux, qux], dtype=object) In [523]: index.get_level_values(second) Out[523]: array([one, two, one, two, one, two, one, two], dtype=object)

7.4.3 Basic indexing on axis with MultiIndex

One of the important features of hierarchical indexing is that you can select data by a partial label identifying a subgroup in the data. Partial selection drops levels of the hierarchical index in the result in a completely analogous way to selecting a column in a regular DataFrame:
In [524]: df[bar] Out[524]: second one two A 0.299368 -0.863838 B 1.289997 0.082423 C -0.281461 0.030711 In [525]: df[bar, one] Out[525]: A 0.299368

7.4. Hierarchical indexing (MultiIndex)

83

pandas: powerful Python data analysis toolkit, Release 0.7.1

B 1.289997 C -0.281461 Name: (bar, one) In [526]: df[bar][one] Out[526]: A 0.299368 B 1.289997 C -0.281461 Name: one In [527]: s[qux] Out[527]: second one 1.063327 two 1.266143

7.4.4 Data alignment and using reindex

Operations between differently-indexed objects having MultiIndex on the axes will work as you expect; data alignment will work the same as an Index of tuples:
In [528]: s + s[:-2] Out[528]: first second bar one -0.047377 two 4.820357 baz one 2.901041 two 0.412107 foo one -0.503810 two -4.427175 qux one NaN two NaN In [529]: s + s[::2] Out[529]: first second bar one -0.047377 two NaN baz one 2.901041 two NaN foo one -0.503810 two NaN qux one 2.126655 two NaN

reindex can be called with another MultiIndex or even a list or array of tuples:
In [530]: s.reindex(index[:3]) Out[530]: first second bar one -0.023688 two 2.410179 baz one 1.450520 In [531]: s.reindex([(foo, two), (bar, one), (qux, one), (baz, one)]) Out[531]:

84

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

first foo bar qux baz

second two one one one

-2.213588 -0.023688 1.063327 1.450520

7.4.5 Advanced indexing with hierarchical index

Syntactically integrating MultiIndex in advanced indexing with .ix is a bit challenging, but weve made every effort to do so. for example the following works as you would expect:
In [532]: df = df.T In [533]: df Out[533]: A first second bar one two baz one two foo one two qux one two 0.299368 -0.863838 0.408204 -1.048089 -0.025747 -0.988387 0.094055 1.262731 B 1.289997 0.082423 -0.055758 0.536580 -0.489682 0.369374 -0.034571 -2.484478 C -0.281461 0.030711 0.109121 1.126203 -0.977349 1.474071 -0.064034 -1.282782

In [534]: df.ix[bar] Out[534]: A B C second one 0.299368 1.289997 -0.281461 two -0.863838 0.082423 0.030711 In [535]: df.ix[bar, two] Out[535]: A -0.863838 B 0.082423 C 0.030711 Name: (bar, two)

Partial slicing also works quite nicely:

In [536]: df.ix[baz:foo] Out[536]: A B C first second baz one 0.408204 -0.055758 0.109121 two -1.048089 0.536580 1.126203 foo one -0.025747 -0.489682 -0.977349 two -0.988387 0.369374 1.474071 In [537]: df.ix[(baz, two):(qux, one)] Out[537]: A B C first second baz two -1.048089 0.536580 1.126203 foo one -0.025747 -0.489682 -0.977349

7.4. Hierarchical indexing (MultiIndex)

85

pandas: powerful Python data analysis toolkit, Release 0.7.1

qux

two one

-0.988387 0.369374 1.474071 0.094055 -0.034571 -0.064034

In [538]: df.ix[(baz, two):foo] Out[538]: A B C first second baz two -1.048089 0.536580 1.126203 foo one -0.025747 -0.489682 -0.977349 two -0.988387 0.369374 1.474071

Passing a list of labels or tuples works similar to reindexing:

In [539]: df.ix[[(bar, two), (qux, one)]] Out[539]: A B C first second bar two -0.863838 0.082423 0.030711 qux one 0.094055 -0.034571 -0.064034

The following does not work, and its not clear if it should or not:
>>> df.ix[[bar, qux]]

The code for implementing .ix makes every attempt to do the right thing but as you use it you may uncover corner cases or unintuitive behavior. If you do nd something like this, do not hesitate to report the issue or ask on the mailing list.

7.4.6 Cross-section with hierarchical index

The xs method of DataFrame additionally takes a level argument to make selecting data at a particular level of a MultiIndex easier.
In [540]: df.xs(one, level=second) Out[540]: A B C first bar 0.299368 1.289997 -0.281461 baz 0.408204 -0.055758 0.109121 foo -0.025747 -0.489682 -0.977349 qux 0.094055 -0.034571 -0.064034

7.4.7 Advanced reindexing and alignment with hierarchical index

The parameter level has been added to the reindex and align methods of pandas objects. This is useful to broadcast values across a level. For instance:
In [541]: midx = MultiIndex(levels=[[zero, one], [x,y]], .....: labels=[[1,1,0,0],[1,0,1,0]]) In [542]: df = DataFrame(randn(4,2), index=midx) In [543]: print df 0 1 one y 0.306389 -2.290613 x -1.134623 -1.561819

86

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

zero y -0.260838 0.281957 x 1.523962 -0.902937 In [544]: df2 = df.mean(level=0) In [545]: print df2 0 1 key_0 zero 0.631562 -0.310490 one -0.414117 -1.926216 In [546]: print df2.reindex(df.index, level=0) 0 1 one y -0.414117 -1.926216 x -0.414117 -1.926216 zero y 0.631562 -0.310490 x 0.631562 -0.310490 In [547]: df_aligned, df2_aligned = df.align(df2, level=0) In [548]: print df_aligned 0 1 one y 0.306389 -2.290613 x -1.134623 -1.561819 zero y -0.260838 0.281957 x 1.523962 -0.902937 In [549]: print df2_aligned 0 1 one y -0.414117 -1.926216 x -0.414117 -1.926216 zero y 0.631562 -0.310490 x 0.631562 -0.310490

7.4.8 The need for sortedness

Caveat emptor: the present implementation of MultiIndex requires that the labels be sorted for some of the slicing / indexing routines to work correctly. You can think about breaking the axis into unique groups, where at the hierarchical level of interest, each distinct group shares a label, but no two have the same label. However, the MultiIndex does not enforce this: you are responsible for ensuring that things are properly sorted. There is an important new method sortlevel to sort an axis within a MultiIndex so that its labels are grouped and sorted by the original ordering of the associated factor at that level. Note that this does not necessarily mean the labels will be sorted lexicographically!
In [550]: import random; random.shuffle(tuples) In [551]: s = Series(randn(8), index=MultiIndex.from_tuples(tuples)) In [552]: s Out[552]: qux one 0.068159 bar two -0.057873 foo one -0.368204 two -1.144073 baz one 0.861209 two 0.800193

7.4. Hierarchical indexing (MultiIndex)

87

pandas: powerful Python data analysis toolkit, Release 0.7.1

qux bar

two one

0.782098 -1.069094

In [553]: s.sortlevel(0) Out[553]: bar one -1.069094 two -0.057873 baz one 0.861209 two 0.800193 foo one -0.368204 two -1.144073 qux one 0.068159 two 0.782098 In [554]: s.sortlevel(1) Out[554]: bar one -1.069094 baz one 0.861209 foo one -0.368204 qux one 0.068159 bar two -0.057873 baz two 0.800193 foo two -1.144073 qux two 0.782098

Note, you may also pass a level name to sortlevel if the MultiIndex levels are named.
In [555]: s.index.names = [L1, L2] In [556]: s.sortlevel(level=L1) Out[556]: L1 L2 bar one -1.069094 two -0.057873 baz one 0.861209 two 0.800193 foo one -0.368204 two -1.144073 qux one 0.068159 two 0.782098 In [557]: s.sortlevel(level=L2) Out[557]: L1 L2 bar one -1.069094 baz one 0.861209 foo one -0.368204 qux one 0.068159 bar two -0.057873 baz two 0.800193 foo two -1.144073 qux two 0.782098

Some indexing will work even if the data are not sorted, but will be rather inefcient and will also return a copy of the data rather than a view:
In [558]: s[qux] Out[558]: L2

88

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

one two

0.068159 0.782098

In [559]: s.sortlevel(1)[qux] Out[559]: L2 one 0.068159 two 0.782098

On higher dimensional objects, you can sort any of the other axes by level if they have a MultiIndex:
In [560]: df.T.sortlevel(1, axis=1) Out[560]: zero one zero one x x y y 0 1.523962 -1.134623 -0.260838 0.306389 1 -0.902937 -1.561819 0.281957 -2.290613

The MultiIndex object has code to explicity check the sort depth. Thus, if you try to index at a depth at which the index is not sorted, it will raise an exception. Here is a concrete example to illustrate this:
In [561]: tuples = [(a, a), (a, b), (b, a), (b, b)] In [562]: idx = MultiIndex.from_tuples(tuples) In [563]: idx.lexsort_depth Out[563]: 2 In [564]: reordered = idx[[1, 0, 3, 2]] In [565]: reordered.lexsort_depth Out[565]: 1 In [566]: s = Series(randn(4), index=reordered) In [567]: s.ix[a:a] Out[567]: a b -1.099248 a 0.255269

However:
>>> s.ix[(a, b):(b, a)] Exception: MultiIndex lexsort depth 1, key was length 2

7.4.9 Swapping levels with swaplevel

The swaplevel function can switch the order of two levels:
In [568]: df[:5] Out[568]: 0 1 one y 0.306389 -2.290613 x -1.134623 -1.561819 zero y -0.260838 0.281957 x 1.523962 -0.902937 In [569]: df[:5].swaplevel(0, 1, axis=0)

7.4. Hierarchical indexing (MultiIndex)

89

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[569]: y x y x 0 1 one 0.306389 -2.290613 one -1.134623 -1.561819 zero -0.260838 0.281957 zero 1.523962 -0.902937

7.4.10 Reordering levels with reorder_levels

The reorder_levels function generalizes the swaplevel function, allowing you to permute the hierarchical index levels in one step:
In [570]: df[:5].reorder_levels([1,0], axis=0) Out[570]: 0 1 y one 0.306389 -2.290613 x one -1.134623 -1.561819 y zero -0.260838 0.281957 x zero 1.523962 -0.902937

7.4.11 Some gory internal details

Internally, the MultiIndex consists of a few things: the levels, the integer labels, and the level names:
In [571]: index Out[571]: MultiIndex([(bar, one), (bar, two), (baz, one), (baz, two), (foo, one), (foo, two), (qux, one), (qux, two)], dtype=object) In [572]: index.levels Out[572]: [Index([bar, baz, foo, qux], dtype=object), Index([one, two], dtype=object)] In [573]: index.labels Out[573]: [array([0, 0, 1, 1, 2, 2, 3, 3], dtype=int32), array([0, 1, 0, 1, 0, 1, 0, 1], dtype=int32)] In [574]: index.names Out[574]: [first, second]

You can probably guess that the labels determine which unique element is identied with that location at each layer of the index. Its important to note that sortedness is determined solely from the integer labels and does not check (or care) whether the levels themselves are sorted. Fortunately, the constructors from_tuples and from_arrays ensure that this is true, but if you compute the levels and labels yourself, please be careful.

7.5 Adding an index to an existing DataFrame

Occasionally you will load or create a data set into a DataFrame and want to add an index after youve already done so. There are a couple of different ways.

90

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

7.5.1 Add an index using DataFrame columns

DataFrame has a set_index method which takes a column name (for a regular Index) or a list of column names (for a MultiIndex), to create a new, indexed DataFrame:
In [575]: data Out[575]: a b c 0 bar one z 1 bar two y 2 foo one x 3 foo two w

d 1 2 3 4

In [576]: indexed1 = data.set_index(c) In [577]: indexed1 Out[577]: a b d c z bar one 1 y bar two 2 x foo one 3 w foo two 4 In [578]: indexed2 = data.set_index([a, b]) In [579]: indexed2 Out[579]: c d a b bar one z 1 two y 2 foo one x 3 two w 4

Other options in set_index allow you not drop the index columns or to add the index in-place (without creating a new object):
In [580]: data.set_index(c, drop=False) Out[580]: a b c d c z bar one z 1 y bar two y 2 x foo one x 3 w foo two w 4 In [581]: df = data.set_index([a, b], inplace=True) In [582]: data Out[582]: c d a b bar one z 1 two y 2 foo one x 3 two w 4

7.5. Adding an index to an existing DataFrame

91

pandas: powerful Python data analysis toolkit, Release 0.7.1

7.5.2 Remove / reset the index, reset_index

As a convenience, there is a new function on DataFrame called reset_index which transfers the index values into the DataFrames columns and sets a simple integer index. This is the inverse operation to set_index
In [583]: df Out[583]: c d a b bar one z 1 two y 2 foo one x 3 two w 4 In [584]: df.reset_index() Out[584]: a b c d 0 bar one z 1 1 bar two y 2 2 foo one x 3 3 foo two w 4

The output is more similar to a SQL table or a record array. The names for the columns derived from the index are the ones stored in the names attribute. reset_index takes an optional parameter drop which if true simply discards the index, instead of putting index values in the DataFrames columns. Note: The reset_index method used to be called delevel which is now deprecated.

7.5.3 Adding an ad hoc index

If you create an index yourself, you can just assign it to the index eld:
df.index = index

7.6 Indexing internal details

Note: The following is largely relevant for those actually working on the pandas codebase. And the source code is still the best place to look at the specics of how things are implemented. In pandas there are a few objects implemented which can serve as valid containers for the axis labels: Index: the generic ordered set object, an ndarray of object dtype assuming nothing about its contents. The labels must be hashable (and likely immutable) and unique. Populates a dict of label to location in Cython to do O(1) lookups. Int64Index: a version of Index highly optimized for 64-bit integer data, such as time stamps MultiIndex: the standard hierarchical index object DateRange: xed frequency date range generated from a time rule or DateOffset. An ndarray of Python datetime objects

92

Chapter 7. Indexing and selecting data

pandas: powerful Python data analysis toolkit, Release 0.7.1

The motivation for having an Index class in the rst place was to enable different implementations of indexing. This means that its possible for you, the user, to implement a custom Index subclass that may be better suited to a particular application than the ones provided in pandas. For example, we plan to add a more efcient datetime index which leverages the new numpy.datetime64 dtype in the relatively near future. From an internal implementation point of view, the relevant methods that an Index must dene are one or more of the following (depending on how incompatible the new object internals are with the Index functions): get_loc: returns an indexer (an integer, or in some cases a slice object) for a label slice_locs: returns the range to slice between two labels get_indexer: Computes the indexing vector for reindexing / data alignment purposes. See the source / docstrings for more on this reindex: Does any pre-conversion of the input index then calls get_indexer union, intersection: computes the union or intersection of two Index objects insert: Inserts a new label into an Index, yielding a new object delete: Delete a label, yielding a new object drop: Deletes a set of labels take: Analogous to ndarray.take

7.6. Indexing internal details

93

pandas: powerful Python data analysis toolkit, Release 0.7.1

94

Chapter 7. Indexing and selecting data

CHAPTER

EIGHT

COMPUTATIONAL TOOLS
8.1 Statistical functions
8.1.1 Covariance
The Series object has a method cov to compute covariance between series (excluding NA/null values).
In [157]: s1 = Series(randn(1000)) In [158]: s2 = Series(randn(1000)) In [159]: s1.cov(s2) Out[159]: 0.019465636696791695

Analogously, DataFrame has a method cov to compute pairwise covariances among the series in the DataFrame, also excluding NA/null values.
In [160]: frame = DataFrame(randn(1000, 5), columns=[a, b, c, d, e]) In [161]: frame.cov() Out[161]: a b c d e a 0.953751 -0.029550 -0.006415 0.001020 -0.004134 b -0.029550 0.997223 -0.044276 0.005967 0.044884 c -0.006415 -0.044276 1.050236 0.077775 0.010642 d 0.001020 0.005967 0.077775 0.998485 -0.007345 e -0.004134 0.044884 0.010642 -0.007345 1.025446

8.1.2 Correlation
Several methods for computing correlations are provided. Several kinds of correlation methods are provided: Method name pearson (default) kendall spearman Description Standard correlation coefcient Kendall Tau correlation coefcient Spearman rank correlation coefcient

All of these are currently computed using pairwise complete observations.

In [162]: frame = DataFrame(randn(1000, 5), columns=[a, b, c, d, e]) In [163]: frame.ix[::2] = np.nan

95

pandas: powerful Python data analysis toolkit, Release 0.7.1

# Series with Series In [164]: frame[a].corr(frame[b]) Out[164]: 0.013306883832198543 In [165]: frame[a].corr(frame[b], method=spearman) Out[165]: 0.022530330121320486 # Pairwise correlation of DataFrame columns In [166]: frame.corr() Out[166]: a b c d e a 1.000000 0.013307 -0.037801 -0.021905 0.001165 b 0.013307 1.000000 -0.017259 0.079246 -0.043606 c -0.037801 -0.017259 1.000000 0.061657 0.078945 d -0.021905 0.079246 0.061657 1.000000 -0.036978 e 0.001165 -0.043606 0.078945 -0.036978 1.000000

Note that non-numeric columns will be automatically excluded from the correlation calculation. A related method corrwith is implemented on DataFrame to compute the correlation between like-labeled Series contained in different DataFrame objects.
In [167]: index = [a, b, c, d, e] In [168]: columns = [one, two, three, four] In [169]: df1 = DataFrame(randn(5, 4), index=index, columns=columns) In [170]: df2 = DataFrame(randn(4, 4), index=index[:4], columns=columns) In [171]: df1.corrwith(df2) Out[171]: one 0.344149 two 0.837438 three 0.458904 four 0.712401 In [172]: df2.corrwith(df1, axis=1) Out[172]: a 0.404019 b 0.772204 c 0.420390 d -0.142959 e NaN

8.1.3 Data ranking

The rank method produces a data ranking with ties being assigned the mean of the ranks for the group:
In [173]: s = Series(np.random.randn(5), index=list(abcde)) In [174]: s[d] = s[b] # so theres a tie In [175]: s.rank() Out[175]: a 2.0 b 3.5

96

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

c d e

1.0 3.5 5.0

rank is also a DataFrame method and can rank either the rows (axis=0) or the columns (axis=1). NaN values are excluded from the ranking.
In [176]: df = DataFrame(np.random.randn(10, 6)) In [177]: df[4] = df[2][:5] # some ties In [178]: df Out[178]: 0 1 0 0.106333 0.712162 1 -1.301869 0.612432 2 -0.899627 0.822023 3 -0.522705 -1.473680 4 0.733147 0.415881 5 0.995001 -1.399355 6 -0.779714 -0.226893 7 -0.635495 -0.621647 8 0.085011 -0.459422 9 -0.557052 0.775425 In [179]: df.rank(1) Out[179]: 0 1 2 3 4 0 3 4 1.5 5 1.5 1 1 6 3.5 5 3.5 2 1 3 5.5 4 5.5 3 5 3 1.5 6 1.5 4 5 4 1.5 6 1.5 5 5 2 3.0 1 NaN 6 1 4 5.0 3 NaN 7 2 3 5.0 4 NaN 8 4 3 2.0 1 NaN 9 2 5 3.0 4 NaN

2 -0.351275 -0.577677 1.506319 -1.726800 -0.026973 0.082244 0.956567 0.406259 -1.660917 0.003794

3 1.176287 0.124709 0.998896 1.555343 0.999488 -1.521795 -0.443664 -0.279002 -1.913019 0.555351

4 -0.351275 -0.577677 1.506319 -1.726800 -0.026973 NaN NaN NaN NaN NaN

5 1.741787 -1.068084 0.259080 -1.411978 0.082219 0.416180 -0.610675 -1.153000 0.833479 -1.169977

5 6 2 2 4 3 4 2 1 5 1

Note: These methods are signicantly faster (around 10-20x) than scipy.stats.rankdata.

8.2 Moving (rolling) statistics / moments

For working with time series data, a number of functions are provided for computing common moving or rolling statistics. Among these are count, sum, mean, median, correlation, variance, covariance, standard deviation, skewness, and kurtosis. All of these methods are in the pandas namespace, but otherwise they can be found in pandas.stats.moments.

8.2. Moving (rolling) statistics / moments

97

pandas: powerful Python data analysis toolkit, Release 0.7.1

Function rolling_count rolling_sum rolling_mean rolling_median rolling_min rolling_max rolling_std rolling_var rolling_skew rolling_kurt rolling_quantile rolling_apply rolling_cov rolling_corr rolling_corr_pairwise

Description Number of non-null observations Sum of values Mean of values Arithmetic median of values Minimum Maximum Unbiased standard deviation Unbiased variance Unbiased skewness (3rd moment) Unbiased kurtosis (4th moment) Sample quantile (value at %) Generic apply Unbiased covariance (binary) Correlation (binary) Pairwise correlation of DataFrame columns

Generally these methods all have the same interface. The binary operators (e.g. rolling_corr) take two Series or DataFrames. Otherwise, they all accept the following arguments: window: size of moving window min_periods: threshold of non-null data points to require (otherwise result is NA) time_rule: optionally specify a time rule to pre-conform the data to These functions can be applied to ndarrays or Series objects:
In [180]: ts = Series(randn(1000), index=DateRange(1/1/2000, periods=1000)) In [181]: ts = ts.cumsum() In [182]: ts.plot(style=k--) Out[182]: <matplotlib.axes.AxesSubplot at 0x10e583e50> In [183]: rolling_mean(ts, 60).plot(style=k) Out[183]: <matplotlib.axes.AxesSubplot at 0x10e583e50>

98

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

They can also be applied to DataFrame objects. This is really just syntactic sugar for applying the moving window operator to all of the DataFrames columns:
In [184]: df = DataFrame(randn(1000, 4), index=ts.index, .....: columns=[A, B, C, D]) In [185]: df = df.cumsum() In [186]: rolling_sum(df, 60).plot(subplots=True) Out[186]: array([Axes(0.125,0.747826;0.775x0.152174), Axes(0.125,0.565217;0.775x0.152174), Axes(0.125,0.382609;0.775x0.152174), Axes(0.125,0.2;0.775x0.152174)], dtype=object)

8.2.1 Binary rolling moments

rolling_cov and rolling_corr can compute moving window statistics about two Series or any combination of DataFrame/Series or DataFrame/DataFrame. Here is the behavior in each case: two Series: compute the statistic for the pairing DataFrame/Series: compute the statistics for each column of the DataFrame with the passed Series, thus returning a DataFrame DataFrame/DataFrame: compute statistic for matching column names, returning a DataFrame For example:
In [187]: df2 = df[:20] In [188]: rolling_corr(df2, df2[B], window=5) Out[188]: A B C D 2000-01-03 NaN NaN NaN NaN 2000-01-04 NaN NaN NaN NaN 2000-01-05 NaN NaN NaN NaN 2000-01-06 NaN NaN NaN NaN

8.2. Moving (rolling) statistics / moments

99

pandas: powerful Python data analysis toolkit, Release 0.7.1

2000-01-07 0.806980 2000-01-10 0.689915 2000-01-11 0.211679 2000-01-12 0.286270 2000-01-13 -0.565249 2000-01-14 0.295310 2000-01-17 0.041252 2000-01-18 0.205705 2000-01-19 0.326449 2000-01-20 0.120893 2000-01-21 0.680531 2000-01-24 0.643667 2000-01-25 0.703188 2000-01-26 0.065322 2000-01-27 -0.429914 2000-01-28 -0.387498

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

-0.911973 -0.609054 -0.383565 0.104075 0.039148 0.501143 0.868636 0.917778 0.933352 0.409255 -0.192045 -0.588676 -0.746130 -0.209789 -0.100807 0.512321

-0.747745 -0.680394 -0.164879 0.345844 0.333921 -0.524100 -0.577590 -0.819271 -0.882750 -0.795062 -0.349044 0.473287 0.714265 0.635360 0.266005 0.592033

8.2.2 Computing rolling pairwise correlations

In nancial data analysis and other elds its common to compute correlation matrices for a collection of time series. More difcult is to compute a moving-window correlation matrix. This can be done using the rolling_corr_pairwise function, which yields a Panel whose items are the dates in question:
In [189]: correls = rolling_corr_pairwise(df, 50) In [190]: correls[df.index[-50]] Out[190]: A B C D A 1.000000 -0.177708 -0.253742 0.303872 B -0.177708 1.000000 -0.085484 0.008572 C -0.253742 -0.085484 1.000000 -0.769233 D 0.303872 0.008572 -0.769233 1.000000

You can efciently retrieve the time series of correlations between two columns using ix indexing:
In [191]: correls.ix[:, A, C].plot() Out[191]: <matplotlib.axes.AxesSubplot at 0x10b800250>

100

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

8.3 Exponentially weighted moment functions

A related set of functions are exponentially weighted versions of many of the above statistics. A number of EW (exponentially weighted) functions are provided using the blending method. For example, where yt is the result and xt the input, we compute an exponentially weighted moving average as yt = (1 )yt1 + xt One must have 0 < 1, but rather than pass directly, its easier to think about either the span or center of mass (com) of an EW moment: =
2 s+1 , s 1 c+1 , c

= span = center of mass

You can pass one or the other to these functions but not both. Span corresponds to what is commonly called a 20day EW moving average for example. Center of mass has a more physical interpretation. For example, span = 20 corresponds to com = 9.5. Here is the list of functions available: Function ewma ewvar ewstd ewmcorr ewmcov Description EW moving average EW moving variance EW moving standard deviation EW moving correlation EW moving covariance

Here are an example for a univariate time series:

In [192]: plt.close(all) In [193]: ts.plot(style=k--) Out[193]: <matplotlib.axes.AxesSubplot at 0x10b811e90> In [194]: ewma(ts, span=20).plot(style=k) Out[194]: <matplotlib.axes.AxesSubplot at 0x10b811e90>

8.3. Exponentially weighted moment functions

101

pandas: powerful Python data analysis toolkit, Release 0.7.1

Note: The EW functions perform a standard adjustment to the initial observations whereby if there are fewer observations than called for in the span, those observations are reweighted accordingly.

8.4 Linear and panel regression

Note: We plan to move this functionality to statsmodels for the next release. Some of the result attributes may change names in order to foster naming consistency with the rest of statsmodels. We will provide every effort to provide compatibility with older versions of pandas, however. We have implemented a very fast set of moving-window linear regression classes in pandas. Two different types of regressions are supported: Standard ordinary least squares (OLS) multiple regression Multiple regression (OLS-based) on panel data including with xed-effects (also known as entity or individual effects) or time-effects. Both kinds of linear models are accessed through the ols function in the pandas namespace. They all take the following arguments to specify either a static (full sample) or dynamic (moving window) regression: window_type: full sample (default), expanding, or rolling window: size of the moving window in the window_type=rolling case. If window is specied, window_type will be automatically set to rolling min_periods: minimum number of time periods to require to compute the regression coefcients Generally speaking, the ols works by being given a y (response) object and an x (predictors) object. These can take many forms: y: a Series, ndarray, or DataFrame (panel model) x: Series, DataFrame, dict of Series, dict of DataFrame or Panel

102

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

Based on the types of y and x, the model will be inferred to either a panel model or a regular linear model. If the y variable is a DataFrame, the result will be a panel model. In this case, the x variable must either be a Panel, or a dict of DataFrame (which will be coerced into a Panel).

8.4.1 Standard OLS regression

Lets pull in some sample data:
In [195]: from pandas.io.data import DataReader In [196]: symbols = [MSFT, GOOG, AAPL] In [197]: data = dict((sym, DataReader(sym, "yahoo")) .....: for sym in symbols) In [198]: panel = Panel(data).swapaxes(items, minor) In [199]: close_px = panel[Close] # convert closing prices to returns In [200]: rets = close_px / close_px.shift(1) - 1 In [201]: rets.info() <class pandas.core.frame.DataFrame> Index: 252 entries, 2011-03-01 00:00:00 to 2012-02-28 00:00:00 Data columns: AAPL 251 non-null values GOOG 251 non-null values MSFT 251 non-null values dtypes: float64(3)

Lets do a static regression of AAPL returns on GOOG returns:

In [202]: model = ols(y=rets[AAPL], x=rets.ix[:, [GOOG]]) In [203]: model Out[203]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <GOOG> + <intercept> Number of Observations: 251 Number of Degrees of Freedom: 2 R-squared: 0.3441 Adj R-squared: 0.3415 Rmse: 0.0136 F-stat (1, 249): 130.6426, p-value: 0.0000 Degrees of Freedom: model 1, resid 249 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------GOOG 0.5059 0.0443 11.43 0.0000 0.4192 0.5927 intercept 0.0017 0.0009 1.97 0.0505 0.0000 0.0034 ---------------------------------End of Summary--------------------------------In [204]: model.beta Out[204]: GOOG 0.505928 intercept 0.001690

8.4. Linear and panel regression

103

pandas: powerful Python data analysis toolkit, Release 0.7.1

If we had passed a Series instead of a DataFrame with the single GOOG column, the model would have assigned the generic name x to the sole right-hand side variable. We can do a moving window regression to see how the relationship changes over time:
In [205]: model = ols(y=rets[AAPL], x=rets.ix[:, [GOOG]], .....: window=250) # just plot the coefficient for GOOG In [206]: model.beta[GOOG].plot() Out[206]: <matplotlib.axes.AxesSubplot at 0x1105c8fd0>

It looks like there are some outliers rolling in and out of the window in the above regression, inuencing the results. We could perform a simple winsorization at the 3 STD level to trim the impact of outliers:
In [207]: winz = rets.copy() In [208]: std_1year = rolling_std(rets, 250, min_periods=20) # cap at 3 * 1 year standard deviation In [209]: cap_level = 3 * np.sign(winz) * std_1year In [210]: winz[np.abs(winz) > 3 * std_1year] = cap_level In [211]: winz_model = ols(y=winz[AAPL], x=winz.ix[:, [GOOG]], .....: window=250) In [212]: model.beta[GOOG].plot(label="With outliers") Out[212]: <matplotlib.axes.AxesSubplot at 0x10fb56450> In [213]: winz_model.beta[GOOG].plot(label="Winsorized"); plt.legend(loc=best) Out[213]: <matplotlib.legend.Legend at 0x10fd02550>

104

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

So in this simple example we see the impact of winsorization is actually quite signicant. Note the correlation after winsorization remains high:
In [214]: winz.corrwith(rets) Out[214]: AAPL 0.997979 GOOG 0.970836 MSFT 0.998111

Multiple regressions can be run by passing a DataFrame with multiple columns for the predictors x:
In [215]: ols(y=winz[AAPL], x=winz.drop([AAPL], axis=1)) Out[215]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <GOOG> + <MSFT> + <intercept> Number of Observations: 251 Number of Degrees of Freedom: 3 R-squared: 0.4532 Adj R-squared: 0.4488 Rmse: 0.0121 F-stat (2, 248): 102.7814, p-value: 0.0000 Degrees of Freedom: model 2, resid 248 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------GOOG 0.4581 0.0542 8.45 0.0000 0.3519 0.5643 MSFT 0.2956 0.0632 4.68 0.0000 0.1718 0.4194 intercept 0.0014 0.0008 1.87 0.0631 -0.0001 0.0029 ---------------------------------End of Summary---------------------------------

8.4.2 Panel regression

Weve implemented moving window panel regression on potentially unbalanced panel data (see this article if this means nothing to you). Suppose we wanted to model the relationship between the magnitude of the daily return and 8.4. Linear and panel regression 105

pandas: powerful Python data analysis toolkit, Release 0.7.1

trading volume among a group of stocks, and we want to pool all the data together to run one big regression. This is actually quite easy:
# make the units somewhat comparable In [216]: volume = panel[Volume] / 1e8 In [217]: model = ols(y=volume, x={return : np.abs(rets)}) In [218]: model Out[218]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <return> + <intercept> Number of Observations: 753 Number of Degrees of Freedom: 2 R-squared: 0.0182 Adj R-squared: 0.0168 Rmse: 0.2859 F-stat (1, 751): 13.8849, p-value: 0.0002 Degrees of Freedom: model 1, resid 751 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------return 3.2352 0.8682 3.73 0.0002 1.5335 4.9369 intercept 0.2273 0.0150 15.19 0.0000 0.1980 0.2567 ---------------------------------End of Summary---------------------------------

In a panel model, we can insert dummy (0-1) variables for the entities involved (here, each of the stocks) to account the a entity-specic effect (intercept):
In [219]: fe_model = ols(y=volume, x={return : np.abs(rets)}, .....: entity_effects=True) In [220]: fe_model Out[220]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <return> + <FE_GOOG> + <FE_MSFT> + <intercept> Number of Observations: 753 Number of Degrees of Freedom: 4 R-squared: 0.7365 Adj R-squared: 0.7355 Rmse: 0.1483 F-stat (3, 749): 697.8591, p-value: 0.0000 Degrees of Freedom: model 3, resid 749 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------return 4.4688 0.4512 9.90 0.0000 3.5845 5.3531 FE_GOOG -0.1416 0.0132 -10.69 0.0000 -0.1675 -0.1156 FE_MSFT 0.4335 0.0133 32.71 0.0000 0.4076 0.4595 intercept 0.1147 0.0110 10.44 0.0000 0.0932 0.1363 ---------------------------------End of Summary---------------------------------

Because we ran the regression with an intercept, one of the dummy variables must be dropped or the design matrix will not be full rank. If we do not use an intercept, all of the dummy variables will be included:
In [221]: fe_model = ols(y=volume, x={return : np.abs(rets)}, .....: entity_effects=True, intercept=False) In [222]: fe_model

106

Chapter 8. Computational tools

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[222]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <return> + <FE_AAPL> + <FE_GOOG> + <FE_MSFT> Number of Observations: 753 Number of Degrees of Freedom: 4 R-squared: 0.7365 Adj R-squared: 0.7355 Rmse: 0.1483 F-stat (4, 749): 697.8591, p-value: 0.0000 Degrees of Freedom: model 3, resid 749 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------return 4.4688 0.4512 9.90 0.0000 3.5845 5.3531 FE_AAPL 0.1147 0.0110 10.44 0.0000 0.0932 0.1363 FE_GOOG -0.0269 0.0111 -2.43 0.0154 -0.0485 -0.0052 FE_MSFT 0.5483 0.0107 51.37 0.0000 0.5274 0.5692 ---------------------------------End of Summary---------------------------------

We can also include time effects, which demeans the data cross-sectionally at each point in time (equivalent to including dummy variables for each date). More mathematical care must be taken to properly compute the standard errors in this case:
In [223]: te_model = ols(y=volume, x={return : np.abs(rets)}, .....: time_effects=True, entity_effects=True) In [224]: te_model Out[224]: -------------------------Summary of Regression Analysis------------------------Formula: Y ~ <return> + <FE_GOOG> + <FE_MSFT> Number of Observations: 753 Number of Degrees of Freedom: 254 R-squared: 0.8122 Adj R-squared: 0.7170 Rmse: 0.1436 F-stat (3, 499): 8.5306, p-value: 0.0000 Degrees of Freedom: model 253, resid 499 -----------------------Summary of Estimated Coefficients-----------------------Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------return 3.7138 0.6877 5.40 0.0000 2.3660 5.0617 FE_GOOG -0.1414 0.0128 -11.03 0.0000 -0.1665 -0.1162 FE_MSFT 0.4325 0.0129 33.66 0.0000 0.4073 0.4577 ---------------------------------End of Summary---------------------------------

Here the intercept (the mean term) is dropped by default because it will be 0 according to the model assumptions, having subtracted off the group means.

8.4.3 Result elds and tests

Well leave it to the user to explore the docstrings and source, especially as well be moving this code into statsmodels in the near future.

8.4. Linear and panel regression

107

pandas: powerful Python data analysis toolkit, Release 0.7.1

108

Chapter 8. Computational tools

CHAPTER

NINE

WORKING WITH MISSING DATA

In this section, we will discuss missing (also referred to as NA) values in pandas. Note: The choice of using NaN internally to denote missing data was largely for simplicity and performance reasons. It differs from the MaskedArray approach of, for example, scikits.timeseries. We are hopeful that NumPy will soon be able to provide a native NA type solution (similar to R) performant enough to be used in pandas.

9.1 Missing data basics

9.1.1 When / why does data become missing?
Some might quibble over our usage of missing. By missing we simply mean null or not present for whatever reason. Many data sets simply arrive with missing data, either because it exists and was not collected or it never existed. For example, in a collection of nancial time series, some of the time series might start on different dates. Thus, values prior to the start date would generally be marked as missing. In pandas, one of the most common ways that missing data is introduced into a data set is by reindexing. For example
In [716]: df = DataFrame(randn(5, 3), index=[a, c, e, f, h], .....: columns=[one, two, three]) In [717]: df[four] = bar In [718]: df[five] = df[one] > 0 In [719]: df Out[719]: one two a 0.059117 1.138469 c -0.280853 0.025653 e 0.863937 0.252462 f 1.053202 -2.338595 h -2.359958 -1.157886

three four -2.400634 bar -1.386071 bar 1.500571 bar -0.374279 bar -0.551865 bar

five True False True True False

In [720]: df2 = df.reindex([a, b, c, d, e, f, g, h]) In [721]: df2 Out[721]: one two three four a 0.059117 1.138469 -2.400634 bar

five True

109

pandas: powerful Python data analysis toolkit, Release 0.7.1

b NaN NaN NaN c -0.280853 0.025653 -1.386071 d NaN NaN NaN e 0.863937 0.252462 1.500571 f 1.053202 -2.338595 -0.374279 g NaN NaN NaN h -2.359958 -1.157886 -0.551865

NaN bar NaN bar bar NaN bar

NaN False NaN True True NaN False

9.1.2 Values considered missing

As data comes in many shapes and forms, pandas aims to be exible with regard to handling missing data. While NaN is the default missing value marker for reasons of computational speed and convenience, we need to be able to easily detect this value with data of different types: oating point, integer, boolean, and general object. In many cases, however, the Python None will arise and we wish to also consider that missing or null. Lastly, for legacy reasons inf and -inf are also considered to be null in computations. Since in NumPy divide-by-zero generates inf or -inf and not NaN, I think you will nd this is a worthwhile trade-off (Zen of Python: practicality beats purity). To make detecting missing values easier (and across different array dtypes), pandas provides the isnull() and notnull() functions, which are also methods on Series objects:
In [722]: df2[one] Out[722]: a 0.059117 b NaN c -0.280853 d NaN e 0.863937 f 1.053202 g NaN h -2.359958 Name: one In [723]: isnull(df2[one]) Out[723]: a False b True c False d True e False f False g True h False Name: one In [724]: df2[four].notnull() Out[724]: a True b False c True d False e True f True g False h True

Summary: NaN, inf, -inf, and None (in object arrays) are all considered missing by the isnull and notnull functions.

110

Chapter 9. Working with missing data

pandas: powerful Python data analysis toolkit, Release 0.7.1

9.2 Calculations with missing data

Missing values propagate naturally through arithmetic operations between pandas objects.
In [725]: a Out[725]: one a 0.059117 b 0.059117 c -0.280853 d -0.280853 e 0.863937 In [726]: b Out[726]: one a 0.059117 b NaN c -0.280853 d NaN e 0.863937

two 1.138469 1.138469 0.025653 0.025653 0.252462

two three 1.138469 -2.400634 NaN NaN 0.025653 -1.386071 NaN NaN 0.252462 1.500571

In [727]: a + b Out[727]: one three a 0.118234 NaN b NaN NaN c -0.561707 NaN d NaN NaN e 1.727874 NaN

two 2.276938 NaN 0.051306 NaN 0.504923

The descriptive statistics and computational methods discussed in the data structure overview (and listed here and here) are all written to account for missing data. For example: When summing data, NA (missing) values will be treated as zero If the data are all NA, the result will be NA Methods like cumsum and cumprod ignore NA values, but preserve them in the resulting arrays
In [728]: df Out[728]: one two a 0.059117 1.138469 b NaN NaN c -0.280853 0.025653 d NaN NaN e 0.863937 0.252462 f 1.053202 -2.338595 g NaN NaN h -2.359958 -1.157886

three -2.400634 NaN -1.386071 NaN 1.500571 -0.374279 NaN -0.551865

In [729]: df[one].sum() Out[729]: -0.66455558290247652 In [730]: df.mean(1) Out[730]: a -0.401016 b NaN

9.2. Calculations with missing data

111

pandas: powerful Python data analysis toolkit, Release 0.7.1

c d e f g h

-0.547090 NaN 0.872323 -0.553224 NaN -1.356570

In [731]: df.cumsum() Out[731]: one two a 0.059117 1.138469 b NaN NaN c -0.221736 1.164122 d NaN NaN e 0.642200 1.416584 f 1.695403 -0.922011 g NaN NaN h -0.664556 -2.079897

three -2.400634 NaN -3.786705 NaN -2.286134 -2.660413 NaN -3.212278

9.2.1 NA values in GroupBy

NA groups in GroupBy are automatically excluded. This behavior is consistent with R, for example.

9.3 Cleaning / lling missing data

pandas objects are equipped with various data manipulation methods for dealing with missing data.

9.3.1 Filling missing values: llna

The llna function can ll in NA values with non-null data in a couple of ways, which we illustrate: Replace NA with a scalar value
In [732]: df2 Out[732]: one two a 0.059117 1.138469 b NaN NaN c -0.280853 0.025653 d NaN NaN e 0.863937 0.252462 f 1.053202 -2.338595 g NaN NaN h -2.359958 -1.157886

three four -2.400634 bar NaN NaN -1.386071 bar NaN NaN 1.500571 bar -0.374279 bar NaN NaN -0.551865 bar

five True NaN False NaN True True NaN False

In [733]: df2.fillna(0) Out[733]: one two three four a 0.059117 1.138469 -2.400634 bar b 0.000000 0.000000 0.000000 0 c -0.280853 0.025653 -1.386071 bar d 0.000000 0.000000 0.000000 0 e 0.863937 0.252462 1.500571 bar f 1.053202 -2.338595 -0.374279 bar

five True 0 False 0 True True

112

Chapter 9. Working with missing data

pandas: powerful Python data analysis toolkit, Release 0.7.1

g 0.000000 0.000000 0.000000 h -2.359958 -1.157886 -0.551865

0 bar

0 False

In [734]: df2[four].fillna(missing) Out[734]: a bar b missing c bar d missing e bar f bar g missing h bar Name: four

Fill gaps forward or backward Using the same lling arguments as reindexing, we can propagate non-null values forward or backward:
In [735]: df Out[735]: one two a 0.059117 1.138469 b NaN NaN c -0.280853 0.025653 d NaN NaN e 0.863937 0.252462 f 1.053202 -2.338595 g NaN NaN h -2.359958 -1.157886

three -2.400634 NaN -1.386071 NaN 1.500571 -0.374279 NaN -0.551865

In [736]: df.fillna(method=pad) Out[736]: one two three a 0.059117 1.138469 -2.400634 b 0.059117 1.138469 -2.400634 c -0.280853 0.025653 -1.386071 d -0.280853 0.025653 -1.386071 e 0.863937 0.252462 1.500571 f 1.053202 -2.338595 -0.374279 g 1.053202 -2.338595 -0.374279 h -2.359958 -1.157886 -0.551865

To remind you, these are the available lling methods: Method pad / fll bll / backll Action Fill values forward Fill values backward

With time series data, using pad/fll is extremely common so that the last known value is available at every time point.

9.3.2 Dropping axis labels with missing data: dropna

You may wish to simply exclude labels from a data set which refer to missing data. To do this, use the dropna method:
In [737]: df Out[737]:

9.3. Cleaning / lling missing data

113

pandas: powerful Python data analysis toolkit, Release 0.7.1

one two three a 0.059117 1.138469 -2.400634 b NaN 0.000000 0.000000 c -0.280853 0.025653 -1.386071 d NaN 0.000000 0.000000 e 0.863937 0.252462 1.500571 f 1.053202 -2.338595 -0.374279 g NaN 0.000000 0.000000 h -2.359958 -1.157886 -0.551865 In [738]: df.dropna(axis=0) Out[738]: one two three a 0.059117 1.138469 -2.400634 c -0.280853 0.025653 -1.386071 e 0.863937 0.252462 1.500571 f 1.053202 -2.338595 -0.374279 h -2.359958 -1.157886 -0.551865 In [739]: df.dropna(axis=1) Out[739]: two three a 1.138469 -2.400634 b 0.000000 0.000000 c 0.025653 -1.386071 d 0.000000 0.000000 e 0.252462 1.500571 f -2.338595 -0.374279 g 0.000000 0.000000 h -1.157886 -0.551865 In [740]: df[one].dropna() Out[740]: a 0.059117 c -0.280853 e 0.863937 f 1.053202 h -2.359958 Name: one

dropna is presently only implemented for Series and DataFrame, but will be eventually added to Panel. Series.dropna is a simpler method as it only has one axis to consider. DataFrame.dropna has considerably more options, which can be examined in the API.

9.3.3 Interpolation
A basic linear interpolate method has been implemented on Series with intended use for time series data. There has not been a great deal of demand for interpolation methods outside of the lling methods described above.
In [741]: fig, axes = plt.subplots(ncols=2, figsize=(8, 4)) In [742]: ts.plot(ax=axes[0]) Out[742]: <matplotlib.axes.AxesSubplot at 0x1122a7910> In [743]: ts.interpolate().plot(ax=axes[1]) Out[743]: <matplotlib.axes.AxesSubplot at 0x10c160210>

114

Chapter 9. Working with missing data

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [744]: axes[0].set_title(Not interpolated) Out[744]: <matplotlib.text.Text at 0x10c2ad490> In [745]: axes[1].set_title(Interpolated) Out[745]: <matplotlib.text.Text at 0x11485d310> In [746]: plt.close(all)

9.4 Missing data casting rules and indexing

While pandas supports storing arrays of integer and boolean type, these types are not capable of storing missing data. Until we can switch to using a native NA type in NumPy, weve established some casting rules when reindexing will cause missing data to be introduced into, say, a Series or DataFrame. Here they are: data type integer boolean oat object For example:
In [747]: s = Series(randn(5), index=[0, 2, 4, 6, 7]) In [748]: s > 0 Out[748]: 0 False 2 True 4 True 6 True 7 True In [749]: (s > 0).dtype Out[749]: dtype(bool)

Cast to oat object no cast no cast

9.4. Missing data casting rules and indexing

115

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [750]: crit = (s > 0).reindex(range(8)) In [751]: crit Out[751]: 0 False 1 NaN 2 True 3 NaN 4 True 5 NaN 6 True 7 True In [752]: crit.dtype Out[752]: dtype(object)

Ordinarily NumPy will complain if you try to use an object array (even if it contains boolean values) instead of a boolean array to get or set values from an ndarray (e.g. selecting values based on some criteria). If a boolean vector contains NAs, an exception will be generated:
In [753]: reindexed = s.reindex(range(8)).fillna(0) In [754]: reindexed[crit] --------------------------------------------------------------------------ValueError Traceback (most recent call last) /Users/wesm/code/pandas/doc/<ipython-input-754-2da204ed1ac7> in <module>() ----> 1 reindexed[crit] /Users/wesm/code/pandas/pandas/core/series.py in __getitem__(self, key) 388 # special handling of boolean data with NAs stored in object 389 # arrays. Since we cant represent NA with dtype=bool --> 390 if _is_bool_indexer(key): 391 key = self._check_bool_indexer(key) 392 key = np.asarray(key, dtype=bool) /Users/wesm/code/pandas/pandas/core/common.py in _is_bool_indexer(key) 321 if not lib.is_bool_array(key): 322 if isnull(key).any(): --> 323 raise ValueError(cannot index with vector containing 324 NA / NaN values) 325 return False ValueError: cannot index with vector containing NA / NaN values

However, these can be lled in using llna and it will work ne:
In [755]: reindexed[crit.fillna(False)] Out[755]: 2 1.314232 4 0.690579 6 0.995761 7 2.396780 In [756]: reindexed[crit.fillna(True)] Out[756]: 1 0.000000 2 1.314232 3 0.000000 4 0.690579 5 0.000000 6 0.995761 7 2.396780

116

Chapter 9. Working with missing data

CHAPTER

TEN

GROUP BY: SPLIT-APPLY-COMBINE

By group by we are refer to a process involving one or more of the following steps Splitting the data into groups based on some criteria Applying a function to each group independently Combining the results into a data structure Of these, the split step is the most straightforward. In fact, in many situations you may wish to split the data set into groups and do something with those groups yourself. In the apply step, we might wish to one of the following: Aggregation: computing a summary statistic (or statistics) about each group. Some examples: Compute group sums or means Compute group sizes / counts Transformation: perform some group-specic computations and return a like-indexed. Some examples: Standardizing data (zscore) within group Filling NAs within groups with a value derived from each group Some combination of the above: GroupBy will examine the results of the apply step and try to return a sensibly combined result if it doesnt t into either of the above two categories Since the set of object instance method on pandas data structures are generally rich and expressive, we often simply want to invoke, say, a DataFrame function on each group. The name GroupBy should be quite familiar to those who have used a SQL-based tool (or itertools), in which you can write code like:
SELECT Column1, Column2, mean(Column3), sum(Column4) FROM SomeTable GROUP BY Column1, Column2

We aim to make operations like this natural and easy to express using pandas. Well address each area of GroupBy functionality then provide some non-trivial examples / use cases.

10.1 Splitting an object into groups

pandas objects can be split on any of their axes. The abstract denition of grouping is to provide a mapping of labels to group names. To create a GroupBy object (more on what the GroupBy object is later), you do the following:
>>> grouped = obj.groupby(key) >>> grouped = obj.groupby(key, axis=1) >>> grouped = obj.groupby([key1, key2])

117

pandas: powerful Python data analysis toolkit, Release 0.7.1

The mapping can be specied many different ways: A Python function, to be called on each of the axis labels A list or NumPy array of the same length as the selected axis A dict or Series, providing a label -> group name mapping For DataFrame objects, a string indicating a column to be used to group. Of course df.groupby(A) is just syntactic sugar for df.groupby(df[A]), but it makes life simpler A list of any of the above things Collectively we refer to the grouping objects as the keys. For example, consider the following DataFrame:
In [358]: df = DataFrame({A : [foo, bar, foo, bar, .....: foo, bar, foo, foo], .....: B : [one, one, two, three, .....: two, two, one, three], .....: C : randn(8), D : randn(8)}) In [359]: df Out[359]: A B 0 foo one 1 bar one 2 foo two 3 bar three 4 foo two 5 bar two 6 foo one 7 foo three

C 0.469112 -0.282863 -1.509059 -1.135632 1.212112 -0.173215 0.119209 -1.044236

D -0.861849 -2.104569 -0.494929 1.071804 0.721555 -0.706771 -1.039575 0.271860

We could naturally group by either the A or B columns or both:

In [360]: grouped = df.groupby(A) In [361]: grouped = df.groupby([A, B])

These will split the DataFrame on its index (rows). We could also split by the columns:
In [362]: def get_letter_type(letter): .....: if letter.lower() in aeiou: .....: return vowel .....: else: .....: return consonant .....: In [363]: grouped = df.groupby(get_letter_type, axis=1)

Note that no splitting occurs until its needed. Creating the GroupBy object only veries that youve passed a valid mapping. Note: Many kinds of complicated data manipulations can be expressed in terms of GroupBy operations (though cant be guaranteed to be the most efcient). You can get quite creative with the label mapping functions.

118

Chapter 10. Group By: split-apply-combine

pandas: powerful Python data analysis toolkit, Release 0.7.1

10.1.1 GroupBy object attributes

The groups attribute is a dict whose keys are the computed unique groups and corresponding values being the axis labels belonging to each group. In the above example we have:
In [364]: df.groupby(A).groups Out[364]: {bar: [1, 3, 5], foo: [0, 2, 4, 6, 7]} In [365]: df.groupby(get_letter_type, axis=1).groups Out[365]: {consonant: [B, C, D], vowel: [A]}

Calling the standard Python len function on the GroupBy object just returns the length of the groups dict, so it is largely just a convenience:
In [366]: grouped = df.groupby([A, B]) In [367]: grouped.groups Out[367]: {(bar, one): [1], (bar, three): [3], (bar, two): [5], (foo, one): [0, 6], (foo, three): [7], (foo, two): [2, 4]} In [368]: len(grouped) Out[368]: 6

By default the group keys are sorted during the groupby operation. You may however pass sort=False for potential speedups:
In [369]: df2 = DataFrame({X : [B, B, A, A], Y : [1, 2, 3, 4]}) In [370]: df2.groupby([X], sort=True).sum() Out[370]: Y X A 7 B 3 In [371]: df2.groupby([X], sort=False).sum() Out[371]: Y X B 3 A 7

10.1.2 GroupBy with MultiIndex

With hierarchically-indexed data, its quite natural to group by one of the levels of the hierarchy.
In [372]: s Out[372]: first second bar one two baz one two

-0.424972 0.567020 0.276232 -1.087401

10.1. Splitting an object into groups

119

pandas: powerful Python data analysis toolkit, Release 0.7.1

foo qux

one two one two

-0.673690 0.113648 -1.478427 0.524988

In [373]: grouped = s.groupby(level=0) In [374]: grouped.sum() Out[374]: first bar 0.142048 baz -0.811169 foo -0.560041 qux -0.953439 Name: result

If the MultiIndex has names specied, these can be passed instead of the level number:
In [375]: s.groupby(level=second).sum() Out[375]: second one -2.300857 two 0.118256 Name: result

The aggregation functions such as sum will take the level parameter directly. Additionally, the resulting index will be named according to the chosen level:
In [376]: s.sum(level=second) Out[376]: second one -2.300857 two 0.118256 Name: result

Also as of v0.6, grouping with multiple levels is supported.

In [377]: s Out[377]: first second bar doo baz foo qux bee bop

third one two one two one two one two

0.404705 0.577046 -1.715002 -1.039268 -0.370647 -1.157892 -1.344312 0.844885

In [378]: s.groupby(level=[first,second]).sum() Out[378]: first second bar doo 0.981751 baz bee -2.754270 foo bop -1.528539 qux bop -0.499427 Name: result

More on the sum function and aggregation later.

120

Chapter 10. Group By: split-apply-combine

pandas: powerful Python data analysis toolkit, Release 0.7.1

10.1.3 DataFrame column selection in GroupBy

Once you have created the GroupBy object from a DataFrame, for example, you might want to do something different for each of the columns. Thus, using [] similar to getting a column from a DataFrame, you can do:
In [379]: grouped = df.groupby([A]) In [380]: grouped_C = grouped[C] In [381]: grouped_D = grouped[D]

This is mainly syntactic sugar for the alternative and much more verbose:
In [382]: df[C].groupby(df[A]) Out[382]: <pandas.core.groupby.SeriesGroupBy at 0x113240d10>

Additionally this method avoids recomputing the internal grouping information derived from the passed key.

10.2 Iterating through groups

With the GroupBy object in hand, iterating through the grouped data is very natural and functions similarly to itertools.groupby:
In [383]: grouped = df.groupby(A) In [384]: for .....: .....: .....: bar A B 1 bar one 3 bar three 5 bar two foo A B 0 foo one 2 foo two 4 foo two 6 foo one 7 foo three name, group in grouped: print name print group

C D -0.282863 -2.104569 -1.135632 1.071804 -0.173215 -0.706771 C D 0.469112 -0.861849 -1.509059 -0.494929 1.212112 0.721555 0.119209 -1.039575 -1.044236 0.271860

In the case of grouping by multiple keys, the group name will be a tuple:
In [385]: for name, group in df.groupby([A, B]): .....: print name .....: print group .....: (bar, one) A B C D 1 bar one -0.282863 -2.104569 (bar, three) A B C D 3 bar three -1.135632 1.071804 (bar, two) A B C D 5 bar two -0.173215 -0.706771

10.2. Iterating through groups

121

pandas: powerful Python data analysis toolkit, Release 0.7.1

(foo, A 0 foo 6 foo (foo, A 7 foo (foo, A 2 foo 4 foo

one) B C D one 0.469112 -0.861849 one 0.119209 -1.039575 three) B C D three -1.044236 0.27186 two) B C D two -1.509059 -0.494929 two 1.212112 0.721555

Its standard Python-fu but remember you can unpack the tuple in the for loop statement if you wish: for (k1, k2), group in grouped:.

10.3 Aggregation
Once the GroupBy object has been created, several methods are available to perform a computation on the grouped data. An obvious one is aggregation via the aggregate or equivalently agg method:
In [386]: grouped = df.groupby(A) In [387]: grouped.aggregate(np.sum) Out[387]: B C D A bar onethreetwo -1.591710 -1.739537 foo onetwotwoonethree -0.752861 -1.402938 In [388]: grouped = df.groupby([A, B]) In [389]: grouped.aggregate(np.sum) Out[389]: C D A B bar one -0.282863 -2.104569 three -1.135632 1.071804 two -0.173215 -0.706771 foo one 0.588321 -1.901424 three -1.044236 0.271860 two -0.296946 0.226626

As you can see, the result of the aggregation will have the group names as the new index along the grouped axis. In the case of multiple keys, the result is a MultiIndex by default, though this can be changed by using the as_index option:
In [390]: grouped = df.groupby([A, B], as_index=False) In [391]: grouped.aggregate(np.sum) Out[391]: A B C D 0 bar one -0.282863 -2.104569 1 bar three -1.135632 1.071804 2 bar two -0.173215 -0.706771 3 foo one 0.588321 -1.901424 4 foo three -1.044236 0.271860

122

Chapter 10. Group By: split-apply-combine

pandas: powerful Python data analysis toolkit, Release 0.7.1

foo

two -0.296946

0.226626

In [392]: df.groupby(A, as_index=False).sum() Out[392]: A C D 0 bar -1.591710 -1.739537 1 foo -0.752861 -1.402938

Note that you could use the reset_index DataFrame function to achieve the same result as the column names are stored in the resulting MultiIndex:
In [393]: df.groupby([A, B]).sum().reset_index() Out[393]: A B C D 0 bar one -0.282863 -2.104569 1 bar three -1.135632 1.071804 2 bar two -0.173215 -0.706771 3 foo one 0.588321 -1.901424 4 foo three -1.044236 0.271860 5 foo two -0.296946 0.226626

10.3.1 Applying multiple functions at once

With grouped Series you can also pass a list or dict of functions to do aggregation with, outputting a DataFrame:
In [394]: grouped = df.groupby(A) In [395]: grouped[C].agg([np.sum, np.mean, np.std]) Out[395]: mean std sum A bar -0.530570 0.526860 -1.591710 foo -0.150572 1.113308 -0.752861

If a dict is passed, the keys will be used to name the columns. Otherwise the functions name (stored in the function object) will be used.
In [396]: grouped[D].agg({result1 : np.sum, .....: result2 : np.mean}) Out[396]: result1 result2 A bar -1.739537 -0.579846 foo -1.402938 -0.280588

On a grouped DataFrame, you can pass a list of functions to apply to each column, which produces an aggregated result with a hierarchical index:
In [397]: grouped.agg([np.sum, np.mean, np.std]) Out[397]: C D mean std sum mean std sum A bar -0.530570 0.526860 -1.591710 -0.579846 1.591986 -1.739537 foo -0.150572 1.113308 -0.752861 -0.280588 0.753219 -1.402938

Passing a dict of functions has different behavior by default, see the next section.

10.3. Aggregation

123

pandas: powerful Python data analysis toolkit, Release 0.7.1

10.3.2 Applying different functions to DataFrame columns

By passing a dict to aggregate you can apply a different aggregation to the columns of a DataFrame:
In [398]: grouped.agg({C : np.sum, .....: D : lambda x: np.std(x, ddof=1)}) Out[398]: C D A bar -1.591710 1.591986 foo -0.752861 0.753219

The function names can also be strings. In order for a string to be valid it must be either implemented on GroupBy or available via dispatching:
In [399]: grouped.agg({C : sum, D : std}) Out[399]: C D A bar -1.591710 1.591986 foo -0.752861 0.753219

10.3.3 Cython-optimized aggregation functions

Some common aggregations, currently only sum, mean, and std, have optimized Cython implementations:
In [400]: df.groupby(A).sum() Out[400]: C D A bar -1.591710 -1.739537 foo -0.752861 -1.402938 In [401]: df.groupby([A, B]).mean() Out[401]: C D A B bar one -0.282863 -2.104569 three -1.135632 1.071804 two -0.173215 -0.706771 foo one 0.294161 -0.950712 three -1.044236 0.271860 two -0.148473 0.113313

Of course sum and mean are implemented on pandas objects, so the above code would work even without the special versions via dispatching (see below).

10.4 Transformation
The transform method returns an object that is indexed the same (same size) as the one being grouped. Thus, the passed transform function should return a result that is the same size as the group chunk. For example, suppose we wished to standardize a data set within a group:
In [402]: tsdf = DataFrame(randn(1000, 3), .....: index=DateRange(1/1/2000, periods=1000), .....: columns=[A, B, C])

124

Chapter 10. Group By: split-apply-combine

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [403]: tsdf Out[403]: <class pandas.core.frame.DataFrame> DateRange: 1000 entries, 2000-01-03 00:00:00 to 2003-10-31 00:00:00 offset: <1 BusinessDay> Data columns: A 1000 non-null values B 1000 non-null values C 1000 non-null values dtypes: float64(3) In [404]: zscore = lambda x: (x - x.mean()) / x.std() In [405]: transformed = tsdf.groupby(lambda x: x.year).transform(zscore)

We would expect the result to now have mean 0 and standard deviation 1 within each group, which we can easily check:
In [406]: grouped = transformed.groupby(lambda x: x.year) # OK, close enough to zero In [407]: grouped.mean() Out[407]: A B C key_0 2000 -0 -0 0 2001 0 0 0 2002 -0 -0 -0 2003 0 -0 0 In [408]: Out[408]: A key_0 2000 1 2001 1 2002 1 2003 1 grouped.std() B 1 1 1 1 C 1 1 1 1

10.5 Dispatching to instance methods

When doing an aggregation or transformation, you might just want to call an instance method on each data group. This is pretty easy to do by passing lambda functions:
In [409]: grouped = df.groupby(A) In [410]: grouped.agg(lambda x: x.std()) Out[410]: B C D A bar NaN 0.526860 1.591986 foo NaN 1.113308 0.753219

But, its rather verbose and can be untidy if you need to pass additional arguments. Using a bit of metaprogramming cleverness, GroupBy now has the ability to dispatch method calls to the groups:

10.5. Dispatching to instance methods

125

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [411]: grouped.std() Out[411]: C D A bar 0.526860 1.591986 foo 1.113308 0.753219

What is actually happening here is that a function wrapper is being generated. When invoked, it takes any passed arguments and invokes the function with any arguments on each group (in the above example, the std function). The results are then combined together much in the style of agg and transform (it actually uses apply to infer the gluing, documented next). This enables some operations to be carried out rather succinctly:
In [412]: tsdf.ix[::2] = np.nan In [413]: grouped = tsdf.groupby(lambda x: x.year) In [414]: grouped.fillna(method=pad) Out[414]: <class pandas.core.frame.DataFrame> DateRange: 1000 entries, 2000-01-03 00:00:00 to 2003-10-31 00:00:00 offset: <1 BusinessDay> Data columns: A 997 non-null values B 997 non-null values C 997 non-null values dtypes: float64(3)

In this example, we chopped the collection of time series into yearly chunks then independently called llna on the groups.

10.6 Flexible apply

Some operations on the grouped data might not t into either the aggregate or transform categories. Or, you may simply want GroupBy to infer how to combine the results. For these, use the apply function, which can be substituted for both aggregate and transform in many standard use cases. However, apply can handle some exceptional use cases, for example:
In [415]: df Out[415]: A B 0 foo one 1 bar one 2 foo two 3 bar three 4 foo two 5 bar two 6 foo one 7 foo three

C 0.469112 -0.282863 -1.509059 -1.135632 1.212112 -0.173215 0.119209 -1.044236

D -0.861849 -2.104569 -0.494929 1.071804 0.721555 -0.706771 -1.039575 0.271860

In [416]: grouped = df.groupby(A) # could also just call .describe() In [417]: grouped[C].apply(lambda x: x.describe()) Out[417]: count mean std min 25% A

50%

75%

max

126

Chapter 10. Group By: split-apply-combine

pandas: powerful Python data analysis toolkit, Release 0.7.1

bar foo

3 -0.530570 5 -0.150572

0.526860 -1.135632 -0.709248 -0.282863 -0.228039 -0.173215 1.113308 -1.509059 -1.044236 0.119209 0.469112 1.212112

The dimension of the returned result can also change:

In [418]: grouped = df.groupby(A)[C] In [419]: def f(group): .....: return DataFrame({original : group, .....: demeaned : group - group.mean()}) .....: In [420]: grouped.apply(f) Out[420]: demeaned original 0 0.619685 0.469112 1 0.247707 -0.282863 2 -1.358486 -1.509059 3 -0.605062 -1.135632 4 1.362684 1.212112 5 0.357355 -0.173215 6 0.269781 0.119209 7 -0.893664 -1.044236

10.7 Other useful features

10.7.1 Automatic exclusion of nuisance columns
Again consider the example DataFrame weve been looking at:
In [421]: df Out[421]: A B 0 foo one 1 bar one 2 foo two 3 bar three 4 foo two 5 bar two 6 foo one 7 foo three

C 0.469112 -0.282863 -1.509059 -1.135632 1.212112 -0.173215 0.119209 -1.044236

D -0.861849 -2.104569 -0.494929 1.071804 0.721555 -0.706771 -1.039575 0.271860

Supposed we wished to compute the standard deviation grouped by the A column. There is a slight problem, namely that we dont care about the data in column B. We refer to this as a nuisance column. If the passed aggregation function cant be applied to some columns, the troublesome columns will be (silently) dropped. Thus, this does not pose any problems:
In [422]: df.groupby(A).std() Out[422]: C D A bar 0.526860 1.591986 foo 1.113308 0.753219

10.7. Other useful features

127

pandas: powerful Python data analysis toolkit, Release 0.7.1

10.7.2 NA group handling

If there are any NaN values in the grouping key, these will be automatically excluded. So there will never be an NA group. This was not the case in older versions of pandas, but users were generally discarding the NA group anyway (and supporting it was an implementation headache).

128

Chapter 10. Group By: split-apply-combine

CHAPTER

ELEVEN

MERGE, JOIN, AND CONCATENATE

pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.

11.1 Concatenating objects

The concat function (in the main pandas namespace) does all of the heavy lifting of performing concatenation operations along an axis while performing optional set logic (union or intersection) of the indexes (if any) on the other axes. Note that I say if any because there is only a single possible axis of concatenation for Series. Before diving into all of the details of concat and what it can do, here is a simple example:
In [620]: df = DataFrame(np.random.randn(10, 4)) In [621]: df Out[621]: 0 1 0 0.469112 -0.282863 1 1.212112 -0.173215 2 -0.861849 -2.104569 3 0.721555 -0.706771 4 -0.424972 0.567020 5 -0.673690 0.113648 6 0.404705 0.577046 7 -0.370647 -1.157892 8 1.075770 -0.109050 9 0.357021 -0.674600

2 -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427 -1.715002 -1.344312 1.643563 -1.776904

3 -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885 -1.469388 -0.968914

# break it into pieces In [622]: pieces = [df[:3], df[3:7], df[7:]] In [623]: concatenated = concat(pieces) In [624]: concatenated Out[624]: 0 1 2 0 0.469112 -0.282863 -1.509059 1 1.212112 -0.173215 0.119209 2 -0.861849 -2.104569 -0.494929 3 0.721555 -0.706771 -1.039575 4 -0.424972 0.567020 0.276232 5 -0.673690 0.113648 -1.478427 6 0.404705 0.577046 -1.715002

3 -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268

129

pandas: powerful Python data analysis toolkit, Release 0.7.1

7 -0.370647 -1.157892 -1.344312 0.844885 8 1.075770 -0.109050 1.643563 -1.469388 9 0.357021 -0.674600 -1.776904 -0.968914

Like its sibling function on ndarrays, numpy.concatenate, pandas.concat takes a list or dict of homogeneously-typed objects and concatenates them with some congurable handling of what to do with the other axes:
concat(objs, axis=0, join=outer, join_axes=None, ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False)

objs: list or dict of Series, DataFrame, or Panel objects. If a dict is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below) axis: {0, 1, ...}, default 0. The axis to concatenate along join: {inner, outer}, default outer. How to handle indexes on other axis(es). Outer for union and inner for intersection join_axes: list of Index objects. Specic indexes to use for the other n - 1 axes instead of performing inner/outer set logic keys: sequence, default None. Construct hierarchical index using the passed keys as the outermost level If multiple levels passed, should contain tuples. levels : list of sequences, default None. If keys passed, specic levels to use for the resulting MultiIndex. Otherwise they will be inferred from the keys names: list, default None. Names for the levels in the resulting hierarchical index verify_integrity: boolean, default False. Check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation ignore_index : boolean, default False. If True, do not use the index values on the concatenation axis. The resulting axis will be labeled 0, ..., n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Without a little bit of context and example many of these arguments dont make much sense. Lets take the above example. Suppose we wanted to associate specic keys with each of the pieces of the chopped up DataFrame. We can do this using the keys argument:
In [625]: concatenated = concat(pieces, keys=[first, second, third]) In [626]: concatenated Out[626]: 0 1 first 0 0.469112 -0.282863 1 1.212112 -0.173215 2 -0.861849 -2.104569 second 3 0.721555 -0.706771 4 -0.424972 0.567020 5 -0.673690 0.113648 6 0.404705 0.577046 third 7 -0.370647 -1.157892 8 1.075770 -0.109050 9 0.357021 -0.674600

2 -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427 -1.715002 -1.344312 1.643563 -1.776904

3 -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988 -1.039268 0.844885 -1.469388 -0.968914

As you can see (if youve read the rest of the documentation), the resulting objects index has a hierarchical index. This means that we can now do stuff like select out each chunk by key:

130

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [627]: concatenated.ix[second] Out[627]: 0 1 2 3 3 0.721555 -0.706771 -1.039575 0.271860 4 -0.424972 0.567020 0.276232 -1.087401 5 -0.673690 0.113648 -1.478427 0.524988 6 0.404705 0.577046 -1.715002 -1.039268

Its not a stretch to see how this can be very useful. More detail on this functionality below.

11.1.1 Set logic on the other axes

When gluing together multiple DataFrames (or Panels or...), for example, you have a choice of how to handle the other axes (other than the one being concatenated). This can be done in three ways: Take the (sorted) union of them all, join=outer. This is the default option as it results in zero information loss. Take the intersection, join=inner. Use a specic index (in the case of DataFrame) or indexes (in the case of Panel or future higher dimensional objects), i.e. the join_axes argument Here is a example of each of these methods. First, the default join=outer behavior:
In [628]: from pandas.util.testing import rands In [629]: df = DataFrame(np.random.randn(10, 4), columns=[a, b, c, d], .....: index=[rands(5) for _ in xrange(10)]) In [630]: df Out[630]: PRh1h uJToG Muad1 WiR40 YK0p6 ZW134 ZRbsV BGx5h zCvoD ZepHk a -1.294524 -0.013960 0.895717 1.431256 0.410835 -0.076467 -1.413681 0.875906 -0.410001 -1.226825 b 0.413738 -0.362543 0.805244 1.340309 0.813850 -1.187678 1.607920 -2.211372 -0.078638 0.769804 c 0.276662 -0.006154 -1.206412 -1.170299 0.132003 1.130127 1.024180 0.974466 0.545952 -1.281247 d -0.472035 -0.923061 2.565646 -0.226169 -0.827317 -1.436737 0.569605 -2.006747 -1.219217 -0.727707

In [631]: concat([df.ix[:7, [a, b]], df.ix[2:-2, [c]], .....: df.ix[-7:, [d]]], axis=1) Out[631]: a b c d BGx5h NaN NaN 0.974466 -2.006747 Muad1 0.895717 0.805244 -1.206412 NaN PRh1h -1.294524 0.413738 NaN NaN WiR40 1.431256 1.340309 -1.170299 -0.226169 YK0p6 0.410835 0.813850 0.132003 -0.827317 ZRbsV -1.413681 1.607920 1.024180 0.569605 ZW134 -0.076467 -1.187678 1.130127 -1.436737 ZepHk NaN NaN NaN -0.727707 uJToG -0.013960 -0.362543 NaN NaN zCvoD NaN NaN NaN -1.219217

11.1. Concatenating objects

131

pandas: powerful Python data analysis toolkit, Release 0.7.1

Note that the row indexes have been unioned and sorted. Here is the same thing with join=inner:
In [632]: concat([df.ix[:7, [a, b]], df.ix[2:-2, [c]], .....: df.ix[-7:, [d]]], axis=1, join=inner) Out[632]: a b c d WiR40 1.431256 1.340309 -1.170299 -0.226169 YK0p6 0.410835 0.813850 0.132003 -0.827317 ZW134 -0.076467 -1.187678 1.130127 -1.436737 ZRbsV -1.413681 1.607920 1.024180 0.569605

Lastly, suppose we just wanted to reuse the exact index from the original DataFrame:
In [633]: concat([df.ix[:7, [a, b]], df.ix[2:-2, [c]], .....: df.ix[-7:, [d]]], axis=1, join_axes=[df.index]) Out[633]: a b c d PRh1h -1.294524 0.413738 NaN NaN uJToG -0.013960 -0.362543 NaN NaN Muad1 0.895717 0.805244 -1.206412 NaN WiR40 1.431256 1.340309 -1.170299 -0.226169 YK0p6 0.410835 0.813850 0.132003 -0.827317 ZW134 -0.076467 -1.187678 1.130127 -1.436737 ZRbsV -1.413681 1.607920 1.024180 0.569605 BGx5h NaN NaN 0.974466 -2.006747 zCvoD NaN NaN NaN -1.219217 ZepHk NaN NaN NaN -0.727707

11.1.2 Concatenating using append

A useful shortcut to concat are the append instance methods on Series and DataFrame. These methods actually predated concat. They concatenate along axis=0, namely the index:
In [634]: s = Series(randn(10), index=np.arange(10)) In [635]: s1 = s[:5] # note were slicing with labels here, so 5 is included In [636]: s2 = s[6:] In [637]: s1.append(s2) Out[637]: 0 -0.121306 1 -0.097883 2 0.695775 3 0.341734 4 0.959726 6 -0.619976 7 0.149748 8 -0.732339 9 0.687738

In the case of DataFrame, the indexes must be disjoint but the columns do not need to be:
In [638]: df = DataFrame(randn(6, 4), index=DateRange(1/1/2000, periods=6), .....: columns=[A, B, C, D]) In [639]: df1 = df.ix[:3]

132

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [640]: df2 = df.ix[3:, :3] In [641]: df1 Out[641]: A B C 2000-01-03 0.176444 0.403310 -0.154951 2000-01-04 -2.179861 -1.369849 -0.954208 2000-01-05 -1.743161 -0.826591 -0.345352 In [642]: df2 Out[642]: A B C 2000-01-06 0.690579 0.995761 2.396780 2000-01-07 3.357427 -0.317441 -1.236269 2000-01-10 -0.487602 -0.082240 -2.182937 In [643]: df1.append(df2) Out[643]: A B 2000-01-03 0.176444 0.403310 2000-01-04 -2.179861 -1.369849 2000-01-05 -1.743161 -0.826591 2000-01-06 0.690579 0.995761 2000-01-07 3.357427 -0.317441 2000-01-10 -0.487602 -0.082240 D 0.301624 1.462696 1.314232

C -0.154951 -0.954208 -0.345352 2.396780 -1.236269 -2.182937

D 0.301624 1.462696 1.314232 NaN NaN NaN

append may take multiple objects to concatenate:

In [644]: df1 = df.ix[:2] In [645]: df2 = df.ix[2:4] In [646]: df3 = df.ix[4:] In [647]: df1.append([df2,df3]) Out[647]: A B C 2000-01-03 0.176444 0.403310 -0.154951 2000-01-04 -2.179861 -1.369849 -0.954208 2000-01-05 -1.743161 -0.826591 -0.345352 2000-01-06 0.690579 0.995761 2.396780 2000-01-07 3.357427 -0.317441 -1.236269 2000-01-10 -0.487602 -0.082240 -2.182937

D 0.301624 1.462696 1.314232 0.014871 0.896171 0.380396

Note: Unlike list.append method, which appends to the original list and returns nothing, append here does not modify df1 and returns its copy with df2 appended.

11.1.3 Ignoring indexes on the concatenation axis

For DataFrames which dont have a meaningful index, you may wish to append them and ignore the fact that they may have overlapping indexes:
In [648]: df1 = DataFrame(randn(6, 4), columns=[A, B, C, D]) In [649]: df2 = DataFrame(randn(3, 4), columns=[A, B, C, D])

11.1. Concatenating objects

133

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [650]: df1 Out[650]: A B C 0 0.084844 0.432390 1.519970 1 0.600178 0.274230 0.132885 2 2.410179 1.450520 0.206053 3 -2.213588 1.063327 1.266143 4 -0.863838 0.408204 -1.048089 5 -0.988387 0.094055 1.262731

D -0.493662 -0.023688 -0.251905 0.299368 -0.025747 1.289997

In [651]: df2 Out[651]: A B C D 0 0.082423 -0.055758 0.536580 -0.489682 1 0.369374 -0.034571 -2.484478 -0.281461 2 0.030711 0.109121 1.126203 -0.977349

To do this, use the ignore_index argument:

In [652]: concat([df1, df2], ignore_index=True) Out[652]: A B C D 0 0.084844 0.432390 1.519970 -0.493662 1 0.600178 0.274230 0.132885 -0.023688 2 2.410179 1.450520 0.206053 -0.251905 3 -2.213588 1.063327 1.266143 0.299368 4 -0.863838 0.408204 -1.048089 -0.025747 5 -0.988387 0.094055 1.262731 1.289997 6 0.082423 -0.055758 0.536580 -0.489682 7 0.369374 -0.034571 -2.484478 -0.281461 8 0.030711 0.109121 1.126203 -0.977349

This is also a valid argument to DataFrame.append:

In [653]: df1.append(df2, ignore_index=True) Out[653]: A B C D 0 0.084844 0.432390 1.519970 -0.493662 1 0.600178 0.274230 0.132885 -0.023688 2 2.410179 1.450520 0.206053 -0.251905 3 -2.213588 1.063327 1.266143 0.299368 4 -0.863838 0.408204 -1.048089 -0.025747 5 -0.988387 0.094055 1.262731 1.289997 6 0.082423 -0.055758 0.536580 -0.489682 7 0.369374 -0.034571 -2.484478 -0.281461 8 0.030711 0.109121 1.126203 -0.977349

11.1.4 More concatenating with group keys

Lets consider a variation on the rst example presented:
In [654]: df = DataFrame(np.random.randn(10, 4)) In [655]: df Out[655]: 0 1 2 0 1.474071 -0.064034 -1.282782 1 -1.071357 0.441153 2.353925

3 0.781836 0.583787

134

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

2 3 4 5 6 7 8 9

0.221471 -0.964980 -1.328865 0.228440 -1.197071 0.306996 1.588931 -0.014805

-0.744471 0.758527 1.729689 -0.845696 -1.340896 1.846883 1.682706 -1.717693 0.888782 0.901805 1.171216 0.520260 -1.066969 -0.303421 -0.858447 -0.028665 0.384316 1.574159 0.476720 0.473424 -0.242861 -0.284319 0.650776 -1.461665

# break it into pieces In [656]: pieces = [df.ix[:, [0, 1]], df.ix[:, [2]], df.ix[:, [3]]] In [657]: result = concat(pieces, axis=1, keys=[one, two, three]) In [658]: result Out[658]: one 0 1 0 1.474071 -0.064034 1 -1.071357 0.441153 2 0.221471 -0.744471 3 -0.964980 -0.845696 4 -1.328865 1.682706 5 0.228440 0.901805 6 -1.197071 -1.066969 7 0.306996 -0.028665 8 1.588931 0.476720 9 -0.014805 -0.284319

two three 2 3 -1.282782 0.781836 2.353925 0.583787 0.758527 1.729689 -1.340896 1.846883 -1.717693 0.888782 1.171216 0.520260 -0.303421 -0.858447 0.384316 1.574159 0.473424 -0.242861 0.650776 -1.461665

You can also pass a dict to concat in which case the dict keys will be used for the keys argument (unless other keys are specied):
In [659]: pieces = {one: df.ix[:, [0, 1]], .....: two: df.ix[:, [2]], .....: three: df.ix[:, [3]]} In [660]: concat(pieces, axis=1) Out[660]: one three two 0 1 3 2 0 1.474071 -0.064034 0.781836 -1.282782 1 -1.071357 0.441153 0.583787 2.353925 2 0.221471 -0.744471 1.729689 0.758527 3 -0.964980 -0.845696 1.846883 -1.340896 4 -1.328865 1.682706 0.888782 -1.717693 5 0.228440 0.901805 0.520260 1.171216 6 -1.197071 -1.066969 -0.858447 -0.303421 7 0.306996 -0.028665 1.574159 0.384316 8 1.588931 0.476720 -0.242861 0.473424 9 -0.014805 -0.284319 -1.461665 0.650776 In [661]: concat(pieces, keys=[three, two]) Out[661]: 2 3 three 0 NaN 0.781836 1 NaN 0.583787 2 NaN 1.729689 3 NaN 1.846883

11.1. Concatenating objects

135

pandas: powerful Python data analysis toolkit, Release 0.7.1

two

4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9

NaN 0.888782 NaN 0.520260 NaN -0.858447 NaN 1.574159 NaN -0.242861 NaN -1.461665 -1.282782 NaN 2.353925 NaN 0.758527 NaN -1.340896 NaN -1.717693 NaN 1.171216 NaN -0.303421 NaN 0.384316 NaN 0.473424 NaN 0.650776 NaN

The MultiIndex created has levels that are constructed from the passed keys and the columns of the DataFrame pieces:
In [662]: result.columns.levels Out[662]: [Index([one, two, three], dtype=object), Int64Index([0, 1, 2, 3])]

If you wish to specify other levels (as will occasionally be the case), you can do so using the levels argument:
In [663]: result = concat(pieces, axis=1, keys=[one, two, three], .....: levels=[[three, two, one, zero]], .....: names=[group_key]) In [664]: result Out[664]: group_key one 0 0 1.474071 1 -1.071357 2 0.221471 3 -0.964980 4 -1.328865 5 0.228440 6 -1.197071 7 0.306996 8 1.588931 9 -0.014805

1 -0.064034 0.441153 -0.744471 -0.845696 1.682706 0.901805 -1.066969 -0.028665 0.476720 -0.284319

two three 2 3 -1.282782 0.781836 2.353925 0.583787 0.758527 1.729689 -1.340896 1.846883 -1.717693 0.888782 1.171216 0.520260 -0.303421 -0.858447 0.384316 1.574159 0.473424 -0.242861 0.650776 -1.461665

In [665]: result.columns.levels Out[665]: [Index([three, two, one, zero], dtype=object), Int64Index([0, 1, 2, 3])]

Yes, this is fairly esoteric, but is actually necessary for implementing things like GroupBy where the order of a categorical variable is meaningful.

11.1.5 Appending rows to a DataFrame

While not especially efcient (since a new object must be created), you can append a single row to a DataFrame by passing a Series or dict to append, which returns a new DataFrame as above.
In [666]: df = DataFrame(np.random.randn(8, 4), columns=[A,B,C,D]) In [667]: df Out[667]:

136

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

0 1 2 3 4 5 6 7

A B C D -1.137707 -0.891060 -0.693921 1.613616 0.464000 0.227371 -0.496922 0.306389 -2.290613 -1.134623 -1.561819 -0.260838 0.281957 1.523962 -0.902937 0.068159 -0.057873 -0.368204 -1.144073 0.861209 0.800193 0.782098 -1.069094 -1.099248 0.255269 0.009750 0.661084 0.379319 -0.008434 1.952541 -1.056652 0.533946

In [668]: s = df.xs(3) In [669]: df.append(s, ignore_index=True) Out[669]: A B C D 0 -1.137707 -0.891060 -0.693921 1.613616 1 0.464000 0.227371 -0.496922 0.306389 2 -2.290613 -1.134623 -1.561819 -0.260838 3 0.281957 1.523962 -0.902937 0.068159 4 -0.057873 -0.368204 -1.144073 0.861209 5 0.800193 0.782098 -1.069094 -1.099248 6 0.255269 0.009750 0.661084 0.379319 7 -0.008434 1.952541 -1.056652 0.533946 8 0.281957 1.523962 -0.902937 0.068159

You should use ignore_index with this method to instruct DataFrame to discard its index. If you wish to preserve the index, you should construct an appropriately-indexed DataFrame and append or concatenate those objects. You can also pass a list of dicts or Series:
In [670]: df = DataFrame(np.random.randn(5, 4), .....: columns=[foo, bar, baz, qux]) In [671]: dicts = [{foo: 1, bar: 2, baz: 3, peekaboo: 4}, .....: {foo: 5, bar: 6, baz: 7, peekaboo: 8}] In [672]: result = df.append(dicts, ignore_index=True) In [673]: result Out[673]: bar baz foo 0 0.040403 -0.507516 -1.226970 1 -1.934370 -1.652499 0.394500 2 0.576897 1.146000 -0.896484 3 2.121453 0.597701 0.604603 4 -1.057909 1.375020 0.967661 5 2.000000 3.000000 1.000000 6 6.000000 7.000000 5.000000

peekaboo qux NaN -0.230096 NaN 1.488753 NaN 1.487349 NaN 0.563700 NaN -0.928797 4 NaN 8 NaN

11.2 Database-style DataFrame joining/merging

pandas has full-featured, high performance in-memory join operations idiomatically very similar to relational databases like SQL. These methods perform signicantly better (in some cases well over an order of magnitude better) than other open source implementations (like base::merge.data.frame in R). The reason for this is careful algorithmic design and internal layout of the data in DataFrame.

11.2. Database-style DataFrame joining/merging

137

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas provides a single function, merge, as the entry point for all standard database join operations between DataFrame objects:
merge(left, right, how=left, on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=True, suffixes=(.x, .y), copy=True)

Heres a description of what each argument is for: left: A DataFrame object right: Another DataFrame object on: Columns (names) to join on. Must be found in both the left and right DataFrame objects. If not passed and left_index and right_index are False, the intersectino of the columns in the DataFrames will be inferred to be the join keys left_on: Columns from the left DataFrame to use as keys. Can either be column names or arrays with length equal to the length of the DataFrame right_on: Columns from the left DataFrame to use as keys. Can either be column names or arrays with length equal to the length of the DataFrame left_index: If True, use the index (row labels) from the left DataFrame as its join key(s). In the case of a DataFrame with a MultiIndex (hierarchical), the number of levels must match the number of join keys from the right DataFrame right_index: Same usage as left_index for the right DataFrame how: One of left, right, outer, inner. Defaults to inner. See below for more detailed description of each method sort: Sort the result DataFrame by the join keys in lexicographical order. Defaults to True, setting to False will improve performance substantially in many cases suffixes: A tuple of string sufxes to apply to overlapping columns. Defaults to (.x, .y). copy: Always copy data (default True) from the passed DataFrame objects, even when reindexing is not necessary. Cannot be avoided in many cases but may improve performance / memory usage. The cases where copying can be avoided are somewhat pathological but this option is provided nonetheless. merge is a function in the pandas namespace, and it is also available as a DataFrame instance method, with the calling DataFrame being implicitly considered the left object in the join. The related DataFrame.join method, uses merge internally for the index-on-index and index-on-column(s) joins, but joins on indexes by default rather than trying to join on common columns (the default behavior for merge). If you are joining on index, you may wish to use DataFrame.join to save yourself some typing.

11.2.1 Brief primer on merge methods (relational algebra)

Experienced users of relational databases like SQL will be familiar with the terminology used to describe join operations between two SQL-table like structures (DataFrame objects). There are several cases to consider which are very important to understand: one-to-one joins: for example when joining two DataFrame objects on their indexes (which must contain unique values) many-to-one joins: for example when joining an index (unique) to one or more columns in a DataFrame many-to-many joins: joining columns on columns.

138

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

Note: When joining columns on columns (potentially a many-to-many join), any indexes on the passed DataFrame objects will be discarded. It is worth spending some time understanding the result of the many-to-many join case. In SQL / standard relational algebra, if a key combination appears more than once in both tables, the resulting table will have the Cartesian product of the associated data. Here is a very basic example with one unique key combination:
In [674]: left = DataFrame({key: [foo, foo], lval: [1, 2]}) In [675]: right = DataFrame({key: [foo, foo], rval: [4, 5]}) In [676]: left Out[676]: key lval 0 foo 1 1 foo 2 In [677]: right Out[677]: key rval 0 foo 4 1 foo 5 In [678]: merge(left, right, on=key) Out[678]: key lval rval 0 foo 1 4 1 foo 1 5 2 foo 2 4 3 foo 2 5

Here is a more complicated example with multiple join keys:

In [679]: left = DataFrame({key1: [foo, foo, bar], .....: key2: [one, two, one], .....: lval: [1, 2, 3]}) In [680]: right = DataFrame({key1: [foo, foo, bar, bar], .....: key2: [one, one, one, two], .....: rval: [4, 5, 6, 7]}) In [681]: merge(left, right, how=outer) Out[681]: key1 key2 lval rval 0 bar one 3 6 1 bar two NaN 7 2 foo one 1 4 3 foo one 1 5 4 foo two 2 NaN In [682]: merge(left, right, how=inner) Out[682]: key1 key2 lval rval 0 bar one 3 6 1 foo one 1 4 2 foo one 1 5

11.2. Database-style DataFrame joining/merging

139

pandas: powerful Python data analysis toolkit, Release 0.7.1

The how argument to merge species how to determine which keys are to be included in the resulting table. If a key combination does not appear in either the left or right tables, the values in the joined table will be NA. Here is a summary of the how options and their SQL equivalent names: Merge method left right outer inner SQL Join Name LEFT OUTER JOIN RIGHT OUTER JOIN FULL OUTER JOIN INNER JOIN Description Use keys from left frame only Use keys from right frame only Use union of keys from both frames Use intersection of keys from both frames

Note that if using the index from either the left or right DataFrame (or both) using the left_index / right_index options, the join operation is no longer a many-to-many join by construction, as the index values are necessarily unique. There will be some examples of this below.

11.2.2 Joining on index

DataFrame.join is a convenient method for combining the columns of two potentially differently-indexed DataFrames into a single result DataFrame. Here is a very basic example:
In [683]: df = DataFrame(np.random.randn(8, 4), columns=[A,B,C,D]) In [684]: df1 = df.ix[1:, [A, B]] In [685]: df2 = df.ix[:5, [C, D]] In [686]: df1 Out[686]: A B 1 -2.461467 -1.553902 2 1.771740 -0.670027 3 -3.201750 0.792716 4 -0.747169 -0.309038 5 0.936527 1.255746 6 0.062297 -0.110388 7 0.077849 0.629498 In [687]: df2 Out[687]: C D 0 0.377953 0.493672 1 2.015523 -1.833722 2 0.049307 -0.521493 3 0.146111 1.903247 4 0.393876 1.861468 5 -2.655452 1.219492 In [688]: df1.join(df2) Out[688]: A B C D 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247 4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492 6 0.062297 -0.110388 NaN NaN 7 0.077849 0.629498 NaN NaN

140

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [689]: df1.join(df2, how=outer) Out[689]: A B C D 0 NaN NaN 0.377953 0.493672 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247 4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492 6 0.062297 -0.110388 NaN NaN 7 0.077849 0.629498 NaN NaN In [690]: df1.join(df2, how=inner) Out[690]: A B C D 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247 4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492

The data alignment here is on the indexes (row labels). This same behavior can be achieved using merge plus additional arguments instructing it to use the indexes:
In [691]: merge(df1, df2, left_index=True, right_index=True, how=outer) Out[691]: A B C D 0 NaN NaN 0.377953 0.493672 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247 4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492 6 0.062297 -0.110388 NaN NaN 7 0.077849 0.629498 NaN NaN

11.2.3 Joining key columns on an index

join takes an optional on argument which may be a column or multiple column names, which species that the passed DataFrame is to be aligned on that column in the DataFrame. These two function calls are completely equivalent:
left.join(right, on=key_or_keys) merge(left, right, left_on=key_or_keys, right_index=True, how=left, sort=False)

Obviously you can choose whichever form you nd more convenient. For many-to-one joins (where one of the DataFrames is already indexed by the join key), using join may be more convenient. Here is a simple example:
In [692]: df[key] = [foo, bar] * 4 In [693]: to_join = DataFrame(randn(2, 2), index=[bar, foo], .....: columns=[j1, j2]) In [694]: df Out[694]: A

key

11.2. Database-style DataFrame joining/merging

141

pandas: powerful Python data analysis toolkit, Release 0.7.1

0 1 2 3 4 5 6 7

-0.308853 -2.461467 1.771740 -3.201750 -0.747169 0.936527 0.062297 0.077849

-0.681087 0.377953 0.493672 -1.553902 2.015523 -1.833722 -0.670027 0.049307 -0.521493 0.792716 0.146111 1.903247 -0.309038 0.393876 1.861468 1.255746 -2.655452 1.219492 -0.110388 -1.184357 -0.558081 0.629498 -1.035260 -0.438229

foo bar foo bar foo bar foo bar

In [695]: to_join Out[695]: j1 j2 bar 0.503703 0.413086 foo -1.139050 0.660342 In [696]: df.join(to_join, on=key) Out[696]: A B C D 0 -0.308853 -0.681087 0.377953 0.493672 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247 4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492 6 0.062297 -0.110388 -1.184357 -0.558081 7 0.077849 0.629498 -1.035260 -0.438229

key foo bar foo bar foo bar foo bar

j1 -1.139050 0.503703 -1.139050 0.503703 -1.139050 0.503703 -1.139050 0.503703

j2 0.660342 0.413086 0.660342 0.413086 0.660342 0.413086 0.660342 0.413086

In [697]: merge(df, to_join, left_on=key, right_index=True, .....: how=left, sort=False) Out[697]: A B C D key j1 j2 0 -0.308853 -0.681087 0.377953 0.493672 foo -1.139050 0.660342 1 -2.461467 -1.553902 2.015523 -1.833722 bar 0.503703 0.413086 2 1.771740 -0.670027 0.049307 -0.521493 foo -1.139050 0.660342 3 -3.201750 0.792716 0.146111 1.903247 bar 0.503703 0.413086 4 -0.747169 -0.309038 0.393876 1.861468 foo -1.139050 0.660342 5 0.936527 1.255746 -2.655452 1.219492 bar 0.503703 0.413086 6 0.062297 -0.110388 -1.184357 -0.558081 foo -1.139050 0.660342 7 0.077849 0.629498 -1.035260 -0.438229 bar 0.503703 0.413086

To join on multiple keys, the passed DataFrame must have a MultiIndex:

In [698]: index = MultiIndex(levels=[[foo, bar, baz, qux], .....: [one, two, three]], .....: labels=[[0, 0, 0, 1, 1, 2, 2, 3, 3, 3], .....: [0, 1, 2, 0, 1, 1, 2, 0, 1, 2]], .....: names=[first, second]) In [699]: to_join = DataFrame(np.random.randn(10, 3), index=index, .....: columns=[j_one, j_two, j_three]) # a little relevant example with NAs In [700]: key1 = [bar, bar, bar, foo, foo, baz, baz, qux, .....: qux, snap] In [701]: key2 = [two, one, three, one, two, one, two, two, .....: three, one] In [702]: data = np.random.randn(len(key1))

142

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [703]: data = DataFrame({key1 : key1, key2 : key2, .....: data : data}) In [704]: data Out[704]: data key1 0 -1.004168 bar 1 -1.377627 bar 2 0.499281 bar 3 -1.405256 foo 4 0.162565 foo 5 -0.067785 baz 6 -1.260006 baz 7 -1.132896 qux 8 -2.006481 qux 9 0.301016 snap In [705]: to_join Out[705]: j_one first second foo one two three bar one two baz two three qux one two three 0.464794 0.683758 1.032814 1.515707 1.397431 -0.135950 0.281151 -0.851985 -1.537770 -0.390201 j_two -0.309337 -0.643834 -1.290493 -0.276487 1.503874 -0.730327 -1.298915 -1.106952 0.555759 1.207122 j_three -0.649593 0.421287 0.787872 -0.223762 -0.478905 -0.033277 -2.819487 -0.937731 -2.277282 0.178690

key2 two one three one two one two two three one

Now this can be joined by passing the two key column names:
In [706]: data.join(to_join, on=[key1, key2]) Out[706]: data key1 key2 j_one j_two j_three 0 -1.004168 bar two 1.397431 1.503874 -0.478905 1 -1.377627 bar one 1.515707 -0.276487 -0.223762 2 0.499281 bar three NaN NaN NaN 3 -1.405256 foo one 0.464794 -0.309337 -0.649593 4 0.162565 foo two 0.683758 -0.643834 0.421287 5 -0.067785 baz one NaN NaN NaN 6 -1.260006 baz two -0.135950 -0.730327 -0.033277 7 -1.132896 qux two -1.537770 0.555759 -2.277282 8 -2.006481 qux three -0.390201 1.207122 0.178690 9 0.301016 snap one NaN NaN NaN

The default for DataFrame.join is to perform a left join (essentially a VLOOKUP operation, for Excel users), which uses only the keys found in the calling DataFrame. Other join types, for example inner join, can be just as easily performed:
In [707]: data.join(to_join, on=[key1, key2], how=inner) Out[707]: data key1 key2 j_one j_two j_three 0 -1.004168 bar two 1.397431 1.503874 -0.478905 1 -1.377627 bar one 1.515707 -0.276487 -0.223762 3 -1.405256 foo one 0.464794 -0.309337 -0.649593

11.2. Database-style DataFrame joining/merging

143

pandas: powerful Python data analysis toolkit, Release 0.7.1

4 0.162565 6 -1.260006 7 -1.132896 8 -2.006481

foo baz qux qux

two 0.683758 -0.643834 0.421287 two -0.135950 -0.730327 -0.033277 two -1.537770 0.555759 -2.277282 three -0.390201 1.207122 0.178690

As you can see, this drops any rows where there was no match.

11.2.4 Overlapping value columns

The merge suffixes argument takes a tuple of list of strings to append to overlapping column names in the input DataFrames to disambiguate the result columns:
In [708]: left = DataFrame({key: [foo, foo], value: [1, 2]}) In [709]: right = DataFrame({key: [foo, foo], value: [4, 5]}) In [710]: merge(left, right, on=key, suffixes=[_left, _right]) Out[710]: key value_left value_right 0 foo 1 4 1 foo 1 5 2 foo 2 4 3 foo 2 5

DataFrame.join has lsuffix and rsuffix arguments which behave similarly.

11.2.5 Joining multiple DataFrame or Panel objects

A list or tuple of DataFrames can also be passed to DataFrame.join to join them together on their indexes. The same is true for Panel.join.
In [711]: df1 = df.ix[:, [A, B]] In [712]: df2 = df.ix[:, [C, D]] In [713]: df3 = df.ix[:, [key]] In [714]: df1 Out[714]: A B 0 -0.308853 -0.681087 1 -2.461467 -1.553902 2 1.771740 -0.670027 3 -3.201750 0.792716 4 -0.747169 -0.309038 5 0.936527 1.255746 6 0.062297 -0.110388 7 0.077849 0.629498 In [715]: df1.join([df2, df3]) Out[715]: A B C D 0 -0.308853 -0.681087 0.377953 0.493672 1 -2.461467 -1.553902 2.015523 -1.833722 2 1.771740 -0.670027 0.049307 -0.521493 3 -3.201750 0.792716 0.146111 1.903247

key foo bar foo bar

144

Chapter 11. Merge, join, and concatenate

pandas: powerful Python data analysis toolkit, Release 0.7.1

4 -0.747169 -0.309038 0.393876 1.861468 5 0.936527 1.255746 -2.655452 1.219492 6 0.062297 -0.110388 -1.184357 -0.558081 7 0.077849 0.629498 -1.035260 -0.438229

foo bar foo bar

11.2. Database-style DataFrame joining/merging

145

pandas: powerful Python data analysis toolkit, Release 0.7.1

146

Chapter 11. Merge, join, and concatenate

CHAPTER

TWELVE

RESHAPING AND PIVOT TABLES

12.1 Reshaping by pivoting DataFrame objects
Data is often stored in CSV les or databases in so-called stacked or record format:
In [760]: df Out[760]: 0 1 2 3 4 5 6 7 8 9 10 11 2000-01-03 2000-01-04 2000-01-05 2000-01-03 2000-01-04 2000-01-05 2000-01-03 2000-01-04 2000-01-05 2000-01-03 2000-01-04 2000-01-05 date variable value 00:00:00 A 0.469112 00:00:00 A -0.282863 00:00:00 A -1.509059 00:00:00 B -1.135632 00:00:00 B 1.212112 00:00:00 B -0.173215 00:00:00 C 0.119209 00:00:00 C -1.044236 00:00:00 C -0.861849 00:00:00 D -2.104569 00:00:00 D -0.494929 00:00:00 D 1.071804

For the curious here is how the above DataFrame was created:
import pandas.util.testing as tm; tm.N = 3 def unpivot(frame): N, K = frame.shape data = {value : frame.values.ravel(F), variable : np.asarray(frame.columns).repeat(N), date : np.tile(np.asarray(frame.index), K)} return DataFrame(data, columns=[date, variable, value]) df = unpivot(tm.makeTimeDataFrame())

To select out everything for variable A we could do:

In [761]: df[df[variable] == A] Out[761]: date variable value 0 2000-01-03 00:00:00 A 0.469112 1 2000-01-04 00:00:00 A -0.282863 2 2000-01-05 00:00:00 A -1.509059

But suppose we wish to do time series operations with the variables. A better representation would be where the columns are the unique variables and an index of dates identies individual observations. To reshape the data into this form, use the pivot function:

147

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [762]: df.pivot(index=date, columns=variable, values=value) Out[762]: variable A B C D date 2000-01-03 0.469112 -1.135632 0.119209 -2.104569 2000-01-04 -0.282863 1.212112 -1.044236 -0.494929 2000-01-05 -1.509059 -0.173215 -0.861849 1.071804

If the values argument is omitted, and the input DataFrame has more than one column of values which are not used as column or index inputs to pivot, then the resulting pivoted DataFrame will have hierarchical columns whose topmost level indicates the respective value column:
In [763]: df[value2] = df[value] * 2 In [764]: pivoted = df.pivot(date, variable) In [765]: pivoted Out[765]: value value2 variable A B C D A B C D date 2000-01-03 0.469112 -1.135632 0.119209 -2.104569 0.938225 -2.271265 0.238417 -4.209138 2000-01-04 -0.282863 1.212112 -1.044236 -0.494929 -0.565727 2.424224 -2.088472 -0.989859 2000-01-05 -1.509059 -0.173215 -0.861849 1.071804 -3.018117 -0.346429 -1.723698 2.143608

You of course can then select subsets from the pivoted DataFrame:
In [766]: pivoted[value2] Out[766]: variable A B C D date 2000-01-03 0.938225 -2.271265 0.238417 -4.209138 2000-01-04 -0.565727 2.424224 -2.088472 -0.989859 2000-01-05 -3.018117 -0.346429 -1.723698 2.143608

Note that this returns a view on the underlying data in the case where the data are homogeneously-typed.

12.2 Reshaping by stacking and unstacking

Closely related to the pivot function are the related stack and unstack functions currently available on Series and DataFrame. These functions are designed to work together with MultiIndex objects (see the section on hierarchical indexing). Here are essentially what these functions do: stack: pivot a level of the (possibly hierarchical) column labels, returning a DataFrame with an index with a new inner-most level of row labels. unstack: inverse operation from stack: pivot a level of the (possibly hierarchical) row index to the column axis, producing a reshaped DataFrame with a new inner-most level of column labels. The clearest way to explain is by example. Lets take a prior example data set from the hierarchical indexing section:
In [767]: tuples = zip(*[[bar, .....: foo, .....: [one, .....: one, bar, foo, two, two, baz, qux, one, one, baz, qux], two, two]])

In [768]: index = MultiIndex.from_tuples(tuples, names=[first, second])

148

Chapter 12. Reshaping and Pivot Tables

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [769]: df = DataFrame(randn(8, 2), index=index, columns=[A, B]) In [770]: df2 = df[:4] In [771]: df2 Out[771]: A B first second bar one 0.721555 -0.706771 two -1.039575 0.271860 baz one -0.424972 0.567020 two 0.276232 -1.087401

The stack function compresses a level in the DataFrames columns to produce either: A Series, in the case of a simple column Index A DataFrame, in the case of a MultiIndex in the columns If the columns have a MultiIndex, you can choose which level to stack. The stacked level becomes the new lowest level in a MultiIndex on the columns:
In [772]: stacked = df2.stack() In [773]: stacked Out[773]: first second bar one A B two A B baz one A B two A B

0.721555 -0.706771 -1.039575 0.271860 -0.424972 0.567020 0.276232 -1.087401

With a stacked DataFrame or Series (having a MultiIndex as the index), the inverse operation of stack is unstack, which by default unstacks the last level:
In [774]: stacked.unstack() Out[774]: A B first second bar one 0.721555 -0.706771 two -1.039575 0.271860 baz one -0.424972 0.567020 two 0.276232 -1.087401 In [775]: stacked.unstack(1) Out[775]: second one two first bar A 0.721555 -1.039575 B -0.706771 0.271860 baz A -0.424972 0.276232 B 0.567020 -1.087401 In [776]: stacked.unstack(0) Out[776]: first bar baz

12.2. Reshaping by stacking and unstacking

149

pandas: powerful Python data analysis toolkit, Release 0.7.1

second one A 0.721555 -0.424972 B -0.706771 0.567020 two A -1.039575 0.276232 B 0.271860 -1.087401

If the indexes have names, you can use the level names instead of specifying the level numbers:
In [777]: stacked.unstack(second) Out[777]: second one two first bar A 0.721555 -1.039575 B -0.706771 0.271860 baz A -0.424972 0.276232 B 0.567020 -1.087401

You may also stack or unstack more than one level at a time by passing a list of levels, in which case the end result is as if each level in the list were processed individually. These functions are intelligent about handling missing data and do not expect each subgroup within the hierarchical index to have the same set of labels. They also can handle the index being unsorted (but you can make it sorted by calling sortlevel, of course). Here is a more complex example:
In [778]: columns = MultiIndex.from_tuples([(A, cat), (B, dog), .....: (B, cat), (A, dog)], .....: names=[exp, animal]) In [779]: df = DataFrame(randn(8, 4), index=index, columns=columns) In [780]: df2 = df.ix[[0, 1, 2, 4, 5, 7]] In [781]: df2 Out[781]: exp A animal cat first second bar one -0.370647 two 1.075770 baz one 0.357021 foo one -0.013960 two 0.895717 qux two 0.410835

B dog -1.157892 -0.109050 -0.674600 -0.362543 0.805244 0.813850

cat -1.344312 1.643563 -1.776904 -0.006154 -1.206412 0.132003

A dog 0.844885 -1.469388 -0.968914 -0.923061 2.565646 -0.827317

As mentioned above, stack can be called with a level argument to select which level in the columns to stack:
In [782]: df2.stack(exp) Out[782]: animal cat first second exp bar one A -0.370647 B -1.344312 two A 1.075770 B 1.643563 baz one A 0.357021 B -1.776904 foo A -0.013960 B -0.006154 two A 0.895717

dog 0.844885 -1.157892 -1.469388 -0.109050 -0.968914 -0.674600 -0.923061 -0.362543 2.565646

150

Chapter 12. Reshaping and Pivot Tables

pandas: powerful Python data analysis toolkit, Release 0.7.1

qux

B A B

-1.206412 0.805244 0.410835 -0.827317 0.132003 0.813850

In [783]: df2.stack(animal) Out[783]: exp A first second animal bar one cat -0.370647 dog 0.844885 two cat 1.075770 dog -1.469388 baz one cat 0.357021 dog -0.968914 foo cat -0.013960 dog -0.923061 two cat 0.895717 dog 2.565646 qux cat 0.410835 dog -0.827317

B -1.344312 -1.157892 1.643563 -0.109050 -1.776904 -0.674600 -0.006154 -0.362543 -1.206412 0.805244 0.132003 0.813850

Unstacking when the columns are a MultiIndex is also careful about doing the right thing:
In [784]: df[:3].unstack(0) Out[784]: exp A B A animal cat dog cat dog first bar baz bar baz bar baz bar baz second one -0.370647 0.357021 -1.157892 -0.6746 -1.344312 -1.776904 0.844885 -0.968914 two 1.075770 NaN -0.109050 NaN 1.643563 NaN -1.469388 NaN In [785]: df2.unstack(1) Out[785]: exp A B A animal cat dog cat dog second one two one two one two one two first bar -0.370647 1.075770 -1.157892 -0.109050 -1.344312 1.643563 0.844885 -1.469388 baz 0.357021 NaN -0.674600 NaN -1.776904 NaN -0.968914 NaN foo -0.013960 0.895717 -0.362543 0.805244 -0.006154 -1.206412 -0.923061 2.565646 qux NaN 0.410835 NaN 0.813850 NaN 0.132003 NaN -0.827317

12.3 Reshaping by Melt

The melt function found in pandas.core.reshape is useful to massage a DataFrame into a format where one or more columns are identier variables, while all other columns, considered measured variables, are pivoted to the row axis, leaving just two non-identier columns, variable and value. For instance,
In [786]: cheese = DataFrame({first : [John, Mary], .....: last : [Doe, Bo], .....: height : [5.5, 6.0], .....: weight : [130, 150]})

12.3. Reshaping by Melt

151

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [787]: cheese Out[787]: first height last 0 John 5.5 Doe 1 Mary 6.0 Bo In [788]: melt(cheese, Out[788]: first last variable 0 John Doe height 1 Mary Bo height 2 John Doe weight 3 Mary Bo weight

weight 130 150 id_vars=[first, last]) value 5.5 6.0 130.0 150.0

12.4 Combining with stats and GroupBy

It should be no shock that combining pivot / stack / unstack with GroupBy and the basic Series and DataFrame statistical functions can produce some very expressive and fast data manipulations.
In [789]: df Out[789]: exp A B A animal cat dog cat dog first second bar one -0.370647 -1.157892 -1.344312 0.844885 two 1.075770 -0.109050 1.643563 -1.469388 baz one 0.357021 -0.674600 -1.776904 -0.968914 two -1.294524 0.413738 0.276662 -0.472035 foo one -0.013960 -0.362543 -0.006154 -0.923061 two 0.895717 0.805244 -1.206412 2.565646 qux one 1.431256 1.340309 -1.170299 -0.226169 two 0.410835 0.813850 0.132003 -0.827317 In [790]: df.stack().mean(1).unstack() Out[790]: animal cat dog first second bar one -0.857479 -0.156504 two 1.359666 -0.789219 baz one -0.709942 -0.821757 two -0.508931 -0.029148 foo one -0.010057 -0.642802 two -0.155347 1.685445 qux one 0.130479 0.557070 two 0.271419 -0.006733 # same result, another way In [791]: df.groupby(level=1, axis=1).mean() Out[791]: animal cat dog first second bar one -0.857479 -0.156504 two 1.359666 -0.789219 baz one -0.709942 -0.821757 two -0.508931 -0.029148 foo one -0.010057 -0.642802

152

Chapter 12. Reshaping and Pivot Tables

pandas: powerful Python data analysis toolkit, Release 0.7.1

qux

two one two

-0.155347 1.685445 0.130479 0.557070 0.271419 -0.006733

In [792]: df.stack().groupby(level=1).mean() Out[792]: exp A B second one 0.016301 -0.644049 two 0.110588 0.346200 In [793]: df.mean().unstack(0) Out[793]: exp A B animal cat 0.311433 -0.431481 dog -0.184544 0.133632

12.5 Pivot tables and cross-tabulations

The function pandas.pivot_table can be used to create spreadsheet-style pivot tables. It takes a number of arguments data: A DataFrame object values: a column or a list of columns to aggregate rows: list of columns to group by on the table rows cols: list of columns to group by on the table columns aggfunc: function to use for aggregation, defaulting to numpy.mean Consider a data set like this:
In [794]: df = DataFrame({A : [one, one, two, three] * 6, .....: B : [A, B, C] * 8, .....: C : [foo, foo, foo, bar, bar, bar] * 4, .....: D : np.random.randn(24), .....: E : np.random.randn(24)}) In [795]: df Out[795]: A B 0 one A 1 one B 2 two C 3 three A 4 one B 5 one C 6 two A 7 three B 8 one C 9 one A 10 two B 11 three C 12 one A 13 one B

C foo foo foo bar bar bar foo foo foo bar bar bar foo foo

D -0.076467 -1.187678 1.130127 -1.436737 -1.413681 1.607920 1.024180 0.569605 0.875906 -2.211372 0.974466 -2.006747 -0.410001 -0.078638

E 0.959726 -1.110336 -0.619976 0.149748 -0.732339 0.687738 0.176444 0.403310 -0.154951 0.301624 -2.179861 -1.369849 -0.954208 1.462696

12.5. Pivot tables and cross-tabulations

153

pandas: powerful Python data analysis toolkit, Release 0.7.1

14 15 16 17 18 19 20 21 22 23

two three one one two three one one two three

C A B C A B C A B C

foo bar bar bar foo foo foo bar bar bar

0.545952 -1.219217 -1.226825 0.769804 -1.281247 -0.727707 -0.121306 -0.097883 0.695775 0.341734

-1.743161 -0.826591 -0.345352 1.314232 0.690579 0.995761 2.396780 0.014871 3.357427 -0.317441

We can produce pivot tables from this data very easily:

In [796]: pivot_table(df, values=D, rows=[A, B], cols=[C]) Out[796]: C bar foo A B one A -1.154627 -0.243234 B -1.320253 -0.633158 C 1.188862 0.377300 three A -1.327977 NaN B NaN -0.079051 C -0.832506 NaN two A NaN -0.128534 B 0.835120 NaN C NaN 0.838040 In [797]: pivot_table(df, values=D, rows=[B], cols=[A, C], aggfunc=np.sum) Out[797]: A one three two C bar foo bar foo bar foo B A -2.309255 -0.486468 -2.655954 NaN NaN -0.257067 B -2.640506 -1.266315 NaN -0.158102 1.670241 NaN C 2.377724 0.754600 -1.665013 NaN NaN 1.676079 In [798]: pivot_table(df, values=[D,E], rows=[B], cols=[A, C], aggfunc=np.sum) Out[798]: <class pandas.core.frame.DataFrame> Index: 3 entries, A to C Data columns: (D, one, bar) 3 non-null values (D, one, foo) 3 non-null values (D, three, bar) 2 non-null values (D, three, foo) 1 non-null values (D, two, bar) 1 non-null values (D, two, foo) 2 non-null values (E, one, bar) 3 non-null values (E, one, foo) 3 non-null values (E, three, bar) 2 non-null values (E, three, foo) 1 non-null values (E, two, bar) 1 non-null values (E, two, foo) 2 non-null values dtypes: float64(12)

The result object is a DataFrame having potentially hierarchical indexes on the rows and columns. If the values column name is not given, the pivot table will include all of the data that can be aggregated in an additional level of hierarchy in the columns:

154

Chapter 12. Reshaping and Pivot Tables

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [799]: pivot_table(df, rows=[A, B], cols=[C]) Out[799]: D E C bar foo bar foo A B one A -1.154627 -0.243234 0.158248 0.002759 B -1.320253 -0.633158 -0.538846 0.176180 C 1.188862 0.377300 1.000985 1.120915 three A -1.327977 NaN -0.338421 NaN B NaN -0.079051 NaN 0.699535 C -0.832506 NaN -0.843645 NaN two A NaN -0.128534 NaN 0.433512 B 0.835120 NaN 0.588783 NaN C NaN 0.838040 NaN -1.181568

You can render a nice output of the table omitting the missing values by calling to_string if you wish:
In [800]: table = pivot_table(df, rows=[A, B], cols=[C]) In [801]: print table.to_string(na_rep=) D E C bar foo bar foo A B one A -1.154627 -0.243234 0.158248 0.002759 B -1.320253 -0.633158 -0.538846 0.176180 C 1.188862 0.377300 1.000985 1.120915 three A -1.327977 -0.338421 B -0.079051 0.699535 C -0.832506 -0.843645 two A -0.128534 0.433512 B 0.835120 0.588783 C 0.838040 -1.181568

Note that pivot_table is also available as an instance method on DataFrame.

12.5.1 Cross tabulations

Use the crosstab function to compute a cross-tabulation of two (or more) factors. By default crosstab computes a frequency table of the factors unless an array of values and an aggregation function are passed. It takes a number of arguments rows: array-like, values to group by in the rows cols: array-like, values to group by in the columns values: array-like, optional, array of values to aggregate according to the factors aggfunc: function, optional, If no values array is passed, computes a frequency table rownames: sequence, default None, must match number of row arrays passed colnames: sequence, default None, if passed, must match number of column arrays passed margins: boolean, default False, Add row/column margins (subtotals) Any Series passed will have their name attributes used unless row or column names for the cross-tabulation are specied For example:

12.5. Pivot tables and cross-tabulations

155

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [802]: foo, bar, dull, shiny, one, two = foo, bar, dull, shiny, one, two In [803]: a = np.array([foo, foo, bar, bar, foo, foo], dtype=object) In [804]: b = np.array([one, one, two, one, two, one], dtype=object) In [805]: c = np.array([dull, dull, shiny, dull, dull, shiny], dtype=object) In [806]: crosstab(a, [b, c], rownames=[a], colnames=[b, c]) Out[806]: b one two c dull shiny dull shiny a bar 1 0 0 1 foo 2 1 1 0

12.5.2 Adding margins (partial aggregates)

If you pass margins=True to pivot_table, special All columns and rows will be added with partial group aggregates across the categories on the rows and columns:
In [807]: df.pivot_table(rows=[A, B], cols=C, margins=True, aggfunc=np.std) Out[807]: D E C bar foo All bar foo All A B one A 1.494463 0.235844 1.019752 0.202765 1.353355 0.795165 B 0.132127 0.784210 0.606779 0.273641 1.819408 1.139647 C 0.592638 0.705136 0.708771 0.442998 1.804346 1.074910 three A 0.153810 NaN 0.153810 0.690376 NaN 0.690376 B NaN 0.917338 0.917338 NaN 0.418926 0.418926 C 1.660627 NaN 1.660627 0.744165 NaN 0.744165 two A NaN 1.630183 1.630183 NaN 0.363548 0.363548 B 0.197065 NaN 0.197065 3.915454 NaN 3.915454 C NaN 0.413074 0.413074 NaN 0.794212 0.794212 All 1.294620 0.824989 1.087016 1.403041 1.188419 1.275851

156

Chapter 12. Reshaping and Pivot Tables

CHAPTER

THIRTEEN

TIME SERIES / DATE FUNCTIONALITY

pandas has proven very successful as a tool for working with time series data, especially in the nancial data analysis space. Over the coming year we will be looking to consolidate the various Python libraries for time series data, e.g. scikits.timeseries, using the new NumPy datetime64 dtype, to create a very nice integrated solution. Everything in pandas at the moment is based on using Python datetime objects. In working with time series data, we will frequently seek to: generate sequences of xed-frequency dates conform or convert time series to a particular frequency compute relative dates based on various non-standard time increments (e.g. 5 business days before the last business day of the year), or roll dates forward or backward pandas provides a relatively compact and self-contained set of tools for performing the above tasks. Note: This area of pandas has gotten less development attention recently, though this should change in the near future.

13.1 DateOffset objects

A DateOffset instance represents a frequency increment. Different offset logic via subclasses: Class name DateOffset BDay Week MonthEnd BMonthEnd QuarterEnd BQuarterEnd YearEnd YearBegin BYearEnd Hour Minute Second Description Generic offset class, defaults to 1 calendar day business day (weekday) one week, optionally anchored on a day of the week calendar month end business month end calendar quarter end business quarter end calendar year end calendar year begin business year end one hour one minute one second

The basic DateOffset takes the same arguments as dateutil.relativedelta, which works like:

157

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [831]: d = datetime(2008, 8, 18) In [832]: d + relativedelta(months=4, days=5) Out[832]: datetime.datetime(2008, 12, 23, 0, 0)

We could have done the same thing with DateOffset:

In [833]: d + DateOffset(months=4, days=5) Out[833]: datetime.datetime(2008, 12, 23, 0, 0)

The key features of a DateOffset object are: it can be added / subtracted to/from a datetime object to obtain a shifted date it can be multiplied by an integer (positive or negative) so that the increment will be applied multiple times it has rollforward and rollback methods for moving a date forward or backward to the next or previous offset date Subclasses of DateOffset dene the apply function which dictates custom date increment logic, such as adding business days:
class BDay(DateOffset): """DateOffset increments between business days""" def apply(self, other): ... In [834]: d - 5 * BDay() Out[834]: datetime.datetime(2008, 8, 11, 0, 0) In [835]: d + BMonthEnd() Out[835]: datetime.datetime(2008, 8, 29, 0, 0)

The rollforward and rollback methods do exactly what you would expect:
In [836]: d Out[836]: datetime.datetime(2008, 8, 18, 0, 0) In [837]: offset = BMonthEnd() In [838]: offset.rollforward(d) Out[838]: datetime.datetime(2008, 8, 29, 0, 0) In [839]: offset.rollback(d) Out[839]: datetime.datetime(2008, 7, 31, 0, 0)

Its denitely worth exploring the pandas.core.datetools module and the various docstrings for the classes.

13.1.1 Parametric offsets

Some of the offsets can be parameterized when created to result in different behavior. For example, the Week offset for generating weekly data accepts a weekday parameter which results in the generated dates always lying on a particular day of the week:
In [840]: d + Week() Out[840]: datetime.datetime(2008, 8, 25, 0, 0) In [841]: d + Week(weekday=4) Out[841]: datetime.datetime(2008, 8, 22, 0, 0)

158

Chapter 13. Time Series / Date functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [842]: (d + Week(weekday=4)).weekday() Out[842]: 4

13.1.2 Time rules

A number of string aliases are given to useful common time series frequencies. We will refer to these aliases as time rules. Rule name WEEKDAY EOM W@MON W@TUE W@WED W@THU W@FRI Q@JAN Q@FEB Q@MAR A@DEC A@JAN A@FEB A@MAR A@APR A@MAY A@JUN A@JUL A@AUG A@SEP A@OCT A@NOV Description business day frequency business month end frequency weekly frequency (mondays) weekly frequency (tuesdays) weekly frequency (wednesdays) weekly frequency (thursdays) weekly frequency (fridays) quarterly frequency, starting January quarterly frequency, starting February quarterly frequency, starting March annual frequency, year end (December) annual frequency, anchored end of January annual frequency, anchored end of February annual frequency, anchored end of March annual frequency, anchored end of April annual frequency, anchored end of May annual frequency, anchored end of June annual frequency, anchored end of July annual frequency, anchored end of August annual frequency, anchored end of September annual frequency, anchored end of October annual frequency, anchored end of November

These can be used as arguments to DateRange and various other time series-related functions in pandas.

13.2 Generating date ranges (DateRange)

The DateRange class utilizes these offsets (and any ones that we might add) to generate xed-frequency date ranges:
In [843]: start = datetime(2009, 1, 1) In [844]: end = datetime(2010, 1, 1) In [845]: rng = DateRange(start, end, offset=BDay()) In [846]: rng Out[846]: <class pandas.core.daterange.DateRange> offset: <1 BusinessDay>, tzinfo: None [2009-01-01 00:00:00, ..., 2010-01-01 00:00:00] length: 262 In [847]: DateRange(start, end, offset=BMonthEnd()) Out[847]:

13.2. Generating date ranges (DateRange)

159

pandas: powerful Python data analysis toolkit, Release 0.7.1

<class pandas.core.daterange.DateRange> offset: <1 BusinessMonthEnd>, tzinfo: None [2009-01-30 00:00:00, ..., 2009-12-31 00:00:00] length: 12

Business day frequency is the default for DateRange. You can also strictly generate a DateRange of a certain length by providing either a start or end date and a periods argument:
In [848]: DateRange(start, periods=20) Out[848]: <class pandas.core.daterange.DateRange> offset: <1 BusinessDay>, tzinfo: None [2009-01-01 00:00:00, ..., 2009-01-28 00:00:00] length: 20 In [849]: DateRange(end=end, periods=20) Out[849]: <class pandas.core.daterange.DateRange> offset: <1 BusinessDay>, tzinfo: None [2009-12-07 00:00:00, ..., 2010-01-01 00:00:00] length: 20

The start and end dates are strictly inclusive. So it will not generate any dates outside of those dates if specied.

13.2.1 DateRange is a valid Index

One of the main uses for DateRange is as an index for pandas objects. When working with a lot of time series data, there are several reasons to use DateRange objects when possible: A large range of dates for various offsets are pre-computed and cached under the hood in order to make generating subsequent date ranges very fast (just have to grab a slice) Fast shifting using the shift method on pandas objects Unioning of overlapping DateRange objects with the same frequency is very fast (important for fast data alignment) The DateRange is a valid index and can even be intelligent when doing slicing, etc.
In [850]: rng = DateRange(start, end, offset=BMonthEnd()) In [851]: ts = Series(randn(len(rng)), index=rng) In [852]: ts.index Out[852]: <class pandas.core.daterange.DateRange> offset: <1 BusinessMonthEnd>, tzinfo: None [2009-01-30 00:00:00, ..., 2009-12-31 00:00:00] length: 12 In [853]: ts[:5].index Out[853]: <class pandas.core.daterange.DateRange> offset: <1 BusinessMonthEnd>, tzinfo: None [2009-01-30 00:00:00, ..., 2009-05-29 00:00:00] length: 5 In [854]: ts[::2].index Out[854]:

160

Chapter 13. Time Series / Date functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

<class pandas.core.daterange.DateRange> offset: <2 BusinessMonthEnds>, tzinfo: None [2009-01-30 00:00:00, ..., 2009-11-30 00:00:00] length: 6

More complicated fancy indexing will result in an Index that is no longer a DateRange, however:
In [855]: ts[[0, 2, 6]].index Out[855]: Index([2009-01-30 00:00:00, 2009-03-31 00:00:00, 2009-07-31 00:00:00], dtype=object)

13.3 Time series-related instance methods

See Also: Reindexing methods Note: While pandas does not force you to have a sorted date index, some of these methods may have unexpected or incorrect behavior if the dates are unsorted. So please be careful.

13.3.1 Shifting / lagging

One may want to shift or lag the values in a TimeSeries back and forward in time. The method for this is shift, which is available on all of the pandas objects. In DataFrame, shift will currently only shift along the index and in Panel along the major_axis.
In [856]: ts = ts[:5] In [857]: ts.shift(1) Out[857]: 2009-01-30 NaN 2009-02-27 0.469112 2009-03-31 -0.282863 2009-04-30 -1.509059 2009-05-29 -1.135632

The shift method accepts an offset argument which can accept a DateOffset class or other timedelta-like object or also a time rule:
In [858]: ts.shift(5, offset=datetools.bday) Out[858]: 2009-02-06 0.469112 2009-03-06 -0.282863 2009-04-07 -1.509059 2009-05-07 -1.135632 2009-06-05 1.212112 In [859]: ts.shift(5, offset=EOM) Out[859]: 2009-06-30 0.469112 2009-07-31 -0.282863 2009-08-31 -1.509059 2009-09-30 -1.135632 2009-10-30 1.212112

13.3. Time series-related instance methods

161

pandas: powerful Python data analysis toolkit, Release 0.7.1

13.3.2 Frequency conversion

The primary function for changing frequencies is the asfreq function. This is basically just a thin, but convenient wrapper around reindex which generates a DateRange and calls reindex.
In [860]: dr = DateRange(1/1/2010, periods=3, offset=3 * datetools.bday) In [861]: ts = Series(randn(3), index=dr) In [862]: ts Out[862]: 2010-01-01 0.721555 2010-01-06 -0.706771 2010-01-11 -1.039575 In [863]: ts.asfreq(BDay()) Out[863]: 2010-01-01 0.721555 2010-01-04 NaN 2010-01-05 NaN 2010-01-06 -0.706771 2010-01-07 NaN 2010-01-08 NaN 2010-01-11 -1.039575 In [864]: ts.asfreq(BDay(), method=pad) Out[864]: 2010-01-01 0.721555 2010-01-04 0.721555 2010-01-05 0.721555 2010-01-06 -0.706771 2010-01-07 -0.706771 2010-01-08 -0.706771 2010-01-11 -1.039575

13.3.3 Filling forward / backward

Related to asfreq and reindex is the fillna function documented in the missing data section.

13.4 Up- and downsampling

We plan to add some efcient methods for doing resampling during frequency conversion. For example, converting secondly data into 5-minutely data. This is extremely common in, but not limited to, nancial applications. Until then, your best bet is a clever (or kludgy, depending on your point of view) application of GroupBy. Carry out the following steps: 1. Generate the target DateRange of interest
dr1hour = DateRange(start, end, offset=Hour()) dr5day = DateRange(start, end, offset=5 * datetools.day) dr10day = DateRange(start, end, offset=10 * datetools.day)

2. Use the asof function (as of) of the DateRange to do a groupby expression

162

Chapter 13. Time Series / Date functionality

pandas: powerful Python data analysis toolkit, Release 0.7.1

grouped = data.groupby(dr5day.asof) means = grouped.mean()

Here is a fully-worked example:

# some minutely data In [865]: minutely = DateRange(1/3/2000 00:00:00, 1/3/2000 12:00:00, .....: offset=datetools.Minute()) In [866]: ts = Series(randn(len(minutely)), index=minutely) In [867]: ts.index Out[867]: <class pandas.core.daterange.DateRange> offset: <1 Minute>, tzinfo: None [2000-01-03 00:00:00, ..., 2000-01-03 12:00:00] length: 721 In [868]: hourly = DateRange(1/3/2000, 1/4/2000, offset=datetools.Hour()) In [869]: grouped = ts.groupby(hourly.asof) In [870]: grouped.mean() Out[870]: key_0 2000-01-03 00:00:00 -0.119068 2000-01-03 01:00:00 0.020282 2000-01-03 02:00:00 0.102562 2000-01-03 03:00:00 -0.106713 2000-01-03 04:00:00 -0.128935 2000-01-03 05:00:00 -0.146319 2000-01-03 06:00:00 -0.002938 2000-01-03 07:00:00 -0.131361 2000-01-03 08:00:00 -0.005749 2000-01-03 09:00:00 -0.399136 2000-01-03 10:00:00 0.097238 2000-01-03 11:00:00 -0.127307 2000-01-03 12:00:00 -0.273955 Name: result

Some things to note about this method: This is rather inefcient because we havent exploited the orderedness of the data at all. Calling the asof function on every date in the minutely time series is not strictly necessary. Well be writing some signicantly more efcient methods in the near future The dates in the result mark the beginning of the period. Be careful about which convention you use; you dont want to end up misaligning data because you used the wrong upsampling convention

13.4. Up- and downsampling

163

pandas: powerful Python data analysis toolkit, Release 0.7.1

164

Chapter 13. Time Series / Date functionality

CHAPTER

FOURTEEN

PLOTTING WITH MATPLOTLIB

Note: We intend to build more plotting integration with matplotlib as time goes on. We use the standard convention for referencing the matplotlib API:
In [871]: import matplotlib.pyplot as plt

14.1 Basic plotting: plot

The plot method on Series and DataFrame is just a simple wrapper around plt.plot:
In [872]: ts = Series(randn(1000), index=DateRange(1/1/2000, periods=1000)) In [873]: ts = ts.cumsum() In [874]: ts.plot() Out[874]: <matplotlib.axes.AxesSubplot at 0x114967f90>

165

pandas: powerful Python data analysis toolkit, Release 0.7.1

If the index consists of dates, it calls gca().autofmt_xdate() to try to format the x-axis nicely as per above. The method takes a number of arguments for controlling the look of the plot:
In [875]: plt.figure(); ts.plot(style=k--, label=Series); plt.legend() Out[875]: <matplotlib.legend.Legend at 0x116fe2e10>

On DataFrame, plot is a convenience to plot all of the columns with labels:

In [876]: df = DataFrame(randn(1000, 4), index=ts.index, .....: columns=[A, B, C, D]) In [877]: df = df.cumsum() In [878]: plt.figure(); df.plot(); plt.legend(loc=best) Out[878]: <matplotlib.legend.Legend at 0x118a43990>

166

Chapter 14. Plotting with matplotlib

pandas: powerful Python data analysis toolkit, Release 0.7.1

You may set the legend argument to False to hide the legend, which is shown by default.
In [879]: df.plot(legend=False) Out[879]: <matplotlib.axes.AxesSubplot at 0x118bd4790>

Some other options are available, like plotting each Series on a different axis:
In [880]: df.plot(subplots=True, figsize=(8, 8)); plt.legend(loc=best) Out[880]: <matplotlib.legend.Legend at 0x118bd0ad0>

14.1. Basic plotting: plot

167

pandas: powerful Python data analysis toolkit, Release 0.7.1

You may pass logy to get a log-scale Y axis.

In [881]: plt.figure(); In [881]: ts = Series(randn(1000), index=DateRange(1/1/2000, periods=1000)) In [882]: ts = np.exp(ts.cumsum()) In [883]: ts.plot(logy=True) Out[883]: <matplotlib.axes.AxesSubplot at 0x118faa790>

168

Chapter 14. Plotting with matplotlib

pandas: powerful Python data analysis toolkit, Release 0.7.1

14.1.1 Targeting different subplots

You can pass an ax argument to Series.plot to plot on a particular axis:
In [884]: fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(8, 5)) In [885]: df[A].plot(ax=axes[0,0]); axes[0,0].set_title(A) Out[885]: <matplotlib.text.Text at 0x1132aafd0> In [886]: df[B].plot(ax=axes[0,1]); axes[0,1].set_title(B) Out[886]: <matplotlib.text.Text at 0x10c4c7850> In [887]: df[C].plot(ax=axes[1,0]); axes[1,0].set_title(C) Out[887]: <matplotlib.text.Text at 0x119acfc10> In [888]: df[D].plot(ax=axes[1,1]); axes[1,1].set_title(D) Out[888]: <matplotlib.text.Text at 0x119bb78d0>

14.1. Basic plotting: plot

169

pandas: powerful Python data analysis toolkit, Release 0.7.1

14.2 Other plotting features

14.2.1 Plotting non-time series data
For labeled, non-time series data, you may wish to produce a bar plot:
In [889]: plt.figure(); In [889]: df.ix[5].plot(kind=bar); plt.axhline(0, color=k) Out[889]: <matplotlib.lines.Line2D at 0x119c071d0>

170

Chapter 14. Plotting with matplotlib

pandas: powerful Python data analysis toolkit, Release 0.7.1

14.2.2 Histogramming
In [890]: plt.figure(); In [890]: df[A].diff().hist() Out[890]: <matplotlib.axes.AxesSubplot at 0x119c30550>

For a DataFrame, hist plots the histograms of the columns on multiple subplots:
In [891]: plt.figure() Out[891]: <matplotlib.figure.Figure at 0x119c53dd0> In [892]: df.diff().hist(color=k, alpha=0.5, bins=50) Out[892]: array([[Axes(0.125,0.536364;0.352273x0.363636), Axes(0.547727,0.536364;0.352273x0.363636)], [Axes(0.125,0.1;0.352273x0.363636), Axes(0.547727,0.1;0.352273x0.363636)]], dtype=object)

14.2. Other plotting features

171

pandas: powerful Python data analysis toolkit, Release 0.7.1

14.2.3 Box-Plotting
DataFrame has a boxplot method which allows you to visualize the distribution of values within each column. For instance, here is a boxplot representing ve trials of 10 observations of a uniform random variable on [0,1).
In [893]: df = DataFrame(np.random.rand(10,5)) In [894]: plt.figure(); In [894]: df.boxplot() Out[894]: <matplotlib.axes.AxesSubplot at 0x119c5f610>

172

Chapter 14. Plotting with matplotlib

pandas: powerful Python data analysis toolkit, Release 0.7.1

You can create a stratied boxplot using the by keyword argument to create groupings. For instance,
In [895]: df = DataFrame(np.random.rand(10,2), columns=[Col1, Col2] ) In [896]: df[X] = Series([A,A,A,A,A,B,B,B,B,B]) In [897]: plt.figure(); In [897]: df.boxplot(by=X) Out[897]: array([Axes(0.1,0.15;0.363636x0.75), Axes(0.536364,0.15;0.363636x0.75)], dtype=object)

You can also pass a subset of columns to plot, as well as group by multiple columns:
In [898]: df = DataFrame(np.random.rand(10,3), columns=[Col1, Col2, Col3]) In [899]: df[X] = Series([A,A,A,A,A,B,B,B,B,B]) In [900]: df[Y] = Series([A,B,A,B,A,B,A,B,A,B]) In [901]: plt.figure(); In [901]: df.boxplot(column=[Col1,Col2], by=[X,Y]) Out[901]: array([Axes(0.1,0.15;0.363636x0.75), Axes(0.536364,0.15;0.363636x0.75)], dtype=object)

14.2. Other plotting features

173

pandas: powerful Python data analysis toolkit, Release 0.7.1

174

Chapter 14. Plotting with matplotlib

CHAPTER

FIFTEEN

IO TOOLS (TEXT, CSV, HDF5, ...)

15.1 Clipboard
A handy way to grab data is to use the read_clipboard method, which takes the contents of the clipboard buffer and passes them to the read_table method described in the next section. For instance, you can copy the following text to the clipboard (CTRL-C on many operating systems):
A B C x 1 4 p y 2 5 q z 3 6 r

And then import the data directly to a DataFrame by calling:

clipdf = read_clipboard(sep=\s*) In [585]: clipdf Out[585]: A B C x 1 4 p y 2 5 q z 3 6 r

15.2 CSV & Text les

The two workhorse functions for reading text les (a.k.a. at les) are read_csv() and read_table(). They both use the same parsing code to intelligently convert tabular data into a DataFrame object. They can take a number of arguments: path_or_buffer: Either a string path to a le, or any object with a read method (such as an open le or StringIO). sep or delimiter: A delimiter / separator to split elds on. read_csv is capable of inferring the delimiter automatically in some cases by snifng. The separator may be specied as a regular expression; for instance you may use s* to indicate arbitrary whitespace. header: row number to use as the column names, and the start of the data. Defaults to 0 (rst row); specify None if there is no header row. names: List of column names to use. If passed, header will be implicitly set to None. skiprows: A collection of numbers for rows in the le to skip. Can also be an integer to skip the rst n rows

175

pandas: powerful Python data analysis toolkit, Release 0.7.1

index_col: column number, or list of column numbers, to use as the index (row labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there is one more data column than there are headers, in which case the rst column is taken as the index. parse_dates: If True, attempt to parse the index column as dates. False by default. date_parser: function to use to parse strings into datetime objects. If parse_dates is True, it defaults to the very robust dateutil.parser. Specifying this implicitly sets parse_dates as True. na_values: optional list of strings to recognize as NaN (missing values), in addition to a default set. nrows: Number of rows to read out of the le. Useful to only read a small portion of a large le chunksize: An number of rows to be used to chunk a le into pieces. Will cause an TextParser object to be returned. More on this below in the section on iterating and chunking iterator: If True, return a TextParser to enable reading a le into memory piece by piece skip_footer: number of lines to skip at bottom of le (default 0) converters: a dictionary of functions for converting values in certain columns, where keys are either integers or column labels encoding: a string representing the encoding to use if the contents are non-ascii, for python versions prior to 3 verbose : show number of NA values inserted in non-numeric columns Consider a typical CSV le containing, in this case, some time series data:
In [586]: print open(foo.csv).read() date,A,B,C 20090101,a,1,2 20090102,b,3,4 20090103,c,4,5

The default for read_csv is to create a DataFrame with simple numbered rows:
In [587]: read_csv(foo.csv) Out[587]: date A B C 0 20090101 a 1 2 1 20090102 b 3 4 2 20090103 c 4 5

In the case of indexed data, you can pass the column number (or a list of column numbers, for a hierarchical index) you wish to use as the index. If the index values are dates and you want them to be converted to datetime objects, pass parse_dates=True:
# Use a column as an index, and parse it as dates. In [588]: df = read_csv(foo.csv, index_col=0, parse_dates=True) In [589]: df Out[589]: A date 2009-01-01 2009-01-02 2009-01-03 a b c B 1 3 4 C 2 4 5

# These are python datetime objects In [590]: df.index Out[590]: Index([2009-01-01 00:00:00, 2009-01-02 00:00:00, 2009-01-03 00:00:00], dtype=object)

176

Chapter 15. IO Tools (Text, CSV, HDF5, ...)

pandas: powerful Python data analysis toolkit, Release 0.7.1

The parsers make every attempt to do the right thing and not be very fragile. Type inference is a pretty big deal. So if a column can be coerced to integer dtype without altering the contents, it will do so. Any non-numeric columns will come through as object dtype as with the rest of pandas objects.

15.2.1 Files with an implicit index column

Consider a le with one less entry in the header than the number of data column:
In [591]: print open(foo.csv).read() A,B,C 20090101,a,1,2 20090102,b,3,4 20090103,c,4,5

In this special case, read_csv assumes that the rst column is to be used as the index of the DataFrame:
In [592]: read_csv(foo.csv) Out[592]: A B C 20090101 a 1 2 20090102 b 3 4 20090103 c 4 5

Note that the dates werent automatically parsed. In that case you would need to do as before:
In [593]: df = read_csv(foo.csv, parse_dates=True) In [594]: df.index Out[594]: Index([2009-01-01 00:00:00, 2009-01-02 00:00:00, 2009-01-03 00:00:00], dtype=object)

15.2.2 Reading DataFrame objects with MultiIndex

Suppose you have data indexed by two columns:
In [595]: print open(data/mindex_ex.csv).read() year,indiv,zit,xit 1977,"A",1.2,.6 1977,"B",1.5,.5 1977,"C",1.7,.8 1978,"A",.2,.06 1978,"B",.7,.2 1978,"C",.8,.3 1978,"D",.9,.5 1978,"E",1.4,.9 1979,"C",.2,.15 1979,"D",.14,.05 1979,"E",.5,.15 1979,"F",1.2,.5 1979,"G",3.4,1.9 1979,"H",5.4,2.7 1979,"I",6.4,1.2

The index_col argument to read_csv and read_table can take a list of column numbers to turn multiple columns into a MultiIndex:

15.2. CSV & Text les

177

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [596]: df = read_csv("data/mindex_ex.csv", index_col=[0,1]) In [597]: df Out[597]: zit year indiv 1977 A B C 1978 A B C D E 1979 C D E F G H I 1.20 1.50 1.70 0.20 0.70 0.80 0.90 1.40 0.20 0.14 0.50 1.20 3.40 5.40 6.40 xit 0.60 0.50 0.80 0.06 0.20 0.30 0.50 0.90 0.15 0.05 0.15 0.50 1.90 2.70 1.20

In [598]: df.ix[1978] Out[598]: zit xit indiv A 0.2 0.06 B 0.7 0.20 C 0.8 0.30 D 0.9 0.50 E 1.4 0.90

15.2.3 Automatically snifng the delimiter

read_csv is capable of inferring delimited (not necessarily comma-separated) les. YMMV, as pandas uses the Sniffer class of the csv module.
In [599]: print open(tmp2.sv).read() year:indiv:zit:xit 1977:A:1.2:0.6 1977:B:1.5:0.5 1977:C:1.7:0.8 1978:A:0.2:0.06 1978:B:0.7:0.2 1978:C:0.8:0.3 1978:D:0.9:0.5 In [600]: read_csv(tmp2.sv) Out[600]: year:indiv:zit:xit 0 1977:A:1.2:0.6 1 1977:B:1.5:0.5 2 1977:C:1.7:0.8 3 1978:A:0.2:0.06 4 1978:B:0.7:0.2 5 1978:C:0.8:0.3 6 1978:D:0.9:0.5

178

Chapter 15. IO Tools (Text, CSV, HDF5, ...)

pandas: powerful Python data analysis toolkit, Release 0.7.1

15.2.4 Iterating through les chunk by chunk

Suppose you wish to iterate through a (potentially very large) le lazily rather than reading the entire le into memory, such as the following:
In [601]: print open(tmp.sv).read() year|indiv|zit|xit 1977|A|1.2|0.6 1977|B|1.5|0.5 1977|C|1.7|0.8 1978|A|0.2|0.06 1978|B|0.7|0.2 1978|C|0.8|0.3 1978|D|0.9|0.5 In [602]: table = read_table(tmp.sv, sep=|) In [603]: table Out[603]: year indiv zit 0 1977 A 1.2 1 1977 B 1.5 2 1977 C 1.7 3 1978 A 0.2 4 1978 B 0.7 5 1978 C 0.8 6 1978 D 0.9

xit 0.60 0.50 0.80 0.06 0.20 0.30 0.50

By speciying a chunksize to read_csv or read_table, the return value will be an iterable object of type TextParser:
In [604]: reader = read_table(tmp.sv, sep=|, chunksize=4) In [605]: reader Out[605]: <pandas.io.parsers.TextParser at 0x1132ae390> In [606]: for chunk in reader: .....: print chunk .....: year indiv zit xit 0 1977 A 1.2 0.60 1 1977 B 1.5 0.50 2 1977 C 1.7 0.80 3 1978 A 0.2 0.06 year indiv zit xit 0 1978 B 0.7 0.2 1 1978 C 0.8 0.3 2 1978 D 0.9 0.5

Specifying iterator=True will also return the TextParser object:

In [607]: reader = read_table(tmp.sv, sep=|, iterator=True) In [608]: reader.get_chunk(5) Out[608]: year indiv zit xit 0 1977 A 1.2 0.60 1 1977 B 1.5 0.50 2 1977 C 1.7 0.80

15.2. CSV & Text les

179

pandas: powerful Python data analysis toolkit, Release 0.7.1

3 4

1978 1978

A B

0.2 0.7

0.06 0.20

15.2.5 Writing to CSV format

The Series and DataFrame objects have an instance method to_csv which allows storing the contents of the object as a comma-separated-values le. The function takes a number of arguments. Only the rst is required. path: A string path to the le to write nanRep: A string representation of a missing value (default ) cols: Columns to write (default None) header: Whether to write out the column names (default True) index: whether to write row (index) names (default True) index_label: Column label(s) for index column(s) if desired. If None (default), and header and index are True, then the index names are used. (A sequence should be given if the DataFrame uses MultiIndex). mode : Python write mode, default w sep : Field delimiter for the output le (default ) encoding: a string representing the encoding to use if the contents are non-ascii, for python versions prior to 3

15.2.6 Writing a formatted string

The DataFrame object has an instance method to_string which allows control over the string representation of the object. All arguments are optional: buf default None, for example a StringIO object columns default None, which columns to write col_space default None, number of spaces to write between columns na_rep default NaN, representation of NA value formatters default None, a dictionary (by column) of functions each of which takes a single argument and returns a formatted string float_format default None, a function which takes a single (oat) argument and returns a formatted string; to be applied to oats in the DataFrame. sparsify default True, set to False for a DataFrame with a hierarchical index to print every multiindex key at each row. index_names default True, will print the names of the indices index default True, will print the index (ie, row labels) header default True, will print the column labels justify default left, will print column headers left- or right-justied The Series object also has a to_string method, but with only the buf, na_rep, float_format arguments. There is also a length argument which, if set to True, will additionally output the length of the Series.

180

Chapter 15. IO Tools (Text, CSV, HDF5, ...)

pandas: powerful Python data analysis toolkit, Release 0.7.1

15.2.7 Writing to HTML format

DataFrame object has an instance method to_html which renders the contents of the DataFrame as an html table. The function arguments are as in the method to_string described above.

15.3 Excel les

The ExcelFile class can read an Excel 2003 le using the xlrd Python module and use the same parsing code as the above to convert tabular data into a DataFrame. To use it, create the ExcelFile object:
xls = ExcelFile(path_to_file.xls)

Then use the parse instance method with a sheetname, then use the same additional arguments as the parsers above:
xls.parse(Sheet1, index_col=None, na_values=[NA])

To read sheets from an Excel 2007 le, you can pass a lename with a .xlsx extension, in which case the openpyxl module will be used to read the le. To write a DataFrame object to a sheet of an Excel le, you can use the to_excel instance method. The arguments are largely the same as to_csv described above, the rst argument being the name of the excel le, and the optional second argument the name of the sheet to which the DataFrame should be written. For example:
df.to_excel(path_to_file.xlsx, sheet_name=sheet1)

Files with a .xls extension will be written using xlwt and those with a .xlsx extension will be written using openpyxl. The Panel class also has a to_excel instance method, which writes each DataFrame in the Panel to a separate sheet. In order to write separate DataFrames to separate sheets in a single Excel le, one can use the ExcelWriter class, as in the following example:
writer = ExcelWriter(path_to_file.xlsx) df1.to_excel(writer, sheet_name=sheet1) df2.to_excel(writer, sheet_name=sheet2) writer.save()

15.4 HDF5 (PyTables)

HDFStore is a dict-like object which reads and writes pandas to the high performance HDF5 format using the excellent PyTables library.
In [609]: store = HDFStore(store.h5) In [610]: print store <class pandas.io.pytables.HDFStore> File path: store.h5 Empty

Objects can be written to the le just like adding key-value pairs to a dict:
In [611]: index = DateRange(1/1/2000, periods=8) In [612]: s = Series(randn(5), index=[a, b, c, d, e])

15.3. Excel les

181

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [613]: df = DataFrame(randn(8, 3), index=index, .....: columns=[A, B, C]) In [614]: wp = Panel(randn(2, 5, 4), items=[Item1, Item2], .....: major_axis=DateRange(1/1/2000, periods=5), .....: minor_axis=[A, B, C, D]) In [615]: store[s] = s In [616]: store[df] = df In [617]: store[wp] = wp In [618]: store Out[618]: <class pandas.io.pytables.HDFStore> File path: store.h5 df DataFrame s Series wp Panel

In a current or later Python session, you can retrieve stored objects:

In [619]: store[df] Out[619]: A B 2000-01-03 -0.173215 0.119209 2000-01-04 -0.861849 -2.104569 2000-01-05 1.071804 0.721555 2000-01-06 -1.039575 0.271860 2000-01-07 0.567020 0.276232 2000-01-10 -0.673690 0.113648 2000-01-11 0.524988 0.404705 2000-01-12 -1.715002 -1.039268

C -1.044236 -0.494929 -0.706771 -0.424972 -1.087401 -1.478427 0.577046 -0.370647

182

Chapter 15. IO Tools (Text, CSV, HDF5, ...)

CHAPTER

SIXTEEN

SPARSE DATA STRUCTURES

We have implemented sparse versions of Series, DataFrame, and Panel. These are not sparse in the typical mostly 0. You can view these objects as being compressed where any data matching a specic value (NaN/missing by default, though any value can be chosen) is omitted. A special SparseIndex object tracks where data has been sparsied. This will make much more sense in an example. All of the standard pandas data structures have a to_sparse method:
In [808]: ts = Series(randn(10)) In [809]: ts[2:-2] = np.nan In [810]: sts = ts.to_sparse() In [811]: sts Out[811]: 0 0.469112 1 -0.282863 2 NaN 3 NaN 4 NaN 5 NaN 6 NaN 7 NaN 8 -0.861849 9 -2.104569 BlockIndex Block locations: array([0, 8], dtype=int32) Block lengths: array([2, 2], dtype=int32)

The to_sparse method takes a kind argument (for the sparse index, see below) and a fill_value. So if we had a mostly zero Series, we could convert it to sparse with fill_value=0:
In [812]: ts.fillna(0).to_sparse(fill_value=0) Out[812]: 0 0.469112 1 -0.282863 2 0.000000 3 0.000000 4 0.000000 5 0.000000 6 0.000000 7 0.000000 8 -0.861849 9 -2.104569 BlockIndex

183

pandas: powerful Python data analysis toolkit, Release 0.7.1

Block locations: array([0, 8], dtype=int32) Block lengths: array([2, 2], dtype=int32)

The sparse objects exist for memory efciency reasons. Suppose you had a large, mostly NA DataFrame:
In [813]: df = DataFrame(randn(10000, 4)) In [814]: df.ix[:9998] = np.nan In [815]: sdf = df.to_sparse() In [816]: sdf Out[816]: <class pandas.sparse.frame.SparseDataFrame> Int64Index: 10000 entries, 0 to 9999 Columns: 0 to 3 dtypes: float64(4) In [817]: sdf.density Out[817]: 0.0001

As you can see, the density (% of values that have not been compressed) is extremely low. This sparse object takes up much less memory on disk (pickled) and in the Python interpreter. Functionally, their behavior should be nearly identical to their dense counterparts. Any sparse object can be converted back to the standard dense form by calling to_dense:
In [818]: sts.to_dense() Out[818]: 0 0.469112 1 -0.282863 2 NaN 3 NaN 4 NaN 5 NaN 6 NaN 7 NaN 8 -0.861849 9 -2.104569

16.1 SparseArray
SparseArray is the base layer for all of the sparse indexed data structures. It is a 1-dimensional ndarray-like object storing only values distinct from the fill_value:
In [819]: arr = np.random.randn(10) In [820]: arr[2:5] = np.nan; arr[7:8] = np.nan In [821]: sparr = SparseArray(arr) In [822]: sparr Out[822]: SparseArray([-1.9557, -1.6589, nan, nan, nan, 0.606 , 1.3342]) IntIndex Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

nan,

1.1589,

0.1453,

184

Chapter 16. Sparse data structures

pandas: powerful Python data analysis toolkit, Release 0.7.1

Like the indexed objects (SparseSeries, SparseDataFrame, SparsePanel), a SparseArray can be converted back to a regular ndarray by calling to_dense:
In [823]: sparr.to_dense() Out[823]: array([-1.9557, -1.6589, nan, nan, 0.606 , 1.3342])

nan,

nan,

1.1589,

0.1453,

16.2 SparseList
SparseList is a list-like data structure for managing a dynamic collection of SparseArrays. To create one, simply call the SparseList constructor with a fill_value (defaulting to NaN):
In [824]: spl = SparseList() In [825]: spl Out[825]: <pandas.sparse.list.SparseList object at 0x1148f9d10>

The two important methods are append and to_array. append can accept scalar values or any 1-dimensional sequence:
In [826]: spl.append(np.array([1., nan, nan, 2., 3.])) In [827]: spl.append(5) In [828]: spl.append(sparr) In [829]: spl Out[829]: <pandas.sparse.list.SparseList object at 0x1148f9d10> SparseArray([ 1., nan, nan, 2., 3.]) IntIndex Indices: array([0, 3, 4], dtype=int32) SparseArray([ 5.]) IntIndex Indices: array([0], dtype=int32) SparseArray([-1.9557, -1.6589, nan, nan, nan, nan, 0.606 , 1.3342]) IntIndex Indices: array([0, 1, 5, 6, 8, 9], dtype=int32)

1.1589,

0.1453,

As you can see, all of the contents are stored internally as a list of memory-efcient SparseArray objects. Once youve accumulated all of the data, you can call to_array to get a single SparseArray with all the data:
In [830]: spl.to_array() Out[830]: SparseArray([ 1. , nan, nan, 2. , 3. , 5. , -1.9557, -1.6589, nan, nan, nan, 1.1589, 0.1453, nan, 0.606 , 1.3342]) IntIndex Indices: array([ 0, 3, 4, 5, 6, 7, 11, 12, 14, 15], dtype=int32)

16.2. SparseList

185

pandas: powerful Python data analysis toolkit, Release 0.7.1

16.3 SparseIndex objects

Two kinds of SparseIndex are implemented, block and integer. We recommend using block as its more memory efcient. The integer format keeps an arrays of all of the locations where the data are not equal to the ll value. The block format tracks only the locations and sizes of blocks of data.

186

Chapter 16. Sparse data structures

CHAPTER

SEVENTEEN

CAVEATS AND GOTCHAS

17.1 NaN, Integer NA values and NA type promotions
17.1.1 Choice of NA representation
For lack of NA (missing) support from the ground up in NumPy and Python in general, we were given the difcult choice between either A masked array solution: an array of data and an array of boolean values indicating whether a value Using a special sentinel value, bit pattern, or set of sentinel values to denote NA across the dtypes For many reasons we chose the latter. After years of production use it has proven, at least in my opinion, to be the best decision given the state of affairs in NumPy and Python in general. The special value NaN (Not-A-Number) is used everywhere as the NA value, and there are API functions isnull and notnull which can be used across the dtypes to detect NA values. However, it comes with it a couple of trade-offs which I most certainly have not ignored.

17.1.2 Support for integer NA

In the absence of high performance NA support being built into NumPy from the ground up, the primary casualty is the ability to represent NAs in integer arrays. For example:
In [338]: s = Series([1, 2, 3, 4, 5], index=list(abcde)) In [339]: s Out[339]: a 1 b 2 c 3 d 4 e 5 In [340]: s.dtype Out[340]: dtype(int64) In [341]: s2 = s.reindex([a, b, c, f, u]) In [342]: s2 Out[342]: a 1 b 2

187

pandas: powerful Python data analysis toolkit, Release 0.7.1

c f u

3 NaN NaN

In [343]: s2.dtype Out[343]: dtype(float64)

This trade-off is made largely for memory and performance reasons, and also so that the resulting Series continues to be numeric. One possibility is to use dtype=object arrays instead.

17.1.3 NA type promotions

When introducing NAs into an existing Series or DataFrame via reindex or some other means, boolean and integer types will be promoted to a different dtype in order to store the NAs. These are summarized by this table: Typeclass floating object integer boolean Promotion dtype for storing NAs no change no change cast to float64 cast to object

While this may seem like a heavy trade-off, in practice I have found very few cases where this is an issue in practice. Some explanation for the motivation here in the next section.

17.1.4 Why not make NumPy like R?

Many people have suggested that NumPy should simply emulate the NA support present in the more domain-specic statistical programming langauge R. Part of the reason is the NumPy type hierarchy: Typeclass numpy.floating numpy.integer numpy.unsignedinteger numpy.object_ numpy.bool_ numpy.character Dtypes float16, float32, float64, float128 int8, int16, int32, int64 uint8, uint16, uint32, uint64 object_ bool_ string_, unicode_

The R language, by contrast, only has a handful of built-in data types: integer, numeric (oating-point), character, and boolean. NA types are implemented by reserving special bit patterns for each type to be used as the missing value. While doing this with the full NumPy type hierarchy would be possible, it would be a more substantial trade-off (especially for the 8- and 16-bit data types) and implementation undertaking. An alternate approach is that of using masked arrays. A masked array is an array of data with an associated boolean mask denoting whether each value should be considered NA or not. I am personally not in love with this approach as I feel that overall it places a fairly heavy burden on the user and the library implementer. Additionally, it exacts a fairly high performance cost when working with numerical data compared with the simple approach of using NaN. Thus, I have chosen the Pythonic practicality beats purity approach and traded integer NA capability for a much simpler approach of using a special value in oat and object arrays to denote NA, and promoting integer arrays to oating when NAs must be introduced.

188

Chapter 17. Caveats and Gotchas

pandas: powerful Python data analysis toolkit, Release 0.7.1

17.2 Integer indexing 17.3 Label-based slicing conventions

17.3.1 Non-monotonic indexes require exact matches 17.3.2 Endpoints are inclusive
Compared with standard Python sequence slicing in which the slice endpoint is not inclusive, label-based slicing in pandas is inclusive. The primary reason for this is that it is often not possible to easily the successor or next element after a particular label in an index. For example, consider the following Series:
In [344]: s = Series(randn(6), index=list(abcdef)) In [345]: s Out[345]: a -1.309989 b -1.153000 c 0.606382 d -0.681101 e -0.289724 f -0.996632

Suppose we wished to slice from c to e, using integers this would be

In [346]: s[2:5] Out[346]: c 0.606382 d -0.681101 e -0.289724

However, if you only had c and e, determining the next element in the index can be somewhat complicated. For example, the following does not work:
s.ix[c:e+1]

A very common use case is to limit a time series to start and end at two specic dates. To enable this, we made the design design to make label-based slicing include both endpoints:
In [347]: s.ix[c:e] Out[347]: c 0.606382 d -0.681101 e -0.289724

This is most denitely a practicality beats purity sort of thing, but it is something to watch out for is you expect label-based slicing to behave exactly in the way that standard Python integer slicing works.

17.4 Miscellaneous indexing gotchas

17.4.1 Reindex versus ix gotchas
Many users will nd themselves using the ix indexing capabilities as a concise means of selecting data from a pandas object:

17.2. Integer indexing

189

pandas: powerful Python data analysis toolkit, Release 0.7.1

In [348]: df = DataFrame(randn(6, 4), columns=[one, two, three, four], .....: index=list(abcdef)) In [349]: df Out[349]: one two three four a -1.407699 1.014104 0.314226 -0.001675 b 0.071823 0.892566 0.680594 -0.339640 c 0.214910 -0.078410 -0.177665 0.490838 d -1.360102 1.592456 1.007100 0.697835 e -1.890591 -0.254002 1.360151 -0.059912 f -0.151652 0.624697 -1.124779 0.072594 In [350]: df.ix[[b, c, e]] Out[350]: one two three four b 0.071823 0.892566 0.680594 -0.339640 c 0.214910 -0.078410 -0.177665 0.490838 e -1.890591 -0.254002 1.360151 -0.059912

This is, of course, completely equivalent in this case to using th reindex method:
In [351]: df.reindex([b, c, e]) Out[351]: one two three four b 0.071823 0.892566 0.680594 -0.339640 c 0.214910 -0.078410 -0.177665 0.490838 e -1.890591 -0.254002 1.360151 -0.059912

Some might conclude that ix and reindex are 100% equivalent based on this. This is indeed true except in the case of integer indexing. For example, the above operation could alternately have been expressed as:
In [352]: df.ix[[1, 2, 4]] Out[352]: one two three four b 0.071823 0.892566 0.680594 -0.339640 c 0.214910 -0.078410 -0.177665 0.490838 e -1.890591 -0.254002 1.360151 -0.059912

If you pass [1, 2, 4] to reindex you will get another thing entirely:
In [353]: df.reindex([1, 2, 4]) Out[353]: one two three four 1 NaN NaN NaN NaN 2 NaN NaN NaN NaN 4 NaN NaN NaN NaN

So its important to remember that reindex is strict label indexing only. This can lead to some potentially surprising results in pathological cases where an index contains, say, both integers and strings:
In [354]: s = Series([1, 2, 3], index=[a, 0, 1]) In [355]: s Out[355]: a 1 0 2 1 3 In [356]: s.ix[[0, 1]]

190

Chapter 17. Caveats and Gotchas

pandas: powerful Python data analysis toolkit, Release 0.7.1

Out[356]: a 1 0 2 In [357]: s.reindex([0, 1]) Out[357]: 0 2 1 3

Because the index in this case does not contain solely integers, ix falls back on integer indexing. By contrast, reindex only looks for the values passed in the index, thus nding the integers 0 and 1. While it would be possible to insert some logic to check whether a passed sequence is all contained in the index, that logic would exact a very high cost in large data sets.

17.4. Miscellaneous indexing gotchas

191

pandas: powerful Python data analysis toolkit, Release 0.7.1

192

Chapter 17. Caveats and Gotchas

CHAPTER

EIGHTEEN

RPY2 / R INTERFACE
Note: This is all highly experimental. I would like to get more people involved with building a nice RPy2 interface for pandas If your computer has R and rpy2 (> 2.2) installed (which will be left to the reader), you will be able to leverage the below functionality. On Windows, doing this is quite an ordeal at the moment, but users on Unix-like systems should nd it quite easy. As a general rule, I would recommend using the latest revision of rpy2 from bitbucket:
# if installing for the first time hg clone http://bitbucket.org/lgautier/rpy2 cd rpy2 hg pull hg update sudo python setup.py install

Note: To use R packages with this interface, you will need to install them inside R yourself. At the moment it cannot install them for you. Once you have done installed R and rpy2, you should be able to import pandas.rpy.common without a hitch.

18.1 Transferring R data sets into Python

The load_data function retrieves an R data set and converts it to the appropriate pandas object (most likely a DataFrame):
In [757]: import pandas.rpy.common as com In [758]: infert = com.load_data(infert) In [759]: infert.head() Out[759]: education age parity 1 0-5yrs 26 6 2 0-5yrs 42 1 3 0-5yrs 39 6 4 0-5yrs 34 4 5 6-11yrs 35 3

induced 1 1 2 2 1

case 1 1 1 1 1

spontaneous 2 0 0 0 1

stratum 1 2 3 4 5

pooled.stratum 3 1 4 2 32

193

pandas: powerful Python data analysis toolkit, Release 0.7.1

18.2 Calling R functions with pandas objects 18.3 High-level interface to R estimators

194

Chapter 18. rpy2 / R interface

CHAPTER

NINETEEN

RELATED PYTHON LIBRARIES

19.1 la (larry)
Keith Goodmans excellent labeled array package is very similar to pandas in many regards, though with some key differences. The main philosophical design difference is to be a wrapper around a single NumPy ndarray object while adding axis labeling and label-based operations and indexing. Because of this, creating a size-mutable object with heterogeneous columns (e.g. DataFrame) is not possible with the la package. Provide a single n-dimensional object with labeled axes with functionally analogous data alignment semantics to pandas objects Advanced / label-based indexing similar to that provided in pandas but setting is not supported Stays much closer to NumPy arrays than pandas larry objects must be homogeneously typed GroupBy support is relatively limited, but a few functions are available: group_mean, group_median, and group_ranking It has a collection of analytical functions suited to quantitative portfolio construction for nancial applications It has a collection of moving window statistics implemented in Bottleneck

19.2 scikits.statsmodels
The main statistics and econometrics library for Python. pandas has become a dependency of this library.

19.3 scikits.timeseries
scikits.timeseries provides a data structure for xed frequency time series data based on the numpy.MaskedArray class. For time series data, it provides some of the same functionality to the pandas Series class. It has many more functions for time series-specic manipulation. Also, it has support for many more frequencies, though less customizable by the user (so 5-minutely data is easier to do with pandas for example). We are aiming to merge these libraries together in the near future.

195

pandas: powerful Python data analysis toolkit, Release 0.7.1

196

Chapter 19. Related Python libraries

CHAPTER

TWENTY

COMPARISON WITH R / R LIBRARIES

Since pandas aims to provide a lot of the data manipulation and analysis functionality that people use R for, this page was started to provide a more detailed look at the R language and its many 3rd party libraries as they relate to pandas. In offering comparisons with R and CRAN libraries, we care about the following things: Functionality / exibility: what can / cannot be done with each tool Performance: how fast are operations. Hard numbers / benchmarks are preferable Ease-of-use: is one tool easier or harder to use (you may have to be the judge of this given side-by-side code comparisons) As I do not have an encyclopedic knowledge of R packages, feel free to suggest additional CRAN packages to add to this list. This is also here to offer a big of a translation guide for users of these R packages.

20.1 data.frame 20.2 zoo 20.3 xts 20.4 plyr 20.5 reshape / reshape2

197

pandas: powerful Python data analysis toolkit, Release 0.7.1

198

Chapter 20. Comparison with R / R libraries

CHAPTER

TWENTYONE

API REFERENCE
21.1 General functions
21.1.1 Data manipulations
pivot_table(data[, values, rows, cols, ...]) pandas.tools.pivot.pivot_table static pivot.pivot_table(data, values=None, rows=None, cols=None, aggfunc=mean, ll_value=None, margins=False) Create a spreadsheet-style pivot table as a DataFrame. The levels in the pivot table will be stored in MultiIndex objects (hierarchical indexes) on the index and columns of the result DataFrame Parameters data : DataFrame values : column to aggregate, optional rows : list Columns to group on the x-axis of the pivot table cols : list Columns to group on the x-axis of the pivot table aggfunc : function, default numpy.mean, or list of functions If list of functions passed, the resulting pivot table will have hierarchical columns whose top level are the function names (inferred from the function objects themselves) ll_value : scalar, default None Value to replace missing values with margins : boolean, default False Add all row / columns (e.g. for subtotal / grand totals) Returns table : DataFrame Create a spreadsheet-style pivot table as a DataFrame. The levels in the

199

pandas: powerful Python data analysis toolkit, Release 0.7.1

Examples >>> df A 0 foo 1 foo 2 foo 3 foo 4 foo 5 bar 6 bar 7 bar 8 bar

B one one one two two one one two two

C small large large small small large small small large

D 1 2 2 3 3 4 5 6 7

>>> table = pivot_table(df, values=D, rows=[A, B], ... cols=[C], aggfunc=np.sum) >>> table small large foo one 1 4 two 6 NaN bar one 5 4 two 6 7

merge(left, right[, how, on, left_on, ...]) concat(objs[, axis, join, join_axes, ...]) pandas.tools.merge.merge

Merge DataFrame objects by performing a database-style join operation by Concatenate pandas objects along a particular axis with optional set logic along the other axes.

static merge.merge(left, right, how=inner, on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=True, sufxes=(.x, .y), copy=True) Merge DataFrame objects by performing a database-style join operation by columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. Parameters left : DataFrame right : DataFrame how : {left, right, outer, inner}, default inner left: use only keys from left frame (SQL: left outer join) right: use only keys from right frame (SQL: right outer join) outer: use union of keys from both frames (SQL: full outer join) inner: use intersection of keys from both frames (SQL: inner join) on : label or list Field names to join on. Must be found in both DataFrames. left_on : label or list, or array-like Field names to join on in left DataFrame. Can be a vector or list of vectors of the length of the DataFrame to use a particular vector as the join key instead of columns right_on : label or list, or array-like

200

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Field names to join on in right DataFrame or vector/list of vectors per left_on docs left_index : boolean, default True Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels right_index : boolean, default True Use the index from the right DataFrame as the join key. Same caveats as left_index sort : boolean, default True Sort the join keys lexicographically in the result DataFrame sufxes : 2-length sequence (tuple, list, ...) Sufx to apply to overlapping column names in the left and right side, respectively copy : boolean, default True If False, do not copy data unnecessarily Returns merged : DataFrame
Examples >>> A lkey 0 foo 1 bar 2 baz 3 foo >>> B rkey 0 foo 1 bar 2 qux 3 bar

value 1 2 3 4

value 5 6 7 8

>>> merge(A, B, left_on=lkey, right_on=rkey, how=outer) lkey value.x rkey value.y 0 bar 2 bar 6 1 bar 2 bar 8 2 baz 3 NaN NaN 3 foo 1 foo 5 4 foo 4 foo 5 5 NaN NaN qux 7

pandas.tools.merge.concat static merge.concat(objs, axis=0, join=outer, join_axes=None, ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False) Concatenate pandas objects along a particular axis with optional set logic along the other axes. Can also add a layer of hierarchical indexing on the concatenation axis, which may be useful if the labels are the same (or overlapping) on the passed axis number Parameters objs : list or dict of Series, DataFrame, or Panel objects If a dict is passed, the sorted keys will be used as the keys argument, unless it is passed, in which case the values will be selected (see below). Any None objects will be dropped silently unless they are all None in which case an Exception will be raised axis : {0, 1, ...}, default 0 The axis to concatenate along 21.1. General functions 201

pandas: powerful Python data analysis toolkit, Release 0.7.1

join : {inner, outer}, default outer How to handle indexes on other axis(es) join_axes : list of Index objects Specic indexes to use for the other n - 1 axes instead of performing inner/outer set logic verify_integrity : boolean, default False Check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation keys : sequence, default None If multiple levels passed, should contain tuples. Construct hierarchical index using the passed keys as the outermost level levels : list of sequences, default None Specic levels (unique values) to use for constructing a MultiIndex. Otherwise they will be inferred from the keys names : list, default None Names for the levels in the resulting hierarchical index ignore_index : boolean, default False If True, do not use the index values on the concatenation axis. The resulting axis will be labeled 0, ..., n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Returns concatenated : type of objects
Notes

The keys, levels, and names arguments are all optional

21.1.2 Pickling
load(path) save(obj, path) Load pickled pandas object (or any other pickled object) from the specied Pickle (serialize) object to input le path

pandas.core.common.load static common.load(path) Load pickled pandas object (or any other pickled object) from the specied le path Parameters path : string File path Returns unpickled : type of object stored in le

202

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.core.common.save static common.save(obj, path) Pickle (serialize) object to input le path Parameters obj : any object path : string File path

21.1.3 File IO
read_table(lepath_or_buffer[, sep, ...]) read_csv(lepath_or_buffer[, sep, header, ...]) ExcelFile.parse(sheetname[, header, ...]) pandas.io.parsers.read_table static parsers.read_table(lepath_or_buffer, sep=t, header=0, index_col=None, names=None, skiprows=None, na_values=None, parse_dates=False, date_parser=None, nrows=None, iterator=False, chunksize=None, skip_footer=0, converters=None, verbose=False, delimiter=None, encoding=None) Read general delimited le into DataFrame Also supports optionally iterating or breaking of the le into chunks. Parameters lepath_or_buffer : string or le handle / StringIO sep : string, default t (tab-stop) Delimiter to use header : int, default 0 Row to use for the column labels of the parsed DataFrame skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) index_col : int or sequence, default None Column to use as the row labels of the DataFrame. If a sequence is given, a MultiIndex is used. names : array-like List of column names na_values : list-like, default None List of additional strings to recognize as NA/NaN parse_dates : boolean, default False Attempt to parse dates in the index column(s) date_parser : function Function to use for converting dates to strings. Defaults to dateutil.parser nrows : int, default None Read general delimited le into DataFrame Read CSV (comma-separated) le into DataFrame Read Excel table into DataFrame

21.1. General functions

203

pandas: powerful Python data analysis toolkit, Release 0.7.1

Number of rows of le to read. Useful for reading pieces of large les iterator : boolean, default False Return TextParser object chunksize : int, default None Return TextParser object for iteration skip_footer : int, default 0 Number of line at bottom of le to skip converters : dict. optional Dict of functions for converting values in certain columns. Keys can either be integers or column labels verbose : boolean, default False Indicate number of NA values placed in non-numeric columns delimiter : string, default None Alternative argument name for sep encoding : string, default None Encoding to use for UTF when reading/writing (ex. utf-8) Returns result : DataFrame or TextParser pandas.io.parsers.read_csv static parsers.read_csv(lepath_or_buffer, sep=, , header=0, index_col=None, names=None, skiprows=None, na_values=None, parse_dates=False, date_parser=None, nrows=None, iterator=False, chunksize=None, skip_footer=0, converters=None, verbose=False, delimiter=None, encoding=None) Read CSV (comma-separated) le into DataFrame Also supports optionally iterating or breaking of the le into chunks. Parameters lepath_or_buffer : string or le handle / StringIO sep : string, default , Delimiter to use. If sep is None, will try to automatically determine this header : int, default 0 Row to use for the column labels of the parsed DataFrame skiprows : list-like or integer Row numbers to skip (0-indexed) or number of rows to skip (int) index_col : int or sequence, default None Column to use as the row labels of the DataFrame. If a sequence is given, a MultiIndex is used. names : array-like List of column names na_values : list-like, default None

204

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

List of additional strings to recognize as NA/NaN parse_dates : boolean, default False Attempt to parse dates in the index column(s) date_parser : function Function to use for converting dates to strings. Defaults to dateutil.parser nrows : int, default None Number of rows of le to read. Useful for reading pieces of large les iterator : boolean, default False Return TextParser object chunksize : int, default None Return TextParser object for iteration skip_footer : int, default 0 Number of line at bottom of le to skip converters : dict. optional Dict of functions for converting values in certain columns. Keys can either be integers or column labels verbose : boolean, default False Indicate number of NA values placed in non-numeric columns delimiter : string, default None Alternative argument name for sep encoding : string, default None Encoding to use for UTF when reading/writing (ex. utf-8) Returns result : DataFrame or TextParser pandas.io.parsers.ExcelFile.parse ExcelFile.parse(sheetname, header=0, skiprows=None, index_col=None, date_parser=None, na_values=None, chunksize=None) Read Excel table into DataFrame Parameters sheetname : string Name of Excel sheet header : int, default 0 Row to use for the column labels of the parsed DataFrame skiprows : list-like Row numbers to skip (0-indexed) index_col : int, default None Column to use as the row labels of the DataFrame. Pass None if there is no such column na_values : list-like, default None parse_dates=False,

21.1. General functions

205

pandas: powerful Python data analysis toolkit, Release 0.7.1

List of additional strings to recognize as NA/NaN Returns parsed : DataFrame

21.1.4 HDFStore: PyTables (HDF5)

HDFStore.put(key, value[, table, append, ...]) HDFStore.get(key) pandas.io.pytables.HDFStore.put HDFStore.put(key, value, table=False, append=False, compression=None) Store object in HDFStore Parameters key : object value : {Series, DataFrame, Panel} table : boolean, default False Write as a PyTables Table structure which may perform worse but allow more exible operations like searching / selecting subsets of the data append : boolean, default False For table data structures, append the input data to the existing table compression : {None, blosc, lzo, zlib}, default None Use a compression algorithm to compress the data If None, the compression settings specied in the ctor will be used. pandas.io.pytables.HDFStore.get HDFStore.get(key) Retrieve pandas object stored in le Parameters key : object Returns obj : type of object stored in le Store object in HDFStore Retrieve pandas object stored in le

206

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.1.5 Standard moving window functions

rolling_count(arg, window[, time_rule]) rolling_sum(arg, window[, min_periods, ...]) rolling_mean(arg, window[, min_periods, ...]) rolling_median(arg, window[, min_periods, ...]) rolling_var(arg, window[, min_periods, ...]) rolling_std(arg, window[, min_periods, ...]) rolling_corr(arg1, arg2, window[, ...]) rolling_cov(arg1, arg2, window[, ...]) rolling_skew(arg, window[, min_periods, ...]) rolling_kurt(arg, window[, min_periods, ...]) rolling_apply(arg, window, func[, ...]) rolling_quantile(arg, window, quantile[, ...]) pandas.stats.moments.rolling_count static moments.rolling_count(arg, window, time_rule=None) Rolling count of number of non-NaN observations inside provided window. Parameters arg : DataFrame or numpy ndarray-like window : Number of observations used for calculating statistic Returns rolling_count : type of caller pandas.stats.moments.rolling_sum static moments.rolling_sum(arg, window, min_periods=None, time_rule=None) Moving sum Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument Rolling count of number of non-NaN observations inside provided window. Moving sum Moving mean O(N log(window)) implementation using skip list Unbiased moving variance Unbiased moving standard deviation Moving sample correlation Unbiased moving covariance Unbiased moving skewness Unbiased moving kurtosis Generic moving function application Moving quantile

21.1. General functions

207

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.stats.moments.rolling_mean static moments.rolling_mean(arg, window, min_periods=None, time_rule=None) Moving mean Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument pandas.stats.moments.rolling_median static moments.rolling_median(arg, window, min_periods=None, time_rule=None) O(N log(window)) implementation using skip list Moving median Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument pandas.stats.moments.rolling_var static moments.rolling_var(arg, window, min_periods=None, time_rule=None) Unbiased moving variance Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument

208

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.stats.moments.rolling_std static moments.rolling_std(arg, window, min_periods=None, time_rule=None) Unbiased moving standard deviation Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument pandas.stats.moments.rolling_corr static moments.rolling_corr(arg1, arg2, window, min_periods=None, time_rule=None) Moving sample correlation Parameters arg1 : Series, DataFrame, or ndarray arg2 : Series, DataFrame, or ndarray window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type depends on inputs DataFrame / DataFrame -> DataFrame (matches on columns) DataFrame / Series -> Computes result for each column Series / Series -> Series pandas.stats.moments.rolling_cov static moments.rolling_cov(arg1, arg2, window, min_periods=None, time_rule=None) Unbiased moving covariance Parameters arg1 : Series, DataFrame, or ndarray arg2 : Series, DataFrame, or ndarray window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type depends on inputs

21.1. General functions

209

pandas: powerful Python data analysis toolkit, Release 0.7.1

DataFrame / DataFrame -> DataFrame (matches on columns) DataFrame / Series -> Computes result for each column Series / Series -> Series pandas.stats.moments.rolling_skew static moments.rolling_skew(arg, window, min_periods=None, time_rule=None) Unbiased moving skewness Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument pandas.stats.moments.rolling_kurt static moments.rolling_kurt(arg, window, min_periods=None, time_rule=None) Unbiased moving kurtosis Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument pandas.stats.moments.rolling_apply static moments.rolling_apply(arg, window, func, min_periods=None, time_rule=None) Generic moving function application Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic func : function Must produce a single value from an ndarray input min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument

210

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.stats.moments.rolling_quantile static moments.rolling_quantile(arg, window, quantile, min_periods=None, time_rule=None) Moving quantile Parameters arg : Series, DataFrame window : Number of observations used for calculating statistic quantile : 0 <= quantile <= 1 min_periods : int Minimum number of observations in window required to have a value time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default=None Name of time rule to conform to before computing statistic Returns y : type of input argument

21.1.6 Exponentially-weighted moving window functions

ewma(arg[, com, span, min_periods, time_rule]) ewmstd(arg[, com, span, min_periods, bias, ...]) ewmvar(arg[, com, span, min_periods, bias, ...]) ewmcorr(arg1, arg2[, com, span, ...]) ewmcov(arg1, arg2[, com, span, min_periods, ...]) pandas.stats.moments.ewma static moments.ewma(arg, com=None, span=None, min_periods=0, time_rule=None) Exponentially-weighted moving average Parameters arg : Series, DataFrame com : oat. optional Center of mass: alpha = com / (1 + com), span : oat, optional Specify decay in terms of span, alpha = 2 / (span + 1) min_periods : int, default 0 Number of observations in sample to require (only affects beginning) time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default None Name of time rule to conform to before computing statistic Returns y : type of input argument
Notes

Exponentially-weighted moving average Exponentially-weighted moving std Exponentially-weighted moving variance Exponentially-weighted moving correlation Exponentially-weighted moving covariance

Either center of mass or span must be specied EWMA is sometimes specied using a span parameter s, we have have that the decay parameter alpha is related to the span as = 1 2/(s + 1) = c/(1 + c)

21.1. General functions

211

pandas: powerful Python data analysis toolkit, Release 0.7.1

where c is the center of mass. Given a span, the associated center of mass is c = (s 1)/2 So a 20-day EWMA would have center 9.5. pandas.stats.moments.ewmstd static moments.ewmstd(arg, com=None, span=None, min_periods=0, bias=False, time_rule=None) Exponentially-weighted moving std Parameters arg : Series, DataFrame com : oat. optional Center of mass: alpha = com / (1 + com), span : oat, optional Specify decay in terms of span, alpha = 2 / (span + 1) min_periods : int, default 0 Number of observations in sample to require (only affects beginning) time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default None Name of time rule to conform to before computing statistic bias : boolean, default False Use a standard estimation bias correction Returns y : type of input argument
Notes

Either center of mass or span must be specied EWMA is sometimes specied using a span parameter s, we have have that the decay parameter alpha is related to the span as = 1 2/(s + 1) = c/(1 + c) where c is the center of mass. Given a span, the associated center of mass is c = (s 1)/2 So a 20-day EWMA would have center 9.5. pandas.stats.moments.ewmvar static moments.ewmvar(arg, com=None, span=None, min_periods=0, bias=False, time_rule=None) Exponentially-weighted moving variance Parameters arg : Series, DataFrame com : oat. optional Center of mass: alpha = com / (1 + com), span : oat, optional Specify decay in terms of span, alpha = 2 / (span + 1) min_periods : int, default 0 Number of observations in sample to require (only affects beginning) time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default None 212 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Name of time rule to conform to before computing statistic bias : boolean, default False Use a standard estimation bias correction Returns y : type of input argument
Notes

Either center of mass or span must be specied EWMA is sometimes specied using a span parameter s, we have have that the decay parameter alpha is related to the span as = 1 2/(s + 1) = c/(1 + c) where c is the center of mass. Given a span, the associated center of mass is c = (s 1)/2 So a 20-day EWMA would have center 9.5. pandas.stats.moments.ewmcorr static moments.ewmcorr(arg1, arg2, com=None, span=None, min_periods=0, time_rule=None) Exponentially-weighted moving correlation Parameters arg1 : Series, DataFrame, or ndarray arg2 : Series, DataFrame, or ndarray com : oat. optional Center of mass: alpha = com / (1 + com), span : oat, optional Specify decay in terms of span, alpha = 2 / (span + 1) min_periods : int, default 0 Number of observations in sample to require (only affects beginning) time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default None Name of time rule to conform to before computing statistic Returns y : type of input argument
Notes

Either center of mass or span must be specied EWMA is sometimes specied using a span parameter s, we have have that the decay parameter alpha is related to the span as = 1 2/(s + 1) = c/(1 + c) where c is the center of mass. Given a span, the associated center of mass is c = (s 1)/2 So a 20-day EWMA would have center 9.5.

21.1. General functions

213

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.stats.moments.ewmcov static moments.ewmcov(arg1, arg2, com=None, time_rule=None) Exponentially-weighted moving covariance Parameters arg1 : Series, DataFrame, or ndarray arg2 : Series, DataFrame, or ndarray com : oat. optional Center of mass: alpha = com / (1 + com), span : oat, optional Specify decay in terms of span, alpha = 2 / (span + 1) min_periods : int, default 0 Number of observations in sample to require (only affects beginning) time_rule : {None, WEEKDAY, EOM, W@MON, ...}, default None Name of time rule to conform to before computing statistic Returns y : type of input argument
Notes

span=None,

min_periods=0,

bias=False,

Either center of mass or span must be specied EWMA is sometimes specied using a span parameter s, we have have that the decay parameter alpha is related to the span as = 1 2/(s + 1) = c/(1 + c) where c is the center of mass. Given a span, the associated center of mass is c = (s 1)/2 So a 20-day EWMA would have center 9.5.

21.2 Series
21.2.1 Attributes and underlying data
Axes index: axis labels Series.values Series.dtype Series.isnull(obj) Series.notnull(obj) pandas.Series.values Series.values Return Series as ndarray Returns arr : numpy.ndarray Return Series as ndarray Data-type of the arrays elements. Replacement for numpy.isnan / -numpy.isnite which is suitable for use on object arrays. Replacement for numpy.isnite / -numpy.isnan which is suitable for use on object arrays.

214

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.dtype Series.dtype Data-type of the arrays elements. Parameters None : Returns d : numpy dtype object See Also: numpy.dtype
Examples >>> x array([[0, 1], [2, 3]]) >>> x.dtype dtype(int32) >>> type(x.dtype) <type numpy.dtype>

pandas.Series.isnull Series.isnull(obj) Replacement for numpy.isnan / -numpy.isnite which is suitable for use on object arrays. Parameters arr: ndarray or object value : Returns boolean ndarray or boolean : pandas.Series.notnull Series.notnull(obj) Replacement for numpy.isnite / -numpy.isnan which is suitable for use on object arrays. Parameters arr: ndarray or object value : Returns boolean ndarray or boolean :

21.2.2 Conversion / Constructors

Series.__init__([data, index, dtype, name, copy]) Series.astype(t) Series.copy() pandas.Series.__init__ Series.__init__(data=None, index=None, dtype=None, name=None, copy=False) One-dimensional ndarray with axis labels (including time series). Labels must be unique and can any hashable type. The object supports both integer- and label-based indexing and provides a host of methods for performing One-dimensional ndarray with axis labels (including time series). Copy of the array, cast to a specied type. Return new Series with copy of underlying values

21.2. Series

215

pandas: powerful Python data analysis toolkit, Release 0.7.1

operations involving the index. Statistical methods from ndarray have been overridden to automatically exclude missing data (currently represented as NaN) Operations between Series (+, -, /, , *) align values based on their associated index values they need not be the same length. The result index will be the sorted union of the two indexes. Parameters data : array-like, dict, or scalar value Contains data stored in Series index : array-like or Index (1d) Values must be unique and hashable, same length as data. Index object (or other iterable of same length as data) Will default to np.arange(len(data)) if not provided. If both a dict and index sequence are used, the index will override the keys found in the dict. dtype : numpy.dtype or None If None, dtype will be inferred copy : boolean, default False Copy input data copy : boolean, default False pandas.Series.astype Series.astype(t) Copy of the array, cast to a specied type. Parameters t : str or dtype Typecode or data-type to which the array is cast. Raises ComplexWarning : : When casting from complex to oat or int. a.real.astype(t).
Examples >>> x = np.array([1, 2, 2.5]) >>> x array([ 1. , 2. , 2.5]) >>> x.astype(int) array([1, 2, 2])

To avoid this, one should use

pandas.Series.copy Series.copy() Return new Series with copy of underlying values Returns cp : Series

216

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.2.3 Indexing, iteration

Series.get(label[, Returns value occupying requested label, default to specied missing value if not default]) present. Series.ix Series.__iter__() Series.iteritems([index])Lazily iterate over (index, value) tuples pandas.Series.get Series.get(label, default=None) Returns value occupying requested label, default to specied missing value if not present. Analogous to dict.get Parameters label : object Label value looking for default : object, optional Value to return if label not in index Returns y : scalar pandas.Series.ix Series.ix pandas.Series.__iter__ Series.__iter__() pandas.Series.iteritems Series.iteritems(index=True) Lazily iterate over (index, value) tuples

21.2.4 Binary operator functions

Series.add(other[, level, ll_value]) Series.div(other[, level, ll_value]) Series.mul(other[, level, ll_value]) Series.sub(other[, level, ll_value]) Series.combine(other, func[, ll_value]) Series.combine_first(other) Binary operator add with support to substitute a ll_value for missing data Binary operator divide with support to substitute a ll_value for missing data Binary operator multiply with support to substitute a ll_value for missing data Binary operator subtract with support to substitute a ll_value for missing data Perform elementwise binary operation on two Series using given function Combine Series values, choosing the calling Seriess values

21.2. Series

217

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.add Series.add(other, level=None, ll_value=None) Binary operator add with support to substitute a ll_value for missing data in one of the inputs Parameters other: Series or scalar value : ll_value : None or oat value, default None (NaN) Fill missing (NaN) values with this value. If both Series are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : Series pandas.Series.div Series.div(other, level=None, ll_value=None) Binary operator divide with support to substitute a ll_value for missing data in one of the inputs Parameters other: Series or scalar value : ll_value : None or oat value, default None (NaN) Fill missing (NaN) values with this value. If both Series are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : Series pandas.Series.mul Series.mul(other, level=None, ll_value=None) Binary operator multiply with support to substitute a ll_value for missing data in one of the inputs Parameters other: Series or scalar value : ll_value : None or oat value, default None (NaN) Fill missing (NaN) values with this value. If both Series are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : Series pandas.Series.sub Series.sub(other, level=None, ll_value=None) Binary operator subtract with support to substitute a ll_value for missing data in one of the inputs Parameters other: Series or scalar value : ll_value : None or oat value, default None (NaN) 218 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Fill missing (NaN) values with this value. If both Series are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : Series pandas.Series.combine Series.combine(other, func, ll_value=nan) Perform elementwise binary operation on two Series using given function with optional ll value when an index is missing from one Series or the other Parameters other : Series or scalar value func : function ll_value : scalar value Returns result : Series pandas.Series.combine_rst Series.combine_first(other) Combine Series values, choosing the calling Seriess values rst. Result index will be the union of the two indexes Parameters other : Series Returns y : Series

21.2.5 Function application, GroupBy

Series.apply(func) Series.map(arg) Series.groupby([by, axis, level, as_index, sort]) pandas.Series.apply Series.apply(func) Invoke function on values of Series. Can be ufunc or Python function expecting only single values Parameters func : function Returns y : Series pandas.Series.map Series.map(arg) Map values of Series using input correspondence (which can be a dict, Series, or function) Parameters arg : function, dict, or Series Invoke function on values of Series. Can be ufunc or Python function Map values of Series using input correspondence (which can be Group series using mapper (dict or key function, apply given function

21.2. Series

219

pandas: powerful Python data analysis toolkit, Release 0.7.1

Returns y : Series same index as caller

Examples >>> x one 1 two 2 three 3 >>> y 1 foo 2 bar 3 baz >>> x.map(y) one foo two bar three baz

pandas.Series.groupby Series.groupby(by=None, axis=0, level=None, as_index=True, sort=True) Group series using mapper (dict or key function, apply given function to group, return result as series) or by a series of columns Parameters by : mapping function / list of functions, dict, Series, or tuple / list of column names. Called on each element of the object index to determine the groups. If a dict or Series is passed, the Series or dict VALUES will be used to determine the groups axis : int, default 0 level : int, level name, or sequence of such, default None If the axis is a MultiIndex (hierarchical), group by a particular level or levels as_index : boolean, default True For aggregated output, return object with group labels as the index. Only relevant for DataFrame input. as_index=False is effectively SQL-style grouped output sort : boolean, default True Sort group keys. Get better performance by turning this off Returns GroupBy object :
Examples

# DataFrame result >>> data.groupby(func, axis=0).mean() # DataFrame result >>> data.groupby([col1, col2])[col3].mean() # DataFrame with hierarchical index >>> data.groupby([col1, col2]).mean()

220

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.2.6 Computations / Descriptive Stats

Series.autocorr() Series.clip([lower, upper, out]) Series.clip_lower(threshold) Series.clip_upper(threshold) Series.corr(other[, method]) Series.count([level]) Series.cumprod([axis, dtype, out, skipna]) Series.cumsum([axis, dtype, out, skipna]) Series.describe() Series.diff([periods]) Series.max([axis, out, skipna, level]) Series.mean([axis, dtype, out, skipna, level]) Series.median([skipna, level]) Series.min([axis, out, skipna, level]) Series.prod([axis, dtype, out, skipna, level]) Series.quantile([q]) Series.skew([skipna, level]) Series.std([axis, dtype, out, ddof, skipna, ...]) Series.sum([axis, dtype, out, skipna, level]) Series.var([axis, dtype, out, ddof, skipna, ...]) Series.value_counts() Lag-1 autocorrelation Trim values at input threshold(s) Return copy of series with values below given value truncated Return copy of series with values above given value truncated Compute correlation two Series, excluding missing values Return number of non-NA/null observations in the Series Cumulative product of values. Cumulative sum of values. Generate various summary statistics of Series, excluding NaN 1st discrete difference of object Return maximum of values Return mean of values Return median of values Return minimum of values Return product of values Return value at the given quantile, a la scoreatpercentile in Return unbiased skewness of values Return unbiased standard deviation of values Return sum of values Return unbiased variance of values Returns Series containing counts of unique values. The resulting Series

pandas.Series.autocorr Series.autocorr() Lag-1 autocorrelation Returns autocorr : oat pandas.Series.clip Series.clip(lower=None, upper=None, out=None) Trim values at input threshold(s) Parameters lower : oat, default None upper : oat, default None Returns clipped : Series pandas.Series.clip_lower Series.clip_lower(threshold) Return copy of series with values below given value truncated

21.2. Series

221

pandas: powerful Python data analysis toolkit, Release 0.7.1

Returns clipped : Series See Also: clip pandas.Series.clip_upper Series.clip_upper(threshold) Return copy of series with values above given value truncated Returns clipped : Series See Also: clip pandas.Series.corr Series.corr(other, method=pearson) Compute correlation two Series, excluding missing values Parameters other : Series method : {pearson, kendall, spearman} pearson : standard correlation coefcient kendall : Kendall Tau correlation coefcient spearman : Spearman rank correlation Returns correlation : oat pandas.Series.count Series.count(level=None) Return number of non-NA/null observations in the Series Parameters level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns nobs : int or Series (if level specied) pandas.Series.cumprod Series.cumprod(axis=0, dtype=None, out=None, skipna=True) Cumulative product of values. Preserves locations of NaN values Extra parameters are to preserve ndarray interface. Parameters skipna : boolean, default True Exclude NA/null values Returns cumprod : Series

222

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.cumsum Series.cumsum(axis=0, dtype=None, out=None, skipna=True) Cumulative sum of values. Preserves locations of NaN values Extra parameters are to preserve ndarray interface. Parameters skipna : boolean, default True Exclude NA/null values Returns cumsum : Series pandas.Series.describe Series.describe() Generate various summary statistics of Series, excluding NaN values. These include: count, mean, std, min, max, and 10%/50%/90% quantiles Returns desc : Series pandas.Series.diff Series.diff(periods=1) 1st discrete difference of object Parameters periods : int, default 1 Periods to shift for forming difference Returns diffed : Series pandas.Series.max Series.max(axis=None, out=None, skipna=True, level=None) Return maximum of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns max : oat (or Series if level specied) pandas.Series.mean Series.mean(axis=0, dtype=None, out=None, skipna=True, level=None) Return mean of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None

21.2. Series

223

pandas: powerful Python data analysis toolkit, Release 0.7.1

If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Extra parameters are to preserve ndarrayinterface. : Returns mean : oat (or Series if level specied) pandas.Series.median Series.median(skipna=True, level=None) Return median of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns median : oat (or Series if level specied) pandas.Series.min Series.min(axis=None, out=None, skipna=True, level=None) Return minimum of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns min : oat (or Series if level specied) pandas.Series.prod Series.prod(axis=None, dtype=None, out=None, skipna=True, level=None) Return product of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns product : oat (or Series if level specied)

224

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.quantile Series.quantile(q=0.5) Return value at the given quantile, a la scoreatpercentile in scipy.stats Parameters q : quantile 0 <= q <= 1 Returns quantile : oat pandas.Series.skew Series.skew(skipna=True, level=None) Return unbiased skewness of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns skew : oat (or Series if level specied) pandas.Series.std Series.std(axis=None, dtype=None, out=None, ddof=1, skipna=True, level=None) Return unbiased standard deviation of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns stdev : oat (or Series if level specied) pandas.Series.sum Series.sum(axis=0, dtype=None, out=None, skipna=True, level=None) Return sum of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Extra parameters are to preserve ndarrayinterface. : Returns sum : oat (or Series if level specied)

21.2. Series

225

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.var Series.var(axis=None, dtype=None, out=None, ddof=1, skipna=True, level=None) Return unbiased variance of values NA/null values are excluded Parameters skipna : boolean, default True Exclude NA/null values level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a smaller Series Returns var : oat (or Series if level specied) pandas.Series.value_counts Series.value_counts() Returns Series containing counts of unique values. The resulting Series will be in descending order so that the rst element is the most frequently-occurring element. Excludes NA values Returns counts : Series

21.2.7 Reindexing / Selection / Label manipulation

Series.align(other[, join, level, copy, ...]) Series.drop(labels[, axis]) Series.reindex([index, method, level, ...]) Series.reindex_like(other[, method]) Series.rename(mapper) Series.select(crit[, axis]) Series.take(indices[, axis]) Series.truncate([before, after, copy]) pandas.Series.align Series.align(other, join=outer, level=None, copy=True, ll_value=None, method=None) Align two Series object with the specied join method Parameters other : Series join : {outer, inner, left, right}, default outer level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level copy : boolean, default True Always return new objects. If copy=False and no reindexing is required, the same object will be returned (for better performance) ll_value : object, default None method : str, default pad Returns (left, right) : (Series, Series) Align two Series object with the specied join method Return new object with labels in requested axis removed Conform Series to new index with optional lling logic, placing Reindex Series to match index of another Series, optionally with Alter Series index using dict or function Return data corresponding to axis labels matching criteria Analogous to ndarray.take, return Series corresponding to requested Function truncate a sorted DataFrame / Series before and/or after

226

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Aligned Series pandas.Series.drop Series.drop(labels, axis=0) Return new object with labels in requested axis removed Parameters labels : array-like axis : int Returns dropped : type of caller pandas.Series.reindex Series.reindex(index=None, method=None, level=None, ll_value=nan, copy=True) Conform Series to new index with optional lling logic, placing NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one and copy=False Parameters index : array-like or Index New labels / index to conform to. Preferably an Index object to avoid duplicating data method : {backll, bll, pad, fll, None} Method to use for lling holes in reindexed Series pad / fll: propagate LAST valid observation forward to next valid backll / bll: use NEXT valid observation to ll gap copy : boolean, default True Return a new object, even if the passed indexes are the same level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level ll_value : scalar, default np.NaN Value to use for missing values. Defaults to NaN, but can be any compatible value Returns reindexed : Series pandas.Series.reindex_like Series.reindex_like(other, method=None) Reindex Series to match index of another Series, optionally with lling logic Parameters other : Series method : string or None See Series.reindex docstring Returns reindexed : Series
Notes

Like calling s.reindex(other.index, method=...)

21.2. Series

227

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.rename Series.rename(mapper) Alter Series index using dict or function Parameters mapper : dict-like or function Transformation to apply to each index Returns renamed : Series (new object)
Notes

Function / dict values must be unique (1-to-1)

Examples >>> foo bar baz >>> FOO BAR BAZ x 1 2 3 x.rename(str.upper) 1 2 3

>>> x.rename({foo : a, bar : b, baz : c}) a 1 b 2 c 3

pandas.Series.select Series.select(crit, axis=0) Return data corresponding to axis labels matching criteria Parameters crit : function To be called on each index (label). Should return True or False axis : int Returns selection : type of caller pandas.Series.take Series.take(indices, axis=0) Analogous to ndarray.take, return Series corresponding to requested indices Parameters indices : list / array of ints Returns taken : Series

228

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.truncate Series.truncate(before=None, after=None, copy=True) Function truncate a sorted DataFrame / Series before and/or after some particular dates. Parameters before : date Truncate before date after : date Truncate after date Returns truncated : type of caller

21.2.8 Missing data handling

Series.dropna() Series.fillna([value, method, inplace]) Series.interpolate([method]) pandas.Series.dropna Series.dropna() Return Series without null values Returns valid : Series pandas.Series.llna Series.fillna(value=None, method=pad, inplace=False) Fill NA/NaN values using the specied method Parameters value : any kind (should be same type as array) Value to use to ll holes (e.g. 0) method : {backll, bll, pad, fll, None}, default pad Method to use for lling holes in reindexed Series pad / fll: propagate last valid observation forward to next valid backll / bll: use NEXT valid observation to ll gap inplace : boolean, default False If True, ll the Series in place. Note: this will modify any other views on this Series, for example a column in a DataFrame. Returns a reference to the lled object, which is self if inplace=True Returns lled : Series See Also: reindex, asfreq Return Series without null values Fill NA/NaN values using the specied method Interpolate missing values (after the rst valid value)

21.2. Series

229

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.interpolate Series.interpolate(method=linear) Interpolate missing values (after the rst valid value) Parameters method : {linear, time} Interpolation method. Time interpolation works on daily and higher resolution data to interpolate given length of interval Returns interpolated : Series

21.2.9 Reshaping, sorting

Series.argsort([axis, kind, order]) Series.order([na_last, ascending, kind]) Series.sort([axis, kind, order]) Series.sort_index([ascending]) Series.sortlevel([level, ascending]) Series.unstack([level]) pandas.Series.argsort Series.argsort(axis=0, kind=quicksort, order=None) Overrides ndarray.argsort. Argsorts the value, omitting NA/null values, and places the result in the same locations as the non-NA values Parameters axis : int (can only be zero) kind : {mergesort, quicksort, heapsort}, default quicksort Choice of sorting algorithm. See np.sort for more information. mergesort is the only stable algorithm order : ignored Returns argsorted : Series pandas.Series.order Series.order(na_last=True, ascending=True, kind=mergesort) Sorts Series object, by value, maintaining index-value link Parameters na_last : boolean (optional, default=True) Put NaNs at beginning or end ascending : boolean, default True Sort ascending. Passing False sorts descending kind : {mergesort, quicksort, heapsort}, default mergesort Choice of sorting algorithm. See np.sort for more information. mergesort is the only stable algorith Returns y : Series Overrides ndarray.argsort. Sorts Series object, by value, maintaining index-value link Sort values and index labels by value, in place. Sort object by labels (along an axis) Sort Series with MultiIndex by chosen level. Data will be Unstack, a.k.a.

230

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.Series.sort Series.sort(axis=0, kind=quicksort, order=None) Sort values and index labels by value, in place. For compatibility with ndarray API. No return value Parameters axis : int (can only be zero) kind : {mergesort, quicksort, heapsort}, default quicksort Choice of sorting algorithm. See np.sort for more information. mergesort is the only stable algorithm order : ignored pandas.Series.sort_index Series.sort_index(ascending=True) Sort object by labels (along an axis) Parameters ascending : boolean, default True Sort ascending vs. descending Returns sorted_obj : Series pandas.Series.sortlevel Series.sortlevel(level=0, ascending=True) Sort Series with MultiIndex by chosen level. Data will be lexicographically sorted by the chosen level followed by the other levels (in order) Parameters level : int ascending : bool, default True Returns sorted : Series pandas.Series.unstack Series.unstack(level=-1) Unstack, a.k.a. pivot, Series with MultiIndex to produce DataFrame Parameters level : int, string, or list of these, default last level Level(s) to unstack, can pass level name Returns unstacked : DataFrame
Examples >>> s one a one b two a two b

1. 2. 3. 4.

21.2. Series

231

pandas: powerful Python data analysis toolkit, Release 0.7.1

>>> s.unstack(level=-1) a b one 1. 2. two 3. 4. >>> s.unstack(level=0) one two a 1. 2. b 3. 4.

21.2.10 Combining / joining / merging

Series.append(to_append) pandas.Series.append Series.append(to_append) Concatenate two or more Series. The indexes must not overlap Parameters to_append : Series or list/tuple of Series Returns appended : Series Concatenate two or more Series. The indexes must not overlap

21.2.11 Time series-related

Series.asfreq(freq[, method]) Series.asof(date) Series.shift(periods, **kwds[, offset]) Series.first_valid_index() Series.last_valid_index() Series.weekday pandas.Series.asfreq Series.asfreq(freq, method=None) Convert this TimeSeries to the provided frequency using DateOffset object or time rule. Optionally provide ll method to pad/backll missing values. Parameters offset : DateOffset object, or string in {WEEKDAY, EOM} DateOffset object or subclass (e.g. monthEnd) method : {backll, pad, None} Method to use for lling holes in new index Returns converted : TimeSeries pandas.Series.asof Series.asof(date) Return last good (non-NaN) value in TimeSeries if value is NaN for requested date. If there is no good value, NaN is returned. 232 Chapter 21. API Reference Convert this TimeSeries to the provided frequency using DateOffset Return last good (non-NaN) value in TimeSeries if value is NaN for Shift the index of the Series by desired number of periods with an Return label for rst non-NA/null value Return label for last non-NA/null value

pandas: powerful Python data analysis toolkit, Release 0.7.1

Parameters date : datetime or similar value Returns value or NaN :

Notes

Dates are assumed to be sorted pandas.Series.shift Series.shift(periods, offset=None, **kwds) Shift the index of the Series by desired number of periods with an optional time offset Parameters periods : int Number of periods to move, can be positive or negative offset : DateOffset, timedelta, or time rule string, optional Increment to use from datetools module or time rule (e.g. EOM) Returns shifted : Series pandas.Series.rst_valid_index Series.first_valid_index() Return label for rst non-NA/null value pandas.Series.last_valid_index Series.last_valid_index() Return label for last non-NA/null value pandas.Series.weekday Series.weekday

21.2.12 Plotting
Series.hist(**kwds[, ax, grid]) Series.plot(**kwds[, label, kind, ...]) pandas.Series.hist Series.hist(ax=None, grid=True, **kwds) Draw histogram of the input series using matplotlib Parameters ax : matplotlib axis object If not passed, uses gca() kwds : keywords To be passed to the actual plotting function 21.2. Series 233 Draw histogram of the input series using matplotlib Plot the input series with the index on the x-axis using matplotlib

pandas: powerful Python data analysis toolkit, Release 0.7.1

Notes

See matplotlib documentation online for more on this pandas.Series.plot Series.plot(label=None, kind=line, use_index=True, rot=30, ax=None, style=-, grid=True, logy=False, **kwds) Plot the input series with the index on the x-axis using matplotlib Parameters label : label argument to provide to plot kind : {line, bar} rot : int, default 30 Rotation for tick labels use_index : boolean, default True Plot index as axis tick labels ax : matplotlib axis object If not passed, uses gca() style : string, default - matplotlib line style to use kwds : keywords To be passed to the actual plotting function
Notes

See matplotlib documentation online for more on this subject Intended to be used in ipython pylab mode

21.2.13 Serialization / IO / Conversion

Series.from_csv(path[, sep, parse_dates, ...]) Series.load(path) Series.save(path) Series.to_csv(path[, index, sep, na_rep, ...]) Series.to_dict() Series.to_sparse([kind, ll_value]) pandas.Series.from_csv classmethod Series.from_csv(path, sep=, , parse_dates=True, header=None, index_col=0, encoding=None) Read delimited le into Series Parameters path : string sep : string, default , Field delimiter Read delimited le into Series

Write Series to a comma-separated values (csv) le Convert Series to {label -> value} dict Convert Series to SparseSeries

234

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

parse_dates : boolean, default True Parse dates. Different default from read_table header : int, default 0 Row to use at header (skip prior rows) index_col : int or sequence, default 0 Column to use for index. If a sequence is given, a MultiIndex is used. Different default from read_table encoding : string, optional a string representing the encoding to use if the contents are non-ascii, for python versions prior to 3 Returns y : Series pandas.Series.load classmethod Series.load(path) pandas.Series.save Series.save(path) pandas.Series.to_csv Series.to_csv(path, index=True, sep=, , na_rep=, header=False, index_label=None, mode=w, nanRep=None, encoding=None) Write Series to a comma-separated values (csv) le Parameters path : string File path nanRep : string, default Missing data repn header : boolean, default False Write out series name index : boolean, default True Write row names (index) index_label : string or sequence, default None Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. mode : Python write mode, default w sep : character, default , Field delimiter for the output le. encoding : string, optional

21.2. Series

235

pandas: powerful Python data analysis toolkit, Release 0.7.1

a string representing the encoding to use if the contents are non-ascii, for python versions prior to 3 pandas.Series.to_dict Series.to_dict() Convert Series to {label -> value} dict Returns value_dict : dict pandas.Series.to_sparse Series.to_sparse(kind=block, ll_value=None) Convert Series to SparseSeries Parameters kind : {block, integer} ll_value : oat, defaults to NaN (missing) Returns sp : SparseSeries

21.3 DataFrame
21.3.1 Attributes and underlying data
Axes index: row labels columns: column labels DataFrame.as_matrix([columns]) DataFrame.dtypes DataFrame.get_dtype_counts() DataFrame.values DataFrame.axes DataFrame.ndim DataFrame.shape pandas.DataFrame.as_matrix DataFrame.as_matrix(columns=None) Convert the frame to its Numpy-array matrix representation. Columns are presented in sorted order unless a specic list of columns is provided. Parameters columns : array-like Specic column order Returns values : ndarray If the DataFrame is heterogeneous and contains booleans or objects, the result will be of dtype=object Convert the frame to its Numpy-array matrix representation. Columns

Convert the frame to its Numpy-array matrix representation. Columns

236

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.dtypes DataFrame.dtypes pandas.DataFrame.get_dtype_counts DataFrame.get_dtype_counts() pandas.DataFrame.values DataFrame.values Convert the frame to its Numpy-array matrix representation. Columns are presented in sorted order unless a specic list of columns is provided. Parameters columns : array-like Specic column order Returns values : ndarray If the DataFrame is heterogeneous and contains booleans or objects, the result will be of dtype=object pandas.DataFrame.axes DataFrame.axes pandas.DataFrame.ndim DataFrame.ndim pandas.DataFrame.shape DataFrame.shape

21.3.2 Conversion / Constructors

DataFrame.__init__([data, index, columns, ...]) DataFrame.astype(dtype) DataFrame.copy([deep]) pandas.DataFrame.__init__ DataFrame.__init__(data=None, index=None, columns=None, dtype=None, copy=False) Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). Arithmetic operations align on both row and column labels. Can be thought of as a dict-like container for Series objects. The primary pandas data structure Parameters data : numpy ndarray (structured or homogeneous), dict, or DataFrame Dict can contain Series, arrays, constants, or list-like objects 21.3. DataFrame 237 Two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). Cast object to input numpy.dtype Make a copy of this object

pandas: powerful Python data analysis toolkit, Release 0.7.1

index : Index or array-like Index to use for resulting frame. Will default to np.arange(n) if no indexing information part of input data and no index provided columns : Index or array-like Will default to np.arange(n) if not column labels provided dtype : dtype, default None Data type to force, otherwise infer copy : boolean, default False Copy data from inputs. Only affects DataFrame / 2d ndarray input See Also: DataFrame.from_records constructor from tuples, also record arrays DataFrame.from_dict from dicts of Series, arrays, or dicts DataFrame.from_csv from CSV les DataFrame.from_items from sequence of (key, value) pairs read_csv
Examples >>> >>> >>> >>> ... d = {col1: ts1, col2: ts2} df = DataFrame(data=d, index=index) df2 = DataFrame(np.random.randn(10, 5)) df3 = DataFrame(np.random.randn(10, 5), columns=[a, b, c, d, e])

pandas.DataFrame.astype DataFrame.astype(dtype) Cast object to input numpy.dtype Parameters dtype : numpy.dtype or Python type Returns casted : type of caller pandas.DataFrame.copy DataFrame.copy(deep=True) Make a copy of this object Parameters deep : boolean, default True Make a deep copy, i.e. also copy data Returns copy : type of caller

238

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.3.3 Indexing, iteration

DataFrame.ix DataFrame.insert(loc, column, value) DataFrame.__iter__() DataFrame.iteritems() DataFrame.pop(item) DataFrame.xs(key[, axis, level, copy]) pandas.DataFrame.ix DataFrame.ix pandas.DataFrame.insert DataFrame.insert(loc, column, value) Insert column into DataFrame at specied location. Raises Exception if column is already contained in the DataFrame Parameters loc : int Must have 0 <= loc <= len(columns) column : object value : int, Series, or array-like pandas.DataFrame.__iter__ DataFrame.__iter__() Iterate over columns of the frame. pandas.DataFrame.iteritems DataFrame.iteritems() Iterator over (column, series) pairs pandas.DataFrame.pop DataFrame.pop(item) Return column and drop from frame. Raise KeyError if not found. Returns column : Series pandas.DataFrame.xs DataFrame.xs(key, axis=0, level=None, copy=True) Returns a cross-section (row or column) from the DataFrame as a Series object. Defaults to returning a row (axis 0) Parameters key : object Some label contained in the index, or partially in a MultiIndex 21.3. DataFrame 239 Insert column into DataFrame at specied location. Raises Exception if Iterate over columns of the frame. Iterator over (column, series) pairs Return column and drop from frame. Returns a cross-section (row or column) from the DataFrame as a Series

pandas: powerful Python data analysis toolkit, Release 0.7.1

axis : int, default 0 Axis to retrieve cross-section on copy : boolean, default True Whether to make a copy of the data Returns xs : Series

21.3.4 Binary operator functions

DataFrame.add(other[, axis, level, ll_value]) DataFrame.div(other[, axis, level, ll_value]) DataFrame.mul(other[, axis, level, ll_value]) DataFrame.sub(other[, axis, level, ll_value]) DataFrame.radd(other[, axis, level, ll_value]) DataFrame.rdiv(other[, axis, level, ll_value]) DataFrame.rmul(other[, axis, level, ll_value]) DataFrame.rsub(other[, axis, level, ll_value]) DataFrame.combine(other, func[, ll_value]) DataFrame.combineAdd(other) DataFrame.combine_first(other) DataFrame.combineMult(other) Binary operator add with support to substitute a ll_value for missing data in Binary operator divide with support to substitute a ll_value for missing data in Binary operator multiply with support to substitute a ll_value for missing data in Binary operator subtract with support to substitute a ll_value for missing data in Binary operator radd with support to substitute a ll_value for missing data in Binary operator rdivide with support to substitute a ll_value for missing data in Binary operator rmultiply with support to substitute a ll_value for missing data in Binary operator rsubtract with support to substitute a ll_value for missing data in Add two DataFrame objects and do not propagate NaN values, so if for a Add two DataFrame objects and do not propagate Combine two DataFrame objects and default to non-null values in frame Multiply two DataFrame objects and do not propagate NaN values, so if

pandas.DataFrame.add DataFrame.add(other, axis=columns, level=None, ll_value=None) Binary operator add with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame

240

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Notes

Mismatched indices will be unioned together pandas.DataFrame.div DataFrame.div(other, axis=columns, level=None, ll_value=None) Binary operator divide with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.mul DataFrame.mul(other, axis=columns, level=None, ll_value=None) Binary operator multiply with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together

21.3. DataFrame

241

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.sub DataFrame.sub(other, axis=columns, level=None, ll_value=None) Binary operator subtract with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.radd DataFrame.radd(other, axis=columns, level=None, ll_value=None) Binary operator radd with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.rdiv DataFrame.rdiv(other, axis=columns, level=None, ll_value=None) Binary operator rdivide with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} 242 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.rmul DataFrame.rmul(other, axis=columns, level=None, ll_value=None) Binary operator rmultiply with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.rsub DataFrame.rsub(other, axis=columns, level=None, ll_value=None) Binary operator rsubtract with support to substitute a ll_value for missing data in one of the inputs Parameters other : Series, DataFrame, or constant axis : {0, 1, index, columns} For Series input, axis to match Series index on ll_value : None or oat value, default None Fill missing (NaN) values with this value. If both DataFrame locations are missing, the result will be missing level : int or name

21.3. DataFrame

243

pandas: powerful Python data analysis toolkit, Release 0.7.1

Broadcast across a level, matching Index values on the passed MultiIndex level Returns result : DataFrame
Notes

Mismatched indices will be unioned together pandas.DataFrame.combine DataFrame.combine(other, func, ll_value=None) Add two DataFrame objects and do not propagate NaN values, so if for a (column, time) one frame is missing a value, it will default to the other frames value (which might be NaN as well) Parameters other : DataFrame func : function ll_value : scalar value Returns result : DataFrame pandas.DataFrame.combineAdd DataFrame.combineAdd(other) Add two DataFrame objects and do not propagate NaN values, so if for a (column, time) one frame is missing a value, it will default to the other frames value (which might be NaN as well) Parameters other : DataFrame Returns DataFrame : pandas.DataFrame.combine_rst DataFrame.combine_first(other) Combine two DataFrame objects and default to non-null values in frame calling the method. Result index will be the union of the two indexes Parameters other : DataFrame Returns combined : DataFrame
Examples >>> a.combine_first(b) as values prioritized, use values from b to fill holes

pandas.DataFrame.combineMult DataFrame.combineMult(other) Multiply two DataFrame objects and do not propagate NaN values, so if for a (column, time) one frame is missing a value, it will default to the other frames value (which might be NaN as well) Parameters other : DataFrame

244

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Returns DataFrame :

21.3.5 Function application, GroupBy

DataFrame.apply(func, **kwds[, axis, ...]) DataFrame.applymap(func) DataFrame.groupby([by, axis, level, ...]) pandas.DataFrame.apply DataFrame.apply(func, axis=0, broadcast=False, raw=False, args=(), **kwds) Applies function along input axis of DataFrame. Objects passed to functions are Series objects having index either the DataFrames index (axis=0) or the columns (axis=1). Returns either a DataFrame (if the function produces another Series) or a Series indexed on either the index or columns if the function produces an aggregated value. Parameters func : function Function to apply to each column axis : {0, 1} broadcast : bool, default False For aggregation functions, return object of same size with values propagated raw : boolean, default False If False, convert each row or column into a Series. If raw=True the passed function will receive ndarray objects instead. If you are just applying a NumPy reduction function this will achieve much better performance args : tuple Positional arguments to pass to function in addition to the array/series Additional keyword arguments will be passed as keywords to the function : Returns applied : Series or DataFrame
Notes

Applies function along input axis of DataFrame. Objects passed to Apply a function to a DataFrame that is intended to operate Group series using mapper (dict or key function, apply given function

Function passed should not have side effects. If the result is a Series, it should have the same index
Examples >>> df.apply(numpy.sqrt) # returns DataFrame >>> df.apply(numpy.sum, axis=0) # equiv to df.sum(0) >>> df.apply(numpy.sum, axis=1) # equiv to df.sum(1)

21.3. DataFrame

245

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.applymap DataFrame.applymap(func) Apply a function to a DataFrame that is intended to operate elementwise, i.e. like doing map(func, series) for each series in the DataFrame Parameters func : function Python function, returns a single value from a single value Returns applied : DataFrame pandas.DataFrame.groupby DataFrame.groupby(by=None, axis=0, level=None, as_index=True, sort=True) Group series using mapper (dict or key function, apply given function to group, return result as series) or by a series of columns Parameters by : mapping function / list of functions, dict, Series, or tuple / list of column names. Called on each element of the object index to determine the groups. If a dict or Series is passed, the Series or dict VALUES will be used to determine the groups axis : int, default 0 level : int, level name, or sequence of such, default None If the axis is a MultiIndex (hierarchical), group by a particular level or levels as_index : boolean, default True For aggregated output, return object with group labels as the index. Only relevant for DataFrame input. as_index=False is effectively SQL-style grouped output sort : boolean, default True Sort group keys. Get better performance by turning this off Returns GroupBy object :
Examples

# DataFrame result >>> data.groupby(func, axis=0).mean() # DataFrame result >>> data.groupby([col1, col2])[col3].mean() # DataFrame with hierarchical index >>> data.groupby([col1, col2]).mean()

246

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.3.6 Computations / Descriptive Stats

DataFrame.clip([upper, lower]) DataFrame.clip_lower(threshold) DataFrame.clip_upper(threshold) DataFrame.corr([method]) DataFrame.corrwith(other[, axis, drop]) DataFrame.count([axis, level, numeric_only]) DataFrame.cumprod([axis, skipna]) DataFrame.cumsum([axis, skipna]) DataFrame.describe() DataFrame.diff([periods]) DataFrame.mad([axis, skipna, level]) DataFrame.max([axis, skipna, level]) DataFrame.mean([axis, skipna, level]) DataFrame.median([axis, skipna, level]) DataFrame.min([axis, skipna, level]) DataFrame.prod([axis, skipna, level]) DataFrame.quantile([q, axis]) DataFrame.skew([axis, skipna, level]) DataFrame.sum([axis, numeric_only, skipna, ...]) DataFrame.std([axis, skipna, level]) DataFrame.var([axis, skipna, level]) pandas.DataFrame.clip DataFrame.clip(upper=None, lower=None) Trim values at input threshold(s) Parameters lower : oat, default None upper : oat, default None Returns clipped : DataFrame pandas.DataFrame.clip_lower DataFrame.clip_lower(threshold) Trim values below threshold Returns clipped : DataFrame pandas.DataFrame.clip_upper DataFrame.clip_upper(threshold) Trim values above threshold Returns clipped : DataFrame Trim values at input threshold(s) Trim values below threshold Trim values above threshold Compute pairwise correlation of columns, excluding NA/null values Compute pairwise correlation between rows or columns of two DataFrame Return Series with number of non-NA/null observations over requested Return cumulative product over requested axis as DataFrame Return DataFrame of cumulative sums over requested axis. Generate various summary statistics of each column, excluding NaN 1st discrete difference of object Return mean absolute deviation over requested axis. Return maximum over requested axis. Return mean over requested axis. Return median over requested axis. Return minimum over requested axis. Return product over requested axis. Return values at the given quantile over requested axis, a la Return unbiased skewness over requested axis. Return sum over requested axis. Return unbiased standard deviation over requested axis. Return unbiased variance over requested axis.

21.3. DataFrame

247

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.corr DataFrame.corr(method=pearson) Compute pairwise correlation of columns, excluding NA/null values Parameters method : {pearson, kendall, spearman} pearson : standard correlation coefcient kendall : Kendall Tau correlation coefcient spearman : Spearman rank correlation Returns y : DataFrame pandas.DataFrame.corrwith DataFrame.corrwith(other, axis=0, drop=False) Compute pairwise correlation between rows or columns of two DataFrame objects. Parameters other : DataFrame axis : {0, 1} 0 to compute column-wise, 1 for row-wise drop : boolean, default False Drop missing indices from result, default returns union of all Returns correls : Series pandas.DataFrame.count DataFrame.count(axis=0, level=None, numeric_only=False) Return Series with number of non-NA/null observations over requested axis. Works with non-oating point data as well (detects NaN and None) Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame numeric_only : boolean, default False Include only oat, int, boolean data Returns count : Series (or DataFrame if level specied) pandas.DataFrame.cumprod DataFrame.cumprod(axis=None, skipna=True) Return cumulative product over requested axis as DataFrame Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA 248 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Returns y : DataFrame pandas.DataFrame.cumsum DataFrame.cumsum(axis=None, skipna=True) Return DataFrame of cumulative sums over requested axis. Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA Returns y : DataFrame pandas.DataFrame.describe DataFrame.describe() Generate various summary statistics of each column, excluding NaN values. These include: count, mean, std, min, max, and 10%/50%/90% quantiles Returns DataFrame of summary statistics : pandas.DataFrame.diff DataFrame.diff(periods=1) 1st discrete difference of object Parameters periods : int, default 1 Periods to shift for forming difference Returns diffed : DataFrame pandas.DataFrame.mad DataFrame.mad(axis=0, skipna=True, level=None) Return mean absolute deviation over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns mad : Series (or DataFrame if level specied)

21.3. DataFrame

249

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.max DataFrame.max(axis=0, skipna=True, level=None) Return maximum over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns max : Series (or DataFrame if level specied) pandas.DataFrame.mean DataFrame.mean(axis=0, skipna=True, level=None) Return mean over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns mean : Series (or DataFrame if level specied) pandas.DataFrame.median DataFrame.median(axis=0, skipna=True, level=None) Return median over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns median : Series (or DataFrame if level specied)

250

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.min DataFrame.min(axis=0, skipna=True, level=None) Return minimum over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns min : Series (or DataFrame if level specied) pandas.DataFrame.prod DataFrame.prod(axis=0, skipna=True, level=None) Return product over requested axis. NA/null values are treated as 1 Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns product : Series (or DataFrame if level specied) pandas.DataFrame.quantile DataFrame.quantile(q=0.5, axis=0) Return values at the given quantile over requested axis, a la scoreatpercentile in scipy.stats Parameters q : quantile, default 0.5 (50% quantile) 0 <= q <= 1 axis : {0, 1} 0 for row-wise, 1 for column-wise Returns quantiles : Series pandas.DataFrame.skew DataFrame.skew(axis=0, skipna=True, level=None) Return unbiased skewness over requested axis. NA/null values are excluded Parameters axis : {0, 1}

21.3. DataFrame

251

pandas: powerful Python data analysis toolkit, Release 0.7.1

0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns skew : Series (or DataFrame if level specied) pandas.DataFrame.sum DataFrame.sum(axis=0, numeric_only=None, skipna=True, level=None) Return sum over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame numeric_only : boolean, default None Include only oat, int, boolean data. If None, will attempt to use everything, then use only numeric data Returns sum : Series (or DataFrame if level specied) pandas.DataFrame.std DataFrame.std(axis=0, skipna=True, level=None) Return unbiased standard deviation over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns std : Series (or DataFrame if level specied)

252

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.var DataFrame.var(axis=0, skipna=True, level=None) Return unbiased variance over requested axis. NA/null values are excluded Parameters axis : {0, 1} 0 for row-wise, 1 for column-wise skipna : boolean, default True Exclude NA/null values. If an entire row/column is NA, the result will be NA level : int, default None If the axis is a MultiIndex (hierarchical), count along a particular level, collapsing into a DataFrame Returns var : Series (or DataFrame if level specied)

21.3.7 Reindexing / Selection / Label manipulation

DataFrame.add_prefix(prex) DataFrame.add_suffix(sufx) DataFrame.align(other[, join, axis, level, ...]) DataFrame.drop(labels[, axis]) DataFrame.filter([items, like, regex]) DataFrame.reindex([index, columns, method, ...]) DataFrame.reindex_like(other[, method, copy]) DataFrame.rename([index, columns, copy]) DataFrame.select(crit[, axis]) DataFrame.take(indices[, axis]) DataFrame.truncate([before, after, copy]) DataFrame.head([n]) DataFrame.tail([n]) pandas.DataFrame.add_prex DataFrame.add_prefix(prex) Concatenate prex string with panel items names. Parameters prex : string Returns with_prex : type of caller pandas.DataFrame.add_sufx DataFrame.add_suffix(sufx) Concatenate sufx string with panel items names Concatenate prex string with panel items names. Concatenate sufx string with panel items names Align two DataFrame object on their index and columns with the Return new object with labels in requested axis removed Restrict frames columns to set of items or wildcard Conform DataFrame to new index with optional lling logic, placing Reindex DataFrame to match indices of another DataFrame, optionally Alter index and / or columns using input function or functions. Return data corresponding to axis labels matching criteria Analogous to ndarray.take, return DataFrame corresponding to requested Function truncate a sorted DataFrame / Series before and/or after Returns rst n rows of DataFrame Returns last n rows of DataFrame

21.3. DataFrame

253

pandas: powerful Python data analysis toolkit, Release 0.7.1

Parameters sufx : string Returns with_sufx : type of caller pandas.DataFrame.align DataFrame.align(other, join=outer, axis=None, level=None, copy=True, ll_value=nan, method=None) Align two DataFrame object on their index and columns with the specied join method for each axis Index Parameters other : DataFrame or Series join : {outer, inner, left, right}, default outer axis : {0, 1, None}, default None Align on index (0), columns (1), or both (None) level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level copy : boolean, default True Always returns new objects. If copy=False and no reindexing is required then original objects are returned. ll_value : scalar, default np.NaN Value to use for missing values. Defaults to NaN, but can be any compatible value method : str, default None Returns (left, right) : (DataFrame, type of other) Aligned objects pandas.DataFrame.drop DataFrame.drop(labels, axis=0) Return new object with labels in requested axis removed Parameters labels : array-like axis : int Returns dropped : type of caller pandas.DataFrame.lter DataFrame.filter(items=None, like=None, regex=None) Restrict frames columns to set of items or wildcard Parameters items : list-like List of columns to restrict to (must not all be present) like : string Keep columns where arg in col == True regex : string (regular expression) Keep columns with re.search(regex, col) == True 254 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Returns DataFrame with ltered columns :

Notes

Arguments are mutually exclusive, but this is not checked for pandas.DataFrame.reindex DataFrame.reindex(index=None, columns=None, method=None, level=None, ll_value=nan, copy=True) Conform DataFrame to new index with optional lling logic, placing NA/NaN in locations having no value in the previous index. A new object is produced unless the new index is equivalent to the current one and copy=False Parameters index : array-like, optional New labels / index to conform to. Preferably an Index object to avoid duplicating data columns : array-like, optional Same usage as index argument method : {backll, bll, pad, fll, None}, default None Method to use for lling holes in reindexed DataFrame pad / fll: propagate last valid observation forward to next valid backll / bll: use NEXT valid observation to ll gap copy : boolean, default True Return a new object, even if the passed indexes are the same level : int or name Broadcast across a level, matching Index values on the passed MultiIndex level ll_value : scalar, default np.NaN Value to use for missing values. Defaults to NaN, but can be any compatible value Returns reindexed : same type as calling instance
Examples >>> df.reindex(index=[date1, date2, date3], columns=[A, B, C])

pandas.DataFrame.reindex_like DataFrame.reindex_like(other, method=None, copy=True) Reindex DataFrame to match indices of another DataFrame, optionally with lling logic Parameters other : DataFrame method : string or None copy : boolean, default True Returns reindexed : DataFrame

21.3. DataFrame

255

pandas: powerful Python data analysis toolkit, Release 0.7.1

Notes

Like calling s.reindex(index=other.index, columns=other.columns, method=...) pandas.DataFrame.rename DataFrame.rename(index=None, columns=None, copy=True) Alter index and / or columns using input function or functions. Function / dict values must be unique (1-to-1). Labels not contained in a dict / Series will be left as-is. Parameters index : dict-like or function, optional Transformation to apply to index values columns : dict-like or function, optional Transformation to apply to column values copy : boolean, default True Also copy underlying data Returns renamed : DataFrame (new object) See Also: Series.rename pandas.DataFrame.select DataFrame.select(crit, axis=0) Return data corresponding to axis labels matching criteria Parameters crit : function To be called on each index (label). Should return True or False axis : int Returns selection : type of caller pandas.DataFrame.take DataFrame.take(indices, axis=0) Analogous to ndarray.take, return DataFrame corresponding to requested indices along an axis Parameters indices : list / array of ints axis : {0, 1} Returns taken : DataFrame pandas.DataFrame.truncate DataFrame.truncate(before=None, after=None, copy=True) Function truncate a sorted DataFrame / Series before and/or after some particular dates. Parameters before : date Truncate before date 256 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

after : date Truncate after date Returns truncated : type of caller pandas.DataFrame.head DataFrame.head(n=5) Returns rst n rows of DataFrame pandas.DataFrame.tail DataFrame.tail(n=5) Returns last n rows of DataFrame

21.3.8 Missing data handling

DataFrame.dropna([axis, how, thresh, subset]) DataFrame.fillna([value, method, inplace]) pandas.DataFrame.dropna DataFrame.dropna(axis=0, how=any, thresh=None, subset=None) Return object with labels on given axis omitted where alternately any or all of the data are missing Parameters axis : {0, 1} how : {any, all} any : if any NA values are present, drop that label all : if all values are NA, drop that label thresh : int, default None int value : require that many non-NA values subset : array-like Labels along other axis to consider, e.g. if you are dropping rows these would be a list of columns to include Returns dropped : DataFrame pandas.DataFrame.llna DataFrame.fillna(value=None, method=pad, inplace=False) Fill NA/NaN values using the specied method Parameters method : {backll, bll, pad, fll, None}, default pad Method to use for lling holes in reindexed Series pad / fll: propagate last valid observation forward to next valid backll / bll: use NEXT valid observation to ll gap Return object with labels on given axis omitted where alternately any Fill NA/NaN values using the specied method

21.3. DataFrame

257

pandas: powerful Python data analysis toolkit, Release 0.7.1

value : any kind (should be same type as array) Value to use to ll holes (e.g. 0) inplace : boolean, default False If True, ll the DataFrame in place. Note: this will modify any other views on this DataFrame, like if you took a no-copy slice of an existing DataFrame, for example a column in a DataFrame. Returns a reference to the lled object, which is self if inplace=True Returns lled : DataFrame See Also: reindex, asfreq

21.3.9 Reshaping, sorting, transposing

DataFrame.sort_index([axis, by, ascending]) DataFrame.delevel(*args, **kwargs) DataFrame.pivot([index, columns, values]) DataFrame.sortlevel([level, axis, ascending]) DataFrame.swaplevel(i, j[, axis]) DataFrame.stack([level, dropna]) DataFrame.unstack([level]) DataFrame.T DataFrame.transpose() Sort DataFrame either by labels (along either axis) or by the values in Reshape data (produce a pivot table) based on column values. Sort multilevel index by chosen axis and primary level. Swap levels i and j in a MultiIndex on a particular axis Pivot a level of the (possibly hierarchical) column labels, returning a Pivot a level of the (necessarily hierarchical) index labels, returning Returns a DataFrame with the rows/columns switched. If the DataFrame is Returns a DataFrame with the rows/columns switched. If the DataFrame is

pandas.DataFrame.sort_index DataFrame.sort_index(axis=0, by=None, ascending=True) Sort DataFrame either by labels (along either axis) or by the values in a column Parameters axis : {0, 1} Sort index/rows versus columns by : object Column name(s) in frame. Accepts a column name or a list or tuple for a nested sort. ascending : boolean, default True Sort ascending vs. descending Returns sorted : DataFrame pandas.DataFrame.delevel DataFrame.delevel(*args, **kwargs) 258 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.pivot DataFrame.pivot(index=None, columns=None, values=None) Reshape data (produce a pivot table) based on column values. Uses unique values from index / columns to form axes and return either DataFrame or Panel, depending on whether you request a single value column (DataFrame) or all columns (Panel) Parameters index : string or object Column name to use to make new frames index columns : string or object Column name to use to make new frames columns values : string or object, optional Column name to use for populating new frames values Returns pivoted : DataFrame If no values column specied, will have hierarchically indexed columns
Notes

For ner-tuned control, see hierarchical indexing documentation along with the related stack/unstack methods
Examples >>> df foo 0 one 1 one 2 one 3 two 4 two 5 two

bar A B C A B C

baz 1. 2. 3. 4. 5. 6.

>>> df.pivot(foo, bar, baz) A B C one 1 2 3 two 4 5 6 >>> df.pivot(foo, bar)[baz] A B C one 1 2 3 two 4 5 6

pandas.DataFrame.sortlevel DataFrame.sortlevel(level=0, axis=0, ascending=True) Sort multilevel index by chosen axis and primary level. Data will be lexicographically sorted by the chosen level followed by the other levels (in order) Parameters level : int axis : {0, 1}

21.3. DataFrame

259

pandas: powerful Python data analysis toolkit, Release 0.7.1

ascending : bool, default True Returns sorted : DataFrame pandas.DataFrame.swaplevel DataFrame.swaplevel(i, j, axis=0) Swap levels i and j in a MultiIndex on a particular axis Returns swapped : type of caller (new object) pandas.DataFrame.stack DataFrame.stack(level=-1, dropna=True) Pivot a level of the (possibly hierarchical) column labels, returning a DataFrame (or Series in the case of an object with a single level of column labels) having a hierarchical index with a new inner-most level of row labels. Parameters level : int, string, or list of these, default last level Level(s) to stack, can pass level name dropna : boolean, default True Whether to drop rows in the resulting Frame/Series with no valid values Returns stacked : DataFrame or Series
Examples >>> s one two a 1. 3. b 2. 4.

>>> s.stack() one a 1 b 2 two a 3 b 4

pandas.DataFrame.unstack DataFrame.unstack(level=-1) Pivot a level of the (necessarily hierarchical) index labels, returning a DataFrame having a new level of column labels whose inner-most level consists of the pivoted index labels. If the index is not a MultiIndex, the output will be a Series (the analogue of stack when the columns are not a MultiIndex) Parameters level : int, string, or list of these, default last level Level(s) of index to unstack, can pass level name Returns unstacked : DataFrame or Series

260

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Examples >>> s one a one b two a two b

1. 2. 3. 4.

>>> s.unstack(level=-1) a b one 1. 2. two 3. 4. >>> df = s.unstack(level=0) >>> df one two a 1. 2. b 3. 4. >>> df.unstack() one a 1. b 3. two a 2. b 4.

pandas.DataFrame.T DataFrame.T Returns a DataFrame with the rows/columns switched. If the DataFrame is homogeneously-typed, the data is not copied pandas.DataFrame.transpose DataFrame.transpose() Returns a DataFrame with the rows/columns switched. If the DataFrame is homogeneously-typed, the data is not copied

21.3.10 Combining / joining / merging

DataFrame.join(other[, on, how, lsufx, ...]) DataFrame.merge(right[, how, on, left_on, ...]) DataFrame.append(other[, ignore_index, ...]) pandas.DataFrame.join DataFrame.join(other, on=None, how=left, lsufx=, rsufx=, sort=False) Join columns with other DataFrame either on index or on a key column. Efciently Join multiple DataFrame objects by index at once by passing a list. Parameters other : DataFrame, Series with name eld set, or list of DataFrame 21.3. DataFrame 261 Join columns with other DataFrame either on index or on a key Merge DataFrame objects by performing a database-style join operation by Append columns of other to end of this frames columns and index, returning a new object.

pandas: powerful Python data analysis toolkit, Release 0.7.1

Index should be similar to one of the columns in this one. If a Series is passed, its name attribute must be set, and that will be used as the column name in the resulting joined DataFrame on : column name, tuple/list of column names, or array-like Column(s) to use for joining, otherwise join on index. If multiples columns given, the passed DataFrame must have a MultiIndex. Can pass an array as the join key if not already contained in the calling DataFrame. Like an Excel VLOOKUP operation how : {left, right, outer, inner} How to handle indexes of the two objects. Default: left for joining on index, None otherwise * left: use calling frames index * right: use input frames index * outer: form union of indexes * inner: use intersection of indexes lsufx : string Sufx to use from left frames overlapping columns rsufx : string Sufx to use from right frames overlapping columns sort : boolean, default False Order result DataFrame lexicographically by the join key. If False, preserves the index order of the calling (left) DataFrame Returns joined : DataFrame
Notes

on, lsufx, and rsufx options are not supported when passing a list of DataFrame objects pandas.DataFrame.merge DataFrame.merge(right, how=inner, on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=True, sufxes=(.x, .y), copy=True) Merge DataFrame objects by performing a database-style join operation by columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored. Otherwise if joining indexes on indexes or indexes on a column or columns, the index will be passed on. Parameters right : DataFrame how : {left, right, outer, inner}, default inner left: use only keys from left frame (SQL: left outer join) right: use only keys from right frame (SQL: right outer join) outer: use union of keys from both frames (SQL: full outer join) inner: use intersection of keys from both frames (SQL: inner join) on : label or list Field names to join on. Must be found in both DataFrames. left_on : label or list, or array-like Field names to join on in left DataFrame. Can be a vector or list of vectors of the length of the DataFrame to use a particular vector as the join key instead of columns 262 Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

right_on : label or list, or array-like Field names to join on in right DataFrame or vector/list of vectors per left_on docs left_index : boolean, default True Use the index from the left DataFrame as the join key(s). If it is a MultiIndex, the number of keys in the other DataFrame (either the index or a number of columns) must match the number of levels right_index : boolean, default True Use the index from the right DataFrame as the join key. Same caveats as left_index sort : boolean, default True Sort the join keys lexicographically in the result DataFrame sufxes : 2-length sequence (tuple, list, ...) Sufx to apply to overlapping column names in the left and right side, respectively copy : boolean, default True If False, do not copy data unnecessarily Returns merged : DataFrame
Examples >>> A lkey 0 foo 1 bar 2 baz 3 foo >>> B rkey 0 foo 1 bar 2 qux 3 bar

value 1 2 3 4

value 5 6 7 8

>>> merge(A, B, left_on=lkey, right_on=rkey, how=outer) lkey value.x rkey value.y 0 bar 2 bar 6 1 bar 2 bar 8 2 baz 3 NaN NaN 3 foo 1 foo 5 4 foo 4 foo 5 5 NaN NaN qux 7

pandas.DataFrame.append DataFrame.append(other, ignore_index=False, verify_integrity=True) Append columns of other to end of this frames columns and index, returning a new object. Columns not in this frame are added as new columns. Parameters other : DataFrame or list of Series/dict-like objects ignore_index : boolean, default False If True do not use the index labels. Useful for gluing together record arrays Returns appended : DataFrame

21.3. DataFrame

263

pandas: powerful Python data analysis toolkit, Release 0.7.1

Notes

If a list of dict is passed and the keys are all contained in the DataFrames index, the order of the columns in the resulting DataFrame will be unchanged

21.3.11 Time series-related

DataFrame.asfreq(freq[, method]) DataFrame.shift(periods, **kwds[, offset]) DataFrame.first_valid_index() DataFrame.last_valid_index() pandas.DataFrame.asfreq DataFrame.asfreq(freq, method=None) Convert all TimeSeries inside to specied frequency using DateOffset objects. Optionally provide ll method to pad/backll missing values. Parameters offset : DateOffset object, or string in {WEEKDAY, EOM} DateOffset object or subclass (e.g. monthEnd) method : {backll, bll, pad, fll, None} Method to use for lling holes in reindexed Series pad / fll: propagate last valid observation forward to next valid backll / bll: use NEXT valid observation to ll methdo Returns converted : DataFrame pandas.DataFrame.shift DataFrame.shift(periods, offset=None, **kwds) Shift the index of the DataFrame by desired number of periods with an optional time offset Parameters periods : int Number of periods to move, can be positive or negative offset : DateOffset, timedelta, or time rule string, optional Increment to use from datetools module or time rule (e.g. EOM) Returns shifted : DataFrame pandas.DataFrame.rst_valid_index DataFrame.first_valid_index() Return label for rst non-NA/null value pandas.DataFrame.last_valid_index DataFrame.last_valid_index() Return label for last non-NA/null value 264 Chapter 21. API Reference Convert all TimeSeries inside to specied frequency using DateOffset Shift the index of the DataFrame by desired number of periods with an Return label for rst non-NA/null value Return label for last non-NA/null value

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.3.12 Plotting
DataFrame.hist(**kwds[, grid]) DataFrame.plot(**kwds[, subplots, sharex, ...]) pandas.DataFrame.hist DataFrame.hist(grid=True, **kwds) Draw Histogram the DataFrames series using matplotlib / pylab. Parameters kwds : other plotting keyword arguments To be passed to hist function pandas.DataFrame.plot DataFrame.plot(subplots=False, sharex=True, sharey=False, use_index=True, gsize=None, grid=True, legend=True, rot=30, ax=None, kind=line, **kwds) Make line plot of DataFrames series with the index on the x-axis using matplotlib / pylab. Parameters subplots : boolean, default False Make separate subplots for each time series sharex : boolean, default True In case subplots=True, share x axis sharey : boolean, default False In case subplots=True, share y axis use_index : boolean, default True Use index as ticks for x axis kind : {line, bar} kwds : keywords Options to pass to Axis.plot
Notes

Draw Histogram the DataFrames series using matplotlib / pylab. Make line plot of DataFrames series with the index on the x-axis using

This method doesnt make much sense for cross-sections, and will error.

21.3. DataFrame

265

pandas: powerful Python data analysis toolkit, Release 0.7.1

21.3.13 Serialization / IO / Conversion

DataFrame.from_csv(path[, header, sep, ...]) DataFrame.from_records(data[, index, ...]) DataFrame.to_csv(path_or_buf[, sep, na_rep, ...]) DataFrame.to_excel(excel_writer[, ...]) DataFrame.to_dict() DataFrame.to_records([index]) DataFrame.to_sparse([ll_value, kind]) DataFrame.to_string([buf, columns, ...]) DataFrame.save(path) DataFrame.load(path) DataFrame.info([verbose, buf]) Read delimited le into DataFrame Convert structured or record ndarray to DataFrame Write DataFrame to a comma-separated values (csv) le Write DataFrame to a excel sheet Convert DataFrame to nested dictionary Convert DataFrame to record array. Index will be put in the Convert to SparseDataFrame Render a DataFrame to a console-friendly tabular output.

Concise summary of a DataFrame, used in __repr__ when very large.

pandas.DataFrame.from_csv classmethod DataFrame.from_csv(path, header=0, sep=, , index_col=0, parse_dates=True, encoding=None) Read delimited le into DataFrame Parameters path : string header : int, default 0 Row to use at header (skip prior rows) sep : string, default , Field delimiter index_col : int or sequence, default 0 Column to use for index. If a sequence is given, a MultiIndex is used. Different default from read_table parse_dates : boolean, default True Parse dates. Different default from read_table Returns y : DataFrame
Notes

Preferable to use read_table for most general purposes but from_csv makes for an easy roundtrip to and from le, especially with a DataFrame of time series data pandas.DataFrame.from_records classmethod DataFrame.from_records(data, index=None, names=None) Convert structured or record ndarray to DataFrame exclude=None, columns=None,

266

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Parameters data : ndarray (structured dtype), list of tuples, or DataFrame index : string, list of elds, array-like Field of array to use as the index, alternately a specic set of input labels to use exclude: sequence, default None : Columns or elds to exclude columns : sequence, default None Column names to use, replacing any found in passed data Returns df : DataFrame pandas.DataFrame.to_csv DataFrame.to_csv(path_or_buf, sep=, , na_rep=, cols=None, header=True, index=True, index_label=None, mode=w, nanRep=None, encoding=None) Write DataFrame to a comma-separated values (csv) le Parameters path_or_buf : string or le handle / StringIO File path na_rep : string, default Missing data representation cols : sequence, optional Columns to write header : boolean, default True Write out column names index : boolean, default True Write row names (index) index_label : string or sequence, default None Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex. mode : Python write mode, default w sep : character, default , Field delimiter for the output le. encoding : string, optional a string representing the encoding to use if the contents are non-ascii, for python versions prior to 3 pandas.DataFrame.to_excel DataFrame.to_excel(excel_writer, sheet_name=sheet1, na_rep=, cols=None, header=True, index=True, index_label=None) Write DataFrame to a excel sheet

21.3. DataFrame

267

pandas: powerful Python data analysis toolkit, Release 0.7.1

Parameters excel_writer : string or ExcelWriter object File path or existing ExcelWriter sheet_name : string, default sheet1 Name of sheet which will contain DataFrame na_rep : string, default Missing data repn cols : sequence, optional Columns to write header : boolean, default True Write out column names index : boolean, default True Write row names (index) index_label : string or sequence, default None Column label for index column(s) if desired. If None is given, and header and index are True, then the index names are used. A sequence should be given if the DataFrame uses MultiIndex.
Notes

If passing an existing ExcelWriter object, then the sheet will be added to the existing workbook. This can be used to save different DataFrames to one workbook >>> writer = ExcelWriter(output.xlsx) >>> df1.to_excel(writer,sheet1) >>> df2.to_excel(writer,sheet2) >>> writer.save() pandas.DataFrame.to_dict DataFrame.to_dict() Convert DataFrame to nested dictionary Returns result : dict like {column -> {index -> value}} pandas.DataFrame.to_records DataFrame.to_records(index=True) Convert DataFrame to record array. Index will be put in the index eld of the record array if requested Parameters index : boolean, default True Include index in resulting record array, stored in index eld Returns y : recarray pandas.DataFrame.to_sparse DataFrame.to_sparse(ll_value=None, kind=block) Convert to SparseDataFrame

268

Chapter 21. API Reference

pandas: powerful Python data analysis toolkit, Release 0.7.1

Parameters ll_value : oat, default NaN kind : {block, integer} Returns y : SparseDataFrame pandas.DataFrame.to_string DataFrame.to_string(buf=None, columns=None, col_space=None, colSpace=None, header=True, index=True, na_rep=NaN, formatters=None, oat_format=None, sparsify=True, nanRep=None, index_names=True, justify=None, force_unicode=False) Render a DataFrame to a console-friendly tabular output. Parameters frame : DataFrame object to render buf : StringIO-like, optional buffer to write to columns : sequence, optional the subset of columns to write; default None writes all columns col_space : int, optional the width of each columns header : bool, optional whether to print column labels, default True index : bool, optional whether to print index (row) labels, default True na_rep : string, optional string representation of NAN to use, default NaN formatters : list or dict of one-parameter functions, optional formatter functions to apply to columns elements by position or name, default None oat_format : one-parameter function, optional formatter function to apply to columns elements if they are oats default None sparsify : bool, optional Set to False for a DataFrame with a hierarchical index to print every multiindex key at each row, default True justify : {left, right}, default None Left or right-justify the column labels. If None uses the option from the conguration in pandas.core.common, left out of the box index_names : bool, optional Prints the names of the indexes, default True force_unicode : bool, default False Always return a unicode result Returns formatted : string (or unicode, depending on data and options) 21.3. DataFrame 269

pandas: powerful Python data analysis toolkit, Release 0.7.1

pandas.DataFrame.save DataFrame.save(path) pandas.DataFrame.load classmethod DataFrame.load(path) pandas.DataFrame.info DataFrame.info(verbose=True, buf=None) Concise summary of a DataFrame, used in __repr__ when very large. Parameters verbose : boolean, default True If False, dont print column count summary buf : writable buffer, defaults to sys.stdout

21.4 Panel
21.4.1 Computations / Descriptive Stats

270

Chapter 21. API Reference

PYTHON MODULE INDEX

p
pandas, 1

271

pandas: powerful Python data analysis toolkit, Release 0.7.1

272

Python Module Index

PYTHON MODULE INDEX

p
pandas, 1

273

pandas: powerful Python data analysis toolkit, Release 0.7.1

274

Python Module Index

INDEX

Symbols
__init__() (pandas.DataFrame method), 237 __init__() (pandas.Series method), 215 __iter__() (pandas.DataFrame method), 239 __iter__() (pandas.Series method), 217

A
add() (pandas.DataFrame method), 240 add() (pandas.Series method), 218 add_prex() (pandas.DataFrame method), 253 add_sufx() (pandas.DataFrame method), 253 align() (pandas.DataFrame method), 254 align() (pandas.Series method), 226 append() (pandas.DataFrame method), 263 append() (pandas.Series method), 232 apply() (pandas.DataFrame method), 245 apply() (pandas.Series method), 219 applymap() (pandas.DataFrame method), 246 argsort() (pandas.Series method), 230 as_matrix() (pandas.DataFrame method), 236 asfreq() (pandas.DataFrame method), 264 asfreq() (pandas.Series method), 232 asof() (pandas.Series method), 232 astype (pandas.Series attribute), 216 astype() (pandas.DataFrame method), 238 autocorr() (pandas.Series method), 221 axes (pandas.DataFrame attribute), 237

concat() (pandas.tools.merge static method), 201 copy() (pandas.DataFrame method), 238 copy() (pandas.Series method), 216 corr() (pandas.DataFrame method), 248 corr() (pandas.Series method), 222 corrwith() (pandas.DataFrame method), 248 count() (pandas.DataFrame method), 248 count() (pandas.Series method), 222 cumprod() (pandas.DataFrame method), 248 cumprod() (pandas.Series method), 222 cumsum() (pandas.DataFrame method), 249 cumsum() (pandas.Series method), 223

D
delevel() (pandas.DataFrame method), 258 describe() (pandas.DataFrame method), 249 describe() (pandas.Series method), 223 diff() (pandas.DataFrame method), 249 diff() (pandas.Series method), 223 div() (pandas.DataFrame method), 241 div() (pandas.Series method), 218 drop() (pandas.DataFrame method), 254 drop() (pandas.Series method), 227 dropna() (pandas.DataFrame method), 257 dropna() (pandas.Series method), 229 dtype (pandas.Series attribute), 215 dtypes (pandas.DataFrame attribute), 237

C
clip() (pandas.DataFrame method), 247 clip() (pandas.Series method), 221 clip_lower() (pandas.DataFrame method), 247 clip_lower() (pandas.Series method), 221 clip_upper() (pandas.DataFrame method), 247 clip_upper() (pandas.Series method), 222 combine() (pandas.DataFrame method), 244 combine() (pandas.Series method), 219 combine_rst() (pandas.DataFrame method), 244 combine_rst() (pandas.Series method), 219 combineAdd() (pandas.DataFrame method), 244 combineMult() (pandas.DataFrame method), 244

E
ewma() (pandas.stats.moments static method), 211 ewmcorr() (pandas.stats.moments static method), 213 ewmcov() (pandas.stats.moments static method), 214 ewmstd() (pandas.stats.moments static method), 212 ewmvar() (pandas.stats.moments static method), 212

F
llna() (pandas.DataFrame method), 257 llna() (pandas.Series method), 229 lter() (pandas.DataFrame method), 254 rst_valid_index() (pandas.DataFrame method), 264 rst_valid_index() (pandas.Series method), 233

275

pandas: powerful Python data analysis toolkit, Release 0.7.1

from_csv() (pandas.DataFrame class method), 266 from_csv() (pandas.Series class method), 234 from_records() (pandas.DataFrame class method), 266

notnull() (pandas.Series method), 215

O
order() (pandas.Series method), 230

G
get() (pandas.io.pytables.HDFStore method), 206 get() (pandas.Series method), 217 get_dtype_counts() (pandas.DataFrame method), 237 groupby() (pandas.DataFrame method), 246 groupby() (pandas.Series method), 220

P
pandas (module), 1 parse() (pandas.io.parsers.ExcelFile method), 205 pivot() (pandas.DataFrame method), 259 pivot_table() (pandas.tools.pivot static method), 199 plot() (pandas.DataFrame method), 265 plot() (pandas.Series method), 234 pop() (pandas.DataFrame method), 239 prod() (pandas.DataFrame method), 251 prod() (pandas.Series method), 224 put() (pandas.io.pytables.HDFStore method), 206

H
head() (pandas.DataFrame method), 257 hist() (pandas.DataFrame method), 265 hist() (pandas.Series method), 233

I
info() (pandas.DataFrame method), 270 insert() (pandas.DataFrame method), 239 interpolate() (pandas.Series method), 230 isnull() (pandas.Series method), 215 iteritems() (pandas.DataFrame method), 239 iteritems() (pandas.Series method), 217 ix (pandas.DataFrame attribute), 239 ix (pandas.Series attribute), 217

Q
quantile() (pandas.DataFrame method), 251 quantile() (pandas.Series method), 225

R
radd() (pandas.DataFrame method), 242 rdiv() (pandas.DataFrame method), 242 read_csv() (pandas.io.parsers static method), 204 read_table() (pandas.io.parsers static method), 203 reindex() (pandas.DataFrame method), 255 reindex() (pandas.Series method), 227 reindex_like() (pandas.DataFrame method), 255 reindex_like() (pandas.Series method), 227 rename() (pandas.DataFrame method), 256 rename() (pandas.Series method), 228 rmul() (pandas.DataFrame method), 243 rolling_apply() (pandas.stats.moments static method), 210 rolling_corr() (pandas.stats.moments static method), 209 rolling_count() (pandas.stats.moments static method), 207 rolling_cov() (pandas.stats.moments static method), 209 rolling_kurt() (pandas.stats.moments static method), 210 rolling_mean() (pandas.stats.moments static method), 208 rolling_median() (pandas.stats.moments static method), 208 rolling_quantile() (pandas.stats.moments static method), 211 rolling_skew() (pandas.stats.moments static method), 210 rolling_std() (pandas.stats.moments static method), 209 rolling_sum() (pandas.stats.moments static method), 207 rolling_var() (pandas.stats.moments static method), 208 rsub() (pandas.DataFrame method), 243

J
join() (pandas.DataFrame method), 261

L
last_valid_index() (pandas.DataFrame method), 264 last_valid_index() (pandas.Series method), 233 load() (pandas.core.common static method), 202 load() (pandas.DataFrame class method), 270 load() (pandas.Series class method), 235

M
mad() (pandas.DataFrame method), 249 map() (pandas.Series method), 219 max() (pandas.DataFrame method), 250 max() (pandas.Series method), 223 mean() (pandas.DataFrame method), 250 mean() (pandas.Series method), 223 median() (pandas.DataFrame method), 250 median() (pandas.Series method), 224 merge() (pandas.DataFrame method), 262 merge() (pandas.tools.merge static method), 200 min() (pandas.DataFrame method), 251 min() (pandas.Series method), 224 mul() (pandas.DataFrame method), 241 mul() (pandas.Series method), 218

N
ndim (pandas.DataFrame attribute), 237 276

S
save() (pandas.core.common static method), 203 Index

pandas: powerful Python data analysis toolkit, Release 0.7.1

save() (pandas.DataFrame method), 270 save() (pandas.Series method), 235 select() (pandas.DataFrame method), 256 select() (pandas.Series method), 228 shape (pandas.DataFrame attribute), 237 shift() (pandas.DataFrame method), 264 shift() (pandas.Series method), 233 skew() (pandas.DataFrame method), 251 skew() (pandas.Series method), 225 sort() (pandas.Series method), 231 sort_index() (pandas.DataFrame method), 258 sort_index() (pandas.Series method), 231 sortlevel() (pandas.DataFrame method), 259 sortlevel() (pandas.Series method), 231 stack() (pandas.DataFrame method), 260 std() (pandas.DataFrame method), 252 std() (pandas.Series method), 225 sub() (pandas.DataFrame method), 242 sub() (pandas.Series method), 218 sum() (pandas.DataFrame method), 252 sum() (pandas.Series method), 225 swaplevel() (pandas.DataFrame method), 260

X
xs() (pandas.DataFrame method), 239

T
T (pandas.DataFrame attribute), 261 tail() (pandas.DataFrame method), 257 take() (pandas.DataFrame method), 256 take() (pandas.Series method), 228 to_csv() (pandas.DataFrame method), 267 to_csv() (pandas.Series method), 235 to_dict() (pandas.DataFrame method), 268 to_dict() (pandas.Series method), 236 to_excel() (pandas.DataFrame method), 267 to_records() (pandas.DataFrame method), 268 to_sparse() (pandas.DataFrame method), 268 to_sparse() (pandas.Series method), 236 to_string() (pandas.DataFrame method), 269 transpose() (pandas.DataFrame method), 261 truncate() (pandas.DataFrame method), 256 truncate() (pandas.Series method), 229

U
unstack() (pandas.DataFrame method), 260 unstack() (pandas.Series method), 231

V
value_counts() (pandas.Series method), 226 values (pandas.DataFrame attribute), 237 values (pandas.Series attribute), 214 var() (pandas.DataFrame method), 253 var() (pandas.Series method), 226

W
weekday (pandas.Series attribute), 233 Index 277