An Introduction To Statistics With Python With

Applications In The Life Sciences 2nd 2nd

Edition Thomas Haslwanter download

https://ebookbell.com/product/an-introduction-to-statistics-with-
python-with-applications-in-the-life-sciences-2nd-2nd-edition-
thomas-haslwanter-47278200

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

An Introduction To Statistics With Python With Applications In The

Life Sciences Thomas Haslwanter

https://ebookbell.com/product/an-introduction-to-statistics-with-
python-with-applications-in-the-life-sciences-thomas-
haslwanter-49052704

An Introduction To Statistical Learning With Applications In Python

1st Gareth James

https://ebookbell.com/product/an-introduction-to-statistical-learning-
with-applications-in-python-1st-gareth-james-50703060

An Introduction To Statistics With Python Thomas Haslwanter

https://ebookbell.com/product/an-introduction-to-statistics-with-
python-thomas-haslwanter-42679398

An Introduction To Algebraic Statistics With Tensors Cristiano Bocci

https://ebookbell.com/product/an-introduction-to-algebraic-statistics-
with-tensors-cristiano-bocci-10519584
An Introduction To Algebraic Statistics With Tensors Cristiano Bocci

https://ebookbell.com/product/an-introduction-to-algebraic-statistics-
with-tensors-cristiano-bocci-10519594

An Introduction To Econometric Theory Measuretheoretic Probability And

Statistics With Applications To Economics A Ronald Gallant

https://ebookbell.com/product/an-introduction-to-econometric-theory-
measuretheoretic-probability-and-statistics-with-applications-to-
economics-a-ronald-gallant-50731408

Biostatistics With R An Introduction To Statistics Through Biological

Data 1st Edition Babak Shahbaba Auth

https://ebookbell.com/product/biostatistics-with-r-an-introduction-to-
statistics-through-biological-data-1st-edition-babak-shahbaba-
auth-2445754

Simulation And Statistics With Excel An Introduction To Business

Students Ibarra Luis Fernando

https://ebookbell.com/product/simulation-and-statistics-with-excel-an-
introduction-to-business-students-ibarra-luis-fernando-56337800

Elements Of Probability And Statistics An Introduction To Probability

With De Finettis Approach And To Bayesian Statistics 1st Edition
Francesca Biagini

https://ebookbell.com/product/elements-of-probability-and-statistics-
an-introduction-to-probability-with-de-finettis-approach-and-to-
bayesian-statistics-1st-edition-francesca-biagini-5200136
Statistics and Computing

Thomas Haslwanter

An Introduction
to Statistics
with Python
With Applications in the Life Sciences
Second Edition
Statistics and Computing

Series Editor
Wolfgang Karl Härdle, Humboldt-Universität zu Berlin, Berlin, Germany
Statistics and Computing (SC) includes monographs and advanced texts on statistical
computing and statistical packages.

More information about this series at https://link.springer.com/bookseries/3022

Thomas Haslwanter

An Introduction to Statistics
with Python
With Applications in the Life Sciences
Second Edition
Thomas Haslwanter
School of Medical Engineering and Applied
Social Sciences
University of Applied Sciences Upper Austria
Linz, Austria

ISSN 1431-8784 ISSN 2197-1706 (electronic)

Statistics and Computing
ISBN 978-3-030-97370-4 ISBN 978-3-030-97371-1 (eBook)
https://doi.org/10.1007/978-3-030-97371-1

1st edition: © Springer International Publishing Switzerland 2016

2nd edition: © Springer Nature Switzerland AG 2022
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
To my two-, three-, and four-legged
household companions: Jean, Felix, and his
sister Jessica: Thank you so much for all the
support you have provided over the years!
Preface

Preface to the First Edition

In the data analysis for my own research work, I was often slowed down by two
things: (1) I did not know enough statistics, and (2) the books available would provide
a theoretical background, but no real practical help. The book you are holding in your
hands (or on your tablet or laptop) is intended to be the book that will solve this very
problem. It is designed to provide enough basic understanding so that you know what
you are doing, and it should equip you with the tools you need. I believe that the
Python solutions provided in this book for the most basic statistical problems address
at least 90% of the problems that most physicists, biologists, and medical doctors
encounter in their work. So if you are the typical graduate student working on your
degree, or a medical researcher analyzing your latest experiments, chances are that
you will find the tools you require here—explanation and source-code included.
This is the reason I have focused on statistical basics and hypothesis tests in
this book, and refer only briefly to other statistical approaches. I am well aware
that most of the tests presented in this book can also be carried out using statistical
modeling. But in many cases, this is not the methodology used in many life science
journals. Advanced statistical analysis goes beyond the scope of this book, and—to
be frank—exceeds my own knowledge of statistics.
My motivation for providing the solutions in Python is based on two considera-
tions. One is that I would like them to be available to everyone. While commercial
solutions like Matlab, SPSS, Minitab etc. offer powerful tools, most can only use
them legally in an academic setting. In contrast, Python is completely free (as in free
beer is often heard in the Python community). The second reason is that Python is the
most beautiful coding language that I have yet encountered; and around 2010 Python
and its documentation matured to the point where one can use it without being an
serious coder. Together, this book, Python, and the tools that the Python ecosystem
offers today provide a beautiful, free package that covers all the statistics that most
researchers will need in their lifetime.

vii
viii Preface

Preface to the Second Edition

Since the publication of the first edition, Python has continuously gained popularity
and become firmly established as one of the foremost programming languages for
statistical data analysis. All the core packages have matured. And thanks to the
stunning development of Jupyter as an interactive programming environment, Python
has become even more accessible for people with little programming background.
To reflect these developments, and to incorporate the suggestions I have received for
improving the presentation of the material, Springer has given me the opportunity to
bring out a new edition of Introduction to Statistics with Python.
Compared to the first edition, the following changes have been made:
• The package pandas and its DataFrames have become an integral part of
scientific Python, as has the Jupyter framework for interactive data environ-
ments. Correspondingly, a bigger amount of space has been dedicated to their
introduction.
• A new package, pingouin, is promising a simplified and more powerful inter-
face for many common statistics function. This package is introduced, and many
application examples have been added.
• The visualization of data has been expanded, including the preparation of
publication-ready graphics.
• The design of experiments and power analyses are discussed in more detail.
• A new section has been added on the confidence intervals of frequently used
statistical parameters.
• A new chapter has been added on finding patterns in data, including an introduction
to the correlation coefficient, cross- and autocorrelation. For an application of
these concepts, a short introduction is given to time series analysis.
As for the first edition, all examples and solutions from this book are again avail-
able online. This includes code samples and example programs, Jupyter Notebooks
with additional or extended information, as well as the data and Python code used to
generate most of the figures. They can be downloaded from https://github.com/tho
mas-haslwanter/statsintro-python-2e.
I hope this book will help you with the statistical analysis of your data, and
convey some of the often really simple ideas behind the sometimes awkwardly named
statistical analysis procedures.

For Whom This Book Is

This book assumes that

• you have some basic programming experience: If you have done no programming
previously, you may want to start out with Python using some of the great links
provided in the text. Starting programming and starting statistics may be a bit
Preface ix

much all at once. However, solutions provided to the exercises at the end of most
chapters should help you to get up to speed with Python.
• you are not a statistics expert: If you have advanced statistics experience, the
online help in Python and the Python packages may be sufficient to allow you to
do most of your data analysis right away. This book may still help you to get started
with Python. However, the book concentrates on the basic ideas of statistics and
on hypothesis tests, and only the last part introduces linear regression modeling
and Bayesian statistics.
This book is designed to give you all (or at least most of) the tools that you
will need for statistical data analysis. I attempt to provide the background you need
to understand what you are doing. I do not prove any theorems, and do not apply
mathematics unless necessary. For all tests, a working Python program is provided.
In principle, you just have to define your problem, select the corresponding program,
and adapt it to your needs. This should allow you to get going quickly, even if you
have little Python experience. This is also the reason why I have not provided the
software as one single Python package; I expect that you will have to tailor each
program to your specific setup (data format, etc.).
This book is organized into three parts
Part I gives an introduction to Python: how to set it up, simple programs to get
started, and tips on how to avoid some common mistakes. It also shows how to
read data from different sources into Python, and how to visualize statistical data.
Part II provides an introduction to statistical analysis; on how to design a study,
power analysis, and how best to analyze data; probability distributions; and an
overview of the most important hypothesis tests. Even though modern statistics
is firmly based in statistical modeling, hypothesis tests still seem to dominate the
life sciences. For each test, a Python program is provided that shows how the test
can be implemented.
Part III provides an introduction to correlation and regression analysis, time
series analysis, and statistical modeling, and a look at advanced statistical analysis
procedures. I have also included tests on discrete data in this section, such as
logistic regression, as they utilize “generalized linear models” which I regard
as advanced. This part ends with a presentation of the basic ideas of Bayesian
statistics.
To achieve all those goals as quickly as possible, the Appendix A of the book
provides hints on how to efficiently develop correct and working code. This should
get you to the point where you can get things done quickly.
x Preface

Acknowledgments

Python is built on the contributions from the user community, and some of the sections
in this book are based on some of the excellent information available on the web.
(Permission has been granted by the authors to reprint their contributions here.)
I especially want to thank the following people:
• Christiane Takacs helped me enormously by polishing the introductory statistics
sections.
• Connor Johnson wrote a very nice blog explaining the results of the statsmodels
OLS command, which provided the basis for the section on Statistical Models.
• Cam Davidson Pilon wrote the excellent open-source e-book Probabilistic-
Programming-and-Bayesian-Methods-for-Hackers. From there, I took the
example of the Challenger disaster to demonstrate Bayesian statistics.
• Fabian Pedregosa’s blog on ordinal logistic regression allowed me to include this
topic, which otherwise would be admittedly beyond my own skills.
I also want to thank Springer Publishing for the chance to bring out the second
edition of this book, and to base the three introductory chapters (Python, Data Import,
and Data Display) to a significant part on the corresponding chapters of my book
Hands-on Signal Analysis with Python.
If you have a suggestion or correction, please send an email to my work address
[email protected]. If I make a change based on your feedback, I will add
you to the list of contributors unless advised otherwise. If you include at least part
of the sentence the error appears in, that makes it easy for me to search. Page and
section numbers are fine, too, but not as easy to work with. Thanks!

Linz, Austria Thomas Haslwanter

August 2022
Contents

Part I Python and Statistics

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 Why Statistics? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Accompanying Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Elements of Scientific Python Programming . . . . . . . . . . . . . . . . . . 15
2.3 Interactive Programming—IPython/Jupyter . . . . . . . . . . . . . . . . . . 28
2.4 Statistics Packages for Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.5 Programming Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3 Data Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.1 Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2 Excel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3 Matlab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.4 Binary Data: NPZ Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.5 Other Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4 Data Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.1 Introductory Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Plotting in Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3 Saving a Figure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4 Preparing Figures for Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.5 Display of Statistical Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

xi
xii Contents

Part II Distributions and Hypothesis Tests

5 Basic Statistical Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.1 Populations and Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.2 Data Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.3 Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.4 Degrees of Freedom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.5 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
6 Distributions of One Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.1 Characterizing a Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.2 Discrete Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.3 Normal Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6.4 Continuous Distributions Derived from the Normal
Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.5 Other Continuous Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.6 Confidence Intervals of Selected Statistical Parameters . . . . . . . . 135
6.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7 Hypothesis Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.1 Typical Analysis Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.2 Hypothesis Tests and Power Analyses . . . . . . . . . . . . . . . . . . . . . . . 144
7.3 Sensitivity and Specificity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.4 Receiver-Operating-Characteristic (ROC) Curve . . . . . . . . . . . . . . 155
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
8 Tests of Means of Numerical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.1 Distribution of a Sample Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
8.2 Comparison of Two Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
8.3 Comparison of Multiple Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8.4 Summary: Selecting the Right Test for Comparing Groups . . . . . 176
8.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
9 Tests on Categorical Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
9.1 Proportions and Confidence Intervals . . . . . . . . . . . . . . . . . . . . . . . . 182
9.2 Tests Using Frequency Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
9.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10 Analysis of Survival Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.1 Survival Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.2 Survival Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10.3 Comparing Survival Curves in Two Groups . . . . . . . . . . . . . . . . . . 202

Part III Statistical Modeling

11 Finding Patterns in Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
11.1 Cross Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
11.2 Correlation Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
11.3 Coefficient of Determination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Contents xiii

11.4 Scatterplot Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

11.5 Correlation Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
11.6 Autocorrelation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
11.7 Time-Series Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
12 Linear Regression Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
12.1 Simple Fits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
12.2 Design Matrix and Formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
12.3 Linear Regression Analysis with Python . . . . . . . . . . . . . . . . . . . . . 237
12.4 Model Results of Linear Regression Models . . . . . . . . . . . . . . . . . . 241
12.5 Assumptions and Interpretations of Linear Regression . . . . . . . . . 257
12.6 Bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
12.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
13 Generalized Linear Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
13.1 Comparing and Modeling Ranked Data . . . . . . . . . . . . . . . . . . . . . . 265
13.2 Elements of GLMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
13.3 GLM 1: Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
13.4 GLM 2: Ordinal Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . 270
13.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
14 Bayesian Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
14.1 Bayesian Versus Frequentist Interpretation . . . . . . . . . . . . . . . . . . . 275
14.2 The Bayesian Approach in the Age of Computers . . . . . . . . . . . . . 277
14.3 Example: Markov-Chain-Monte-Carlo Simulation . . . . . . . . . . . . 278
14.4 Summing Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280

Appendix A: Useful Programming Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Appendix B: Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Appendix C: Equations for Confidence Intervals . . . . . . . . . . . . . . . . . . . . . 321
Appendix D: Web Ressources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
Abbreviations

ACF Auto-Correlation Function

AIC Akaike Information Criterion
ANOVA ANalysis Of VAriance
ARIMA Autoregressive Integrated Moving Average model
ARMA Autoregressive Moving Average model
BIC Bayesian Information Criterion
CDF Cumulative Distribution Function
CI Confidence Interval
CQ Code Quantlet
DF/DOF Degrees of Freedom
EOL End Of Line
GLM Generalized Linear Models
GUI Graphical User Interface
HDF5 Hierarchical Data Format 5
HSD Honest Significant Difference
HTML HyperText Markup Language
IDE Integrated Development Environment
IQR Inter-Quartile-Range
ISF Inverse Survival Function
ISO International Organization for Standardization
ISP2e https://github.com/thomas-haslwanter/statsintro-python-2e/tree/mas
ter/src/code_quantlets
JPEG Joint Photographic Experts Group
KDE Kernel Density Estimation
MCMC Markov Chain–Monte Carlo
NAN Not-A-Number
NPV Negative Predictive Value
OLS Ordinary Least Squares
PACF Partial Auto-Correlation Function
PDF Probability Density Function
PMF Probability Mass Function

xv
xvi Abbreviations

PNG Portable Network Graphics

PPF Percentile Point Function
PPV Positive Predictive Value
QQ-Plot Quantile-Quantile Plot
ROC Receiver Operating Characteristic
RVS Random Variate Sample
SARIMAX Seasonal Autoregressive Integrated Moving Average Exogenous
model
SD Standard Deviation
SE/SEM Standard Error (of the Mean)
SF Survival Function
SQL Structured Query Language
SS Sum of Squares
SVG Scalable Vector Graphics
TSA Time Series Analysis
Tukey HSD Tukey Honest Significant Difference test
 Part I
Python and Statistics

The first part of the book presents an introduction to statistics based on Python. It
is impossible to cover the whole language in thirty or forty pages, so if you are
a beginner, please see one of the excellent Python introductions available on the
Internet for details. Links are given below. This part is a kick-start for Python; it
shows how to install Python under Windows, Linux, or MacOS, and walks step-by-
step through documented programming examples. Tips are included to help avoid
some of the problems frequently encountered while learning Python.
Because most of the data for statistical analysis are commonly obtained from text
files, Excel files, or data preprocessed by Matlab, the third chapter presents simple
ways to import these types of data into Python.
The last chapter of Part I illustrates various ways of visualizing data in Python.
Since the flexibility of Python for interactive data analysis has led to a certain com-
plexity that can frustrate new Python programmers, code samples for various types
of interactive plots should help future Pythonistas to avoid these problems.
Chapter 1
Introduction

Statistics is the explanation of variance in the light of what

remains unexplained (anon).

1.1 Why Statistics?

Every day, we are confronted with situations with uncertain outcomes, and must
make decisions based on incomplete data: “Should I run for the bus? Which stock
should I buy? Which man should I marry? Should I take this medication? Should
I have my children vaccinated?” Some of these questions are beyond the realm
of statistics (“Which person should I marry?”), because they involve too many
unknown variables. But in many situations, statistics can help extract maximum
knowledge from information given, and clearly spell out what we know and what
we don’t know. For example, it can turn a vague statement like “This medication
may cause nausea,” or “You could die if you don’t take this medication” into a spe-
cific statement like “Three patients in one thousand experience nausea when taking
this medication,” or “If you don’t take this medication, there is a 95% chance that
you will die.”
Without statistics, the interpretation of data can quickly become massively flawed.
Take, for example, the estimated number of German tanks produced during World
War II, also known as the “German Tank Problem.” The estimate of the number of
German tanks produced per month from standard intelligence data was 1550; how-
ever, the statistical estimate based on the number of tanks observed was 327, which
was very close to the actual production number of 342 (http://en.wikipedia.org/wiki/
German_tank_problem).
Similarly, using the wrong tests can also lead to erroneous results.

c Springer Nature Switzerland AG 2022 3

T. Haslwanter, An Introduction to Statistics with Python, Statistics and Computing,
https://doi.org/10.1007/978-3-030-97371-1_1
4 1 Introduction

In general, statistics will help to

• Clarify the question.
• Identify the variable and the measure of that variable that will answer that ques-
tion.
• Determine the required sample size.
• Describe variation.
• Make quantitative statements about estimated parameters.
• Make predictions based on your data.
Reading the book: Statistics was originally invented—like so many other
things—by the famous mathematician C. F. Gauss, who said about his own work,
“Ich habe fleissig sein müssen; wer es gleichfalls ist, wird eben so weit kommen.”
(“I had to work hard; if you work hard as well, you, too, will be successful.”) Just as
reading a book about playing the piano won’t turn you into a great pianist, simply
reading this book will not teach you statistical data analysis. If you don’t have your
own data to analyze, you need to do the exercises included. Should you become frus-
trated or stuck, you can always check the sample solutions provided in Appendix.
Exercises: Solutions to the exercises provided at the end of most chapters can be
found in Appendix. In my experience, very few people work through large numbers
of examples on their own, so I have not included additional exercises in this book.
If the information here is not sufficient, additional material can be found in other
statistical textbooks and on the web.
Books: There are a number of good books on statistics. My favorite is (Altman
1999): it does not dwell on computers and modeling, but gives an extremely useful
introduction to the field, especially for life sciences and medical applications. Many
formulations and examples in this manuscript have been taken from that book. A
more modern book, which is more voluminous and, in my opinion, a bit harder to
read, is (Riffenburgh 2012). Kaplan (2009) provides a simple introduction to modern
regression modeling. If you know your basic statistics, a very good introduction
to Generalized Linear Models can be found in Dobson and Barnett (2018), which
provides a sound, advanced treatment of statistical modeling.
WWW: On the web, you will find very extensive information on statistics in
English at

• http://www.statsref.com/
• http://www.vassarstats.net/
• http://www.biostathandbook.com/
• http://onlinestatbook.com/2/index.html
• http://www.itl.nist.gov/div898/handbook/index.htm.

A good German web page on statistics and regulatory issues is

http://www.reiter1.com/.
I hope to convince you that Python provides clear and flexible tools for most of
the statistical problems that you will encounter, and that you will enjoy using it.
1.3 Accompanying Material 5

1.2 Conventions

The following conventions will be used.

• Text that is to be typed in on the computer is written in Courier font, e.g.,

plot(x,y).
• Optional text in command-line entries is expressed with <...>,
e.g., <InstallationDir>.
• Names referring to computer programs and applications are written in italics, e.g.,
Jupyter.
• Italics will also be used when introducing new terms or expressions for the first
time.
• Really important points that should definitely be remembered are indicated as
follows:

This will be important!

1.3 Accompanying Material

All the examples and solutions shown in this book are available online. This
includes code samples and example programs, Jupyter Notebooks with additional
or extended information, as well as the data and Python code used to generate many
of the figures. They can be downloaded from GitHub
https://github.com/thomas-haslwanter/statsintro-python-2e
which is organized in folders:

data Raw data required for running the programs.

resources Images used by this repository.
ipynbs Jupyter Notebooks with examples, and with additional or
extended information that goes beyond the content pre-
sented in the book.
src/exercise_solutions Solutions to the exercises that are presented at the end of
most chapters.
src/listings Programs that are explicitly listed in this book.
src/figures Code used to generate the Python figures in the text.
Unless noted otherwise, the source code for
Python figures is available in the source file
F<chapter-#>_<figure-#>_xxx.py. For exam-
ple, Fig. 8.4 can be generated with the Python code in
F8_4_anovaOneway.py
src/code_quantlets Additional code samples.
6 1 Introduction

In this book, references to the src/code_quantlets-directory in this reposi-

tory will be abbreviated with <ISP2e>. Make sure also to look at the file
Errata.pdf in the top folder of that archive, which will be kept up to date with
corrections to any mistakes that are discovered after publication of the book.
Packages on GitHub are called repositories, and can easily be copied to
your computer: when git is installed on your computer, simply type
git clone <RepositoryName> (here, the repository name of
statsintro-python-2e given above) in a command terminal, and the whole
repository—code as well as data—will be “cloned” to your system. Alternatively,
you can download a ZIP-archive from there to your local system.
Chapter 2
Python

Python is free, consistently and completely object oriented, and has a large num-
ber of (free) scientific toolboxes (e.g., http://www.scipy.org/). It is used by Google,
NASA, and many others. Information on Python can be found under http://www.
python.org/. If you want to use Python for scientific applications, currently the best
way to get started is with a Python distribution, either WinPython, or Anaconda
from Continuum Analytics. These distributions contain the complete scientific and
engineering development software for numerical computations, data analysis, and
data visualization based on Python. They also come with Qt graphical user inter-
faces, and the interactive scientific/development environment Spyder. If you already
have experience with Matlab, the article NumPy for Matlab Users (https://numpy.
org/devdocs/user/numpy-for-matlab-users.html) provides an overview of the simi-
larities and differences between the two languages.
Python is a very-high-level dynamic object-oriented programming language
(Fig. 2.1). It is designed to be easy to program and easy to read. It was started
in 1980, and has since gained immense popularity in a broad range of fields from
web development, system administration, and in science and engineering. Python is
open source, and has become one of the most successful programming languages.
There are three reasons why I have switched from other programming languages to
Python:
1. It is the most elegant programming language that I know.
2. It is free.
3. It is powerful.

© Springer Nature Switzerland AG 2022 7

T. Haslwanter, An Introduction to Statistics with Python, Statistics and Computing,
https://doi.org/10.1007/978-3-030-97371-1_2
8 2 Python

Fig. 2.1 The Python Logo

2.1 Getting Started

2.1.1 Distributions and Packages

The Python core distribution contains only the essential features of a general pro-
gramming language. Python itself is an interpretative programming language, with
no optimization for working with vectors or matrices, or for producing plots. Pack-
ages which extend the abilities of Python must be loaded explicitly. The most impor-
tant packages for scientific applications are numpy, which makes working with vec-
tors and matrices fast and efficient, and matplotlib, which is the most common
package used for producing graphical output. SciPy contains important scientific
algorithms. And pandas has become widely adopted for statistical data analysis. It
provides DataFrames which are labeled, 2D data structures, making work with data
more flexible and intuitive. Python can be used with different front-ends (Fig. 2.2):
from the command line or terminal; interactively, using Jupyter and IPython; and
from “Integrated Development Environments” (IDEs).
IPython provides the tools for interactive data analysis, and Jupyter provides the
different frontends for IPython. IPython lets you quickly display graphs and change
directories, explore the workspace, provides a command history, etc.

Fig. 2.2 Python can be used from different front-ends: from the command line (left), interactively
(center), and from an IDE (right)
2.1 Getting Started 9

Fig. 2.3 The structure of the most important Python packages for statistics

Figure 2.3 shows the modular structure of the most important Python packages
that are used in this book.
To facilitate the use of Python, so-called Python distributions collect matching
versions of the most important packages, and I strongly(!) recommend using one
of these distributions when getting started. Otherwise, one can easily become over-
whelmed by the huge number of Python packages available. My favorite Python
distributions are
• WinPython recommended for Windows users. At the time of writing, the latest
version was 3.10.5
https://winpython.github.io/
• Anaconda by Continuum, for Windows, Mac, and Linux. The latest Anaconda
version at the time of writing was 2020.11, with Python 3.9.
https://www.anaconda.com/products/individual

I am presently using WinPython, which is free and customizable. Anaconda also

runs under Mac OS and Linux, and is free for educational purposes.
The Python code samples in this book expect a Python version ≥ 3.6.
The programs included in this book have been tested under Windows and Linux.
Under Windows, the following package versions have been used:

• python 3.8.9 ... basic Python installation.

• numpy 1.20.2+mkl ... for working with vectors and arrays.
• scipy 1.6.2 ... all the essential scientific algorithms.
• matplotlib 3.4.1 ... the de facto standard module for plotting and visualization.
• pandas 1.2.4 ... adds DataFrames (imagine powerful spreadsheets) to Python.
• seaborn 0.11.1 ... statistical visualization package and visualization.
• pingouin 0.4.0 ... easy-to-use statistics functions to perform the most widely used
statistical tests.
• statsmodels 0.12.2 ... for advanced statistical modeling.
• ipython 7.22.0 ... for interactive work.
• jupyter 1.0.0 ... for interactive work environments, e.g., the JupyterLab, Jupyter
Notebook, or the Qt console.
10 2 Python

All of these packages come with the WinPython and Anaconda distributions.
Additional packages, which may be required by individual applications, can easily
be installed using pip or conda.
a) PyPI—The Python Package Index
The Python Package Index (PyPI) (https://pypi.org/) is a repository of software for
the Python programming language and contains more than 390’000 projects!
Packages from PyPI can be installed easily from the Windows command shell
(cmd) or the Linux terminal with
pip install <package>

To update a package, use

pip install <package> -U

To get a list of all the Python packages installed on your computer, type
pip list

And to show information about a particular package type

pip show <package>

Anaconda uses conda, a more powerful installation manager. But pip also works
with Anaconda.

2.1.2 Installation of Python

While Python and the required packages can be installed manually, it is typically
much easier to start out with a complete Python distribution.
a) Under Windows
Neither WinPython nor Anaconda require administrator rights for installation.
WinPython In the following, I assume that <WinPythonDir> is the installation
directory for WinPython.
Tip: Do NOT install WinPython into the Windows program directory (typically
C:\Program Files or C:\Program Files (x86)), because this can lead
to permission problems during the execution of WinPython.
• Download WinPython from https://winpython.github.io/.
• Run the downloaded .exe-file, and install WinPython into the
<WinPythonDir> of your choice. (On my own system, I place all programs
that do not modify the Windows Registry, such as WinPython, vim, and ffmpeg,
into a folder C:\Programs.)
• After the installation, make a change to your Windows Environment, by typing
Win -> env -> Edit environment variables for your
account (Note that this is different from the system environment!):
2.1 Getting Started 11

– Add the directories

<WinPythonDir>\python-3.8.9.amd64;
<WinPythonDir>\python-3.8.9.amd64\Scripts\;
(or whatever your Python version number is) to your PATH. (This makes
Python and IPython accessible from the standard Windows command line,
which can be reached quickly by typing Win+cmd.)
– Remove the default
%USERPROFILE%\AppData\Local\Microsoft\WindowsApps from
the PATH (since it contains a misleading python.exe-link).
– If you do have administrator rights, you should activate
<WinPythonDir>\WinPython Control Panel.exe ->
Advanced -> Register Distribution.
(This associates .py-files with this Python distribution.)
Anaconda
• Download Anaconda from
https://www.anaconda.com/distribution/.
• Follow the installation instructions from the web page. During the installation,
allow Anaconda to make the suggested modifications to your environment PATH.
• After the installation: in the Anaconda Launcher, click update (besides the apps),
in order to ensure that you are running the latest version.
Installing additional packages When I have had difficulties installing additional
packages, I have been saved more than once by the pre-compiled packages Christoph
Gohlke, available under http://www.lfd.uci.edu/~gohlke/pythonlibs/: from there you
can download the <program>.whl file for your current version of Python, and
then install it simply with pip install <program>.whl.
a) Under Linux
The following procedure works on Linux Mint 20.1:
• Download the most recent version of Anaconda.
• Open terminal, and navigate to the location where you downloaded the file to.
• Install Anaconda with bash Anaconda<xx>-y.y.y-Linux-x86.sh
• Update your Linux installation with sudo apt-get update.
Notes to Anaconda
• You do not need root privileges to install Anaconda if you select a user writable
install location, such as ~/Anaconda.
• After the self-extraction is finished, you should add the Anaconda binary directory
to your PATH environment variable.
• As all of Anaconda is contained in a single directory, uninstalling Anaconda is
easy: you simply remove the entire install location directory.
• If any problems remain, Mac and Unix users should look up Johansson’s instal-
lations tips:
https://github.com/jrjohansson/scientific-python-lectures
12 2 Python

b) Under Mac OS X
• Go to https://www.anaconda.com/distribution/.
• Choose the Mac installer (make sure you select the Mac OS X Python 3.x Graph-
ical Installer), and follow the instructions listed beside this button.
• After the installation: in the Anaconda Launcher, click update (besides the Apps),
in order to ensure that you are running the latest version.
After the installation, the Anaconda icon should appear on the desktop. No admin
password is required. This downloaded version of Anaconda includes the Jupyter
Notebook, Jupyter Qt console, and the IDE Spyder.
To see which packages (e.g., numpy, scipy, matplotlib, and pandas) are featured
in your installation, look up the Anaconda Package List for your Python version. For
example, the Python-installer may not include seaborn. To add an additional pack-
age, e.g., seaborn, open the terminal, and enter pip install seaborn.

2.1.3 Installation of R and rpy2

If you have not used R previously, you can safely skip this section. However, if you
are already an avid R user, the following adjustments will allow you to also harness
the power of R from within Python, using the package rpy2.
a) Under Windows
Also, R does not require administrator rights for installation. You can download
the latest version (at the time of writing R 4.1.0) from http://cran.r-project.org/, and
install it into the <RDir> installation directory of your choice.
• After the installation of R, add the following two variables to your Windows Envi-
ronment, by typing
Win -> env -> Edit environment variables for your account:

– R_HOME=<RDir> R-4.1.0
– R_USER=<YourLoginName>
The first entry is required for rpy2. The last entry is not really necessary, just
better style.
with Anaconda While WinPython comes with rpy2 installed, Anaconda comes
without rpy2. So after the installation of Anaconda and R, you should install rpy2
with
• conda install -c conda-forge rpy2.
b) Under Linux
• After the installation of Anaconda, install R and rpy2 with
conda install -c conda-forge rpy2.
2.1 Getting Started 13

2.1.4 Python Resources

My favorite introductory book for scientific applications of Python is Scopatz and

Huff (2015). However, that book does not provide any information on statistics. If
you have some programming experience, the book you are currently reading may be
all you need to get the statistical analysis of your data going. If required, very good
additional information can be found on the web, where tutorials as well as good free
books are available online. The following links are all recommendable sources of
information for starting with Python:

• Python Scientific Lecture Notes If you don’t read anything else, read this!
http://scipy-lectures.org/
• NumPy for Matlab Users Start here if you have Matlab experience
https://numpy.org/doc/stable/user/numpy-for-matlab-users.html
also check
http://mathesaurus.sourceforge.net/matlab-numpy.html
• Lectures on scientific computing with Python Great IPython notebooks, from JR
Johansson.
https://github.com/jrjohansson/scientific-python-lectures
• The Python tutorial The official introduction
http://docs.python.org/3/tutorial
• 7 Steps to Python My own efforts to smoothen the first steps into Python
https://work.thaslwanter.at/py_intro/

When running into a problem while developing a new piece of code, most of
the time I just google; thereby, I stick primarily to the official Python documenta-
tion pages and to http://stackoverflow.com. Also, I have found Python user groups
surprisingly active and helpful!

2.1.5 A Simple Python Program

a) Hello World
Python Shell The simplest way to start Python is to type python on the command
line. (When I say command line, I refer in Windows to the command shell started
with cmd, and in Linux or Mac OS X to the terminal.) Then you can already
start to execute Python commands, e.g., the command to print “Hello World” to the
screen: print('Hello World'). On my Windows computer, this results in
Python 3.8.9 (tags/v3.8.9:0a7dcbd, May 3 2021, 17:27:52)...
Type "help", "copyright", "credits" or "license" for more...
>>> print('Hello World')
Hello World
>>>
14 2 Python

However, most of the time it is more recommendable to start with the IPython/
Jupyter Qt console described in more detail in Sect. 2.3. For example, the Jupyter Qt
console is an interactive programming environment which offers a number of advan-
tages. When you type print( in the Qt console, you immediately see information
about the possible input arguments for the command print.
Python Modules are files with the extension .py, and are used to store Python
commands in a file for later use. Let us create a new file with the name
helloWorld.py, which contains the single line
print('Hello World')

This file can now be executed by typing python helloWorld.py on the

command line.
On Windows, you can actually run the file by double-clicking it, or by simply
typing helloWorld.py, if the extension .py is associated with the local Python
installation. On Linux and Mac OS X, the procedure is slightly more involved. There,
the file needs to contain an additional first line specifying the path to the Python
installation.
#! \usr\bin\python
print('Hello World')

On these two systems, you also have to make the file executable, by typing
chmod +x helloWorld.py

before you can run it with helloWorld.py.

b) square_me
To increase the level of complexity, let us write a Python module that includes a
function definition and prints out the square of the numbers from zero to five. (More
on functions in Sect. 2.2.5.) We call the file L2_1_square_me.py, and it con-
tains the following lines:

Listing 2.1: square_me.py

1 # This file shows the square of the numbers 0-5.

2

3 def squared(x=10):
4 return x**2
5

6 for ii in range(6):
7 print(ii, squared(ii))
8

9 print(squared())

Let me explain what happens in this file, line-by-line:

1 The first line starts with “#”, indicating a comment-line.

2.2 Elements of Scientific Python Programming 15

3–4 These two lines define the function squared, which takes the variable x as
input, and returns the square (x**2) of this variable. If the function is called
with no input, x is by default set to 10. This notation makes it very simple to
define default values for function inputs.
Note: The range of the function is defined by the indentation! This is a fea-
ture loved by many Python programmers, but often found a bit confusing by
newcomers. Here, the last indented line is line 4, which ends the function
definition.
6–7 Here, the program loops over the first 6 numbers. Also, the range of the for-
loop is defined by the indentation of the code.
In line 7, each number and its corresponding square are printed to the output.
9 This command is not indented, and therefore is executed after the for-loop
has ended. It tests if the function call with “()”, which uses the default param-
eter for x, also works, and prints the result.

Notes:

• Since Python starts at 0, the loop in line 6 includes the six numbers from 0 to 5.
• In contrast to some other languages, Python distinguishes the syntax for function
calls from the syntax for addressing elements of an array, etc.: function calls, as
in line 7, are indicated with round brackets (...); and individual elements of
arrays or vectors are addressed by square brackets [...].

2.2 Elements of Scientific Python Programming

Compared to the simple example above, real-world applications involve not only
individual numbers but also vectors and matrices. These, together with the most
important Python data- and file-structures, will be described in this section.

2.2.1 Python Datatypes

Python offers a number of powerful data structures, and it pays off to make yourself
familiar with them. The most common ones are

• Lists to group objects of the same types.

• Numpy Arrays to work with numerical data. (numpy also offers the data type
np.matrix. However, in my experience np.array is the way to go, since
many numerical and scientific functions will not accept input data in matrix
format.)
• Tuples to group objects of different types.
• Dictionaries for named, structured data sets.
• pandas DataFrames for simple import and export of data, and for statistical data
analysis.
16 2 Python

For simple programs, you will mainly work with lists and arrays. Dictionaries are
used to group related information together. And tuples are used primarily to return
multiple parameters from functions.
In the following, we will use for Python code the input/output formatting of
IPython which will be presented in Sect. 2.3.
List [] Lists are typically used to collect items of the same type (numbers,
strings, ...). They are “mutable”, i.e., their elements can be modified.
Note that “+” concatenates lists.
In [1]: myList = ['abc', 'def', 'ghij']

In [2]: myList.append('klm')

In [3]: myList
Out[3]: ['abc', 'def', 'ghij', 'klm']

In [4]: myList2 = [1,2,3]

In [5]: myList3 = [4,5,6]

In [6]: myList2 + myList3

Out[6]: [1, 2, 3, 4, 5, 6]

Array [] vectors and matrices, for numerical data manipulation. They are defined
in numpy. Note that vectors and 1D arrays are different: vectors CANNOT be
transposed! With arrays, “+” adds the corresponding elements; and the array-
method .dot performs a scalar multiplication. (Since Python 3.5, scalar multi-
plications can also be performed with the operator “@”.)
In [7]: import numpy as np
...: myArray2 = np.array(myList2)
...: myArray3 = np.array(myList3)

In [8]: myArray2 + myArray3

Out[8]: array([5, 7, 9])

In [9]: myArray2.dot(myArray3)
Out[9]: 32

In [10]: myArray2 @ myArray3

Out[10]: 32

Tuple ( ) A collection of different things. Once created, tuples cannot be modi-

fied. (This really irritated me when I started to work with Python. But since I use
tuples almost exclusively to return parameters from functions, this has not turned
out to be any real limitation.)
In [11]: import numpy as np

In [12]: myTuple = ('abc', np.arange(0,3,0.2), 2.5)

In [13]: myTuple[2]
Out[13]: 2.5
2.2 Elements of Scientific Python Programming 17

Dictionary { } Dictionaries are unordered (key/value) collections of content, where

the content is addressed as dict['key']. Dictionaries can be created with the
command dict, or by using curly brackets {...}:
In [14]: myDict = dict(one=1, two=2, info='some information')

In [15]: myDict2 = {'ten':1, 'twenty':20, info':'more

information'}

In [16]: myDict['info']
Out[16]: 'some information'

In [17]: myDict.keys()
Out[17]: dict_keys(['one', 'info', 'two'])

DataFrame Data structure optimized for working with named, statistical data. It
is defined in pandas. (See Sect. 2.2.4.)

2.2.2 Indexing and Slicing

The rules for addressing individual elements in Python lists, tuples, or numpy arrays
have been nicely summarized by Greg Hewgill on stackoverflow1 :
a[start:end] # items start through end-1
a[start:] # items start through the rest of the array
a[:end] # items from the beginning through end-1
a[:] # a copy of the whole array

There is also the step value, which can be used with any of the above:
a[start:end:step] # start up to end, by step

The key points to remember are that indexing starts at 0, not at 1; and the :end
value represents the first value that is not in the selected slice. So, the difference
end - start is the number of elements selected (if step is 1, the default).
start or end may be a negative number. In that case the count goes from the end
of the array instead of the beginning. So
a[-1] # last item in the array
a[-2:] # last two items in the array
a[:-2] # everything except the last two items

As a result, a[:5] gives you the first five elements (Hello in Fig. 2.4), and a[-5:]
the last five elements (World).

1 http://stackoverflow.com/questions/509211/explain-pythons-slice-notation.
18 2 Python

Fig. 2.4 The expressions in the top row indicate possible notations to select elements of a list or
an array. Indexing starts at 0, and slicing does not include the last value

2.2.3 Numpy Vectors and Arrays

numpy is the Python module that makes working with numbers efficient. It is com-
monly imported with
import numpy as np

By default, it produces vectors. The commands most frequently used to generate

numbers are as follows:

np.zeros generates numpy arrays containing zeros. Note that it takes only one(!)
input. If you want to generate a matrix of zeroes, this input has to be a tuple or a
list, containing the number of rows/columns!
In [1]: import numpy as np

In [2]: np.zeros(3)
# by default numpy-functions generate 1D-vectors
Out[2]: array([ 0., 0., 0.])

In [3]: np.zeros( [2,3] )

Out[3]: array([[ 0., 0., 0.],
[ 0., 0., 0.]])

np.ones generates numpy arrays containing ones.

np.random.randn generates normally distributed numbers, with a mean of 0 and
a standard deviation of 1. To produce reproducible random numbers, you have to
specify the starting point for the random number generation, for example, with
np.random.seed(...), using an integer number of your choice.
np.arange generates a range of numbers. Parameters can be
start, end, steppingInterval. Note that the end value is excluded! While
this can sometimes be a bit awkward, it has the advantage that consecutive
sequences can be easily generated, without any overlap, and without missing
any data points:
In [4]: np.arange(3)
Out[4]: array([0, 1, 2])

In [5]: xLow = np.arange(0, 3, 0.5)

2.2 Elements of Scientific Python Programming 19

In [6]: xHigh = np.arange(3, 5, 0.5)

In [7]: xLow
Out[7]: array([ 0., 0.5, 1., 1.5, 2., 2.5])

In [8]: xHigh
Out[8]: array([ 3., 3.5, 4., 4.5])

np.linspace generates linearly spaced numbers

In [9]: np.linspace(0, 10, 6)
Out[9]: array([ 0., 2., 4., 6., 8., 10.])

np.array generates a numpy array from given numerical data, and is a convenient
notation to enter small matrices
In [10]: np.array([[1,2], [3,4]])
Out[10]: array([ [1, 2],
[3, 4] ])

There are a few points that are peculiar to Python, and that are worth noting:

matrices are simply “lists of lists”. Therefore, the first element of a matrix gives
you the first row, the second element the second row, etc.:
In [11]: Amat = np.array([ [1, 2],
[3, 4] ])

In [12]: Amat[0]
Out[12]: array([1, 2])

Warning: A vector is not the same as a 1D matrix! This is one of the few features
of Python that is not intuitive (at least to me), and can lead to mistakes that are
hard to find. For example, vectors cannot be transposed, but matrices can.
In [13]: x = np.arange(3)

In [14]: Amat = np.array([ [1,2], [3,4] ])

In [15]: x.T == x
Out[15]: array([ True, True, True])
# This indicates that a vector stays a vector, and that
# the transposition with ''.T'' has no effect on
# its shape

In [16]: Amat.T == Amat

Out[16]: array([[ True, False],
[False, True]])

np.r_ Useful command to quickly construct small row vectors. But I only use
it to try things out quickly. In my programs, I prefer the clearer but equivalent
np.array([...])
20 2 Python

In [17]: np.r_[1,2,3]
Out[17]: array([1, 2, 3], dtype=int32)

np.c_ Useful command to quickly build up small column vectors. Note that
column-vectors can also be generated with the command np.newaxis:
In [18]: np.c_[[1.5,2,3]] # note the double brackets!
Out[18]: array([[1.5],
[2. ],
[3. ]])

In [19]: x[:, np.newaxis]

Out [19]: array([[0],
[1],
[2]])

np.atleast_2d Converts a vector (which cannot be transposed; see above) to the

corresponding 2D array (which can be transposed):
In [20]: x = np.arange(5)

In [21]: x
Out[21]: array([0, 1, 2, 3, 4])

In [22]: x.T
Out[22]: array([0, 1, 2, 3, 4]) # no effect on 1D-vectors

In [23]: x_2d = np.atleast_2d(x)

In [24]: x_2d.T
Out[24]: array([[0],
[1],
[2],
[3],
[4]])

np.column_stack An elegant command to generate column matrices:

In [25]: x = np.arange(3)

In [26]: y = np.arange(3,6)

In [27]: np.column_stack( (x,y) )

Out[27]: array([[0, 3],
[1, 4],
[2, 5]])

2.2.4 pandas DataFrames

pandas (http://pandas.pydata.org/) is a widely used Python package, and provides

data structures suitable for statistical analysis and data manipulation. It also adds
2.2 Elements of Scientific Python Programming 21

Fig. 2.5 pandas DataFrame

functions that facilitate data input, data organization, and data manipulation. pandas
is commonly imported with
import pandas as pd
The official pandas documentation contains a very good “Getting started” section:
https://pandas.pydata.org/docs/getting_started/.
a) Basic Syntax of DataFrames
Especially in statistical data analysis (read “data science”), panelled data structures
(→ “pandas”) have turned out to be immensely useful. To handle such labeled data
in Python, pandas introduces so-called “DataFrame” objects. A DataFrame is a 2D
labeled data structure with columns of potentially different types. You can think of
it as a spreadsheet or SQL table (see Fig. 2.5). DataFrames are the most commonly
used pandas objects.
For statistical analysis, pandas becomes really powerful when combined with the
package statsmodels (https://www.statsmodels.org/).
pandas DataFrames can have some distinct advantages over numpy arrays:
• A numpy array requires homogeneous data. In contrast, with a pandas
DataFrame you can have a different data type (float, int, string, datetime, etc.)
in each column (Fig. 2.6).
• pandas has built-in functionality for a lot of common data-processing applica-
tions: for example, easy grouping by syntax, easy joins (which are also really
efficient in pandas), and rolling windows.
• DataFrames, where the data can be addressed with column names, can help a lot
in keeping track of your data.
22 2 Python

Fig. 2.6 Demonstration of some features of pandas DataFrames in JupyterLab (see Sect. 2.3.2).
Unlike np.array, a pd.DataFrame can be used to combine different data types (here, strings,
floats, integers, and dates: commands 2–4). It also can work efficiently with date and time data
(commands 2 and 6, also note the output of “6” in the “Output View” on the right. This is a new
feature of JupyterLab)

In addition, pandas has excellent tools for data input and output.
Let me start with a specific example, by creating a DataFrame with three columns
called “Time”, “x”, and “y”:
import numpy as np
import pandas as pd

t = np.arange(0, 10, 0.1)

x = np.sin(t)
y = np.cos(t)

df = pd.DataFrame({'Time':t, 'x':x, 'y':y})

In pandas, rows are addressed through indices, and columns through their name.
To address the first column only, you have two options:
df.Time
df['Time']

To extract two columns at the same time, put the variable names in a list. With the
following command, a new DataFrame data is generated, containing the columns
Time and y:
data = df[['Time', 'y']]
2.2 Elements of Scientific Python Programming 23

After reading in the data, it is good practice to check if the data have been read
in correctly. The first or last few rows can be displayed with
data.head()
data.tail()

For example, the following statement shows the 5th − 10th row (note that these
are 6 rows):
data[4:10]

as 10 − 4 = 6. (I know, the array indexing takes some time to get used to. It helps
me to think of the indices as pointers to the elements, and that they start at 0. see
Fig. 2.4)
The handling of DataFrames is somewhat different from the handling of numpy
arrays. For example, (numbered) rows and (labeled) columns can be addressed
simultaneously as follows:
df[['Time', 'y']][4:10]

The standard row/column notation can be used by applying the method iloc:
df.iloc[4:10, [0,2]]

Finally, sometimes one wants direct access to the data, not to the DataFrame.
This can be achieved with
data.values

which returns a numpy array if all data have the same data type.
b) Note: Data Selection
While pandas’ DataFrames are similar to numpy arrays, their philosophy is differ-
ent. The numpy syntax comes from the mathematical description of n-dimensional
matrices. In contrast, pandas has its origin in the data analysis of column-oriented
database information. Some of the differences between the two that you should
watch out for are

• numpy handles “rows” first. For example, data[0] is the first row of an array.
• pandas starts with the columns. For example, df['values'][0] is the first ele-
ment of the column 'values'.
• If a DataFrame has labeled rows, one can extract, for example, the row “row_label”
with df.loc['row_label']. If one wants to address a row by its number, e.g.,
row number “15”, one can use df.iloc[15]. And iloc can be used to address
“rows/columns”, e.g., df.iloc[2:4,3].
• Slicing of rows also works, e.g., df[0:5] for the first 5 rows. A sometimes con-
fusing convention is that if you want to slice out a single row, e.g., row “5”, you
have to use df[5:6]. df[5] raises an error!
24 2 Python

Code:ISP_showPandas.py2 demonstrates some of pandas

powerful functions for handling missing data, and for grouping and pivoting data
items.

2.2.5 Functions, Modules, and Packages

Python has three different levels of modularization:

Functions are defined by the keyword def, and can be defined anywhere in
Python. They return the object in the return statement, typically at the end of
the function.
Modules are files with the extension .py. Modules can contain function- and
variable- definitions, as well as valid Python statements.
Packages are folders containing multiple Python modules, and must contain a
file named __init__.py. For example, numpy is a Python package. Since
packages are mainly important for grouping a larger number of modules, they
won’t be discussed in this book.
a) Functions
A function is a set of statements that take inputs, do some specific computation, and
produce output. The idea is to group commonly or repeatedly done tasks and make
a function, so that instead of writing the same code again and again for different
inputs we can call the function. In Python, functions can be declared at any point in
a program with the command def.
A short application example is given in Listing 2.2. Note that in the function
definition so-called “type hints” are used (line 11) to indicate input and return type.
They are optional, but make the code easier to read and understand.
Listing 2.2: python_module.py

1 """ Demonstration of a Python Function """

2
3 # author: Thomas Haslwanter
4 # date: June-2022
5

6 # Import standard packages

7 import numpy as np
8 from typing import Tuple
9

10

11 def income_and_expenses(data : np.ndarray) -> Tuple[float,

float]:

2 <ISP2e>/02_GettingStarted/ISP_showPandas.py.
2.2 Elements of Scientific Python Programming 25

12 """Find the sum of the positive numbers, and the sum of

the negative ones.
13

14 Parameters
15 ----------
16 data : numpy array (,n)
17 Incoming and outgoing account transactions
18

19 Returns
20 -------
21 income : Sum of the incoming transactions
22 expenses : Sum of the outgoing account transactions
23 """
24

25 income = np.sum(data[data>0])
26 expenses = np.sum(data[data<0])
27
28 return (income, expenses)
29

30

31 if __name__=='__main__':
32 testData = np.array([-5, 12, 3, -6, -4, 8])
33

34 # If only real banks would be so nice ;)

35 if testData[0] < 0:
36 print('Your first transaction was a loss and is
dropped.')
37 testData = np.delete(testData, 0)
38 else:
39 print('Congratulations: Your first transaction is a
gain!')
40

41 (my_income, my_expenses) = income_and_expenses(testData)

42 print(f'You have earned {my_income:5.2f} EUR, ' + \
43 f'and spent {-my_expenses:5.2f} EUR.')
A detailed description of what happens in this piece of code is given below.
The example in Listing 2.2 shows how functions can be defined and used.
• 1: Module header, commonly written as a multiline comment
("""< x x x > """).
• 3/4: Author and date information (should be separate from the module header).3
• 7: The required Python packages have to be imported explicitly. Here, numpy will
be required, and it is customary to import numpy as np.
• 8: The command Tuple from the package typing will be used in the “type hints”
for the upcoming function. Type hints give hints on the type of the object(s) the
function is using and for its return. They are optional, but improve the readability
of code.
• 9/10: Keep 2 empty lines before function definitions.

3 For the rest of the book, the “author/date” information will be left away, to keep the program
listings more compact.
26 2 Python

• 11: Function signature.

• 12–23: Multiline comment describing the function. It should also include infor-
mation about the parameters the function takes, and about the return elements.
• 11–28: Function definition. Note that in Python the function block is defined by
the indentation, not by any brackets or end statements! This is a feature that irri-
tates many Python novices, but really helps to keep code clear and nicely for-
matted. Important: Python makes a difference between a tab and the equivalent
amount of spaces. This can lead to errors which are really hard to detect, so use a
good IDE that automatically converts tabs to spaces!
• 25:
– The sum command is taken from numpy, so it has to be preceded by np.
– In Python, function arguments are indicated by round brackets (...), whereas
elements of lists, tuples, vectors, and arrays are indicated by square brackets
[...].
– In numpy, you can select elements of an array either with an index (see line
35), or with a Boolean array (lines 25–26).
• 28: Python also uses round brackets to form groups of elements, so-called
“tuples”. And the return statement does the obvious things: it returns elements
from a function.
• 31: Here, quite a few new aspects of Python are introduced:
– Just like function definitions, if-loops or for-loops use indentation to define
their context.
– A convention followed by most Python coders is to prefix variables or methods
that are supposed to be treated as a non-public part of the Python code with an
underscore, for example, _geek or __name__.
– Here, we check the variable with the name __name__, which is automatically
generated by the Python interpreter and indicates the context of a module eval-
uation. If the module is run as a Python script, __name__ is automatically set
to __main__. But if a module is imported (see, e.g., Listing 2.3), it is set to the
name of the importing module. This way it is possible to add code to a function
that is only used when the module is executed, but not when the functions in
this module are imported by other modules (see below). This is a nice way to
test functions defined in the same module.
• 32: Definition of a numpy array.
• 41: The two elements returned as a tuple from the function
income_and_expenses can be simultaneously assigned to two different Python
variables, here to (my_income, my_expenses).
• 42: While there are different ways to produce formatted strings, the “f-strings”
that were introduced with Python 3.6 are probably the most elegant: curly brackets
{} indicate values that will be inserted. The optional expression after the colon
contains formatting statements: here :5.2f indicates “express this number as a
2.2 Elements of Scientific Python Programming 27

float, with 5 digits, 2 of which are after the comma”.4 The corresponding values
are then passed into the f-string for formatted output. And the '\' at the end of
the line indicates a line continuation.
b) Modules
To execute the module L2_2_python_module.py from the command line, type
python L2_2_python_module.py. In Windows, if the extension “.py” is
associated with the Python program, it suffices to double-click the module, or to
type python_module.py on the command line. In WinPython, the association
of the extension “.py” with the Python function can be set by the WinPython Control
Panel.exe, by the command Register Distribution ... in the menu Advanced.
To run a module in IPython, use the magic function %run:
In [1]: %run L2_2_python_module
Your first transaction was a loss and is dropped.
You have earned 23.00 EUR, and spent 10.00 EUR.
Note that you either have to be in the directory where the function is defined, or
you have to give the full path name.
If you want to use a function or variable that is defined in a different module,
you have to import that module. This can be done in three different ways. For the
following example, assume that the other module is called new_module.py, and
the function that we want from there new_function.
• import new_module: The function can then be accessed with
new_module.new_function().
• from new_module import new_function: In this case, the function can be
called directly new_function().
• from new_module import *: This imports all variables and functions from
new_module into the current workspace; again, the function can be called directly
with new_function(). However, use of this syntax is discouraged! It clutters up
the current workspace, and one risks overwriting existing variables with the same
name as an imported variable.
If you import a module multiple times, Python recognizes that the module is
already known and skips later imports.
The next example shows you how to import functions from one module into
another module:
Listing 2.3: python_import.py

1 """ Demonstration of importing a Python module """

2

3 # Import standard packages

4 import numpy as np
5

6 # additional packages: this imports the function from above

4 https://pyformat.info/ contains all the details of formatted output in Python.

28 2 Python

7 import L2_2_python_module as py_func

8

9 # Generate test-data
10 testData = np.arange(-5, 10)
11

12 # Use a function from the imported module

13 out = py_func.income_and_expenses(testData)
14

15 # Show some results

16 print(f'You have earned {out[0]:5.2f} EUR, '+\
17 f' and spent {-out[1]:5.2f} EUR.')

• 7: The module L2_2_python_module (that we have just discussed above) is

imported, as py_func.
• 13: To access the function income_and_expenses from the module
py_func, module- and function-name have to be given:
py_func.income_and_expenses(...). Note that out here contains both return-
variables.

2.3 Interactive Programming—IPython/Jupyter

2.3.1 Workflow

The best way to start a program is to take a paper and pencil and explicitly write
down the algorithms to be implemented! This helps to clarify the required program-
ming steps, which parameters have to be provided explicitly, and which have to be
calculated during the execution of the program. In most cases, this is also the most
efficient way to start the development of a new program.
The next step is to work out the command syntax. In Python, this is best done
with IPython/Jupyter. IPython (http://ipython.org/) provides a programming envi-
ronment that is optimized for interactive computing with Python, similar to the
command line in Matlab. It comes with a command history, interactive data visual-
ization, command completion, and a lot of features that make it quick and easy to
try out code.
Once the individual steps are working, one can use the IPython command
%history to get the commands used. One can use either copy/paste that history, or
save it directly to a file with
%history -f [fname]

Then one can switch to an integrated development environment (IDE), in my

case Wing, to generate the final, working program.
The example in Fig. 2.7 shows the first steps for a program that generates a
sine wave. Underlining the required parameters helps me to see which parameters
need to be defined at the beginning of the program. And spelling out each step
explicitly, e.g., the generation of a time-vector in line 4 in Fig. 2.7, clarifies which
2.3 Interactive Programming—IPython/Jupyter 29

Fig. 2.7 Still, the best start to the successful development of a new program!

additional parameters arise in the program implementation. This approach speeds up

the implementation of a program and is an important first step in avoiding mistakes.

2.3.2 Jupyter Interfaces

While IPython can also be run in a terminal-environment, its full power becomes
available with Jupyter. In 2013 the IPython Notebook, a browser-based frontend for
Python, became a very popular way to share research and results in the Python com-
munity. In 2015, the development of the frontend became its own project, called
Project Jupyter (https://jupyter.org/). Today, Jupyter can be used not only with
Python, but also with Julia, R, and more than 100 other programming languages.
The most important interfaces provided by Jupyter are

• Qt console.
• Jupyter Notebook.
• JupyterLab.

They can be started from a terminal with the command

jupyter <viewer>

where viewer is qtconsole, notebook, or lab.

a) Qt Console
The Qt console (see Fig. 2.8) is my preferred way to start coding, especially to figure
out the correct command syntax. It provides immediate feedback on the command
syntax, and good text completion for commands, file names, and variable names.
30 2 Python

Fig. 2.8 The Qt console, displaying parameter tips for the current command, np.arange

b) Jupyter Notebook
The Jupyter Notebook is a browser-based interface, which is especially well suited
for teaching, documentation, and collaborations. It allows you to combine a struc-
tured layout, equations in the popular LaTeX format, and images, and can include
resulting graphs and videos, as well as the output from Python commands (see
Fig. 2.9). Packages such as plotly (https://plot.ly/) or bokeh (https://bokeh.org/) build
on such browser-based advantages, and allow easy construction of interactive inter-
faces inside Jupyter Notebooks.
Code samples accompanying this book are also available as Jupyter Notebooks,
and can be downloaded from
https://github.com/thomas-haslwanter/statsintro-python-2e.
c) JupyterLab
JupyterLab is the successor to the Jupyter Notebook. As Fig. 2.10 shows, it extends
the Notebook with very useful capabilities such as a file browser, easy access to
commands and shortcuts, and flexible image viewers. The file format stays the same
as the Notebook, and both are saved as .ipynb-files.
2.3 Interactive Programming—IPython/Jupyter 31

Fig. 2.9 The Jupyter Notebook makes it easy to share research, formulas, and results

2.3.3 Personalizing IPython/Jupyter

When working on a new problem, I always start out with the Qt console (see
Fig. 2.8).
In the following, <mydir> has to be replaced with your home directory (i.e., the
directory that opens up when you run cmd in Windows, or terminal in Linux).
And <myname> should be replaced by your name or your userID.
To start up IPython in a folder of your choice, and with personalized startup
scripts, proceed as follows:
a) In Windows
• Type Win+R, and start a command shell with cmd.
• In the newly created command shell, type
ipython profile create.
(This creates the directory <mydir>\.ipython.)
32 2 Python

Fig. 2.10 JupyterLab is the successor to Jupyter Notebook. The blue labels indicate new features
in JupyterLab

• Add the Variable IPYTHONDIR to your environment (see Sect. 2.1.2), and set
it to <mydir>\.ipython. This directory contains the startup commands for
your IPython-sessions.
• Into the startup folder
<mydir>\.ipython\profile\_default\startup,
place a file for example with the name <myname>.py, containing the startup
commands that you want to execute every time that you launch IPython. My per-
sonal startup file contains the following lines, which will import frequently used
packages:
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy import stats

• Generate a file “ipy.bat” in <mydir>, containing

jupyter qtconsole

• To customize the jupyter qtconsole type

jupyter notebook -generate-config.
This creates the file jupyter_qtconsole_config.py in your Jupyter
folder. The Jupyter folder is in the subfolder ~/.jupyter in your home direc-
tory. In this file you find multiple options to configure your Qt Console, e.g., the
distance between commands, the editor used, and the header displayed at the pro-
gram start.
(The same procedure can be used to customize the jupyter notebook and
jupyter lab.)
2.3 Interactive Programming—IPython/Jupyter 33

To see all Jupyter Notebooks that come with this book, for example, do the fol-
lowing:
• Type Win+R, and start a command shell with cmd.
• Run the commands
cd <ipynb-dir>
jupyter lab

where <ipynb-dir> is the directory where all the Jupyter Notebooks are
stored.
• Again, if you want, you can put this command-sequence into a batch-file.
b) In Linux

• Start a Linux terminal with the command terminal.

• In the newly created command shell, execute the following command:
ipython

(This generates a folder .i python.)

• Into the subfolder .ipython/profile_default/startup, place a file
with, e.g., the name 00<myname>.py, containing the lines
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
import pandas as pd

• In your .bashrc file (which contains the startup commands for your shell-
scripts), enter the lines
alias ipy='jupyter qtconsole'
IPYTHONDIR='~/.ipython'

• To see all Jupyter Notebooks, do the following:

– Go to <mydir>.
– Create the file ipynb.sh, containing the lines
#!/bin/bash
cd <ipynb-dir>
jupyter lab

– Make the file executable, with chmod 755 ipynb.sh.

You can now start “your” IPython by just typing ipy, and JupyterLab by typing
ipynb.sh.
34 2 Python

c) In Mac OS X

• Start the Terminal either by manually opening Spotlight or the shortcut

CMD + SPACE and entering Terminal and search for “Terminal”.
• In Terminal, execute ipython, which will generate a folder under
<mydir>/.ipython.
• Enter the command pwd into the Terminal. This lists <mydir>; copy this for
later use.
• Now open Anaconda and launch an editor, e.g., spyder-app or TextEdit. Create
a file containing the command lines you regularly use when writing code (you
can always open this file and edit it). For starters, you can create a file with the
following command lines:
import pandas as pd
import os
os.chdir('<mydir>/.ipython/profile_<myname>')

• The next steps are somewhat tricky. Mac OS X by default hides the folders
that start with “.” (They can be shown with cmd-shift-.). So to access
.ipython, open File -> Save as..... Now open a Finder window,
click the Go menu, select Go to Folder, and enter
<mydir>/.ipython/profile_default/startup. This will open a
Finder window with a header named “startup”. On the left of this text, there
should be a blue folder icon. Drag and drop the folder into the Save as... win-
dow open in the editor. IPython has a README file explaining the naming
conventions. In our case, the file should begin with 00-, so we could name it
00-<myname>. .
• Open your .bash_profile (which contains the startup commands for your
shell scripts), and enter the line
alias ipy='jupyter qtconsole'.
• To see all IPython Notebooks, do the following:
– Go to <mydir>.
– Create the file ipynb.sh, containing the lines
#!/bin/bash
cd <ipynb_dir>
jupyter lab

– Make the file executable, with chmod 755 ipynb.sh.

2.3.4 Sample Interactive Session

An important aspect of data analysis is interactive, visual inspection of the data.

My personal preference for this, and for working out the syntax of the commands
needed, is to start data analysis in the Jupyter Qt console.
2.3 Interactive Programming—IPython/Jupyter 35

In this example, I start my IPython sessions from the command line, with the
command jupyter qtconsole. (Under WinPython: if you have problems start-
ing Jupyter from the cmd console, use the WinPython Command Prompt instead—it
is nothing else but a command terminal with the environment variables set such that
Python is readily found.)
To get started with Python and IPython, let me go step-by-step through the
IPython session in Fig. 2.11:

Fig. 2.11 Sample session in the Jupyter Qt console

36 2 Python

• IPython starts out listing the versions of IPython and Python that are used.
• In [1]: It is customary to import numpy as np, and matplotlib.pyplot, the mat-
plotlib module containing all the plotting commands, as plt. Note that by hitting
CTRL+Enter, one can execute multiline commands. (The command sequence
gets executed after the next empty line.)
• In [2]: The command t = np.arange(0,10,0.1) generates a vector from 0 to
10, with a step size of 0.1. arange is a command in the numpy package.
• In [3]: Calculates omega. Note that the value of pi is only defined in numpy,
and does not exist in Python!
• In [4]: Since t is a vector, and sin is a function from numpy, the sine value is
calculated automatically for each value of t.
• In [5]: The “IPython magic function” pwd stands for “print working directory”,
and does just that. Tasks common with interactive computing, such as directory
changes (%cd), bookmarks for directories (%bookmark), and inspection of the
workspace (%who and %whos), are implemented as “IPython magic functions”.
If no Python variable with the same name exists, the “%” sign can be left away,
as here.
• In [7]: All the plotting commands are in the package plt. IPython generates plots
by default in the Jupyter Qt console, as shown in Fig. 2.11. Generating graphics
files is also very simple: here, I generate the PNG-file “Sinewave.png”, with a
resolution of 200 dots-per-inch.

I have mentioned above that matplotlib handles the graphics output. In Jupyter,
you can switch between inline graphs and output into an external Graphics win-
dow with %matplotlib inline and %matplotlib qt5 (see Fig. 2.12).
(Depending on your version of Python, you may have to replace %matplotlib
qt5 with %matplotlib or with %matplotlib tk.) An external graphics win-
dow allows zooming and panning, gets the cursor position (which can help to find
outliers), and gets interactive input with the command plt.ginput. matplotlib’s
plotting commands closely follow Matlab conventions.

2.3.5 Converting Interactive Commands into a Python

Program

IPython is very helpful in working out the command syntax and sequence. The next
step is to turn these commands into a Python program with comments that can be
run from the command line. This section introduces a number of Python conventions
and syntax features.
For me, an efficient way to turn IPython commands into a script is to

• first obtain the command history with the command %hist or %history. (With
the option -f, you can save the history directly with the desired filename.)
• copy the history into a good integrated development environment (IDE): my pre-
ferred IDE is Wing (http://www.wingware.com/), because it provides a very com-
Random documents with unrelated
content Scribd suggests to you:
animals belonging to the articulate type, the Insecta possess the
highest instincts. To this class belong the following orders: I.
Neuroptera,[14] or lace-winged insects, of which the Dragon-fly, or
Devil’s Darning Needle, is a good representative. II. The
Orthoptera[15] (straight-winged). They have four wings, the front
pair thick and narrow, overlapping along the back; the hind pair
broad, net-veined and folded upon the abdomen. The representative
forms are Crickets, Grasshoppers, Locusts, and Cockroaches. III.
Hemiptera[16] (half winged). To this order belong the wingless Bed-
bug, the Squash-bug, the Seventeen-year Locust, and the Cochineal.
Coccus-cacti[17]. The Mexican cochineal insect is of great value as
a dye, and from it the most beautiful scarlet and crimson colors are
obtained. The female is wingless, and, as an uncomely parasite, lives
and feeds on cactus plants, especially those of the nopal[18] species,
native in Mexico and Peru. The male only is represented in the
diagram, and magnified somewhat larger than life. They are
comparatively few in number, and of no commercial value. The
plants are cultivated for the purpose, and the care of the insects,
which increase very rapidly, is an industry giving employment to
thousands of laborers, and in some parts of Mexico the product of
the cochineal farms is among the most valuable of their exports.
Cortes[19] in the sixteenth century received instructions from the
Spanish court to obtain cochineal in as large quantities as possible.
The export became very large, both Spaniards and others becoming
skilled in the use of the beautiful dye stuff, nearly a hundred years
before its real nature was known. The dried insect being very small,
and crushed in preparing it for market, it was supposed to be the
seed of some plant; and it was not until in 1703 that its true nature
was discovered by microscopic observations. The industry still
flourishes in Mexico, but both plant and the insect have been taken
to other countries, and do well. The annual export of cochineal from
the Canary islands has, in the present century, amounted to over
6,000 tons, valued at more than $4,000,000. The manner of
collecting the insects is very simple. When of sufficient age, some
already dead, and others yet alive, they are brushed off into bags,
and the living killed by holding them either in boiling water or heated
ovens, and then exposed in the sun till quite dry. The dried insects
have the form and appearance of irregular fluted and concave
grains, of which it is estimated there are 70,000 in a pound.

THE COCHINEAL.

There are several species of these insects, alike in form and

habits, but not alike useful. Some, as the Scale insect, are a great
annoyance to gardeners, and destructive to our house plants.
Others, as the Wax insect, live on certain tropical trees, and soon
entomb themselves in a mass of glutinous matter that oozes from
the small twigs of the tree, and which furnishes them both food and
shelter. As they are marvelously prolific, a single female, according
to the estimates of entomologists, being succeeded by many millions
of descendants in less than a year, when a colony has possession of
a tree, every tender branch is soon punctured, and the abundant
resinous juices that flow out envelop it in a coating often half an
inch thick. These branches, and also what falls to the ground, are
collected, and the wax, melted off, is prepared for the market. From
this source the shellac of commerce, so extensively used, is
obtained. This curious and useful insect, like its congener, the
cochineal, secretes a coloring substance, but of different tints, and
less valuable. The Lac insect is a native of Siam, Assam, Burmah,
Bengal, and Malabar. For some years the average annual imports
into Britain have been a little over 600 tons of the lac dye, and more
than a thousand tons of lac, including the several varieties. This
industry also gives employment to many thousands of people.

METAMORPHOSIS OF A BEETLE.

Ex.—A, larva; b, pupa; c, beetle.

IV.—Coleoptera[20] (sheath-winged). Beetles are innumerable,

about ninety thousand species being recognized. Their anterior or
upper wings, useless for flight, are composed of a hard, horn-like
substance known as chitine, and meet in a straight line on the top of
the back. The posterior wings are thin, membranous, and, when
folded, out of sight. They have usually two pairs of laterally moving
mandibles, or jaws, and in their development undergo several
metamorphoses. We see the egg, the larva, or grub, in different
stages of its growth, the chrysalis, and the imago,[21] or complete
beetle. Entomologists have spent much time and labor in making
collections, and classifying them according to their peculiarities of
form or habits. If many are repulsive, and most plain in form and
color, some are beautiful, and worthy of our admiration. Beetles,
especially in their larva stage of development are very voracious,
and as most of their species live on fruits, leaves, and stems of
plants, they are often destructive of crops, and even of forests.
Millions of vigorous, valuable trees have been assailed, and stripped
of their leaves fast as they appeared, or literally bored to death. All
know the ravages of the potato beetle on our American tuber, that
has to be assiduously defended to prevent the entire destruction of
the crop. Some known as Goliath and Hercules beetles are large,
often measuring six inches in length, exclusive of their long
antennæ. The “Diamond beetles” of Brazil are adorned with the
most brilliant colors, showing a beautiful metallic luster, and the
elytra,[22] or chitine sheaths, of this species are now largely used in
the manufacture of personal ornaments.[23]
V. Diptera (two-winged), or Flies, number about 24,000 varieties.
Among these are the Mosquito, Hessian fly, Daddy-long-legs, Flea,
and common House fly. They usually have one pair of fully
developed wings, the second pair being rudimentary, although a few,
as the fleas, are wingless. They pass through a complete
metamorphosis, the larvæ being usually footless maggots, with the
breathing holes in the posterior part of the body; the pupa are either
encased in the dry skin of the larvæ or are naked.
VI. Lepidoptera[24] (scaly-winged), or Butterflies and Moths, are
distinguished by four wings, covered on both sides by minute scales.
The butterflies fly by day and have knobbed antennæ, while the
moths fly by night and have feathery antennæ. Among the moths
one of the most interesting is that of the silk worm.[25] The
physiology of the insect and its metamorphoses reveal nothing very
peculiar, and its habits need not be mentioned farther than to say,
the larva eats voraciously, with short intervals of abstinence, until full
grown, which stage is reached in about a month. During the last ten
days the silk germ is elaborated, the eggs laid, and then the
spinning and winding soon begin. To complete the cocoon requires
at most only about three days. The larva then becomes a chrysalis,
and in due time the moth emerges from its cell.
VII. Hymenoptera (membrane-winged) comprises the Ichneumon
and Gall fly, Ants, Wasps, Bees, in all about 25,000 species. This
order includes the most social of the insects. They have four wings,
which in flying they fasten together by means of small hooks on the
edges. The females are usually provided with a sting or borer. The
Gall fly produces the gall nuts or oak balls so common on oak trees.
The Ichneumon fly introduces its eggs underneath the skin of the
caterpillar.[26]
Ants[27] live in communities. They are divided into fertile females,
males and infertile females. Among the ants, the mining ants, which
make long galleries in the earth, and carpentering ants, which
perforate solid timber, are the best known classes. Some species,
like the white ants of the tropics, the termites, are famous for their
ravages.
The bees are divided into queens or females, drones or males,
and workers. Each community or swarm has one queen, which lays
the eggs. The bee is provided with a formidable sting.
This curious weapon of attack and defense is here magnified with
the adjacent parts. It consists of an extensile sheath with two needle
shaped darts that are exceedingly sharp. This spear is furnished with
barbs near the point, and when it pierces the skin, if thrust with
violence, it sometimes remains, not only making the wound more
painful, but, having been wrenched from the bee, frequently causes
its death. The sting is connected with a little sac containing a
poisonous liquid which is thrown into the wound and increases the
pain.
 STING OF A BEE.

SUB-KINGDOM VIII.—TUNICATA.
This very small class of animals is distinguished by the leathery
sac-like covering, from which they take their name of Tunicata
(having a tunic). The Ascidian[28] is the best known representative.
It is found fastened to rocks, shells, crabs, and other bodies. These
animals are both simple and compound; the latter are often
phosphorescent. They have neither feet, head nor shell, but a
shapeless body with apertures at both ends.

SUB-KINGDOM IX.—VERTEBRATA.
This division includes the most perfect animals. Their chief
distinguishing characteristics are an internal skeleton; a backbone; a
dorsal nervous cord, separated from the body cavity; a complete
circulation, and limbs not exceeding four. There are about 25,000
living species, beside the numberless host now extinct.
Lowest of the vertebrates, and closely related to the true fishes,
are the Lancelet and Lamprey.[29] The former is a lance-shaped
animal having no skeleton, but boasting the rudiment of a backbone
in a string-like cartilaginous cord. The organs are very simple, the
heart being a long sac in which colorless blood circulates. It breathes
by taking in water through the mouth and letting it out through the
gill-slits. The Lamprey belongs to the pouch-gilled vertebrates or
Marsipobranchii.[30] It is an eel-shaped animal of about three feet in
length.
The round, soft mouth is suctorial, the tongue acting as a piston.
By suction it can anchor itself to a rock, and allow the long body to
float freely, without being carried by the current from the place.
There are seven gill openings on each side, and the whole breathing
apparatus so arranged that the animal can live some days out of the
water. In some parts of England it is in demand for the table, and
“lamprey pie”[31] is esteemed a great luxury. The American lamprey
is similar, and the flesh good, but in less demand in our markets.
During the breeding season those about the estuaries go up stream
as do the shad, and by rolling stones together construct large conical
nests[32] for the protection of their young. With apparently little
adaptation for such architecture they accomplish wonders in that
line.
 THE LAMPREY.

Class I.—Pisces. The first clearly defined division of vertebrates is

that of the fishes. They are regarded as, in some respects, the
lowest of vertebrate animals. They are credited with having the least
intelligence and sensibility. Their eyes, though often large, are nearly
motionless in their sockets, are protected by no eye-lids, and are
without the expression usual even in the animal eye. If they have
ears the external parts are wanting. Sounds may reach the auditory
nerve and be heard through the cranium. The other senses, as taste,
touch and smell, are but slightly developed. But they are admirably
adapted to the element in which they live, and the mode of life for
which they were created. In no other organisms is the evidence of
design in the adaptation of means to the ends contemplated, more
apparent. In the number of species and variety of forms they exceed
all other vertebrate animals.
The skeleton of a fish is usually divided into four parts: the head,
respiratory organs, vertebral column and limbs.
The head is very suggestive, and of itself presents a profitable
study. It is not hung or poised on a neck, but attached immediately
to the body. In most species it is large, making a large mouth
possible, but pointed, to lessen the resistance met in passing
through the water. In some the eyes are quite near the nasal organ,
in others farther back. In some they face laterally, in others upward.
As there is no nictating[33] membrane there is neither winking nor
the shedding of tears. Both jaws are, to some extent, movable and
provided with osseous teeth that are usually sharp and of a spike-
like form.
From the heads here presented the operculum,[34] or gill cover, is
removed to show those delicate respiratory organs. The branchiæ,
or gills, are situated at the sides of the head just back of the eyes,
and consist of numerous and very vascular[35] plates, arranged in
double fringe-like rows, fixed or attached at the base only, and so
constructed, in all respects, as to expose as much surface as
possible. These gills are covered with innumerable small blood
vessels, to which blood is pumped from the heart, there to receive
the needed supply of oxygen. The oxygen is obtained from what air
circulates through the water, and not by a decomposition of the
water, as some have supposed. For some species the modicum of
oxygen, thus obtained, seems insufficient, and they come to the
surface for more.

HEADS OF FISHES WITH GILL-COVERING REMOVED.

Notice another peculiarity. There being no neck or long gullet, the

principal digestive organs are packed in the cavity near the
capacious mouth, and this leaves the whole of the posterior tapering
part of the body for strong muscles, that can vigorously move the
caudal extremity from side to side as a propelling oar. The spine is
so jointed as to allow a free horizontal and but little vertical motion.
The bones of fishes are less compact than those of the higher
orders of animals, but quite elastic; and in some species small bones
are distributed through the flesh, giving additional firmness to
portions that lack muscular strength. As the peculiar breathing
apparatus of the fish is adapted to its element and mode of life, so is
every part and appendage of the whole structure. Adaptation reigns
through the whole. The elongated tapering body, its scaly covering
exquisitely adjusted, the material and position of the fins, all attest
the intelligent and beneficent purpose of the Creator.
The Elasmobranchii[36] (strap-gilled) have a cartilaginous
skeleton, rough skin, and uncovered gills. The Shark, the Saw-fish
and the Ray are representatives of this order.
The Ganoidei (enameled scales) formed one of the largest orders
in ancient geological history, but they have now but few
representatives, such as the Sturgeon, Gar-pike and Mud-fish. Their
characteristics are a skeleton not completely ossified, ventral fins
placed far back, and the tail heterocercal, that is, having the upper
lobe larger than the lower.
The Teleosti[37] (perfect bone) form the largest order, including
nearly all our common fishes. The characteristics of the order are an
osseous or bony skeleton; gills protected by a gill cover or
operculum; and an equally divided tail. Lowest among the Teleosts is
the common eel. It has an elongated, cylindrical, thick-skinned body,
and is destitute of ventral fins. There are many species, and they are
widely distributed, living in both salt and fresh water. One species is
electrical.[38] The cat fishes of this order have long threads hanging
from their jaws, and are noted for their peculiar methods of
protecting their young; one species is electric, having cells arranged
in layers over the body.
Other peculiar types are the Blind or Cave fishes,[39] living in the
waters of caves; the Lamp fishes which take their name from the
luminous spots arranged along the sides, and which are supposed to
light the recesses where they live, and the Flying fishes.
There are at least two genera, and more than thirty species of
“flying fish,” a name given to all those which have the pectoral fins
so large that they are sustained in short, seeming flights through the
air. They do not really fly, as they have not muscular power in their
fins to beat the air as birds do. But when extended the fins help bear
them up, and the impulse received at the start, sufficient to give
them the elevation they reach, may be supplemented by the use of
the caudal fin as in swimming. Some naturalists claim to have
noticed a movement of the other fins, but the preponderance of
testimony is that these are of service only as parachutes or the
wings of a paper kite. The Herring, Shad, Salmon, Pike, Perches,
Bass, Mackerel and Cod are valuable food fishes, belonging to the
Teleosts. The Sword fishes, in which the upper jaw is developed into
a long, sword-like projection, used as a weapon of defense and
offense, is an interesting member of the order. The Climbing fishes,
noted for being able to live out of water, and the Nest Builders,
which make homes from the weeds of the Sargossa Sea,[40] are
peculiar types.
Another curious Teleost is the Sea Horse. This peculiar animal has
its name (Hippocampus) from the shape of the head, that has some
resemblance to that of a horse. There is no other resemblance. The
short body, without legs, is covered with angular spinous plates. Its
fishy part is a long prehensile tail, but it has neither dorsal nor
caudal fins.
The last order of fishes is the Dipnoi, or Lung fishes. They are
characterized by the possession of two lungs, as well as gills. The
gills are used in respiration when the fishes are under water, but
when out of it, and burrowing in the mud as they often do, the lungs
are put into service. They are also known as mud fishes, from their
habits of encasing themselves in the mud.
 THE SEA HORSE.

End of Required Reading for May.

 HOW TO WIN.

BY FRANCES E. WILLARD,
President National W. C. T. U.

CHAPTER III.
But—as I was saying when the stern old gentleman was pleased
to interrupt me—I am to give you reasons why you are to cultivate
your specialty. And I claim, first (as has been implied already), that
you should do this because you have a specialty to cultivate. (This,
on the principle of the old cook book, which begins its “Recipe for
Broiling Hares,” with the straightforward exhortation: “First catch
your hare.”) The second reason is, because you will then work more
easily and naturally, with the least friction, with the greatest pleasure
to yourself and the most advantage to those around you. “Paddle
your own canoe,” but paddle it right out into the swift, sure current
of your strongest, noblest inclination. Thirdly, by this means you will
get into your cranium, in place of aimless reverie, a resolute aim.
This is where your brother has had his chief intellectual advantage
over you. Quicker of wit than he, far less unwieldy in your mental
processes, swifter in judgment, and every whit as accurate, you still
have felt, when measuring intellectual swords with him, that yours
was in your left hand, that his was in his right; and you have felt this
chiefly, as I believe, because from the dawn of thought in his sturdy
young brain, he has been taught that he must have a definite aim in
life if he ever meant to swell the ranks of the somebodies upon this
planet, while you have been just as sedulously taught that the
handsome prince might whirl past your door “’most any day,” lift you
to a seat beside him in his golden chariot and carry you off to his
castle in Spain.
And of course you dream about all this; why shouldn’t you? Who
wouldn’t? But, my dear friends, dreaming is the poorest of all
grindstones on which to sharpen one’s wits. And to my thinking, the
rust of woman’s intellect, the canker of her heart, the “worm i’ the
bud” of her noblest possibilities has been this aimless reverie; this
rambling of the thoughts; this vagueness, which when it is finished,
is vacuity. Let us turn our gaze inward, those of us who are not
thorough-going workers with brain or hand. What do we find? A wild
chaos; a glimmering nebula of fancies; an insipid brain-soup where a
few lumps of thought swim in a watery gravy of dreams, and, as
nothing can come of nothing, what wonder if no brilliancy of
achievement promises to flood our future with its light? Few women,
growing up under the present order of things, can claim complete
exemption from this grave intellectual infirmity.
Somehow one falls so readily into a sort of mental indolence;
one’s thoughts flow onward in a pleasant, gurgling stream, a sort of
intellectual lullaby, coming no-whence, going no-whither. Only one
thing can help you if you are in this extremity, and that is what your
brothers have—the snag of a fixed purpose in this stream of
thought. Around this will soon cluster the dormant ideas, hopes, and
possibilities that have thus far floated at random. The first one in the
idle stream of my life was the purpose, lodged there by my life’s best
friend, my mother, to have an education. Then, later on, Charlotte
Bronte’s “Shirley” was a tremendous snag in the stream to me.
Around that brave and steadfast character clustered a thousand new
resolves. I was never quite so steeped in reveries again, though my
temptations were unusual; my “Forest Home,” by a Wisconsin river,
offering few reminders to my girlish thought, of the wide, wide world
and its sore need of workers. The next jog that I got was from the
intellectual attrition of a gifted and scholarly woman who asked me
often to her home and sent me away laden with volumes of
Wordsworth, Niebuhr and the British essayists. Margaret Fuller Ossoli
was another fixed point—shall I not rather say a fixed star?—in the
sky of my thought, while Arnold of Rugby, to one who meant to
make teaching a profession, was chief of all. Well, is it possible that
any word I have here written may set some of you thinking—that’s
it, set you, a fixed purpose rather than a floating one—about a
definite object in life toward which, henceforth, you may bend a
steady, earnest gaze? I am not speaking of a thorough intellectual
training only. It is rather to the life-work, which only a lifetime can
fully compass, that I would direct your thoughts. Rather than that
you should fail to have a fixed purpose concerning it, I would that
your mental attitude might be like the one confided to me by a
charming Philadelphia girl, whose letter of this morning has the
following naïve statement:
“I feel such an aching in me to do or be something
uncommon, and yet a kind of awful assurance that I never
shall.”
Nor do I here refer to that general knowledge of household arts
which forms the sole acquirement set forth in the regulation
“Women’s Department” of the bygone age newspaper, which in
many localities remains in this like the boulder of a past epoch.
It was once thought to be a high virtue for women, no matter
how lofty in station or how ample of fortune, to do their own work
with the needle. Homer represents Penelope spinning, surrounded
by her maids, and classic art abounds with illustrations of like
character. But the virtues of one age often become the mistakes of
the next. When loom, needle and broom were woman’s only
weapons, she did well to handle them deftly, no matter what her
rank, for they were her bread-winning implements, and fortune has
been proverbially fickle in all ages. But men, by their “witty
inventions,” have perpetually encroached on “woman’s sphere.”
Eli Whitney, with his cotton gin, Elias Howe, with his sewing
machine, and a hundred other intricate-brained mechanics who have
set steel fingers to do in an hour what women’s fingers could not
accomplish in a year; all these have combined to revolutionize the
daily cares of the gentler sex. With former occupations gone, and
the world’s welcome ready when they succeed in special vocations
new to them, it becomes not only the privilege but the sacred duty
of every woman to cultivate and utilize her highest gift. There is no
more practical form of philanthropy than this, for every one who
makes a place for herself “higher up” leaves one lower down for
some other woman who, but for the vacancy thus afforded her in
the world’s close crowded ranks, might be tempted into paths of sin.
There is an army of poor girls wholly dependent, for a livelihood,
upon the doing of house work. They have no other earthly resource
between them and the poor house or haunt of infamy. There is
another class to whom an honorable support can come only by
sewing or millinery work. Whoever then fits herself for some
employment involving better pay and higher social recognition,
graduates out of these lower grades and leaves them to those who
can not so advance, has helped the world along in a substantial way,
because she has added to the sum of humanity’s well being.
To young women in wealthy homes, these considerations should
come with even greater convincing force. As David Swing has wisely
said to his own rich congregation:
“The rhetoric thrown at women of property for not doing ‘their
own work’ could only be useful in an age of fashionable idleness, but
in a busy age it is a part of nature’s law that what are called the
‘better classes’ shall leave for the poorer classes some labor to be
done, just as the Mosaic law left some sheaves in the field for the
gleaner. The world’s work is to be apportioned according to the need
and capability of its workers, and the higher order of power must not
encroach upon the task which nature seems to have set apart for
the employment and support of the less capable.”
Let it not be concluded that I have meant to speak lightly of the
intricate, skilled labor involved in making healthful and attractive that
bright, consummate flower of a Christian civilization—the home. I
have felt that this theme has been so often treated that it needed no
amplification at my hands, but I will add that, having been
entertained in scores of homes belonging to “exceptional women,”
“women with a career,” etc., my testimony is that for
wholesomeness, heartsomeness, and every quality that superadds
home-making to housekeeping, I have never seen their superiors,
and seldom, take them all in all, their peers. But as a rule, these
women have earned the “wherewithal” to make a home, by the
exercise of some good gift of brain or hand, and thus having been
enabled to put a proxy in the kitchen, they direct, but do not attend
to the minutiæ of their daily household cares.
Cultivate, then, your specialty, because the independence thus
involved will lift you above the world’s pity to the level of its respect,
perchance its honor. Understand this first, last, and always: The
world wants the best thing. It wants your best. It needs you as a
significant figure to give its ciphers value; to designate as an
example; to serve up in a eulogy, perchance to shine in the galaxy
by whose light alone its centuries maintain their places in the
firmament of history. I know this may strike you as contradiction, for
the paradox of paradoxes is this crotchety but kind, narrow-minded
but just old world in which you and I are cast away, like Æneas in
the domain of Dido. The effrontery of “Madame Grundy” passes all
comprehension, and would be laughable if it were not so sad. She
tells us women distinctly that we positively shall not do for society
the thing we can do best; she declares that if we attempt it we shall
be frowned down, and practically ostracised, if not utterly made
away with, and then, if we go right on and succeed, she trumpets
our names from sea to shore, showers us with greenbacks, and nods
her conventional old head with a knowing “I told you so.” And per
contra, while on one hand this same unreasonable old lady cripples
our attempts to succeed, on the other she snubs us for not doing so.
In fact, she is so poor a mathematician that she has never yet so
much as tried to learn the value of the “unknown quantity.” The
mute Milton is, to her, indeed “inglorious.” Her code of ethics
recognizes just one crime (not mentioned in the Decalogue), and it
is Failure. Her law is written on a single table—it is a table of stone—
and it reads thus: “Succeed and live; make shipwreck of success,
and die.”
And so, young friends, fold away your talents in a napkin if you
choose; the world will not openly reprove you. She will never urge
you to bring out your hidden treasure, but she knows right well
when you defraud her, and the relentless old tyrant will punish you,
with tireless lash, because you did not bring all your tithes into the
storehouse of the common good, because you lived “beneath your
privilege;” because, for yourself (which means for her), you did not
“covet earnestly the best gifts.” She will cut you on the public street
when she would have shown you all her teeth in smiles. She will
send poverty on your track, when you might have sat down at her
banquet an honored guest. Yes, the world wants the best thing; your
best, and she will smite you stealthily if you do not hand over your
gift. Now last, but not least (under the head of reasons for seeking
to know your true vocation as a human being), let me bring forward
the rationale of the bread-and-butter argument. In sooth, no writer
or speaker may omit it with impunity, if he would retire in good order
from an American audience. Briefly, then, your specialty, well
trained, is your best bread-winning implement, and she who earliest
grasps this, and who firmest holds it, comes off best in the race. “Be
not simply good, be good for something,” said Henry D. Thoreau. A
bright eyed girl of eighteen used to come to me on Friday evenings
to give me German lessons. To be sure, I have lived in Germany, and
she has never been out of Illinois, but then that language is not my
specialty, while it is hers. “How is it that though so young, you have
made yourself independent?” I inquired of her one day. Listen to the
reply: “My mother was always quoting this saying of Carlyle: ‘The
man who has a sixpence commands the world—to the extent of that
sixpence.’ I early laid this sentiment to heart. Besides, when I was
fifteen years old, I heard a sermon on the text: ‘This one thing I do.’
Being of a practical turn of mind, I made an application of which the
preacher, perhaps, had no intention. I thought, why not in everyday
affairs as well as in religion do one thing well, rather than many
things indifferently, and in that way secure the magic sixpence of
Carlyle! My father was a rich man then, but I resolved to prepare
myself to teach the German language, of which I was very fond, by
way of a profession. When the Chicago fire came we lost our
property, but I discovered that I could not only support myself, but
help my father to many a convenient sixpence, because, in
prosperous days I had forearmed myself with a cultivated specialty.”
As she told me this, I thought how, from widely different premises
and conditions in life young people may reach similar conclusions.
For instance, on the top of the great St. Bernard, I said to the
“Hospitable Father,” a noble young monk, “How is it that you, so
gifted and well taught, are spending your life away up here among
eternal snows?” And I shall never forget his look of exaltation as he
simply answered, “’Tis my vocation.”
After all, this is the vital question: With what sort of a weapon will
you ward off the attacks of the blood-hound Poverty, which Dame
Fortune is pretty sure to let on everybody’s track sooner or later, that
she may try his mettle, and learn what manner of spirit he is of? In
times like these, when men’s hearts are failing them for fear, when
riches are saved the trouble of “taking to themselves wings” by the
faithless cashiers and bookkeepers who are adepts at furnishing
these flying implements, and, above all, when labor is coming to be
king, the question “What will you do?” has fresh significance.
Remember, going forth from the uncertain Eden of your dreams, into
the satisfying pleasures of honest, hard work, “the world is all before
you, where to choose.” Will you share some other woman’s home,
and help her make it beautiful? No task more noble or more needed
awaits the thoughtful worker of to-day. The world exists but for the
sake of its homes. Will you bestow your hand upon some fine
æsthetic industry, as drawing, designing, engraving, telegraphing,
phonographing, photographing? Will you be an architect? a printer?
an editor? Will you enter one of the three learned professions?
Braver women than you or I have won a foot-hold for us in each of
them; as to the brain-hold, that is our affair. I will not now pursue
the question further. Only the “Cyclopædia of Woman’s Occupations”
(a book I recommend to your attention) can exhaust it, and with it
exhaust you and the world’s work, too, for that matter!
After all, it doesn’t so much signify what you may do as that you
do it well, whatever it may be. For the value of skilled labor is
estimated on a democratic basis, nowadays. President Eliot, of
Harvard University, the cook in the Parker House restaurant, and
Mary L. Booth, who edits Harper’s Bazar, each receive $4,000 per
year.
Think a moment. Will you be led to say: “The good old ways are
good enough for me,” and so drop into the swollen ranks of
teacherdom, or rattle awhile on a martyrized piano, and then set up
for a musician, though you have not a particle of music in throat or
finger-tips? Or will you stay at home and let papa support you until
you grow tired of doing nothing and expecting nothing, and proceed
to marry some man whom you endure rather than love, just to get
decently out of your dilemma?
Nay, I do you injustice. Few girls who breathe the free air of our
western prairies will be so cowardly. I may not construct your
horoscope, but this much I will venture—that when you marry, no
matter what you find, you will seek not a name, behind which to
cover up the insignificance of your own; not a “good provider,” to
feed and clothe one who has learned how to feed and clothe herself;
not a “natural protector,” to shield you in his plaidie, the gallant,
gallant laddie, from the cauld, cauld blast; but you will seek (and
may heaven grant that you shall find) that rarest, choicest, most
elusive prize of man’s existence, as of woman’s; one which—
mournfully I say it—the modern marriage is by no means certain to
involve, namely, a mate. At this juncture, shrewd mater familias
whispers to pater: “That’s the first orthodox word she’s said.” Some
youth throws down the magazine and mutters to himself: “There, I
knew it would come to this! Look at the absurdity of these women!
Why, they preach up all sorts of trades and professions, and then
they come back, at last, to the ‘good old way’ they have forsaken,
and advise every young lady to get a situation in a school of one
scholar, and her board thrown in.”
Meanwhile, heroic Hypatia sits near by, and “musing in maiden
meditation, fancy free,” on a “career,” murmurs within herself, “To
this complexion must it come at last!”
Peace, peace, good friends! This seeming inconsistency is readily
explained. In this century, when the wage of battle has cost our land
an army of her sons, when widows mourn, and unwedded
thousands are forced to meet the hard-faced world (from which
rose-water theorists would shield them), America is coming to the
rescue of her daughters! For the nearer perfect—that is, the more
Christian—a civilization has become, the more carefully are the
exceptional classes of society provided for. All our philanthropic
institutions under state or private patronage illustrate this. In less
enlightened days, your ideal woman composed the single, grand
class for which public prejudice set itself to provide. She was to be
the wife and mother, and she was carefully enshrined at home. But,
happily, this is the world’s way no longer. The exceptions are so
many, made by war, by the thousand misunderstandings and cross-
purposes of social intercourse, by the peculiar features of the
transition period in which we live, by the absurdly extravagant
customs of our day, and the false notions of both men and women—
that not to provide for them would be a monstrous meanness, if not
a crime. And the provision made in this instance is the most rational,
indeed, the only rational one which it is in the power of society or
government to make for any save the utterly incapable, namely: a
fair chance for self-help. Nor (to pursue the line of our argument still
further) can we forget that skeleton hand which, in utter disregard
of “the proprieties” in destiny’s drama, thrusts itself so often into the
charmed domestic circle, and snatches the beloved “provider” away
forever, while it sets gaunt famine by the fireside in his stead? Can
we forget that, in ten thousand families, wives are this moment
waiting in suspense and agony the return of wretched husbands to
homes made hideous by the drunkard’s sin—wives whose work of
brain or hand alone keeps their children from want, now that their
“strong staff is broken, and their beautiful rod?” There are delicate
white fingers turning the page on which I print these words, that will
never wear the marriage ring; there are slight forms bending over
my friendly lines, which, not far down the years, will be clothed in
widow’s weeds. Alas, there are as surely others, who, when they
have been wooed and won, shall find that they are worse than
widowed. And what of these three classes of women, sweet and
helpless? Clearly, to all of them I am declaring a true and blessed
gospel, in this good news concerning honest independence and
brave self-help! Clearly, also, no one is wise enough to go through
the assembly of my readers, and tell me who, in future years, shall
need a bread-winning weapon with which to defend herself and
perchance, also, the helpless ones between whom and the world
there may be no arm but hers. But it is a principle in public as well
as private economy, that the wisest foresight provides for the
remotest contingency, and thus, in its full force, all that I have been
saying applies to every woman who may read this article on “How to
Win.” Suppose that many of you, dear girls, are destined to a downy
nest, instead of a strong-winged flight. What then? Will the years
spent in making the most of the best powers with which God has
endowed you be worse employed than if you had given them to
fashion and frivolity? Those “ad interim” years which separate the
graduate’s diploma from the bride’s marriage certificate, can they
possibly be invested better than in the acquisition of some useful
trade or dignified profession? And then, aside from this, I would help
the youngest of you to remember (even in the bewildered years of
her second decade) what noble Margaret Fuller said: “No woman
can give her hand with dignity, or her heart with loyalty, until she
has learned how to stand alone.” It is not so much what comes to
you as what you come to, that determines whether you are a winner
in the great race of life. Never forget that the only indestructible
material in destiny’s fierce crucible is character. Say this, not to
another—say it to yourself; utter it early, and repeat it often: Fail me
not, thou.
 THE LIFE OF MINERALS.

BY M. J. THOULET.

Translated for The Chautauquan from the Révue Scientifique.

The definition given to-day to mineralogy places it among the

exact sciences. Long continued study has shown that it possesses all
the inflexibility of chemistry, of physics, of mathematics. The work of
one making a specialty of this subject is similar to that of a
millwright who collects the different pieces, forged and cast and
prepared in various ways by other workmen, and arranges them all
in their proper relations to one another, joins them, and forms the
mill with all its complication of machinery in good order, ready to run
without friction, without jar. The mineralogist gathers up the facts
and theories wrought out by workers in other fields of science,
studies their variations, their agreements and connections,
demonstrates their presence and their union in inorganic bodies, and
sums up and announces all the results of his labor in the form of
laws which shall be exact rules for events past, present, and future;
for a science incapable of foreseeing and foretelling is not a true
science. Mineralogy is not chemistry, nor physics nor mathematics,
any more than the millwright is the smith or the smelter. It is a
distinct science pursuing a particular aim, and which, although
borrowing from other sciences certain of their results, nevertheless
possesses its own individuality. It might be said to be a direct
application of these three sciences, together with geology, to the
study of the life of minerals.
I have just used a very significant expression: The life of
minerals. Others have used it before me. “Not only do stones live,
but they suffer from sickness, from old age, and death,” wrote
Cardan in the sixteenth century. And he was right. Eternal matter
performs an eternal cycle; the incessant variations which it
experiences; the movement which is never arrested, which from
modification to modification, from transformation to transformation
draws it along without a single moment of rest; the continual births
and deaths and resurrections are life. Every man, every animal,
every plant, and every stone obey without any power to resist, and
they are all borne along without relaxation or repose toward a vortex
whose beginning and ending are concealed within the shadows of
eternity. There is no difference between the mineral and vegetable,
or animal. Inorganic life is identical with organic life, varying only in
degree.
From the moment which we call birth, that is to say at the
commencement of one of these periods of transformation, our eyes
see, hour by hour, moment by moment, the living being develop.
The atoms entering into its construction seek like atoms to which
they ally themselves, and molecules combine with other molecules.
What matters it about the form of being? Simple or complicated, the
law is the same, and it is obeyed. The individual appears with its
own chemical constitution, its own form and look, its own variations,
all decided under its predetermined conditions. Among these
conditions a single one is variable, but the equilibrium is constantly
preserved; the individual changes from time to time in its own
appointed way, but it never ceases to exist.
In the same manner as organic life bears the impress of its
surroundings, so do minerals submit themselves to external
influences. The one perhaps is more frail, more delicate, less able to
resist, more susceptible to impressions; the natural forces of the
other, more powerful because they are simpler, yield less readily to
circumstances. Both alike are forced to take their part in the great
concert of forces in which they fare only infinitely feeble notes; both
alike are influenced by the majestic assembly of powers which act
upon them, and upon which they, in turn, also act, conformably to
one of the first laws of matter, that of equality between action and
reaction.
Let us take any mineral whatever and subject it to a constantly
increasing temperature. We notice first that it undergoes a change of
form. Cease the application of heat and it will gradually resume its
former shape. Let us heat it again, and more intensely. All the
properties of the matter which constitutes it become changed, some
quickly, some slowly, and it is incapable now of taking back its first
appearance. Its crystalline form is different, and its mechanical
elasticity, its hardness, and its electric properties; even its color is
changed. We will still increase the heat. The molecules disperse,
following certain directions, and, following others, gather themselves
together. Suddenly a limit, varying according to the chemical
composition, the crystalline type, or the pressure, is broken; the
solid, beginning to melt, becomes a liquid. Heat it still higher and we
shall see new phenomena appearing, volatilization and dissolution.
Another limit is passed and the atom, becoming isolated, is free
henceforward from the laws of chemistry, and must now obey laws
yet unknown, the task of discovering and formulating which is
awaiting some worker in the realms of physics or mechanics.
The dissolution of a mineral, is it not death? Every abrupt limit of
all the powers of a body is death, and all death precedes a
resurrection.
As a child, which at the same moment when it opens its eyes
upon the light and utters its first cry, begins already to die, so with
the mineral scarcely formed, death commences. Feldspar, which
constitutes in great part the soil pressed by our feet, under the
influence of air and of water, of drought by day and dews by night,
of the heat of summer and the cold of winter, of all agents
mechanical, chemical and physical acting upon it, is reduced to its
elements by a series of almost insensible transformations. Its
fragments are broken to still finer bits, and when they have become
dust disintegration still goes on, and gradually the silicon, the
aluminum, the iron, the lime, the magnesium, and the potassium
which composed them form clay. The iron oxydizes, the silicon
separates itself, is dissolved by rain and carried off by the streams.
Each element then enters into a new combination; sometimes it
again becomes part of a stone; sometimes it helps to form the
structure of a plant; sometimes that of a man. Where can birth,
signifying the beginning of all existence, be placed, or where shall
we find any real death? I perceive only periods of life.
Of old, naturalists made more frequent and much stronger
affirmations than they do to-day. Confidence in self is the property of
youth; maturity learns to doubt, which is the beginning of wisdom,
provided that it does not remain content, but rather compels man to
seek with increased ardor the truth which seems to fly from him.
The ancients placed between the animal and the vegetable limits
which in reality did not exist. Up to the present time limits of the
same nature have been set between organic and inorganic life. But
in proportion as we examine minerals we shall see the differences
disappear and the resemblances increase. Man is born of parents;
the whole animal and vegetable worlds are perpetuated in obedience
to the laws of reproduction, each after his own kind. It was this
absolute identity between parent and offspring that separated
distinctly the other kingdoms from the mineral; but recently a
scientist has discovered that the same fixed law is established in this
department of life also. M. Gernez prepared a solution consisting of
octahedral borax in five equivalents of water, and rhomboidal borax
in ten equivalents of water. The two bodies, excepting their
proportion of water, had the same chemical composition. The liquid,
treated with suitable precaution, remained perfectly limpid, and he
could place in it fragments of all imaginable substances, without
causing it to give rise to any remarkable phenomena. But when even
an infinitely small crystal of octahedral borax was dropped into it,
the temperature rose, and in a few minutes all the octahedral borax
contained in the solution took the crystalline form. Meanwhile, the
rhomboidal borax was held in solution, and in order to crystallize it in
its turn, there was needed only the contact of a rhomboidal crystal.
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com