Coding Theorems of Classical and Quantum

Information Theory Second Edition Parthasarathy

pdf download

https://ebookname.com/product/coding-theorems-of-classical-and-
quantum-information-theory-second-edition-parthasarathy/

Get Instant Ebook Downloads – Browse at https://ebookname.com

Instant digital products (PDF, ePub, MOBI) available
Download now and explore formats that suit you...

Classical and Quantum Information Theory An

Introduction for the Telecom Scientist 1st Edition
Emmanuel Desurvire

https://ebookname.com/product/classical-and-quantum-information-
theory-an-introduction-for-the-telecom-scientist-1st-edition-
emmanuel-desurvire/

Fundamentals of information theory and coding design

1st Edition Roberto Togneri

https://ebookname.com/product/fundamentals-of-information-theory-
and-coding-design-1st-edition-roberto-togneri/

Quantum Information Theory and the Foundations of

Quantum Mechanics 2nd Edition Christopher G. Timpson.

https://ebookname.com/product/quantum-information-theory-and-the-
foundations-of-quantum-mechanics-2nd-edition-christopher-g-
timpson/

Principles of Cell Proliferation 1st Edition J. R.

Heath

https://ebookname.com/product/principles-of-cell-
proliferation-1st-edition-j-r-heath/
Lead Acid Battery Technologies Fundamentals Materials
and Applications Electrochemical Energy Storage and
Conversion 1st Edition Joey Jung (Editor)

https://ebookname.com/product/lead-acid-battery-technologies-
fundamentals-materials-and-applications-electrochemical-energy-
storage-and-conversion-1st-edition-joey-jung-editor/

The Private Lives of Trees 1st Edition Alejandro Zambra

https://ebookname.com/product/the-private-lives-of-trees-1st-
edition-alejandro-zambra/

Energy Capitals Local Impact Global Influence 1st

Edition Joseph A. Pratt

https://ebookname.com/product/energy-capitals-local-impact-
global-influence-1st-edition-joseph-a-pratt/

Weight Gain During Pregnancy Reexamining the Guidelines

1 Har/Com Edition Committee To Reexamine Iom Pregnancy
Weight Guidelines

https://ebookname.com/product/weight-gain-during-pregnancy-
reexamining-the-guidelines-1-har-com-edition-committee-to-
reexamine-iom-pregnancy-weight-guidelines/

Warhammer Fantasy Roleplay 4th Edition Dominic Mcdowall

https://ebookname.com/product/warhammer-fantasy-roleplay-4th-
edition-dominic-mcdowall/
The Cinematic Whitechapel Documents of Contemporary Art
Campany

https://ebookname.com/product/the-cinematic-whitechapel-
documents-of-contemporary-art-campany/
~ TEXTS Ai'iD READINGS 45
~ IN j\IATHEMATICS

Coding Theorems of Classical and

Quantum Information Theory
Second Edition

~HINDUSTAN
U UlJUBOOKAGENCY
TEXTS AND READINGS
IN MATHEMATICS
45
Co ding Theorems of
Classical and Quantum
Information Theory
Second Edition
Texts and Readings in Mathematics

Advisory Editor
C. S. Seshadri, Chennai Mathematical Institute, Chennai.

Managing Editor
Rajendra Bhatia, Indian Statistical Institute, New Delhi.

Editors
V. Balaji, Chennai Mathematical Institute, Chennai.
R. B. Bapat, Indian Statistical Institute, New Delhi.
V S. Borkar, Tata Inst. of Fundamental Research, Mumbai.
Probal Chaudhuri, Indian Statistical Institute, Kolkata.
 Coding Theorems of
Classical and Quantum
Information Theory
Second Edition

K. R. Parthasarathy
Indian Statistical Institute
NewDelhi

[Ldgl@ HINDUSTAN
U U!J UBOOK AGENCY
Published by

Hindustan Book Agency (lndia)

P 19 Green Park Extension
New Delhi 110 016
lndia

email: [email protected]
www.hindbook.com

Copyright © 2013, Hindustan Book Agency (India)

No part of the material protected by this copyright notice may be

reproduced or utilized in any form or by any means, electronic or mechanical,
including photocopying, recording or by any information storage and retrieval
system, without written permission from the copyright owner, who has also
the sole right to grant licences for translation into other languages and
publication thereof.

All export rights for this edition vest exclusively with Hindustan Book Agency
(India) . Unauthorized export is a violation of Copyright Law and is subject
to legal action.

ISBN 978-93-80250-41-0 ISBN 978-93-86279-59-0 (eBook)

DOI 10.1007/978-93-86279-59-0
 To

Shyama
Preface

The logarithmic connection between entropy and probability was first enun-
ciated by L.E. Boltzmann (1844-1906) in his kinetic theory of gases. His
famous formula for entropy S is S = k log W (as engraved on his tombstone in
Vienna) where k is a constant and W is the number of possible microstates
corresponding to the macroscopic state of a system of particles in agas.
Ignoring the constant k and replacing log W by -log P(E) where P(E) is the
probability of an event E in the probability space (n, F, P) of a statistical
experiment, C. E. Shannon (1916- 2001) looked upon -logP(E) as a measure
of the information gained about the probability space from the occurrence of
E. If X is a simple random variable on this probability space assuming the
values al , a2, ... , ak from a finite set with P( X = aj) = Pj for each j then the
famous Shannon entropy H(X) = - Lj Pj log Pj is the expected information
about (n, F, P) gained from observing X. Cent red around this idea of
entropy a mathematical theory of communication was woven by Shannon in
a celebrated pair of papers in the 1948 volume of the Bell System Technical
Journal. Here Shannon established two fundamental coding theorems about
the optimal compressibility of a text in its storage and the optimal capacity of
a channel in communicating a text after encoding.

The modern approach to information theory is to view a text in any al-

phabetic language as a finite time realization of a stochastic process in discrete
time with values in a finite set (calIed alphabet) and consider the quantity
-~ log P(XO , Xl, ... , Xn-l) as the rate at which information is generated by the
text Xo , Xl, ... , Xn-l during the per iod [0, n -lJ. Under fairly general conditions
this rate exhibits an asymptotic stability property as n becomes large. Through
the papers of B. Mcmillan, A. Feinstein, L. Breiman, J. Wolfowitz and others
it is now known that an appeal to this stability property enlarges the scope
of Shannon's coding theorems. This gets enriched further by exploiting the
Kryloff-Bogoliouboff theory of disintegrating an invariant probability measure
into its ergodic components. The first three chapters of this little book are
devoted to Shannon's coding theorems and their enriched versions. However,
we have not touched upon the coding theorems in their most general form as
presented in the book of Te Sun Han [14J .
 A decade after the appearance of Shannon's famous work, A. N. Kol-
mogorov (1903- 1987) demonstrated, rat her dramatically, how the not ion of
the expected rate of generation of entropy or information assumes an intel-
ligence of its own and yields a nonspectral invariant for the classification of
dynamical systems. Since very little extra effort is involved in presenting this
beautiful work I have taken the liberty of including it as a small digression.
In 1932, while laying the mathematical foundations for quantum me-
chanics, John von Neumann (1903- 1957) introduced the fruitful notion of en-
tropy for the state of a quantum system. If p is the density operator of the
state of a quantum system then its von Neumann entropy S(p) is defined
by S(p) = -Tr p log p. Through the work of A. S. Holevo, B. Schumacher,
W . D. We s tmoreland and others as outlined in the book of Nielsen and Chuang
[24] the reader can recognize the role of von Neumann entropy in attempts to
formulate and establish quantum versions of the coding theorems of Shannon
when classical messages are encoded as quantum states and decoding is done
by generalized measurements. Our last and the fourth chapter is devoted to
a self-contained account of these coding theorems in the quantum avatar as
described in the elegant work of A.Winter in his 1999 paper [48].
A large part of the first three chapters of this book does not use anything
more than Chebyshev's inequality. The ergodic theorem, martingale theorem
and decomposition of an invariant probability measure into its ergodic compo-
nents are used in arriving at the more sophisticated versions of the classical
coding theorems. The last chapter demands not hing more than a knowledge of
operators in a finite dimensional Hilbert space.
The preseot exposition has evolved through the courses of lectures I had
given at the Indian Statistical Institute, Calcutta in 1961, the Tata Institute of
Fundamental R esearch, Mumbai in 2001 and 2002, the Institute of Mathemat-
ical Sciences, Chennai in 2001 and 2005, the Ramanujan Institute of Advanced
Study in Mathematics at the University of Madras in 2005 and Chungbuk Na-
tional University, Cheongju, Korea in 2005. I am grateful to C. R. Rao who
suggested to me in 1959 the study of information theory for my PhD thesis and
J. Radhakrishnan, R. Parimala, R. Balasubramanian, M. Krishna, V. Arvind,
S. Parvathi , K. Parthasarathy, V. Thangaraj and Un Cig Ji who were instru-
mental in organising these lectures in a congenial atmosphere. I thank Anil
Shukla for his elegant TEX of my notes with patience in spite of my repeated
requests for changes and corrections. Thanks to the careful proof-reading by P.
Vanchinathan a significant control over the number of grammatical, typograph-
ical and TEX errors has been exercised. The support given by my colleagues at
the Delhi Centre of the Indian Statistical Institute is gratefully acknowledged.

Indian Statistical Institute K. R. Parthasarathy

Delhi Centre January 2007
New Delhi - 110016
India
Preface to the revised edition

The essential feature of the revised edition is the inclusion of a new chapter
devoted to the Knill-Laflamme theory of quantum error correction and its
consequences in the construction of t-error correcting quantum codes. Our
approach is based on the unification of classical and quantum error correcting
codes through imprimitivity systems for finite group actions.

Many typographical error corrections and some minor changes have been
made in the text of the first edition.

I have greatly benefited from discussions with V. Arvind and Harish

Parthasarathy. Ajit Iqbal Singh has rendered valuable help in carefully reading
the manuscript and suggesting many improvements. Anil Kumar Shukla has
Texed the revised manuscript showing tremendous patience in fulfilling my re-
quests for repeated changes in the text. The continued support of my colleagues
in the institute has enabled the completion of this revision in reasonable time.
To all of them I express my sincere thanks.

Indian Statistical Institute K. R. Parthasarathy

Delhi Centre September 2012
New Delhi - 110 016
India
Contents

Preface vii

Preface to the revised edition ix

1 Entropy of Elementary Information Sources 1

1.1 Uniquely decipherable and irreducible codes. 1
1.2 The Huffman code . . . . . . . . 9
1.3 Entropy and conditional entropy . . . 12
1.4 Entropy and size . . . . . . . . . . . . 19
1.5 Shannon's characterization of entropy 23

2 Stationary Information Sources 27

2.1 Language as a stochastic process . . . . . . . . . . . . . . .. 27
2.2 The ergodie theorem and the martingale convergence theorem 29
2.3 The Shannon- McMillan- Breiman theorem . 34
2.4 Noiseless coding theorem for ergodie sources 41
2.5 An integral r epresentation. . . . . . . . . . . 44
2.6 The noiseless coding theorem . . . . . . . . . 48
2.7 The Kolmogorov- Sinai entropy of a dynamical system 54

3 Communication in the Presence of Noise 61

3.1 Elementary communication channels . 61
3.2 Converse of the coding theorem 69
3.3 Latin square channels . . . . . . . . . 75
3.4 Sequences of channels . . . . . . . . . 79
3.5 Ergodie and stationary capacities of stationary channels . 85
3.6 Ergodicity of stationary channels . . 86
3.7 Latin square channels visited again. 89

4 Quantum Coding Theorems 93

4.1 Classical and quantum probability 93
4.2 The Dirac notation . . . . . . . . 97
4.3 Elementary quantum information sources 99
4.4 Some properties of von Neumann entropy 103
xii Contents

4.5 Elementary classical-quantum communication channels 116

4.6 Entropy typical projections . . . .. . 119
4.7 Two elementary inequalities. . . . . . . . . . 122
4.8 The greedy algorithm for cq-channels . .. . 124
4.9 The coding theorem for product cq-channels 127

5 Quantum Error Correction 135

5.1 A model of noise and the Knill-Laflamme theorem . . ... 135
5.2 A quantum circuit for the Knill-Laflamme theorem 141
5.3 Imprimitivity systems and error correcting quantum codes 145
5.4 t-error correcting quantum codes . . . .. . . . . . . . . . 163

Bibliography 111

Index 115
Chapter 1

Entropy of Elementary
Information Sources

1.1 Uniquely decipherable and irreducible codes

We begin with an elementary analysis of maps from a finite set into the free
semigroup generated by another finite set and develop a terminology appropri-
ate to information theory.
Consider any finite set A of cardinality a denoted as #A = a. We say
that A is an alphabet of size a and call any element x in A as a letter from the
alphabet A. Any element W = (Xl, X2, .. . , xn ) in the n-fold cartesian product
An of copies of Ais called a word of length n, the latter denoted by l(w). It is
customary to express such a word as w = XIX2 ... Xn by dropping the brackets
and commas. Denote

U Ar
00

S(A) =
r=l

and for any WI = XIX2 ... Xnl E Anl, W2 = YIY2 ... Yn 2 E An 2 define the product
word WIW2 by WIW2 = XIX2.·· Xnl YIY2· .. Yn 2· Thus l(WIW2) = l(wI) + l(w2).
Clearly, this multiplication is associative.1t makes S(A) a semigroup without an;
identity element. We call S(A) the free semigroup or word semigroup generated
by the alphabet A.
Let A, B be alphabets of sizes a, b respectively. A one-to-one (or injective)
map f : A ~ S(B) is called a code with message alphabet A and encoding
alphabet B. When B is the two point set {O, I} such a code f is called a binary
code. Any word in the range of a code f is called a basic code word. Start with
a code f : A ~ S(B) and extend it uniquely to a map j : S(A) ~ S(B) by
putting
2 Chapter 1. Entropy of Elementary Information Sources

for any word w = XIX2 ... X n in S(A) . Then f is said to be a uniquely deci-
pherable code if its extension j is also one to one. The code f is said to be
irreducible if for any two letters x and y in A , f(y) #- f(x) and f(y) cannot
be expressed as f(y) = f(x)w for any word w in S(B). A simple examination
shows that an irreducible code is uniquely decipherable.
We shall now establish a necessary condition for a code f : A -+ S(B) to
be uniquely decipherable.

Theorem 1.1.1 (Sardinas and Patterson [40]) Let A , B be alphabets of sizes

a, b respectively and let f : A -+ S(B) be a uniquely decipherable code. Then

L b-l(f(x» :::; 1. (1.1.1)

xEA

where l(w) denotes the length of the word w.

Proof. Let

L max {l(f(x)) I xE A},

er = #{xll(f(x))=r} .

Then the left hand side of (1.1.1) can be expressed as

L b-l(f(x» L L
L
b-l(f(x»
xEA r=l x :l(f(x»=r

r=l

where P is the polynomial defined by

L
P(z) = Lerz r .
r=l

Define
N(k) = # {j(w) Iw E S(A), l(j(w)) = k} ,
the cardinality of the set of all code words of length k. Clearly, N(k) :::; bk for
k = 1,2, .. . Thus the power series

L N(k)zk
00

F(z) = 1 +
k=l

converges to an analytic function in the open disc {z I Izl < b- 1 } . Introduce

the convention that N(O) = 1 and N(k) = 0 if k < O. Since every code word
1.1. Uniquely decipherable and irreducible codes 3

J(w) of length k 2: 2 can be expressed as ](W1)!(X) for some W1 E S(A) and

letter x E A where l(](wd) = k - rand l(J(x)) = r for some 1 :'S r :'S L it
follows that

N(k) = N(k - 1)C1 + N(k - 2)C2 + ... + N(k - L)CL if k 2: 1.

Multiplying by zk on both sides of this relation and summing over k = 1, 2, ...

we get F(z) -1 = F(z)P(z). Thus F(z) = (1- p(z))-l is analytic in the open
disc {z I Izl < b- 1 } . In other words the polynomiall - P(z) has no zeros in
the disc {z I Izl < b- 1 }. We also have 1 - P(O) = 1. Thus the real polynomial
1- P(t) in the real variable tremains positive in [0, b- 1 ). Hence 1- P(b- 1 ) 2: 0
which is same as the inequality (1.1.1).
o
Our next result is a converse of Theorem 1.1.1.

Theorem 1.1.2 Let m(x), x E A be a positive integer-valued function satisfying

L b-m(x) :'S 1 (1.1.2)

xEA

where b = #B. Then there exists an irreducible (and hence uniquely decipher-
able) code! : A -t S(B) such that m(x) = l(J(x)) '<I xE A.

Proof. Define

L maxm(x),
xEA
Ar {x I m(x) = r},
Cr #A r , 1:'S r :'S L.
Then A = UrA r is a partition of A into disjoint sets and Cl + C2 + ... + CL = a,
the size of A. Then (1.1.2) can be expressed as

This implies

Thanks to the first inequality above we can and do select a subset Sl c B c

S(B) such that #Sl = Cl :'S b. Suppose we have selected subsets Sj c Bj c
S(B) such that #Sj = Cj and no word in Sj is an extension of any word in
Sl U S2 U··· U Sj-1 for j = 2, ... , r - 1. The number of words in Br which are
not extensions of any word in Sl U S2 U ... U Sr-1 is equal to br - C1br-1 -
c2br - 2 - ... - Cr-1b 2: Cr· Thus we can and do select a subset Sr c B r c
S(B) such that #Sr = Cr. We continue this procedure till we reach SL. Then
4 Chapter 1. Entropy of Elementary Information Sources

U~=l Sr c S(B) is a subset of words in which no word is an extension of another

and # U~=l Sr = Cl + C2 + .. . +CL = a. Let now f be any bijection from A
onto U~=ISr. Then f is an irreducible code with the required properties. 0

Remark 1.1.3 When b = 2, the inequality (1.1.2) is known as Kraft's inequality

in the computer science literature.
The proofs of Theorem 1.1.1 and Theorem 1.1.2 yield a necessary and
sufficient condition for the existence of a uniquely decipherable code when the
lengths of the basic code words are given. However, the choice of such a uniquely
decipherable code is far from unique. In order to narrow down the choice of a
uniquely decipherable code it is necessary to introduce an optimality criterion
for the code by examining the statistics of frequencies with which the different
letters of the alphabet A appear when a language with the alphabet A is used
to write long texts. With this aim we introduce a formal definition.

Definition 1.1.4 An elementary information source (EIS) is a pair (A, J.L) where
Ais an alphabet and J.L is a probability distribution on A, Le., J.L : A -+ [0,1] is
a map satisfying E J.L(x) = 1.
xEA
As an example we may consider A to be the union of the set of all 26 let-
ters a, b, . . . ,Z of the English language, the set of all punctuation marks like full
stop ., comma , , question mark ?, . . . etc and a symbol for the 'space' between
successive words. By making an analysis of the frequencies with which letters
from A appear in different pages of books one can construct a model distribu-
tion J.L so that the EIS (A, J.L) is an approximation of the English language. Such
a model can be constructed for any language after a suitable alphabetization
of its script.
Let now (A, J.L) be an EIS and let B be an encoding alphabet with #A = a,
#B = b. Consider any uniquely decipherable code f : A -+ S(B). Then l(J(x))
considered as a function of x E A is a positive integer-valued random variable
on the probability space (A, J.L) with expectation

Z(J) = lEl(J(.)) = L: l(J(x))J.L(x)

xEA

which is called the mean code word length of the code f. Write
L(J.L) = min{I(J) I f : A -+ S(B) is a uniquely decipherable code} .

In view of Theorem 1.1.1 and Theorem 1.1.2 we have

L(J.L) = min {L: m(x)J.L(x) Im : A -+ Z+, L:

xEA xEA
b-m(x) :::; 1} (1.1.3)

where Z+ = {1, 2, 3, ... }. It is desirable to construct a uniquely decipherable

code f : A -+ S(B) for which Z(J) = L(J.L), Le., a code which has the least mean
codeword length.
1.1. Uniquely decipherable and irreducible codes 5

Our next result provides a good estimate of L(J-l).

Theorem 1.1.5 Let (A, J-l) be an EIS and let b be the size of the eneoding
alphabet B. Then

- EXEA J-l(x) 10gJ-l(x) L() - E XEA J-l(x) 10gJ-l(x)

log b S J-l < log b + 1. (1.1.4)

Proof. Let m : A ~ Z+ be a map satisfying the inequality EXEA b-m(x) S 1.

Define the probability distribution v on A by
b-m(x)
v(X) = =---~
'" b-m(y)·
L...yEA

Write T = EyEA b-m(y) so that T S 1. We have

L m(x)J-l(x) = (log b)-l {-lOg T - L J-l(x) log V(X)} . (1.1.5)

xEA xEA

Without loss of generality we assume that J-l(x) > 0 V x. We write

- LJ-l(x)logv(x)
xEA
= - LJ-l(x)logJ-l(x) -log
xEA
rr
xEA
V(X)){l(X)
(~
J-l
Sinee a weighted geometrie mean does not exeeed the eorresponding weighted
arithmetic mean we have

rr
xEA
(V(X)){l(X) S L
J-l(x) xEA
v(x) J-l(x)
J-l(x)
= 1.

Thus
-L J-l(x) log v(x) 2': - L J-l(x) 10gJ-l(x).
xEA xEA

Sinee T S 1 it now follows from (1.1.5) that

L m(x)J-l(x) 2': -(log b)-l L J-l(x) log J-l(x)

xEA xEA

whieh proves the left hand part of the inequality (1.1.4).

To prove the right hand part of (1.1.4) eonsider the unique positive integer
m(x) satisfying
m(x) - 1 < -logJ-l(x) S m(x)
log b
for eaeh xE A. Onee again we assume that J-l(x) > 0 V x. Then

xEA x
6 Chapter 1. Entropy of Elementary Information Sources

Hence by Theorem 1.1.2 there exists an irreducible (and hence uniquely deci-
pherable) code f : A -+ 8(B) for which l(f(x)) = m(x) V x. Hence

Z(f) = L m(x)p,(x)
xEA

< ~
L...J
(1 _ log p,(X)) p,(x)
log b
xEA
- LXEA p,(x) logp,(x)
= . log b + 1,
proving the right hand part of (1.1.4). o
Remark 1.1.6 We can express the inequality (1.1.4) as

p,(x) log2 p,(x) p,(x) log2 p,(x)

- LXEA
< L()
p, <
--===.:.,:-'--,-:--:...=.-'--'--'-
- LXEA
+ 1. (1.1.6)
log2 b - log2 b
The special case of the binary alphabet B = {O, I} with b = 2 is of great
practical importance. In this case (1.1.4) takes the form
(1.1. 7)
where
(1.1.8)
xEA
is called the Shannon entropy of the EIS (A, p,) or, simply, the probability
distribution ft. Thus H(ft) is the expectation ofthe randorn variable -log2 p,(.) .
The variance of this randorn variable, narnely,

O"~ = L p,(x) (log2 p,(x))2 - H(p,)2

xEA

is also a very useful quantity in the development of our subject. In fact the
randorn variable - log2 p,(.), its expectation H (p,) and its standard deviation 0"Jl.
play an important role in the understanding of coding theorems of information
theory.

Exercise 1.1.7 For any alphabet A let P(A) denote the set of all probability
distributions on A. Then the following holds :
(i) 0 ~ H(p,) ~ log #A V p, E P(A) . Here equality obtains on the left hand
side if and only if p, is degenerate, i.e., p,(x) = 1 for so me xE A. Equality
obtains on the right hand side if and only if p, is the uniform distribution
on A, i.e., p,(x) = (#A)-l V xE A.
(ii) For any pre-assigned value H for H{p,) the maximum of 0"Jl. is attained at
a distribution of the form
p if x = Xo,
p,{x) = { 2=L.f .../..
#A-l 1 X I Xo
1.1. Uniquely decipherable and irreducible codes 7

where 0 :s: p :s: 1. In particular,

max (Jp. = max VPQ {IOg2 E(#A - I)}

p.EP(A) p21/2 q

where q = 1 - p. Furthermore,

max(Jp.
p.

< {~ + ~ log2(#A - I)} log2 e.

In particular there exist absolute positive constants k 1 and k 2 (indepen-

dent of #A) such that

(Hint: For both (i) and (ii) use the method of Lagrange multipliers.)

Exercise 1.1.8 Let A = {O, I} and let f.tp(O) = 1- p, f.tp(1) = p where 0 :s: p :s: 1.
Then (A, f.tp) is called a Bernoulli source. Its entropy H(f.tp) = -p log2 P -
(1 - p) log2(1 - p) is a continuous function of p in the unit interval [O,IJ.
Here we interpret x log2 x to be 0 when x = O. Furthermore H(f.tp) increases
monotonically from 0 to 1 as p increases from 0 to 1/2. Thus, for every 0 :s:
() :s: 1, there exists a unique p(()) in the interval [0, 1/2J such that H(f.tp((J») = ().
See figure below:

(0,0) (0.5.0) (1,0)

P

Figure 1.1
8 Chapter 1. Entropy of Elementary Information Sources

Exercise 1.1.9 If (A, J.l) and (B, v) are two elementary information sources define
their product (A x B, J.l ® v) by

J.l ® v((x, y)) = J.l(x)v(y), xE A, y E B.

Then H(J.l ® v) = H(J.l) + H(v) .

If 0< h < 00 write h = lhJ + {h} where lhJ and {h} are respectively the
integral and fractional parts of hand consider the alphabet

A = {O, l} lh J+1

with the product probability distribution

0lhJ
J.l=J.l I /2 ®J.lp({h})

where J.lp(O) and J.lI/2 are as in Exercise 1.1.8. Then H(J.l) = h. In other words,
for any 0 < h < 00 there exists a product of Bernoulli sources with Shannon
entropy h .
We now conclude this section with a heuristic discussion. The important
inequality (1.1.7) teIls us that the Shannon entropy H(J.l) measures the amount
of space (in terms of codeword length) needed for encoding or storing a 'mes-
sage' from the EIS (A, J.l). We may interpret H(J.l) as the expected information
from (A, J.l). Now consider a general probability space (n, F, J.l). Such a proba-
bility space describes a statistical experiment. In such an experiment watch the
occurrence of an event E E F. The occurrence of such an event throws light
on the prob ability space and provides 'information'. Suppose this information
is measured by a nonnegative quantity I(E) . If the event E is certain to occur,
i.e., J.l(E) = 1 no information is gained by watehing E. In other words I(E) = 0
whenever J.l(E) = 1. The rarer an event, its occurrence throws greater light on
the probability space. In other words I(E) is a monotonie decreasing function
of J.l(E). If EI and E 2 are two independent events it is desirable to have the
property I(E I n E 2 ) = I(E I ) + I(E2 ). These properties are fulfilled if we put
I(E) == -k log J.l(E) for some positive constant k. As a normalization we choose
I(E) = 1 when J.l(E) = 1/ 2. This suggests that I(E) = -log2 J.l(E) is a suitable
measure of the information provided by watehing the occurrence of the event
E.
When the sampie space n is partitioned into disjoint events Ej E F ,
j = 1, 2, .. . , n so that n = U']=I E j then (n, F, J.l) is approximated by (n, A, J.l)
where A is the algebra generated by the events E j , j = 1,2, . . . ,n. This
data gives rise to the elementary information source ({E I ,E2 , ... ,En },J.l)
with the probability distribution J.l(Ej ) , j = 1, 2, ... , n and Shannon entropy
n
- L J.l( E j ) log2 J.l( E j ) which is the average information .provided by the ap-
j=l
proximate experiment (n, A, J.l) .
1.2. The HuJJman code 9

1.2 The Huffman code

We shall now describe a simple and nice algorithm due to Huffman [16] for
constructing an optimal irreducible code f : A --* S( {O, I}) for which Z(J) =
L(J..L) for a given probability distribution J..L on an alphabet A where Z(J) and
L(J..L) are as defined in Section 1.1. Without loss of generality we assume that
A = {I, 2, ... ,al, J..L(i) = Pi and PI ~ P2 ~ . .. ~ Pa> 0.

Proposition 1.2.1 Let f : {I, 2, . . . ,a} --* S( {O, I})" be an irreducible code with
a
l(J(i» = Li, 1 S; i S; a such that L(J..L) =L liPi. Then lt S; h ::; . .. S; la-2 S;
i=1
la-I = La ·

Proof. Clearly, lt S; l2 S; .. . S; La . Indeed, if this were not so and li > li+1 for
some i we can interchange the code words f (i) and f (i + 1) and construct a new
irreducible code 9 for which g(i) = f(i + 1), g(i + 1) = f(i) and g(j) = f(j) for
all j different from i and i + 1. Then Z(g) < Z(J), a contradiction. Now suppose
that la-I< La. Then the word f(a) can be expressed as f(a) = WIW2 where
r)
l(WI) = la-I, l(w = la -la-I· Since f is irreducible WI is not an extension of
any of the words f(i) , i = 1,2, ... , a - 1. By dropping W2 we can construct a
new irreducible code 9 for which g(i) = f(i) Vi S; a - 1 and g(a) = WI . Then
Z(g) < Z(J), a contradiction. 0

Theorem 1.2.2 Consider the contracted EIS ({ 1,2, ... , a-l}, v) where v( i) = Pi,
1 S; i S; a - 2, v(a -1) = Pa-I + Pa where PI ~ P2 ~ .. . ~ Pa as in Proposition
1.2.1. Suppose 9 : {I, 2, ... , a - I} --* S( {O, I}) is an irreducible code for which
Z(g) = L(L') . Now define fo : {I, 2, ... ,a} --* S( {O, I}) by fo( i) = g( i) if 1 S; i S;
a - 2, fo(a - 1) = g(a - 1)0, fo(a) = g(a - 1)1. Then fo is an irreducible code
für which l(fo) = L(J..L) where J..L(i) = Pi , 1 ::; i ~ a .

Proof. Let f : {I, 2, ... , a} --* S( {O, I}) be an irreducible code for which Z(J) =
L(J..L) . By Proposition 1.2.1, lt S; h S; .. . S; la-I = la where li = l(J(i». By
Theorem 1.1.1 and the fact that irreducibility implies unique decipherability
we have
2- 11 + 2- 12 + ... + 2- la - 2 + 2- la - 1 +1 ::; 1.
By Theorem 1.1.2 there exists an irreducible code
h: {1,2, .. . ,a -I} --* S({O, I}) for which l(h(i» = Li if 1 S; i S; a - 2 and
l(h(a - 1» = la-I - 1. Thus
a-2
L liPi + (la-I - 1) (Pa-I + Pa) ~ L(v) = Z(g)
i=1
a-2
L l(g(i»Pi + l(g(a - 1»(Pa-1 + Pa).
i=1
(1.2.1)
10 Chapter 1. Entropy of Elementary Information Sources

The map fo : {I, 2, ... , a} -+ S( {O, I}) is an irreducible code with

a-2
IUo) = Ll(g(i))Pi + (l(g(a -1)) + I)(Pa-l + Pa)
i=l

a-2
= LliPi + la-l(Pa-l + Pa). (1.2.2)
i=l

Inequalities (1.2.1) and (1.2.2) imply

a-2
L(Ji-) L l(g(i))pi + (l(g(a - 1)) + I)(Pa-l + Pa)
i=l

IUo).

Remark 1.2.3 According to Theorem 1.2.2 the construction of an optimal ir-

°
reducible binary code (i.e., a code with encoding alphabet {O, I}) for an EIS
({1,2, ... ,a},Ji-) where Ji-(i) = Pi, Pl ~ P2 ~ ... ~ Pa > reduces to the
construction of an optimal irreducible binary code for the contracted or re-
duced EIS ({1,2, ... ,G - 1},v) where v(i) = Pi, i = 1,2, ... ,a - 2 and
v(a - 1) = Pa-l + Pa. However, it may be necessary to rearrange the points
of the reduced alphabet in order of decreasing probabilities. This reduces the
problem inductively and finally to an EIS with a two-point alphabet. We shall
illustrate by concrete examples.

°
Example 1.2.4 If A = {I, 2, 3}, Ji-(i) = Pi, Pl ~ P2 ~ P3 > consider the reduced
EIS ({I, 2}, v) with v(l) = Pl, v(2) = P2 + P3. Then we have two cases :

Case 1: Pl ~ P2 + P3· Then fo(l) = 0, fo(2) = 10, fo(3) = 11.

Case 2: P2 + P3 > Pl· Then fo(l) = 1, fo(2) = 00, fo(3) = 01.

1.2. The Huffman code 11

Example 1.2.5 Consider the EIS ({I, 2, 3, 4, 5, 6, 7}, (0.3,0.2,0.15,

0.15,0.1,0.05,0.05)) .

A 1 2 3 4 5 6 7
0.3 0.2 0.15 0.15 0.1 0.05 0.05
f-t 0 0
0
O~'
, __
A

o. ~ ___ >_ ~

0.2
-- > -

Figure 1.2

The code word corresponding to 6 i.e., 10(6) is to be read along the dotted line
6
from left to right. Note that I: 2 fo (i) = 1.
i=l

The Huffman code 10 is given by the following table:

lo(i)
1 00
2 10
3 010
4 011
5 110
6 1110
7 1111

Exercise 1.2.6 Let ({1 , 2, ... ,a},f-t) be an EIS and let

hp. : {I, 2, ... ,a} -+ S( {O, I}) be a Huffman code constructed by the inductive
algorithm determined by Theorem 1.2.2. Then
a
LTI(h,,(i)) = 1.
i=l
12 Chapter 1. Entropy of Elementary Information Sources

(Rint : Use induction on a.)

Exercise 1.2.7 When A = {O, 1,2, ... , 2n - I} and p, is the uniform distribution
then the binary expansion map f : A -+ {O, l}n is a Ruffman code. In this case
L(p,) = H(p,) = n.

1.3 Entropy and conditional entropy

Let A be an alphabet and let P(A) denote the set of all prob ability distribu-
tions on A. Any element p, E P(A) is thus a map from A into [0,1] satisfying
l: p,(x) = 1. Thus P(A) is a compact convex subset of the real linear space
xEA
]RA . Define the map H : P(A) -+ [0,00] by

H(p,) =- L p,(x) log2 p,(x), P, E P(A). (1.3.1)

xEA

Theorem 1.1.5, Remark 1.1.6 and the heuristic discussion at the end of Section
1.1 indicate that the Shannon entropy can be interpreted as a reasonable mea-
sure of the 'information content' of the EIS (A, J1). We shall now make a more
detailed investigation of the properties of H. Hereafter by the symbol 'log' we
shall understand that it is the logarithm with respect to the base 2. Thus the
naturallogarithm will be expressed as loge' Consider the function

cp
(t) = {
°
-t loge t if
if
0 < t :S 1,
t = 0.
(1.3.2)

in the interval [0,1]. Since cp'(t) = -(1 + loge t) and cp"(t) = -Cl it follows
that cp is a continuous and strictly concave function which increases from to°
°
e- l in [0, e- l ], attains its maximum value e- l at e- l and decreases from e- l
to in [e- l , 1]. See Figure 1.3 below :

Figure 1.3
1.3. Entropy and conditional entropy 13

Then (1.3.1) can be expressed as

H(J-L) = (loge 2)-1 L cp(J-L(x)). (1.3.3)

xEA

Proposition 1.3.1 The map H : P(A) -+ [0,00) is continuous and for any Pi > 0,
i= 1,2, ... , k with LPi = 1 and any J-Li E P(A), i = 1,2, ... , k the following
holds:

(1.3.4)

where equality on the left obtains if and only if J-Ll = J-L2 = ... = J-Lk.
Furthermore, for any J-L E P(A), 0 :::; H(J-L) :::; log #A where (i) H(J-L) = 0
if and only if J-L is degenerate at a point; (ii) H(J-L) = log #A if and only if J-L is
the uniform distribution on A.

Proof. The continuity of H(J-L) in the variable J-L E P(A) is immediate from the
continuity of the function cP and (1.3.3). The strict concavity of cP implies that

where equality is possible if and only if J-Li(X) is independent of i. Summing

over x E A and multiplying by (loge 2)-1 on both sides we get the left hand
part of (1.3.4). Observing that

-logLPiJ-Li(x):::; -logpjJ-Lj(x) \:j x E A, 1:::; j:::; k,

multiplying both sides by Pj J-Lj (x) and adding over j and x we get the right
hand part of (1.3.4). Clearly, H(J-L) 2: 0 and H(J-L) = 0 if and only if cp(J-L(x)) = 0
\:j x E A, i.e., J-L(x) = 0 or 1 \:j x, i.e., J-L is degenerate at some point Xo in A. To
prove the last part of the proposition we shall repeat the argument involving
the weighted geometrie and arithmetic means in the proof of Theorem 1.1.5.
Without loss of generality we assurne that J-L(x) > 0 \:j xE A. Let v be another
probability distribution on A. Then

rr
xEA
[V(X)]JL(X) :::; LJ-L(X)v(x) = 1
J-L(x) xEA J-L(x)

where equality holds if and only if v(x) == J-L(x). Taking log we get

H(J-L):::; - LJ-L(x)logv(x).
xEA
14 Chapter 1. Entropy of Elementary Information Sources

Choosing v to be the uniform distribution in A, Le., v( x) = (#A) -1 'Vx we get

H(J.L) :S log #A with equality if and only if J.L is also the uniform distribution.
o
In practice elementary information sources arise in different ways. In order
to describe them it is convenient to consider a general statistical experiment
whose events and chances of occurrence are described by a probability space
(n, F, P) consisting of a sampIe space n, a sub a-algebra of subsets of n and
a probability measure P on F. Very often the statistical experiment (n, F, P)
can be difficult to perform owing to several reasons like cost, inaccessbility in
collecting data etc., but partial experiments like (n, A, P) involving events from
a sub a-algebra A C F may be executed. For example, a particular random
variable X on (n, F, P) with values in a finite set or some measurable space
like IR, IRn etc., may be observed and in such a case the sub a-algebra of all
events concerning X becomes accessible. As a special case consider a simple
random variable X on (n, F, P) assuming values in a finite set A. If we write
J.L(x) = P(X = x) = PX-l({x}) then (A, J.L) is an EIS with entropy H(J.L) which
is also denoted by H(X). We say that the simple random variable X yields
H(X) bits of information about (n, F, P). If Xl, X 2, ... ,Xk are k such simple
random variables assuming values in alphabets Al, A 2, . . . , Ak respectively then
we get an EIS with the alphabet Al x A 2 X . . . X A k and J.L(XI,X2, . . . ,Xk) =
P(Xi = Xi, i = 1,2, ... , k) whose entropy will be denoted by H(X l X 2 ... Xk).
(Here we have dropped the commas and no multiplication is implied. )This same
entropy is sometimes denoted by H(A I A 2 ... Ad. The simple random variable
X mentioned above yields Ex = X-I ({ X} ), X E A satisfying Ex n E y = 0 if
xi- y , uxEx = Ü, Ex E F. In other words 7r = {Ex , XE A} is a finite partition
of n into measurable sets from F . The subalgebra A generated by the events
Ex,x E A, is denoted by A(7r) or A(X) and one also writes H(X) = H(A) =
H(7f).
Let now A C F be an arbitrary sub a-algebra. After knowing the result
of the partial experiment in the form of a sampIe point w from (n, A, P) one
can look at the conditional probability P(E I A)(w) for any E E F. Let now
7f = {EI, E 2, .. . , E k } be a finite measurable partition of n. By adjusting the

°
definitions of P(Ei I A) on a set of probability we may assurne that the
following properties hold:
(i) P(Ei I A)(w) = P(Ei,w) is an A-measurable random variable with values
in [0,1] for each i = 1,2, ... , k.
k
(ii) I: P(Ei ,w) = 1 'VwEn.
i=I

(iii) J P(Ei ,w)P(dw) = P(Ei n F) 'V F E A, 1:S i :S k.

F
Thus we get a 'random' EIS (7f,P(.,w)) with entropy
k
H w (7f) = - LP(Ei,w)logP(Ei,w).
i=I
Random documents with unrelated
content Scribd suggests to you:
We cultivate such a use of our eyes, as indeed of all our faculties, as
will on the whole lead to the most profitable results. As a rule, the
particular impression is not so important as what it represents.
Sense impressions are simply the symbols or signs of things or
ideas, and the thing or the idea is more important than the sign.
Accordingly, we are accustomed to interpret lines, whenever we can,
as the representations of objects. We are well aware that the canvas
or the etching or the photograph before us is a flat surface in two
dimensions, but we see the picture as the representation of solid
objects in three dimensions. This is the illusion of pictorial art. So
strong is this tendency to view lines as the symbols of things that if
there is the slightest chance of so viewing them, we invariably do so;
for we have a great deal of experience with things that present their
contours as lines, and very little with mere lines or surfaces. If we
view outlines only, without shading or perspective or anything to
definitely suggest what is foreground and what background, it
becomes possible for the mind to supply these details and see
foreground as background, and vice versa.
A good example to begin with is Fig. 8. These outlines will probably
suggest at first view a book, or better a book cover, seen with its
back toward you and its sides sloping away from you; but it may
also be viewed as a book opened out toward you and presenting to
you an inside view of its contents. Should the change not come
readily, it may be facilitated by thinking persistently of the
appearance of an open book in this position. The upper portion of
Fig. 9 is practically the same as Fig. 8, and if the rest of the figure
be covered up, it will change as did the book cover; when, however,
the whole figure is viewed as an arrow, a new conception enters,
and the apparently solid book cover becomes the flat feathered part
of the arrow. Look at the next figure (Fig. 10), which represents in
outline a truncated pyramid with a square base. Is the smaller
square nearer to you, and are the sides of the pyramid sloping away
from you toward the larger square in the rear? Or are you looking
into the hollow of a truncated pyramid with the smaller square in the
background? Or is it now one and now the other, according as you
 decid
e to
see
it?
Here
(Fig.
13) is
a
skelet
on
box
which
you
may
conce
ive as
made
of
wires
outlini
ng
the
Fig. 8.—This drawing may be sides. Fig. 9.—When
viewed as the representation of a Now this figure is
book standing on its half-opened the viewed as an
covers as seen from the back of front, arrow, the
the book; or as the inside view of or upper or
an open book showing the pages. side feathered end
neare seems flat;
st to me, seems directed downward and to the when the rest of
left; again, it has shifted its position and is no the arrow is
longer the front, and the side which appears to covered, the
be the front seems directed upward and to the feathered end
right. The presence of the diagonal line makes may be made to
the change more striking: in one position it runs project or
from the left-hand rear upper corner to the recede like the
right-hand front lower corner; while in the other book cover in
it connects the left-hand front upper corner Fig. 8.
with the right-hand rear lower corner.

Fig. 10.—The smaller square may be

regarded as either the nearer face of a
projecting figure or as the more distant
face of a hollow figure.
 Fig. 11.—This represents an
ordinary table-glass, the bottom
of the glass and the entire rear
side, except the upper portion,
being seen through the
transparent nearer side, and the
rear apparently projecting above
the front. But it fluctuates in
appearance between this and a
view of the glass in which the
bottom is seen directly, partly
from underneath, the whole of
the rear side is seen through the
transparent front, and the front
projects above the back.
 Fig. 12.—In this scroll the left
half may at first seem concave
and the right convex, it then
seems to roll or advance like a
wave, and the left seems
convex and the right concave,
as though the trough of the
wave had become the crest, and
vice versa.
 Figs. 13, 13a, and 13b.—The two methods of viewing Fig.
13 are described in the text. Figs. 13a and 13b are added to
make clearer the two methods of viewing Fig. 13. The
heavier lines seem to represent the nearer surface. Fig. 13a
more naturally suggests the nearer surface of the box in a
position downward and to the left, and Fig. 13b makes the
nearer side seem to be upward and to the right. But in
spite of the heavier outlines of the one surface, it may be
made to shift positions from foreground to background,
although not so readily as in Fig. 13.

Fig. 14.—Each member of this frieze represents a relief

ornament, applied upon the background, which in cross-
section would be an isosceles triangle with a large obtuse
angle, or a space of similar shape hollowed out of the solid
wood or stone. In running the eye along the pattern, it is
interesting to observe how variously the patterns fluctuate
from one of these aspects to the other.
Figs. 15, 15a, and 15b.—The two views of Fig. 15 described
in the text are brought out more clearly in Figs. 15a and
15b. The shaded portion tends to be regarded as the nearer
face. Fig. 15a is more apt to suggest the steps seen as we
ascend them. Fig. 15b seems to represent the hollowed-out
structure underneath the steps. But even with the shading
the dual interpretation is possible, although less obvious.
Fig. 15 will probably seen at first glimpse to be the view of a flight of
steps which one is about to ascend from right to left. Imagine it,
however, to be a view of the under side of a series of steps; the view
representing the structure of overhanging solid masonwork seen
from underneath. At first it may be difficult to see it thus, because
the view of steps which we are about to mount is a more natural
and frequent experience than the other; but by staring at it with the
intention of seeing it differently the transition will come, and often
quite unexpectedly.
 Fig. 16.—This interesting figure (which is
reproduced with modifications from Scripture—
The New Psychology) is subject in a striking way
to interchanges between foreground and
background. Most persons find it difficult to
maintain for any considerable time either aspect
of the blocks (these aspects are described in the
text); some can change them at will, others must
accept the changes as they happen to come.
 Figs. 17, 17a, and 17b.—How many
blocks are there in this pile? Six or
seven? Note the change in
arrangement of the blocks as they
change in number from six to seven.
This change is illustrated in the text.
Figs. 17a and 17b show the two phases
of a group of any three of the blocks.
The arrangement of a pyramid of six
blocks seems the more stable and is
usually first suggested; but hold the
page inverted, and you will probably
see the alternate arrangement (with,
however, the black surfaces still
forming the tops). And once knowing
what to look for, you will very likely be
 able to see either arrangement,
whether the diagram be held inverted
or not. This method of viewing the
figures upside down and in other
positions is also suggested to bring out
the changes indicated in Figs. 13, 13a,
13b, and in Figs. 15, 15a, 15b.
The blocks in Fig. 16 are subject to a marked fluctuation. Now the
black surfaces represent the bottoms of the blocks, all pointing
downward and to the left, and now the black surfaces have changed
and have become the tops pointing upward and to the right. For
some the changes come at will; for others they seem to come
unexpectedly, but all are aided by anticipating mentally the nature of
the transformation. The effect here is quite striking, the blocks
seeming almost animated and moving through space. In Fig. 17 a
similar arrangement serves to create an illusion as to the real
number of blocks present. If viewed in one way—the black surface
forming the tops of the blocks—there seem to be six arranged as in
Fig. 18; but when the transformation has taken place and the black
surfaces have become the overhanging bottoms of the boxes, there
are seven, arranged as in Fig. 19. Somewhat different, but still
belonging to the group of ambiguous figures, is the ingenious
conceit of the duck-rabbit shown in Fig. 20. When it is a rabbit, the
face looks to the right and a pair of ears are conspicuous behind;
when it is a duck, the face looks to the left and the ears have been
changed into the bill. Most observers find it difficult to hold either
interpretation steadily, the fluctuations being frequent, and coming
as a surprise.
 Figs. 18 and 19.

Fig. 20.—Do you see a duck or a rabbit,

or either? (From Harper's Weekly,
originally in Fliegende Blätter.)
All these diagrams serve to illustrate the principle that when the
objective features are ambiguous we see one thing or another
according to the impression that is in the mind's eye; what the
objective factors lack in definiteness the subjective ones supply,
while familiarity, prepossession, as well as other circumstances
influence the result. These illustrations show conclusively that seeing
is not wholly an objective matter depending upon what there is to be
seen, but is very considerably a subjective matter depending upon
the eye that sees. To the same observer a given arrangement of
lines now appears as the representation of one object and now of
another; and from the same objective experience, especially in
instances that demand a somewhat complicated exercise of the
senses, different observers derive very different impressions.
Not only when the sense-impressions are ambiguous or defective,
but when they are vague—when the light is dim or the forms
obscure—does the mind's eye eke out the imperfections of physical
vision. The vague conformations of drapery and make-up that are
identified and recognized in spiritualistic séances illustrate extreme
instances of this process. The whitewashed tree or post that
momentarily startles us in a dark country lane takes on the guise
that expectancy gives it. The mental predisposition here becomes
the dominant factor, and the timid see as ghosts what their more
sturdy companions recognize as whitewashed posts. Such
experiences we ascribe to the action of suggestion and the
imagination—the cloud "that's almost in shape like a camel," or "like
a weasel," or "like a whale." But throughout our visual experiences
there runs this double strain, now mainly outward and now mainly
inward, from the simplest excitements of the retina up to the realms
where fancy soars freed from the confines of sense, and the
objective finds its occupation gone.
 NATURE STUDY IN THE
PHILADELPHIA NORMAL SCHOOL.
By L. L. W. WILSON, Ph. D.
When it was first proposed to me to write for the Popular Science
Monthly a brief account of the biological laboratories in the
Philadelphia Normal School, and of the Nature work carried on under
my direction in the School of Observation and Practice, I felt that I
could not do justice either to the place or the work; for, in my
judgment, the equipment of the laboratories and the work done in
connection with them are finer than anything else of the kind either
in this country or abroad—a statement which it seemed to me that I
could not make with becoming modesty. But, after all, it is not great
Babylon that I have built, but a Babylon builded for me, and to fail to
express my sense of its worth is to fail to do justice to Dr. W. P.
Wilson, formerly of the University of Pennsylvania, to whom their
inception was due; to Mr. Simon Gratz, president of the Board of
Education, who from the beginning appreciated their value, and
without whose aid they never would have taken visible form; to the
principals of the two schools, and, above all, to my five assistants,
whose knowledge, zeal, and hard work have contributed more than
anything else to the rapid building up of the work.
The Laboratories and their Equipment.—The rooms occupied by the
botanical and zoölogical departments of the normal school measure
each seventy by twenty feet. A small workroom for the teachers cuts
off about ten feet of this length from each room. In the middle of
the remaining space stands a demonstration table furnished with hot
and cold water. Each laboratory is lighted from the side by ten
windows. From them extend the tables for the students. These give
plenty of drawer space and closets for dissecting and compound
microscopes. Those in the zoölogical room are also provided with
sinks. Each student is furnished with the two microscopes, stage and
eyepiece micrometers, a drawing camera, a set of dissecting
instruments, glassware, note-books, text-books, and general
literature.
The walls opposite the windows are in both rooms lined with cases,
in which there is a fine synoptic series.
In the botanical laboratory this systematic collection begins with
models of bacteria and ends with trees. In other cases, placed in the
adjoining corridor, are representatives, either in alcohol or by means
of models, of most of the orders of flowering plants, as well as a
series illustrating the history of the theory of cross-fertilization, and
the various devices by which it is accomplished; another, showing
the different methods of distribution of seeds and fruits; another, of
parasitic plants; and still another showing the various devices by
means of which plants catch animals.
As an example of the graphic and thorough way in which these
illustrations are worked out, the pines may be cited. There are
fossils; fine specimens of pistillate and staminate flowers in alcohol;
cones; a drawing of the pollen; large models of the flowers; models
of the seeds, showing the embryo and the various stages of
germination; cross and longitudinal sections of the wood; drawings
showing its microscopic structure; pictures of adult trees; and
samples illustrating their economic importance. For the last, the
long-leaved pine of the South is used, and samples are exhibited of
the turpentine, crude and refined; tar and the oil of tar; resin; the
leaves; the same boiled in potash; the same hatcheled into wool;
yarn, bagging and rope made from the wool; and its timber split,
sawn, and dressed.
The series illustrating the fertilization of flowers begins with a large
drawing, adapted by one of the students from Gibson, showing the
gradual evolution of the belief in cross-fertilization from 1682, when
Nehemiah Grew first declared that seed would not set unless pollen
reached the stigma, down to Darwin, who first demonstrated the
advantages of cross-fertilization and showed many of the devices of
plants by which this is accomplished. The special devices are then
illustrated with models and large drawings. First comes the
dimorphic primrose; then follows trimorphic Lythrum, to the beautiful
model of which is appended a copy of the letter in which Darwin
wrote to Gray of his discovery:

"But I am almost stark, staring mad over Lythrum.... I should

rather like seed of Mitchella. But, oh, Lythrum!
"Your utterly mad friend,
"C. Darwin."

Models of the cucumber, showing the process of its formation, and

the unisexual flowers complete this series. Supplementing this are
models and drawings of a large number of flowers, illustrating
special devices by which cross-fertilization is secured, such as the
larkspur, butter and eggs, orchids, iris, salvia, several composites,
the milkweed, and, most interesting of all, the Dutchman's pipe. This
is a flower that entices flies into its curved trumpet and keeps them
there until they become covered with the ripe pollen. Then the hairs
wither, the tube changes its position, the fly is permitted to leave,
carrying the pollen thus acquired to another flower with the same
result.
Pictures and small busts of many naturalists adorn both of the
rooms. Of these the most notable is an artist proof of Mercier's
beautiful etching of Darwin. Every available inch of wall space is thus
occupied, or else, in the botanical laboratory, has on it mounted
fungi, lichens, seaweeds, leaf cards, pictures of trees, grasses, and
other botanical objects.
The windows are beautiful with hanging plants from side brackets
meeting the wealth of green on the sill. Here are found in one
window ferns, in another the century plant; in others still, specimens
of economic plants—cinnamon, olive, banana, camphor. On the
tables are magnificent specimens of palms, cycads, dracænas, and
aspidistras, and numerous aquaria filled with various water plants.
Most of these plants are four years old, and all of them are much
handsomer than when they first became the property of the
laboratory. How much intelligent and patient care this means only
those who have attempted to raise plants in city houses can know.
The zoölogical laboratory is quite as beautiful as the botanical, for it,
too, has its plants and pictures. It is perhaps more interesting
because of its living elements. Think of a schoolroom in which are
represented alive types of animals as various as these: amœba,
vorticella, hydra, worms, muscles, snails and slugs of various kinds,
crayfish, various insects, including a hive of Italian bees, goldfish,
minnows, dace, catfish, sunfish, eels, tadpoles, frogs, newts,
salamanders, snakes, alligators, turtles, pigeons, canaries, mice,
guinea-pigs, rabbits, squirrels, and a monkey! Imagine these living
animals supplemented by models of their related antediluvian forms,
or fossils, by carefully labeled dissections, by preparations and
pictures illustrating their development and mode of life; imagine in
addition to this books, pamphlets, magazines, and teachers further
to put you in touch with this wonderful world about us, and you will
then have some idea of the environment in which it is the great
privilege of our students to live for five hours each week.
In addition to these laboratories there is a lecture room furnished
with an electric lantern. Here each week is given a lecture on
general topics, such as evolution and its problems, connected with
the work of the laboratories.
The Course of Study pursued by the Normal Students.—Botany: In
general, the plants and the phenomena of the changing seasons are
studied as they occur in Nature. In the fall there are lessons on the
composites and other autumn flowers, on fruits, on the ferns,
mosses, fungi, and other cryptogams. In the winter months the
students grow various seeds at home, carefully drawing and
studying every stage in their development. Meanwhile, in the
laboratory, they examine microscopically and macroscopically the
seeds themselves and the various food supplies stored within. By
experimentation they get general ideas of plant physiology,
beginning with the absorption of water by seeds, the change of the
food supply to soluble sugar, the method of growth, the functions,
the histology, and the modifications of stem, root, and leaves. In the
spring they study the buds and trees, particularly the conifers, and
the different orders of flowering plants.
The particular merit of the work is that it is so planned that each
laboratory lesson compels the students to reason. Having once thus
obtained their information, they are required to drill themselves out
of school hours until the facts become an integral part of their
knowledge.
For the study of fruits, for example, they are given large trays, each
divided into sixteen compartments, plainly labeled with the name of
the seed or fruit within. Then, by means of questions, the students
are made to read for themselves the story which each fruit has to
tell, to compare it with the others, and to deduce from this
comparison certain general laws.
After sufficient laboratory practice of this kind they are required to
read parts of Lubbock's Flower, Fruit, and Leaves, Kerner's Natural
History of Plants, Wallace's Tropical Nature, and Darwinism, etc.
Finally, they are each given a type-written summary of the work, and
after a week's notice are required to pass a written examination.
Zoölogy: The course begins in the fall with a rather thorough study
of the insects, partly because they are then so abundant, and partly
because a knowledge of them is particularly useful to the grade
teacher in the elementary schools.
The locust is studied in detail. Tumblers and aquaria are utilized as
vivaria, so that there is abundant opportunity for the individual study
of living specimens. Freshly killed material is used for dissection, so
that students have no difficulty in making out the internal anatomy,
which is further elucidated with large, home-made charts, each of
which shows a single system, and serves for a text to teach them
the functions of the various organs as worked out by modern
physiologists.
They then study, always with abundant material, the other insects
belonging to the same group. They are given two such insects, a
bug, and two beetles, and required to classify them, giving reasons
for so doing. While this work is going on they have visited the
beehive in small groups, sometimes seeing the queen and the drone,
and always having the opportunity to see the workers pursuing their
various occupations, and the eggs, larvæ, and pupæ in their
different states of development. Beautiful models of the bees and of
the comb, together with dry and alcoholic material, illustrate further
this metamorphosis, by contrast making clearer the exactly opposite
metamorphosis of the locust.
At least one member of each of the other orders of insects is
compared with these two type forms, and, although only important
points are considered at all, yet from one to two hours of laboratory
work are devoted to each specimen. This leisurely method of work is
pursued to give the students the opportunity, at least, to think for
themselves. When the subject is finished they are then given a
searching test. This is never directly on their required reading, but
planned to show to them and to their teachers whether they have
really assimilated what they have seen and studied.
After this the myriapods, the earthworm, and peripatus are studied,
because of their resemblance to the probable ancestors of insects.
In the meantime they have had a dozen or more fully illustrated
lectures on evolution, so that at the close of this series of lessons
they are expected to have gained a knowledge of the methods of
studying insects, whether living or otherwise, a working hypothesis
for the interpretation of facts so obtained, and a knowledge of one
order, which will serve admirably as a basis for comparison in much
of their future work.
They then take up, more briefly, the relatives of the insects, the
spiders and crustaceans, following these with the higher
invertebrates, reaching the fish in April. This, for obvious reasons, is
their last dissection. But with living material, and the beautiful
preparations and stuffed specimens with which the laboratory is
filled, they get a very general idea of the reptiles, birds, and
mammals. This work is of necessity largely done by the students out
of school hours. For example, on a stand on one of the tables are
placed the various birds in season, with accompanying nests
containing the proper quota of eggs. Books and pamphlets relating
to the subject are placed near. Each student is given a syllabus
which will enable her to study these birds intelligently indoors and
out, if she wishes to do so.
In the spring are taken up the orders of animals below the insect,
and for the last lesson a general survey of all the types studied gives
them the relationships of each to the other.
The Course of Study pursued in the School of Practice.—In addition to
the plants and animals about them, the children study the weather,
keeping a daily record of their observations, and summarizing their
results at the end of the month. In connection with the weather and
plants they study somewhat carefully the soil and, in this connection,
the common rocks and minerals of Philadelphia—gneiss, mica schist,
granite, sandstone, limestones, quartz, mica, and feldspar.
As in the laboratories, so here the effort is made to teach the
children to reason, to read the story told by the individual plant, or
animal, or stone, or wind, or cloud. A special effort is made to teach
them to interpret everyday Nature as it lies around them. For this
reason frequent short excursions into the city streets are made.
Those who smile and think that there is not much of Nature to be
found in a city street are those who have never looked for it. Enough
material for study has been gathered in these excursions to make
them a feature of this work, even more than the longer ones which
they take twice a year into the country.
Last year I made not less than eighty such short excursions, each
time with classes of about thirty-five. They were children of from
seven to fourteen years of age. Without their hats, taking with them
note-books, pencils, and knives, they passed with me to the street.
The passers-by stopped to gaze at us, some with expressions of
amusement, others of astonishment; approval sometimes, quite
frequently the reverse. But I never once saw on the part of the
children a consciousness of the mild sensation that they were
creating. They went for a definite purpose, which was always
accomplished.
The children of the first and second years study nearly the same
objects. Those of the third and fourth years review this general
work, studying more thoroughly some one type. When they enter
the fifth year, they have considerable causal knowledge of the
familiar plants and animals, of the stones, and of the weather. But,
what is more precious to them, they are sufficiently trained to be
able to look at new objects with a truly "seeing eye."
The course of study now requires general ideas of physiology, and,
in consequences, the greater portion of their time for science is
devoted to this subject. I am glad to be able to say, however, that it
is not "School Physiology" which they study, but the guinea-pig and
The Wandering Jew!
In other words, I let them find out for themselves how and what the
guinea-pig eats; how and what he expires and inspires; how and
why he moves. Along with this they study also plant respiration,
transpiration, assimilation, and reproduction, comparing these
processes with those of animals, including themselves.
The children's interest is aroused and their observation stimulated by
the constant presence in the room with them of a mother guinea-pig
and her child. Nevertheless, I have not hesitated to call in outside
materials to help them to understand the work. A series of lessons
on the lime carbonates, therefore, preceded the lessons on
respiration; an elephant's tooth, which I happened to have, helped
to explain the guinea-pig's molars; and a microscope and a frog's leg
made real to them the circulation of the blood.
In spite of the time required for the physiology, the fifth-year
children have about thirty lessons on minerals; the sixth-year, the
same number on plants; and the seventh-year, on animals; and it
would be difficult to decide which of these subjects rouses their
greatest enthusiasm.
 PRINCIPLES OF TAXATION.[6]
By the Late Hon. DAVID A. WELLS.

XX.—THE LAW OF THE DIFFUSION OF TAXES.

PART I.

No attempt ought to be made to construct or formulate an

economically correct, equitable, and efficient system of taxation
which does not give full consideration to the method or extent to
which taxes diffuse themselves after their first incidence. On this
subject there is a great difference of opinion, which has occasioned,
for more than a century, a vast and never-ending discussion on the
part of economic writers. All of this, however, has resulted in no
generally accepted practical conclusions; has been truthfully
characterized by a leading French economist (M. Parieu) as marked
in no small part by the "simplicity of ignorance," and from a
somewhat complete review (recently published[7]) of the conflicting
theories advanced by participants one rises with a feeling of
weariness and disgust.
The majority of economists, legislators, and the public generally
incline to the opinion that taxes mainly rest where they are laid, and
are not shifted or diffused to an extent that requires any recognition
in the enactment of statutes for their assessment. Thus, a tax
commission of Massachusetts, as the result of their investigations,
arrived at the conclusion that "the tendency of taxes is that they
must be paid by the actual persons on whom they are levied." But a
little thought must, however, make clear that unless the
advancement of taxes and their final and actual payment are one
and the same thing, the Massachusetts statement is simply an
evasion of the main question at issue, and that its authors had no
intelligent conception of it. A better proposition, and one that may
even be regarded as an economic axiom, is that, regarding taxation
as a synonym for a force, as it really is, it follows the natural and
invariable law of all forces, and distributes itself in the line of least
resistance. It is also valuable as indicating the line of inquiry most
likely to lead to exact and practical conclusions. But beyond this it
lacks value, inasmuch as it fails to embody any suggestions as to the
best method of making the involved principle a basis for any general
system for correct taxation; inasmuch as "the line of least
resistance" is not a positive factor, and may be and often is so
arranged as to make levies on the part of the State under the name
of taxation subservient to private rather than public interests. Under
such circumstances the question naturally arises, What is the best
method for determining, at least, the approximative truth in respect
to this vexed subject? A manifestly correct answer would be: first, to
avoid at the outset all theoretic assumptions as a basis for
reasoning; second, to obtain and marshal all the facts and conditions
incident to the inquiry or deducible from experience; third, recognize
the interdependence of all such facts and conclusions; fourth, be
practical in the highest degree in accepting things as they are, and
dealing with them as they are found; and on such a basis attention
is next asked to the following line of investigations.
It is essential at the outset to correct reasoning that the distinction
between taxation and spoliation be kept clearly in view. That only is
entitled to be called a tax law which levies uniformly upon all the
subjects of taxation; which does not of itself exempt any part of the
property of the same class which is selected to bear the primary
burden of taxation, or by its imperfections to any extent permits
such exemptions. All levies or assessments made by the State on the
persons, property, or business of its citizens that do not conform to
such conditions are spoliations, concerning which nothing but
irregularity can be predicated; nothing positive concerning their
diffusion can be asserted; and the most complete collection of
experiences in respect to them can not be properly dignified as "a
science." And it may be properly claimed that from a nonrecognition
or lack of appreciation of the broad distinction between taxation and
spoliation, the disagreement among economists respecting the
diffusion of taxes has mainly originated.
With this premise, let us next consider what facts and experiences
are pertinent to this subject, and available to assist in reaching
sound conclusions; proceeding very carefully and cautiously in so
doing, inasmuch as territory is to be entered upon that has not been
generally or thoroughly explored.
The facts and experiences of first importance in such inquiry are that
the examination of the tax rolls in any State, city, or municipality of
the United States will show that surprisingly small numbers of
persons primarily pay or advance any kind of taxes. It is not
probable that more than one tenth of the adult population or about
one twentieth of the entire population of the United States ever
come in contact officially with a tax assessor or tax collector. It is
also estimated that less than two per cent of the total population of
the United States advance the entire customs and internal revenue
of the Federal Government.
In the investigations made in 1871, by a commission created by the
Legislature of the State of New York to revise its laws relative to the
assessment and collection of taxes, it was found that in the city of
New York, out of a population of over one million in the above year,
only 8,920 names, or less than one per cent of this great multitude
of people, had "any household furniture, money, goods, chattels,
debts due from solvent debtors, whether on account of contract,
note, bond, or mortgage, or any public stocks, or stocks in moneyed
corporations, or in general any personal property of which the
assessors could take cognizance for taxation"; and further, that not
over four per cent, or, say, forty thousand persons out of the million,
were subject to any primary tax in respect to the ownership of any
property whatever, real or personal; while only a few years
subsequent, or in 1875, the regular tax commissioners of New York
estimated that of the property defined and described by the laws of
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookname.com