Susskind Notes (those parts that were published)

Contents
Conservation of Information, Energy, Entropy, and Temperature 1.......................................................... 1 1.1 Introduction ....................................................................................................................................... 1 conservation of distinction, or conservation of information. .......................................................... 3 1.2 The 1st Law of Thermodynamics: Energy is conserved ....................................................................... 3 1.3 Entropy is an information-theoretic concept. ..................................................................................... 3 Entropy is measured in bits .................................................................................................................. 4 The general definition of entropy ......................................................................................................... 4 1.4 Temperature ...................................................................................................................................... 5 1.5 The 2nd Second Law of Thermodynamics: Entropy always increases ................................................... 5 2 The Bolzmann Distribution .................................................................................................................... 6 2.1 Solving for the Lagrange Multipliers ................................................................................................... 8 2.2 Helmholtz Free Energy ....................................................................................................................... 8 3 Fluctuations .......................................................................................................................................... 9 Note on Boltzmann constant k = 1 ..................................................................................................... 10 4 Control Parameters............................................................................................................................. 10 4.1 The Adiabatic Theorem and the First Law ......................................................................................... 10 Adiabatic Invariants Entropy .............................................................................................................. 10 4.2 Processes at Fixed Temperature....................................................................................................... 11 Part II Some Simple Applications ............................................................................................................ 12 5 Ideal Gas ............................................................................................................................................. 12

Conservation of Information, Energy, Entropy, and Temperature 1 1.1 Introduction

Statistical mechanics of often thought of as the theory of how atoms combine to form gases, liquids, solids, and even plasmas and black body radiation. But it is both more and less than that.

Statistical mechanics is a useful tool in many areas of science in which large a number of variables have to be dealt with using statistical methods. My son, who studies neural networks, uses it. I have no doubt that some of the financial wizards at AIG and Lehman Brothers used it. Saying that Stat Mech is the theory of gasses is rather like saying calculus is the theory of planetary orbits. Stat Mech is really a particular type of probability theory. Coin flipping is a good place to start. The probabilities for heads (H) and tails (T) are both equal to 1/2. Why do I say that? One answer is that the symmetry between H and T means their probabilities are equal. Here is another example. Lets take a die (as in dice) and color the six faces red, yellow, blue, green, orange, and purple (R, Y, B, G,O, P). The obvious cubic symmetry of the die dictates that the probabilities are equal to 1/6. But what if we dont have a symmetry to rely on? How do we assign a priori probabilities? Suppose for example instead of the coloring scheme that I indicated above, I chose to color the purple face red. Then there would be only five colors. Would the probability of throwing a given color be 1/5? After all, if I just write (R, Y, B, G,O), the 5 names are just as symmetric as the original 6 names. Nonsense, you say: the real symmetry is among the 6 faces, and that is so. But what if there really is no obvious symmetry at all, for example if the die is weighted in some unfair way? In that case we would have to rely on a bunch of details such as the precise way the die was thrown by the hand that threw it, the wind, maybe even the surface that the die lands of (can it bounce?). As is often the case, we have to think of the system in question as part of a bigger system. But what about the bigger system? How do we assign its probabilities. Here is another idea that involves some dynamics. Suppose there is a law of motion (in this example time is discrete) that takes a configuration and in the next instant replaces is by another unique configuration. For example R B, B Y, Y G, G O, O P, P R. I can then ask what fraction of the time does that die spend in each configuration? The answer is 1/6. In fact there are many possible laws for which the answer will be the same. For example, R B, B G, G P, P O, O Y, Y R or R Y, Y P, P G, G O, O B, B R. But what about the law R B, B G, G R, P O, O Y, Y P. In this case there are two trajectories through the space of states. If we are oN one of them we dont jump to the other. So the probability will depend on the relative probability for beginning on the two cycles. In this last case there is a conserved quantity. Suppose we assign the number 1 to R,B,G and 0 to O,Y,P. Lets call this quantity the Zilch. Obviously Zilch is conserved. Whenever we have a conserved quantity like Zilch, either we have to specify its value. Thats ok, there are conserved quantities in nature, the most important in statistical mechanics being energy. But once we specify all the conserved Zilches, we can proceed as usual, and say that all the states on a trajectory of fixed Zilch-numbers are equally probable. That sounds good, but there are lots of counterexamples. Here is one. R R, B R, G R, P O R, Y R. No matter where you begin, you go to red in the next instant. There are no R,

conserved quantities but obviously the probability after a short time is completely unequal; only red is possible.

conservation of distinction, or conservation of information.

This last law has something odd about it. Is an example of a law that does not respect the conservation of distinction, or conservation of information. In the previous examples distinct starting points always lead to distinct outcomes. But in this case we quickly lose track of where we started. Trajectories merge! One of the central principles of classical mechanics (it has a quantum analog) is that information (distinctions) is conserved. This principle is so important to the validity of stat mech and thermodynamics that I would call it the minus first law of thermo (the zeroth law is already taken.) What the conservation of distinctions says is that trajectories in phase space never run into each other. Even stronger, if you start with a volume in phase space and follow it using the Hamiltons equations of motion, the volume is conserved. That suggests that for a closed system, a priori probability is uniform over the phase space. We will come back to this.

1.2 The 1st Law of Thermodynamics: Energy is conserved

The First Law of Thermodynamics is that energy is conserved. For a closed system (note: 1), dE/dt = 0 If a system is composed of two very weakly interacting subsystems (note: 2), the energy is additive, E = E1 + E2 Interactions between the subsystems can exchange energy between them. Thus the energy of either snbsystem is not conserved. In equilibrium the energy of a subsystem fluctuates although the total energy is fixed. Note 1: By a closed system we mean one with a time independent Hamiltonian and with insulating walls so that no energy can enter or leave the system from outside Note 2: Weakly interacting means that the energy of interaction is very small compared to the separate energies of the subsystems by themselves. If the Hamiltonian is H1 + H2 + Hint then the nnmerical value of Hint should be negligible compared with either H1 or H2.

1.3 Entropy is an information-theoretic concept.

Entropy is an information-theoretic concept. It is a measure of how much we DONT know about a system. Our ignorance may be due to the fact that the degrees of freedom are too small, numerous, and rapidly changing, or just plain laziness. In either case entropy measures our ignorance. Suppose a system can be in any of N states. If we know nothing about the state of the system (complete ignorance) the entropy, S, is defined to be S = logN Suppose that we know only that the system is an any of M states with M < N. Then we have some nontrivial knowledge. In this case the entropy is defined to be S = log M

Evidently, the more we know, the less the entropy.

Entropy is measured in bits

Here are some examples. Let the system be n distinguishable coins. Each coin is either H (heads), or T (tails). A state consists of a sequence of Hs and Ts.H H T H T T H Suppose we know nothing at all. The total number of states is 2n and the entropy is S = log2n = n log2 Note that by taking the log we have made entropy proportional to the number of degrees of freedom (extensive). Entropy is measured in bits. If the entropy is S = log 2n = n log 2, one says that there are n bits of entropy. Another example is that we know everything that there is to know about the coins. In that case we know the exact state and M = 1. The entropy is zero. Next, suppose we know that all the coins but one are H but we dont know which one is T. The system can be in any of n states and the entropy is S = logn In this case the entropy is not additive but that is because there are strong correlations (to be defined later). Homework Problem: y Suppose that n is even and that we know that half the coins are H and half are T, but that is all we know. What is the entropy? y For large n what is the entropy if we know that a fraction f of the coins are H and fraction 1-f are T?

The general definition of entropy

In each case we are given a probability distribution on the space of states. If we label the states with index i then the probability to be in state i is called P(i). In the examples P(i) is either zero for those states not possible, and P(i) = 1/M for the possible states. We can write a formula for S in terms of P(i), 1.1 The general definition of entropy Those states with P = 0 dont contribute and those with P = 1/M contribute 1/M log M. Since there are M such states the sum gives log M. For a general probability distribution we require only that: 1.2   requirements for a general probability distribution The general definition of entropy is given by 1.1. Obviously 0 P(i)1 which implies that each contribution in 1.1 is positive or zero.

The entropy defined by 1.1 roughly measures the log of the number of states that have nonnegligible probability in the distribution P(i). In other words exp S is the width of the distribution. Note that it is zero if and only if P = 1 for a single state and vanishes for all others. Homework Problems: 1. Suppose you have a set of N coins. Each coin independently has a probability i/3 to be heads and 2/3 to be tails. What is the total entropy? 2. A variable q has a uniformly spaced spectrum of values with very small spacing S. The sum over states can be accurately approximated by: Suppose that the probability distribution for q is proportional to entropy?

What is the

1.4 Temperature
The average energy associated with a probabilty distribution (I will just call it E) is given by 1.3 where Ei is the energy of the state i. Now suppose we have a one-parameter family of probability distributions labeled by the average energy P(i; E). For each value of E P(i; E) satisfies the usual requirements of a probability distribution. Later we will think of it as the thermal equilibrium distribution for given average E. But for now it is just a one-parameter family. At each value of E we can compute the entropy so that S becomes a function of E, S(E) Consider the amount of energy that is needed to increase the entropy by one bit (by log 2). It is given by 1.4 We call the quantity dE/dS the temperature, T. 1.5 Slogan: Apart from a factor of log 2, the temperature is the amount of energy needed to increase the entropy by one bit. For example, if you erase a bit of information from your computer you are really transferring it from the computer out into the atmosphere, where it shows up as some heat. How much heat? The answer is T Iog 2. Except in very unusual circumstances, the temperature is always positive, i,e,. entropy is a monotonically increasing function of energy.

1.5 The 2ndSecond Law of Thermodynamics: Entropy always increases

The second Law of Thermodynamics is that Entropy always increases. We can state it in the following way: When a closed system which is out of thermal equilibrium comes to equilibrium the entropy of the system increases. We will come back to the reasons for this, but for now lets accept it.

From the 2nd law we can prove that heat always flows from hot to cold. Consider two isolated systems , A and B, at different temperatures. Let them have energies, temperatures, and entropies EA, EB, TA, TB, SA, SB Without loss of generality we can assume that TB> TA Now bring them into contact so that energy (heat) flows between them. Suppose a small quantity of energy is exchanged. The total change in energy must be zero. Therefore 1.6  Since they must tend to equilibrium, if the entropy is not maximum it must increase. Hence 1.7  We can use 1.6 to eliminate SB from 1.7. We find 1.8 Since the system B is initially the hotter of the two, (TB TA) is positive. Therefore dSA and also TAdSA are positive. Equation 1.6 then tells us that TBdSB is negative. Equivalently, energy flows from hot to cold as equilibrium is established. The final equilibrium configuration in which energy has stopped flowing, must have TA = TB. In other words, temperature must be uniform in a system in thermal equilibrium.

2 The Bolzmann Distribution

If a system A is in contact with (weakly interaction with) a much larger system called the bath, then it can exchange energy with the bath. After a long time combined system will have jumped around over all states with the given total energy. There will be equal probability for all the various ways the energy can be shared. This is the assumption of chaos and although it is very difficult to prove for a given system, it is almost certainly true for almost all systems. Whenever it is true the small system A will be in thermal equilibrium with a certain probability distribution for being in the state i. (Note that i refers to states of the small system A and not the combined system.) We will illustrate the consequences of this principle by choosing the bath to be N - 1 copies of A so that all together we have a total system consisting of N copies of A.  labels of N copies of system A states of each copy energy of each state of each copy Let us define the occupation numbers n. to be the number of copies that occupy the state i. The ni satisfy two conditions, 2.1 total #copies add up to N 2.2 total energy adds to a fixed value The chaotic assumption tells us that all of the configurations of the total system with a given total energy are equally likely. Thus the probability for a given partitioning of the energy (given set of nI) is equal to the number of configurations with that set of nI. How many distinct configurations of the total system are there with given This is a combinatoric problem that I will leave to you. The answer is ?

2.3

# of distinct configurations of

The important point is that when N and n become large, subject to the constraints 2.1 and 2.2, then the quantity in 2.3 becomes very sharply peaked around some set of occupation numbers. Before we compute the occupation numbers that maximize 2.3, let me introduce some changes of notation. let P(i) to be the fraction of copies in state i. E be the average energy of a copy let Then 2.1 and 2.2 take a form identical to 1.2 and 1.3, namely 2.1a 2.2a Now we will assume that N and n are very large and use Stirlings approximation ( ) 2.3. But first let us take its logarithm (Maximizing a positive quantity is the same as maximizing its log.)  We want to maximize this subject to 2.1 and 2.2. Substituting n/N = P and Etotal = NE we find that this is equivalent to maximizing subject to  and In other words the probability distribution for thermal equilibrium maximizes the entropy subject to the constraint of a given average energy. In order for find the P(i) we use the method of Lagrange multiplier, to Implement the two constraints. The two multipliers are called and . Thus we maximize 2.4 At the end we choose , so that the constraints are satisfied. Differentiating with respect to P(i) and setting the result to zero gives 2.5 This is the Boltzmann distribution. Let us define Then 2.5 has the familiar form 2.6 Boltzmann distribution

2.1 Solving for the Lagrange Multipliers

Solving for is equivalent to solving for Z which is done by setting the sum of the P to 1. This gives the famous formula for the partition function, 2.7 Next consider the equation which becomes

Let us use 2.8

and we obtain

This equation can he used to fix

in terms of the average energy E.

The Lagrange multiplier is, of course, the inverse temperature. We will demonstrate this, but first let us derive another familiar thermodynamic formula.

2.2 Helmholtz Free Energy

Using we can write the entropy in the form

The quantity A = -T log Z is called the Helmholtz free energy (reminder: T = 1/ .) Thus we find or 2.9 The Helmholtz free energy satisfies

Using dE = TdS we find 2.10 2.3 Why is T = 1/ ? We have proposed two definitions of temperature. The first is equation 1.5 (T=dE/dS)the second is the inverse of the Lagrange multiplier (T=1/ ) We would like to see that they are really the same. Consider a small change in the energy of a system,

Now use 2.9 in the form E = ST T logZ to get Using 2.9 again, and the definition of A, we find that the second and third term cancel, leaving 2.11 This is of course equivalent to 1.5. Thus the two definitions of temperature are the same.

3 Fluctuations
So far we have been deriving classical thermodynamics from statistical mechanics. We go beyond thermodynamics when we consider fluctuations of quantities about their averages. Such fluctuations are observablefor example Einsteins theory of the Brownian motion. In this section I will illustrate by considering the fluctuations of the energy of a system in contact with a heat bath. Given a probability distribution P(x) the fluctuation in x (called x)is defined by 3.1 which is also equal to 3.2 where means average (langle rangle) For any function f(x) the average is defined by sum is replaced by integral in the obvious way. . If x is continuous then the

Let us consider the fluctuation of energy of a system in equilibrium. We use the following: and  The first identity is the usual identification of average energy in terms of the derivative of Z. The second identity is derived the same way as the first, noting that each derivative acting on e(- E) brings down a factor of -E. Thus is given by

But this expression is equivalent to Using (T=1/ ) we get

Now note that (dE/dT) the specific heat of the system. Call it C. The final identity is   3.3 Thus we find that the fluctuation of the energy is proportional to the specific heat.

Note on Boltzmann constant k = 1

It may seem odd that the fluctuations should be so large. But that is because we have set the Boltzmann constant to 1. If we put k back into the equation it becomes 3.4 where specific heat of the system Boltzmans constant Since k is a very small number. In units of meters, kilograms, seconds, and degrees, Kelvin, 4 = 1.4 x 10-23. Nevertheless, despite their small magnitude, fluctuations can be measured and they provide a method for determining Boltzmanns constant.

4 Control Parameters
So far we have considered closed systems characterized by constant values of the parameters. These parameters include all the parameters in the Lagrangian such as the masses of particles, the values of external electric and magnetic fields, and the shape and volume of the containers that enclose the system. Some of these parameters such as the volume of the system and the external fields may be controllable from the outside, for example by moving pistons to change the volume. We will call such macroscopic control variables Xm. For simplicity we will consider the case of only one X although the principles are the same for several of them. If you want to think of a specific example, X can represent the volume of the system.

4.1 The Adiabatic Theorem and the First Law

An adiabatic process means two things. 1. First of all it means that the system is isolated so that no energy in the form of heat can flow into or out of the system. 2. Secondly, it means that the control parameters are varied very slowly. In that situation the system will remain in equilibrium throughout the process, although typically the energy and temperature will change.

Adiabatic Invariants Entropy

Those things which do not change during a adiabatic process are called adiabatic invariants. It is a theorem that entropy is an adiabatic invariant. It is true in classical mechanics but it is most easily understood in quantum mechanics. Consider the energy levels of the system Ei. In general they will depend on X: Ei = Ei (X). y In general if a sudden change is made in X the system will not remain in an eigenstate of energy. y But if the change is arbitrarily slow then the adiabatic theorem says that the system remains in an eigenstate, simply tracking the slow time dependence of the instantaneons energy levels.

In fact the levels will not cross over or disappear, if the change is adiabatc The implication is that the probability function P(i) is constant for each level, even as the energy along the way gradually varies. y Obvionsly if P(i) is constant, so is the entropy. That is why entropy is an adiabatic invariant. y y Now consider the change in energy of the system during the adiabatic process. That energy change is by definition, the work done on the system by changing X. The most familiar example is the work done in slowly compressing a gas in a insulated container. If the change in X is small (call it dX) we may assume the work done is small (call it dW). We can express the above idea in equations.
In the general case of several control parameters this becomes
Let us define the conjugate variables, Yn, to the Xn, by the formula
4.1 and it follows that 4.2 The most familiar example of (X, Y) is volume and pressure (V,P).

Let us suppose that an infinitesimal adiabatic change is followed by a second process in which energy is added to the system in the form of heatin other words a second process in which the control parameters are constant but the entropy changes. For this second process dE=TdS so that the combined effect of the work (adiabatic process) and the added heat give a change in energy, 4.3 First Law of Thermodynamics This relation is called the First Law of Thermodynamics, but it is really an expression of energy conservation.

4.2 Processes at Fixed Temperature

We have defined the conjugate variables Y such as pressure in terms of adiabatic or constant entropy processes. We can also define them in terms of constant temperature (isothermal). Such processes usually mean that the system is in contact with a large heat bath which is so big that its temperature does not change. First we need a calculus theorem that I leave for you to prove. Suppose we have a function S(T, X) of two variables T and X. Let there also be a second function E(T, X). By solving for T in terms of S, X, we can think of E as a function of S and X.

Homework Problem: Prove the following identity:



This identity is general but in the case where S, T, E have their usual thermodynamic meaning, we can use

to get

Finally, defining this can be written

Part II Some Simple Applications 5 Ideal Gas

Consider a container of volume V containing N identical point particles, each with mass m. In order to apply the reasoning of Part I we must know how to define the state of the system and how to sum over states. Begin with a single particle. The state of a classical system is described by a point in its phase space, i.e., a value for each generalized coordinate and each generalized momentum. Thus in the present case a value for each of 3N coordinates restricted to the interior of the container, and 3N momenta (unrestricted). To sum over states we imagine replacing the phase space by a discrete collection of small cells. The phase-space-volume (units of momentum times length to the power 3N) of each cell tends to zero at the end of the calculation. Quantum mechanics suggests that we take it to be ( )3N . Thus we replace the sum over states by 5.0 An ideal gas is one composed of non-interacting particles. The energy of a state at point x,p is 5.1 and the partition function is 5.2 The factor 1/N! is put in to avoid over counting configurations of identical particles. For example if there are two particles, there is no difference between the configuration particle- one at point x.p, particle-two at x.p and the configuration particle-two at point x.p, particle-one at x.p. Thus we divide by the number of equivalent configurations which in this case is 2 and in the general case is N!. We will use Stirlings approximation, N! = NNe-Nin evaluating the partition function.

The x integrals in 5.2 are trivial since the integrand does not depend on x. It just gives V3N When combined with the 1/N! it gives

Since N/V is the particle density, call it p these factors combine to give

Notice how the N and V dependence nicely combine to give an expression which only depends on the density which we will keep fixed as the nnmber of particles tends to infinity. The momentum integral in 5.2 is a gaussian integral over 3N variables. In fact it is the 3N power of the one dimensional integral  The final result for the partition function is 5.3 If we want to explicitly exhibit the dependence on volume we replace p by N/V. 5.4 Homework Problem: y Given the partition function in 5.4, compute the A, E, S, P as functions of the temperature. Derive the energy per particle and the ideal gas law P = pT. y What is the average speed (magnitude of velocity) of a particle? y On a PV diagram (pressure on one axis, volume on the other) what are the curves of constant temperature (isotherms) and constant entropy (adiabats)?