Mathematical Preparation for Finance

A wild ride through mathematics

Kaisa Taipale

Contents

- Preface

- Chapter 1 Introduction to probability

- 1.1 Brief introduction to sets

- 1.1.1 Sizes of sets via bijections

- 1.1.2 “Paradoxes” of set theory

- 1.1.3 Notation for set theory

- 1.2 Axioms of probability

- 1.2.1 Definitions

- 1.3 Counting

- 1.3.1 Triangular numbers

- 1.3.2 Factorials

- 1.3.3 Combinations and permutations

- 1.4 Binomial theorem

- 1.5 Geometric and arithmetic series

- 1.6 Binomial trees

- 1.7 Continuity property

- 1.8 Compound experiments

- Chapter 2 Geometry problems in probability

- 2.1 Function families

- 2.2 Derivatives and integrals

- 2.2.1 Volume and area

- 2.2.2 Differentiation as infinitesimal approximation

- 2.2.3 Average rate of change and Mean Value Theorem

- 2.2.4 Integration and its disguises

- 2.3 Buffon’s needle problem

- 2.4 Stick-breaking problems

- Chapter 3 Probability rules

- 3.1 Basic rules of probability

- 3.1.1 Complements and inclusion-exclusion

- 3.1.2 Example of inclusion-exclusion principle

- 3.2 Conditional probability

- 3.3 Bayes’ theorem

- Chapter 4 First applications in finance

- 4.1 Binomial stock pricing

- 4.1.1 Additive model

- 4.1.2 Multiplicative model

- 4.2 Arbitrage

- 4.3 Short- and long-term approximation

- 4.3.1 Long-term approximation

- 4.3.2 First differential equation

- Chapter 5 Discrete random variables and transformations of variables

- 5.1 Discrete random variables

- 5.1.1 Linearity of expectation – best thing ever

- 5.1.2 Multiplicativity of expectation?

- 5.2 Variance: average of squared deviations

- 5.3 Series

- 5.3.1 Convergence, divergence, well-defined

- 5.4 Moment generating functions

- 5.5 Linear and affine linear transformations

- 5.6 Specific discrete random distributions

- 5.6.1 Bernoulli

- 5.6.2 Binomial

- 5.6.3 Geometric

- 5.6.4 Hypergeometric

- 5.6.5 Poisson

- 5.6.6 Negative binomial

- 5.6.7 Transformations of discrete random variables

- Chapter 6 Continuous Random Variables

- 6.1 Geometry problems

- 6.1.1 Max or min in unit square

- 6.1.2 Absolute values

- 6.1.3 Circles: you mean I need to remember trig substitution?

- 6.2 Expected value and variance

- 6.3 Transformations of continuous random variables

- 6.4 Markov and Chebyshev

- 6.5 Important distributions

- 6.5.1 Exponential distribution and Poisson

- 6.5.2 Normal distribution and lognormal distribution

- 6.6 Central Limit Theorem

- Chapter 7 Random walks

- 7.1 Simple symmetric random walk

- 7.1.1 Expectation and variance

- 7.2 Asymmetric random walks

- 7.2.1 Expected value and variance

- 7.3 Scaling time or space

- 7.3.1 Special properties

- 7.4 Arithmetic Brownian motion

- 7.5 Geometric Brownian motion

- 7.6 Solving Brownian motion problems

- Chapter 8 Linear algebra

- 8.1 What is a vector? What is a matrix?

- 8.2 Linear combinations, matrix multiplication, and planes

- 8.2.1 Linear combinations

- 8.2.2 Matrix multiplication and dot products

- 8.2.3 Planes and parameterizations

- 8.3 Useful inequalities for vectors

- 8.4 Some vectors in finance

- 8.5 Linear transformations

- 8.6 Bases

- 8.7 Applications to financial math

- 8.8 Invertible transformations

- Chapter 9 Spectral theorem and portfolio management

- 9.1 Orthogonal matrices and orthonormal bases

- 9.2 Gram-Schmidt orthogonalization

- 9.3 Rotation and scaling

- 9.4 Complex Numbers

- 9.4.1 Taylor series

- 9.5 Changes of basis and coordinates

- 9.5.1 Changing basis alone

- 9.5.2 Changing linear transformations into a new basis

- 9.6 Eigenvalues and eigenvectors

- 9.7 Quadratic forms and definiteness

- 9.8 Power series of matrices

- 9.9 Applications to financial math

- 9.10 Complex vectors

- 9.11 Complex matrices

- 9.12 The spectral theorem

- 9.13 Singular value decomposition

- 9.14 Applications of SVD

- Chapter 10 Joint distributions

- 10.1 Jointly distributed discrete random variables

- 10.1.1 Marginal probability mass function

- 10.1.2 Examples of discrete jointly distributed random variables

- 10.1.3 Conditional probability mass function

- 10.1.4 Independence

- 10.1.5 Multivariate versions

- 10.2 Jointly distributed continuous random variables

- 10.2.1 Marginal and conditional probability density functions

- 10.3 Covariance and correlation, again

- 10.4 Multivariate change of variables

- 10.5 Bivariate and multivariate normal

- 10.6 Visualizing the bivariate normal distribution

- Chapter 11 Optimization and Newton’s method

- 11.1 Single-variable Unconstrained optimization

- 11.1.1 Single-variable minima and maxima

- 11.1.2 Single-variable constrained optimization

- 11.2 Newton’s method, single-variable

- 11.2.1 Newton’s method for single-variable optimization

- 11.3 Multivariable Taylor approximations

- 11.3.1 Zeroes of multivariate functions

- 11.4 Multivariate optimization

- 11.4.1 Unconstrained optimization

- 11.4.2 Multivariate Newton’s method for optimization

- 11.4.3 Constrained optimization

- Chapter 12 Differential equations

- 12.1 Equilibrium: the concept

- 12.2 A single ordinary differential equation

- 12.3 So I can model a caribou population: what about money?

- 12.4 Systems of Differential Equations

- 12.5 Equilibria

- 12.6 Straight-line solutions

- 12.7 Back to Black-Scholes for a minute

Preface

This book is being written for students in my Preparation for Financial Mathematics class at the University of Minnesota.  It’s an idiosyncratic tour through probability, calculus, and linear algebra, attempting to mix in financial applications throughout.  Unlike many authors, I am making no attempt whatsoever to write a “self-contained” exposition: I’m assuming you know some calculus and linear algebra and are willing to look up a lot.  Perhaps it is better to view this as a workbook than a reference book.

My students next take a course, FM 5011, in which probability is presented in terms of measure theory.  You’ll deal with the sigma-algebra that consists of sets of events and you’ll put a probability measure on that -algebra.  Then, in the rest of the first day, you’ll proceed through probability distributions for random variables, noting the binomial, Poisson, uniform, and normal distributions among others.  You’ll review joint distributions and expectations.  That’s lecture one of 5011; by lecture two you’ll cover stochastic processes.

In this book, we will take a more non-technical approach and get our hands dirty.  The aim is that you will understand all those distributions, and have a beginning familiarity with Brownian motion, so that you can assimilate the technical notions presented in 5011.

This book has benefited from the feedback of my students through several years: in essentially random order, I’d like to thank them here.  In 2017-2018, Yanqiu Tan, Tianzi Guo, Thara Ali Said, Jacob Gotto, David Rokhinson, Ameya Phadke, and Alec Hamer all made suggestions for improvement.  In previous years, Joseph Ogega, Chong Wang, and Guining Zhang made suggestions – and since they’re all graduated, you should hire them for great math finance jobs.  In 2018-2019, I need to thank Boris Alyurov and Michele Knudsen.

Chapter 1

Introduction to probability

We’re plunging right into probability.  Several generations of students have now found that probability is subtle, confusing, and requires a different mindset than calculus and linear algebra.  You want a full year to absorb these subtleties!

Broadly, our goal for this semester is to gain a firm foundation in basic probability so that we can proceed to understanding geometric Brownian motion, which is widely used as a basic model for stock prices.  At the end of the semester (end of chapter 7) I want you to understand the transition from random walk to Brownian motion, and how these relate to the Black-Scholes differential equation and the binomial tree model for option pricing.  The goal for this first chapter is to get you acquainted with the basics of probability, grounding you in the idea of looking at a sample space and finding the probability of an event occuring in this sample space when you conduct a chance experiment.  You will get a lot of practice with such problems.

By the end of this chapter, I want you to understand the axioms of probability:

Definition 0.1 The probability measure on a sample space is denoted by .  It assigns to each set in the sample space a probability, satisfying the following axioms:

- for each subset .

- when is equal to the sample space.

- for every collection of pairwise disjoint subsets .

The set notation is explained in Section 1.1 and the probability terminology is explained in Section 1.2.

I also want you to be able to solve problems like the following:

Eight teams are in the semifinals of an international badminton tournament.  The eight teams consist of two teams each from China, India, Denmark, and Korea.  What is the probability that the two teams from each country end up playing against each other in each of the semifinal matches?  That is, China 1 plays China 2, Denmark 1 plays Denmark 2, etc.

For these, you’ll need the counting techniques outlined in Sections 1.3 and 1.4.

1.1 Brief introduction to sets

1.1.1 Sizes of sets via bijections

A finite set is one that has a positive integer number of elements.  These are the sets we got comfortable with in kindergarten and first grade: five apples, seven chairs, a million dollars.  (Well...)  Another way to say that a set is finite is to say that its elements are in bijection with a set for a positive integer.  A “bijection” is a matching that pairs each element in with one and exactly one element in .  Then we use the notation for cardinality of a set to write that the size of is .  (Note that this is the same notation as absolute value and magnitude, so you must understand the context of the statement to correctly interpret .)

More generally, a bijection between two sets and is a matching (function, map) that is injective and surjective.  A map is injective, or “one-to-one,” if in means that in .  This means no two distinct elements in can map to the same element in .  A map is surjective, or “onto,” if for every there’s an such that .  This means every element in is “hit” by the map from the set .

Why make things so complicated?  The concept of bijection is useful in talking about sizes of infinity.  For instance, the set of natural numbers is a certain “size” of infinity.  We call this size of infinity countable, because we can count through it.  Any other set we can put in bijection with the counting numbers will also be called countably infinite.

Put the integers in bijection with .  You can do it!  Can you reconcile your bijection with your intuition that there should be somehow twice as many integers as natural numbers?  Infinity is mysterious.

Another “size” of infinity comes up when looking at uncountable sets.  How many real number are there?  Can you put the real numbers in bijection with the natural numbers ?

Prove to yourself that you can’t put and in bijection.

The set of real numbers is an uncountable set, and so is the set of irrational numbers – let’s call that for now.  (Irrational numbers are numbers that can’t be expressed as a ratio of two integers.  Examples include and .  The set of rational numbers is denoted by .)

With uncountable sets like the real numbers we can construct very weird subsets that can’t be assigned a probability measure consistent with how probability should work.  Basically, we can break the intuitive notion we have of the size or “measure” of a set: if we proceed naively, we could have a set that looks like it’s of size 1 and size 0 at the same time, in which case and all is lost.  That’s why -algebras are introduced in later classes: using -algebras and more sophisticated ideas we can formalize the notion of measure and make sure probability works over uncountable sets.  Our approach in this book: ignore this problem.  Let’s stick with “nice” subsets of uncountable spaces like intervals and their complements.  But before we stop considering complicated sets forever, I’ll throw out a few situations that seem paradoxical, to show you why you’ll need another year of math to truly understand measure theory.

1.1.2 “Paradoxes” of set theory

Above, I asked you to put in bijection with .  Hopefully you did that!  Now, think about the rational numbers, .  This is a countable set, as you can see from the picture below:

In particular, that means that the set of rational numbers between zero and one is countable.  Let’s consider the cardinality of the set of all real numbers between zero and one.  It’s uncountable, and here’s how to prove it.  I use what’s called Cantor’s diagonalization argument.  Assume you can list all the numbers between zero and one, writing them out as decimals.  Put that list together: A clever friend comes along and says, “I can give you a number between zero and one that is not on that list!”  You say, “Impossible – I listed them all!”  She says, “No: I’ll give you a number whose th digit after the decimal is one different than the th digit of the th number on your list, for every digit.”  You are flabbergasted...  but she is right!  The number she gives you is between zero and one and is not on your list, no matter what was on your list.

This proves that is an uncountable set of numbers.  Moreover, it proves is uncountable.  But we know that the length of the interval is one.  Somehow we have a countable set of rational numbers between zero and one and an uncountable set of irrational numbers between zero and one and their “sizes” add to one.  It turns out that the measure of is one and the measure of is zero.

From this example you might get the idea that countable sets are always of measure zero or that uncountable sets have positive measure.  Nope.  Here’s another puzzle, again involving the mathematician Georg Cantor:

We can define a set called the Cantor middle-thirds set.  Take the interval and remove the middle interval .  Then from the two remaining intervals, remove the middle third.  Keep removing the middle third of each remaining interval (forever).  You’re left with a dusting of points.  How big is it?  Well, what do you mean by “how big”?

The “length” of the interval (its measure) can be found by using a geometric series.  You start with an interval of length 1, take out 1/3, take out 2/9, take out 4/27....

So this is a set of measure zero (again, we have no rigorous definition here).  But how many elements does it have?

Take a look at the picture below to see a way to label each point in the set: This gives a ternary representation of each point – ternary means “base three,” as opposed to binary or decimal.  For each step of the middle-interval-subtraction, write 0 if the point is in the left interval, 1 if it is in the middle, and 2 if the point is in the right interval.  (These numbers are in green in the picture.)  The points in the Cantor set then consist of every number that has an expression only in 0s and 2s.  For example, and , while starts with and so is not in the Cantor set.  Now you can use Cantor’s diagonalization argument to prove is uncountable: try to make a list of every number in the Cantor set and have your clever friend come by and give you a number that is in the Cantor set but not on your list!

The Cantor middle-thirds set, then, is uncountable but of measure zero.  What’s the probability of choosing a point in the Cantor set if you choose a number randomly between zero and one?  That’s for next year!

The bottom line, for us: we will only consider sets that are discrete and of measure zero, or sets that are made from intervals in .  An interval or or for has measure and an uncountable number of points, and we don’t have to deal with any paradoxes.

1.1.3 Notation for set theory

We express that is an element of the set by writing .  As you may have noticed above, we often use curly brackets , to indicate we are discussing a set: is the set of integers one through , while would be the set of numbers that satisfy the condition that is an integer.  Read the sign (or, in some books, the : sign) as “such that.”  There are often many ways to write the same set: , , or for .

You will need to be comfortable with the interactions of sets as well as finding the cardinality of a given set or the measure of an “easy” set (one consisting of intervals).  Unions and intersections of sets are the two big concepts.  The union of two sets is the set of all elements in either set: .  The intersection of sets is what you think it is: .  Visually, this is

If you have a countably infinite number of sets, like , you can write

Two sets are disjoint if they don’t overlap – if their intersection is empty.  We can write , using to represent the empty set.

We also write for the complement of a set .  This is the set of all elements that are not in .  For instance, if and , then contains the elements in that are less than zero: .  Notice that .

Last, we have notation for set “subtraction.”  We can pick out all elements of a set that are not also in by writing .

Many of these concepts will be revisited in section 3.1.  This should be enough to get us started, though, and as you work with probability problems you will start to see why set theory is so useful.

1.2 Axioms of probability

1.2.1 Definitions

When you conduct a probability experiment or a chance experiment, you toss the die or flip the coin or throw the dart – whatever it may be.  Then an event occurs: the outcome of your experiment.  The set of all possible outcomes of your experiment is called the sample space, , and an outcome in the sample space will be denoted .

A sample space with a collection of events and an assignment of probabilities to the events is called a probability space.  The probability measure on a sample space is denoted by .  It assigns to each set in the sample space a probability, satisfying the following axioms:

- for each subset .

- when is equal to the sample space.

- for every collection of pairwise disjoint subsets .

Notice that the set can be a single event or a more.  For instance, you can consider the event of rolling a six when you roll a die (set of size one) or the event of rolling an odd number (set of size three).

A probability measure on is defined by assigning a probability to every event in a finite or countably infinite , so that and , and letting

Some specific examples and terminology: Two events are mutually exclusive if they can’t both happen.  For instance, you can’t roll a two and a five on the same roll of a die.  You can’t flip both heads and tails when you flip a coin.

A lot of the problems you’ll encounter in the next few pages will feature finitely many outcomes which are all equally likely.  Then by the axioms of probability you can deduce that for and each event has probability

where is the number of outcomes in set .

1.3 Counting

Let’s go back to the questions asked at the beginning of the section:

Eight teams are in the semifinals of an international badminton tournament.  The eight teams consist of two teams each from China, India, Denmark, and Korea.  What is the probability that the two teams from each country end up playing against each other in each of the semifinal matches?  That is, China 1 plays China 2, Denmark 1 plays Denmark 2, etc.

In one throw, you roll two dice.  You win if the sum of the numbers on the two dice is 8 or more.  What’s the probability that you win?

How can you solve these problems?  You probably have an intuitive approach, but I am going to push you to try a systematic and algorithmic approach.  Learning this systematic approach will help you solve many probability problems, I promise.  Here it is:

If each individual event in a finite sample space is equally likely,

1. Identify the sample space precisely and mathematically.

2. Write out the event or set of events whose probability you want.  Use the same notation and setup as you used when identifying the sample space .

3. Calculate the size of and the size of .

4. Divide: .

To do this, you need to know how to count.

1.3.1 Triangular numbers

We’ll start with the dice problem and work through the problem-solving algorithm I suggested.

Step one: the sample space, the set of all possible outcomes of the experiment, is the set of combinations of numbers on the two dice.  To be careful, I’ll make one a red die and one a blue die – I want to keep track of them.  Why?  If the outcomes are written (number on red die, number on blue die), then and are both possible outcomes.  If you don’t keep track of the dice, you may think that the combination of 1 and 2 is as likely as the combination of 3 and 3, instead of being twice as likely.

So, , where is the number on the red die and is the number on the blue die.

Step two: identify the events that give us a “win.”  We can write this as , and draw both and :

The elements of the set are highlighted with yellow.

Step three: How big is ?  See our previous drawing.  has 36 elements, all equally likely, so .

How do we find ?  You see the triangle of size highlighted with yellow in the sketch of .  You can count by hand, or you can use the wonder of triangular numbers.  The triangular numbers are

Figure 1.1

I want a formula for these numbers so that I don’t need to count by hand.  Look at and do a demonstration that can be made into a proof:

Figure 1.2

This picture proof is easily made rigorous to give

Step four:

1.3.2 Factorials

For many counting problems we need to know the number of possible orderings of a set of objects.  Factorials are the building blocks of ordered counting.  Say you need to know how many distinct five-letter strings can be made from the set of letters , using each letter once.  (In computer science a string is a sequence of characters.)  It’s clear that you have five choices for the first letter of the string, four choices remaining for the second letter, three for the third, and so on.  Counting in this way you could make

different strings from without repeating any letters.  We read this as “five factorial.”  For a positive integer ,

and by convention we say . On the vocabulary side, we have just computed the number of permutations of the set .  Notice I didn’t say “ordered permutations” – permutations are by definition ordered.

Now let’s try the badminton problem:

Eight teams are in the semifinals of an international badminton tournament.  The eight teams consist of two teams each from China, India, Denmark, and Korea.  What is the probability that the two teams from each country end up playing against each other in each of the semifinal matches?  That is, China 1 plays China 2, Denmark 1 plays Denmark 2, etc.

Step 1: Identify the sample space precisely and mathematically.  First, come up with your own sample space.

There are several possibilities for the sample space, but I will use all 8-tuples of teams – that is, all strings that can be created from

Order matters because in this set-up I am assuming that the first team named plays the second, the third plays the fourth, and so on.  I will write this as an 8-tuple, where the are all distinct teams.

(Compare this to the choice you made.  What do you think?)

Step 2: Write out the event or set of events whose probability you want.  Use the same notation and setup as you used when identifying the sample space .  We’re looking for the probability that each country plays itself in the semifinals, and we are looking for 8-tuples that satisfy this outcome condition.  Examples of this include and .  Thus is the set of 8-tuples in with and from the same country, and from the same country, and from the same country, and and from the same country.

Step 3: Calculate the size of and the size of .  The size of is easy:

The size of is more subtle: you have 8 choices for the first team and 1 choice for the second (if the first team is K1, the second must be K2 and vice versa).  Then 6 choices for the third team and 1 for the fourth.  Continue on to get

Another way to think about the size of : there are orders for the countries within the 8-tuple and given every country order ( or ) there are ways of ordering the teams ( then versus then ).  This gives immediately.

Step 4: Divide: .  Thus

A problem-solving note: you might argue that there are just four games, so counting as a different event than is redundant.  On a physical level this is true, but it turns out that “overcounting” in our sample space helps us organize events nicely and we cancel out the “overcount” in our characterization of .  Sometimes it’s much easier to use some seeming “overcount” rather than the most parsimonious presentation of the events.

1.3.3 Combinations and permutations

What if we wanted instead ordered subsets of items?  For instance, all the possible 4-tuples of badminton teams that would go to the quarterfinals, or all the distinct two-letter strings we could make by picking two elements from without repetition?  This second example involves picking two elements in order: for the first character in the string we have five choices, for the second we have four choices.  We write this number

of which I prefer , or in China we call it an arrangement instead and write

Generalize this to creating a -letter string from distinct elements: count down from choices for the first letter to choices for the last of the letters.  We get

Combinations instead refer to selections of items from for which order does not matter.  Imagine that you grab two socks from a drawer full of socks and you want to see if they are a pair.  Or you could choose three letters from and not care about their order.  We know how to count the distinct strings we get by choosing three letters in order (it’s ) and now just need to divide by orderings of the three letters (a factorial, ).  Thus the number of ways to grab three letters from is

Here I point out a warning: I’ve also seen , , , , and .  This is horrible, as it seems that any combination of , 5, and 3 is used somewhere.  DON’T USE THESE! Simply use the binomial notation .  It is unambiguous and best.  We set notation for the number of ways to pick items (unordered) from items to be

Take this as your definition, and check out Section 1.4 to see why I called this binomial notation.

1.4 Binomial theorem

The coefficients of the expansion of the innocent little expression , for an integer, will appear so many times in the rest of your life that you will be astounded.  These numbers are truly essential in doing probability or financial mathematics.

Expand a few instances of where is a positive integer: you’ll start to notice a pattern.  In school you may have learned that you can cleverly get these coefficients via Pascal’s triangle.  First, Pascal’s triangle alone:

You may remember that expands nicely as

as long as is a positive integer.  Since we defined , we have .  Here is how this compares with Pascal’s triangle:

Pause here to ponder more deeply, and experiement with these problems:

Specialize to the situation , : what does this imply about the relationship between binomial coefficients and ?

How is related to ?

Consider flipping a fair coin times.  What is the probability that you get heads and tails?

If you evaluate at (that’s what I mean when I say “specialize” – it means “look at the special case”), you get .  Looking at that with the binomial expansion, you see that

Knowing that the sum of binomial coefficients from to is actually comes in quite handy when you’re looking at random walks and coin-flipping problems.

Grabbing items out of items is the same as leaving items behind, so .  Thinking about what you want and what you don’t want is a great problem-solving tool – it sometimes lets you flip a problem into a more tractable problem.

It’s easy to compute how many ways you can get heads out of tosses – that’s our binomial coefficient!  So there are ways to get heads and tails from total tosses.  That’s not a probability, though.  How many total patterns of heads and tails are there?  The sum over all of is as we just saw above, so the size of the sample space of all coin flip sequences is and thus the probability of exactly heads is

1.5 Geometric and arithmetic series

You may remember arithmetic and geometric series from algebra class sometime in the past.  First we’ll review their definitions and some terminology and then I’ll justify why we are talking about them in a probability and finance context.

An arithmetic sequence is a sequence with a constant difference between each two consecutive terms (so for all ).  An arithmetic series then is the sum of such terms,

The triangular numbers discussed earlier are one of the most basic arithmetic sequences, since you’re looking at sums of , , (so ).

Arithmetic sequences and series mainly come up as a tool in probability problems – you find you need to add up a bunch of numbers all separated by 37, and then you do it.  In finance, true arithmetic sequences come up mainly when considering constant payouts (if you’re not considering the time value of money).  Many financial instruments give a constant payout or demand a recurring constant fee, and if you’re ignoring the time value of money you might consider using an arithmetic sequence to model this.  In this era of near-zero interest rates, using an arithmetic sequence to model my monthly Crossfit gym payments is reasonable in the short term.

A geometric sequence is a sequence with a constant ratio between each two consecutive terms (so for all ).  The geometric sum is then

You’ve probably encountered this when looking at compound interest – classic problems you might remember doing would involve compounding monthly, daily, etc.  In each of those situations, you are multiplying your current principal by for the appropriate interest rate .  If you need an infinite number of terms in your sum, you get

Geometric sequences also come up a lot in probability problems, and in fact there’s a probability distribution called a “geometric distribution”.  What’s the probability that you need exactly rolls of a die until you roll the number 6 for the first time?  It’s

What’s the probability that it will take less than five rolls to get that 6 for the first time?  You’ll sum up these probabilities (since they correspond to mutually exclusive events) and that’ll be a geometric series.

There are nice formulas for sums of these sequences.  Derive them:

You know that

Now use what you know about triangular numbers and sums of constants to come up with a closed formula for this sum, depending on , , and .

You know that

I’m calling this sum for a reason.  Do something clever here: compare and .  In fact, look at .  Write out the terms of , then solve for .  This will give you a nice closed form presentation for .

Terminology break: “closed form” and “closed formula” mean a function where you can just plug in a few numbers and directly compute an answer, as opposed to something recursive, or a formula with a limit in it, or where you have to do an infinite number of operations.  Example: you could get an expression like

which as written implies you’ve got to sum up an infinite number of terms.  On the other hand, you could use geometric series to work out that

The expression is something you can do in one “move”.

1.6 Binomial trees

Now let’s put into financial action a bit of what you’ve learned.  We’ll talk about stock movements here, foreshadowing random walks and Brownian motion.  Make sure you understand every detail here, as these foundational ideas do need to be foundational if you want to do financial math!

Consider a stock that either goes up in price or down in price with equal probability.  What is the probability that it goes up times and down times?

A few ingredients to consider: the probability the stock goes up is 1/2; the probability the stock goes down is 1/2; the number of paths with up-steps and down-steps is .  So the probability of going up exactly times (or down exactly times) is

Consider a stock whose price goes up by ten dollars or goes down by ten dollars, with equal probability, on any given day.  Its initial price is .  If in days it has gone up times and gone down times, what is its final price?  Since a stock price can’t be negative, what is the probability that the stock price is zero?

Write for the price in dollars on day :

What’s the probability of this price?  If we don’t care about negative prices, the probability of price on day is just

from the previous problem.  However, stock prices can’t go negative (although you can short stocks).

The real-life probability of a stock price of zero is basically zero, as you can’t buy and sell stocks with share price zero.  Even if a company goes bankrupt and stock is still bought and sold, it’s not at a price of zero – Blockbuster Video’s stock traded at prices from 4 to 23 cents a share for a while after it went into bankruptcy.  When stock prices fall to near zero they’re commonly delisted from stock markets, and then move to over-the-counter pink sheets (a very weird market).

This is a situation in which our model and reality conflict.  An additive model of stock prices gives plenty of situations in which stock prices go to zero or negative numbers.  If you use such a model, you’ll need to decide if you want to take for your price on day , and whether the probability of the stock price going to zero really reflects the probability that the company will go bankrupt.

Consider a stock whose price goes up by ten percent or goes down by ten percent, with equal probability, on any given day.  Its initial price is .  If in days it has gone up times and gone down times, what is its final price?  What is the probability that the stock’s price is zero after days?

In this multiplicative model, the price on day is

Since we’re multiplying a positive number by more positive numbers (percent increase/decrease), will never be zero in this model!

1.7 Continuity property

“Probability is a continuous set function.”  What does that mean?  Why do we care?

Our short discussion here will be a foreshadowing of measure theory, a topic not covered in this book!  On a first reading, you might choose to skip this section – but you may find it useful to ponder the complexities of this topic.

Let

be a nondecreasing sequence of sets.  For the union , use the notation .  Then the continuity property is

Essentially, the theorem is that you can interchange the probability function and the limit.  Limits characterize continuity, which is why this is called the continuity property of probability.  You can also translate this result to a non-increasing sequence, in which case .

What is an example of a set of sets like this?  Here’s an example made up for this situation – you probably won’t encounter it in real life.  Say you’ve got a game of darts that keeps getting tougher, and you’re terrible at darts.  The dart board is a disk of radius one foot; we’ll assume you can hit the board with a dart every time but your aim is random on the dart board (every point is equally likely).  Here are my made-up rules: On the first round, you need to hit the center of the disk, a region with radius 1/2 foot, to score.  To score on the second round, you need to hit the disk in the middle with radius 1/4.  To score on the third round, you need to hit the disk with radius 1/8.  To score on the th round, you need to hit the disk with radius .  Since you are so bad at darts, the probability of hitting the circular region with radius feet, , is

What’s the probability you score as the radius goes to zero?  This probability is zero: the probability of scoring on round , , is .  Since

we know

A related question is this: what’s the probability that you score infinitely many times?  That’s also zero.  The Borel-Cantelli Lemma gives this extension:

Let be an infinite sequence of subsets of the sample space .  Define the set as the set of events that occur in infinitely many .  Then if

we have

So what’s the probability that you score infinitely many times in my made-up dart game?  Zero.

You can also get from this a corollary about things that always happen instead of never happening. Let be an infinite sequence of subsets of the sample space , all of which are independent.  Define the set as the set of events that occur in all for , for some , and such that doesn’t occur in only finitely many .  Then if

we have

For an example of this type of situation, consider some poor student doomed to toss a coin forever.  Let’s say the event occurs if in the th block of 100 coin tosses, you get 100 heads in a row.  Each block of 100 coin tosses is independent of every other block of 100 coin tosses, and .  We can see that

so by the lemma .  That means that with probability one you’ll get a run of 100 heads in an infinite series of coin tosses.

Don’t ignore this topic even though this section seems very small.  Turns out the Borel-Cantelli lemma comes up again when we look at Brownian motion.

1.8 Compound experiments

A compound experiment consists of several elementary experiments that are independent of each other.  For instance, if you toss three coins that are physically independent from each other, you’re conducting a compound probability experiment.

We’ll revisit the concept of independence when we discuss random variables.  For now, use your intuitive understanding of the idea: two events are independent if they don’t influence each other.

The probability of and is

as long as the events in are physically independent of the events in .  Notice that some sources use the notation to mean the intersection of events and , and we will use this notation too.

Notice that the formula itself has no reference to a sample space here.  Why not?  We may mix and match sample spaces when discussing independent experiments.  For instance, if we want to know the probability of rolling a three on a six-sided and getting heads on a coin toss, we could look at the sample space and do a counting-based argument, or we could use the rule above to find

using independent calculations from the sample space of possible die rolls and coin tosses.

Another version of a compound experiment involves repeated independent probability experiments.  Here are three different examples:

(A) Toss a coin seven times in a row.  What is the probability that you get the sequence ?

(B) You take turns tossing a coin with a friend.  The first person to toss tails owes the other one dollar and when tails appears, the game ends.  What is the probability that you toss tails first and owe your friend a dollar?

(C) You hire an undergraduate to toss a fair coin for you an infinite number of times.  If the undergraduate gets heads times in a row, or tails times in a row, we call it a run of length .  Prove that with probability one the undergraduate will get a run of length at some point in this experiment.

(D) You are gambling on coin tosses: on each turn, you flip a single fair coin.  Every time you gets heads, you pay her a dollar, and every time you get tails, she gives you a dollar.  You have $500.  What is the probability that you have $1000 before you run out of money?

Problem (A) is well within your reach.  The probability that you get the sequence can be found using counting techniques or by multiplying the probabilities of independent outcomes.  Make sure you can do this both ways.

Problem (B) has an infinite sample space

Questions for you: Is this countable or uncountable?  What is the probability that tails is never tossed at all?  Hints: notice that all the events in the first subset of are mutually exclusive, so you can sum up their probabilities.  See if you can calculate the probability of each event and then sum them.

Problem (C) can be done using the Borel-Cantelli theorem discussed earlier – see if you can use this for a proof!

Problem (D) is an example of a stopping time problem.  These types of problems were incredibly important in the beginnings of probability: in some sense, modern probability theory was born out of gambling, and when you are out of money you can’t gamble any more.  Figuring out the probability of reaching net worth before net worth zero is pretty imporant, even in modern capitalism.  The start-up concept of “runway,” for instance, is how much time you have before your start-up goes bankrupt, and depends on how much money you have and how fast you are spending it (your “burn rate”).  Figuring out your runway is in essence a stopping time calculation.

Chapter 2

Geometry problems in probability

All the questions we addressed in the last chapter happened to be discrete probability problems.  However, a lot of probability problems are geometric problems, with calculations taking place on a space with an uncountable number of points: throwing darts at a dartboard or dropping needles on a lined page, for instance.  The probability of hitting a particular point is zero, for reasons we’ll discuss, but the tools of geometry and calculus can help us.

Why are these problems of interest to someone who cares about finance?  First, many problems in finance can be fruitfully considered by looking at continuous analogs.  While stocks are priced in fractions of a cent on the New York Stock Exchange, the Black-Scholes equation for option pricing takes a continuous analogue and works rather well.  Brownian motion relies on a continuous limit of discrete processes.  Being able to use both continuous and discrete techniques as appropriate is an important tool in mathematical finance.

To solve geometric probability problems, you need to dredge up your knowledge of area and volumes.  Calculating the size of an event set in set with an uncountably infinite number of points is the calculation of an area or volume.  Sometimes it’s enough to know that the area of a disk of radius is , and sometimes you will need to integrate.  By the end of this chapter I want you to be practiced in solving probability problems that require geometric techniques, like this:

Consider the square with corners at , , , and .  Pick a point in this square at random.  What is the probability that the product of the coordinates, , is greater than one half?

This will require some review of calculus which will be done in the sections on function families (Section 2.1) and derivatives and integrals (Section 2.2).  We’ll also do a tiny bit of linear algebra in talking about affine linear transformations (Section 5.5).  You may find that you need extra resources to shore up your memory of these topics or you may find them almost easy enough to skip – do the right thing.  At the end of the chapter we’ll examine some famous problems that use geometric techniques.

2.1 Function families

Here is a quick run-through of functions you’ll encounter in this book:

- Linear functions and affine linear functions

- Quadratic functions, also known as conics

- Polynomials of all degrees

- Rational functions

- Power functions

- Exponential functions

- Logarithmic functions

- Trigonometric functions and their inverses

- Compositions of all the above

If you need a refresher on any of these, check out Paul’s Online Calculus (and pre-calculus) Notes.  Another extraordinarily effective way to refresh your memory is to volunteer for a few shifts of homework help for middle-school and high-school students: you can often find a local library or Boys & Girls club that offers homework help to kids.  Trying to explain algebra and pre-calculus concepts to a twelve- or seventeen-year-old will force you to think about these ideas as you never have before.

2.2 Derivatives and integrals

In this book, I’m assuming you took a calculus class in the past.  You may need a reference, though.  Any calculus book should have differentiation and integration covered, so find your favorite calculus book (or the cheapest).

Instead of rewriting a calculus textbook, I will concentrate on pointing out connections – building a higher-level appreciation of calculus and how it relates to probability and discrete mathematics.  Calculus as we are taught it fundamentally takes place in the continuous world.  We can’t take the derivative at a discontinuous point of a function.  This continuous point of view is always an approximation for finance, but it’s very powerful!

2.2.1 Volume and area

Mathematicians don’t tend to differentiate (haha!)  between area, volume, hypervolume, length: to a mathematician, they are all some sort of generalized “area.”  Think about why you see the notation for absolute value of a number, length of an interval, size of a set, and determinant of a matrix.  How are these quantities related or not?  Realizing the underlying unity of these ideas you were taught in different times and places will give you a lot of mathematical power.  In this section, I want to point out how calculus relates to simple ideas of circumference or perimeter, area, and volume.  Where will you put this to work, you ask?  In transformations of random variables, in integration by parts, in deducing probability density functions from the structure of a probability problem, and in short-term and long-term approximation of financial and other quantities.

Circles and spheres.  You remember that the area of a circle with radius is and the circumference is .  Notice that

The derivative of area is circumference.  For spheres, the volume of a sphere with radius is , while surface area is .  Again, differentiating the higher-dimensional volume gives the volume in the dimension one lower, if we expand what “volume” means to us:

Why does this work?  A picture of the two-dimensional disk might help:

Notice that we use the notation for a very small change in the variable,

Squares.  Try the naive analogue for a square with side length : area of this square is and perimeter is – the relationship doesn’t seem to hold!  Can you see why?

Now look carefully at what you are actually doing here:

It turns out the procedure is working just fine – we are simply solving a different problem.  The change in side length gives us only half the perimeter of the square.

A more accurate analogue comes when we realize that our thinking about the square needs to be adjusted to match the reasoning for the circle.  Let’s use a new variable, , and draw a new picture:

Here, notice that the area of the square is and the perimeter is .  Our change of viewpoint shows that the relationship we’d conjectured earlier holds.

2.2.2 Differentiation as infinitesimal approximation

The previous discussion of area and volume and their relationship might prompt you to look at how quickly a function changes as you increase or decreases .  That’s a derivative.  Translating the sentence into mathematics, you considered the change

as the quantity

changed.  The ratio is

which might start ringing alarm bells in your brain.  That’s right: a common definition for the derivative is

Our drawings in the previous discussions can be made into mathematically precise arguments using this definition!

This can be turned around.  We can predict the value of a “few moments” later, when the input is .  We want to predict .  Turn our definition of the derivative inside-out to get the following:

If is our output, then we can write the approximation (not equation!)

Since we’re working with a single input, , and often graph this on the x-y plane, we can add and write

This allows us to figure out the change in output and relate it to the change in :

\Delta y \approx f^{\prime }(x) \Delta x.

Relate this to our discussion of volume and area to cement your understanding.

To make this all more precise, we can characterize how “good” or “bad” the approximation is.  We introduce little-oh notation, .  Taking our definition from the National Institute of Standards and Technology, we say

- In words, if becomes insignificant relative to as approaches infinity.

- With more symbols, “for all there exists some such that for all .  The value of must not depend on , but may depend on .”

- With limits, this means .

This means we can write the equation

2.2.3 Average rate of change and Mean Value Theorem

For a differentiable function,

for some value of between and .  This means that at some point , the derivative of is equal to the average rate of change.  A simple picture illustrates this nicely:

Remember, this requires that is a differentiable function – in particular, has to be continuous (at least near ).  This Mean Value Theorem is very convenient for proofs and calculations in the continuous world, but it won’t help you in the discrete world of stock prices.  Just because the price of a share of Microsoft (MSFT) wass $50.05 at 12:55 pm on June 17, 2016 and was $50.03 at 1:00 pm on the same day doesn’t mean that it had a derivative of 2 cents per five minutes at some point in the middle!

You can derive many of the rules of differentiation using the short-term approximation formula, and doing so is actually very good practice to prepare for the world of stochastic calculus.

Example: to derive the power rule using this idea of short-term approximation, we need the binomial theorem:

Using , look at

Then short-term approximation tells us that

Since we’d like to solve for , rearrange:

which simplifies nicely to

I’ll rewrite it even more pointedly:

As , this gives us , the power rule we learned in calculus.  You can make this more precise by using limits throughout and using the notation to keep track of the error!

2.2.4 Integration and its disguises

Integration takes on many roles in the mathematical foundations of finance:

- Integration is a useful tool for calculation in solving probability problems.

- Riemann sums and their limits occur in problems and in proofs in probability.

- Probabilistic methods (Monte Carlo techniques) can be used for numerical integration, including value-at-risk and expected shortfall calcuations.

- Understanding the relationships between integrals (), Riemann sums (), and probabilistic or statistical ideas like expected values and moments is crucial to your ability to progress in mathematical finance.

Let’s begin with the first comment: integration is necessary to solve geometric probability problems.  Consider the square with corners at , , , and .  Pick a point in this square at random.  What is the probability that the product of the coordinates, , is greater than one half?

In the last chapter, we dealt with finite and countably infinite sample spaces.  There, we could assign a probability to each individual possible outcome and sum over these outcomes.  Here, we could try this but we encounter a problem.  What’s the probability that I pick the point exactly?  A mathematical point has area zero, and the chance that I hit exactly that point is zero.  However, the chance that I pick a point in a continuous region of the unit square is the area of (because the area of the unit square is one, and when we say “pick a point at random” without specifying further, every point is equally likely).  How can this be?  How can the sum of infinitely many zeroes be a positive number?

You actually encountered this in calculus.  Recall that

for instance, and

for any continuous function .  The integral over a point is always zero even though the integral over an interval may not be.  This is because is uncountable as you proved in Chapter 1, and uncountable sets present dramatically more subtle situations than finite or countable sets.

Back to our problem: Pick a point in the unit square at random.  What is the probability that the product of the coordinates, , is greater than one half?

Step 1: identify our sample space .  It’s all the points in the unit square.

Step 2: identify the set of outcomes we’re interested in.  Here

Step 3: Calculate the sizes of and .  We know has area 1.  What is the area of ?  Draw a picture:

Find the shaded area using an integral.  Notice that it’s the integral of :

We’re done!  yay!  WRONG. While trivial in this case, remember to divide by the size of the original sample space.

Step 4: Calculate .  Since , we have

More examples of probability problems that use integration will be given in the last two sections of this chapter.

2.3 Buffon’s needle problem

Buffon’s needle problem is a classic probablistic problem.  It was first written down by Georges-Louis Leclerc, Comte de Buffon, a French intellectual who lived in the 1700s.  He asked the following question in 1733: Consider a floor ruled with parallel lines.  Drop a need onto the floor.  What is the probability that the needle will cross one of the lines?

There are two situations to consider, one with a needle shorter than the distance between the lines and one with a needle longer than the distance between the lines.  We’ll start by considering a short needle.

An illustration to set our notation is useful:

The length of the needle is , the distance between the lines on the floor is , and the angle that the needle makes with the lines is .  In a given probability experiment (dropping the needle once) we’ll label the distance from the middle of the needle to the nearest line on the floor by .

There are several ways to solve Buffon’s problem.  For simplicity, I will present a method that follows our format of specifying , finding the event set that we desire, and comparing the sizes of these sets.

Using the notation of the illustration, we write

For yourself, check the following: Why don’t we specify how far “over” the needle falls (horizontal position)?  Why don’t we use , or even ?  The way we’ve set this up, there are two positions for the needle for each angle !

The event set , then, is when

1. the needle is close enough to the line to possible cross the line, and

2. the angle of the needle is such that it actually does cross the line.

Mathematically, write

Notice that you could draw as a rectangle in the plane or the plane.  Likewise, you can draw the set if you choose a given orientation of the needle.  To avoid confusion, let’s call the set of positions of the needle with measured only clockwise .

Once you draw , you notice that the area of is , and to find the area of you must integrate either with respect to or with respect to .  One choice is clearly easier than the other :

Then

The long-needle version of the problem (with ) can be solved in a similar way, but since the needle can cross more than one line a little more analysis is needed.

For now, we will ignore the long needle version of Buffon’s problem.  We’ll come back to it in the section about joint probability density functions, a great tool which will allow the solution to be given in just a line or two.

2.4 Stick-breaking problems

Stick-breaking problems give students of probability all kinds of trouble.  They’re great!  Deceptively simple, these stick-breaking problems illustrate how small changes to a problem can dramatically change the techniques necessary to solve it.  They also show that these simple mathematical puzzles are actually structures that show up in all sorts of natural situations, like Dirichlet processes.  Bizarrely, the last version is apparently an interview question at Goldman-Sachs.  (See http://www.wallstreetoasis.com/blog/interview-questions-ib-analyst-goldman-sachs for reference.)

Version one: A stick of length is broken in two places.  The break points are independent of each other, and chosen at random (uniformly) on the stick.  What is the probability that a triangle can be formed using the three pieces of the stick?

Version two: You and a friend are breaking the stick now: you each grasp one end of the stick and, together, break it at one point.  Then you take the piece of the stick left in your hand and break it again.  What is the probability that a triangle can be formed using the three pieces of the stick?

Version three: You and a friend are breaking the stick now: you each grasp one end of the stick and, together, break it at one point.  Then you take the longer of the two pieces and break it again.  What is the probability that a triangle can be formed using the three pieces of the stick?

So....  solutions?  Try these problems first and then look at the solutions one by one, as you might be able to clear up misunderstandings about the structure of the problem by looking at solutions individually.

Before discussing solutions, I want to bring up a subtle but important point: you can use variables in this problem to represent either lengths or locations.  That is, might represent the length of a piece of the stick, or it might represent a location along the stick.  This makes a big difference in that locations are given with respect to a single origin point.

Solution to version one: First, let .  We can rescale and just say that inches (or meters) is 1 unit of our own measurement system.  Call the distance from one end of the stick to the nearest breakpoint , and the distance from the other end of the stick to its nearest breakpoint .  The two breaks happen at the exact same point of the stick with probability zero, so we don’t have to worry about the contribution from breaking the stick into only two pieces.  We’ll have three pieces of the stick after breaking, of lengths , , and .  The most important thing to notice here is the domain for and : and can be any values between 0 and such that , and on this triangular set, every point is as likely as any other!

To form a triangle, these lengths must satisfy the triangle inequality (the sum of any two sides of a triangle must be larger than the third side):

Simplify to get

Since we know and , and also , graph the sample space and the event space .  You can use symmetry to see that

Solution to version three: You and a friend each grasp one end of the stick and, together, break it at one point.  Then you take the longer piece and break it.

Here we will use locations rather than lengths.  Again rescale the stick so it’s length one, and label one end of the stick zero and the other end one.  The first breakpoint occurs anywhere along the stick.  If , then the second breakpoint will be chosen at random somewhere in the interval , while if the second breakpoint is chosen at random on the interval .  Now, the problem is that the possible values of depend on – these are not independent quantities anymore, and is not uniformly distributed on !  We only have the tools for uniform distributions at this point.

To solve the problem today, we need to turn it into a problem with uniform distributions (every point as likely as every other point).  The alternative is to learn about continuous random variables and probability density functions, which won’t happen immediately.  So....  Let’s use this rescaling trick again.

- If , then let and , for .  In essence, we’re letting be whatever it is and then treating the stick of length as its own new stick to rescale to length 1, and tells us where along this the next breakpoint is.

- if , let and , again in .  Again, we now have representing the fraction of the longer stick at which the next break happens.

- Putting these situations together, we have chosen randomly (uniformly) in the interval , and we have chosen uniformly in the interval , and we’ve transformed the variables so that each point in the unit square is as likely as any other.

Now we have to do some algebra and calculus, because we need to figure out which points in the unit square correspond to which give a triangle.  Use the triangle inequalities for and convert:

For , we have three lengths: , , and .  The triangle inequalities are

and

Simplifying,

and

Convert to and :

and

This gives us two inequalities,

and

We must integrate between these two curves:

Now carry out the same process for and add the answers to get the total probability, which is

The solution to the second stick problem is just – adapt the argument above to get this answer!

Chapter 3

Probability rules

3.1 Basic rules of probability

3.1.1 Complements and inclusion-exclusion

Essentially every concept in basic set theory gives rise to an analogous rule in probability.

If two sets of events and are not disjoint, there is an overlap .  This intersection of and is overcounted when we look at the union of and , so we must remove it to find the size of .  In symbols,

Probabilistically, this implies

We encountered disjoint sets earlier: and are disjoint if .  In this case, the sizes of the sets add:

This is a special case of the previous observation because .  When and are disjoint sets, then we call the events in and mutually exclusive events.  The probability of either of two mutually exclusive events is the sum of the probabilities:

which is again a special case because the probability of nothing happening () is zero.

Another set-theoretic idea is that of the complement of a set .  We consider as a subset of some sample space , so .  Then the complement of is everything in except the elements of :

From the axioms of probability, we know that for the entire sample space.  From the definition of , we know that and are disjoint sets that partition .  Since , we also have .  This implies

Take the preceding rules a little further to deduce the inclusion-exclusion rule.  Instead of two sets and , what if you’ve got multiple subsets of , and you want to compute ?  If all the are pairwise disjoint you can simply sum the probabilities of the individual sets .  However, if there are overlaps, you need to consider intersections, triple intersections, quadruple intersections...

Probabilistically,

All useful tools in your toolbox!!

3.1.2 Example of inclusion-exclusion principle

A classic type of problem asks questions like this:

- At a theater men leave their hats at the hat check counter.  The irresponsible clerk loses everyone’s ticket and so after the perfomance, he randomly returns hats to men who come back to retrieve them.  What is the probability that no men get the correct hat returned?  (Back when men wore hats and left them at a hat check before a movie, women also wore hats but kept them on as the hat was considered part of the woman’s outfit.)

- You’re sending thank-you notes after your wedding.  After writing them all, you have a pile of notes and a pile of addressed envelopes.  A “helpful” friend stuffs all the envelopes and sends the notes but doesn’t realize the two piles are not in the same order – so random letters are put in random envelopes.  What is the probability that at least one person gets the correct thank-you note?

- Rabbits are playing in a field outside their burrows.  They are suddenly surprised by an eagle, so each rabbit runs to a separate hole and escapes into a burrow.  What is the probability that no rabbit escapes down its own hole?[1]

These are variations on what might be called the “derangement” problem.  Derangement does not refer to being a deranged person, despite what your probability homework might lead you to believe or feel.  Instead, a derangement is a permutation with no fixed points – a rearrangement of objects so that no object stays in the same place.

Let’s outline a general way of solving these problems.  Say there are elements in the set of objects (hats, letters, rabbits).  Let be the event that object ends up in the correct place (hat with correct man, letter with correct envelope, rabbit in own burrow).  This is a key point: rather than looking at all possible arrangements of hats, look instead at hat only.

Set-theoretically, then, the event that any hat returns to its correct owner is .  We can use inclusion-exclusion to find .  Here are the pieces: , , , etc.  Combine them to get:

Notice this simplifies:

so

This is the probability that at least one object goes to the right place.  To find the probability that no objects go to the right place, subtract the above from 1.

3.2 Conditional probability

Define the probability of the event given that event happens as

Think about how to visualize this: I see it as

Notice that re-writing this gives

At times this is a useful way to compute the probability of two events happening simultaneously.

As an example, let’s revisit a problem you did via counting in Chapter 1.

Eight teams are in the semifinals of an international badminton tournament.  The eight teams consist of two teams each from China, India, Denmark, and Korea.  What is the probability that the two teams from each country end up playing against each other in each of the semifinal matches?  That is, China 1 plays China 2, Denmark 1 plays Denmark 2, etc.

You can nicely do this with conditional probabilities instead of counting techniques:

Some team is picked to play in match 1 with probability 1.  This team is from country A. Given that country A is picked to play in the first match, the probability that their opponent will be from the same country is .  Given these events, any remaining team can be picked to play in the second semifinal match – say it’s from country B. Given all these, the probability that country B’s opponent is also from country B is .  Given...

Maybe you see a pattern here!  In the end, we get that the desired probability is

Much less counting needed!

3.3 Bayes’ theorem

Bayes’ theorem is an extraordinarily useful tool, allowing its user to adjust estimates of probabilities when new evidence is made available.  The theorem uses conditional probability to find the probability of a hypothesis given evidence .  Start with the definition of conditional probability, which tells us:

Do a bit of algebra to notice the following:

From here, since we want to find , we can proceed in two ways: we can find the “odds form” of Bayes’ theorem or we can find a formula for directly.  In either case, we will need to consider the probability that our hypothesis event did not occur: we will write , for the complement of in a set-theoretic sense or the negation of in the context of logic.

The direct formula for Bayes’ theorem is often used: use algebra to get from

to

For the final step, use the law of total probability to write as a sum by splitting the event over mutually exclusive and :

This gives

a formula memorized by probability and statistics students around the world.  The method can also be generalized to multiple hypotheses as long as they’re mutually exclusive.

The odds form is also easily derived: simply take the ratio of

and

The final result is

Why bother with this slightly less direct formulation?  Because it presents the information with a nice structure.

- First, it states Bayes’ result as “prior odds” times a “likelihood ratio”, giving the “posterior odds” in which the evidence has been taken into account.

- Second, the odds format often allows you to use natural frequencies and avoid calculating probabilities.  Natural frequencies provide information like, “It rains on average once every ten days,” or, “An estimated 1 out of 133 Americans has celiac disease.”

The only danger is that odds really are ratios of occurrences of dissimilar events – days with rain versus days without rain, for instance – rather than the fractions we’re most familiar with, like days with rain as a proportion of all days.  Check what sets you’re dealing with if you get confused about whether you have an odds ratio or fraction: odds compare the sizes of two sets that partition a larger set, while fractions and probabilities deal with the size of a subset in the numerator compared to the size of the whole set in the denominator.

Back to the statement of the theorem: The prior odds of the hypothesis event occurring are

In the examples above, the prior odds of rain are or , and the a priori odds of an American having celiac disease are or .  Taking evidence into account involves multiplying by the likelihood ratio, which gives information about how likely the evidence is if the hypothesis does or doesn’t hold true.  If , the posterior odds describing the likelihood of in the face of are increased – the evidence has added to the probability that the hypothesis is true.  Many of the most surprising results from Bayes’ theorem, though, come because the likelihood ratio is not what people expect.

First let’s consider a somewhat famous example that illustrates both the difficulties many people have with probablistic reasoning and a good use of Bayes’ theorem.

(Gigerenzer, “Calculated Risks,” 2004) Doctors were told the following: you’ve got a patient with a positive mammogram ().  She is between 40 and 50 years old, with no symptoms or family history of breast cancer.  The a priori probability that such a woman has breast cancer () is 0.8 percent.  If she’s got breast cancer, the mammogram will be positive 90% of the time, and if she does not have breast cancer, there’s still a 7% chance that she has a positive mammogram.  Given all this information, what’s the probability that your patient actually has breast cancer?

What do you estimate, reader?  Gerd Gigerenzer, a cognitive psychologist, asked doctors to do this experiment.  Their estimates ranges from 1 percent to 90 percent.  As a patient, how would you feel if you were told that there is 90% chance you have this cancer – especially when the true answer is around 9%?!

Using the probabilities as stated, we know that the prior odds that this patient has breast cancer are .  (Small odds!)  The likelihood ratio is .  This is big, but only about a factor of 13.  The posterior odds, then, are .  When we solve for the probability , we find

See Girgerenzer’s book for more examples of how bad we are at estimating probabilities.  These cognitive psychology experiments are very useful in showing us where we fall into cognitive traps, overestimating the probability of rare events, underestimating the probability of common events, and showing “anchoring” tendencies, in which we estimate that quantities are close to numbers coming from preconceived notions.  While much of finance involves careful calculations in spreadsheets using approved models, we in numerical fields must still make quick estimates to check the validity of our work.  Avoiding these cognitive traps will make you more effective in many ways!

1. Asked by Jarrad Smith on MathForum.org in 1999 ↑

Chapter 4

First applications in finance

It is amazing to me that a simple model – the binomial tree model for stock prices – is so effective in pricing various types of financial derivatives.  In this chapter, we’ll consider financial ideas (arbitrage and stock pricing) and additional tools useful for finance (approximation, including Taylor series).  We’ll also see why we need to step up our sophistication and introduce concepts like random variables and their expected values in the next chapter.

4.1 Binomial stock pricing

At 2:05 pm on a Tuesday, a stock has a given price listed on the New York Stock Exchange.  As orders come in, that price will go up, go down, or stay the same.  A first approximation – the binomial tree model for stock prices – is to pick a unit of time and to model the stock price at each unit step.

4.1.1 Additive model

The first idea you might have would be to start with an initial stock price, , and then add dollars or subtract dollars on each step.  At time one, then, the price would be or .  Here, let .  In constructing the model, you’d want to find the appropriate probability of the stock price increasing and the probability of the price decreasing.  Continue this over subsequent time steps.

Notice that the price at any time step is the sum of two arithmetic series.

As always, you should evaluate the benefits and drawbacks of such a model.  A benefit is that the model is quite simple, and easy to extend to any number of time steps desired.  A huge drawback is that the model allows negative stock prices.  This is unrealistic: while we can short a stock, the price of the stock itself can never be negative.  In addition, consider the sizes of and : if is $ 400, a change of $ 1.00 in the stock price is not that significant, while if is $4, a change of $ 1.00 is an enormous change in value.

A last factor to consider is that this is a discrete model of a market that trades in continuous time.  Whether this is a benefit or drawback is to be argued – what do you think?  We will keep discussing the relationship between continuous and discrete models: remember that a theme of our initial probability discussion was the back-and-forth between combinatorial and geometric probability problems.

This additive tree model is really not very good for stock prices, but we shouldn’t dismiss it entirely.  It will lead us on a random walk through the central limit theorem, normal distributions as a limit of repeated Bernoulli trials, and finally arithmetic Brownian motion.

4.1.2 Multiplicative model

To fix at least one problem with the additive binomial tree model, we can change to a multiplicative pricing model.  The initial price changes by a percentage: with probability the stock price increases by a factor of and with probability the stock price decreases by a factor of .  In general, we set .  You will see why when we discuss arbitrage.

Construct the binomial tree as follows, then:

The stock price will never fall below zero in this model, and changes in price are proportional to the original price.  This is still a discrete-time model, but financial mathematicians have found it quite useful.  Here, notice that the maximum price at each time makes up a geometric sequence with common ratio and the minimum price at each time makes up a geometric sequence with common ratio .

From this model, you can do two things given , and : you can find the probability that the stock has a certain price at a given time step, and you can find all the possible prices at a given time step.

What are all the possible prices for this stock at time step ?

Use the binomial theorem to get all prices at time .  must take on a value that looks like for some between zero and .  Ask yourself: does refer to steps “up” or steps “down” in price?

Is this model realistic?  Can we simply say any numbers for , , and ?  Are these quantities related?  What are the pros and cons?

4.2 Arbitrage

The no-arbitrage principle in finance and economics answers one of these questions.  The no-arbitrage principle claims that “there is no free lunch” – you can’t make guaranteed money.  If the no-arbitrage principle holds, then is uniquely determined by and .  We can show this rigorously later.

To make this discussion easier, we need to introduce the concepts of random variable and then expected value.  Notice that in each of the binomial tree illustrations above there are notations along the bottom: .  The notation means essentially “the outcome of the probability experiment at step .”  It is a random variable, which is a confusing name for the following reasons: in much of math, a variable is a letter that we use in expressing a mathematical condition like or , and the variables can take on any values that satisfy the condition, but in statistics, the random variable can only take on values in , and the random variable comes with a certain probability distribution.  In the next chapter we will talk about such distributions and in later study you’ll encounter the definition of a random variable as a measurable function for a measure space.  For now, it’s enough to know that is a random variable that can take on each outcome that appears in step 3 of a binomial tree.  A random variable has to take on a numerical value, not a value like or .

The next big idea is expected value.  The expected value of a random variable is the weighted average of all the possible outcomes.  Let’s consider only discrete random variables with outcomes .  Each outcome has a particular probability, , and the sum of these probabilities is 1.  The expected value of , then, is

which weights each outcome according to its probability.  For example, let’s say you flip a fair coin and you let the random variable be 0 if the coin comes up heads, 1 if it comes up tails.  The expected value of the outcome of is , from .  Again, a funny name, as you’d never expect to get the value 1/2 from flipping a coin!  But 1/2 is the weighted average of the outcomes we allowed for .

Back to no arbitrage: the “no free lunch” principle.  One of the crazy paradoxes of finance is that people participate in modern finance because they want to make money, but the no-arbitrage principle states that everything is fairly priced by the market and so there is no guaranteed risk-free money to be made.  One way to think about this is as follows: on a very small time scale, there may be inaccuracies in pricing in the market, but these inaccuracies will be found and exploited by so-called arbitrageurs, and thus supply and demand will push the inaccuracies to be corrected.  Thus, in the long term, financial instruments are priced fairly.  This philosophy has had enormous success because it seems to work pretty well over larger timescales, and because it has allowed us to get a unique “correct” price for many common financial instruments.  A price that everyone agrees on makes trade much easier!

More concretely, the no-arbitrage principle would tell us that if the stock under consideration is fairly priced initially (at time zero) then the expected value of the stock at time one would be the same as its initial price.  Why is this?  Say the expected value of the stock at time one was definitely bigger than at time zero.  You’d buy a lot of stock at time zero!  (So would other people...)  Similarly, if the expected value of the stock at time one was less than the value at time zero, you wouldn’t buy it at all: you’d say it was overpriced and you’d put your money elsewhere.

Mathematically, this implies that

Since this expected value is just a weighted average, let’s see what this means for one step of the multiplicative binomial tree:

which simplifies to

finally giving

This gives us the promised relatinoship between , and – and astoundingly requires no price information at all.

In this era of historically low interest rates, what we’ve done is even almost right.  For most of financial history, though, we have needed to consider whether we should just put our money in the bank rather than investing in stocks.  The interest you can earn on cash is referred to as the “time value of money.”

4.3 Short- and long-term approximation

We discussed the idea of short-term approximation in Chapter 2, in Section 2.2.  As we alluded to, the idea of predicting value of a “few moments” later, when the input is , is quite powerful.  How does this apply to finance?  Take a moment to ponder this question and brainstorm some answers.

If is the output of a function, then we can write the approximation (not equation!)

We can rewrite this in many ways, including:

and

What are our assumptions in making these short-term predictions?

This approximation is really only valid if is a continuous and differentiable function.  If is not continuous in the area of our apprximation, the approximation is total nonsense – and may be nowhere near each other.  If is not differentiable at a point where we’d like to look at this approximation, then won’t exist or make sense.

Financially, the idea of short-term approximation comes about because we’d like to know the price of a stock tomorrow, for instance.  At the beginning of this chapter we discussed binomial tree models for stock prices, which are necessarily discrete.  Even though it’s not “true”, we can invent a continous function for the price of a stock depending on time, and we can approximate the discrete situation by using calculus and the continuous model.  The Black-Scholes model, for instance, is a continuous (or stochastic) model of the price of an option on a stock.  The partial differential equation

is called the Black-Scholes equation, implicitly describing the price of a European call option in terms of , the price of the underlying stock; time ; standard deviation of stock returns ; and a risk-free interest rate (that time value of money!).  All of these derivatives can be discretized using the idea of short-term approximation.

The short-term approximation is a linear approximation.  Let’s interpret the equation

as a linear approximation for the price of a stock at time given the starting price at time .  Say is 11 am on a Monday, and we want to guess where the price will be after lunch (at 1 pm) if the stock behaves in a predictable way.  Then we could consider approximating from the morning’s data, or from last week’s data, or some other data set.  This number would give us how fast the stock price is sinking or rising.  The linear approximation is easy then.

4.3.1 Long-term approximation

We want to combine short-term predictions to make long-term predictions.  How do we do this?  What do we mean?

Our most important tool is the fundamental theorem of calculus.  The fundamental theorem of calculus relates integrals and derivatives.  One version of the FTC in words is, “net change is the sum of rates of change over the interval.”  We can write this in symbols below.

Example 3.1 Explore

An example: you want to look at total change in a stock price from time to time .  That’s .  One way to look at the change in price is to look at how it stepped up and stepped down at every moment between time and time .  If you sum up all the changes, for each moment , you get a version of the fundamental theorem of calculus.

Can you convince yourself that the fundamental theorem of calculus is a reinterpretation of the short-term approximation discussed above?

How do you relate the discrete and continuous versions of each of these concepts?  What is each useful for?

4.3.2 First differential equation

A differential equation is an equation that relates quantities and their derivatives – for instance, the quantity and its derivative with respect to , .

I want you to learn how to read these differential equations, and identify the assumptions implicit in the mathematics.

For example, what does this differential equation say?

It says, “the rate of change of with respect to is proportional to the value of at every value of .”  Think through the implications.  Let’s say is positive, for simplicity.  Then if is negative, must also be negative.  If is positive, must also be positive.  If , then the rate of change of is also zero.

What is the solution to a differential equation?  It’s a function that makes the differential equation true.  For instance, makes the equation true: is always true.  On the other hand, is not a solution to this differential equation if , as but .

How can we solve this differential equation?  I’ll group the methods as numerical methods, analytic methods, and graphical methods.  Numerical methods find approximate numerical solutions and are great if you’ve got some computational power at your disposal.  Analytic methods find you exactly-accurate formulas.  In this class, many differential equations will have analytic solutions (closed formulas) but in real life many differential equations don’t have analytics solutions at all.  Last, graphical methods like slope fields can help you understand the behavior of solutions and might give guidance as to what further techniques you’d like to pursue.

Let’s look at our first differential equation in a financial context.  The most basic finance calculation many people do is the calculation of accrual of interest on principal.  Using for principal and for time in days, we can write

for the daily interest rate.  Note that could be positive (money in the bank) or negative (you owe the bank money).

Say (you start with $100) and the daily interest rate is .  Then if interest is accrued daily, you end up with at the end of the day.  But if interest is compounding continuously, this is only an approximation – a linear, short-term approximation.  What’s the exact amount of interest you end up with?

Here’s where an analytic solution comes in handy.  We’ll have to integrate here.  Do a bit of manipulation to convert

to

Use your calculus knowledge to solve:

where is a constant.  Exponentiate to get

If we know that , then we should have – ah, is the initial amount of principal.  (For the general solution of the differential equation we can drop the absolute value on the left-hand side as if we’re starting with positive amounts of money, all future values will be positive, and if we start with and compound debt, we’ll just get more debt.)  To find the exact value of , then, we just evaluate:

Last, let’s look at a graphical exploration of the differential equation.  This is a slope field: at points , we draw little lines of slope .

This allows us to quickly understand the qualitative behavior of solutions to

These solutions must always “follow the arrows,” so it’s easy to sketch solutions to the differential equation as well!

Later we’ll consider numerical methods for solving differential equations, and we’ll see that Euler’s method, for instance, is really just a version of long-term approximation applied to differential equations.

Chapter 5

Discrete random variables and transformations of variables

This chapter will combine a number of concepts that aren’t usually discussed in conjunction.  First, we’ll talk about discrete random variables, expected values, and variance.  Second, we’ll notice that series keep coming up.  We’ll talk about series in general, power series, and Taylor series.  I’ll throw in some simple but important finance applications.  Last, we’ll talk about linear and affine linear transformations of variables.  This will be our first step toward incorporating linear algebra and multivariable calculus.

5.1 Discrete random variables

It’s finally time to look seriously at random variables.  A random variable is a function from to : it always takes on numerical values.  Remember that is the set of possible outcomes of a probability experiment, so writing out a random variable as a function is a way of assigning a numerical value to each outcome of the probability experiment.  In general, we don’t know what the outcome of a probability experiment is until the experiment is carried out – that’s why we use or another variable name to represent this outcome before knowing what it is!

Here’s are some examples of random variables.  When you flip a coin, you get heads or tails , but the random variable associated to your coin flip might be , with

or

or taking its value in any other two-element set in .  The numerical values depend on the problem you’re trying to solve.  For instance, we could play a game where you flip a coin and if it’s heads I give you five dollars and if it’s tails you give me eight dollars.  You could decide then to represent the outcomes of the probability experiment (heads and tails) as taking values in the profit-loss space represented by , and using the tools we’ll build you can easily and rigorously decide whether this is a game you want to play or not.  You might write for the random variable representing your winnings, with

A discrete random variable is one that takes values in a finite or countably infinite subset of .  The values that the random variable can take make up the range of the random variable, often denoted .  (Convince yourself that any random variable taking values on a continuous interval of can’t be a discrete random variable, using this definition.)  The best way to get a feel for discrete random variables is to do examples.

(Aside: As you do problems, I highly recommend writing out what random variable you want to consider, very explicitly.  A major cause of mistakes is mixing up the random variable you really want to consider and some other related quantity – for instance, doing a calculation with the outcomes rather than the outcomes in our coin-tossing game above, or mixing up “number of winning coin flips it takes to win a game” with “total number of coin flips it takes to win a game”.)

Example 1.1 Flip a coin three times.  Let be the random variable representing how many times heads appears in your three flips.  Then , and we call the range of .

Example 1.2 Roll two dice simultaneously.  Consider the random variable given by taking the maximum roll.

Here, I would say is the value on the first die and is the outcome of the second die, and I’d say is the random variable we’d like to consider.  The range of is .

Alone, the concept of the discrete random variable doesn’t seem very subtle or important.  However, it’s going to facilitate the discussion of

- the probability mass function,

- the expected value of a random variable, and

- the variance of a random variable.

These are basic probabilistic concepts which are very important in finance: once we consider portfolio optimization, you’ll look to maximize your expected return (the expected value of the portfolio in the future) and minize the volatility of the portfolio (the standard deviation of the value of the portfolio).

Let’s define these concepts.

First, for a discrete random variable we can find the probability mass function, for .

Second, we can define the expected value (expectation, mean) using the probability mass function:

Each outcome is weighted by its probability.  The expected value function is written using the or notation, and when everyone reading knows which random variable we’re talking about, then we can use the notation (mu) for mean.  Notice that for a constant, and convince yourself that as well.  Figure out what is for , constants.

Third, we can define variance:

Meditate on this definition for a while: it’s the expected value of the square of the difference of from the mean .  Hmmmm....  how to understand this?  Notice that again we have the function which we can apply to any random variable, and we have the notation when it is clear which random variable is under discussion.  Also, we can define the standard deviation here:

Example 1.3 Again toss a coin three times and let be the number of heads appearing.

Following the definitons above, we first compute the probability mass function (pmf).

We’re counting the number of ways heads can appear in three tosses, then multiplying by the probability of any combination.  It’s better to just write , you’ll see.

Second comes expected value: it is

which evaluates to

Third, variance.  If we use our formula above for variance, we need to compute the expected value of a new random variable, .  The variance is

Frankly, that last calculation was mildly annoying.  There’s got to be a better way.  Many advances in mathematics have been made through rigorous laziness – can we simplify this calculation?

5.1.1 Linearity of expectation – best thing ever

Fortunately, we can simplify our calculations of variance.  The expected value function is linear!  That is,

as long as , are finite numbers.

Sketch a proof of this fact: consider two discrete random variables and and call their sum .  You’ll have to look at the probability of the joint event “ and ,” which you can write as .  Figure out the pmf for and go from there.

In particular, this has the following lovely consequence: if we let ,

This is lovely: let’s do another example calculation using this new formula for variance.

Example 1.4 Consider those two dice that you rolled earlier.  Remember one die had a value and the other a value of , and we want to examine .  The range of is .

The pmf for is .  I got this by writing down the first few probabilities, noticing a pattern, and then mathematically formalizing it.

The expected value of is easier to compute using the formula for the pmf than it is to compute by force.

Some magic happened in there: I used my knowledge of sums of consecutive integers and sums of consecutive squares.  You’ll remember the triangular numbers from a previous chapter, but you may not know the sum of squares formula.  Here it is:

This is something I re-memorize when working through this section, and then forget when it’s not important.  Do what you like, but it is helpful to know by rote.

Variance is an even worse calculation, but at least we can simplify it by using our nice new formula.  var(X) = E[Z2] - 2

= k=16 k2 2k-136 - ( 16136 )2

Here you’ll need the sum of cubes.  You know what’s really cool?

The sum of the first cubes is the square of the th triangular number.  So

This is called Nicomachus’ theorem sometimes – Nicomachus lived in what’s now Jordan, in the Middle East, in the first century BCE. The theorem was also discovered by Al-Karaji, an Arab mathematician, and Aryabhata, an Indian mathematician, all at or before the year 1000.  (See https://www.math.nmsu.edu/ davidp/bridge.pdf for more information.)  Then it was discovered in France in the 1300s.  Thinking about the history of mathematical achievement and financial thought is humbling and instructive.  What drove these three mathematicians in different areas to figure out this number-theoretic theorem that also pops up in probability?  How did Thales of Miletus have the idea to become the first options or futures guy?  Why did the prophet Mohammed think so much about the ethics of futures contracts?  While synthetic CDOs are pretty new, the basics of hedging and portfolio management have been thought about for thousands of years.

Back to variance: Using the identity, we can simplify the variance calculation to

or

This is about .  Does this make sense?  Always do a quick “gut check” to catch errors – the variance is not bigger than all possible outcome values, the variance is not negative...  looks good!

Notice all this use of series.  Calculations of variance and expected value for discrete random variables often require a lot of series, and in addition, we can package expectation and variance as the first two moments in a moment generating function.  If you’d like to start reviewing series, skip to Section 5.3.  We’ll first discuss properties of expectation and variance just a bit more.

5.1.2 Multiplicativity of expectation?

It is so cool that expectation is linear.  Really.  If we were very lucky, expectation would also factor nicely: and would be equal.  Is this true?

To see whether this would be true in general or not, go back to probability itself: does probability factor?  That is, do we have for discrete random variables and taking values in the appropriate ranges?

Remember the discussion in Section 1.8 of compound experiments.  Before having the language of random variables, we argued that if two events and are independent, then .  Now we can translate this into the language of random variables and define independence of discrete random variables as follows:

Definition 1.1 Two discrete random variables and are independent if for all in the range of and all in the range of , we have

If and are not independent, this won’t be true for all combinations of and – there will be some event for which this equation is false.

A very simple example: flip a coin once.  Let be the number of heads, and be the number of tails.  The probability of getting heads and tails on the same flip is zero (the events are mutually exclusive), while .  It’s easy to see that and are related just by thinking about it: relates the two nicely.

Another way of looking at this is to say that the joint probability mass function of independent random variables and factors into the probability mass functions for and .

Look at the definition of expected value and see if you can prove the following fact for discrete random variables: If and are independent random variables, then

as long as both and exist and are finite.

This fact is also true for continuous random variables, using the following defintion of independence:

Definition 1.2 Two random variables and are independent if for all in the range of and all in the range of , we have

You’ll see why we needed to change the equalities to inequalities when we discuss continuous random variables.

5.2 Variance: average of squared deviations

Variance is a subtle quantity, initially hard to grasp.  It might help to contrast it with other measures of variability in a set of data.  We want to look at how a data set is scattered, and there are many ways to do that:

- the range of the data is useful (max-min) but not very stable with respect to outliers;

- quartiles can give you a sense of the data but don’t give one statistic by which to compare different data sets;

- the average of absolute values of differences from the mean gives a fine statistic of variability.

However, in the end one method has come to rule them all: average of squared deviation.

Let’s explore measures of “scatter” through an example.  We’ll look at three random variables, , , and , with probability mass functions given in the figure:

I’ve engineered each random variable so that they all have the same mean, .  The deviation of each outcome from the mean would be , and similarly for and .  Try taking a weighted average of these deviations to give some measure of “scatter.”  Why is it zero each time?  Because linearity of expectation means that .  Fine, use absolute deviation instead: take the expected value of to get the absolute mean deviation.  Calculations for , , and :

This seems fine, but for better or worse it doesn’t give much weight to outliers.  It’s a statistic used in some time series calculations where you don’t want to give too much weight to outliers, but maybe in some situations where you want to measure “scatter” weighting the outliers is part of the point!

Variance is instead the mean of squared deviations.  Squaring the deviations results in positive numbers, so there’s no cancellation of deviations to worry about.  Take a look: the calculations of etc. are below.

Outcomes further from the mean have a big effect on variance!  Look at that !  Take the square root to get the standard deviation, which has the same units as the original random variable.  This is the standard measure of “scatter” for data sets and probability distributions.  The name “standard deviation” comes about because of this historical community decision, even though for some applications you might decide to use an alternative statistic.

5.3 Series

A power series is a function that is an endless sum of monomials – like a polynomial, but of infinite degree:

A power series can take in a real number or a complex number.  Here we will consider only power series whose input is real; further on we’ll look at functions of complex numbers.  However, throughout the book we will only consider real coefficients .

Why consider these power series?  Two reasons: to set a good framework for approximations (Taylor series), and to understand moment generating functions.  We’ll concentrate on the first reason in this subsection.

Power series are a natural generalization of the “short-term approximation” or linear approximation of the last chapter.  Often a more accurate short-term approximation can be gained by taking a higher-degree polynomial as the approximating function.  We’ll talk about quadratic approximation, for instance, and some of you may have encountered cubic splines for numerical interpolation.  These are polynomials, rather than power series, but you’ll find that it’s really powerful to have the power series framework in which to consider all these approximations.

Example 3.1 Let be a function that can be differentiated as many times as we need.  Determine a third degree polynomial so that , , , and .

The example above brings us directly to Taylor series, power series representations of a function at .  If we want to write a power series representation of a function at using a Taylor series, each coefficient is given by the following formula:

Here, is the derivative of at , for , and .  Basically, we are trying to make the slope, convexity, and higher derivatives of our approximation match the original function as well as possible around the point .  Then if all the derivatives exist and this infinite series converges, we have

In the following example, each series converges for all .

Example 3.2 Find the Taylor series at for .

By contrast, the next two series converge only for .  What goes wrong at the edge of this interval?

Example 3.3 Find the Taylor series for (geometric series) and (a harmonic series).  (Why am I not asking for the Taylor series for ?)

5.3.1 Convergence, divergence, well-defined

Three big concepts come up in working with series: convergence, divergence, and well-definedness.  I just threw you into finding Taylor series and used the word “convergence” without defining it, because I expect the reader (you) to have some experience of these concepts.  But let’s look carefully at them:

The Taylor series for at is .

- For , for instance, this series works beautifully and sums to 2.  (Use a geometric argument to justify if you like).  The series converges, as it sums to a unique, finite, real number.

- What happens at , though?  does not sum to any finite number – it goes to infinity.  We say that this series diverges.  This reflects the behavior of the original function; isn’t defined because it prompts you to divide by zero.

- What about the series at ?  It’s no problem to compute .  However, .  Now, is this or ?  or something else?  This sum is not well-defined: it doesn’t have a unique, unambiguous answer.  It is also divergent, then, because it’s not convergent!

5.4 Moment generating functions

Remember that is a pretty remarkable function.  It is the only function that is it’s own derivative.  You can phrase that as a differential equation if you like:

or

What does this differential equation mean for the Taylor series at ?  Well, since the equation gives

we can deduce for all .  Since as , we know , , , and .  Hence the Taylor series for is

Why do we care about this?  Because this series provides a great package for expectation, variance, etc.  Let’s define the moment generating function for the random variable by

Generating functions are used in combinatorics and all sorts of other fields to package numerical information in a series.  The variable is called a formal variable, and basically it is a placeholder that is there simply to give you its degree (from ).  This degree is part of the packaging that the generating function does.  For the moment generating function, the coefficient of is the th moment.  The th moment of is the expectation of , .

Check it out:

So the moment generating function of is defined to be

Let’s consider a discrete random variable that takes only positive integer values.  It has a probability mass function that takes the range and gives an output for each .  We can combine this with the generating function calculation above to calculate each , using , carefully regrouping terms:

You can see that the first moment, the coefficient of when is expanded, gives you .  The second moment gives you which you can use to get .  Higher moments won’t be discussed now, but I will say that as we study different special discrete and continuous random variables, you’ll see that we can classify their distributions using moment generating functions.

5.5 Linear and affine linear transformations

Above, we discussed the linearity of expectation.  You can prove that

for constants.  This means that expectation is a linear operator: it satisfies

and

the two conditions for linearity.  You are familiar with several other linear operators, too:

and

so differentiation is a linear operator, and integration is as well (convince yourself!).

Quick quiz Is the function linear?  Is squaring linear?  What about sine or cosine?  What about variance?

You’ll notice that none of the above mentioned functions or operators are linear!  For instance, because squaring (taking to ) is not linear, variance is not linear either.  Check for yourself to see

What happened to the ?  Do the math to check it out.

A linear function is not exactly what you might think, though.  Quick quiz: which of these is linear?

Right: although both equations have a line as the graph, only one is linear in the technical sense.  (We abuse this langugae all the time.)  We say that is a linear equation and is linear because describes a linear vector space in .

Definition 5.1 A vector space, or linear space, is a set of vectors that is closed under addition and scalar multiplication.

In particular, the set is linear because if a point and a point both satisfy the equation, then you can check that also satisfies the equation.  Also, if satisfies the equation, you can check that also satisfies the equation for any .  Dead giveaway: the graph of is a line that goes through the origin.

By contrast, if you look at the equation , you can see that these properties – closed under addition, closed under multiplication – are not true of the points that satisfy the equation.  But the graph is a line!  Yes: this equation is an affine linear equation.  It’s a shift of a linear equation, and its graph is a shift of a linear space.  We call these shifts affine transformations.

Comfort with transformations of functions translates very well to probability applications.  By transformations, we mean stretching, scaling, and translating functions.  We can accomplish these tasks and more using vectors and matrices.  Let’s bravely step into the two-dimensional world and consider vectors in .

Rewrite as .

Notice that the above equation involves shifting and shifting and scaling .  We can write this using two equations, one for and one for , or using matrices and vectors.

Set of equations: and gives the desired shift in the previous example.  Using matrices and vectors, we can write this as

This is scaling (matrix multiplication) and shift (affine transformation).

You need to get comfortable alternating between viewing as a vector and as a point in the plane.

Quiz: In general, if we scale in the -direction by and the -direction by , and shift by , we can write: what?

To ponder: We could do these computations before without matrices; why bother?  To be discussed...  How can you “go backwards” and undo a scaling and translation?

5.6 Specific discrete random distributions

5.6.1 Bernoulli

The Bernoulli random variable and Bernoulli distribution are a building block for other discrete random variables and their distributions.  The Bernoulli random variable takes on the value of 1 with probability or with probability , with .  Here we identify , the set of possible outcomes, with the range of the random variable .  Notice this distribution obeys the axioms of probability: with our identification of and , we have

we have

and

and we have

because the outcomes and are disjoint.

Bernoulli random variables are useful for the endless variety of coin-flipping problems, but you’ll also find them useful as indicator random variables for solving more complicated problems.  As you’ll see you can use sums and products of Bernoulli random variables to come up with many of the following discrete random variables.

The expectation of is

and the variance is

5.6.2 Binomial

The name comes from the binomial expansion, of course, which we’ve seen so many times:

The binomial distribution asks, in coin flips, what is the probability of heads?  Use the Bernoulli trials just discussed to rephrase this: in Bernoulli trials, what is the probability of successes?

Very explicitly, a random variable distributed binomially is a random variable depending on the parameter , taking on values in .  This can be interpreted as the number of successes in trials, where each trial is independent and success has probability and failure has probability .  The probability of succeeding exactly times in trials is then

This expression is the probability mass function for the binomial random variable.

To figure out the expected value of , let’s use the idea of indicator random variables.  Let be the result of the th trial, with if you succeed and if you fail.  We already know and .  Then counts the number of successes you have.  Using linearity of expectation, we know that

This is an easy computation, since .  We have

Likewise, the variance computation for is easy if we use the property of variance and the fact that each trial is independent, so for .  (Independence is crucial here!!)  We can just sum the variances of the indicator random variables to get

At points in this math finance journey you may wonder, where is the finance?  Here I refer you to the pioneering paper of Cox, Ross, and Rubinstein, in which they use a binomial distribution to provide a discrete model for derivatives prices that is easily implemented on a computer and in the continuous limit produces the Black-Scholes-Merton model for pricing European options.  Go read it right now and be amazed that you can read some foundational literature!  Notice that one reason the Cox-Ross-Rubinstein and Black-Scholes-Merton basic models converge is that for large values of , it’s possible to approximate the (discrete) binomial distribution using the (continuous) normal distribution.  We’ll discuss this when we talk about the Central Limit Theorem.

5.6.3 Geometric

Say you’re doing a series of Bernoulli trials again, but now you’re considering how many trials you must carry out until you reach your first success.  Another way to think about this is how many failures occur before the first success.  The probability mass function for either of these is built on ideas of conditional probability:

- you assume for that you fail times and then succeed on the th trial, for in , or

- you assume for that you fail times and then succeed on the next trial, for in .

As before, let the probability of success on a given trial be and the probability of failure be .  Writing out the first few probabilites in the probability mass function, verify that

It’s easy to package this into one function:

Follow the same process for :

Instead of using indicator random variables to calculate expectation and variance, we’ll use what we know about series.  For , we can do the following calculation (and if or , we don’t need to calculate anything!):

The identity we used here is that for , is the derivative of .

5.6.4 Hypergeometric

Now say you’ve got red balls and white balls in an urn ( total items) and you’re picking out items all at once.  What’s the probability of red balls in the you pick out?  This is the hypergeometric random variable .

While I haven’t seen hypergeometric used a ton in financial mathematics, Henriksson and Merton used the distribution in their 1981 paper, “On Market Timing and Investment Performance.  II.” It also comes up in various exams you may need to pass on your career path.  Very briefly, Henriksson and Merton look at what kind of results you’d get by choosing investments at random from losing investments and winning investments.  If forecasters do no better (or do worse!)  than the hypergeometric distribution, then their methods are not useful.  Of course there is much more to the paper; check it out to see for yourself.

What’s the probability mass function for , the number of red balls among balls picked out of an urn of red balls and white balls?  The probability that is the ways to choose red balls times the number of ways to choose white balls, over the total number of ways to pick balls from .  That’s

To find the expected value of the number of red balls picked out of the urn of total balls, use indicator variables again.  Let be 1 if the th ball is red and if the th ball is white.  Then the total number of red balls is .  We can use linearity of expectation (best thing ever!)  to break this into computing the and summing.  The probability that is one is

and so

Thus

Use indicator variables to find the variance, too – you find that

Work through that and check your work:

5.6.5 Poisson

The Poisson process is a bit different than the others discussed so far.  It’s not simply a combinatorial extension of Bernoulli trials.  Instead, the Poisson random variable is often used for events happening in time.  How many earthquakes happen in a given time period?  How many times does volatility spike over a certain time interval?

The Poisson distribution has been used to model market shocks via jump processes (adding jumps into a model of the market, with frequency described by a Poisson distribution).

Define a Poisson random variable as one that takes on the values with the probability mass function

Here is a parameter often called the event rate or rate parameter.  It gives the average number of events per time interval.

5.6.6 Negative binomial

How many MBAs must a financial firm interview before finding exactly good candidates?  This is an example using the negative binomial distribution from Analysis of Financial Time Series by Tsay.  If each candidate is independent and each applicant has probability of being a good fit, and the total number of interviews necessary is the random variable , then in the literature both and may be described as having the negative binomial distribution.  I’ll delve into this a bit because it’s good practice with discrete probability, not because of any particular use for financial math.

First let’s look at the probability mass function for , the total number of interviews necessary to get exactly good candidates.  Each interview is a Bernoulli trial with probability of success and probability of failure .  The number of good candidates desired is a fixed positive integer!  With the set-up given, where is the total number of interviews/Bernoulli trials necessary, the last interview or trial must be a success (if it were a failure, either it was not a necessary interview because good candidates had already been identified, or not enough good candidates were yet identified and so the trials should continue).  Call this last interview the th interview.  In the last interviews, there must have been successes.  How many ways can those be arranged?  Use a binomial coefficient to count this: ways.  There are total successful trials, each with probability , and unsuccessful trials, each with probability .  Thus

for (You should check for yourself that for smaller than the probability must be zero.)

We can split this random variable up as the sum of geometric random variables , where each is the number of interviews necessary to get from success to success .  Use linearity of expectation and the fact that to see that

Note that these are independent, as well, so we can find variance this way too:

If you’re reading a variety of sources on your probability journey, you might notice that Tijms, Tsay, and Grinestead and Snell define the negative binomial distribution as I have above.  On the other hand, Wikipedia defines the negative binomial distribution as the number of successes you see until a predetermined number of failures has occurred – so if you flip “failure” and “success”, the definition in Wikipedia is looking at the random variable instead of .  (To repeat the set-up at the start of the section, is the number of failures occurring before successes, and is the total number of trials necessary for successes.)  This gives a slightly different presentation of the probability mass function because something else is being counted.  I will not flip my successes and failures to match Wikipedia, but let’s look at the probability mass function for , the number of failures until successes.

Let’s think this through: if there are no failures, just successes, then and .  If there is one failure and successes, then and .  If there are failures and successes, .

5.6.7 Transformations of discrete random variables

One last thing that may help you grapple with discrete random variables is this theorem:

Theorem 6.1 If is a discrete random variable and , then has the probability mass function

and has expected value

for the range of values that can take.

This is a very streamlined and useful theorem that applies to any type of transoformation of !

Chapter 6

Continuous Random Variables

In the previous chapter we considered Poisson random variables, for instance the number of earthquakes that occur in two years.  While the number of earthquakes is necessarily discrete – an integer value – the time between two earthquakes can take values on a continuous domain.  Times and distances are natural settings for continuous random variables.  We often see continuous random variables coming up from geometry questions.  In addition, continuous settings are often an nice idealized environment in which to approximate solutions to discrete financial problems.  Stock prices technically don’t take values in a continuous range, but using a continuous approximation for the distribution of stock prices allows for very fast computations.

Continuous random variables are defined by the existence of a probability density function, or pdf, that characterizes the behavior of the random variable.  This pdf must satisfy the following properties:

and

Two of these equations come from the axioms of probability: if could take on negative values, we could get negative probabilities (from the definition of ), and if the integral of over was not 1, we’d be violating the axiom .

The function is called the cumulative distribution function or cdf for the random variable .  Notice a few things about :

because of that axiom, and by the Fundamental Theorem of Calculus.  These will both come in handy!

A few more technical notes: since we here make the decision to define as an integral of a pdf, will be continuous for all and differentiable at all but a finite number of points (differentiable almost everywhere).  You can take a slightly different approach and define to be a continuous random variable if it has a cdf that is continuous for all and differentiable at all but a countable number of points, but to think through the ramifications you need some measure-theoretic tools that I don’t want to develop in this course.

6.1 Geometry problems

Geometry is a great place to start an examination of continuous random variables.  For a little while, we can work in the sanitized and idealized world of pure geometric shapes – we’ll delve into messy data soon enough.  In this section, we’ll just do example after example to illustrate how to work with pdfs and cdfs.

6.1.1 Max or min in unit square

Pick a point at random in the unit square, and in .  What is the probability that the maximum of and is less than any given value ?

To rephrase that, we can invent a new continuous random variable, .  Now we ask, what is the cumulative distribution function ?

This may be familiar from previous geometry problems, and there’s a good chance you’ve already done this calculation.  Draw a picture and notice that for any given , the set of points with both and less than is a square with one vertex at the origin and side lengths .  The cumulative distribution function is in fact

Check that when we’ve got , and that when we’ve got .

Working backward from this cdf, we can find the corresponding probability density function .  It’s just the derivative of , by the Fundamental Theorem of Calculus and the definition of .  Thus

Notice that this is not continuous, and we don’t care.  Moreover, if you’re very sharp you’ll say, hey, the function was not differentiable at or – how can you assign a value to there if you’re just taking the derivative of ?  I would respond, you’re right that is not differentiable at zero or 1.  To make nicer, I’m just assigning values to and that are consistent with the demands that and .  You can assign other values to and if you like!  Yes, this leads to the troubling realization that “the” pdf of a random variable is not really unique, but measure theory will (someday) assure us that it won’t change any probabilities.  The values of the integrals are not affected.

Your turn: find the pdf and cdf for the continuous random variable , where for a point chosen at random in the unit square.

6.1.2 Absolute values

Another example problem is finding the cumulative distribution function and probability density function for , where is a point chosen at random in the unit square .  The absolute value in this definition of is a complicating factor, which makes this a good example!

Again, we’ll use basic geometry to figure out the cumulative distribution function (cdf) for .  First, start with what you want to know and substitute:

Draw a picture of the unit square with the lines and , remembering that is a constant of your choice, then figure out what set satisfies both desired inequalities.  (A mental game you can play is this: imagine an outsider gives you a series of constants to try for , ranging from negative to positive numbers.  What scenarios occur?)  In this case, we want all the points that are within distance from the line .  Call this set of points .  For a value of between and , this has area .  Thus

giving us our cdf:

Differentiating to get the pdf, we have

Notice that this probability density function isn’t continuous, and it takes values greater than one.  Those are both fine!  Probability density functions need not be continuous, and since they are not actually probabilities, they can also take values greater than one.

6.1.3 Circles: you mean I need to remember trig substitution?

Here’s another example, inspired by a homework problem in Tijms’ “Understanding Probability” book: Imagine you’re on the High Roller Ferris wheel in Las Vegas.  It is 550 feet high.  The power goes out and you are stopped at a random point.  What is the probability that you are stuck at a height greater than 412. 5 feet?

Our strategy will go as follows:

1. Figure out what random variable we want to know about.

2. Find a way to convert this into a random variable about which we have more information.

3. Use the information about the cdf of to find the cdf of .

4. Use the cdf of to find the desired probability.

We want the cdf of the height in feet of our car on the Ferris wheel.  The difficulty is that is not uniformly distributed – because of the shape of the Ferris wheel, we move faster through heights near 275 feet than heights near the top or bottom of the Ferris wheel.  What related quantity is uniformly distributed?  The angle of the spoke supporting our car on the Ferris wheel!  The angle from the vertical of the spoke supporting our car moves smoothly from zero radians from the vertical, at the bottom of the wheel, to radians from the bottom vertical, at the top of the wheel.  We can use symmetry to consider the random variable uniformly distributed on the interval , and then see height as .

Since is uniformly distributed on , we have

for .  Thus

6.2 Expected value and variance

The expected value of a continuous random variable is very analogous to the discrete case, just using an integral instead of the corresponding sum:

Likewise, while the variance of a continuous random variable is again defined as , it can again by linearity be written as :

6.3 Transformations of continuous random variables

Above you’ve used the fact that the probability density function is the derivative of the cumulative distribution function several times, and we can use this again to prove a general theorem for nice transformations of continuous random variables.  Here, “nice” means monotonic.  A function is monotonically increasing on an interval if its derivative is always positive on that interval, and it’s monotonically decreasing if its derivative is always negative on that interval.  You’ll see where this is a useful condition.

Let be your transformation of the continuous random variable .  Assume you know the probability density function for , and thus the cumulative distribution function.  What’s the probability density function of ?  Find it via calculus: first, .  If only we could get alone, as we understand !

Aha – if we can invert the function , this is possible.  That’s where “monotonic” comes in.  If a function of is monotonic on an interval, it can be inverted on that interval.  There’s a unique function such that and .  The graph of the function passes the “horizontal line test” you may have learned at some point – a horizontal line intersects with the graph of at only one point in the interval, which means that the graph of the reflection over the line is a function.  That reflection is the graph of the inverse.

Back to the proof: if is a strictly increasing function,

Thus by the fundamental theorem of calculus and the chain rule,

If is strictly decreasing,

and by the fundamental theorem of calculus and the chain rule hY(y) = ddy P(Y<y) = ddy g-1(y) fX(x) dx

= -fX(g-1(y))ddy g-1(y). We can make the formula more compact by writing

6.4 Markov and Chebyshev

Here I’ll introduce two inequalities that do happen to hold for both discrete and continuous random variables.

Markov’s inequality says that if is a random variable taking only nonnegative values, then for any we have

Notice the lack of conditions on the random variable !

We can cleverly transform this into a related inequality, Chebyshev’s inequality, by letting and using an absolute value.  Chebyshev’s inequality says that for with mean and variance ,

Why bother with these inequalities?  First, they give us rough but useful bounds on probabilities.  You can detect lies and mistakes with these bounds.  Second, they are very useful in various proofs.  Neither inequality has a lot of hypotheses that the random variable must satisfy, and so you can apply them in many situations.

6.5 Important distributions

Here, since we’re focusing on applications to financial mathematics, the discussion will first relate the exponential distribution to the Poisson distribution you saw last chapter, and then discuss the normal and lognormal distributions and their applications in financial math.  Last, we’ll touch on a few other continuous distributions that come up in data analysis and finance.

6.5.1 Exponential distribution and Poisson

A continuous random variable has the exponential distribution with parameter if it has the probability density function

One common use of the exponential distribution is in modeling the time until a rare event occurs, or the time between rare events.  Conceptually, this is why it’s related to the discrete Poisson distribution – we’ll formalize this in a theorem below.

The expected value of is , and the variance of is .  The cumulative distribution function is

Computing all of these is a good exercise (do it!).

Unfortunately, there is notational ambiguity when people write about the exponential distribution.  Another way of expressing the probability density function is

where is the mean and the standard deviation of the exponential random variable.  When this notation is used, is called a survival parameter.  This comes from models in which the random variable represents the time a system or organism manages to survive.  How do you know which notation an author is using?  Look at units: and must have the same units.  Say is the time before some event happens.  Then will have time units and will have units “events per unit time”.

Example 5.1 Imagine that the time to failure of a type of hard drive widely used by a large company is exponentially distributed, and on average the hard drives break every eight years.  What is the probability that a hard drive will break this year?

First identify the parameter.  We have “on average the hard drive breaks in eight years”, so years.  That means .  Then , which is an approximately percent chance the hard drive will break in a year.

What’s the chance that a hard drive will break in eight years or less?  Same process: , which is approximately percent.  Why isn’t this fifty percent?  Because the mean and the median are different quantities.  The exponential distribution has a long right tail, in this case pulling the mean to 8 years while the median time to breakdown is about 5. 5 years.  (Can you find the exact median time to failure, using the definition of the median as such that ?)

A very important property of the exponential distribution is memorylessness.  This is an amazing property that roughly says that it doesn’t matter when you start counting: the time to your rare event is always (probabilistically) the same.  More precisely,

for all .  This is also easy to prove!  Take a look:

The exponential distribution is the only continuous distribution with this property!  (Which discrete distribution also has this property?)

Example 5.2 You’re considering the hard drive that fails on average every eight years again.  One of the hard drives has been working for three years.  What is the probability that it will fail in the next year if failure is truly modeled exponentially?

Yep, memorylessness says that the previous three years of service don’t matter.  They don’t make failure any more or less likely, so again you have the approximately percent chance of failure in the next year.

In practice, failure rates are usually increasing or decreasing, rather than constant (as in a pure exponential distribution).  Many components are more likely to fail as they age.  Amazingly, spacecraft tend to be less likely to fail as they get older (see “Impact of the space environment on spacecraft lifetimes” by Baker and Baker).  This happens when catastrophic events are likely to reveal themselves early, and if they don’t occur the device is likely to work a long time.  In these cases, the memorylessness property does not hold, and another distribution is more appropriate.

Last, let’s make formal the relationship between the exponential distribution and the Poisson distribution.  Let be exponentially distributed random variables, all independent and identically distributed with parameter .  Interpret each as the time between event and event in a sequence of events.  For any , define as the number of events occuring in the time interval .  Then we have the following theorem:

Theorem 5.1 With as above interpreted as waiting times between events, and the number of events occurring in the interval , we have

for .  Notice that this means is Poisson with .

Likewise, you can prove that if a sequence of events is Poisson-distributed, then the waiting times between events are exponentially distributed.

It is very useful to be able to switch back and forth between consideration of the Poisson and exponential distributions.  You saw some examples above, but I’ll give three examples to be very explicit about which distribution to use when.  Here let be the number of events in time interval and let be the waiting time until the first event.

- What’s the probability that no events occur in the time interval ?  Use either: you’re looking for (probability of no events) or for (probability that first event happens after this interval).

- What’s the probability that five or more events happen in the interval ?  Poisson: it’s easy to take .  It is less easy to figure out the sum of s, where for is the waiting time from the th event to the th event.

- What’s the probability that you wait more than minutes for the event?  Exponential – it’s tailor-made for this question.  Here, you really care about the continuous random variable of time, rather than the number of events that may occur before or after minutes.

In the second situation, you might think you could just look at the sum of five or more exponentially distributed random variables.  The part I find annoying about this approach is that you need to figure out and discard the probabilities that , , etc., and then look only at the probabilities that for .  It’s just awfully complicated.  However, it does bring up a question: what’s the distribution of a sum of independent exponentially distributed random variables?

Write for the sum of exponentially distributed random variables.  The sum has the Erlang distribution.  We can use convolution for continuous random variables to find the distribution for (although we’ll call it for the next two paragraphs to cut down on the subscripts needed).  Convolving the probability density functions for and gives the probability density function for .  Define the convolution of probability density functions and to be

The is the symbol for the convolution operation on two functions, just as denotes addition and and denote multiplication in .  We’ll put convolution to use right away by finding the distribution for the sum of two exponentially distributed random variables that have the same parameter .  Just to keep our calculation cleaner, call the exponentially distributed random variables and and their sum .

Let and be the probability density functions for the exponentially distributed random variables.  Then:

Wait, what?  How do you get to change the limits of integration from and to and ?  The secret is that

and likewise for .  The part of the function that is zero is really important: whenever , so whenever .  Similarly, whenever .  Put those together: the integrand is zero if or if , so the integrand is only non-zero if .  Having cleared that up, finish the calculation:

You can extend this to the sum of independent exponentially distributed random variables by induction.

6.5.2 Normal distribution and lognormal distribution

Here we’ll talk about three things: the standard normal distribution, the normal distribution in general, and then the lognormal distribution.  This discussion may motivate you to understand why transformations of random variables are important to understand more abstractly.

A continuous random variable has the standard normal distribution if it has the probability density function

for .  Here we are using for the random variable and for the pdf very consciously – these are traditional notations that indicate we’re dealing with the standard normal rather than a more general normal distribution.  The standard normal distribution is the only normal distribution with mean and variance .  The cumulative distribution function for is

There is no nicer formula for this cdf; in particular, there is no closed form.  Today we often use technology to compute values of , but we also make use of the traditional z-table for looking up values of given values of .

A continuous random variable that is normally distributed with mean and variance can be obtained as an affine linear transformation of .  Let’s introduce a nice notation for indicating that a random variable is normally distributed with mean and variance : write

It’s a property of normal distributions that if , then .  In particular, we can see that is a transformation of a standard normal random variable , as by the above property .  This also tells us how to standardize a normal random variable :

brings us back to the standard normal.

Why do we care?  The probability density function for has a lot of symbols:

for .  (Note .)  The cumulative distribution function, again, has no closed form:

There aren’t tables to look up these values except when and .  Before the availability of computers, using the affine linear transformation to get back to a standard normal cumulative distribution function was the only way to calculate values for , and even now, it’s often still the easiest way as algorithms for computing are optimized in computational software.

Another really nice property of normally distributed random variables is that if and are independent and normally distributed with and , then is also normally distributed.  Because and are independent, both their means and variances add, and .

Now let’s look at the lognormal distribution.  A continuous random variable is lognormally distributed if .  Another way to look at this is that for some that is normally distributed.  This relationship tells us that the cumulative distribution function for a lognormally distributed is

for .  As a consequence, we get the probability density function

From this you can deduce that and .  Self-quiz: since , why don’t we have equal to ?  This is actually quite important to understand in the context of finance.

6.6 Central Limit Theorem

The Central Limit Theorem has been called “the queen of theorems” in probability, and indeed it is amazing – and one of the primary reasons we care about the normal distribution.  In essence, the Central Limit Theorem says that under certain conditions any large enough number of independent and identically distributed random variables sum to a random variable that has an approximately normal probability density function.

Let’s be more precise.  Let be a sequence of independent, identically distributed, real-valued random variables with mean and standard deviation .  Consider

for any natural number ; it’s a partial sum of the s.

You, personally, can prove using linearity of expectation and properties of variance that and .  It’s worth pointing out that as these quantities also approach infinity (if ).  So it’s not correct to talk about the limiting distribution for , despite the temptation.  Instead, consider the standardization of :

Theorem 6.1 The Central Limit Theorem: as , the distribution of the standardization of converges to the standard normal distribution.

Another way to state this is to say:

Theorem 6.2 For independent and identically distributed random variables with mean and variance , and ,

How big must be for the normal approximation to be “good”?  In many situations, an of at least 30 will be big enough, but truly it depends on the original distribution of the .  In addition, there is another technical problem when the have a discrete distribution.  Approximating the resulting discrete distribution of or by a continuous distribution means that the event should correspond to for – but different values of give different approximations.  The tradition is to use and not worry too much.

Chapter 7

Random walks

Random walks bring us from discrete probability to continuous motion.  We know how to look at the results of sequential coin flips.  We now know about the normal distribution, as well.  Last, we know about the Central Limit Theorem, which gives us some results about sums of independent identically distributed random variables.  However, in finance we care about the final distributions of stock prices and returns and we often care about the path taken as well.  And we know that this financial info inhabits a netherworld between discrete and continuous

7.1 Simple symmetric random walk

A simple symmetric walk has a name that makes sense.  Consider a walk along a line of integers.  You start at zero and move right or left one integer unit with equal probability.  Mathematically, we can say that for the th step , we have pmf

The simple symmetric random walk is an integer-time stochastic process where each is the sum of the steps that occurred before time :

We set .  Then notice that each gives the position at time (or after steps, since we’re taking unit-time steps).  If you want to know the distance from the origin at time , you look at , while the number of steps taken is just .  We’ll come back to the notation when we need it, and for simplicity just talk about for now.

The simple symmetric walk is a great setting in which to ask a lot of questions:

- What’s ?

- What’s ?

- What’s the probability that falls below a certain value or hits a certain higher value?

- How long would it take on average to reach a certain value?

We won’t answer all of these questions yet, but you can imagine why they’re important and why they came up: if I bet $20 on each round of blackjack on the Strip in Las Vegas, how long do I expect to play until I go broke?  (Tip: you can find lower-priced tables if you go to old Las Vegas and get off the Strip).  If I switch to $5 tables, how long could I expect to play?  I do need to throw in the caveat that these questions more properly belong in the asymmetric random walk section due to the house advantage.

7.1.1 Expectation and variance

The expectation and variance of the simple symmetric random walk are easy calculations that show off the power of the linearity of expectation and the delight of independent random variables.  Remember that each step is of size one.  Since this walk is symmetric and starts at zero, notice that for all .  (Check!)  Thus

Variance is another easy calculation: you should check that (check the signs!), and remember that the steps are independent from each other.

Thus the standard deviation of the position after independent steps is , which may or may not be surprising to you.  While simple, this property of the simple symmetric random walk will play a large role below.

For finite , we can approximate the distribution of using the Central Limit Theorem.  We can say that for large ,

It’s useful to look at a picture of this set of random walks.  Here I generate 180 simple symmetric random walks in Python:

Notice that there’s a cone of possible paths, with the position bounded above by and below by every step of the way.  If we plot a histogram of the values of these 180 paths, we see that the validity of approximating position using the normal distribution begins to emerge:

As we consider asymmetrical random walks, we want to see how the cone of possible paths and the histogram of endpoints change, and as we transition to smaller and smaller steps in the walk, we want to maintain this cone of possible paths.

7.2 Asymmetric random walks

The first generalization of the simple symmetric random walk is the random walk with asymmetric probabilities.  All we do is consider

where .  How does this look?

Here is a picture of a random walk with one step of size one for every unit of time, with :

We can look at the following simulation of 180 such random walks.  Note the black line across the top, corresponding to :

Notice that changing and doesn’t change the cone of possible paths – the paths are still bounded by above and below at step – but it does radically change the general trajectory of the paths.  Let’s quantify this by looking at the expected value and variance of .

7.2.1 Expected value and variance

Since , we have the following calculation of expectation :

We plotted this as a line on the figure consisting of simulated walks above.  Here’s another example, with .

The variance calculation is mildly more interesting:

and with independence of the steps,

For what value of is this variance maximized?  What does that mean in terms of the walks we can take?

You can calculate the value of maximizing using calculus or using symmetry of the parabola : no matter what method you use, variance is maximized for .  Figure out how to justify this to yourself using the pictures above.  Also confirm to yourself that variance is minimized for or , and understand why.

7.3 Scaling time or space

Instead of adjusting the probabilities, we could scale the frequency of the steps or the size of the steps.  Let’s return to the symmetric random walk so that we can mess with one idea at a time!

Return to the symmetric probability setting , but take two steps in every unit time – or three, or .  Before, each step took one time unit, and we conflated “number of steps” and “total time units”.  Now we need to be more careful.  Let’s say that we take steps in one unit of time, so in units of time we take steps in the random walk.  Another way to think about this is that in one unit of time we take steps, each taking time length .  We’d like to design a discrete-time stochastic process that just gives a finer and finer random walk, without blowing up the position of the walker (as changes we want variance of for any particular time to stay constant).

For example, here are twenty random walks that each have

- a step distance of 1 unit per step

- ten steps per unit time (so , while .)

- all over total steps or 100 units of time

Look how some paths get near or just 20 time units in.  The variance of this random walk process is much larger than our previous random walks: for this particular set of 20 trials, we have a variance at time 100 of .  Variance is about ten times bigger than the time length of the random walk, and that’s no coincidence.  What if we let ?  Then variance would be about at time .

As grows, we’ll simply move away from the origin more rapidly as we walk – the variance of the position at time will get larger as gets larger.  This does not preserve the properties of random walks that we wanted.  We want a finer random walk (more steps per unit time) that keeps variance at time constant even as changes.  Scaling only the frequency of steps we take (time) while leaving the length of the step (space) at one results in a variance of at time .

Solution: as we scale time, we must scale space as well.  If we move more often we’ve got to take smaller steps in order to keep variance constant with respect to step frequency.  We want the variance of our stochastic process to be at time , not .  This can be accomplished by scaling distance to take out that extra :

Take a look.  In this set of 30 random walks, we took

- ten steps per unit time (so , while )

- each step of length about

- all over total steps or 100 units of time.

At time 100, the variance for the final endpoints of these random walks is – much closer to 100, as desired

7.3.1 Special properties

Three special properties stand out for these random walks that will also be very important in the discussion of Brownian motion:

- Idependent increments: for , and are independent.

- Expected value is zero for our symmetric set-up, , and variance is proportional to interval: .

- By the Central Limit Theorem, as grows large, is well-approximated by a normal distribution with the above mean and variance.

Once we transition to Brownian motion, we’ll see that independence of increments and the normality of the distribution at a given time are important carryovers.  The difference between our random walks and Brownian motion is that Brownian motion is continuous.

7.4 Arithmetic Brownian motion

Here we will only talk about one-dimensional Brownian motions.  (When we say one dimension, we mean one space dimension – we’re not counting time.)  It’s possible to look at two-dimensional or multi-dimensional Brownian motions, which could be useful when modeling a portfolio of stocks, for instance.

Brownian motion is a continuous-time process.  In the random walks above, we only took steps at times that were multiples of for an integer.  They were discrete-time stochastic processes.  Brownian motion involves steps at all times for .

Let’s build up from the simple symmetric random walk, .  Let be the scaled symmetric random walk defined before.  We can write a scaled random walk with drift and “diffusion coefficient” as

Recall this will have the properties that

- For , we have independent increments and .

- For , the distribution of depends only on .

- In particular, for each , is distributed as .

Take the limit as , and get a stochastic process.  Taking the limit preserves the properties of stationary and independent increments which are normally distributed.  So let be the limiting process

Now we have a real-valued process , , with

- Idependent increments: for , and are independent.

- For , the increment has distribution .

- is continuous with probability one!

This is called a Brownian motion with drift and diffusion coefficient .  The “standard” version, with and , is called standard Brownian motion or the Wiener process, and we often denote this by .

The relationship between any two random walks and is just a change of coordinates, scaling space and time if and are different and scaling and if they differ.  The Wiener process (Brownian motion) is the limit of a standardized random walk as goes to infinity (as step size goes to zero).  Thus Brownian motion is the continuous-time limit of a random walk.

Note that if we’re being very specific, we could call this an arithmetic Brownian motion.  Here we are drawing the distinction between arithmetic and geometric, and this corresponds exactly to the distinction between an additive binomial tree and a multiplicative binomial tree.

7.5 Geometric Brownian motion

Geometric Brownian motion is the next logical step.  If is an (arithmetic) Brownian motion, we can make a geometric Brownian motion by defining

This is again a continuous stochastic process, and we can show it’s got properties similar to the arithmetic Brownian motion .  The process has stationary and independent increments, just like .  On the other hand, you can see that at time zero we start with , any number we like, although we generally use a positive number in financial modeling.  (In particular we don’t use because that would result in a constant function.)  Last, since at a particular we have normally distributed , we also have a lognormally distributed .  Everything you learned about the lognormal distribution applies at each moment in time to a geometric Brownian motion.

Previously, we saw that (arithmetic) Brownian motion comes about from scaling and taking the continuous limit of an (additive) random walk.  Geometric Brownian motion comes instead as the limit of a multiplicative random walk: look at an initial stock price, , multiplied by factors at each time step:

If we model these as independent and identically distributed random variables, applying the logarithm allows us to write the equation additively and then the Central Limit Theorem applies.

The random variables are still independent and identically distributed.  You can imagine taking more and more multiplicative factors at shorter and shorter time steps, just as we did with arithmetic Brownian motion.

7.6 Solving Brownian motion problems

How do we solve these problems?

Example 6.1 With a standard Brownian motion , solving problems is often easy!  Since , we can standardize and normalize with ease.  Let be the standard normal continuous random variable.  Take a look:

Here we’re definitely using symmetry of the normal distribution.  A step up:

Now we’re using stationary increments: we know .  Mildly more difficult:

Since , this probability is between and using a -table. And now let’s combine some conditions:

Here I write to really emphasize that I’m looking at an increment that does not overlap with , even though they have the same distribution.  The distribution of what happens between times 2 and 3 doesn’t depend on prior behavior, even though actual position at time 3 certainly depends on position at time 2!  So I’ll end with a question we can’t solve with current tools:

Do you see any way to transform this into a question about non-overlapping, and thus independent, increments?  I don’t!  In fact, that position at time 4 is influenced by position at time 2.  We’ll need to figure out how, which will bring us to considering the covariance matrix of at integer times.

Questions about Brownian motion with drift are similar, although standardizing requires another step.

Example 6.2 Let be a Brownian motion with drift and diffusion , so

Find

At this point I always stop to check some things: has what distribution again?

Since , we’ll call it and write

Let’s get back to the calculation:

This is approximately

Likewise, questions about geometric Brownian motion usually just involve taking a logarithm and then following the steps shown above.  In list form,

- take appropriate logarithms to go from geometric Brownian motion to arithmetic Brownian motion

- translate from arithmetic Brownian motion to normal random variables

- standardize those normal random variables

- use a z-table and/or symmetry of the normal distribution to get numbers.

As illustrated above, you might get stuck if you can’t separate your computations into computations about independent increments – but otherwise the world is yours!

Chapter 8

Linear algebra

While I’m assuming you’ve encountered vectors and matrices in previous math classes, we’ll start with a short review.  Then in the next few chapters, we’ll cover elements of linear algebra, multivariable calculus, and differential equations that provide a nice base for financial math.  Financial information is almost always multivariate: as a portfolio manager, you manage multiple assets; as a risk analyst, you look at multiple risks.  Combining these classical multivariate topics with statistics will give you access to powerful mathematical techniques.

8.1 What is a vector? What is a matrix?

Most generally, a vector is an element of a vector space.  Often, we care about the vector spaces or , in which case

- the vector is an -tuple of real or complex numbers (a list in which order matters), or

- (equivalently) a direction with a magnitude.

There are examples that are rather different, but we’ll save those for discussion later.

We’ll use the notation for a vector, and can write for instance

for a vector whose components for are all real numbers.

A vector space or linear space is a set of vectors that is closed under scalar multiplication and vector addition.  This word “scalar” refers to a number that scales things – a stretch factor – and so for a real vector space, a scalar is a real number, and for a complex vector space, a scalar is a complex number.  Notice that any vector space must include the zero vector (which we write ) because if scaling a vector by any scalar is an operation that keeps the output in the vector space, we’ve got to be able to scale by zero.

Here are some examples of vectors and vector spaces:

Example 1.1 Think of all the vectors in that have zero for their -coordinate.  Call that vector space , and write as

Check that you can add or subtract two such vectors and still have .  Check that you can multiply the vector by any number in and stay in .  Remember multiplication of a vector by a scalar is done element by element, so .

Example 1.2 Another vector space is given by all the vectors such that .  You can graph this.  Do you know what surface this gives in ?[1]  Vectors in this vector space include and .  Check for yourself that this vector space is closed under scalar multiplication and vector addition (that is, for all , for scalars ).

If a vector space is contained in another vector space, we say it is a vector subspace of the larger vector space.  In our examples immediately above, both and are vector subspaces of , but since neither nor contains the other one, neither is a vector subspace of the other.  However, both and contain the intersection , the set of vectors in with and .  That vector space, , is given by

In fact, that’s a line through the origin given by and , and we could rewrite it as

If you like parameterizations (I do!)  we could use a parameter and write this yet one more way:

What is a matrix?  Mechanically, a matrix is made by stacking vectors as rows or columns to make a rectangular array of numbers.  In this book we’ll most frequently encounter matrices of numbers, but we also make matrices of symbols or expressions (you’ll notice this in the section on rotation matrices, for instance).

An example of a matrix of real numbers would be

This is a two by three matrix.  An example of a matrix of complex numbers would be

This is also a matrix.  When we need a very general ( by ) matrix called , we can write

You can multiply matrices by scalars (just multiply each element by the scalar) and you can add matrices (also element by element).  The observant among you might think, hmm, does that mean we can make a vector space out of matrices?  Yes, you can!  The vector space of all matrices with real entries is often called , or , or .  Somehow you have to indicate the dimensions of the matrix and what the entries are allowed to be.

8.2 Linear combinations, matrix multiplication, and planes

8.2.1 Linear combinations

First, let’s look at the idea of linear combinations of vectors and relate it to matrices.  Here’s a very simple motivating example:

The equation is a linear combination of the variables and .  It can be expanded as a dot product,

or as a product of matrices,

Here is another motivating example: An example of a linear combination of the vectors and is given by

Using matrix multiplication, we could also write

8.2.2 Matrix multiplication and dot products

Linear combinations of vectors can always be expressed via matrix multiplication, and matrix multiplication is built out of dot products.  The dot product of two real vectors and in is

We are only defining dot product for real vectors right now (complex vectors will show up in Section 9.10).  For real vectors, , so dot product is commutative (not true for complex inner product!).  Multiplication by scalars (real numbers) distributes over dot product, too: .  Often with commutativity we talk about associativity (for instance ), but for dot product this does not make sense: is not an operation that makes sense, as you can’t dot a vector and a scalar.

Geometrically, the dot product is related to the angle between the vectors and .  Pick the smallest possible angle between the two vectors, between zero and .  Define

If and are perpendicular to each other, then the angle between them is radians and .

This uses the magnitude of and : for , we can define the magnitude by

This should look a lot like Euclidean distance to you (if you’re not sure, just think about ).  It is exactly that, the distance from the tail of at the origin to the tip of the vector .  Notice too then that .  This will be useful.

Matrix multiplication, then, is built from dot products as follows: Let be an matrix with rows given by through .  Let be an matrix with columns given by through .  We can write the matrix multiplication as

Make sure you know the difference between a row vector and column vector!  A row vector looks like

for instance, while a column vector might be

Often people (like me) are rather sloppy and switch back and forth between writing a vector in as a row or column vector depending on how much paper they have.  That’s not terrible.  But being clear in calculations about whether you’re using a row vector or column vector is important.  For instance, what is the difference between multiplying two vectors as matrices and taking the dot product between two vectors with the same shape?

Example 2.1 Let and .  Then compute the following, with the knowledge that the stands for transpose (exchange rows and columns, or flip over the diagonal):

Now check your answers in the footnote.  [2]

8.2.3 Planes and parameterizations

In this section we’ll talk about equations of planes, but maybe it’s good to start with the equation of a line as a test case.  You can write the equation of a line in in several ways.  One of my favorite ways to write a line in is to parameterize it.  I will use the parameter .  It’s just a scalar.  Then we need a direction for the line – call it – and a point that lies in the line – call that .  A set-theoretic description of the points in the line, then, is

When we’re just at the point , and as ranges through the rest of , we just scale along the vector .

This concept of parameterization will be very useful in exploring planes and other multivariate curves, surfaces, solids, etc.  Among other things, parametrization can help us understand the notion of dimension both in linear and nonlinear contexts.

We can use matrix multiplication or dot product to write the equation of a plane in .  For instance, the equation can be rewritten as .  This is a logical condition – a constraint – that picks out a certain set of points in .  The plane given by is an affine subspace of , not a vector space in .  Why?  First, check to see if the space is closed under scalar multiplication and vector addition.  (Remember zero is a scalar!)  What do you find?  Second, ponder this question:

Example 2.2 What is the difference between a point in this plane and a vector in this plane?  Find an example of a point in the plane and a vector lying in the plane.

You probably found your own examples, but and are two points in the plane – they satisfy the equation .  By contrast, is a vector in the plane.  A few different ways to say that: it’s a vector that lies parallel to the plane when based at the origin, so we can translate it to lie in the plane; it is a vector that goes between two points in the plane; it’s a vector perpendicular to the normal vector , which is enough to characterize the plane since we’re in .

A plane can be parametrized using two variables, for instance and , because it’s a two-dimensional object.  For example, let , , and .  Notice this satisfies our previous Cartesian equation, !  We can write the parametric vector-valued equation like this:

If we wish to drop the and write a function , we can write the below expressions as well as the previous:

or

Notice here that I’ve now written the expression as a linear shift of the linear combination of two vectors that lie in the plane, or as the linear shift of a matrix product.  While simply a cosmetic rewrite, this points out a larger lesson.  Explore this through examples:

Look at

How can this product be expanded?  Write the product in two ways, as a single row vector and as the linear combination of three row vectors in .

Do this again for another product:

Again, rewrite the product as a single column vector in and as the linear combination of two column vectors in .

What do you notice about the linear combinations and vector spaces that result?

This leads to the useful concepts of row space and column space, explored a little later.  ***

8.3 Useful inequalities for vectors

It’s important to mention a few special inequalities that are useful for linear algebra and for finance.  First, the triangle inequality.  Back in Chapter 2, we encountered a version of the triangle inequality: two sides of a triangle have to sum to a number larger than or equal to the third side of the triangle.  Now let’s use vectors to express the same idea.  Look at a triangle with sides , , and , with all vectors in .  Then the triangle inequality is

Why is this discussed right after all the material on dot products for real vectors?  Because you can rewrite those magnitudes as dot products and use the Cauchy-Schwarz inequality to prove the triangle inequality in an elegant way.

The Cauchy-Schwarz inequality is really fundamental to any inner product space – the dot product is an example of a more general inner product.  Using the notation we’ve been using in this chapter, the Cauchy-Schwarz inequality says

You can prove this using the geometric characterization of the dot product as , where is the angle between and .

Now I suggest you try using the Cauchy-Schwarz inequality to prove the triangle inequality.  Use the fact that you can rewrite as .  Can you finish the proof?

8.4 Some vectors in finance

Linear algebra is used all over finance, and here I’ll introduce four vectors that are useful in our further applications of linear algebra.  First, we can represent a portfolio of stocks (or other assets) with the vector .  Interpreted, this means we have shares of stock , for stocks .  These numbers can be real numbers: a negative entry would indicate holding a short position on the stock, and we can have non-integer entries via buying fractional shares or through investing in a mutual fund or exchange-traded fund.

Second, we could construct vectors that represent what happens to the price of these assets under a particular economic scenario.  Say we’re looking at a possible change in regulations, or in energy prices, or just have a forecast of what all the asset prices will be in one week.  Then we could represent what happens under this hypothetical future scenario using a vector

where represents the change in price of a share of asset under the scenario.  (We could equally well write a vector in which represents the price of a share of asset under the scenario, rather than the change in price.)

Notice that both our portfolio vector and our single-scenario change-in-price vector (or single-scenario price vector ) both have entries.  This is because assets are under consideration.  Also notice that gives the expected change in the value in the portfolio given the occurrence of the scenario under consideration, while gives the overall value of the portfolio under the given scenario.

A third type of vector we could invent would look at the potential prices of a stock or asset under various scenarios.  For instance, what would happen to a pharmaceutical company’s stock price if (I’ll date myself) the Affordable Care Act is repealed?  if President Trump decreases the Food and Drug Administration’s regulatory responsibilities and allows “fast-tracking” of new drugs”?  if the rules for H1B visa applicants are changed substantially?  if changing trade relations with India and China change the profile of the export market?  Any of these changes could affect the stock price of a pharmaceutical company, and you might want to look at these scenarios using linear algebra as a first pass analysis.  Say for stock we consider scenarios.  Then a vector could represent the change in price of the asset under each of the scenarios, or a vector could represent the net price of the asset under each of the scenarios.

Fourth, an essential part of risk management and forecasting is working with the probabilities of these future scenarios.  You may use “expert judgement” to come up with the probabilities of these future scenarios, or you may use the no-arbitrage principle to come up with probabilities based on the price changes forecast by your “expert judgement.”  Either way, with scenarios under consideration you’d want a vector

with each element giving the probability of the th scenario.  Note that if you’re considering this situation where these scenarios cover all future events, you must have the probabilities adding up to one by the axioms of probability:

This is the first time in our consideration of vectors in finance that we’ve had a condition like this, and it’s a little special – finally probability is starting to intersect with our multivariate work!

Again let’s look at a few matrix or dot products that are relevant.  The product

would give the expected value of the change in price of the asset under the scenarios under consideration.  If you’re confident in your scenario analysis and the given probabilities and was positive, you’d have an expected profit on your hands!  Buy that asset!  It’s not necessarily a guaranteed profit – under some scenarios you might still lose money – but it may be a “good bet” in investment terms.  If you are looking at no-arbitrage pricing and probabilities, you’d expect to have

with no expected profit.  (The product would just give the expected value, rather than change in value, of asset over all the scenarios under consideration.)

Questions:

How would you represent a portfolio with 6 shares of stock , 5 of stock , and a short on shares of stock ?

Can you find a probability vector that will give you an expected profit of zero given a scenario vector for stock of ?  What about for ?  What problems do you encounter for the second one, and why?[3]

8.5 Linear transformations

Consider a general matrix , with rows and columns.  Figure out some way to remember that rows come first, columns second – easy for some to remember, but I’m a person who used to have trouble keeping left and right straight.  My solution:

Royal Crown Cola reminds me that rows come first, columns second!

Quiz yourself:

- If we multiply the matrix on the left by a row vector with        [4] elements, then we get a row vector with elements.

- If we multiply on the right by a column vector with elements, then we get a column vector with        [5] elements.

In this way, we can think of multiplication by as a function that transforms row vectors in to row vectors in , or as a function that transforms column vectors in to column vectors in .

Example 5.1 Is multiplication by a matrix , on either the left or the right, a linear operation?  (If necessary, remind yourself what linear means!)

We call these functions linear transformations, because they are nice general ways of transforming yourself from one linear space to another linear space.

Go back to our example of a row vector multiplied by a matrix

We say that represents a linear transformation

and we write the action as

With this notation it’s easy to see that multiplying by a matrix is a linear operation: for any by properties of matrix multiplication, so .

Example 5.2 What is the domain of the function with

What is the range, or image, of this function?  The domain is and range is...  Well, in this case it’s all of but that takes some work to prove.  We need to develop some more machinery.

The concept of range is a bit sophisticated for linear transformations.  We call the possible outputs of the image of , and we notice it consists of all linear combinations of the rows of .  The notation for this is or .  Here, is the transpose and we write this because historically mathematicians are prejudiced in favor of multiplying with the matrix on the left and the vector on the right, and converting to this format means taking .  Reconcile this with the notation for column space below.  The term for “all possible linear combinations” is span.  In sentences, we can say, The image of is the linear span of the rows of , or alternatively, The image of is the row span of the matrix .  If the rows of are written as , then we can write for this row span.

Example 5.3 Repeat this analysis instead multiplying by a column vector on the right: the linear transformation

written as

Here you get the column span of the matrix for the image of .  If the columns of are , we can write for the span of the columns, or we can write for the image of the linear transformation .

How do we know the dimension of the row span or the column span?  We need to look at linear independence of rows and columns.  This will give us the idea of the of a matrix.  We’ll define rank in section 8.6, but first, we’ll set up some financial concepts.

When we talked about the span of a set of vectors above, we were making a vector space by simply defining, for ,

For instance, the linear span of the is .

Example 5.4 Is the row span of a matrix a vector space?  Is the column span of a matrix a vector space?

Let be a vector subspace of .  The set of vectors orthogonal to every vector in is also a vector subspace.  We denote this space by and call it the orthogonal complement of :

You should prove to yourself that is closed under scalar multiplication and vector addition.

Example 5.5 How does this show that the set of all solutions to the equation is a vector subspace of ?  This vector subspace is called the right null space of the matrix .

Example 5.6 How does this show that the set of all solutions to the equation is a vector subspace of ?  This vector subspace is called the left null space of the matrix .

We define the following vocabulary:

- A function is one-to-one if whenever , we have .  Example: , where , is one-to-one.  If , you know and there are no other choices.  Non-example: is not one-to-one, because if I tell you , you can reply, “Clearly must equal either 1 or !  Which would you like?”

- A function is onto if for all in there’s an in so that .  The function maps on to its image, covering all the points in .  Example: again (why?).  Non-example: again (why?).  More subtle non-example: .  Here .  Points like are in the image of this function, but is ?  No!  So not all points of are covered by this function; is not onto.  Be careful of domain and range here.

- For a function , take a point in that is in the image of .  The of this point in is the set of points in that map to under .  For instance, for , the preimage of is , the set of all points whose square is .  For a multivariable example, consider .  Here .  Take any point in and try to “go backward”: the preimage of is the empty set, the preimage of is , and the preimage of is the circle in .

8.6 Bases

If is the linear span of a set of vectors , we call these a spanning set for .  A minimal spanning set for is such a set with as few elements as possible: if you remove any vector from a minimal spanning set, the remaining vectors will no longer span .  We call a minimal spanning set a basis for .

Theorem 6.1 Let be a basis of .  Then the vectors are linearly independent: that is, no can be written as a linear combination of the remaining vectors.  Equivalently, the only way to write the zero vector as a linear combination of the ,

is to take all coefficients .  In addition, every basis for has exactly vectors.

Example 6.1 Use contradiction to prove the first part: if we could write (for instance) as a linear combination of the other vectors, then (what?)

Example 6.2 Challenge: prove that every basis for must have the same number of vectors.

The dimension of a vector space is the number of vectors in a basis for the vector space.

We can easily work with the linear span of a set of vectors by writing the vectors as the rows of an matrix.  Go through the following questions and give your best answers:

Example 6.3

- Does swapping two rows of a matrix change the row space of the resulting matrix?

- Does replacing row of a matrix with row minus row change the row space of the matrix?

- Does multiplying a row in a matrix by a real number change the row space of the resulting matrix?

The answer to all of the above is no, because each of these is a linear combination of row vectors, and the row space (the span of the row vectors) is closed under linear combinations (scalar multiplication and vector addition).  This means that we can use row reduction techniques to solve matrix equations of the form on paper.  The goal is to streamline old techniques for solving systems of equations by row reducing the augmented matrix to have in row echelon form.

Row echelon form is a special form of a matrix: at the bottom of the matrix, rows with only zeroes; all other rows have first nonzero entry 1, which we call a “leading 1”; all entries above and below leading 1s are zero.  Not only does this allow us to solve equations of the form , but it allows us to easily see the rank of a matrix , and this form gives an easy way of determining the dimension of the right null space (just the number of rows of zeroes).

Example 6.4 Find the space of solutions to

Hint: Using the symbol for transpose (swapping rows and columns), we can change to .  This makes the equation easier to deal with using methods discussed in class.

(The solution space is one-dimensional – we’ve got three variables and two conditions on them, so one degree of freedom in solutions.  Check that your work gives you something like , , free.  We can write this answer parametrically as

Example 6.5 Show that

has no solution.

(Find a contradiction.)

Example 6.6 Consider the following three investment opportunities, showing the net profit under four possible outcomes:

Show that these do not offer the possibility of arbitrage, and determine the price of the following investment opportunity: .  Hint: Use the theorem of no arbitrage!  You can show there’s no possibility of arbitrage (“free money”) directly, by reducing the matrix and showing there’s no solution, or you can find a probability vector that satisfies case (2) of the No Arbitrage theorem.  The existence of this probability vector shows that there can be no arbitrage!

The fundamental theorem of linear algebra relates the four “fundamental linear subspaces” of an matrix .  These four subspaces are the right and left null spaces, the column space, and the row space.  I’ll remind you that L: Rn Rm

x Ax and R: Rm Rn

y yA = AT yT .

Theorem 6.2 The dimension of the row space of plus the dimension of right null space of equals .

The dimension of the column space of plus the dimension of left null space of equals .

Theorem 6.3 The dimension of row space of is the same as the dimension of column space of , and these are both equal to the rank of matrix .

Example 6.7 Challenge: think about how you could get a set of orthogonal basis vectors for a vector space from any basis provided.  Pick a vector to start with, then use the projection formula discussed a few classes ago to find the next one.  How do you get the third basis vector?  Remember it must be orthogonal to both of the previous vectors!  Explain how to proceed in the case of vectors.

Example 6.8 Challenge two: can you modify this process to produce an orthonormal basis?  That means all basis vectors are orthogonal to each other, and moreover each is a unit vector.

This will be discussed more in the next chapter.

8.7 Applications to financial math

The principle of no arbitrage in finance has many different phrasings.  One that I like is that if two assets have the same risk and the same cash flow, they’ve got to sell at the same price.  Otherwise you could make money from exploiting the difference in price between the two.  Another way people express the no arbitrage principle is to say that you can’t make more money than the market without taking on more risk, and yet another is that there is “no free lunch.”

Now, is the no arbitrage principle true?  Well, not exactly.  The “efficient market hypothesis” says that if there’s an arbitrage opportunity (the chance to make money without risk) then the market will notice, people will take advantage of it, and prices will then adjust to eliminate that opportunity.  This means that in an efficient market, prices reflect information accurately.  (Look up strong, semi-strong, and weak efficiency elsewhere!)  The capital asset pricing model (CAPM) and Black-Scholes options pricing model are both built on the principle of no arbitrage, and that’s why we need to understand it.  Moreover, CAPM and Black-Scholes are really useful out in the real world.  But like all models, they’re wrong, as is a really strict no-arbitrage statement.  Robert Shiller, for instance, looked at changes in dividend prices and their effect on share prices of assets.He found that share prices “overreact.”  Look up the work of Fama, Schiller, and Hansen, who jointly won the Nobel Prize in Economics in 2013 for their (separate) work on asset pricing.  You’ll find that the truth about asset prices is a lot more complicated than no-arbitrage – but you can’t understand what is really going on without knowing this basic principle.

We will formulate a “No Arbitrage” theorem via linear algebra.  First, we assume that we can form an matrix of net profits, describing what will happen to stocks under outcomes or scenarios.  (Entry is the net profit for stock under scenario .)

Example 7.1 What are the possible net profit vectors for all the portfolios I could create?  Test yourself by translating this into the language of linear algebra.  What vector space am I asking for?

Consider the situation given by

We have three stocks and four scenarios under examination.

Example 7.2 Can we invest in a way that produces the vector ?  How do you figure out the answer to this?  How do you interpret the answer once you have it?

Example 7.3 Can we invest in a way that produces the vector ?

Example 7.4 Let’s add another investment opportunity, the “savings account.”  This opportunity is an idealized situation in which you put in a dollar and get a dollar.  This of course ignores interest rates for the moment.  How do you add this information to the matrix?

Example 7.5 Given the new matrix, can you invest in a way that produces the vector ?

Example 7.6 What is the cost of such a portfolio?  Interpret this carefully, looking at net profit vectors and the savings vector and considering the cost of each.

We can find the cost of such a portfolio by solving for a probability vector that satisfies

and then taking the dot product of our portfolio vector with this probability vector .

By the axioms of probability, a probability vector has only non-negative entries and its entries sum to one.  In particular, if the vector must have non-positive elements to satisfy the matrix equation given above, we can create a portfolio with no negative elements and at least one positive element which costs no money – and that’s arbitrage.

Let’s go back to vector spaces so that we can gather some techniques for more efficiently solving the matrix equations above.  With new language, we can also restate the “no arbitrage” theorem in terms of linear algebra and provide some strategies for proof.

Using the new language of linear algebra, we can restate the no-arbitrage theorem as the following: EITHER

- the row space of contains a non-negative vector with at least one positive element, OR

- the orthogonal complement of the row space of (the right null space) contains a vector whose elements are all strictly positive.

I use incorrect capitalization here to emphasize that these outcomes are mutually exclusive.  To test your understanding, ask yourself: which of these cases is the case with arbitrage?  How do you know?  In the no-arbitrage scenario, what’s true about the probabilities of the scenarios?

Initially we considered the situation with scenarios and stocks, so

- each portfolio has entries,

- the matrix of prices is an matrix,

- and the probability vector has entries.

Figuring out which case of the no-arbitrage theorem holds involves either finding with some strictly positive entry (making some money!)  and the corresponding portfolio that guarantees us this risk-free money, or finding the probability vector that gives .

To consider cost or the “savings account” approach, add a row of ones to the bottom of .  Call the new matrix .  Add an entry to the end of the portfolio vector , representing how much money you’re putting in the savings account, and call the resulting vector .  If the “savings account” interpretation is distateful to you (and it does have some drawbacks), consider this a mathematical way of requiring that be a probability vector: that is, .  The reason you might not like the “savings account” interpretation is that it does not quite line up with the idea that the matrix entries of are net change in stock price, or profits.  The row of ones does not represent absolute profit (making $3 per share of stock ) but instead leaves the amount unchanged in each scenario.

Example 7.7 Prove to yourself that if we have a probability vector so that , then .

Example 7.8 Show that if we have a probability vector with only positive entries, then we must have negative entries in the portfolio vector .  (This is one part of the proof of the no arbitrage theorem.)

Example 7.9 Challenge: prove that if we have a vector whose entries are all non-negative, and at least one of whose entries is positive, so that is non-negative and has at least one strictly positive entry, then there can be no strictly positive probability vector so that .

8.8 Invertible transformations

This section mainly emphasizes ideas of rank and determinant, and reinforces what we’ve learned about transformations.  All matrices in this section are square matrices.

Theorem 8.1 If the rank of an matrix is , then the linear transformation defined by is both one-to-one and onto.  If the rank of is less than , then the linear transformation is neither one-to-one or onto.  Thus the linear transformation is invertible if and only if the matrix has rank .

A square matrix is of full rank if and only if its determinant nonzero; a determinant of zero means the matrix is degenerate (the matrix gives a transformation that is not onto – the image is contained in a smaller linear subspace of the target space).

In much of linear algebra before this class, you’ve probably concentrated on full rank matrices, which give invertible transformations.  That is because they give systems of equations easy to solve with matrix methods: if is full rank and , then .  This is extraordinarily important, and also not that interesting!

One concept that I personally find interesting, though, and that will be useful in several more contexts is the idea that the determinant of a matrix gives the change in volume that the transformation induced by the transformation .  Maybe some illustrations will help: in two dimensions,

and in three dimensions

This holds for transformations , in fact.  Think of it this way: take the unit hypercube and look at its image under the transformation .  The image of the unit hypercube under will have -dimensional volume .  This also says something about transformations that are not invertible: if a square matrix is not invertible, then the determinant is zero – the transformation is not onto and one-to-one.  That means that the image of the unit hypercube has zero -volume, which means the unit hypercube in was squashed into a lower-dimensional subspace by the transformation .  Remember this when we start talking about principal component analysis, dimension reduction techniques, and singular value decomposition!!

1. A plane that goes through the origin! ↑

2. (Dot product is , second product makes no sense for dimension reasons, third is , and fourth is , the same as the dot product.) ↑

3. Remember that probabilities must all be positive and less than or equal to 1, and for a probability vector elements must all add to one. ↑

4. ↑

5. ↑

Chapter 9

Spectral theorem and portfolio management

9.1 Orthogonal matrices and orthonormal bases

An matrix is called orthogonal if its rows are orthonormal: that is, all rows are perpendicular to each other and all have length one as vectors in .  For instance,

is an orthonormal matrix (check!).  More generally, if is an matrix with rows , then is orthonormal if for all between 1 and and for all .

Geometrically, what does an orthogonal matrix do?

Think about the consequences of the comments above: if is orthonormal, then

where is the identity matrix.

Example 1.1 Prove this!

This means that : is the inverse of .

Example 1.2 What does this mean for ?

Example 1.3 Prove the columns of are orthonormal.

Example 1.4 Compare the length of a vector with the length of the vector : carry out the calculations necessary to do so.

Example 1.5 Compare the angle between and to the angle between and by using the formula that relates dot products and angles between vectors.

If you did all the questions above, you proved the following theorem:

Theorem 1.1 Let be a linear transformation from to represented by an orthogonal matrix .  Then preserves the lengths of vectors and the angles between vectors.  Equivalently, for all column vectors , .

9.2 Gram-Schmidt orthogonalization

To find an orthonormal basis from any collection of basis vectors, use Gram-Schmidt orthogonalization.  The idea is elegant and simple: pick a first vector and make it length one.  Pick the next vector and take the component of it orthogonal to the first, then normalize.  Pick a third vector and take the component of it orthogonal to the first two, and so on.

First, of course, we need to define what the projection of a vector onto another vector is.  Imagine that a projection is the shadow of on the line in direction if the sun is “directly overhead.”

The algebraic formulation is

Notice that this vector has a direction (it goes in the direction of ) and a magnitude (the magnitude of the projection is , where is the angle between the vectors and .  You could figure out this definition of projection yourself by looking at the natural geometric expression and using to prove the formula in terms of the dot product.

Why do we need projection for Gram-Schmidt orthogonalization?  It turns out that will give the component of that is perpendicular, or orthogonal, to .  By subtracting off the part of that is in the direction of , we’re left only with the part that is perpendicular to .  This is what we want for “orthogonalization.”

Formalize our descriptions with mathematical language: To find a Gram-Schmidt orthogonal basis or orthonormal basis for the space spanned by a set of vectors ,

- Pick one vector to start with: let’s choose .  Normalize it (make it length one) via

if you want an orthonormal basis.  Then the first vector in your orthogonal basis is , and the first vector in the orthonormal basis is .

- Pick the next one to deal with: I choose .  Subtract off the component in the direction of :

Normalize the result if you want an orthonormal basis (that makes ).

- Take and subtract off the components in the directions of and :

Normalize this if you want an orthonormal basis ( )

- Repeat until done!

Order sort of matters: no matter what order you go in, you’ll get an orthogonal or orthonormal basis, but if you use a different order than a friend, you’ll very often get a different basis.  Check out the picture below:

**add picture****

Example 2.1 Why does this work?  Why doesn’t the process lead to vectors outside the vector space spanned by the original set of vectors? Use the words “linear combination” in your solution.

9.3 Rotation and scaling

A particular class of invertible linear transformations is given by rotation.  Yes, rotation: just pick up everything in your vector space and rotate it by radians in some particular plane.  The rotation matrix that gives this transformation is

Check for yourself that this is an orthonormal matrix.  What is its determinant?

More things to check for yourself:

- What is ?  What is ?  Yes, they happen to be the same!  Why?

- What is ?  What is ?  What is ?  Yes, they happen to be the same!  Why?  You’ll need your angle-sum formulas from trigonometry to see this.

- What do you think should be?  Can you come up with a definition?

- What is for a positive integer?  Note that this denotes carrying out matrix multiplication times ( times itself times), not raising the elements in the matrix to the th power.  With this nice rotation matrix , what’s a nice way to streamline this calculation?  If you deduced , you’d be right.

Rotation matrices certainly extend to rotations in , but are more complicated to write out.  No matter what, if we’re rotating only by an angle , a single parameter, there will be some axis that is invariant under the rotation in .  In , you rotate by around a point (the origin).  In , you rotate by around some line.  For instance,

rotates by around the -axiz – the plane is rotated but points stay untouched.

You may have noticed that whether in or it is dramatically easier to calculate for any rational number than it would be for a general matrix.  This is not a coincidence.  The beautiful geometry of a rotation matrix allows a very simple interpretation of these operations.  We’ll be able to put this to use in Taylor series of matrices, among other applications.  But you know that not every matrix is a rotation – we need some wider techniques.

The next baby step is to consider scaling as well, and look at the matrix

Notice we’ve given this matrix a name, .  Notice, too, that now we’ve introduced another bit of ambiguity: if you look carefully, .  This is the same equivalence that occurs in polar coordinates.

Consider what happens when two such matrices and are multiplied: geometric reasoning tells you that the angles add and the scalars and multiply.  When you multiply out the matrices, though, what happens?

Example 3.1 Multiply out and .  How does this agree with your geometric reasoning?

Raising matrices to a power is also easy using our new notation:

This formula essentially uses polar coordinates, if you remember those!

How does this work for fractional powers?  Almost the same way!  Notice that when you take a square root, you still have the choice of positive or negative that you would in the real numbers.

Example 3.2 Take the square root of Make sure you get two different answers.  Express those answers in form.

This may be reminding you of complex numbers.....

9.4 Complex Numbers

At some point in your math education you encountered the frustration of wanting to understand the solution to .  Hopefully at about the same time you encountered the idea of complex and imaginary numbers.  We define the imaginary number to be ; notice that is a fine imaginary number as well and as well.  (Remember that is only a function if it has a unique output for each input , and we define the unique output to be the “positive” number that satisfies the equation .)

You may have also encountered the idea of the complex plane at about this time.  The complex plane is a representation of that is topologically equivalent to .  The way to see this is that every number can be represented as a pair by viewing the -axis as the “real” axis and the -axis as the “imaginary” axis.

Complex numbers add together just like vectors in , as , and multiplication by real scalars also behaves the same.  The difference between and lies in their multiplicative structures.  In , we might pick out dot product as the way to multiply and , but in we’ve got a rule we want satisfied – that – and dot product clearly doesn’t do the right thing!!

To illuminate the multiplicative structure of the complex numbers, we’ll pick out two equivalent ways of thinking about complex numbers.  First, the complex number behaves like the matrix

What I mean by “behaves like” is that if you carry out matrix multiplication here, you’ll get a structure that is equivalent to complex multiplication:

corresponds with

Example 4.1 How can you write this matrix as ?  Challenge: think about the ways in which you can extract from the information you have, and how it may be constrained by the way in which you write it.

Example 4.2 Think of three ways in which you can find the reciprocal of the complex number represented by a matrix Remember, the reciprocal of the number is It is the multiplicative inverse.

The questions above ask you to think about the second way to represent a complex number: instead of using Cartesian coordinates via , you can use polar coordinates via , or with Euler’s formula,

Euler’s formula gives another way to link the geometry of with the matrix representation and the Cartesian representation of the same complex number.

9.4.1 Taylor series

Why are we bothering with series, especially right now, right after complex numbers?  The use of power series representations for functions makes it very easy to extend these functions to the complex numbers.  If I ask you what is, you will probably be unable to answer the question using your unit-circle understanding of trigonometry.  You can just plug into the power series definition, though!  Convergence is still a big deal, but we can prove that convergence in real numbers is the “same” as convergence in the complex numbers.  That is, if converges for , then converges for and .

Example 4.3 Use your power series representation for to write the series for .  Separate the terms with odd degree from the terms with even degree to prove that

9.5 Changes of basis and coordinates

9.5.1 Changing basis alone

We are used to working in the standard basis for , with vectors , through .  However, there might be a situation in which we’d like to work in a different basis – maybe an eigenbasis.

9.5.2 Changing linear transformations into a new basis

Example 5.1 In , call our standard basis and denote another basis by .  Let consist of and Say we’ve got a linear transformation that we’ve written as in terms of the standard basis .  How can we write in terms of this new basis?  That is, what’s the matrix for in terms of the basis ?

Introduce a transition matrix that relates the two bases.  has entries defined by

In our example, since , , and , we get the matrix

Notice the columns here!!

A few calculations show, then, that if our transformation in the standard basis is written and the same transformation in the new basis is written , then

or in particular

This operation is called conjugation.

Example 5.2 To continue our example, say that

and is as above.  Then the matrix is

Check this!

This means that takes the vector to the vector .  Notice that this is in the standard basis!

Example 5.3 Say we want to write a linear transformation in terms of the new basis given by

an orthogonal basis for .  Let be given by

What is ?  What is the matrix for in the new basis?

If two matrices and are conjugates of each other, that is, there’s an so that or so that , then we say that and are similar matrices.

Ideally, we can find a diagonal matrix so that .  This is very nice because

- Diagonal matrices are easy to multiply: they simply scale rows or columns, depending on which side of the produce they’re on.

- In particular, is very easy to calculate – you just raise each diagonal entry to the th power.

- Calculations like are suddenly easy too.  Since , , times.  Notice most of the s cancel with .

- The concept of diagonalizability (being able to find a diagonal similar matrix) is going to be very useful when we look at eigenvalues and eigenvectors!

Example 5.4 Draw a picture in of the standard basis and of the basis given by and .  Draw a picture of the vector , and express it in terms of and .

Example 5.5 If has coordinates of with respect to the basis given in the previous problem, what are the coordinate of in the standard basis?

Example 5.6 Using again the basis given by and , what matrix will change a vector in standard coordinates to a vector in coordinates with respect to ?  Use that matrix to find the coordinates of .

9.6 Eigenvalues and eigenvectors

Let be a linear transformation from to .  A nonzero vector is an eigenvector of if

for some scalar .  The number is the eigenvalue of associated to the eigenvector .  For an matrix, there are up to distinct eigenvectors and eigenvalues.  As you’ll see, we’ll care a lot how many distinct eigenvalues a matrix has.

The geometric meaning of the eigenvectors and eigenvalues of a real matrix is that an eigenvector gives a direction in which a matrix transformation gives a “pure stretch,” and the associated eigenvalues gives the magnitude of this stretch.  At least, this is true if the eigenvalue is a real number and the eigenvector is a real vector.  Direction and magnitude – sounds like a vector – why do we need the pair?  The eigenvector truly is just a basis vector for a particular vector subspace, while the eigenvalue can be either positive or negative, indicating pure stretch when positive and a stretch and reflection when negative.

Example 6.1 Find the eigenvalues and eigenvectors of the matrix

solely by thinking (no calculating allowed!!).  Link this understanding visually or geometrically with scaling in the and directions.

Example 6.2 Find the eigenvalues and eigenvectors of the matrix

To do this, use the characteristic polynomial to find out for what values of the matrix is singular.  The solutions to

are the eigenvalues of .  In this situation, you’ll have two or fewer eigenvalues: check that you get and as solutions.  Write and to keep track of them, and then find the associated eigenvectors using the definition.  We know that we must have , so our first equation is

implying .  A solution to this is ; does this also satisfy the second equation,

Check: is true!  Thus

is a fine eigenvector for .  In fact, any scalar multiple of this would work; check that

satisfies the definition of eigenvector for any .  I’ll let you calculate .  Check your answer in the footnote.[1]

Geometric and algebraic multiplicity of eigenvalues are useful and subtly different ideas that come into play when you have fewer than eigenvalues for an matrix.  Eigenvectors of a matrix are a basis of the null space of – that’s why finding the roots of the characteristic polynomial works to find eigenvalues.  We know there is a relationship between the dimension of the (right) nullspace and and the rank of the matrix; let’s explore this relationship and geometric and algebraic multiplicities by example.

Example 6.3 Consider the matrix

Check to see that the two eigenvalues are and , and that each of and have a nullspace of dimension 1.

Example 6.4 Consider the matrix

The characteristic polynomial is .  The only root, , occurs twice – the algebraic multiplicity is two because the exponent of the term is two.  The geometric multiplicity is the dimension of the nullspace of .  Since

is rank one, its nullspace has dimension one by the rank-nullity theorem.  That means the geometric multiplicity of the eigenvalue is one.

These multiplicities play into whether we can find a basis of eigenvectors.  If an matrix has eigenvectors, we have a basis of eigenvectors and we can diagonalize the matrix.  Else we can’t!  Let’s think through this: Consider a specific transformation and there’s a basis consisting of distinct eigenvectors .  Let’s find the matrix of with respect to basis .  Thinking through this carefully, we see that the matrix of in the matrix must be the matrix

Call this diagonal matrix .  Notice that if we use our knowledge from earlier, this matrix is easy to find:

for the matrix of in the standard basis and the transition matrix defined above.

Example 6.5 Silly example: consider the matrix

The only eigenvalue of this matrix is two, with both algebraic and geometric multiplicity two: that means that even though , we can pick linearly independent eigenvectors

Can you diagonalize the matrix ?  Not a trick question!

Example 6.6 Use the characteristic polynomial to find the eigenvalues of

Find the corresponding eigenvectors of .  Then find the matrix that satisfies

Example 6.7 Use the characteristic polynomial to find the eigenvalues of

Find the corresponding eigenvectors of .  Then find the matrix that satisfies .

How are these two examples different?

Example 6.8 Last, look at the matrix

Check that you can find three linearly independent eigenvectors even though one of the eigenvalues has algebraic multiplicity two.

Remarkably, if the eigenvalues of a matrix are all distinct, then the corresponding eigenvectors are automatically linearly independent, thus providing a basis for

So far we’ve ignored one sticky case: the case of complex eigenvalues.  Look at the matrix

Check that you get roots of that are complex: .  No coincidence that they occur in a conjugate pair!  The fundamental theorem of algebra guarantees that any degree polynomial with real coefficients will have exactly solutions when you count with multiplicity (explored above) and count complex solutions, and that complex solutions will always come in conjugate pairs because has real coefficients, while any other product will have complex coefficients.  But how can this make sense if we are considering “real-world” applications?

First, go through the work of finding the eigenvectors for the matrix we just considered.  You’ll find eigenvectors

for some (so if you got a different-looking answer, just check that it’s a multiple by some ) and

again for .  Notice that these eigenvectors also come in a conjugate pair.  That’s also not coincidence.  Second, let’s try to interpet this as a rotation and scaling using properties of complex numbers.  *******

9.7 Quadratic forms and definiteness

Having eigenvalues and eigenvectors also allows us to discuss quadratic forms and positive definite, negative definite, and indefinite matrices much more easily.

A quadratic form is a homogeneous degree-two polynomial with real coefficients.  Homogeneous means that every term of the polynomial has the same degree; for example, is a homogeneous polynomial of degree two and thus a quadratic form.  Using summation symbols, we could write a quadratic form in variables as

More excitingly, though, we can write any quadratic form using vectors and matrices: let , and convince yourself that you can write

for a unique real symmetric matrix .  What must the entries of be?  Compare terms between our two expressions: we must have and we must have .

We call a quadratic form and its associated matrix positive definite if

for all non-zero vectors in .  If instead we have the weaker condition

then both the form and the matrix are positive semidefinite.  Likewise, if

for all non-zero , we call both and negative definite, and if

they’re negative semidefinite.  If none of the above, call and indefinite.

9.8 Power series of matrices

Using the diagonalization of a matrix we can more easily work with power series.  For instance, if , then for ,

We can make this into a matrix of power series, and then if the power series converges to for each , we get

This is easy, and works for real and complex eigenvalues!

We don’t always get convergence, though – if some of the eigenvalues are too big for convergence this won’t work.  For instance, the power series expansion of only converges if the eigenvalues of are of absolute value less than one.

Example 8.1 Calculate using power series where

9.9 Applications to financial math

Often you might be interested in how two quantities vary together – their covariance.  If you have a number of quantities whose covariances are of interest, you can assemble a covariance matrix.  Each entry is the covariance between quantity and quantity .  Since , the covariance matrix is symmetric.

Principal component analysis is mathematically simply the act of finding the eigenvectors and eigenvalues of the covariance matrix.  I think it’s fair to say that the first step in principal component analysis is simply linear regression – finding the best-fit line through the data.

Since the covariance matrix is symmetric, the Spectral Theorem will tell us (soon!)  that when all the eigenvalues are distinct, the eigenvectors are actually all orthogonal.  Since we can choose eigenvectors of whatever length we like (it’s direction that is not up to us), that means it’s easy to find an orthonormal basis of eigenvectors for a real symmetric matrix.

- Eigenvectors come up in analyzing components of Brownian motion in many dimensions.

- Power series and their truncations come up a lot when looking at normal distributions of prices, for instance.

9.10 Complex vectors

Complex vectors are very similar to real vectors, but there are a few key differences. First, the similarities:

- Multiplication by a scalar works the same.

- Addition of vectors works the same, and is commutative.

- The formula for length is analogous (but not quite the same!).

Here is the first difference: We want a formula for length or magnitude of a complex vector so that

in analogy with real vectors.  But what is for a complex number?

We want to be the “absolute value” or length of , and looking at that magnitude would be .  An easy way to get this is by using the complex conjugate to get the length:

This means that the right way to define length of a complex vector is using the inner product

using

This also allows us to define the inner product of two different vectors:

BIG CHANGE here: we are now talking about the inner product of two vectors, rather than the dot product, and this inner product is not commutative.  Check it yourself!

Example 10.1 Given and , compare and .  What is their relationship?

It turns out that

9.11 Complex matrices

In the previous section, we determined that to take the inner product of two complex vectors we need to use complex conjugates: basically,

in matrix notation.  Here, we want to maintain some of the structure we had in real matrix land — to check if a real matrix is orthonormal, for instance, we just see if is equal to the identity matrix.  This relies on dot products, so needs to be modified for the complex world.

Define the Hermitian transpose of a matrix by

That is, we take the complex conjugate of every entry and take the transpose of the whole matrix.  You can do this to any matrix: for example, the Hermitian transpose of

is

Also notice that

if the product of complex matrices made sense in the first place.  You can prove this by using properties of the transpose.

We call a “complex orthogonal” matrix if it is square () and has rows which satisfy

for and

Another name for this is “unitary.”  Such a matrix satisfies , so we have

(Notice that you can prove as well, as .)

9.12 The spectral theorem

An complex matrix is called Hermitian if .  Any real symmetric matrix is Hermitian; all the diagonal entries of a Hermitian matrix must be real, even if the rest of the entries are complex.  (Check this yourself!  Notice that the diagonal entries of the matrix must all be equal to their conjugates.)

Hermitian matrices are very special.  Following is a non-exhaustive list of their special properties.

Proposition 12.1 Let be an Hermitian matrix.  Then for column vectors

Proposition 12.2 Every eigenvalue of a Hermitian matrix is a real number.

Proposition 12.3 Every eigenvalue of a real symmetric matrix is a real number, and the eigenvectors of such a matrix can always be chosen to be real eigenvectors.

Proposition 12.4 If and are eigenvectors of a Hermitian matrix, and the corresponding eigenvalues are distinct, then and are orthogonal.

Proposition 12.5 If is an Hermitian matrix with distinct eigenvalues then all of these eigenvalues are real and we can choose the corresponding eigenvectors so that they are an orthonormal basis for .  If is the matrix whose columns are those eigenvectors, then

If is real and symmetric, then the eigenvectors can be taken to be real, and they form an orthonormal basis for .

Theorem 12.1 If is an Hermitian matrix with eigenvalues listed according to multiplicity, then all of these eigenvalues are real.  If is one of these eigenvalues and has multiplicity , then there are corresponding eigenvectors that are an orthonormal to each other.  There is an orthonormal basis for consisting of eigenvectors of .  If is the matrix whose columns are those eigenvectors, then

If is real and symmetric, then the eigenvectors can be taken to be real, and they form an orthonormal basis for .

9.13 Singular value decomposition

We have talked a lot about eigenanalysis of square matrices.  For any square matrix , you can now find the eigenvalues and eigenvectors and have some intuitive notion about how these relate to the geometry of the linear transformation given by the matrix .  In particular, you know that you can think of the linear transformation as “scaling” by the eigenvector in the direction of the corresponding eigenvector .

This point of view works well in many situations.  (You will use it to analyze systems of linear first-order differential equations and to classify critical points of surfaces in three-dimensional space, for instance.)  Even if one’s data does not appear in a square matrix, relationships between different quantities can often be analyzed in this way by carrying out eigenanalysis of a corresponding covariance matrix, which will be square.

However, we need a new tool when we are looking at transformations from to .  For instance, we may want to look at a rectangular matrix of stock prices under different scenarios, or we might want to analyze how economic factors influence GDP of geographically related provinces or countries.  Singular value decomposition can even be used to analyze voting patterns in the US Senate, showing interesting information about party affiliation.  All of these involve rectangular arrays of data, and invite interpretation based on the meaning of the domain space and the range space.

Singular value decomposition of a rectangular matrix will factor into three matrices:

will be an almost-diagonal matrix – it will have the same dimensions as , so need not be square, but will be zero everywhere but the diagonal.  On the diagonal will appear the singular values of , explained below.  and will be orthogonal matrices that give rotation or reflection on or , as appropriate.

Here’s the fast version of the process of SVD:

- To find the orthonormal basis of that will give the columns of the matrix , we find the eigenvectors of .

- We order the by the magnitude of their eigenvalues: that means we have , and the matrix is

- The singular values that give the diagonal entries of are .

- We can find the columns of , which also give (part of?)  an orthonormal basis of , by solving for all the .  If , we can fill in the remaining by Gram-Schmidt orthogonalization.

- Then the matrix is

Since we define the singular values of by , for an eigenvector of , we can follow the calculations to prove some of the claims made above.  For an eigenvector of with eigenvalue ,

At the same time, using the definition of singular value we’ve got

Since the are orthonormal, by the Spectral Theorem, this tells us that either (when in the calculation above) or , when .  That implies the are also orthonormal, which we claimed above but didn’t justify.

This gives you a way of finding the SVD decomposition of a matrix , but what does it all mean?

9.14 Applications of SVD

So many!

1. The second eigenvector can be any multiple of .  ↑

Chapter 10

Joint distributions

10.1 Jointly distributed discrete random variables

For simplicity, we’ll start by considering two jointly distributed random variables at a time.  Once we’ve established these definitions, we can easily extend to jointly distributed random variables.

We say that and are jointly distributed discrete random variables if

- both and are defined on the same sample space , and

- there is a probability measure that satisfies the axioms of probability, so that

- we have a function called the joint probability mass function of and .

Notice that is a function defined on .  Remember that a random variable or is a function from sample space to that assigns numerical values to outcomes of probability experiments.  In particular, then, where and .  This seems a little pedantic but it’s nice to avoid confusion by trying precision from the start!

We can call or a random vector.  You can see how linear algebra will start emerging...

10.1.1 Marginal probability mass function

The marginal probability mass functions are what we get by looking at only one random variable and letting the other roam free.  For jointly distributed random variables , we sum over all possible values of to get the marginal for :

Similarly, sum over all possible values of to get the marginal for :

You can think of these as collapsing back to single-variable probability.  This definition has its foundation in one of the initial rules of probability we learned: if are mutually exclusive events and , then .  Think about how this gives the marginal probability mass functions above.

10.1.2 Examples of discrete jointly distributed random variables

Let’s do a pretty simple example first: we’ll return to our coin-tossing roots.

Toss a quarter and a dime into the air.  We’ll let and be the indicator random variables for heads on the dime and the quarter, respectively.  That means that if the dime shows heads and if the dime shows tails, and for the quarter showing heads, for tails.

The space of outcomes is pretty limited here: just four outcomes with equal probability.  So

Moreover, and are independent.  Check for yourself and see that .

Here is a transformation of the two random variables and .  Let be the number of heads showing wehn you’ve tossed the two coins, and let be the amount of money showing tails.  For instance, if the dime shows heads and the quarter shows tails, and cents.  What is the joint pmf of and ?

It’s nice to draw a picture here:

Notice how this table allows us to see the joint pmf for and and also shows the marginal pmfs nicely.

Are and independent?  Definitely not – simply by reasoning about the problem, we know that how much money shows tails is related to how many coins show heads.  We’ll return to this example in the next few sections.

Our coin-tossing problem didn’t have any nice formulas to work with.  For a contrasting discrete probability problem, we can return to dice!

Roll two dice.  Let be the maximum roll and the minimum roll.  What is the joint probability mass function for and ?

Again, let’s draw a table illustrating the joint pmf and the marginal pmfs:

*****add picture*******

Notice that we can come up with nice formulas for the marginal pmfs:

and

for .

10.1.3 Conditional probability mass function

Rather than looking at the probability of given outcomes for regardless of the value of (or vice versa), maybe we’d like to look at the probability of a given outcome of given a particular value of .  That is, maybe we want .  Using our previous rules for conditional probability, we know that

as long as is not zero.  Following through the definitions, we can find a conditional probability mass function as well:

for .  Likewise,

for .

This also allows us to define conditional expected value:

Let’s apply this to the example of rolling two dice that we considered a few paragraphs ago.  Remember is the max of our roll of two dice, while is the minimum of the roll of two dice.  We can find a nice equation for conditional probability of given any value :

For example, if we want the probability mass function for conditioned on , we’d evaluate and find:

Check this result using techniques from the past.

10.1.4 Independence

Build off of our old rule for independence: events and are independent if , so and are independent (jointly distributed) discrete random variables if and only if

for all .  Again, you can often use reasoning to guide you to whether two random variables are independent or not.

10.1.5 Multivariate versions

Extend everything I just said by looking at random vectors

taking values in .

Notice that we can write the expected value of a random vector as the vector of expected values:

10.2 Jointly distributed continuous random variables

Again, we have the concept of random vector , and again we’ll do everything in the two-dimensional case first.  The main difference from the last section is that continuous random variables have a probability density function and a cumulative distribution function to define, and so that’s what we’ll do!

Definition 2.1 The continuous random variables and have a joint probability density function if there exists a probability measure that satisfies

with for all and

A great fact: for a “nice” region , we have

Remember that in single-variable probability we were able to find the pdf from the cdf by differentiating.  Now we need partial derivatives, because we have two variables to consider.  Using the multivariate fundamental theorem of calculus, we can see that

Let’s do an example to see how this works in practice:

Example 2.1 I will give you a probability density function with a mystery parameter :

The restrictions on the domain mean that the random vector takes values in a triangle:

Let’s find the parameter that ensures that this pdf respects the laws of probability: we need

Notice the bounds of integration here: why are they what they are?  We want to integrate only over a triangle , not over the square .  Do the integration to find that this implies , so .  Thus the pdf is

10.2.1 Marginal and conditional probability density functions

We have a familiar pattern: take everything you learned from discrete random variables, turn the sum into an integral, and you’ve got properties of continuous random variables if correctly interpreted.  So, we’ve got marginal pdfs

and

Example 2.2 Let’s revisit the example of and jointly distributed with pdf

Integrate to find the marginal pdf , and think carefully what bounds you want:

for and else. Likewise,

for and else.  These are both functions of a single variable, and so only that variable can appear in the expression for the marginal pdf.

Remember that the marginal pdfs must also satisfy the laws of probability, so we have to have and and

Again, if is a random vector with a joint density function , then we can define the conditional probability density function for a particular value of as

when and zero elsewhere.  If we do this, we can define conditional expected value:

This is the expected value of given that .  However, if we are very careful here we may find some problems, and one of them is a very famous “paradox” in probability, the Borel-Kolomogorov paradox.

10.3 Covariance and correlation, again

Covariance and correlation were mentioned briefly earlier.  Remember that we looked at the variance of a sum of independent random variables in Chapter 5.  Now, we care about non-independent random variables (jointly distributed random variables!).  So...

is one way to define covariance between and .  There are of course more convenient formulas.  Another definition of covariance is

Use linearity of expectation to prove for yourself that this implies

If you need to compute the covariance of two random variables, this is often the easiest way to do it.

Covariance thus defined for two (one-dimensional) random variables is great, but in the multivariate situation we’ll want to look at the covariance matrix!  (This is exciting!  Trust me!)  For a random vector

where all the are jointly distributed, we define the covariance matrix as having entries for .  This is always a square matrix, and because of the symmetry of covariance, it’s always a symmetric matrix as well.  Be aware that terminology varies from field to field and book to book: some people call this a variance-covariance matrix, as you can see that the terms on the diagonal are all .  Notice that if all the variables are independent of each other, will be a diagonal matrix.

Another cute way to use linear algebra to write the covariance matrix of a random vector is to say that

Again, you can use properties of expectation to write

Last, we can prove that covariance matrices are always positive semidefinite by using the following trick.  Remember one characterization of a matrix being positive semidefinite was that for all non-zero vectors .  Well, for any random vector with mean ,

The quantity will be some scalar, which we call in the last line of our calculation, and since is always positive or at worst zero, we’re set!

Covariance has the drawback of being intimately related to the units and magnitudes of the random variables under consideration.  Correlation is the unitless sister to covariance, and more easily allows comparisons between different types of information.  Write .  Then as before we write for the correlation of and ,

Remember that you can prove

and in fact you can now prove that using the Cauchy-Schwarz inequality!  Moreover, we can define the correlation matrix as the matrix whose entries at spot are .

Mentioning the Cauchy-Schwarz inequality should give you flashbacks to the section on vector inequalities (Section 8.3).  In fact, covariance is conceptually a lot like an inner product, and in particular like the dot product for real vectors.  Remember that the dot product for real vectors satisfied a few properties:

- Symmetry: just as for we have , we have .

- Linearity: just as , we have .

- Positive-definiteness: just as and if and only if , we have and if and only if is (more or less) constant.[1]

This is actually useful for financial applications.

10.4 Multivariate change of variables

In the linear algebra sections of the book, we spent significant effort on changing bases.  Now we’re going to change variables.  This is conceptually similar and will use some linear algebra, but the huge difference is that we don’t have to make linear changes of variables!  We can make all sorts of changes of variables!

First, the technical details; then some examples.  Consider continuous random variables with a continuous joint distribution.  Call their joint probability density function and let it be defined on a domain .  Say we want to transform these random variables into new random variables, , using some one-to-one differentiable functions from to :

for .  Here is some new domain (could be the same as or different).  Since these are one-to-one and differentiable, there is an inverse transformation that has for each .  This takes points in to points in .  (Think about how this is the analogue of the “monotonically increasing” or “monotonically decreasing” condition in the single-variable case.)

Along with the inverse of the transformation, we need to know how the transformation changes the variables infinitesimally.  We understand this change through using the Jacobian,

Determinants are intimately related to volume – remember the determinant of a three by three matrix is the volume of the parallelopiped spanned by the three columns of the matrix, for instance.  (This was discussed in section 8.8.)  The partial derivatives measures how much is changing with respect to .  The combination of the determinant of the matrix of derivatives can be thought of as measuring the change of volume that is forced by the transformation.

Given all this set-up, the joint probability density function of the random variables is

There are essentially two actions here: you need to substitute in the new variables by plugging in each , and then you need to compensate for “stretching” caused by the transformation using .

10.5 Bivariate and multivariate normal

The bivariate normal distribution is a great place to start exploring multivariate normal distributions, as we can actually draw pictures.  Let’s start with a definition and a derivation.

Definition 5.1 A random vector of jointly distributed random variables has the bivariate normal distribution if is normally distributed for every pair with .

While a bit abstract, this is actually a very useful characterisation of bivariate normal distributions.  You might wonder, though, what the probability density function is.  That’s fair.  Let’s start, as in the single-variable case, with the simplest situation – here, two independent though jointly-distributed standard normal random variables.  Thus we consider and , with .  Then the pdf for their bivariate normal distribution is

for .  Notice this is no big surprise; since and are independent, their joint density function is just the product of their individual density functions.

As in the single-variable case, we can transform our way from this straightforward density function to any other bivariate normal density function.  How would we get to the probability density function for where , , and ?

One way to look at this is to transform the standard normal to have the desired parameters, by using affine linear transformations:

We shift to get the desired mean and stretch to get the desired variance .  Check for yourself that has the desired mean and variance!  Since and are linear combinations of normal random variables and , and will also be normally distributed because of our definition previously.  Notice that , as well.

How do we find the joint density function of and from this?  Use the multivariate change of variables formula discussed earlier.  In our current situation, we have inverse functions

and

These give the Jacobian

Putting this all together, we can get the horrible formula

Why do I call this horrible?  Because it takes a long time to type, and some people find it easy to mess up the recall of the formula on exams due to its length.  Use the power of linear algebra to simplify this expression!  If we write

then we can write instead

The best part is that this formula generalizes to a -dimensional multivariate normal: if is -dimensional, then

The only change is in the power of ; the structure of the density function remains the same.  Of course, here is the covariance matrix with .

You need to familiarize yourself with the probability density function for the bivariate and multivariate normal distributions to call yourself a financial mathematician.  They’re foundational.  However, in practice you won’t use these equations explicitly all that often.  You’ll often be able to rely on computer implementations of the multivariate normal to simulate returns or other quantities that you’re modeling with a normal distribution, and you’ll be able to use properties of the multivariate normal distribution to solve other problems without going to the pdf.  For instance, you might want to do a basic calculation like this:

Example 5.1 You’re considering two stocks and modeling their returns using a joint bivariate distribution.  Based on historical data, stock A has returns with mean and variance , while stock B has returns with mean and variance .  What is the probability that a portfolio that invests equally in both stocks will have returns greater than ?

Using standardization and z-tables will be a fine technique for solving many multivariate normal problems, and I would be negligent not to discuss these problems here.  On the other hand, you can find this material in many probability texts and I urge you to visit them for examples (see for instance Rosencrantz).

So as not to duplicate commonly-available standard materials, I’ll turn instead to visualization and some topics I feel haven’t been covered very well in the texts I’ve looked at.

10.6 Visualizing the bivariate normal distribution

We’ll concentrate on the bivariate normal distribution just because it’s easy to graph, but these ideas again generalize to higher dimensions!

You can graph the probability density function of a bivariate normal as a surface, with :

***add pic*****

This gives you a nice visual representation of the fact that if and have a bivariate normal distribution then any is also normally distributed (for ).

****add pic*****

Contour plots are a nice way of visualizing this pdf: we can take slices for constant values of and project them onto the plane, using different colors to show different values of :

*****add pic*********

At first glance this all seems a bit obvious, but the power of mathematics only comes into play if you dig deeper and question the obvious.  Why do we always get ellipses?  Do we always get ellipses?  If we have ellipses, what does their shape tell us?  (Any ideas?  Ponder these questions for three minutes and then read on, or if you’re truly dedicated to understanding, get the Python worksheet or grab your pencil and experiment.)

Do we always get ellipses?  Well, yes and no.  You don’t get an ellipse if and are equal, for instance.  More generally, you don’t get an ellipse if the covariance matrix is singular, but in that case you also can’t use the usual equation for the probability density function because is not defined and you have a divide-by-zero error at the very beginning (the factor).  If the covariance matrix is full rank, which must be the case when you’re using the pdf given, you get an ellipse.

Why an ellipse?  To answer this, we need quadratic forms.  Each of these level sets is the set of points that satisfy .  Let’s solve:

I don’t want to write out all the parts of this constant, so make a new constant .

This is a quadratic form!!  Bet you thought you wouldn’t need Section 9.7 again.  Hah.  The matrix for this quadratic form is .  What do we know about ?  Well, we know it’s positive definite.  How do we know that?  is positive definite, with eigenvalues and corresponding eigenvectors .  Then has eigenvalues for corresponding eigenvectors , and all those eigenvalues are also positive – so must also be positive definite.

Speaking of these eigenvectors, they provide the directions of the major and minor axes of the ellipses under consideration.  (In higher dimensions, they give all the axes of the “covariance ellipsoids”.)  The eigenvector corresponding to the biggest eigenvalue of gives the major axis of the ellipse (the long dimension of the ellipse) and the eigenvector corresponding to the smaller eigenvalue gives the minor axis of the ellipse, in this bivariate case.

1. To be precise here, I should actually say “almost everywhere constant” or “almost surely constant” – constant everywhere but a set of measure zero.  But you need measure theory for this. ↑

Chapter 11

Optimization and Newton’s method

In the single-variable portion of the course, we emphasized short- and long-term predictions (differentiation and integration) along with single-variable probability.  Then we learned about linear algebra with real and complex numbers, mixing that up with joint distributions of random variables.  Now we’re going to head toward different kinds of approximation: approximating solutions to equations via Newton’s method and approximating scalar-valued functions themselves using power series and Taylor polynomials.

11.1 Single-variable Unconstrained optimization

As a bit of motivation and a setting for these techniques, let’s start with optimization for functions with .  The single-variable case is very familiar to you, as it’s what a first calculus course emphasizes, but I’d like to call out the parts that are particularly useful when we move to functions defined on for .

11.1.1 Single-variable minima and maxima

Long ago and possibly far far away in your first calculus class, you learned how to find local and global maxima and minima of functions by

- finding the points at which or at which does not exist, and then

- classifying such points as local maxima, minima, or saddles using either the first or second derivative tests,

- and comparing all the values and the behavior of the function to determine global maxima and minima (if they exist).

To find the points where , you can use the bisection method or Newton’s method if it’s not straightforward to simply solve the equation.  To classify the critical points as maxima, minima, or saddle points, you can look at the first derivative on either side of the point (does the function change from increasing to decreasing, decreasing to increasing, or not change direction?)  or you can use the second derivative (check if the function is concave up, concave down, or neither at the point ).  Let’s do a few examples.

Example 1.1 Here’s an example that won’t exactly come up in finance, but which is an illustration of the analytical methods you can use to find local extrema.  Think about the function .

Identify critical points first: when , so for all for .  How can we classify the critical points?

The first derivative test would tell us to look at where is increasing (where ) and where is decreasing (where <0).  Well, we know for all , so is always non-decreasing – so all those critical points are actually saddles by the first derivative test.

Another check: if we look at , we see that does alternate between being concave up () and concave down ().  But at , .  So what does this tell us?  Actually, nothing.  The second derivative test is indeterminate, because each critical point is an inflection point as well.

A slight modification of the function is very different.  If you consider , you’ll find forces .  Looking at this geometrically, you can see that two sets of values solve this: and for .  So we again have an infinite set of critical points.

How do we classify these critical points?  Since on the intervals where , we have increasing on and all translates, and decreasing on and its translates.  Increasing to decreasing...  that’s a max, and so each is a max.  Decreasing to increasing makes a minimum, and so are all minima.

If you don’t want to think like that, check the second derivative.  , and so for all , which then are all minima.  Similarly, for , so those are maxima.

To know whether your extrema are local or global extrema,

- Check whether the function goes to infinity or negative infinity as goes to infinity or negative infinity, and check to see if the function has any asymptotes.

- Make a list of the value of the function at the maxes and mins.

- Compare the two parts above and pick out the local and global maxes and mins.

For instance, a function whose highest-degree term has a positive coefficient will go to positive infinity as goes to or , because of the even degree.  That means it cannot have a global maximum at any particular value of in , because you can always find a bigger and a bigger output.

11.1.2 Single-variable constrained optimization

In the single-variable case, it’s actually usually really easy to deal with contraints, because they just can’t be that complicated.  You can find the minima or maxima of a function on an interval or some subset of an interval.  For instance, you can find the mins and maxes of on the interval by using our previous work (which tells us is a critical point in this interval) and then comparing the output of there with the value of at the ends of the interval:

With a picture it’s very easy to see what’s going on.  Even in the absence of a picture we can follow the following procedure as long as is continuous on .

To find the absolute maximum and minimum of on :

- Find the critical points of in the interior .  (You don’t need to classify them.)

- Evaluate at the critical points and evaluate and .  Write these down in a list.

- Circle the biggest value, circle the smallest value.  These give you your absolute min and absolute max!

There is a theorem that applies here:

Theorem 1.1 (Extreme Value Theorem) As long as is continuous on , will attain an absolute maximum and an absolute minimum on the interval .

The idea that you should look at the interior of the contrained region and then look at the boundaries will carry over into the multivariable setting.

11.2 Newton’s method, single-variable

Many equations can’t be solved exactly.  Even many polynomials can’t be solved exactly!  Newton’s method provides a way of approximating roots of any differentiable function.  You might think that Newton’s method is thus not very versatile...  but you just spent a few hours (I hope) finding for various functions .  That means you were finding roots of the function .  Newton’s method can be generalized into a variety of optimization techniques, even inspiring methods in machine learning.  Again, we’ll start with the familiar single-variable version (and look at some failures of the method) and then generalize.

Newton’s method relies on our old idea of short-term approximation or linear approximation, in essence “running it backward” to find roots by assuming that there is some close to our initial guess such that .  If we have an approximate solution , so , we want a better solution (one so that is ideally closer to zero than ).  Use the basic equation for short-term prediction to estimate this new :

This suggests we consider

which we can use in .  In fact, this often works!  And if it works once, why not try again and find an even better ?

Newton’s method, then, is an iterative process that solves by finding a sequence using

When successful, this sequence of s converges to a root .  In general, you don’t have to iterate too many times; if the root you are seeking to approximate is of multiplicity one and your initial guess was a good one, then the number of correct digits after the decimal roughly doubles after every iteration.  This is proved using power series, which we will discuss further.

Example 2.1 Find an approximate value for .  (Hint: this is the same as solving the equation .)

Example 2.2 Find an approximate value for the solution of .

Newton’s method is fairly straightforward when it works.  It fails to work for several reasons.  Two important “failure modes” are when for some in the iterative process, and when the sequence of fails to converge.

Visualize the first failure mentioned above: when , the graph of has a horizontal tangent line, and so the next step of Newton’s method (finding the intersection of the horizontal tangent line with the -axis) cannot be carried out.

The second failure is very interesting mathematically, though outside the scope of this class.  It leads directly into what’s called “dynamical systems theory” and “chaos theory”.

A note on conditions for convergence of Newton’s method: You can prove that for a function there exists an open interval around a root of so that for any in , Newton’s method converges if is continuously differentiable over and for all in .  Basically, if is “nice” and is small, we can get convergence.

Example 2.3 Analyze and explain why , for instance, leads to a sequence that does not converge.  We have a theorem that guarantees conditions under certain conditions.  Why doesn’t the theorem apply?

Example 2.4 Analyze .  Start with .  What happens?  Draw a picture.

This is an example of a period-two solution.  Newton’s method oscillates between two solutions forever.  Instead of starting with , start with or another similarly small value.  Do you get a different solution?  What happens?  You should see an example of “sensitivity to initial conditions.”  While this seems somewhat theoretical, this comes up a lot in financial modeling.  If you’re creating a model more complicated than a linear regression, you may come up with something that gives you pretty different results depending on your initial conditions.  “Robustness” and “sensitivity analysis” are both words that come up in this context.  Often professionals in mathematics prefer robust models, models that don’t give wildly different results given slightly different inputs, because so many inputs coming from the real world are estimates or adjustments.

Example 2.5 Consider the function and try any initial value between and .  What happens?  Draw a picture if you can!

This is an example of “chaotic” behavior.  Chaos is avoided by most financial mathematicians, but some instead lean into it: Benoit Mandelbrot, for instance, has put forward the view that traditional stochastic calculus is fundamentally inaccurate for financial mathematics and that chaos theory and in particular fractal self-similarity are better for describing the behavior of financial markets.  Check out his book, “The Mis-behavior of Markets,” to learn more.

11.2.1 Newton’s method for single-variable optimization

Instead of using Newton’s method to find , we can use it to find and thus find extrema more directly.  Check for yourself that now the iteration is

Since you’ll have computed in the process, this will give a quick and streamlined way of finding and classifying extrema as long as .  If , you won’t be able to use this method at all – the iteration will diverge quickly!

Put this into practice by finding the absolute minimum of to four decimal places without first finding the critical points of .

11.3 Multivariable Taylor approximations

You will notice that everything above used and maybe , first and second derivatives of single-variable functions.  These pieces of information can be encapsulated in linear approximations and quadratic approximations of the function .  To generalize the methods above to multiple variables, we need multivariate versions of this.

We will start with degree-two approximations of single-variable functions to establish our notation and the ideas, then move to multivariable functions .  Our primary multivariable examples will be functions because these can be drawn on paper!

How do you write the degree-two polynomial approximation to any function ?  Dredge up memories from Taylor approximation and remind yourself now.  Remember that the Taylor series of a function or at a point is given by the power series

By just truncating this power series at some point, we can get fine degree- approximations near the input :

This includes our linear approximation,

and the quadratic approximation

We’d like to do something similar for a multivariable function .  Write or for the (scalar) output of the function.

What is the multivariable equivalent of the first derivative?

It is called the gradient, and it assembles all the first derivatives of in one vector of partial derivatives: the gradient of is

You can evaluate at a point by plugging in :

At a point , this gradient vector indicates the direction of fastest change – the direction in which you’d walk from to experience the greatest “slope.”

What is the multivariable equivalent to the second derivative?  Instead of a vector, we now need a matrix.  This matrix is called the Hessian, and it’s a delightful matrix of all the second derivatives of .

What is true about the Hessian matrix ?

- If is a real-valued function of real variables, is a real matrix.

- It is symmetric by Clairaut’s theorem, as long as is second-differentiable around .

- Its eigenvalues and eigenvectors at the point tell us about the properties of the function near .  In particular, we can classify critical points of using the Hessian evaluated at the critical point.

The multivariable quadratic approximation to at can be written

or in a slightly different notation where is the small change from point ,

Try applying this yourself:

Example 3.1 Find the quadratic approximation to the function at the point .

These quadratic approximations are useful in a variety of contexts, not just optimization.  In finance, option pricing methods include delta-approximations (-approximation) which is a linear approximation, delta-gamma approximation (-approximation), and then delta-gamma-theta methods.  These are all developed to estimate option price movements when the price of the underlying is also changing.  For an option on a single stock, single-variable methods suffice...  but of course you want to work with portfolios!

11.3.1 Zeroes of multivariate functions

You might ask yourself if you could find the roots of a function using something like Newton’s method.  Well, kind of.  The problem with finding roots of a function is that often zeroes of such a function are not isolated.  For instance, what do the zeroes of look like?  Any points in the unit circle give .  Or if you want to know where is zero – it’s an entire surface in .  When is a polynomial, the study of sets of points that give is an entire area of mathematics called algebraic geometry (the field in which I got my PhD!).  It’s very beautiful mathematics, and there are even practical applications in optimization, cryptography (elliptic curves, for instance), and genetics (!).  However, algebraic geometry is really beyond the scope of this course.

One idea I will pull out from algebraic geometry is an idea that starts in multivariable calculus and linear algebra.  In linear algebra, we saw that a system of linear equations in variables has a unique solution if the corresponding matrix is full rank.  If you have more equations than variables, you may have an “overdetermined” system – too many conditions may mean no solutions.  If you have fewer equations than variables, you may have an “underdetermined” system – too much freedom means that solutions are not unique.  You’ve got a positive-dimensional space of solutions, in linear algebra.  When you have nonlinear equations, it’s not quite so clear, but a similar idea carries through.  Looking at , you’ve got one condition or constraint in a three-dimensional ambient space.  That leaves two degrees of freedom, and so you’ve got two-dimensional solution set.  This reasoning can be extended.

Anyhow, we’ll leave this discussion here.  A multivariate Newton’s method for finding zeroes of just is not so easy because that zero set won’t just be a point.  However, we can use Newton’s method more fruitfully in the setting of optimization.  Isolated maxes and mins of are much more common, and very useful in finance.  Maximizing the return of a portfolio, minimizing the variance – classic applications.

11.4 Multivariate optimization

11.4.1 Unconstrained optimization

A critical point of the function is a point at which , or at which is not differentiable.  For a differentiable function , how can you interpret this in terms of “steepest ascent”?[1]  (What does this look like for a differentiable function , for instance?)  Maxima and minima of can appear only at critical points.

We can classify a critical point of as a maximum, minimum, or saddle by looking at .  This is the multivariate version of the second derivative test.  Specifically, if this matrix is

- positive definite, then is a strict minimum

- negative definite, then is a strict maximum

- positive/negative semidefinite, then is a non-strict minimum/maximum

- indefinite, then is a saddle point

Wonderfully, we can get the definiteness of a matrix from looking at its eigenvalues!

Quiz yourself:

- If the eigenvalues of a matrix are positive, the matrix is                   .

- If the eigenvalues of a matrix are negative, the matrix is                   .

- If one of the eigenvalues is zero, then the matrix is                    or                   , depending on

- If the matrix has eigenvalues of opposite signs, then the matrix is                   .

I’ll write this as a theorem so it’s official (and this can carry over to constrained optimization):

Theorem 4.1 If is a function with continuous second partial derivatives on a set , and is an interior point of that is also a critical point of , then the eigenvalues of the matrix determine whether is a (local) maximum, minimum, or saddle point of .

Again, if you’d like to check whether points are globally optimal, you’ll need to understand the behavior of the function (continuous or not?  bounded or not?)  and then examine the values of the function at the critical points and as inputs go off to infinity in various directions.

11.4.2 Multivariate Newton’s method for optimization

Let’s return to the consideration of Newton’s method for optimization.  Why is Newton’s method useful when we’re considering optimization?  Look at the linear and quadratic approximations to at :

Now, if happens to be a critical point of the function , then .  But the point of Newton’s method is that we create an iterative process that starts with a point that is not the minimum (if we knew the minimum, we wouldn’t need to optimize!).  So, rather than set and solve, we want to be slightly cleverer.  Instead set and use that to give us a hint.

If we set this equal to zero and solve, we’d get

Iterate!

11.4.3 Constrained optimization

Either you want maxes/mins on the boundary of a region, or in a bounded region.....

1. If you said that means there is some sort of a “flat point” at , you may have the right idea. ↑

Chapter 12

Differential equations

Goals of this chapter

- find equilibrium solutions of basic differential equations

- classify equilibrium points of systems of first-order equations, at least roughly!

This topic is mostly qualitative, but involves elements of review from earlier topics as well.  Keep an eye out for:

- short-term prediction

- long-term prediction

- Euler’s method

- a possible application of Newton’s method

- optimization

- eigenvalues and eigenvectors

12.1 Equilibrium: the concept

Differential equations allow us to study how quantities change in relation to each other, as they’re equations that set out the relationships between given derivatives.  Equilibrium is a useful concept to group our exploration: it’s defined as “a state in which opposing forces or influences are balanced” according to Google’s source on August 27, 2018.  In the world of differential equations, think of equilibrium as a solution to a differential equation in which quantities are not increasing or decreasing, but instead are constant...  a solution in which derivatives are zero!

12.2 A single ordinary differential equation

Consider a basic differential equation like

This is the sort of equation we considered at the beginning of the semester: we were able to solve such an equation using the method of long-term approximation or using analytical methods:

so

and thus

In general, humans are more interested in gaining or losing money than keeping it the same, but it’s actually extraordinarily useful for mathematicians and analysts to understand equilibria, points at which the rate of change of a quantity is zero.  For this equation under consideration, the equilibrium solution would be when , which is when (by the design of the equation), which occurs only when the function .

Equilibrium analysis is useful because if we consider a smoothly-changing , then can’t change signs without passing through zero!  (When I say “smoothly-changing” I mean that in is a continuous and differentiable function.  This is stricter than necessary but will serve our purposes.)  In fact, for a single differential equation like this which depends only on , we can draw a phase line that illustrates the behavior of by looking at the sign of .

Example 2.1 Consider the example

For what values of is the system at equilibrium?  That is, when is ?  Draw a phase line and indicate where is positive and where it is negative.

The equation above is an example of a logistic differential equation.  The logistic equation is very useful in population ecology, and there is written

In this situation is population as a function of time, is the growth rate of the population, and is the carrying capacity of the environment.

Example 2.2 Practice reading the differential equation: change in population is proportional to the rate parameter times the population times one minus the ratio of population to carrying capacity.  What are the equilibrium values for the differential equation?  Draw a phase line and characterize the behavior of the population with various initial conditions.  redReview: Remember initial value problems were introduced in September.

To carry out heuristic analysis of these differential equations, another great graphical tool is the slope field or direction field.  Slope fields can be generated by computer or drawn by hand.  To draw a slope field for a single differential equation of the form , draw the - and -axes and then for every point draw a small tick mark with slope .

Here is an example: the slope field for is

To use a slope field for qualitative analysis of solutions to a differential equation, start at an initial value of your choice and simply follow the arrows!  This traces a solution curve, and this curve is the graph of a solution to the differential equation.

Recall that we discussed Euler’s method for numerically solving these differential equations.  Euler’s method puts together many short-term predictions to make a long-term prediction.  How does that work again?  Review!

Example 2.3 Use Euler’s method or the method of long-term predictions to estimate if you know and you start at .  Use two steps.

Example 2.4 (Challenge) Solve analytically.

12.3 So I can model a caribou population: what about money?

A good qualitative understanding of differential equations is essential to progressing toward the Ito calculus and Black-Scholes equation.  Let’s take a look at the Black-Scholes equation for a moment:

Here is the price, when the stock price is and time is , of a European call option struck at price with an expiration date of .  The parameter is the volatility of the stock’s returns.  This equation is a partial differential equation (PDE) as opposed to the ordinary differential equations (ODEs) we’re considering in this document.  But the principles of “reading” a differential equation remain the same!  A very first analysis can go as follows:

- If we considered a simplified under some extraordinary condition in which did not vary according to at all (imagine simply holding constant but letting time run) we’d end up with the equation

Look familiar?  Yes, it’s just the exponential growth from the first page of this document!

- If we let stay constant and just let vary, as a thought experiment, we’d get

This is a second-degree differential equation, as it has those second derivatives.  Ordinary autonomous second-degree differential equations can be split into two ordinary first-degree differential equations; check out this concept in the next chapter.

Understanding the uses and pitfalls of the Black-Scholes equation is a significant endeavor in financial math!  Having an acquaintance with ordinary differential equations gives a mental context for dealing with Black-Scholes.

12.4 Systems of Differential Equations

Systems of differential equations: use everything you know about linear algebra and transfer it to the differential equation setting!  Let’s start with something simple to get an idea of why this might work.

Example 4.1 Consider the system of equations dxdt = 3x

dydt = -2y. Alone, you’d be happy to solve either of these equations.  You would get and .  This is still perfectly reasonable.

We can rewrite the system of differential equations using the language of matrices:

Then the solution is

Solutions to this differential equation, then, are curves in xy-space: they depend on (they are parametrized by ).

To draw a slope field or phase plane for a two-dimensional system of equations, take a point in the plane and draw a tiny vector in the direction of

***Notice something: no equation I write today has a appearing explicitly on the right-hand side.  Instead, they are of the form .  These are called autonomous differential equations.  It is possible in math to deal with non-autonomous systems

but that is definitely outside the scope of this class!!

Example 4.2 Systems of differential equations can be linear (the functions describing the derivatives are linear and of form , for and real numbers at least for today) or non-linear.  Here’s an example of a linear system:

Here is a phase plane with some solutions included (for positive only):

By contrast, here’s a nonlinear system:

Notice that I can’t write it using the matrix format!  Here’s the slope field, with some solutions sketched.  Which solution is wrong?  Why is it wrong?  Where are common sources of error when using Euler’s method?

12.5 Equilibria

We can find “equilibrium points” (we now call them fixed points instead) for these differential equations just as we found equilibrium solutions for single differential equations.  Just solve for the points where and are both zero!  For a linear system of differential equations, the origin will always be an/the only fixed point.  (Why?)  For nonlinear systems you’ve got to use whatever techniques you can to find these solutions.

Example 5.1 For

solve

and get the curve

(A whole curve…!  Can you see this on the slope field?  What would it mean?) Then solve

to get

Put these together: if both are true,

This has many solutions, but is one of them and fits into our picture.  If that is true, then and that gives us our fixed point.  Messy!  This is just one of many fixed points since is periodic.

Look at this fixed point in the picture: how do the solutions relate to the fixed points?  How do they behave together?  It looks like solutions “tend toward’’ the fixed point.  When solutions tend toward a fixed point, the fixed point is called a sink.  When solutions tend away from a fixed point, the fixed point is called a source.  When nearby solutions zoom past a fixed point, careening toward it and then rushing away, the fixed point is called a saddle.[1]

Let’s deal with a simpler situation:

Example 5.2 Consider

This has two equilibria.  Solve to get , and to get .  Then , so .  This has the solutions of and .  That means is an equilibrium point and is also an equilibrium point.

12.6 Straight-line solutions

Looking at phase planes of systems of linear first-order differential equations, you can see that there are often straight-line solutions – solution curves that follow a straight line into or out of the origin.  This is no accident.  Turns out they run along eigenvectors.  In fact, the solutions to a system of two linear DEs

where has real eigenvalues with corresponding eigenvectors are completely classified by

(Actually, we can do the same if the eigenvectors are complex, but we have to pick off the real parts of the solution by using the identity

and combining this with the complex eigenvectors.)

Example 6.1 Consider the system

The matrix has eigenvalues , : (2-) (-6-) +7 = 0

2+4 -5 = 0

(+5) (-1) =0 . It’s got eigenvectors

and

(check yourself!).  So the straight-line solutions are

and

Look at the phase plane:

Here I did not draw the whole straight line solution in either situation: I drew only the solution starting (with ) at an initial value and continuing for positive .  I did this because then you can see that the solutions associated with go “in” toward the origin (as decreases when increases) and the solutions associated with go “out” from the origin (as increases when increases).

12.7 Back to Black-Scholes for a minute

In the first chapter I mentioned that second-degree differential equations can often be split into two first-degree differential equations.  The most classic example of this is the motion of a spring with a mass on the end, possibly with some external forcing.  Back to physics for a moment!

The equation for the displacement of a spring in this situation is

where is the mass at the end of the spring, is some damping coefficient (is your spring moving through air, oil, peanut butter?), is the spring constant ( stiffness), and describes the external forcing.[2]  Use a very simple trick to rewrite this second-order equation as two first-order equations:

Look: I made up a variable name for !  Now we get

This is not quite linear, but there are standard mathematical tools to deal with this.  Moreover, if (in other words, there’s no external forcing) you’ve got all the information you need to solve

1. How does this remind you of minima, maxima, and saddles when we look at optimizing functions like ? ↑

2. Compare it to Black-Scholes:

There are some similarities; the big difference is that Black-Scholes has derivatives with respect to time and stock price.
↑

Table of Contents

Frontmatter

Chapter 1: Introduction to probability

Chapter 2: Geometry problems in probability

Chapter 3: Probability rules

Chapter 4: First applications in finance

Chapter 5: Discrete random variables and transformations of variables

Chapter 6: Continuous Random Variables

Chapter 7: Random walks

Chapter 8: Linear algebra

Chapter 9: Spectral theorem and portfolio management

Chapter 10: Joint distributions

Chapter 11: Optimization and Newton's method

Chapter 12: Differential equations