Some Basic Statistical Tools

Outline for Today

  1. Note on methods of learning
  2. Convergence for sequences of random variables
  3. Break
  4. Concentration inequalities
  5. Proof of the weak law of large numbers
  6. (If time) Proof techniques

Methods of Learning

(Disclaimer: I am not a trained epistemologist)

Testimony

Perception

(Scientific Method)

Reason

(Mathematical Proof)

  1. Read or hear a piece of knowledge
  2. Decide whether to believe it (e.g. based on authority of speaker)
  3. Learn by believing it
  1. Formulate a question
  2. Formulate a hypothesis
  3. Test the hypothesis with an experiment
  4. Learn by analyzing the results of the experiment
  1. Formulate a question
  2. Create a conjecture that answers the question
    1. Define all concepts
    2. State as a theorem
  3. Prove or disprove the conjecture
  4. Learn by considering the theorem/false conjecture and the proof

Most RL research fits into the scientific method.

Review

Given a probability space \((\Omega, \mathcal{F}, P)\), and a measurable space \((E, \mathcal{E})\), an \(E\)-valued random variable is a measurable function \(X: \Omega \to E\).

\(\omega \in \Omega\)

\(\Omega\)

\(\mathcal{F} = \sigma(\{\quad\}) = \{\Omega,\quad, \quad, \emptyset\}\)

\(\mathcal{F}\)

Review: Equality of Random Variables

Review: Convergence

For a (deterministic) sequence \(\{x_n\}\), we say

\[\lim_{n \to \infty} x_n = x\]

or

\[x_n \to x\]

if, for every \(\epsilon > 0\), there exists an \(N\) such that \(|x_n - x| < \epsilon\) for all \(n > N\).

Random Variables: Sure Convergence

\[X_n(\omega) \to X(\omega) \quad \forall \, \omega \in \Omega\]

Random Variables:

Almost Sure Convergence

\(X_n \stackrel{a.s.}{\to} X\) if and only if \(P(\{\omega: X_n(\omega) \to X\}) = 1\), that is, \(X_n\) converges to \(X\) except possibly on a zero-measure set.

Does sure convergence imply almost sure convergence?

Random Variables:

Convergence in Probability

\(X_n \to_p X\) if \(P(\{\omega : |X_n(\omega) - X(\omega) | > \epsilon\}) \to 0\) for any fixed \(\epsilon > 0\).

Does \(X_n \stackrel{a.s}{\to} X\) imply \(X_n \to_p X\)?

Does \(X_n \to_p X\) imply \(X_n \stackrel{a.s}{\to} X\)?

No.

But there exists a subsequence \(n_k\) such that \(X_{n_k} \stackrel{a.s.}{\to} X\).

Random Variables:

Convergence in Probability

Random Variables:

Weak Convergence

\(X_n \stackrel{D}{\to} X\) if \(F_{X_n}(\alpha) \to F_{X}(\alpha)\) for each fixed \(\alpha\) that is a continuity point of \(F_X\).

"Weak convergence", "convergence in distribution", and "convergence in law" all mean the same thing.

Summary

Convergence:

  • Sure ("pointwise")
     
  • Almost Sure
     
  • In Probability

     
  • Weak ("in distribution"/"in law")

\[X_n \to X \iff X_n(\omega) \to X(\omega) \quad \forall \, \omega \in \Omega\]

\[X_n \stackrel{a.s}{\to} X \iff P(\{\omega: X_n(\omega) \to X\}) = 1\]

\(X_n \to_p X \iff P(\{\omega : |X_n(\omega) - X(\omega) | > \epsilon\}) \to 0 \quad \forall \epsilon > 0\)

\(X_n \stackrel{D}{\to} X \iff F_{X_n}(\alpha) \to F_{X}(\alpha)\) for each continuity point.

Break: Convergence of MC integration

\[Q_N \to \frac{\pi}{4} \text{ (sure)?}\]

 

\[Q_N \stackrel{a.s.}{\to} \frac{\pi}{4} \text{?}\]

 

\[Q_N \to_p \frac{\pi}{4} \text{?}\]

 

\[Q_N \stackrel{D}{\to} \frac{\pi}{4} \text{?}\]

\[Q_N \equiv \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_{X_{i,1}^2 + X_{i,2}^2 \leq 1} (X_i)\]

\(X_i \sim U([0,1]^2)\)

(Intuitively)

Concentration Inequalities

Concentration inequalities take the form

\[P(X \geq t) \leq \phi(t)\]

where \(\phi\) goes to zero (quickly) as \(t \to \infty\)

Intuition: If an r.v. has a finite variance, the probability that a random variable takes a value far from its mean should be small

Concentration Inequalities

Markov's Inequality:

If \(X \geq 0\), then \[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]

Concentration Inequalities

Chebyshev's Inequality:

Let \(X\) be any real-valued random variable with \(\text{Var}(X) < \infty\). Then

\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\text{.}\]

Very general, but still loose

Concentration Inequalities

Chebyshev's Inequality

Concentration Inequalities

Moment generating function: \(M_X(t) \equiv E[e^{tX}]\)

Chernoff Bound: If the moment-generating function \(M_X\) exists, then

\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]

and

\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]

Tighter than Markov and Chebyshev

Bonus Slide: Moment Generating Functions

To get the \(n\)th moment from \(M_X(t)\), differentiate it \(n\) times and set \(t=0\).

Because:

Concentration Inequalities

Name

Requirements

Bound

Chebyshev

\(\text{Var}(X) < \infty\)

\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\]

Markov

\(X\geq 0\),  \(\text{E}[X]\) exists

\[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]

Chernoff

\(M_X\) exists

\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]

\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]

Exercise

Let \(Y\) be a r.v. that takes values in \([-1,1]\) with mean -0.5. Give an upper bound on the probability that \(Y \geq 0.5\).

Name

Requirements

Bound

Chebyshev

\(\text{Var}(X) < \infty\)

\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\]

Markov

\(X\geq 0\),  \(\text{E}[X]\) exists

\[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]

Chernoff

\(M_X\) exists

\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]

\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]

(Weak) Law of large numbers

Let \(X_i\) be independent identically distributed r.v.s with mean \(\mu\) and variance \(\sigma^2\). If \(Q_N \equiv \frac{1}{N} \sum_{i=1}^N X_i\), then \(Q_N \to_p \mu\).

Proof:

Law of Large Numbers

Two somewhat astounding takeaways:

1. Standard deviation decays at \(\frac{1}{\sqrt{N}}\) regardless of dimension.

2. You can estimate the "standard error" with \[SE = \frac{s}{\sqrt{N}}\]

where \(s\) is the sample standard deviation.