Some Basic Statistical Tools
Outline for Today
- Note on methods of learning
- Convergence for sequences of random variables
- Break
- Concentration inequalities
- Proof of the weak law of large numbers
- (If time) Proof techniques
Methods of Learning
(Disclaimer: I am not a trained epistemologist)
Testimony
Perception
(Scientific Method)
Reason
(Mathematical Proof)
- Read or hear a piece of knowledge
- Decide whether to believe it (e.g. based on authority of speaker)
- Learn by believing it
- Formulate a question
- Formulate a hypothesis
- Test the hypothesis with an experiment
- Learn by analyzing the results of the experiment
- Formulate a question
- Create a conjecture that answers the question
- Define all concepts
- State as a theorem
- Prove or disprove the conjecture
- Learn by considering the theorem/false conjecture and the proof
Most RL research fits into the scientific method.
Review
Given a probability space \((\Omega, \mathcal{F}, P)\), and a measurable space \((E, \mathcal{E})\), an \(E\)-valued random variable is a measurable function \(X: \Omega \to E\).
\(\omega \in \Omega\)
\(\Omega\)
\(\mathcal{F} = \sigma(\{\quad\}) = \{\Omega,\quad, \quad, \emptyset\}\)
\(\mathcal{F}\)
Review: Equality of Random Variables
Review: Convergence
For a (deterministic) sequence \(\{x_n\}\), we say
\[\lim_{n \to \infty} x_n = x\]
or
\[x_n \to x\]
if, for every \(\epsilon > 0\), there exists an \(N\) such that \(|x_n - x| < \epsilon\) for all \(n > N\).
Random Variables: Sure Convergence
\[X_n(\omega) \to X(\omega) \quad \forall \, \omega \in \Omega\]
Random Variables:
Almost Sure Convergence
\(X_n \stackrel{a.s.}{\to} X\) if and only if \(P(\{\omega: X_n(\omega) \to X\}) = 1\), that is, \(X_n\) converges to \(X\) except possibly on a zero-measure set.
Does sure convergence imply almost sure convergence?
Random Variables:
Convergence in Probability
\(X_n \to_p X\) if \(P(\{\omega : |X_n(\omega) - X(\omega) | > \epsilon\}) \to 0\) for any fixed \(\epsilon > 0\).
Does \(X_n \stackrel{a.s}{\to} X\) imply \(X_n \to_p X\)?
Does \(X_n \to_p X\) imply \(X_n \stackrel{a.s}{\to} X\)?
No.
But there exists a subsequence \(n_k\) such that \(X_{n_k} \stackrel{a.s.}{\to} X\).
Random Variables:
Convergence in Probability
Random Variables:
Weak Convergence
\(X_n \stackrel{D}{\to} X\) if \(F_{X_n}(\alpha) \to F_{X}(\alpha)\) for each fixed \(\alpha\) that is a continuity point of \(F_X\).
"Weak convergence", "convergence in distribution", and "convergence in law" all mean the same thing.
Summary
Convergence:
- Sure ("pointwise")
- Almost Sure
- In Probability
- Weak ("in distribution"/"in law")
\[X_n \to X \iff X_n(\omega) \to X(\omega) \quad \forall \, \omega \in \Omega\]
\[X_n \stackrel{a.s}{\to} X \iff P(\{\omega: X_n(\omega) \to X\}) = 1\]
\(X_n \to_p X \iff P(\{\omega : |X_n(\omega) - X(\omega) | > \epsilon\}) \to 0 \quad \forall \epsilon > 0\)
\(X_n \stackrel{D}{\to} X \iff F_{X_n}(\alpha) \to F_{X}(\alpha)\) for each continuity point.
Break: Convergence of MC integration
\[Q_N \to \frac{\pi}{4} \text{ (sure)?}\]
\[Q_N \stackrel{a.s.}{\to} \frac{\pi}{4} \text{?}\]
\[Q_N \to_p \frac{\pi}{4} \text{?}\]
\[Q_N \stackrel{D}{\to} \frac{\pi}{4} \text{?}\]
\[Q_N \equiv \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_{X_{i,1}^2 + X_{i,2}^2 \leq 1} (X_i)\]
\(X_i \sim U([0,1]^2)\)
(Intuitively)
Concentration Inequalities
Concentration inequalities take the form
\[P(X \geq t) \leq \phi(t)\]
where \(\phi\) goes to zero (quickly) as \(t \to \infty\)
Intuition: If an r.v. has a finite variance, the probability that a random variable takes a value far from its mean should be small
Concentration Inequalities
Markov's Inequality:
If \(X \geq 0\), then \[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]
Concentration Inequalities
Chebyshev's Inequality:
Let \(X\) be any real-valued random variable with \(\text{Var}(X) < \infty\). Then
\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\text{.}\]
Very general, but still loose
Concentration Inequalities
Chebyshev's Inequality
Concentration Inequalities
Moment generating function: \(M_X(t) \equiv E[e^{tX}]\)
Chernoff Bound: If the moment-generating function \(M_X\) exists, then
\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]
and
\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]
Tighter than Markov and Chebyshev
Bonus Slide: Moment Generating Functions
To get the \(n\)th moment from \(M_X(t)\), differentiate it \(n\) times and set \(t=0\).
Because:
Concentration Inequalities
Name
Requirements
Bound
Chebyshev
\(\text{Var}(X) < \infty\)
\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\]
Markov
\(X\geq 0\), \(\text{E}[X]\) exists
\[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]
Chernoff
\(M_X\) exists
\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]
\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]
Exercise
Let \(Y\) be a r.v. that takes values in \([-1,1]\) with mean -0.5. Give an upper bound on the probability that \(Y \geq 0.5\).
Name
Requirements
Bound
Chebyshev
\(\text{Var}(X) < \infty\)
\[P(|X-E[X]| \geq t) \leq \frac{\text{Var}(X)}{t^2}\]
Markov
\(X\geq 0\), \(\text{E}[X]\) exists
\[P(X \geq t) \leq \frac{E[X]}{t}\quad \forall \, t \geq 0\]
Chernoff
\(M_X\) exists
\[P(X \geq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t > 0\]
\[P(X \leq a) \leq \frac{E[e^{tX}]}{e^{ta}} \quad \forall\, t < 0\]
(Weak) Law of large numbers
Let \(X_i\) be independent identically distributed r.v.s with mean \(\mu\) and variance \(\sigma^2\). If \(Q_N \equiv \frac{1}{N} \sum_{i=1}^N X_i\), then \(Q_N \to_p \mu\).
Proof:
Law of Large Numbers
Two somewhat astounding takeaways:
1. Standard deviation decays at \(\frac{1}{\sqrt{N}}\) regardless of dimension.
2. You can estimate the "standard error" with \[SE = \frac{s}{\sqrt{N}}\]
where \(s\) is the sample standard deviation.
050 Some Basic Statistical Tools
By Zachary Sunberg
