ESE 531: Statistical Learning and Inference

Course Description

This graduate-level course covers fundamental concepts in statistical inference and estimation theory, including parameter estimation, hypothesis testing, and Bayesian methods.

Topics

• Properties of random samples – definition of a random sample, population distribution, definition of a statistic, sampling distribution, sample mean, sample variance, probability inequalities, convergence of random variables, weak and strong law of large numbers, central limit theorem, delta method,
• Point estimation – maximum likelihood estimation, method of moments estimation, sufficient statistics, exponential families, single and multi-parameter estimation, methods for estimation (e.g., analytical, and numerical optimization, expectation maximization), point estimation under parameter constraints, Gaussian mixture models,
• Evaluation of estimators – mean-squared error criterion, bias-variance tradeoff, unbiased estimators, minimum variance criterion; existence of minimum variance unbiased estimators, Fisher information, Cramer-Rao bounds, properties of maximum likelihood estimators, predictive performance,
• Linear models – definition and properties, maximum likelihood estimation for linear models, noise model, colored noise model, best linear unbiased estimator
• Bayesian estimators – Bayesian vs. frequentist philosophies, Bayes’ theorem, conjugate priors, MAP estimation, MMSE estimation, Jeffrey’s prior, examples of Bayesian inference, approximate Bayesian inference, Laplace approximation, Monte Carlo methods, Metropolis-Hastings algorithm,
• Detection theory – Binary hypothesis testing, type-I and type-II errors, simple and composite hypothesis testing, Neyman-Pearson theorem, likelihood ratio test, receiver operating characteristics, probability of error, multiple hypothesis testing, generalized likelihood ratio test, examples of various detectors

Course Materials

Lectures

• Lecture 1: Random Samples, Probability Inequalities, Limit Theorems
• Lecture 2 (coming soon!)

Homework Assignments

• Homework 1: Properties of Random Samples
• Homework 2 (coming soon!)

Recommended Supplementary Reading

• Fundamentals of Statistical Signal Processing: Estimation Theory, S. M. Kay, Prentice Hall, 1993
• Fundamentals of Statistical Signal Processing: Detection Theory, S. M. Kay, Prentice Hall, 1998
• Statistical Inference, 2nd ed., G. Casella and R. Berger, Duxbury Press, 2002

Prerequisites

• Calculus
• Linear algebra
• Probability theory