Multivariate Statistics: Lectures

Statistics Course

Course Overview

This page brings together graduate-level lectures on multivariate statistics, covering the foundations of statistical estimation, hypothesis testing, linear and logistic regression, Principal Component Analysis (PCA), and Factor Analysis. Each lecture is available as a downloadable PDF. Selected lectures also include video recordings in French. This material builds on the foundations established in Measure & Probability and provides the statistical groundwork for Machine & Deep Learning.

1 Basic Statistics

PDF and video of the lecture on Basic Statistics, covering the foundational concepts of statistical inference.
📄 PDF 🎬 Vidéo (FR)
We begin with statistical estimation, the cornerstone of inferential statistics, aiming to infer characteristics of a population from sample data. We define the statistical framework, including the sample space, the parameter space, estimands, and estimators, and we explore the mean squared error of an estimator along with estimators of expectation and variance, focusing on Gaussian random variables. We then study confidence intervals, providing a range of plausible values for population parameters, with propositions for the expectation and variance of Gaussian random variables, Gaussian approximation techniques, and the case of Bernoulli’s parameter. We move on to hypothesis testing, covering null and alternative hypotheses, error rates, the concept of power, and tests for Gaussian random variables with known and unknown standard deviations. We clarify the role of statistic and test statistic in decision-making processes within hypothesis tests. Finally, we explore likelihood functions, evaluating the plausibility of parameter values given observed data, with a focus on the Gaussian distribution.

Course Outline:

  1. Statistical estimation
    1. Statistical framework
    2. Mean squared error of an estimator
    3. Estimators of expectation and variance
  2. Confidence interval
  3. Hypothesis testing
    1. Hypothesis tests
    2. Hypothesis tests for Gaussian random variables
  4. Statistic and test statistic
  5. Likelihood

2 Linear Fitting and Regression

PDF and video of the lecture on Linear Fitting and Regression, fundamental concepts in statistical modeling.
📄 PDF 🎬 Vidéo (FR)
We begin with unidimensional input scenarios, distinguishing between linear fitting (deterministic points, finding the best-fitting line through the least-squares method) and linear regression (where the output is random). We explore the connection between least-squares fitting and projection, introduce the determination coefficient for assessing the goodness of fit, and discuss regression errors, parameter estimation, and linear prediction. We then extend our analysis to multidimensional input spaces, finding the best-fitting hyperplane, discussing the decomposition of output variance, and covering multidimensional regression errors, parameter estimation, residual analysis, and prediction error assessment. Finally, we introduce the Gaussian linear regression model, deriving maximum likelihood estimators for the parameters under the assumption of Gaussian errors, providing a probabilistic framework for linear regression analysis.

Course Outline:

  1. Unidimensional input
    1. Linear fitting
    2. Linear regression
    3. Linear prediction
  2. Multidimensional input
    1. Multidimensional linear fitting
    2. Multidimensional linear regression
    3. Multidimensional linear prediction
  3. Gaussian model
    1. Maximum likelihood estimators

3 Logistic Regression

PDF and video of the lecture on Logistic Regression, a pivotal technique for analyzing discrete outcome variables.
📄 PDF 🎬 Vidéo (FR)
We begin with binary logistic regression, defining the model with the logit and sigmoid functions mapping probabilities. We explicate the likelihood function, the log-likelihood function, and discuss efficient numerical methods for calculating the maximum likelihood estimators. We then transition to binary logistic prediction, leveraging the learned model to predict outcomes for new observations. Finally, we explore multiclass logistic regression, a sophisticated extension for scenarios with more than two categorical outcomes, defining the model and deriving its maximum likelihood estimators, and we cover multiclass logistic prediction, extending the principles of binary prediction to multiple categorical outcomes.

Course Outline:

  1. Binary logistic regression
    1. Binary logistic regression model
  2. Binary logistic prediction
  3. Multiclass logistic regression
    1. Multiclass logistic regression model
    2. Multiclass logistic prediction

4 Principal Component Analysis (PCA)

PDF of the lecture on Principal Component Analysis (PCA), a fundamental technique for dimensionality reduction in multivariate analysis.
📄 PDF
We start with PCA for random observed variables, elucidating the spectral decomposition of the covariance matrix and introducing the concept of principal component transformation, accompanied by key definitions, lemmas, and propositions. We then discuss PCA’s adaptability to deterministic observed variables, outlining the spectral decomposition of empirical covariance matrices and exploring essential aspects such as variance reduction and orthogonality. The technique aims to reduce redundancy among observed variables while preserving the essential information of the data structure.

Course Outline:

  1. PCA for random observed variables
  2. PCA for deterministic observed variables

5 Factor Analysis

PDF of the lecture on Factor Analysis, a nuanced approach to uncovering latent variables underlying observed variables.
📄 PDF
We begin with an overview of the Factor Analysis model, including observations, common factors, and factor loadings. We explore the existence of factor analysis models, scaling considerations, and the non-uniqueness of factor loadings. We discuss factor interpretation techniques, including factor rotation methods such as the Varimax criterion. Finally, we cover various estimation methods for factor loadings, encompassing the principal component method, the principal factor method, and the maximum likelihood method. Factor Analysis facilitates a deeper understanding of data structures by revealing the underlying factors that explain the observed correlations.

Course Outline:

  1. Existence of factor analysis model
  2. Scaling in factor analysis
  3. Rotation of factors
  4. Factors interpretation
  5. Estimating loadings

Book: Preliminary Version Available

The lectures on this page are based on the first preliminary version of our book, available as a preprint on the HAL open archive: M. K. Karray, B. Błaszczyszyn, L. Darlavoix — Data Science: From Statistics to Machine Learning and Deep Learning, with Applications to Wireless Networks. Preliminary Version (Preprint). HAL open archive, October 2024. The book covers a broader scope including Multivariate Statistics (the content of this page), Machine Learning, and Applications to Wireless Networks. See also Machine & Deep Learning and Wireless Networks for the corresponding pages.

About These Topics

These graduate-level lectures on multivariate statistics are designed for students and researchers seeking a rigorous mathematical foundation for data analysis, statistical modeling, and machine learning applications. The material adopts a measure-theoretic approach, building on the foundations of probability theory and providing the statistical groundwork for advanced topics in machine learning and deep learning. The course covers the foundational concepts of statistical estimation, including estimators, confidence intervals, hypothesis testing, and likelihood functions. It then develops linear regression in both univariate and multivariate contexts, with the least-squares method, the Gaussian linear model, and maximum likelihood estimators. Logistic regression is introduced for discrete outcome variables, covering both binary and multiclass classification, with the sigmoid and softmax functions and the log-likelihood function. The lectures conclude with Principal Component Analysis (PCA) for dimensionality reduction via spectral decomposition of the covariance matrix, and Factor Analysis for uncovering latent variables through factor loadings, factor rotation (including the Varimax criterion), and various estimation methods (principal component, principal factor, and maximum likelihood). The material draws on the book Data Science: From Statistics to Machine Learning and Deep Learning, with Applications to Wireless Networks by Karray, Błaszczyszyn, and Darlavoix, available as a preprint on the HAL open archive.

Last Updated on 24 mai 2026 by Mohamed Kadhem KARRAY