3 The algorithm of estimation: Ronald Fisher and maximum likelihood

As the sample size increases, the sample mean converges toward the true population mean (𝜇 = 5). Although estimates from small samples fluctuate because of random sampling variation, those fluctuations diminish as additional observations are incorporated, illustrating the consistency of the sample mean.

The logistic (sigmoid) function maps the linear predictor onto a probability between 0 and 1. The midpoint occurs at 𝑧 = 0, corresponding to a predicted probability of 0.5.

Fitted regression line for the synthetic study-hours data set. Each point represents an observed pair of study hours and exam score, while the line represents the conditional mean estimated by maximum likelihood under the normal-error model. The vertical distance between each point and the line is its residual. Because maximizing the Gaussian likelihood produces the same coefficient estimates as minimizing the sum of squared residuals, this is also the ordinary least squares regression line.

• Maximum likelihood estimation is a general-purpose method for estimating model parameters by finding the values that make the observed data most probable under an assumed probability model. Once a model is specified, maximizing its likelihood produces parameter estimates, making MLE still—more than a century later—one of the most widely used estimation frameworks in statistics and machine learning.
• Fisher's approach to inference reoriented statistical reasoning away from subjective prior beliefs and toward the evidence contained in the observed data. Rather than asking what should be believed before seeing the data, MLE asks which parameter values make the observed sample most plausible.
• Likelihood measures the plausibility of parameter values given the observed data, reversing the usual probability perspective of evaluating the probability of the data under fixed parameters. This shift made likelihood the central object of Fisher's theory of statistical inference.
• Consistency means that an estimator converges to the true parameter value as the sample size increases. Fisher viewed this property as essential because it ensures that more data lead to increasingly accurate estimates.
• Efficiency refers to achieving the smallest possible variance among unbiased estimators, allowing the available data to be used as effectively as possible.
• Sufficiency ensures that an estimator captures all the information in the sample relevant to the parameter being estimated, permitting data reduction without sacrificing inferential accuracy.
• Fisher argued that MLE satisfied the desirable properties of consistency, efficiency, and sufficiency more reliably than competing approaches of his era, particularly Bayesian inference and the method of moments. Although modern Bayesian methods have addressed many of Fisher's criticisms, the philosophical distinction between likelihood-based and Bayesian inference remains central to statistics today.
• Logistic regression demonstrates MLE in action by estimating the coefficients that maximize the likelihood of the observed binary outcomes. As one of the earliest statistical learning methods founded on maximum likelihood estimation, logistic regression remains one of the most enduring and widely used applications of Fisher's likelihood framework.
• Linear regression, under the assumption of normally distributed errors, illustrates that maximizing the likelihood is mathematically equivalent to minimizing the sum of squared residuals using ordinary least squares, thereby revealing the likelihood foundation beneath one of statistics' most familiar methods.
• The likelihood framework extends naturally from parameter estimation to model comparison. Likelihood ratio tests compare nested models using their maximized likelihoods, while information criteria such as AIC and BIC begin with the maximum likelihood and penalize unnecessary model complexity to guard against overfitting.

FAQ