Estimation and inference: formulas
Pick a formula to see it explained with worked examples and a solver that shows every step.
About this topic
Estimation is making a good guess about a number from data. Inference is saying how sure you are about it. On this page you will find 34 pages, such as Fisher information and Cramér–Rao bound, Posterior is proportional to likelihood × prior and Kullback–Leibler divergence.
Each page explains the idea in plain words and lets you try your own numbers in a solver that shows every step. Try Fisher information and Cramér–Rao bound first.
Formulas in this topic
- Fisher information and Cramér–Rao boundThe Fisher information is I(θ)=-E[dfrac∂²ln f(X;θ)∂θ²],
- Posterior is proportional to likelihood × priorIn Bayesian inference the posterior distribution of a parameter is proportional to the likelihood times the pr
- Normal–normal conjugate updateWith a normal prior μsim N(μ_0,1/τ_0) (precision τ_0) and n observations with mean bar x from N(m
- Kullback–Leibler divergenceThe Kullback–Leibler divergence D_KL(P\|Q)=Σ p_iln(p_i)/(q_i) measures how different the distributi
- Shannon entropyThe Shannon entropy of a distribution is H(X)=-Σ p_iln p_i, the average surprise of an outcome. It satisfi
- Mutual informationThe mutual information of two variables is I(X;Y)=H(X)+H(Y)-H(X,Y). It measures how much knowing one reduces t
- Sufficiency (factorisation theorem)A statistic T(x) is sufficient for θ if the sample density factorises as f(xmidθ)=g(T(x),θ)h
- Rao–Blackwell theoremIf T is sufficient and hatθ is an estimator, then tildeθ=E(hatθmid T) is also an estimator
- Neyman–Pearson lemmaFor testing a simple null H_0:θ=θ_0 against a simple alternative H_1:θ=θ_1, the most power
- Multiple testing: Bonferroni and Benjamini–HochbergWhen m hypotheses are tested, the Bonferroni correction tests each at level (α)/(m). The Benjamini–H
- Akaike and Bayesian information criteriaFor a model with k parameters fitted to n observations with maximised likelihood hat L: AIC=2k-2ln\
- Ridge regressionRidge regression adds a penalty λ\|boldsymbolβ\|² to least squares. The solution is hatboldsy
- Bias–variance decompositionThe expected squared prediction error of an estimator hat f splits as El[(y-hat f)²r]=operatorna
- Logistic function and its derivativeThe logistic function p=dfrac11+e^(-eta) maps any real eta to a probability between 0 and 1. Its deriva
- Softmax and cross-entropyThe softmax turns scores z_i into probabilities, softmax(z)_i=dfrace^(z_i)Σ_je^(z_j), a
- Principal component analysisPrincipal component analysis finds the eigenvectors mathbf v_i of the covariance matrix Σmathbf v_i=la
- Expectation–maximisationThe EM algorithm fits models with hidden variables Z by repeating two steps: the E-step computes Q(θmid\
- Survival analysisIn survival analysis the hazard is h(t)=(f(t))/(S(t)), the cumulative hazard is H(t)=-ln S(t), and the K
- Time series: AR(1)The autoregressive model AR(1) is X_t=φ X_t-1+varepsilon_t with independent noise varepsilon_t. It is s
- Kalman filter gainIn the Kalman filter, the gain K=PH^(T)(HPH^(T)+R)⁻¹ decides how much of the measurement error t
- Chernoff and Hoeffding boundsHoeffding's inequality bounds the deviation of a sum of independent bounded variables: P(S_n-ES_n≥ tri
- Bootstrap standard errorThe bootstrap estimates the standard error of a statistic by resampling the data with replacement B times, com
- Bayes' theoremBayes' theorem reverses a conditional probability: P(Amid B)=(P(Bmid A)P(A))/(P(B)), with P(B) found
- Mean squared errorThe mean squared error of an estimator hatθ is MSE(hatθ)=E[(hatθ-θ)²bi
- Maximum likelihood estimationThe maximum likelihood estimate is the value hatθ of the parameter that makes the observed data most pro
- MLE: exponential rateFor independent observations x_1,dots,x_n from an exponential distribution with rate λ, the maximum lik
- MLE: normal mean and varianceFor a normal sample, the maximum likelihood estimates are hatμ=bar x and hatσ²=dfrac1nΣ(x_i-\
- Confidence interval for a proportionAn approximate confidence interval for a population proportion is hat p± z_α/2sqrtdfrachat p(1-
- Two-sample pooled varianceWhen two independent samples come from populations with the same variance, the pooled variance is s_p²=dfr
- ANOVA sum of squares and F statisticIn one-way ANOVA the total variation splits as SS_T=SS_B+SS_W (between groups plus within groups), and the tes
- Coefficient of determinationThe coefficient of determination is R²=1-(SSE)/(SST), the fraction of the variation in y explained by
- Multiple regression (normal equations)In multiple regression mathbf y=Xboldsymbolβ+boldsymbolvarepsilon, the least-squares coefficients are
- Likelihood-ratio test statisticThe likelihood-ratio statistic compares the maximised likelihood under the null hypothesis with the unrestrict
- Residual variance of a simple regressionThe residual variance of a simple linear regression is s²=(SSE)/(n-2), where SSE=Σ(y_i-hat y_i)^2
Estimation and inference questions
What makes a good estimate?
One that is close to the true value on average and does not jump around too much.
Does the solver show every step?
Yes. Each page shows the working line by line, then checks the answer so you can see if yours matches.