Definition of a quantitle:

Let the cdf of \(y\) be \(F_Y=P(Y\leq y)\). The \(q\)th quantile of Y is then \(Q_Y(q)=F_Y^{-1}(q)=inf\{y:F_Y(y)\leq q\}\). I.e. it is the smallest value in the distribution such that the cdf is greater than the quantile. E.g. the 90% quantile is the first value of the distribution where the cdf exceeds 0.9.

Frequentist Quantile Regression

In frequentist statistics, the parameters are fixed. The following is taken from the bayesQR paper,

Consider the standard linear model, \[y_i=x_i^T\beta+\epsilon_i\] The conditional mean model is obtained by assuming that \(E(\epsilon|x)=0\). The likelihood (how likely parameter values are given the data) is then, \[l(x_i,y_i,\beta, \epsilon, \sigma)=\Pi_{i=1}^{n} \dfrac{1}{2\pi \sigma^2}exp\left(-\dfrac{(y_i-x_i^T\beta)^2}{\sigma^2}\right)\] and the log likelihood is, \[L(x_i,y_i,\beta, \epsilon, \sigma)\propto -\sum_{i=1}^n(y_i-x_i^T\beta)^2,\]

So we see that to maximise the log likelihood (a negative number), we can convert it to a minimisation problem - in which case the regression coefficients can be obtained by solving \[\hat\beta=argmin_{\beta}\sum_{i=1}^n (y_i-x_i^T\beta)^2.\]

If we consider QR, then the optimisation problem needs to depend on \(q\). To do this, we introduce the check function, \(p_q(.)\),

\[ p_q = \begin{cases} q &\quad\text{if residuals}\geq 0 \implies y_i\geq x_i^T\beta\\ 1-q &\quad\text{if residuals}< 0 \implies y_i<x_i^T \beta \\ \end{cases} \]

So that now the optimisation problem is, \[\hat\beta=argmin_{\beta}\sum_{i=1}^n p_q(y_i-x_i^T\beta)=argmin_{\beta}\left(\sum_{i: y_i \geq x_i^T\beta}q|y_i-x_i^T\beta|+\sum_{i: y_i < x_i^T\beta}(1-q)|y_i-x_i^T\beta| \right)\] Note that if \(q=0.5\) (median regression) then this reduces to \(\hat\beta=argmin_{\beta}\sum|y_i-x_i^T\beta|\).

The quantreg R package uses frequentist methodology to estimate quantile regression coefficients. It offers 3 methods to solve the above minimisation problem:

  1. Simplex method (default) using method=br: Uses the fact that solutions are focused on the vertices of the constraint set.

  2. Frisch-Newton interior point method using method="fn" or method="pfn" for very large problems: Instead of focusing on the vertices, it traverses the interior of the feasible region.

  3. Sparse regression quantile fitting using method="sfn" or Sparse version of the above, it is efficient when the design matrix has many zeros (e.g. if the predictors contain several factors).

Bayesian Quantile Regression

See Yu et al. and APTS notes that explain Bayesian quantile regression whereby a likelihood function that is based on the asymmetric Laplace distribution (ALD) is used.

The Laplace distribution with density \(f(z)=\frac{1}{2\sigma} exp\left(-\frac{|z-\mu|}{\sigma}\right)\) has the nice property that the MLE of \(\mu\) is the sample median. The asymmetric Laplace distribution is a generalisation of the Laplace distribution whereby the MLE of \(\mu\) is now the sampled quantile of \(z\).

The AL distribution is similar to the normal distribution but is “spikier” and is allowed to be asymmetric. The pdf for the ALD is, \[f(y | x_i^T\beta, \sigma, q)=\frac{q(1-q)}{\sigma} \exp \left(-p_q\left(\frac{y- x_i^T\beta}{\sigma}\right)\right).\] Notice that this contains the check function \(p_q(.)\).

If we assume place an ALD prior on the residuals with \(\sigma=1\) then the likelihood of our model becomes \[L(y|\beta)=q^n(1-q)^n exp\left(-\sum_{i=1}^n p_q(y_i-x_i^T\beta) \right).\]

We can then combine this with a prior to get our posterior, and use the standard methods to approximate this.

\[\psi(\beta,\sigma|y,x,q)\propto \pi(\beta,\sigma)\Pi_{i=1}^n \text{ALD}(y_i | x_i^T\beta, \sigma,q).\]

Bayesian quantile regression can be performed in R using bayesQR, which places weak priors on \(\beta\) and \(\sigma\) and using a Gibbs sampler to estimate the model parameters. Users need to specify the number of MCMC draws, and there is an option to use adaptive lasso variable selection.

Coefficient estimation: Marginal change in \(q\)th quantile due to marginal change in \(x\).