Pearson correlation coefficient

From neurov.is/on

Original content imported from the Wikipedia article Pearson correlation coefficient.

In statistics, the Pearson product-moment correlation coefficient (sometimes referred to as the PMCC, and typically denoted by r) is a measure of the correlation (linear dependence) between two variables X and Y, giving a value between +1 and −1 inclusive. It is widely used in the sciences as a measure of the strength of linear dependence between two variables. It was developed by Karl Pearson from a similar but slightly different idea introduced by Francis Galton in the 1880s.^[1]^[2] The correlation coefficient is sometimes called "Pearson's r."

1 Definition
2 Mathematical properties
3 Interpretation
- 3.1 Geometric interpretation
- 3.2 Interpretation of the size of a correlation
4 Inference
5 Pearson's correlation and least squares regression analysis
6 Sensitivity to the data distribution
7 Calculating a weighted correlation
8 Removing correlation
9 References

[edit] Definition

Pearson's correlation coefficient between two variables is defined as the covariance of the two variables divided by the product of their standard deviations:

$\rho_{X,Y}={\mathrm{cov}(X,Y) \over \sigma_X \sigma_Y} ={E[(X-\mu_X)(Y-\mu_Y)] \over \sigma_X\sigma_Y},$

The above formula defines the population correlation coefficient, commonly represented by the Greek letter ρ (rho). Substituting estimates of the covariances and variances based on a sample gives the sample correlation coefficient, commonly denoted r :

$r = \frac{\sum ^n _{i=1}(X_i - \bar{X})(Y_i - \bar{Y})}{\sqrt{\sum ^n _{i=1}(X_i - \bar{X})^2} \sqrt{\sum ^n _{i=1}(Y_i - \bar{Y})^2}}.$

An equivalent expression gives the correlation coefficient as the mean of the products of the standard scores. Based on a sample of paired data (X_i, Y_i), the sample Pearson correlation coefficient is

$r = \frac {1}{n-1} \sum ^n _{i=1} \left( \frac{X_i - \bar{X}}{s_X} \right) \left( \frac{Y_i - \bar{Y}}{s_Y} \right)$

where

$\frac{X_i - \bar{X}}{s_X}, \bar{X}, \text{ and } s_X$

are the standard score, sample mean, and sample standard deviation.

[edit] Mathematical properties

The absolute value of both the sample and population Pearson correlation coefficients are less than or equal to 1. Correlations equal to 1 or -1 correspond to data points lying exactly on a line (in the case of the sample correlation), or to a bivariate distribution entirely supported on a line (in the case of the population correlation). The Pearson correlation coefficient is symmetric: corr(X,Y) = corr(Y,X).

A key mathematical property of the Pearson correlation coefficient is that it is invariant to changes in location and scale. That is, we may transform X to a + bX and transform Y to c + dY, ahere a, b, c, and d are constants, without changing the correlation coefficient (this fact holds for both the population and sample Pearson correlation coefficients).

The Pearson correlation can be expressed in terms of uncentered moments. Since μ_X = E(X), σ_X² = E[(X − E(X))²] = E(X²) − E²(X) and likewise for Y, and

since

$E[(X-E(X))(Y-E(Y))]=E(XY)-E(X)E(Y),\,$

we may also write

$\rho_{X,Y}=\frac{E(XY)-E(X)E(Y)}{\sqrt{E(X^2)-(E(X))^2}~\sqrt{E(Y^2)- (E(Y))^2}}.$

Alternative formulae for the sample Pearson correlation coefficient are also available:

$r_{xy}=\frac{\sum x_iy_i-n \bar{x} \bar{y}}{(n-1) s_x s_y}=\frac{n\sum x_iy_i-\sum x_i\sum y_i} {\sqrt{n\sum x_i^2-(\sum x_i)^2}~\sqrt{n\sum y_i^2-(\sum y_i)^2}}.$

The above formula conveniently suggests a single-pass algorithm for calculating sample correlations, but, depending on the numbers involved, it can sometimes be numerically unstable.

[edit] Interpretation

The correlation coefficient ranges from −1 to 1. A value of 1 implies that a linear equation describes the relationship between X and Y perfectly, with all data points lying on a line for which Y increases as X increases. A value of −1 implies that all data points lie on a line for which Y decreases as X increases. A value of 0 implies that there is no linear correlation between the variables.

More generally, note that

$(X_i-\bar{X})(Y_i-\bar{Y}) > 0$

if and only if X_i and Y_i lie on the same side of their respective means. Thus the correlation coefficient is positive if X_i and Y_i tend to be simultaneously greater than, or simultaneously less than, their respective means. The correlation coefficient is negative if X_i and Y_i tend to lie on opposite sides of their respective means.

[edit] Geometric interpretation

For centered data (i.e., data which have been shifted by the sample mean so as to have an average of zero), the correlation coefficient can also be viewed as the cosine of the angle between the two vectors of samples drawn from the two random variables.

Some practitioners prefer an uncentered (non-Pearson-compliant) correlation coefficient. See the example below for a comparison.

As an example, suppose five countries are found to have gross national products of 1, 2, 3, 5, and 8 billion dollars, respectively. Suppose these same five countries (in the same order) are found to have 11%, 12%, 13%, 15%, and 18% poverty. Then let x and y be ordered 5-element vectors containing the above data: x = (1, 2, 3, 5, 8) and y = (0.11, 0.12, 0.13, 0.15, 0.18).

By the usual procedure for finding the angle between two vectors (see dot product), the uncentered correlation coefficient is:

$\cos \theta = \frac { \bold{x} \cdot \bold{y} } { \left\| \bold{x} \right\| \left\| \bold{y} \right\| } = \frac { 2.93 } { \sqrt { 103 } \sqrt { 0.0983 } } = 0.920814711.$

Note that the above data were deliberately chosen to be perfectly correlated: y = 0.10 + 0.01 x. The Pearson correlation coefficient must therefore be exactly one. Centering the data (shifting x by E(x) = 3.8 and y by E(y) = 0.138) yields x = (−2.8, −1.8, −0.8, 1.2, 4.2) and y = (−0.028, −0.018, −0.008, 0.012, 0.042), from which

$\cos \theta = \frac { \bold{x} \cdot \bold{y} } { \left\| \bold{x} \right\| \left\| \bold{y} \right\| } = \frac { 0.308 } { \sqrt { 30.8 } \sqrt { 0.00308 } } = 1 = \rho_{xy},$

as expected.

[edit] Interpretation of the size of a correlation

Correlation	Negative	Positive
None	−0.09 to 0.0	0.0 to 0.09
Small	−0.3 to −0.1	0.1 to 0.3
Medium	−0.5 to −0.3	0.3 to 0.5
Large	−1.0 to −0.5	0.5 to 1.0

Several authorsTemplate:Who have offered guidelines for the interpretation of a correlation coefficient. Cohen (1988),^[3] has observed, however, that all such criteria are in some ways arbitrary and should not be observed too strictly. The interpretation of a correlation coefficient depends on the context and purposes. A correlation of 0.9 may be very low if one is verifying a physical law using high-quality instruments, but may be regarded as very high in the social sciences where there may be a greater contribution from complicating factors.

[edit] Inference

Early work on the distribution of the sample correlation coefficient was carried out by R. A. Fisher^[4]^[5] and A. K. Gayen.^[6] Confidence intervals and hypothesis tests relating to ρ are usually carried out using the Fisher transformation:

$F(r) = {1 \over 2}\log{1+r \over 1-r} = \operatorname{arctanh}(r).$

If F(r) is the Fisher transformation of r, and n is the sample size, then

$z = \sqrt{n-3}F(r)$

is a z-score for r which approximately follows a standard normal distribution under the null hypothesis of no linear association (ρ = 0), given the assumption that the sample pairs are independent and identically distributed and follow a bivariate normal distribution. Thus an approximate p-value for the hypothesis ρ=0 can be obtained from a normal probability table. For example, if z = 2.2 is observed and a two-sided p-value is desired, the p-value is 2Φ(−2.2) = 0.028, where Φ is the standard normal cumulative distribution function.

To obtain a confidence interval for ρ, we first compute a confidence interval for z, then invert the Fisher transformation to the correlation scale. For example, suppose we observe r = 0.3 with a sample size of n=50, and we wish to obtain a 95% confidence interval for ρ. The z-score is z = 2.12, so the confidence interval on the z-scale is 2.12 ± 1.96, or (0.16, 4.08). Converting back to the correlation scale yields (0.02, 0.53).

[edit] Pearson's correlation and least squares regression analysis

The square of the sample correlation coefficient, which is also known as the coefficient of determination, estimates the fraction of the variance in Y that is explained by X in a linear regression analysis. As a starting point, the total variation in the Y_i around their average value can be decomposed as follows

$\sum_i (Y_i - \bar{Y})^2 = \sum_i (Y_i-\hat{Y}_i)^2 + \sum_i (\hat{Y}_i-\bar{Y})^2,$

where the $\hat{Y}_i$ are the fitted values from the regression analysis. This can be rearranged to give

$1 = \frac{\sum_i (Y_i-\hat{Y}_i)^2}{\sum_i (Y_i - \bar{Y})^2} + \frac{\sum_i (\hat{Y}_i-\bar{Y})^2}{\sum_i (Y_i - \bar{Y})^2}.$

The two summands above are the fraction of variance in Y that is explained by X (right) and that is unexplained by X (left).

Next, we apply a property of least square regression analysis, that the sample covariance between $\hat{Y}_i$ and $Y_i-\hat{Y}_i$ is zero. Thus, the sample correlation coefficient between the observed and fitted response values in the regression can be written

$\begin{align} r(Y,\hat{Y}) &= \frac{\sum_i(Y_i-\bar{Y})(\hat{Y}_i-\bar{Y})}{\sqrt{\sum_i(Y_i-\bar{Y})^2\cdot \sum_i(\hat{Y}_i-\bar{Y})^2}}\\ &= \frac{\sum_i(Y_i-\hat{Y}_i+\hat{Y}_i+\bar{Y})(\hat{Y}_i-\bar{Y})}{\sqrt{\sum_i(Y_i-\bar{Y})^2\cdot \sum_i(\hat{Y}_i-\bar{Y})^2}}\\ &= \frac{\sum_i(Y_i-\hat{Y}_i)(\hat{Y}_i-\bar{Y}) +(\hat{Y}_i+\bar{Y})^2}{\sqrt{\sum_i(Y_i-\bar{Y})^2\cdot \sum_i(\hat{Y}_i-\bar{Y})^2}}\\ &= \sqrt{\frac{\sum_i(\hat{Y}_i-\bar{Y})^2}{\sum_i(Y_i-\bar{Y})^2}}. \end{align}$

Thus

$r(Y,\hat{Y})^2 = \frac{\sum_i(\hat{Y}_i-\bar{Y})^2}{\sum_i(Y_i-\bar{Y})^2}$

is the proportion of variance in Y explained by a linear function of X.

[edit] Sensitivity to the data distribution

The population Pearson correlation coefficient is defined in terms of moments, and therefore exists for any bivariate probability distribution for which the population covariance is defined and the marginal population variances are defined and are non-zero. Some probability distributions such as the Cauchy distribution have undefined variance and hence ρ is not defined if X or Y follows such a distribution. In some practical applications, such as those involving data suspected to follow a heavy-tailed distribution, this is an important consideration. However, the existence of the correlation coefficient is usually not a concern; for instance, if the range of the distribution is bounded, ρ is always defined.

In the case of the bivariate normal distribution the population Pearson correlation coefficient characterizes the joint distribution as long as the marginal means and variances are known. For most other bivariate distributions this is not true. Nevertheless, the correlation coefficient is highly informative about the degree of linear dependence between two random quantities regardless of whether their joint distribution is normal^[1]. The sample correlation coefficient is the maximum likelihood estimate of the population correlation coefficient for bivariate normal data, and is asymptotically unbiased and efficient, which roughly means that it is impossible to construct a more accurate estimate than the sample correlation coefficient if the data are normal and the sample size is moderate or large. For non-normal populations, the sample correlation coefficient remains approximately unbiased, but may not be efficient. The sample correlation coefficient is a consistent estimator of the population correlation coefficient as long as the sample means, variances, and covariance are consistent (which is guaranteed when the law of large numbers can be applied).

Like many commonly-used statistics, the sample statistic r is not robust^[7], so its value can be misleading if outliers are presentTemplate:Page number^[8]^[9]. Specifically, the PMCC is neither distributionally robust,Template:Citation needed nor outlier resistant^[7] (see Robust statistics#Definition). Inspection of the scatterplot between X and Y will typically reveal such a situation, and in such cases it may be advisable to use a robust measure of association if the purpose of a study is simply to assess statistical dependence.

Statistical inference for Pearson's correlation coefficient is sensitive to the data distribution. Exact tests, and asymptotic tests based on the Fisher transformation can be applied if the data are approximately normally distributed, but may be misleading otherwise. In some situations, the bootstrap can be applied to construct confidence intervals, and permutation tests can be applied to carry out hypothesis tests. These non-parametric approaches may give more meaningful results in some situations where bivariate normality does not hold. However the standard versions of these approaches rely on exchangeability.

A stratified analysis is one way to either accommodate a lack of bivariate normality, or to isolate the correlation resulting from one factor while controlling for another. If W represents cluster membership or another factor that it is desirable to control, we can stratify the data based on the value of W, then calculate a correlation coefficient within each stratum. The stratum-level estimates can then be combined to estimate the overall correlation while controlling for W.^[10]

[edit] Calculating a weighted correlation

Suppose observations to be correlated have differing degrees of importance that can be expressed with a weight vector w. To calculate the correlation between vectors x and y with the weight vector w (all of length n),^[11]^[12]

Weighted mean:

$\operatorname{m}(x; w) = {\sum_i w_i x_i \over \sum_i w_i}$

Weighted covariance

$\operatorname{cov}(x,y;w) = {\sum_i w_i (x_i - \operatorname{m}(x; w)) (y_i - \operatorname{m}(y; w)) \over \sum_i w_i }$

Weighted correlation

$\operatorname{corr}(x,y;w) = {\operatorname{cov}(x,y;w) \over \sqrt{\operatorname{cov}(x,x;w) \operatorname{cov}(y,y;w)}}$

[edit] Removing correlation

It is always possible to remove the correlation between random variables with a linear transformation, even if the relationship between the variables is nonlinear. A presentation of this result for population distributions is given by Cox & Hinkley.^[13]

A corresponding result exists for sample correlations, in which the sample correlation is reduced to zero. Suppose a vector of n random variables is sampled m times. Let X be a matrix where $X i, j$ is the jth variable of sample i. Let $Z m, m$ be an m by m square matrix with every element 1. Then D is the data transformed so every random variable has zero mean, and T is the data transformed so all variables have zero mean and zero correlation with all other variables - the moment matrix of T will be the identity matrix. This has to be further divided by the standard deviation to get unit variance. The transformed variables will be uncorrelated, even though they may not be independent.

$D = X -\frac{1}{m} Z_{m,m} X$

$T = D (D^T D)^{-\frac{1}{2}}$

where an exponent of -1/2 represents the matrix square root of the inverse of a matrix. The covariance matrix of T will be the identity matrix. If a new data sample x is a row vector of n elements, then the same transform can be applied to x to get the transformed vectors d and t:

$d = x - \frac{1}{m} Z_{1,m} X$

$t = d (D^T D)^{-\frac{1}{2}}$

[edit] References

↑ ^1.0 ^1.1 J. L. Rodgers and W. A. Nicewander. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59–66, Feb 1988.
↑ Stigler, Stephen M. (1989). "Francis Galton's Account of the Invention of Correlation". Statistical Science 4 (2). http://www.jstor.org/stable/2245329.
↑ Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.)
↑ Fisher, R.A. (1915). "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population". Biometrika 10 (4): 507–521.
↑ Fisher, R.A. (1921). "On the probable error of a coefficient of correlation deduced from a small sample" (PDF). Metron 1 (4): 3–32. http://hdl.handle.net/2440/15169. Retrieved on 2009-03-25.
↑ Gayen, A.K. (1951). "The frequency distribution of the product moment correlation coefficient in random samples of any size draw from non-normal universes". Biometrika 38: 219–247.
↑ ^7.0 ^7.1 Wilcox, Rand R. (2005). Introduction to robust estimation and hypothesis testing. Academic Press.
↑ Devlin, Susan J; Gnanadesikan, R; Kettenring J.R. (1975). "Robust Estimation and Outlier Detection with Correlation Coefficients". Biometrika 62 (3): 531–545. http://www.jstor.org/stable/2335508.
↑ Huber, Peter. J. (2004). Robust Statistics. Wiley.
↑ Multivariable Analysis- A Practical Guide for Clinicians. 2nd Edition. Mitchell H. Katz. University of California, San Francisco. ISBN 9780521549851. ISBN 052154985X DOI: 10.2277/052154985X
↑ http://sci.tech-archive.net/Archive/sci.stat.math/2006-02/msg00171.html
↑ A MATLAB Toolbox for computing Weighted Correlation Coefficients
↑ Cox, D.R., Hinkley, D.V. (1974) Theoretical Statistics, Chapman & Hall (Appendix 3) ISBN 0412124203

[thirteenways-0] 1.0 ^1.1 J. L. Rodgers and W. A. Nicewander. Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1):59–66, Feb 1988.

[1] Stigler, Stephen M. (1989). "Francis Galton's Account of the Invention of Correlation". Statistical Science 4 (2). http://www.jstor.org/stable/2245329.

[Cohen88-2] Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.)

[3] Fisher, R.A. (1915). "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population". Biometrika 10 (4): 507–521.

[4] Fisher, R.A. (1921). "On the probable error of a coefficient of correlation deduced from a small sample" (PDF). Metron 1 (4): 3–32. http://hdl.handle.net/2440/15169. Retrieved on 2009-03-25.

[5] Gayen, A.K. (1951). "The frequency distribution of the product moment correlation coefficient in random samples of any size draw from non-normal universes". Biometrika 38: 219–247.

[wilcox-6] 7.0 ^7.1 Wilcox, Rand R. (2005). Introduction to robust estimation and hypothesis testing. Academic Press.

[7] Devlin, Susan J; Gnanadesikan, R; Kettenring J.R. (1975). "Robust Estimation and Outlier Detection with Correlation Coefficients". Biometrika 62 (3): 531–545. http://www.jstor.org/stable/2335508.

[8] Huber, Peter. J. (2004). Robust Statistics. Wiley.

[9] Multivariable Analysis- A Practical Guide for Clinicians. 2nd Edition. Mitchell H. Katz. University of California, San Francisco. ISBN 9780521549851. ISBN 052154985X DOI: 10.2277/052154985X

[10] ttp://sci.tech-archive.net/Archive/sci.stat.math/2006-02/msg00171.html

[11] A MATLAB Toolbox for computing Weighted Correlation Coefficients

[12] Cox, D.R., Hinkley, D.V. (1974) Theoretical Statistics, Chapman & Hall (Appendix 3) ISBN 0412124203

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

Pearson correlation coefficient

From neurov.is/on

Contents

[edit] Definition

[edit] Mathematical properties

[edit] Interpretation

[edit] Geometric interpretation

[edit] Interpretation of the size of a correlation

[edit] Inference

[edit] Pearson's correlation and least squares regression analysis

[edit] Sensitivity to the data distribution

[edit] Calculating a weighted correlation

[edit] Removing correlation

[edit] References

Views

Personal tools

Navigation

Search

Toolbox