Musings

A random collection

Archive for the ‘statistics’ Category

STAT: Multivariate

Distribution function for random variable \mathbf{X} = (X_1, X_2, \ldots, X_n)

    F_{\mathbf{X}}(x_1, x_2, \ldots, x_n) = P(X_1 \leq x_1, X_2 \leq x_2, \ldots, X_n \leq x_n)
    F_{\mathbf{X}}(x_1,\ldots,x_d,\infty\ldots,\infty) = F_{X_1,\ldots,X_d}(x_1,\ldots,x_d)
    F_{\mathbf{X}}(x_1,\ldots,x_d,\ldots,-\infty) = 0

Density Function

    f(x_1,x_2,\ldots,x_n) \equiv \frac{\partial^n}{\partial x_1 \partial x_2 \ldots \partial x_n} F(x_1,x_2,\ldots,x_n)
    f_{X_1,X_2,\ldots,X_k}(x_1,x_2,\ldots,x_k) = \int_{-\infty}^\infty \cdots \int_{-\infty}^\infty f(x_1,x_2,\ldots,x_n) dx_{k+1}\ldots dx_n

Expectation

    E[g(X_1,\ldots,X_n)] = \int_{-\infty}^{\infty} \ldots \int_{-\infty}^{\infty} g(x_1,\ldots,x_n) f(x_1,\ldots,x_n) dx_1 \ldots dx_n

Independence

  • P(X_1 \in A_1, \ldots, X_n \in A_n) = P(X_1 \in A_1) \ldots P(X_n \in A_n)
  • F(x_1,x_2,\ldots,x_n) = \prod_i F_{X_i}(x_i)
  • f(x_1,x_2,\ldots,x_n) = \prod_i f_{X_i}(x_i)
  • E[\prod_i g_i(X_i)] = \prod_i E[g_i(X_i)]
  • Given independent Xi‘s such that Y=\sum X_i, \quad M_Y[t] = \prod_i M_{X_i}[t]

Multivariate Moment Generating Function

    M_{\mathbf{X}}(\mathbf{t}) = E[e^{\mathbf{t}'\mathbf{X}}] = E[e^{\sum X_i t_i}]

Independent Xi:

    M_{\mathbf{X}}(\mathbf{t}) = \prod M_{X_i}(t_i)

Conditional Density
Let \mathbf{X} = (\mathbf{X_1} : \mathbf{X_2}), \mathbf{X_1} = (x_1,\ldots,x_k), \mathbf{X_2} = (x_{k+1},\ldots,x_n)

    f_{\mathbf{X_1}|\mathbf{X_2}}(\mathbf{x}_1|\mathbf{x}_2) = \frac{f(\mathbf{x}_1,\mathbf{x}_2)}{f_{\mathbf{X}_2}(\mathbf{x}_2)}

Covariance Matrix

  • \boldsymbol{\Sigma}_{\mathbf{X}} = \text{Cov}(\mathbf{X},\mathbf{X}) = E[(\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^T], \quad \text{ where } \boldsymbol{\mu} = E[\mathbf{X}]
  • \boldsymbol{\Sigma}_{\mathbf{X}} = \text{Cov}(\mathbf{X},\mathbf{X}) = E[\mathbf{X}\mathbf{X}^T]-\boldsymbol{\mu}\boldsymbol{\mu}^T
  • \text{Cov}(\mathbf{X},\mathbf{X}) = E[\text{Cov}(\mathbf{X},\mathbf{X})|\mathbf{Y}] + \text{Cov}(E[\mathbf{X}|\mathbf{Y}],E[\mathbf{X}|\mathbf{Y}])
  • Let \mathbf{Y} = \mathbf{a}+\mathbf{B}\mathbf{X}, then \boldsymbol{\mu}_{\mathbf{Y}} = \mathbf{a}+\mathbf{B}\boldsymbol{\mu}_{\mathbf{X}} and \text{Cov}(\mathbf{Y},\mathbf{Y}) = \boldsymbol{\Sigma}_{\mathbf{Y}} = \mathbf{B} \boldsymbol{\Sigma}_{\mathbf{X}} \mathbf{B}^T
  • Non-negative Definite, i.e., \mathbf{b}^T \boldsymbol{\Sigma}_{\mathbf{X}} \mathbf{b} \geq 0, \quad \forall \mathbf{b} \in \mathbb{R}^n
    Let Y = \mathbf{b}^T \mathbf{X} = \sum b_i X_i, then 0 \leq \text{Var}(Y) = \mathbf{b}^T \boldsymbol{\Sigma}_{\mathbf{X}} \mathbf{b}
  • \mathbf{b} = \mathbf{1} = (1,\ldots,1)^T, Y = \mathbf{1}^T \mathbf{X} = \sum X_i then \text{Var}[\sum X_i] = \mathbf{1}^T \boldsymbol{\Sigma}_{\mathbf{X}} \mathbf{1} = \sum \text{Var}[X_i] + 2 \sum \sum_{i<j} \text{Cov}(X_i,X_j)

Standardization of random variable
Since \boldsymbol{\Sigma}_{\mathbf{X}} is non-negative definite, there exists a matrix A such that

    \mathbf{A}\mathbf{A}^T = \boldsymbol{\Sigma}_{\mathbf{X}}

In addition, if it is positive definite, then A is invertible. Now let

    \mathbf{Z} = \mathbf{A}^{-1}(\mathbf{X}-\boldsymbol{\mu}_{\mathbf{X}})

Then

  • \boldsymbol{\mu}_{\mathbf{Z}} = 0
  • \boldsymbol{\Sigma}_{\mathbf{Z}} = \mathbf{I}
  • Z is standardized: a) Zi are uncorrelated, b) E[Zi] = 0; c) Var[Zi] = 1

Another Standardization of random variable
Let us say

    \mathbf{Z} = \mathbf{D}^{-1}(\mathbf{X}-\boldsymbol{\mu}_{\mathbf{X}})
    where \mathbf{D} = \text{diagonal}(D_{ii}) = \text{diagonal}(\sqrt{\Sigma_{ii}}) = \text{diagonal}(\sigma_i)

i.e. Z_i = \frac{X_i-\mu_{X_i}}{\sigma_i}

Here too Z is standardized: a) but Zi are not uncorrelated, b) E[Zi] = 0; c) Var[Zi] = 1

Actually, covariance matrix of Z is same as correlation matrix of X :- \boldsymbol{\rho}_{\mathbf{X}}

  • \boldsymbol{\Sigma}_{\mathbf{Z}} = \mathbf{D}^{-1} \boldsymbol{\Sigma}_{\mathbf{X}}(\mathbf{D}^{-1})^T = \boldsymbol{\rho}_{\mathbf{X}} \neq \text{diagonal}
  • (\boldsymbol{\Sigma}_{\mathbf{Z}})_{ij} = \sum_k \sum_l (D^{-1})_{ik} \sigma_{kl} (D^{-1})_{lj} = \frac{\sigma_{ij}}{\sigma_{ii}\sigma_{jj}} = \rho_{ij}

Change of variable – multivariate
Given:

  1. C be the smallest open set in \mathbb{R}^d so that P(\mathbf{X} \in C) = 1
  2. Let \mathbf{Y} = \mathbf{y}(\mathbf{X}), \quad \mathbf{y} \colon C \to \mathbb{R}^d is one-to-one and continuously differentiable
  3. y has an inverse x. So that X = x(Y)
  4. X has probability density function fX

Then probability density function of Y is given by:

    f_{\mathbf{Y}}(\mathbf{y}) = f_{\mathbf{X}}(\mathbf{x}(\mathbf{y})) \vert \det\left(\frac{\partial \mathbf{x}}{\partial \mathbf{y}}\right) \vert

    Here \left(\frac{\partial \mathbf{x}}{\partial \mathbf{y}}\right) is the Jacobian Matrix defined by
    \left(\frac{\partial \mathbf{x}}{\partial \mathbf{y}}\right)_{ij} = \frac{\partial x_i}{\partial y_j}

Advertisements

Written by curious

August 11, 2010 at 9:37 pm

Posted in statistics

STAT: Bivariate

Distribution function for random variable \mathbf{X} = (X_1, X_2)

    F_{\mathbf{X}}(x_1,x_2) = P(X_1 \leq x_1, X_2 \leq x_2)
    F_{\mathbf{X}}(x_1,\infty) = F_{X_1}(x_1), \quad F_{\mathbf{X}}(\infty,x_2) = F_{X_2}(x_2)
    F_{\mathbf{X}}(x_1,-\infty) = F_{\mathbf{X}}(-\infty,x_2) = 0

For a_1 \leq b_1, a_2 \leq b_2

    P(a_1 < X_1 \leq b_1, a_2 < X_2 \leq b_2) = F(b_1,b_2) + F(a_1,a_2) - F(a_1,b_2) - F(b_1,a_2)

Density Function

    f(x_1,x_2) \equiv \frac{\partial^2}{\partial x_1 \partial x_2} F(x_1,x_2)

    \displaystyle \lim{h \to 0} \frac{P(x_1 < X_1 \leq x_1+h,x_2 < X_2 \leq x_2+h)}{h^2} = f(x_1,x_2)

Marginal Density

    f_{X_1}(x_1) = \int_{-\infty}^{\infty} f(x_1,x_2) dx_2 ; \quad f_{X_2}(x_2) = \int_{-\infty}^{\infty} f(x_1,x_2) dx_1

Expectation

    E[g(X_1,X_2)] = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} g(x_1,x_2) f(x_1,x_2) dx_1 dx_2

Covariance

    \text{Cov}(X_1,X_2) = E[(X_1-\mu_1)(X_2-\mu_2)] = E[X_1 X_2] - \mu_1 \mu_2
    \text{Cov}(a_1+b_1 X_1, a_2+b_2 X_2) = b_1 b_2 \text{Cov}(X_1,X_2)

Variance

    \text{Var}(X_1+X_2) = \text{Var}(X_1) + \text{Var}(X_2) + \text{Cov}(X_1,X_2)

Pearson Correlation

    \rho(X_1, X_2) = \frac{\text{Cov}(X_1,X_2)}{\sqrt{\text{Var}[X_1]\text{Var}[X_2]}}
    \rho(a_1+b_1 X_1, a_2+b_2 X_2) = \text{sign}(b_1 b_2) \rho(X_1,X_2)
    \text{sign}(b) = \begin{cases} 1, & b > 0, \\ 0, & b = 0 \\ -1, & b < 0 \end{cases}

Independence

  • P(X_1 \in A_1, X_2 \in A_2) = P(X_1 \in A_1) P(X_2 \in A_2)
  • F(x_1,x_2) = F_{X_1}(x_1)F_{X_2}(x_2), \quad \forall x_1,x_2
  • f(x_1,x_2) = f_{X_1}(x_1)f_{X_2}(x_2), \quad \forall x_1,x_2
  • f(x_1,x_2)dx_1 dx_2 = f_{X_1}(x_1) dx_1 f_{X_2}(x_2) dx_2
  • E[g_1(X_1)g_2(X_2)] = E[g_1(X_1)] E[g_2(X_2)]
    Show that
    1) given this condition implies independence (P(AB) = P(A)P(B)). Use E[1A] = P(A)
    2) given independence, this equality follows. Just follows from definition of E.

Uncorrelated
X1 and X2 are uncorrelated if Cov(X1, X2) = 0

    E[X_1 X_2] = E[X_1] E[X_2]
  • Independent implies Uncorrelated
  • Uncorrelated does not necessarily imply Independent
    Only for linear functions of the form g(x) = a+bx, E[g(X_1)g(X_2) = E[g(X_1)]E[g(X_2)]
    Independence requires it to hold for all functions not just linear functions.
  • Independent ⊂ Uncorrelated
  • Example: Intra-day stock returns are dependent (not independent) and correlated (not uncorrelated, Cov ≠ 0).
    Inter-day Stock returns (from yesterday and today) are uncorrelated but not independent (still dependent).

Conditional Distribution

    F_{X_1|X_2}(x_1|x_2) = P(X_1 \leq x_1 | X_2 = x_2) = \left(\frac{\partial}{\partial x_2} F(x_1,x_2)\right)/f_{X_2}(x_2)

Conditional Density

    f_{X_1|X_2}(x_1|x_2) = \frac{\partial}{\partial x_1} F_{X_1|X_2}(x_1|x_2) = \frac{f(x_1,x_2)}{f_{X_2}(x_2)}
    \implies f_{X_1|X_2}(x_1|x_2) f_{X_2}(x_2) = f(x_1,x_2)
    f_{X_1}(x_1) = \int_{-\infty}^{\infty} f_{X_1|X_2}(x_1|x_2) f_{X_2}(x_2) dx_2

Conditional Expectation

  • E[g(X_1)|X_2] = \int_{-\infty}^{\infty} g(x_1) f_{X_1|X_2}(x_1|x_2) dx_1
  • E[g_1(X_1)g_2(X_2)|X_2] = g_2(X_2)E[g_1(X_1)|X_2]
    E[g(X_2)|X_2] = g(X_2)
  • E[E[g(X_1)|X_2]] = E[g(X_1)]

If X1 and X2 are independent,

    E[g(X_1)|X_2] = E[g(X_1)]

Conditional Variance

    \text{Var}[X_1|X_2] \equiv E[(X_1-E[X_1|X_2])^2|X_2]
    \text{Var}[X_1] = E[\text{Var}(X_1|X_2)] + \text{Var}(E[X_1|X_2])
    E[\text{Var}(X_1|X_2)] \leq \text{Var}[X_1]

Written by curious

August 9, 2010 at 9:57 am

Posted in statistics

STAT: Pareto Distribution

Written by curious

August 6, 2010 at 1:41 pm

Posted in statistics

STAT: Weibull Distribution

Weibull Distribution PDF

a=5, \gamma=2

Written by curious

August 6, 2010 at 1:32 pm

Posted in statistics

STAT: Inverse Gamma Distribution

Inverse Gamma PDF

Written by curious

August 6, 2010 at 1:15 pm

Posted in statistics

STAT: Log Gamma Distribution

Log Gamma PDF

Written by curious

August 6, 2010 at 12:46 pm

Posted in statistics

STAT: Double Exponential Distribution

Written by curious

August 6, 2010 at 12:20 pm

Posted in statistics