LMSE Estimation and Least Squares

Hello,

Consider the problem:
Given observation Y, estimate X via a linear approach: \hat{X}=HY+b
such that

(1) J1=E[ (X-\hat{X})^T (X-\hat{X}) ] is minimized.
(2) J1 is minimized, and we know that Y=Mx + n
(3) J2=E[ (Y- M \hat{X})^T (Y- M \hat{X}) is minimized

Solution for (1):
The solution of this Linear Mean Square Error (LMSE) error problem is
given by b=m_x+H m_y, where H is solution of K_yy H = K_yx.

[Notation: m_x mean of vector X, K_yx=E[ (y-m_y)^T (y-m_y ] ]

Solution for (2):
Calculation of K_yy and K_yx leads to
H=M K_x (MK_xM^T+K_n)^(-1)

Solution for (3):
Solution for this least squares problem is
H=(M^T M)^(-1) M^T

Question:
- What are the differences and common points of the estimator obtained
in case of the LMSE and the least squares case? How are they related?
- Solution for the weighted least squares problem is given by H=(M^T
S^(-1) M)^(-1) M^T S^(-1) , where for weighting S often the
autocorrelation matrix K_n is taken. Can we motivate the choice of the
weight S by statistical considerations?

Thank you!

Michael

.

Relevant Pages

  • Re: Minimizing the Frobenius norm
    ... The v are linearly related to the T so it is indeed a linear ... Otherwise you have a nonlinear least squares problem and need a corresponding ... or, should the optimal sum of differences not be small, ... port/n2f or port/n2g ...
    (sci.math.num-analysis)
  • Re: Separable nonlinear least squares optimizer with simple bounds on linear and nonlinear variables
    ... solves a non-linear least squares problem, ... imposes simple bounds both on the linear and nonlinear variables, ... exploit the separability of my model. ...
    (sci.math.num-analysis)
  • Re: Linear Equations Similar to 1D Poisson
    ... how to deal with a linear system with matrix ... plus the constraint sum x=1 ... try to solve this harmless least squares problem directly, ... you write the coefficients of the linear equation on top of the matrix. ...
    (sci.math.num-analysis)
  • Re: Matrix / Estimation stuff
    ... > anything to do with the jacobian matrix above? ... linearise non-linear problems. ... linear least squares problem min|b-Ax| is to solve the normal ... the non-linear least squares problem minimises the linear model of the ...
    (sci.math.num-analysis)