Chapter 2

22 mai 2017 be seen from the second derivative (if it exists). 6. Page 7. DMM summer 2017 ... Calculate the gradient of fA
optimization

The Matrix Cookbook

15 nov. 2012 determinant derivative of inverse matrix
matrixcookbook

b − Ax2. 2. = (b − Ax)T (b − Ax). = bT b − (Ax)T b − bT Ax + xT AT Ax. = bT b − 2bT Ax + xT AT Ax Because mixed second partial derivatives satisfy.
lecture

The Matrix Cookbook

determinant derivative of inverse matrix
Matrix Cookbook

Techniques of Integration

apparent that the function you wish to integrate is a derivative in some EXAMPLE 8.1.2 Evaluate ∫ sin(ax + b) dx assuming that a and b are constants ...
calculus Techniques of Integration

Week 3 Quiz: Differential Calculus: The Derivative and Rules of

Question 2: Find limx→2f(x): f(x) = 1776. (A) +∞. (B) 1770. (C) −∞. (D) Does not exist! (E) None of the above. Answer: (E) The limit of any constant

Assignment 2 — Solutions

If a1b2 = a2b1 show that this equation reduces to the form y′ = g(ax + by). Solution Substituting a2 = λa1 and b2 = λb1 into the equation yields:.
Assignment Solutions

Introduction to Linear Algebra 5th Edition

To see that action I will write b1
linearalgebra

1 mar. 2016
ORF S Lec gh

Order and Degree and Formation of Partial Differential Equations

When a differential equation contains one or more partial derivatives of an (viii) z = ax e' +. 1. 2. Sol. (1) We are given z = (2x + a) (2 y + b).
partial differential equations unit

267621

Lecture 8 Notes

Let (x) =cTx wherecis a constant vector. Then, from (x) =nX j=1c jxj; we have @@x k=nX j=1c j@xj@x k=nX j=1c jjk=ck; and therefore r(x) =c:

Now, let

'(x) =xTBx=nX i=1n X j=1b ijxixj: Then @'@x k=nX i=1n X j=1b ij@(xixj)@x k nX i=1n X j=1b ij(ikxj+xijk) nX i=1n X j=1b ijxjik+nX i=1n X j=1b ijxijk nX j=1b kjxj+nX i=1b ikxi = (Bx)k+nX i=1(BT)kixi = (Bx)k+ (BTx)k:

We conclude that

r'(x) = (B+BT)x: 1

Gradient of the 2-Norm of the Residual Vector

From kxk2=px Tx; and the properties of the transpose, we obtain kbAxk22= (bAx)T(bAx) =bTb(Ax)TbbTAx+xTATAx =bTb2bTAx+xTATAx =bTb2(ATb)Tx+xTATAx: Using the formulas from the previous section, withc=ATbandB=ATA, we have r(kbAxk22) =2ATb+ (ATA+ (ATA)T)x:

However, because

(ATA)T=AT(AT)T=ATA; this simplies to r(kbAxk22) =2ATb+ 2ATAx= 2(ATAxATb):

Positive Deniteness ofATA

LetAbemn, withmnand rank(A) =n. Then, ifx6=0, it follows from the linear independence ofA's columns thatAx6=0. We then have x

TATAx= (Ax)TAx=kAxk22>0;

since the norm of a nonzero vector must be positive. It follows thatATAis not only symmetric, but positive denite as well.

Hessians of Inner Products

The Hessian of the function'(x), denoted byH'(x), is the matrix with entries h ij=@2'@x i@xj:

Because mixed second partial derivatives satisfy

2'@x i@xj=@2'@x j@xi as long as they are continuous, the Hessian is symmetric under these assumptions. In the case of'(x) =xTBx;whose gradient isr'(x) = (B+BT)x, the Hessian isH'(x) = B+BT. It follows from the previously computed gradient ofkbAxk22that its Hessian is

2ATA. Therefore, the Hessian is positive denite, which means that the unique critical point

x, the solution to the normal equationsATAxATb=0, is aminimum.

In general, if the Hessian at a critical point is

2 positive denite, meaning that its eigenvalues are all positive, then the critical point is a local minimum. negative denite, meaning that its eigenvalues are all negative, then the critical point is a local maximum.

Lecture 8 Notes

Let (x) =cTx wherecis a constant vector. Then, from (x) =nX j=1c jxj; we have @@x k=nX j=1c j@xj@x k=nX j=1c jjk=ck; and therefore r(x) =c:

Now, let

'(x) =xTBx=nX i=1n X j=1b ijxixj: Then @'@x k=nX i=1n X j=1b ij@(xixj)@x k nX i=1n X j=1b ij(ikxj+xijk) nX i=1n X j=1b ijxjik+nX i=1n X j=1b ijxijk nX j=1b kjxj+nX i=1b ikxi = (Bx)k+nX i=1(BT)kixi = (Bx)k+ (BTx)k:

We conclude that

r'(x) = (B+BT)x: 1

Gradient of the 2-Norm of the Residual Vector

From kxk2=px Tx; and the properties of the transpose, we obtain kbAxk22= (bAx)T(bAx) =bTb(Ax)TbbTAx+xTATAx =bTb2bTAx+xTATAx =bTb2(ATb)Tx+xTATAx: Using the formulas from the previous section, withc=ATbandB=ATA, we have r(kbAxk22) =2ATb+ (ATA+ (ATA)T)x:

However, because

(ATA)T=AT(AT)T=ATA; this simplies to r(kbAxk22) =2ATb+ 2ATAx= 2(ATAxATb):

Positive Deniteness ofATA

LetAbemn, withmnand rank(A) =n. Then, ifx6=0, it follows from the linear independence ofA's columns thatAx6=0. We then have x

TATAx= (Ax)TAx=kAxk22>0;

since the norm of a nonzero vector must be positive. It follows thatATAis not only symmetric, but positive denite as well.

Hessians of Inner Products

The Hessian of the function'(x), denoted byH'(x), is the matrix with entries h ij=@2'@x i@xj:

Because mixed second partial derivatives satisfy

2'@x i@xj=@2'@x j@xi as long as they are continuous, the Hessian is symmetric under these assumptions. In the case of'(x) =xTBx;whose gradient isr'(x) = (B+BT)x, the Hessian isH'(x) = B+BT. It follows from the previously computed gradient ofkbAxk22that its Hessian is

2ATA. Therefore, the Hessian is positive denite, which means that the unique critical point

x, the solution to the normal equationsATAxATb=0, is aminimum.

In general, if the Hessian at a critical point is

2 positive denite, meaning that its eigenvalues are all positive, then the critical point is a local minimum. negative denite, meaning that its eigenvalues are all negative, then the critical point is a local maximum.