## Chapter 2

22 mai 2017 be seen from the second derivative (if it exists). 6. Page 7. DMM summer 2017 ... Calculate the gradient of fA
optimization

## The Matrix Cookbook

15 nov. 2012 determinant derivative of inverse matrix
matrixcookbook

b − Ax2. 2. = (b − Ax)T (b − Ax). = bT b − (Ax)T b − bT Ax + xT AT Ax. = bT b − 2bT Ax + xT AT Ax Because mixed second partial derivatives satisfy.
lecture

## The Matrix Cookbook

determinant derivative of inverse matrix
Matrix Cookbook

## Techniques of Integration

apparent that the function you wish to integrate is a derivative in some EXAMPLE 8.1.2 Evaluate ∫ sin(ax + b) dx assuming that a and b are constants ...
calculus Techniques of Integration

## Week 3 Quiz: Differential Calculus: The Derivative and Rules of

Question 2: Find limx→2f(x): f(x) = 1776. (A) +∞. (B) 1770. (C) −∞. (D) Does not exist! (E) None of the above. Answer: (E) The limit of any constant

## Assignment 2 — Solutions

If a1b2 = a2b1 show that this equation reduces to the form y′ = g(ax + by). Solution Substituting a2 = λa1 and b2 = λb1 into the equation yields:.
Assignment Solutions

## Introduction to Linear Algebra 5th Edition

To see that action I will write b1
linearalgebra

1 mar. 2016
ORF S Lec gh

## Order and Degree and Formation of Partial Differential Equations

When a differential equation contains one or more partial derivatives of an (viii) z = ax e' +. 1. 2. Sol. (1) We are given z = (2x + a) (2 y + b).
partial differential equations unit

146443 ### Lecture 8 Notes

Let (x) =cTx wherecis a constant vector. Then, from (x) =nX j=1c jxj; we have @@x k=nX j=1c j@xj@x k=nX j=1c jjk=ck; and therefore r(x) =c:

### Now, let

'(x) =xTBx=nX i=1n X j=1b ijxixj: Then @'@x k=nX i=1n X j=1b ij@(xixj)@x k nX i=1n X j=1b ij(ikxj+xijk) nX i=1n X j=1b ijxjik+nX i=1n X j=1b ijxijk nX j=1b kjxj+nX i=1b ikxi = (Bx)k+nX i=1(BT)kixi = (Bx)k+ (BTx)k:

### We conclude that

r'(x) = (B+BT)x: 1

### Gradient of the 2-Norm of the Residual Vector

From kxk2=px Tx; and the properties of the transpose, we obtain kbAxk22= (bAx)T(bAx) =bTb(Ax)TbbTAx+xTATAx =bTb2bTAx+xTATAx =bTb2(ATb)Tx+xTATAx: Using the formulas from the previous section, withc=ATbandB=ATA, we have r(kbAxk22) =2ATb+ (ATA+ (ATA)T)x:

### However, because

(ATA)T=AT(AT)T=ATA; this simplies to r(kbAxk22) =2ATb+ 2ATAx= 2(ATAxATb):

### Positive Deniteness ofATA

LetAbemn, withmnand rank(A) =n. Then, ifx6=0, it follows from the linear independence ofA's columns thatAx6=0. We then have x

### TATAx= (Ax)TAx=kAxk22>0;

since the norm of a nonzero vector must be positive. It follows thatATAis not only symmetric, but positive denite as well.

### Hessians of Inner Products

The Hessian of the function'(x), denoted byH'(x), is the matrix with entries h ij=@2'@x i@xj:

### Because mixed second partial derivatives satisfy

2'@x i@xj=@2'@x j@xi as long as they are continuous, the Hessian is symmetric under these assumptions. In the case of'(x) =xTBx;whose gradient isr'(x) = (B+BT)x, the Hessian isH'(x) = B+BT. It follows from the previously computed gradient ofkbAxk22that its Hessian is

#### 2ATA. Therefore, the Hessian is positive denite, which means that the unique critical point

x, the solution to the normal equationsATAxATb=0, is aminimum.

### In general, if the Hessian at a critical point is

2 positive denite, meaning that its eigenvalues are all positive, then the critical point is a local minimum. negative denite, meaning that its eigenvalues are all negative, then the critical point is a local maximum.

### Lecture 8 Notes

Let (x) =cTx wherecis a constant vector. Then, from (x) =nX j=1c jxj; we have @@x k=nX j=1c j@xj@x k=nX j=1c jjk=ck; and therefore r(x) =c:

### Now, let

'(x) =xTBx=nX i=1n X j=1b ijxixj: Then @'@x k=nX i=1n X j=1b ij@(xixj)@x k nX i=1n X j=1b ij(ikxj+xijk) nX i=1n X j=1b ijxjik+nX i=1n X j=1b ijxijk nX j=1b kjxj+nX i=1b ikxi = (Bx)k+nX i=1(BT)kixi = (Bx)k+ (BTx)k:

### We conclude that

r'(x) = (B+BT)x: 1

### Gradient of the 2-Norm of the Residual Vector

From kxk2=px Tx; and the properties of the transpose, we obtain kbAxk22= (bAx)T(bAx) =bTb(Ax)TbbTAx+xTATAx =bTb2bTAx+xTATAx =bTb2(ATb)Tx+xTATAx: Using the formulas from the previous section, withc=ATbandB=ATA, we have r(kbAxk22) =2ATb+ (ATA+ (ATA)T)x:

### However, because

(ATA)T=AT(AT)T=ATA; this simplies to r(kbAxk22) =2ATb+ 2ATAx= 2(ATAxATb):

### Positive Deniteness ofATA

LetAbemn, withmnand rank(A) =n. Then, ifx6=0, it follows from the linear independence ofA's columns thatAx6=0. We then have x

### TATAx= (Ax)TAx=kAxk22>0;

since the norm of a nonzero vector must be positive. It follows thatATAis not only symmetric, but positive denite as well.

### Hessians of Inner Products

The Hessian of the function'(x), denoted byH'(x), is the matrix with entries h ij=@2'@x i@xj:

### Because mixed second partial derivatives satisfy

2'@x i@xj=@2'@x j@xi as long as they are continuous, the Hessian is symmetric under these assumptions. In the case of'(x) =xTBx;whose gradient isr'(x) = (B+BT)x, the Hessian isH'(x) = B+BT. It follows from the previously computed gradient ofkbAxk22that its Hessian is

#### 2ATA. Therefore, the Hessian is positive denite, which means that the unique critical point

x, the solution to the normal equationsATAxATb=0, is aminimum.

### In general, if the Hessian at a critical point is

2 positive denite, meaning that its eigenvalues are all positive, then the critical point is a local minimum. negative denite, meaning that its eigenvalues are all negative, then the critical point is a local maximum.