Chapter 2
22 mai 2017 be seen from the second derivative (if it exists). 6. Page 7. DMM summer 2017 ... Calculate the gradient of fA
optimization
The Matrix Cookbook
15 nov. 2012 determinant derivative of inverse matrix
matrixcookbook
Gradients of Inner Products
b − Ax2. 2. = (b − Ax)T (b − Ax). = bT b − (Ax)T b − bT Ax + xT AT Ax. = bT b − 2bT Ax + xT AT Ax Because mixed second partial derivatives satisfy.
lecture
The Matrix Cookbook
determinant derivative of inverse matrix
Matrix Cookbook
Techniques of Integration
apparent that the function you wish to integrate is a derivative in some EXAMPLE 8.1.2 Evaluate ∫ sin(ax + b) dx assuming that a and b are constants ...
calculus Techniques of Integration
Week 3 Quiz: Differential Calculus: The Derivative and Rules of
Question 2: Find limx→2f(x): f(x) = 1776. (A) +∞. (B) 1770. (C) −∞. (D) Does not exist! (E) None of the above. Answer: (E) The limit of any constant
week answers
Assignment 2 — Solutions
If a1b2 = a2b1 show that this equation reduces to the form y′ = g(ax + by). Solution Substituting a2 = λa1 and b2 = λb1 into the equation yields:.
Assignment Solutions
Introduction to Linear Algebra 5th Edition
To see that action I will write b1
linearalgebra
1 Theory of convex functions
1 mar. 2016
ORF S Lec gh
Order and Degree and Formation of Partial Differential Equations
When a differential equation contains one or more partial derivatives of an (viii) z = ax e' +. 1. 2. Sol. (1) We are given z = (2x + a) (2 y + b).
partial differential equations unit
The Matrix Cookbook
[ http://matrixcookbook.com ]Kaare Brandt Petersen
Michael Syskind Pedersen
Version: September 5, 2007
What is this?These pages are a collection of facts (identities, approxima- tions, inequalities, relations, ...) about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference . Disclaimer:The identities, approximations and relations presented here were obviously not invented but collected, borrowed and copied from a large amount of sources. These sources include similar but shorter notes found on the internet and appendices in books - see the references for a full list. Errors:Very likely there are errors, typos, and mistakes for which we apolo- gize and would be grateful to receive corrections at cookbook@2302.dk. Its ongoing:The project of keeping a large repository of relations involving matrices is naturally ongoing and the version will be apparent from the date in the header. Suggestions:Your suggestion for additional content or elaboration of some topics is most welcome at cookbook@2302.dk. Keywords:Matrix algebra, matrix relations, matrix identities, derivative of determinant, derivative of inverse matrix, differentiate a matrix. Acknowledgements:We would like to thank the following for contributions and suggestions: Bill Baxter, Christian Rishøj, Douglas L. Theobald, Esben Hoegh-Rasmussen, Jan Larsen, Korbinian Strimmer, Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, Loic Thibaut, Ole Winther, Stephan Hattinger, and Vasile Sima. We would also like thank The Oticon Foundation for funding our PhD studies. 1CONTENTS CONTENTS
Contents
1 Basics 5
1.1 Trace and Determinants . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 The Special Case 2x2 . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Derivatives 7
2.1 Derivatives of a Determinant . . . . . . . . . . . . . . . . . . . . 7
2.2 Derivatives of an Inverse . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Derivatives of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Derivatives of Matrices, Vectors and Scalar Forms . . . . . . . . 9
2.5 Derivatives of Traces . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 Derivatives of vector norms . . . . . . . . . . . . . . . . . . . . . 13
2.7 Derivatives of matrix norms . . . . . . . . . . . . . . . . . . . . . 13
2.8 Derivatives of Structured Matrices . . . . . . . . . . . . . . . . . 13
3 Inverses 16
3.1 Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Exact Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Implication on Inverses . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Generalized Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Pseudo Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Complex Matrices 22
4.1 Complex Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 22
5 Decompositions 25
5.1 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . 25
5.2 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 25
5.3 Triangular Decomposition . . . . . . . . . . . . . . . . . . . . . . 27
6 Statistics and Probability 28
6.1 Definition of Moments . . . . . . . . . . . . . . . . . . . . . . . . 28
6.2 Expectation of Linear Combinations . . . . . . . . . . . . . . . . 29
6.3 Weighted Scalar Variable . . . . . . . . . . . . . . . . . . . . . . 30
7 Multivariate Distributions 31
7.1 Student"s t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
7.2 Cauchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.3 Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.4 Multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.5 Dirichlet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.6 Normal-Inverse Gamma . . . . . . . . . . . . . . . . . . . . . . . 32
7.7 Wishart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.8 Inverse Wishart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Petersen & Pedersen, The Matrix Cookbook, Version: September 5, 2007,Page 2
CONTENTS CONTENTS
8 Gaussians 34
8.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
8.2 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
8.3 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
8.4 Mixture of Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . 39
9 Special Matrices 40
9.1 Orthogonal, Ortho-symmetric, and Ortho-skew . . . . . . . . . . 40
9.2 Units, Permutation and Shift . . . . . . . . . . . . . . . . . . . . 40
9.3 The Singleentry Matrix . . . . . . . . . . . . . . . . . . . . . . . 41
9.4 Symmetric and Antisymmetric . . . . . . . . . . . . . . . . . . . 43
9.5 Orthogonal matrices . . . . . . . . . . . . . . . . . . . . . . . . . 44
9.6 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . . . . 45
9.7 Toeplitz Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
9.8 The DFT Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
9.9 Positive Definite and Semi-definite Matrices . . . . . . . . . . . . 47
9.10 Block matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
10 Functions and Operators 51
10.1 Functions and Series . . . . . . . . . . . . . . . . . . . . . . . . . 51
10.2 Kronecker and Vec Operator . . . . . . . . . . . . . . . . . . . . 52
10.3 Solutions to Systems of Equations . . . . . . . . . . . . . . . . . 53
10.4 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
10.5 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
10.6 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
10.7 Integral Involving Dirac Delta Functions . . . . . . . . . . . . . . 58
10.8 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
A One-dimensional Results 59
A.1 Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 A.2 One Dimensional Mixture of Gaussians . . . . . . . . . . . . . . . 60B Proofs and Details 62
B.1 Misc Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Petersen & Pedersen, The Matrix Cookbook, Version: September 5, 2007,Page 3
CONTENTS CONTENTS
Notation and Nomenclature
AMatrix
A ijMatrix indexed for some purpose A iMatrix indexed for some purpose A ijMatrix indexed for some purpose A nMatrix indexed for some purposeorThe n.th power of a square matrix
A -1The inverse matrix of the matrixA A +The pseudo inverse matrix of the matrixA(see Sec. 3.6) A1/2The square root of a matrix (if unique), not elementwise
(A)ijThe (i,j).th entry of the matrixA A ijThe (i,j).th entry of the matrixA [A]ijTheij-submatrix, i.e.Awith i.th row and j.th column deleted aVector a iVector indexed for some purpose a iThe i.th element of the vectora aScalar ?zReal part of a scalar ?zReal part of a vector ?ZReal part of a matrix ?zImaginary part of a scalar ?zImaginary part of a vector ?ZImaginary part of a matrix det(A) Determinant ofATr(A) Trace of the matrixA
diag(A) Diagonal matrix of the matrixA, i.e. (diag(A))ij=δijAij vec(A) The vector-version of the matrixA(see Sec. 10.2.2) sup Supremum of a set ||A||Matrix norm (subscript if any denotes what norm) ATTransposed matrix
A ?Complex conjugated matrix A HTransposed and complex conjugated matrix (Hermitian)A◦BHadamard (elementwise) product
A?BKronecker product
0The null matrix. Zero in all entries.
The Matrix Cookbook
[ http://matrixcookbook.com ]Kaare Brandt Petersen
Michael Syskind Pedersen
Version: September 5, 2007
What is this?These pages are a collection of facts (identities, approxima- tions, inequalities, relations, ...) about matrices and matters relating to them. It is collected in this form for the convenience of anyone who wants a quick desktop reference . Disclaimer:The identities, approximations and relations presented here were obviously not invented but collected, borrowed and copied from a large amount of sources. These sources include similar but shorter notes found on the internet and appendices in books - see the references for a full list. Errors:Very likely there are errors, typos, and mistakes for which we apolo- gize and would be grateful to receive corrections at cookbook@2302.dk. Its ongoing:The project of keeping a large repository of relations involving matrices is naturally ongoing and the version will be apparent from the date in the header. Suggestions:Your suggestion for additional content or elaboration of some topics is most welcome at cookbook@2302.dk. Keywords:Matrix algebra, matrix relations, matrix identities, derivative of determinant, derivative of inverse matrix, differentiate a matrix. Acknowledgements:We would like to thank the following for contributions and suggestions: Bill Baxter, Christian Rishøj, Douglas L. Theobald, Esben Hoegh-Rasmussen, Jan Larsen, Korbinian Strimmer, Lars Christiansen, Lars Kai Hansen, Leland Wilkinson, Liguo He, Loic Thibaut, Ole Winther, Stephan Hattinger, and Vasile Sima. We would also like thank The Oticon Foundation for funding our PhD studies. 1CONTENTS CONTENTS
Contents
1 Basics 5
1.1 Trace and Determinants . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 The Special Case 2x2 . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Derivatives 7
2.1 Derivatives of a Determinant . . . . . . . . . . . . . . . . . . . . 7
2.2 Derivatives of an Inverse . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Derivatives of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . 9
2.4 Derivatives of Matrices, Vectors and Scalar Forms . . . . . . . . 9
2.5 Derivatives of Traces . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6 Derivatives of vector norms . . . . . . . . . . . . . . . . . . . . . 13
2.7 Derivatives of matrix norms . . . . . . . . . . . . . . . . . . . . . 13
2.8 Derivatives of Structured Matrices . . . . . . . . . . . . . . . . . 13
3 Inverses 16
3.1 Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.2 Exact Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3 Implication on Inverses . . . . . . . . . . . . . . . . . . . . . . . . 19
3.4 Approximations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.5 Generalized Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Pseudo Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4 Complex Matrices 22
4.1 Complex Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 22
5 Decompositions 25
5.1 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . 25
5.2 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . 25
5.3 Triangular Decomposition . . . . . . . . . . . . . . . . . . . . . . 27
6 Statistics and Probability 28
6.1 Definition of Moments . . . . . . . . . . . . . . . . . . . . . . . . 28
6.2 Expectation of Linear Combinations . . . . . . . . . . . . . . . . 29
6.3 Weighted Scalar Variable . . . . . . . . . . . . . . . . . . . . . . 30
7 Multivariate Distributions 31
7.1 Student"s t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
7.2 Cauchy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.3 Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.4 Multinomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.5 Dirichlet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.6 Normal-Inverse Gamma . . . . . . . . . . . . . . . . . . . . . . . 32
7.7 Wishart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
7.8 Inverse Wishart . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33Petersen & Pedersen, The Matrix Cookbook, Version: September 5, 2007,Page 2
CONTENTS CONTENTS
8 Gaussians 34
8.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
8.2 Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
8.3 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
8.4 Mixture of Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . 39
9 Special Matrices 40
9.1 Orthogonal, Ortho-symmetric, and Ortho-skew . . . . . . . . . . 40
9.2 Units, Permutation and Shift . . . . . . . . . . . . . . . . . . . . 40
9.3 The Singleentry Matrix . . . . . . . . . . . . . . . . . . . . . . . 41
9.4 Symmetric and Antisymmetric . . . . . . . . . . . . . . . . . . . 43
9.5 Orthogonal matrices . . . . . . . . . . . . . . . . . . . . . . . . . 44
9.6 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . . . . 45
9.7 Toeplitz Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
9.8 The DFT Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
9.9 Positive Definite and Semi-definite Matrices . . . . . . . . . . . . 47
9.10 Block matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
10 Functions and Operators 51
10.1 Functions and Series . . . . . . . . . . . . . . . . . . . . . . . . . 51
10.2 Kronecker and Vec Operator . . . . . . . . . . . . . . . . . . . . 52
10.3 Solutions to Systems of Equations . . . . . . . . . . . . . . . . . 53
10.4 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
10.5 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
10.6 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
10.7 Integral Involving Dirac Delta Functions . . . . . . . . . . . . . . 58
10.8 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
A One-dimensional Results 59
A.1 Gaussian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 A.2 One Dimensional Mixture of Gaussians . . . . . . . . . . . . . . . 60B Proofs and Details 62
B.1 Misc Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Petersen & Pedersen, The Matrix Cookbook, Version: September 5, 2007,Page 3