Instructor: ChoJui Hsieh Office location: Mathematical Sciences Building (MSB) 4232 Email: chohsieh@ucdavis.edu Office hours: Tuesday 6pm7pm 
TA: Puyudi Yang Email: pydyang@ucdavis.edu Office hours: Wednesday 1:15pm2:15pm 
Date 
Topic 
Readings and links 
Lectures 
Assignments 
Tues 9/25

Mathemetical Background 



Thurs 9/28 
Introduction to Optimization

N&W Chapter 1, 2.1 B&V Chapter 1, 2 


Tues 10/3 
Gradient descent, convergence, autodifferentiation

B&V Chapter 9.19.3 N&W Chapter 2.2 

Thurs 10/5 
Line Search

N&W Chapter 3.13.3 

Newton's method



Conjugate Gradient Method, NewtonCG



Stochastic Gradient Descent, Variance Reduction, Adagrad, Adam



Gradient descent for constrained and nonsmooth problems



QuasiNewton methods, BFGS, LBFGS



Coordinate Descent



Primaldual relationships, KKT condition



Dual gradient ascent, Augmented Lagrangian, ADMM

