Fall 2017: CS/ECE/ME 532 — Matrix Methods in Machine Learning

Professor Rebecca Willett

willett@discovery.wisc.edu
3537 Engineering Hall

Class logistics:

Class location: 1610 Engineering Hall
Class time: 11-12:15pm Mondays and Wednesdays

Course Topics:

This course is an introduction to machine learning that focuses on matrix methods
and features real-world applications ranging from classi cation and clustering to denoising and data analysis. Mathematical topics covered include: linear equations, regression, regularization, the singular value decomposition, and iterative algorithms. Machine learning topics include: the lasso, support vector machines, kernel methods, clustering, dictionary learning, neural networks, and deep learning. Students are expected to have taken a course in calculus and have exposure to numerical computing (e.g. Matlab, Python, Julia, R). Appropriate for graduate students or advanced undergraduates.

Prerequisites:

(MATH 222 and (ECE 203 or CS 200, 300, 302)) or (graduate or professional standing)

Textbook:

Matrix Methods in Data Mining and Pattern Recognition by Lars Elden. Textbook is freely available for anybody on the UW-Madison network: http://epubs.siam.org/doi/book/10.1137/1.9780898718867. The textbook will be supplemented with additional notes and readings.

Tentative schedule:

The course will follow a standard 150 minutes/week lecture format (two 75-minute lectures per week). The course is divided in three roughly equal parts:

  1. Least Squares: matrix and block-matrix products, norms, linear independence, least squares regression, vector derivatives and PSD matrices, subspaces and orthogonality, Gram-Schmidt, classification using least-squares, cross-validation, overtting. Midterm 1.
  2. Singular Value Decomposition: PCA, low-rank approximation, the pseudoinverse, multi-objective trade-offs, pareto curves, alternate norm choices, regularization, convexity, support vector machines, simple iterative methods, soft thresholding. Midterm 2.
  3. Machine Learning: stochastic gradient descent, max-margin and kernalized SVM, neural networks, deep learning, perceptron, convolutional networks, image identification, unsupervised learning, clustering, dictionary learning, K-means, spectral clustering. Final Project.

Learning Outcomes:

This applies to both graduate and undergraduate students enrolled in the class. Upon successful completion of this course, students will: 

  • Understand the linear algebraic and analytic concepts underlying many modern machine learning models. Includes both mathematical rigor as well as geometrical intuition. 
  • Formulate a wide variety of machine learning problems as optimization models and solve them numerically. Understand practical implications of norm choice, regularization, and convexity. 
  • Investigate an applied machine topic not explicitly covered in class and produce a research project that explains, analyzes, and discusses the topic.

Evaluation:

Graduate and undergraduate students will be expected to perform at the graduate level and will be evaluated equally. All students will be evaluated by regular homework assignments, exams, and a final project. The fi nal grade will be allocated to the di fferent components as follows: 

  • Homework: 20%. There are roughly weekly homework assignments (about 10 total). Homework problems include both mathematical derivations and proofs as well as more applied problems that involve writing code and working with real or synthetic data sets. 
  • Exams: 40%. Two midterm exams (20% each), to conclude Parts I and II. No fi nal exam. 
  • Final project: 40%. Students will work in groups (up to 3 students per group) to investigate a machine learning problem or technique using tools learned in class.

Letter grades will be assigned using the following hard cutoff s:

  • A: 93% or higher
  • AB: 87% or higher
  • B: 80% or higher
  • BC: 70% or higher
  • C: 60% or higher
  • D: 50% or higher
  • F: less than 50%

We reserve the right to curve the grades, but only in a fashion that would improve the grade earned by the stated rubric.

Academic integrity:

Students are strongly encouraged to work together on homework assignments, but each student must submit his or her own writeup. Plagiarism of material written by classmates, book or article authors, or web posters is prohibited. Students must work independently on exams. Academic integrity will be strictly enforced. http://students.wisc.edu/doso/acadintegrity.html