The objective of this course is to introduce the student computational methods required by engineers, mathematicians, physicists and economists to explore complex systems. Mathematical models developed to explore complex systems can be rarely “solvable” algebraically and hence computational methods have been developed. This course introduces such methods that range from techniques for system of linear equations, nonlinear equations, approximation of functions, interpolation, clustering, least square data fitting and classification, differentiation and integration. More emphasis will be put on applied linear algebra topics which are prerequisite for Artificial Intelligence, Machine Learning, and other advanced courses. We will make use of Matlab programming to implement and analyze the methods.
Course Information: Syllabus
Topics | Readings | # of lectures |
---|---|---|
Approximation errors and approximating single variable functions | ||
Floating point number system and error in number representation, review of derivatives, Taylor Series, finding optima of single variable functions | Ch 3, 4 NME | 1 |
Finding roots of single variable functions – Bisection, Secant and Newton-Raphson Method | Ch 5, 6 NME | 1 |
Vectors and Matrices | ||
Vectors - review of vector notation, vector operations, linear and affine multivariable
functions, complex vectors, complexity of vector computations, applications: vector representation of data (e.g., images, documents, timeseries, features), vector representation of linear and affine functions (e.g., regression, Linear (Taylor) approximation of multivariable function functions) |
Ch 1,2 VMLS | 2 |
Norms and distances - Euclidean norm and distances, properties (Cauchy-Schwarz and
triangle inequalities, Pythagorean theorem), statistical measurements of data: average,
rms, standard deviation, and angle between vectors and correlation, covariance;
representation of hyperplanes, application: single variable linear regression, k-means clustering |
Ch 3,4 VMLS | 2 |
Direct Methods for Solving System of Linear Equations | ||
Solving system of linear equations using LU decomposition, application:
Polynomial interpolation and Vandermonde matrix, applications of solving system of linear
equations |
Ch 8 VMLS | 2 |
Matrix Inverses: Left and right inverses, solving system of linear equations using matrix inverses, Gram matrix and Pseudo-inverse | Ch 5, 11 VMLS | 2 |
Orthogonality and Least Square Methods | ||
Basis, orthogonality and inner products: basis and change of basis, Orthogonal basis,
Gram-Schmidt, modified-Gram Schmidt algorithms, QR decomposition of matrices, *Householder
reflections, application: solving system of linear equations using QR factorization, *lower dimensional data representation |
Ch 5, 10, 11 VMLS | 2 |
Linear least-Squares: solution to over-determined systems, normal equation and pseudo
inverse of a matrix, computing pseudo inverse using QR and Cholesky factorization, solving
least squares using matrix-vector derivates, application: data fitting and least-square regression, feature engineering, Least-square classification, regularized least square data fitting, *least square function approximation |
Ch 12-14 VMLS; Ch 17 NME | 3 |
*Interpolation | ||
Interpolation using monomial and Lagrange bases will be discussed in Linear equation lecture. *Interpolation using other basis functions: Newton, Legendre, Chebyshev bases, Hermite interpolation, cubic spline interpolation | Ch 18 NME | 2 |
Numerical Differentiation and Integration | ||
Finite divided difference approximation of derivatives, Trapezoidal rule, Simpson’s rule | Ch 22, 23 NME | 1 |
Problem Condition, Algorithm Stability | Ch 6, 7 (notes) | 2 |
We will learn theory, concepts, and applications on how to extract useful information from huge amounts of data.
Course Information: Syllabus
Week | Session | Topics | Resources | Assignments |
---|---|---|---|---|
Week-1 | Session-1 | Data Matrix, Attributes, Vector Recap, Basic Statistics, Distributions, PDF, CDF | https://dataminingbook.info/resources/ Book: DMML, Chapter 1 and Chapter 2 |
Assignment-1 |
Week-2 | Session-2 | Multivariate Gaussian, Covariance Matrix, Geometry of the multivariate normal, Diagonalization of Covariance Matrix | https://dataminingbook.info/resources/ Book: DMML, Chapter 1 and Chapter 2 |
|
Week-3 | Session-3 | Frequent Itemset Mining, The Market-Basket Model, Mining Association Rules, Finding Frequent Pairs, A-Priori Algorithm, FP Growth, *Eclat algorithm | http://www.mmds.org/#book Book: MMDS, Chapter 6 |
Assignment-2 |
Week-4 | Session-4 | Mining Data Streams, General Stream Processing Model, Sampling from a Data Stream, *Queries over a (long) Sliding Window | http://www.mmds.org/#book Book: MMDS, Chapter 4 |
|
Week-5 | Session-5 | Analysis of Large Graphs: Link Analysis, PageRank, Topic Specific Page rank, *Sim Rank | http://www.mmds.org/#book Book: MMDS, Chapter 5 |
Assignment-3 |
Week-6 | Session-6 | Recommender Systems, Content-based Systems, Collaborative Filtering | http://www.mmds.org/#book/>Book: MMDS, Chapter 9 | |
Week-7 | Session-7 | Recommender Systems, Latent Factor Models, SVD | http://www.mmds.org/#book Book: MMDS, Chapter 9, 11 |
Assignment-4 |
Week-8 | Session-8 | Application of SVD in recommender system, *SVD for dimension reduction | http://www.mmds.org/#book Book: MMDS, Chapter 9, 11 |
|
Week-9 | Session-9 | Analysis of Large Graphs: Community Detection, Betweenness, Modularity, Graph Partitioning, *Graph Cut, Spectral Partitioning | http://www.mmds.org/#book Book: MMDS, Chapter 10 |
Assignment-5 |
Week-10 | Session-10 | Map-Reduce and the New Software Stack | http://www.mmds.org/#book Book: MMDS, Chapter 10 |
Assignment-6 |
Week-11 | Session-11 | *Finding Similar Items: Locality Sensitive Hashing, *Distance Measure, *MinHashing | http://www.mmds.org/#book Book: MMDS, Chapter 3 |
|
Week-12 | Session-12 | TBA | TBA |
We have entered the era of big data. This deluge of data calls for automated methods of data analysis, which is what machine learning or pattern recognition provides. Machine learning as a set of methods that can automatically detect patterns in data, and then use the uncovered patterns to predict future data, or to perform other kinds of decision making under uncertainty (such as planning how to collect more data!). This introductory pattern recognition/machine learning course will give an overview of many popular models and algorithms used in modern machine learning - both statistical and non-statistical. The course will give the student the basic ideas and intuition behind these methods, as well as a more formal understanding of how and why they work. Students will have an opportunity to experiment with machine learning techniques and apply them on real life problems.
Course Information: Syllabus
# of Lectures | Topics |
---|---|
1 | Machine Learning/Pattern Recognition basics: Supervised and Unsupervised Learning - Classification, Clustering and Regression, Parametric and Non-parametric Models, Curse of Dimensionality, Over-fitting, and Model Selection, Performance Measures |
1 | Data: Attribute types, Basic Statistical description of Data, Review of probability theory |
2 | Bayesian Decision Theory: Likelihood Ratio Test, Bayes Risk; Bayes, ML and MAP Criteria, Naive Bayes classifier |
1 | Normal Variables and its Discriminant Analysis |
1 | Parametric Density Estimation: MLE, Bayesian Density Estimation |
1 | Nonparametric Density Estimation: Kernel Density Estimators and Nearest Neighbor Method |
1 | Regression: Linear Regression Analysis and Bayesian Linear Regression |
2 | Decision Trees and Random Forests, Ensemble Methods: Bagging and Boosting |
4 | Feature Selection and Extraction, Dimensionality Reduction : PCA and SVD |
3 | Linear Models for Classification: Fisher's Linear Discriminant, Support Vector Machines |
2 | Introduction to Graphical Models: Bayesian Networks, Exact and Approximate Inference Methods |
2 | *Introduction to Reinforcement Learning: Policy gradient, Q-learning |
Course Information: Syllabus
Weeks | Topics | Lectures | Presentation Topics |
---|---|---|---|
Week-1 | Neural Network Basics, Multilayer Perceptron, Linear Classifiers, Loss calculation, Log likelihood loss, Cross Entropy Loss, Softmax Classifier, Different Activation Functions and their Derivatives | 2 | |
Week-2 | Gradient Descent, Chain Rule for Derivatives, Back Propagation, Update Rule, Implementation of Multilayer Perceptron from Scratch that uses back propagation | 2 | |
Week-3 | Convolutional Neural Network, Filters, Kernels, Convolutional Layer, Max Pool Layer, Activation Function ReLU, Batch Normalization, Implementation of CNN from Scratch | 2 | |
Week-4 | Capacity, Overfitting, Under fitting, Regularization, Weight Decay, Dropout, Batch Normalization, Convolutional AutoEncoder, Semantic Segmentation, Different up-sampling method (Deconvolution, Reverse Maxpool) | 2 | Presentation: Semantic Segmentation Presentation 1. Segnet 2. FCN-8 |
Week-5 | Attention, Where CNN pays attention for classification Concept: Class Activation Map (CAM) |
2 | 1. GradCAM Learn to Pay Attention |
Week-6 | Object Detection, Object localization , Region Proposal, Regional Convolutional Neural Network (R-CNN) , Mask R-CNN | 2 | 1. YOLO 2.Fast R-CNN 3.Faster R-CNN |
Week-7 | Word Embedding, Word2vec, Negative Sampling, Character Level Embedding, Sentence Level Embedding | 2 | 1. Attention all you need 2. BERT |
Week-8 | LSTM/GRU for language model, Neural Machine Translation, LST/GRU + Attention, Image Captioning | 2 | 1. Show, Attend, and Tell |
Week-9 | Self-Attention, Transformer for Neural Machine Translation | 2 | 1. Transformer-XL |
Week-10 | Introduction to Graph Embedding, Node2vec, Graph Convolution Network | 2 | 1. Representation Learning on Graphs: Method and Application |
Week-11 | Graph Neural Network (GNN) style Embedding, Graph Attention Network (GAT) style embedding | 2 | 1. GraphSage |
Week-12 | Advanced Topics Variational Auto Encoder, Generative Adversarial Network, Few/Zero Shot Learning | 2 |
An introduction to the basic principles, techniques, and applications of Artificial Intelligence. Coverage includes perception and learning, searching and logical inference and knowledge base. Methods used in this course will have wide applications in different artificial intelligent systems such as expert system, robotics, computer vision, and natural language processing. Students will have practical experience in designing and implementing components of an intelligent system.
Course Information: Syllabus
Topics | Descriptions | # of lectures |
---|---|---|
Introduction | Intelligent agents: a discussion on what Artificial Intelligence is about and different types of AI agents | 2 |
Search | Optimization on a Discrete state-space - Uninformed and informed search methods – BFS, DFS, IDS, A*, and IDA* search methods | 3 |
Constraint Satisfaction Search | Constraint Satisfaction Problems (CSP), Arc consistency algorithm | 3 |
Local search | Hill Climbing, Simulated Annealing, Genetic algorithms, Swarm intelligence – Particle Swarms, Ant Colony Optimization | 3 |
Logical Reasoning | Propositional logic, Reasoning - Forward and Backward Chaining, *First order Logic and Reasoning | 2 |
Optimization | Review of Linear Algebra and Calculus for Multivalued Functions. Optimization of multivariable functions, Directional Derivatives, Gradient, Hessian, Gradient-based Optimization, Numerical Differentiation | 3 |
Machine Learning I | Supervised learning, Regression, Classification methods - formulation of Linear Regression, Logistic regression, Linear classifiers | 3 |
Machine Learning II | Neural Networks, Backpropagation, Regression and Multiclass Classification, Training of Neural Networks | 3 |