Prog Lang and Data Structure

Mach Learning and Data Mining

CSC | 411 | Mach Learning and Data Mining | assessment | 

2013_Final.pdf

2012_Final.pdf

CSC 411: Lecture 04: Logistic Regression Raquel Urtasun & Rich Zemel University of Toronto Sep 23, 2015 Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 1 / 16 Today Key Concepts: I Logistic Regression I Regularization I Cross validation Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 2 / 16 Logistic Regression An alternative: replace the sign ( · ) with the sigmoid or logistic function We assumed a particular functional form: sigmoid applied to a linear function of the data y ( x )= σ ( w T x + w 0 ) where the sigmoid is deﬁned as σ ( z )= 1 1 + exp( - z ) 0 0.5 0 1 The output is a smooth function of the inputs and the weights Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 3 / 16 Logistic Regression We assumed a particular functional form: sigmoid applied to a linear function of the data y ( x )= σ ( w T x + w 0 ) where the sigmoid is deﬁned as σ ( z )= 1 1 + exp( - z ) I One parameter per data dimension (feature) I Features can be discrete or continuous I Output of the model: value y ∈ [0 , 1] I This allows for gradient-based learning of the parameters: smoothed version of the sign ( · ) Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 4 / 16 Shape of the Logistic Function Let’s look at how modifying w changes the function shape 1D example: y = σ ( w 1 x + w 0 ) Demo Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 5 / 16 Probabilistic Interpretation If we have a value between 0 and 1, let’s use it to model the posterior p ( C =0 | x )= σ ( w T x + w 0 ) with σ ( z )= 1 1 + exp( - z ) Substituting we have p ( C =0 | x )= 1 1 + exp ( - w T x - w 0 ) Supposed we have two classes, how can I compute p ( C =1 | x )? Use the marginalization property of probability p ( C =1 | x )+ p ( C =0 | x )=1 Thus ( show matlab ) p ( C =1 | x )=1 - 1 1 + exp ( - w T x - w 0 ) = exp( - w T x - w 0 ) 1 + exp ( - w T x - w 0 ) Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 6 / 16 Conditional likelihood Assume t ∈{ 0 , 1 } , we can write the probability distribution of each of our training points p ( t (1) , ··· , t ( N ) | x (1) , ··· x ( N ) ) Assuming that the training examples are sampled IID : independent and identically distributed p ( t (1) , ··· , t ( N ) | x (1) , ··· x ( N ) )= N Y i =1 p ( t ( i ) | x ( i ) ) We can write each probability as p ( t ( i ) | x ( i ) ) = p ( C =1 | x ( i ) ) t ( i ) p ( C =0 | x ( i ) ) 1 - t ( i ) =  1 - p ( C =0 | x ( i ) )  t ( i ) p ( C =0 | x ( i ) ) 1 - t ( i ) We might want to learn the model, by maximizing the conditional likelihood max w N Y i =1 p ( t ( i ) | x ( i ) ) Convert this into a minimization so that we can write the loss function Urtasun & Zemel (UofT) CSC 411: 04-Prob Classif Sep 23, 2015 7 / 16

CSC | 411 | Mach Learning and Data Mining | lecture notes | CSC 411: Lecture 04: Logistic 

Support Vector Machines CSC 411 Tutorial November 10, 2015 Tutor: Renjie Liao Many thanks to Jake Snell, Yujia Li and Kevin Swersky for much of the following material. Brief Review of SVMS Out[40]: Click here to toggle on/off the raw code. Geometric Intuition Out[48]: Margin Derivation Out[57]: Margin Derivation Compute the distance of an arbitrary point in the (+) class to the separating hyperplane. If we let denote the class of , then the distance becomes We can set for the point closest to the decision boundary, leading to the problem: SVM Problem But scaling and doesn't change . or equivalently: Non-linear SVMs For a linear SVM, . We can just as well work in an alternate feature space: . Out[29]: http://i.imgur.com/WuxyO.png Non-linear SVMs Out[31]: SVM with polynomial kernel visualization http://www.youtube.com/watch?v=3liCbRZPrZA SVMs vs Logistic Regression Logistic Regression Out[32]: [<matplotlib.lines.Line2D at 0x109413090>] Logistic Regression Assign probability to each outcome Train to maximize likelihood Linear decision boundary SVMs Out[33]: SVMs Enforce a margin of separation Train to find the maximum margin Linear decision boundary Comparison Logistic regression wants to maximize the probability of the data. The greater the distance from each point to the decision boundary, the better. SVMs want to maximize the distance from the closest points to the decision boundary. Doesn't care about points that aren't support vectors. A Different Take Consider an alternate form of the logistic regression decision function: A Different Take Suppose we don't actually care about the probabilities. All we want to do is make the right decision. We can put a constraint on the likelihood ratio, for some constant : A Different Take Take the log of both sides: Recalling that and : But is arbitrary, so set it s.t. : A Different Take So now we have . But this may not have a unique solution, so put a quadratic penalty on the weights to make the solution unique: By asking logistic regression to make the right decisions instead of maximizing the probability of the data, we derived an SVM. Likelihood Ratio The likelihood ratio drives this derivation: Different classifiers assign different costs to . LR Cost Choose Out[34]: <matplotlib.text.Text at 0x109348810> LR Cost Minimizing is the same as minimizing the negative log-likelihood objective for logistic regression!

Courses in CSC

Popular Documents from University of Toronto Scarborough

University of Toronto Scarborough

University of Toronto Scarborough

University of Toronto Scarborough

University of Toronto Scarborough

University of Toronto Scarborough