Machine Learning 101

" />

# Machine Learning 101
## Model Assessment in R
### Sarah Romanes   &nbsp;@sarah_romanes
### 17-Oct-2018 &nbsp;bit.ly/rladies-sydney-ML-2

---

]
.column.slide-in-right[.content.vmiddle[
.sliderbox.shade_main.pad1[
.font5[Welcome!]
]
]]

---

.column.bg-main1[.content.vmiddle.center[

# Overview

### This two part R-Ladies workshop is designed to give you a small taster into the large field that is known as .orange[**Machine Learning**]! Last week, we covered supervised learning techniques, and this week we will cover model assessment.

### You can always visit last weeks slides at .orange[bit.ly/r-ladies-ML-1] with the corresponding repo [here](https://github.com/sarahromanes/r-ladies-ML-1).

]]
.column.bg-main3[.content.vmiddle.center[
<img src="images/machinewar.png", width="70%">

##### [SMBC](https://www.smbc-comics.com/comic/rise-of-the-machines)

]]

---

# .purple[Recap - What *is* Machine Learning?]

### Machine learning is concerned with finding functions `$Y=f(X)+\epsilon$` that best **predict** outputs (responses), given data inputs (predictors).

</center>

### Mathematically, Machine Learning problems are simply *optimisation* problems, in which we will use .purple[] to help us solve!

---

]
.column.slide-in-right[.content.vmiddle[
.sliderbox.shade_main.pad1[
.font5[Performance Metrics]
]
]]

---

.column.bg-main1[.content[

# Regression - revisited

### Recall the simulated dataset `Income`, which looked at the relationship between `Education` (years) and `Income` (thousands).

```r
data <- read.csv("data/Income.csv")
head(data)
```

```
##   Education   Income
## 1  10.00000 26.65884
## 2  10.40134 27.30644
## 3  10.84281 22.13241
## 4  11.24415 21.16984
## 5  11.64548 15.19263
## 6  12.08696 26.39895
```

### How can we assess how well a regression model fits the data?

]]

]]

---

.column.bg-main1[.content[

# Is a straight line the best fit?

## Last week we considered fitting a .orange[Linear] model to this data, resulting in the following line...

#### (Revise how this was fit [here](https://sarahromanes.github.io/r-ladies-ML-1/#9).)

]]

]]

---

.column.bg-main1[.content[

# Is a straight line the best fit?

## ... however there are many types of curves we could have fit to this dataset!

]]

]]

---
class: split-two white

.column.bg-main1[.content[

# Is a straight line the best fit?

## ... however there are many types of curves we could have fit to this dataset!

## Consider now the .orange[*polynomial*] curve of degree 17. Does it fit better than our straight line?

]]

]]

---

.column.bg-main1[.content[

# Regression metrics: the RMSE

# The RMSE is the .orange[Root Mean Squared Error], which is a metric of how far away our fitted line lies away from the observations (on average).

]]
.column[.content.vmiddle.center[
<img src="images/rmse.png", width="80%">
]]

---

.column.bg-main1[.content[

# Regression metrics: the RMSE

```r
fit.lm <- lm(data=data, Income~Education)
fit.poly <- lm(data=data, Income~poly(Education, 17))

fitted.vals.lm <- predict(fit.lm, 
 data.frame(Education=data$Education))
fitted.vals.poly <- predict(fit.poly, 
 data.frame(Education=data$Education))
```

#### RMSE - Linear Model

```r
sqrt(mean((data$Income-fitted.vals.lm)^2))
```

```
## [1] 5.461576
```
#### RMSE - Polynomial Model

```r
sqrt(mean((data$Income-fitted.vals.poly)^2)) 
```

```
## [1] 2.803986
```

]]
.column.bg-main3[.content.vmiddle.center[

## We can find the RMSE of the two models proposed before.

## .orange[The polynomial fit has a lower RMSE. Does this mean we should use it?]

]]

---
# .purple[Bias and Variance]

## In order to minimise test error on new data points, statistical theory (out of the scope of this workshop) tells us that we need to select a function that achieves *low variance* and *low bias*.

## - .pink[**Variance**] refers to the amount by which our predictions would change if we estimated using a different training set. The more flexible the model, the higher the variance.

## - .pink[**Bias**] refers to the error that introduced by the approximation we are making with our model (represent complicated data by a simple model). The more simple the model, the higher the bias.

---

# High variance (over fitted) models can be problematic...

---

# ... as well as high bias (under fitted) models!

---

# .purple[The Bias-Variance Tradeoff]

## As you may have guessed, there is a trade off between increasing variance (flexibility) and decreasing bias (simplicity) and vice versa.

</center>

### This is known as the .pink[Bias-Variance Tradeoff], and will be revisited in the next section!

---
class: split-two white

.column.bg-main1[.content[

# The middle ground!

## For this dataset, we might consider the `loess` model as a nice middle ground between bias and variance.

```r
*fit.loess <- loess(data=data, Income~Education)
fitted.vals.loess <- predict(fit.loess,
 data.frame(
 Education=data$Education))
rmse.loess <- sqrt(mean((data$Income-fitted.vals.loess)^2)) 
rmse.loess
```

```
## [1] 3.806408
```

]]

]]

---

.column.bg-main1[.content[

# The classification setting

### Recall the dataset `Exam`, where two exam scores are given for each student, and a Label represents whether they passed or failed the course.

```r
data<- read.csv("data/Exam.csv", header=T)
head(data,4)
```

```
##      Exam1    Exam2 Label
## 1 34.62366 78.02469     0
## 2 30.28671 43.89500     0
## 3 35.84741 72.90220     0
## 4 60.18260 86.30855     1
```

### How do we assess model accuracy for a classifier?
]]

.column[.content.vmiddle.center[
<img src="index_files/figure-html/unnamed-chunk-11-1.png" width="504" />
]]

---
class: split-two white

.column.bg-main1[.content[

# Classification metrics

# The .orange[Resubstitution Error Rate] is a measure of the proportion of data points we predict correctly when we try to predict all the points we used to fit the model.

]]
.column[.content.vmiddle.center[
<img src="images/rser.png", width="80%">
]]

---

.column.bg-main1[.content[

# The Resubstitution Error Rate

```r
*fit.glm <- glm(data=data,
* Label ~ .,
* family=binomial(link="logit"))
```
 
]]
.column.bg-main3[.content.vmiddle.center[
# First, we fit a `glm` against all predictors like we did last week.
]]

---

.column.bg-main1[.content[

# The Resubstitution Error Rate

```r
fit.glm <- glm(data=data, 
 Label ~ ., 
 family=binomial(link="logit"))

*fitted.vals <- round(predict(fit.glm,
* newdata = data[, c("Exam1", "Exam2")],
* type="response"))
```
]]
.column.bg-main3[.content.vmiddle.center[
# We then predict the values of all the data points we used to build the model.
#### (Remember, for `glm`, we predict the *probability* of being class 1, and then round).

]]

---

.column.bg-main1[.content[

# The Resubstitution Error Rate

```r
fit.glm <- glm(data=data, 
 Label ~ ., 
 family=binomial(link="logit"))

fitted.vals <- round(predict(fit.glm,
 newdata = data[, c("Exam1", "Exam2")], 
 type="response")) 
*err <- mean(fitted.vals!=data$Label)
err
```

```
## [1] 0.11
```
]]
.column.bg-main3[.content.vmiddle.center[
# We can then find the .orange[*resubstitution error rate*] by looking at the proportion of labels that were not correctly predicted.

]]

---

.column.bg-main1[.content[

# The Confusion Matrix

```r
fit.glm <- glm(data=data, 
 Label ~ ., 
 family=binomial(link="logit")) 
fitted.vals <- round(predict(fit.glm, 
 newdata = data[, c("Exam1", "Exam2")], 
 type="response"))
```

```r
library(caret)
*confusion.glm <- confusionMatrix(as.factor(fitted.vals),
* as.factor(data$Label))
confusion.glm$table
```

```
##           Reference
## Prediction  0  1
##          0 34  5
##          1  6 55
```
]]
.column.bg-main3[.content.vmiddle.center[
## We can examine how our model predicted all the data points, using `confusionMatrix` from the `caret` package. Note, that we are required to put in factor inputs.

]]

---

.column.bg-main1[.content[

# The Confusion Matrix

```r
library(caret)
*confusion.glm <- confusionMatrix(as.factor(fitted.vals),
* as.factor(data$Label))
*confusion.glm$table
```

```
##           Reference
## Prediction  0  1
##          0 34  5
##          1  6 55
```
]]
.column.bg-main3[.content.vmiddle.center[

## Looking at the `table` output, reading *vertically*, we can assess model performance. Out of the 40 failures in our dataset, `glm` sucessfully predicts 34. Out of the 60 passes in our data, `glm` sucessfully predicts 55.
]]

---

# Your turn!

## 
---

.column.bg-main1[.content[

# Classifying and assessing model fit for Diabetes data

```r
data <- read.csv("data/diabetes.csv")
dim(data)
```

```
## [1] 768   9
```

```r
str(data)
'data.frame':	768 obs. of  9 variables:
 $ pregnant : int  6 1 8 1 0 5 3 10 2 8 ...
 $ glucose  : int  148 85 183 89 137 116 78 115 197 125 ...
 $ hg       : int  72 66 64 66 40 74 50 0 70 96 ...
 $ thickness: int  35 29 0 23 35 0 32 0 45 0 ...
 $ insulin  : int  0 0 0 94 168 0 88 0 543 0 ...
 $ bmi      : num  33.6 26.6 23.3 28.1 43.1 25.6 31 35.3 30.5 0 ...
 $ pedigree : num  0.627 0.351 0.672 0.167 2.288 ...
 $ age      : int  50 31 32 21 33 30 26 29 53 54 ...
*$ class    : int  1 0 1 0 1 0 1 0 1 1 ...
```

]]

.column.bg-main3[.content.vmiddle.center[

# Build a machine learning model to predict whether or not the patients in the dataset have diabetes (`class`).

# Further, assess the performance of .orange[*your model*] on classifying `class`.
]]

---

]
.column.slide-in-right[.content.vmiddle[
.sliderbox.shade_main.pad1[
.font5[Cross Validation]
]
]]

---

# Building reliable and accurate models is paramount in machine learning...

---

# ... however, it can be hard to get new data to assess model performance!

<img src="images/newdata.jpg", width="70%">
---

.column.bg-main1[.content[

# Cross Validation

##  Often, we want to see how well our model can predict new data points. However, it is often impossible to get completely new data.

## So, we split our data into .orange[*training]* and .orange[*testing*] sets to evaluate performance, treating the *testing* data as new data points.

]]
.column[.content.vmiddle.center[
<img src="images/Cv1.png", width="80%">
]]

---

.column.bg-main1[.content[

# Cross Validation

## To do this, we split our data into .orange[K-Folds].

## The first fold is treated as a testing set, and the method is fit on the remaining K − 1 folds.

]]
.column[.content.vmiddle.center[
<img src="images/Cv1.png", width="80%">
]]

---

.column.bg-main1[.content[

# Cross Validation

## The misclassification error rate is then computed on the observations in the held-out fold.

## This procedure is .orange[repeated K] times; each time, a different group of observations is treated as a testing set.

]]
.column[.content.vmiddle.center[
<img src="images/Cv2.png", width="80%">
]]

---

.column.bg-main1[.content[

# Cross Validation

# The .orange[CV error rate] is then calculated as the average of these K error rates.

]]
.column[.content.vmiddle.center[
<img src="images/Cv3.png", width="80%">
]]

---

# .purple[How many folds?]

</center>

#### Generally, .pink[k between 5 and 10] avoids over-training the model (variance), whilst avoiding too few training points (bias).

---

.column.bg-main1[.content[

# 5-fold CV for Diabetes `knn` model

```r
library(class) # for KNN function

K <- 5 # number of folds
n <- nrow(data) # number of data points

X <- data[,-9] # predictors
y <- as.factor(data$class) # response
```

]]
.column.bg-main3[.content.vmiddle.center[
# Let's perform a .orange[5-fold] CV for the `knn` model.  First, we split up our response and predictors.
]]

---

.column.bg-main1[.content[