CONFUSION MATRIX/TYPE 1 ERROR AND TYPE 2 ERROR
In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix,[9] is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning one (in unsupervised learning it is usually called a matching matrix). Each row of the matrix represents the instances in an actual class while each column represents the instances in a predicted class, or vice versa — both variants are found in the literature.[10] The name stems from the fact that it makes it easy to see whether the system is confusing two classes (i.e. commonly mislabeling one as another).
Confusion Matrix with errors
A confusion matrix (or error matrix) is usually used as the quantitative method of characterising image classification accuracy. It is a table that shows correspondence between the classification result and a reference image. I.e., to create the confusion matrix we need the ground truth data, such as cartographic information, results of manually digitizing an image, field work/ground survey results recorded with a GPS-receiver.
AN example in daily life
1) We are getting a business problem 2) Gathering data 3) Cleaning the data 4) Building all kinds of outstanding models, right? Then, we are getting output in probabilities. Wait Wait Wait! How can we say it’s an outstanding model? One way we can say this is by measuring the effectiveness of the model. Better the effectiveness, better the performance of the model. This is where the term Confusion matrix comes into the picture.
A confusion matrix is a performance measurement technique for Machine learning classification problems. It’s a simple table which helps us to know the performance of the classification model on test data for the true values are known.Consider we are doing telecom churn modelling. Our target variable is churn (binary classifier). There are two possible predicted classes: ‘yes’ and ‘no’. ‘Yes’ means churn (leaving the network) and ‘No’ means not churn (not leaving the network). Below is our confusion matrix table
- The classifier made a total of 200 predictions (200 customers’ records were analyzed ).
- Out of 200 customers, the classifier predicted ‘yes’ 160 times, and ‘no’ 40 times.
- In reality, 155 customers are churn, and 45 customers are not churn .
Let’s see the important terms associated with this confusion matrix with the above example
True Positives (TP): These are the people in which we predicted yes (churn), and they are not leaving the network (not churn)
True Negatives (TN): We predicted no, and they are not leaving the network.
False Positives (FP): We predicted yes, but they are not leaving the network (not churn). It is also known as a “Type 1 error”
False Negatives (FN): We predicted no, but they are actually leaving the network (churn). It is also known as a “Type 2 error”….Just incorporated into our confusion table and added both row and columns
Accuracy: How often is the classifier correct?
Accuracy = (TP +TN)/total
Misclassification Rate: Overall, how often is it wrong? It is also called “Error rate”
Misclassification rate = (FP+FN)/total
True Positive Rate (TPR): When it’s actually yes, how often does it predict yes?. It is also known as “Sensitivity” or “Recall”
TPR or Recall = TP/actual yes
False Positive Rate (FPR): When it’s actually no, how often does it predict yes?
FPR = FP/actual no
True Negative Rate (TNR): When it’s actually no, how often does it predict no?. It is also known as “Specificity”
TNR = TN/actual no
Precision: When it predicts yes, how often is it correct?
Precision = TP/Predicted: YES
Prevalence: How often does the yes condition actually occur in our sample?
Prevalence = Actual YES/Total
TYPE I ERROR:
This type of error can prove to be very dangerous. Our system predicted no attack but in real attack takes place, in that case no notification would have reached the security team and nothing can be done to prevent it. The False Positive cases above fall in this category and thus one of the aim of model is to minimize this value.
TYPE II ERROR:
This type of error are not very dangerous as our system is protected in reality but model predicted an attack. the team would get notified and check for any malicious activity. This doesn’t cause any harm. They can be termed as False Alarm.
- Accuracy: The values of confusion matrix are used to calculate the accuracy of the model. It is the ratio of all correct predictions to overall predictions (total values)
Accuracy = (TP + TN)/(TP + TN + FP + FN)
2. Precision: (True positives / Predicted positives) = TP / TP + FP
3. Recall: (True positives / all actual positives) = TP / TP + FN
4. Specificity: (True negatives / all actual negatives) =TN / TN + FP
5. Misclassification: (all incorrect / all) = FP + FN / TP + TN + FP + FN
It can also be calculated as -> 1-Accuracy
