Also known as the Blue Tarps Projet
Table of Contents
- Introduction
- Methods
- Model Fitting
- KNN
- LDA
- QDA
- Logistic Regression
- Random Forest
- SVM
- Model Performance
- KNN
- LDA
- QDA
- Logistic Regression
- Random Forest
- SVM
- Conclusion
Project Intro
This was our final project for Statistical Learning. The project uses images from Haiti after the 2010 earthquake. Aid efforts were trying to quickly identify blue tarps as locations with a lot of displaced people and the goal was to provide food and water to the people in these concentrated locations who need aid the most. The goal of the project was to find the best performing algorithm for detecting blue tarps in images of Haiti from above. The full code can be found on my GitHub here.
Methods
- R and RStudio were used for this project. Microsoft Excel was also used for cleaning and processing the holdout data.
- The classification methods were: KNN, LDA, QDA, Logistic Regression, Random Forest and SVM. Largely used caret, dplyr and ROCR for all of these algorithms and dplyr
- Data was preprocessed using “center” and “scale” for KNN and SVM.
- Changed the thresholds for some of the algorithms based on whether or not I was satisfied with the false and true positive rates.
- Most of the pre-processing was combining and cleaning the various tables of the holdout data into one spreadsheet. Added the class “tarp” to the data for the purpose of prediction
- Split the training data into a test-train split of .25-.75. Trained the training section and had them predict the testing section over 10 folds for cross validation. Then these algorithms had adjusted thresholds if necessary and then ultimately were used to predict the holdout data.
Model Fitting
| Method | Accuracy on Train Data | TP Rate | FP Rate |
|---|---|---|---|
| KNN(K=9) | 99.76% | 96.02% | 0.12% |
| LDA | 98.36% | 77.69% | 0.96% |
| QDA | 98.94% | 89.24% | 0.74% |
| Logistic Regression | 99.13% | 94.42% | 0.44% |
| Random Forest (mtry=1, ntree=500) |
99.47% | 96.71% | 0.44% |
| SVM (Sigma= 2, C=100) |
99.62% | 95.82% | 0.14% |
Model Performance
| Method | Accuracy on Train Data | TP Rate | FP Rate |
|---|---|---|---|
| KNN(K=9) | 99.31% | 88.32 | 0.61% |
| LDA | 98.27% | 80.57% | 1.61% |
| QDA | 98.06% | 83.2% | 1.83% |
| Logistic Regression | 98.99% | 98.83% | 1.01% |
| Random Forest (mtry=1, ntree=500) |
99.47% | 74.1% | 0.35% |
| SVM (Sigma= 2, C=100) |
98.8% | 42.46% | 0.79% |
Conclusion
- I decided that Logistic Regression performed the best. The two main things I valued were high true positive rates, and low false positive rates. I aimed for a false positive rate around 1% and a true positive rate around 90% as ideal. I would put the ranking at 1. Logistic Regression 2. KNN 3. LDA 4. QDA 5. Random Forest 6. SVM. While SVM and Random Forest had the lowest false positive rates, I could not look past their poor true positive rates which put them at the bottom of the list.
- I decided that false positives were the worst case scenario in this situation. A true positive being the best case scenario because that means people who needed aid received it. While a true negative prevents resources from being wasted. A false negative is an unfortunate situation because people who could have received aid did not, but I view that better than a false positive because a false positive helps no one while also wasting limited resources. So I made an effort to reduce false positive rates to avoid what I thought to be the worst possible scenario in this situation. It’s a delicate ethical balance.
- The discrepancy between performance across training and hold out data sets came as a surprise to me. It did not seem consistent for the models between datasets. I imagine the poorer performing algorithms were overfit for the training dataset. Should be something to note when deciding thresholds.
- I think it should be noted that a small amount of the data is actually blue tarps. I think this may have impacted the accuracy of algorithms such as SVM and Random Forest. They both boasted high overall accuracy rates but low true positive rates. That innate skew in data points may have played a role in the discrepancy between their accuracy and their true positive rates.