Month: July 2020

Binary Classification Model for Truck APS Failure Using TensorFlow Take 3

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In the previous Scikit-Learn iterations, we constructed and tuned machine learning models for this dataset using the Scikit-Learn and the XGboost libraries. We also observed the best accuracy result that we could obtain using the tuned models with the training, validation, and test datasets.

In iteration Take1, we constructed and tuned machine learning models for this dataset using TensorFlow with three layers. We also observed the best result that we could obtain using the tuned models with the validation and test datasets.

In iteration Take2, we provided more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We increased the population of the minority class from approximately 0.1% to approximately 33% of the training instances. We then decreased the population of the majority class to equal to the minority class. Furthermore, we also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In this Take3 iteration, we will construct and tune machine learning models for this dataset using TensorFlow with four layers. At the same time, we will leverage the SMOTE technique to augment the dataset for training purposes. Furthermore, we will observe the best result that we can obtain using the tuned models with the validation and test datasets.

ANALYSIS: From the previous Scikit-Learn iterations, the optimized XGBoost model processed the testing dataset with a recall metric of 98.66% with a low Type II error rate.

From this Take1 iteration, the performance of the three-layer TensorFlow model achieved a recall score of 77.20% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 75.20%, which was consistent with the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 55.46% with a high Type II error rate.

From iteration Take2, the performance of the three-layer TensorFlow model achieved a recall score of 87.60% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 96.40%, which was much better than the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 85.06% with a lower Type II error rate than the previous iteration.

From this Take3 iteration, the performance of the four-layer TensorFlow model achieved a recall score of 83.60% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 93.20%, which was much better than the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 95.73% with a lower Type II error rate than the previous iteration.

CONCLUSION: For this dataset, the model built using TensorFlow did better than the previous iteration with the help of SMOTE. We should consider using TensorFlow and SMOTE to model this dataset further.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.

Binary Classification Model for Truck APS Failure Using TensorFlow Take 2

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In the previous Scikit-Learn iterations, we constructed and tuned machine learning models for this dataset using the Scikit-Learn and the XGboost libraries. We also observed the best accuracy result that we could obtain using the tuned models with the training, validation, and test datasets.

In iteration Take1, we constructed and tuned machine learning models for this dataset using TensorFlow with three layers. We also observed the best result that we could obtain using the tuned models with the validation and test datasets.

In this Take2 iteration, we will attempt to provide more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We will increase the population of the minority class from approximately 0.1% to approximately 33% of the training instances. We will then decrease the population of the majority class to equal to the minority class. Furthermore, we will observe the best sensitivity/recall score that we can obtain using the tuned models with the training and test datasets.

ANALYSIS: From the previous Scikit-Learn iterations, the optimized XGBoost model processed the testing dataset with a recall metric of 98.66% with a low Type II error rate.

From this Take1 iteration, the performance of the three-layer TensorFlow model achieved a recall score of 77.20% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 75.20%, which was consistent with the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 55.46% with a high Type II error rate.

From this Take2 iteration, the performance of the three-layer TensorFlow model achieved a recall score of 87.60% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 96.40%, which was much better than the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 85.06% with a lower Type II error rate than the previous iteration.

CONCLUSION: For this dataset, the model built using TensorFlow did better than the previous iteration with the help of SMOTE. We should consider using TensorFlow and SMOTE to model this dataset further.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.

Binary Classification Model for Truck APS Failure Using TensorFlow Take 1

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In the previous Scikit-Learn iterations, we constructed and tuned machine learning models for this dataset using the Scikit-Learn and the XGboost libraries. We also observed the best accuracy result that we could obtain using the tuned models with the training, validation, and test datasets.

In this Take1 iteration, we will construct and tune machine learning models for this dataset using TensorFlow with three layers. We will observe the best accuracy result that we can obtain using the tuned models with the validation and test datasets.

ANALYSIS: From the previous Scikit-Learn iterations, the optimized XGBoost model processed the testing dataset with a recall metric of 98.66% with a low Type II error rate.

From this Take1 iteration, the performance of the three-layer TensorFlow model achieved a recall score of 77.20% with the training dataset. After a series of tuning trials, the TensorFlow model processed the validation dataset with a recall score of 75.20%, which was consistent with the prediction from the training result. When configured with the optimized parameters, the TensorFlow model processed the test dataset with a recall score of 55.46% with a high Type II error rate.

CONCLUSION: For this dataset, the model built using TensorFlow did not achieve a satisfactory result, but we should consider using TensorFlow to model further.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.

Binary Classification Model for Truck APS Failure Using Scikit-Learn Take 4

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In iteration Take1, we constructed and tuned machine learning models for this dataset using the Scikit-Learn library. We also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In iteration Take2, we attempted to provide more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We increased the population of the minority class from approximately 0.1% to approximately 33% of the training instances. Furthermore, we also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In iteration Take3, we constructed and tuned machine learning models for this dataset using the XGBoost library. We also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In this Take4 iteration, we will attempt to provide more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We will decrease the population of the majority class to be the same as the minority class of the training instances. Furthermore, we will observe the best sensitivity/recall score that we can obtain using the tuned models with the training and test datasets.

ANALYSIS: From iteration Take1, the performance of the machine learning algorithms achieved an average recall metric of 59.26%. Two algorithms (Extra Trees and Random Forest) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 68.53%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 66.40%. Furthermore, the optimized model processed the test dataset with a score of 71.73% with a high Type II error rate.

From iteration Take2, the performance of the machine learning algorithms achieved an average recall metric of 98.21%. Two algorithms (Extra Trees and k-Nearest Neighbors) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 99.72%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 80.40%. Furthermore, the optimized model processed the test dataset with a score of 82.40% with a high Type II error rate.

From iteration Take3, the performance of the XGBoost algorithm achieved a baseline recall metric of 75.86%. After a series of tuning trials, the XGBoost model completed the training phase and achieved a score of 99.72%. When configured with the optimized learning parameters, the XGBoost model processed the validation dataset with a score of 72.80%. Furthermore, the optimized model processed the test dataset with a score of 78.93% with a high Type II error rate.

From this Take4 iteration, the performance of the XGBoost algorithm achieved a baseline recall metric of 96.67%. After a series of tuning trials, the XGBoost model completed the training phase and achieved a score of 96.80%. When configured with the optimized learning parameters, the XGBoost model processed the validation dataset with a score of 97.20%. Furthermore, the optimized model processed the test dataset with a score of 98.66% with a low Type II error rate.

CONCLUSION: For this iteration, the XGBoost model achieved the best overall results using the training and test datasets. For this dataset, we should consider using the Extra Trees algorithm for further modeling and testing activities.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.

Binary Classification Model for Truck APS Failure Using Scikit-Learn Take 3

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In iteration Take1, we constructed and tuned machine learning models for this dataset using the Scikit-Learn library. We also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In iteration Take2, we attempted to provide more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We up-sample’ed the minority class from approximately 0.1% to approximately 33% of the training instances. Furthermore, we also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In this Take3 iteration, we will construct and tune machine learning models for this dataset using the Scikit-Learn library. We will observe the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

ANALYSIS: From iteration Take1, the performance of the machine learning algorithms achieved an average recall metric of 59.26%. Two algorithms (Extra Trees and Random Forest) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 68.53%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 66.40%. Furthermore, the optimized model processed the test dataset with a score of 71.73% with a high Type II error rate.

From iteration Take2, the performance of the machine learning algorithms achieved an average recall metric of 98.21%. Two algorithms (Extra Trees and k-Nearest Neighbors) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 99.72%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 80.40%. Furthermore, the optimized model processed the test dataset with a score of 82.40% with a high Type II error rate.

From this Take3 iteration, the performance of the XGBoost algorithm achieved an average recall metric of 75.86%. After a series of tuning trials, the XGBoost model completed the training phase and achieved a score of 99.72%. When configured with the optimized learning parameters, the XGBoost model processed the validation dataset with a score of 72.80%. Furthermore, the optimized model processed the test dataset with a score of 78.93% with a high Type II error rate.

CONCLUSION: For this iteration, the XGBoost model achieved the best overall results using the training and test datasets. For this dataset, we should consider using the Extra Trees algorithm for further modeling and testing activities.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.

Scott Adams on Loserthink, Part 3

In the book, Loserthink: How Untrained Brains Are Ruining America, Scott Adams analyzed and discussed ways to teach us how to eliminate our biases and to sharpen our ability to think critically.

These are some of my favorite quotes and takeaways from reading the book.

How can we minimize our tendency for Loserthink by thinking like an artist?

What to watch out for: Failure of Imagination.

“If you can’t imagine any other explanation for a set of facts, it might be because you are bad at imagining things.”

How can we minimize our tendency for Loserthink by thinking like a historian?

What to watch out for: Assuming the history we learned is accurate

“If you believe you learned an accurate version of history in school, you are probably wrong.”

What to watch out for: Allowing history to have a stronghold on us

“If bad memories are keeping you from being happy, try crowding out the destructive memories with new and interesting thoughts. Stay busy, in mind and body, and time is on your side.”

“History (even the fake kind) can be useful for persuading others through guilt. But don’t make the mistake of persuading yourself that history should matter to your choices today.”

“Focusing on the past when the present offers sufficient paths to success is Loserthink. It is better to focus on your own systems for success, and when you succeed, watch how winning fixes most problems.”

What to watch out for: History Repeats

“History doesn’t repeat, at least not in any way you can use to accurately predict the future. (The exceptions are simple situations.)”

What to watch out for: The Slippery Slope

“Belief in slippery slopes is Loserthink. It is more useful to look at forces and counterforces to see where things are likely to end up.”

Watch to watch out for: Privacy is Overrated.

“If you think more privacy is always better, that is a case of Loserthink. Every situation is different. Sometimes privacy is the problem that prevents the solutions.”

Binary Classification Model for Truck APS Failure Using Scikit-Learn Take 2

Template Credit: Adapted from a template made available by Dr. Jason Brownlee of Machine Learning Mastery.

SUMMARY: The purpose of this project is to construct a predictive model using various machine learning algorithms and to document the end-to-end steps using a template. The Truck APS Failure dataset is a binary classification situation where we are trying to predict one of the two possible outcomes.

INTRODUCTION: The dataset consists of data collected from heavy Scania trucks in everyday usage. The system in focus is the Air Pressure system (APS), which generates pressurized air that supports functions such as braking and gear changes. The dataset’s positive class consists of component failures for a specific component of the APS system. The negative class consists of trucks with failures for components not related to the APS. The training set contains 60000 examples in total, in which 59000 belong to the negative class and 1000 positive class. The test set contains 16000 examples.

The challenge is to minimize the total cost of a prediction model the sum of “Cost_1” multiplied by the number of Instances with type 1 failure and “Cost_2” with the number of instances with type 2 failure. The “Cost_1” variable refers to the cost resulted from a redundant check by a mechanic at the workshop. Meanwhile, the “Cost_2” variable refers to the cost of not catching a faulty truck. The cost of Type I error (cost_1) is 10, while the cost of the Type II error (cost_2) is 500.

In iteration Take1, we constructed and tuned machine learning models for this dataset using the Scikit-Learn library. We also observed the best sensitivity/recall score that we could obtain using the tuned models with the training and test datasets.

In this Take2 iteration, we will attempt to provide more balance to this imbalanced dataset by using “Synthetic Minority Oversampling TEchnique” or SMOTE for short. We will up-sample the minority class from approximately 0.1% to approximately 33% of the training instances. Furthermore, we will observe the best sensitivity/recall score that we can obtain using the tuned models with the training and test datasets.

ANALYSIS: From iteration Take1, the performance of the machine learning algorithms achieved an average recall metric of 59.26%. Two algorithms (Extra Trees and Random Forest) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 68.53%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 66.40%. Furthermore, the optimized model processed the test dataset with a score of 71.73% with a high Type II error rate.

From this Take2 iteration, the performance of the machine learning algorithms achieved an average recall metric of 95.79%. Two algorithms (Extra Trees and k-Nearest Neighbors) produced the top results after the first round of modeling. After a series of tuning trials, the Random Forest model completed the training phase and achieved a score of 99.67%. When configured with the optimized learning parameters, the Random Forest model processed the validation dataset with a score of 80.40%. Furthermore, the optimized model processed the test dataset with a score of 82.40% with a high Type II error rate.

CONCLUSION: For this iteration, the Extra Trees model achieved the best overall results using the training and test datasets. For this dataset, we should consider using the Extra Trees algorithm for further modeling and testing activities.

Dataset Used: APS Failure at Scania Trucks Data Set

Dataset ML Model: Binary classification with numerical attributes

Dataset Reference: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

One potential source of performance benchmark: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks

The HTML formatted report can be found here on GitHub.