Article by: Antonio Rubio, Project Engineer, Braking Systems in Applus IDIADA
Review Part One | Review Part Two
The field of artificial intelligence (AI) has made significant progress in recent years, with applications ranging from natural language processing to computer vision. In recent years, Applus IDIADA Brakes department has presented several studies about artificial intelligence application for detection of brake noises. In this paper, Applus IDIADA presents the research done in this area, but focusing on the development of an AI model for predicting subjective ratings for squeal brake noises based on objective measurements collected through the instrumentation in a typical Brake Noise Durability programme. Subjective ratings are based on human opinions and can be challenging to quantify. Objective measurements, on the other hand, can be objectively quantified and provide a more reliable basis for prediction.
The first part of the article introduced the data processing, whereas the second part focused on the AI model creation. This third part, on the other hand, explains the methodology to validate the AI model.
1 Model Validation
1.1 Background
Validation experiments are conducted to assess the model’s performance. The results are compared with evaluations from different vehicles, various drivers, and assessments made during the training process. The overall accuracy and specific accuracies for different ratings are analysed.
Accuracy is calculated comparing model prediction and subjective rating from the driver.
During the model training phase, it is also evaluated the performance of both models, named “test”, to select best performance model before validation. This evaluation is performed with the 20% of the selected dataset or 1734 noise events. In test phase, the classification model achieves an accuracy of 70%, and the regression model yields an RMSE of 0.51. RMSE, Root Mean Square Error, measures the average deviation between the predicted values of a model and the actual values in a dataset. A lower RMSE value indicates a better fit of the model to the data, with smaller prediction errors. Conversely, a higher RMSE value suggests a larger average deviation between the predicted and actual values, indicating poorer model performance.
As can be seen, the accuracy results evaluating the 892 noise events from the dataset is around 70% for the reference driver.
1.2 Data set used for validation
Similar to the dataset used for training, validation data comes from reference driver from several years of testing at Applus IDIADA during Brake Noise Durability programmes performed in Mojacar (Almeria). Data set for validation come from same years and tests used to train the model.
The dataset coming from the same reference driver is used. Data used for validation is extracted and separated before the training the model. Number of noise events per rating were distributed according to the frequency in which they appear in the original data set (table 6). A total of 892 noise events were used.
Accuracy is calculated comparing all model prediction vs rating assigned by the driver. A percentage of all correct rating predictions is calculated to get the accuracy. The % error with a difference of 1 rating, 2 rating and 3 rating is also calculated.
Table 6 Validation data set number of noise events for reference driver
It can be seen that there are more subjective ratings available in the data set with high ratings than for low ratings.
1.3 Results
Results for the reference driver are presented in terms of accuracy. Accuracy is calculated comparing the subjective rating prediction from the model with the actual ones of the drivers, meaning a 100% of accuracy a correct prediction (same as driver) of the model for all subjective ratings. In addition, the % of ratings not correctly assigned with a difference error of 1 rating, 2 rating and 3 rating is calculated (table 7). Also, it is calculated accuracy per rating (table 8).
Table 7 Accuracy result for reference driver
Table 8 Accuracy result rating for reference driver
A confusion matrix is created in order to check model performance (figure 8). Confusion matrix or error matrix is a table that is used to evaluate the performance of the model. It compares the predicted classes of the model to the actual classes. In the confusion matrix performed, the rows represent the actual value of rating, and the columns represent the prediction of the model. For each cell, there is the number of values to the current actual vs predicted value. In addition, predicted vs actual values can be checked in figure 9.
Figure 8 Confusion matrix results for reference driver
Figure 9 Model results for reference driver
It can be seen that:
- Close to 70% of prediction rating are same as the driver rating.
- Rating discrepancies between model and driver rating are mainly with a 1 rating error (close to 30%).
- Rating discrepancies between model and driver rating more than 2 points are minimal.
It can be seen that:
- Accuracy for rating 9, rating 8 and rating 7 is around 70%.
- Accuracy for rating 6 or lower decreases to 50% or lower.
About Applus IDIADA
With over 25 years’ experience and 2,450 engineers specializing in vehicle development, Applus IDIADA is a leading engineering company providing design, testing, engineering, and homologation services to the automotive industry worldwide.
Applus IDIADA is located in California and Michigan, with further presence in 25 other countries, mainly in Europe and Asia.
