Applied Sciences | Free Full-Text

3. Evaluation and Results

For the Anaconda Navigator platform with Jupyter Notebook, different machine-learning techniques used in further research were modeled [35], which were decision tree, support vector machine, random forest, naive Bayes, logistic regression, k-nearest neighbors, neuronal network, gradient boosted, and smart bands. Of the data to model, 80% was used for training and 20% for testing.

Jupyter Notebook is a widely used tool in the machine-learning community, where you can import a set of libraries, train a dataset classification classifier, and evaluate the model by just using a few lines of code in Python with its library Scikit-learn.

Likewise, the optimization of hyperparameters with the Scikit-learn optimization library was used. Hyperparameters are parameters that are not directly learned by the learning algorithm. The basic hyperparameter tuning models are manual search, grid search, and random search. Random search was used for training since it allows us to find equal or better models in computing time.

The configuration used for the hyperparameters in the case of gradient boosted was the loss function to optimize with the “log_loss” option, the criterion with the process to measure the quality of a division with the “friedman_mse” option, the “learning_rate” with 0.1, “max_depth”: 3, “min_samples_leaf”: 1, “min_samples_split”: 2, “n_estimators”: 100, “random_state”: none, “subsample”: 1.0, “tol”: 0.0001, “validation_fraction”: 0.1, and with the other default parameters.

Next, the tests were carried out, and the results of the modeling that was executed for the different techniques that were previously chosen were obtained using the CSV file as input data and made up of the motor competence variables.

To evaluate the model [36], accuracy in Equation (1), recall in Equation (2), precision in Equation (3), and the f1-score in Equation (4) were used. For many classes C_i, fp_i represents false positive, tp_i true positive, fn_i false negative, and tn_i true negative.

$A c c u r a c y = \frac{{t p}_{i} + {t n}_{i}}{{t p}_{i} + {t n}_{i} {+ f p}_{i} + {f n}_{i}}$

(1)

$R e c a l l = \frac{\sum_{i = 1}^{l} \frac{{t p}_{i}}{{t p}_{i} + {f n}_{i}}}{l}$

(2)

$P r e c i s i o n = \frac{\sum_{i = 1}^{l} \frac{{t p}_{i}}{{t p}_{i} + {f p}_{i}}}{l}$

(3)

$F 1 - s c o r e = 2 \times \frac{(P r e c i s i o n) \times (R e c a l l)}{(P r e c i s i o n) + (R e c a l l)}$

(4)

Table 4 and Table 5 compare the classical and optimized machine-learning techniques results with selected hyperparameters and the accuracy, f1-score, recall, and precision metrics for both schoolchildren males and females.

In the results of the previous tables, it can be identified that, for the classification techniques, those that gave the best results concerning accuracy for the case of males and females were gradient boosting, whose values were the highest, indicating a better adjustment to the estimated prediction with a value of 0.95. For the f1-score metrics, the algorithms gave similar values of 0.92 between males and females. In the case of recall, it was 0.92. The confusion matrix of the gradient boosting machine-learning algorithm with the highest score found is shown in Figure 7.

Figure 8 and Figure 9 show a graph of the ROC-AUC curves of gradient boosting for males and females.

Observing the results of the ROC curves, in the case of male schoolchildren, the “Low” class shows an intense elevation towards the upper left corner of the graph and has a high area under the curve (AUC) of 0.98; this indicates that the model has a high sensitivity to detect the “Low” classification of motor competence. The false positive rate is low, suggesting the model does not misidentify many schoolchildren with the “Low” motor competence classification.

In the case of the female schoolchildren, the “High” class shows an intense elevation towards the upper left corner of the graph and has a high area under the curve (AUC) of 0.98; this indicates that the model has a high sensitivity to detect the classification “High” motor competence. The false positive rate is low, suggesting the model does not misidentify many schoolchildren with the “High” motor competence classification. On the other hand, the dark blue dotted line refers to the fact that the test has been bad, which is not the case of the study.

Deployment

A mobile app was developed in Android Studio using the Flutter framework. The code structure was organized by the best mobile application development practices and following the design pattern recommended by Flutter, which is the model–view–controller (MVC) design pattern. Firebase real-time database was used for storage.

The primary interfaces are shown in Figure 10, where you have the login screen and the options menu.

Figure 11 shows the interface that allows entering the data corresponding to the student; the body mass index (BMI) will be able to be calculated automatically. Additionally, table is provided to identify the student’s BMI classification. Likewise, the interface that allows entering the data captured with the smart band, such as speed, cadence, stride, and steps of the schoolchildren, is shown. It contains an Artificial Intelligence function that allows you to locate your position within the percentiles generated from the sample of schoolchildren.

The process was evaluated through the prototype of the Software Quality Systemic Model (SQSM). This model is planned based on six standardized international quality characteristics through a set of characteristic categories and metrics, which measure and evaluate the software quality of a product; this model is made into an instrument for measuring great value covering essential aspects of software quality.

It was carried out in the selected educational centers to evaluate the application’s functionalities. The information collected meets expectations as it explains the population’s needs.

Once the various interviewees were integrated, we proceeded with the evaluation method based on the Systemic Quality Model, which includes 11 categories. We have selected three specific classes related to the software: functionality, reliability, and usability because these apply to the case study. Table 6 shows a detailed description of each category with the interviews.

Considering the levels of satisfaction achieved in the categories of functionality, reliability, and usability, by calculating the arithmetic mean, it is observed that the degree of satisfaction of the application for the specialists of the educational centers is 77.09%. By contrasting this result with the quality level, it is evident that the application meets significant standards.

4. Discussion

The study’s objective was to classify the motor competence of schoolchildren performed in a school using smart bands according to age range and sex, using machine-learning techniques optimized with hyperparameters acceptably for the classification indicated through the passing results in the tests carried out.

The results of the study have shown that according to the motor competence tests carried out, step cadence values obtained range from 96 to 29 steps in both sexes and the number of steps ranges from 1437 to 455, with speed from 4.75 to 1.16 and the stride from 88 to 71. Furthermore, it was verified that the measurements obtained decreased rapidly with age, being more pronounced in females than in males.

In this context, the cut-off points suggested in other recent research [37] were taken as a basis, where the study proposed percentiles of the number of steps in a day. The percentiles indicate p75 above average. In essence, percentiles, regardless of the method used, can be applied to establish improvement goals, especially for schoolchildren below the 25th percentile [38]. For example, the schoolchildren in this study classified below the p25th percentile in the tests performed showed higher BMI values.

Precisely, these results coincide with those found in [4], when it states that the measurements obtained decrease rapidly with age. This information could help promote strategies for developing physical activity in schoolchildren in educational centers to improve their performance. However, a limitation found in the study was the need for a broader set of tests to be performed.

Furthermore, the results of the study have shown that using machine-learning techniques such as those evaluated by Himi et al. [14,39], then optimized with hyperparameters like work carried out by Yadav et al. [38] and Yang et al. [40], they achieve reasonable classifications for schoolchildren according to their motor competence. It can also be seen that the naïve Bayes algorithm obtains the lowest results [41,42].

It was determined that the most suitable supervised machine-learning technique would be the gradient-boosted model, which has obtained the best accuracy of 0.95 with an f-score of 0.93, recall of 0.92, and precision of 0.94, which was used in works such as [43,44]. This has been corroborated by the ROC-AUC curves, where the “Low” classes for males and “High” for females show a substantial elevation towards the upper left corner of the graph and have a high area under the curve of 0.98; this indicates that the model has a high sensitivity to detect motor competence classification.

This study represents a significant contribution with the use of wearable devices for data capture and classification using machine-learning algorithms optimized with hyperparameters for motor competence in schoolchildren that can support people interested in finding more precise ways to evaluate motor competence with the support of technology.

5. Conclusions

In conclusion, smart bands have shown great potential to improve the motor competence of schoolchildren through machine learning and hyperparameters. The gradient-boosted algorithm is a good model for classifying schoolchildren. Motor competence is a crucial aspect of the development of children and adolescents, and smart bands can facilitate this process by providing personalized feedback. By harnessing the power of technology, smart bands can help children develop essential motor skills, leading to better physical health and overall well-being. As more research is conducted in this field, we expect to see more advancements in smart band technology, which will undoubtedly positively impact child development.

Finally, the mobile software product built based on the proposed model was validated using the prototype of the Software Quality Systemic Model (SQSM) based on three specific categories: functionality, reliability, and usability, obtaining 77.09%

Expanding the determining attributes for classifying motor competence in schoolchildren is recommended in future work. Additionally, constructing the application of other classification algorithms is essential to compare results and assess efficiency, as well as deep learning. Different approaches, such as transfer learning, can be used to adapt domain data to train at high fidelity.

Source link