Check out the new article: Data Science and Machine Learning (Part 18): The battle of Mastering Market Complexity, Truncated SVD Versus NMF.
Truncated Singular Value Decomposition (SVD) and Non-Negative Matrix Factorization (NMF) are dimensionality reduction techniques. They both play significant roles in shaping data-driven trading strategies. Discover the art of dimensionality reduction, unraveling insights, and optimizing quantitative analyses for an informed approach to navigating the intricacies of financial markets.
Let’s be real for a second, In most real-world applications many datasets used to build machine-learning models have a very large number of features or variables(dimensions). High-dimensional data can lead to all sorts of challenges such as Increased computation complexity, the risk of overfitting, and difficulties in visualization; That dataset you usually use with 5 independent variables! that’s not what the big guys in AI-Algorithmic trading do.
Say you collect all the MT5(38) built-in indicator buffers, You end up with 56 buffers worth of data, This dataset is now huge.
Curse of Dimensionality
This curse is real, and for those who don’t believe, try to implement a Linear regression model with a lot of correlated independent variables.
The presence of highly correlated features can cause the machine learning models to capture the noise and specific patterns present in the training data, which may not generalize well to new, unseen data.
Author: Omega J Msigwa
