Understanding Model Personality Through Hyperparameters #1: Hyperparameters Are Not About “Improving Performance,” but About “Controlling Runaway Behavior”
Deep insights into the hyperparameters of LightGBM and Extra Trees based on the tuning order
Disclaimer: As a freelance analyst, my posts exclusively cover the AI methodologies of Portfolio123 (P123), without addressing individual stock recommendations.
This Post is included in Section 4: How to Effectively Choose Algorithms and Tune Hyperparameters.
The table of contents is as follows:
AI-Driven Quant Investment Strategies
Hyperparameters Are Not About “Improving Performance,” but About “Controlling Runaway Behavior”
Introduction
So far in several series, we have compared LightGBM and Extra Trees from several different angles.
In Part 1, we conducted a “personality diagnosis” of the two models.
We examined the basic characteristics of each model, including resistance to overfitting, robustness to outliers, robustness to noisy targets, handling of interactions between factors, linear and nonlinear patterns, multicollinearity, and suitability for small-cap universes and the S&P 500.
In Part 2, we discussed how to interpret model results.
We looked at RMSE, Pearson correlation, Spearman correlation, Avg%(H-L), Feature Importance, SD, Turn%, and return curves.
In particular, we confirmed that in financial machine learning, it is not enough for prediction error to be small. We also need to examine the spread between the top-ranked and bottom-ranked stocks, the stability of the return curve, turnover, standard deviation, and other metrics in a comprehensive way.
In Part 3, we visualized the learning steps of LightGBM and Extra Trees.
We saw that LightGBM is a sequential model that corrects the errors of previous trees with subsequent trees, while Extra Trees is a model that builds many highly randomized trees independently and gains stability by averaging them.
In Part 4, we discussed the true role of validation, or cross-validation.
Validation data is not where the model is directly updated. Rather, it is where we examine whether the model can withstand future data.
We also covered early stopping, the number of folds, the strictness of validation, and why ensembles of weak models can be powerful.
At this point, we have a fairly clear picture of the basic personalities of LightGBM and Extra Trees, how to read their results, how their learning processes work, and what role validation plays.
The next major topic should be “features”.
In other words, we need to discuss what kinds of fundamental indicators, price momentum indicators, analyst estimate indicators, quality indicators, value indicators, growth indicators, and risk indicators should be fed into the model.
However, before moving on to that major topic, there is one important preliminary step.
That is the topic we will begin covering now: “hyperparameters”.
Before increasing or modifying features, we first need to understand the “control knobs” on the model side.
This is because even if we use the same features, the model’s behavior can change significantly depending on the hyperparameter settings.
Even when using the same data, the same target, and the same features, the following can happen:
LightGBM may try to predict too sharply.
Extra Trees may average too much and become too dull.
The scores of top-ranked stocks may become unstable.
Results may look good in cross-validation but collapse in backtesting.
RMSE may improve while Avg% (H-L) does not improve.
Many of these differences are deeply related to hyperparameter settings.
Therefore, in Part 5, we will organize the major hyperparameters of LightGBM and Extra Trees in an order that is close to the actual order in which we would examine them during tuning.
In this first post, before getting into individual parameters, we will first review the overall picture.
(1) What Are Hyperparameters?
In machine learning models, there are broadly two types of “settings.”
The first type is what the model automatically learns from the data.
For example, which feature to use at each split, what threshold to split on, and what prediction value to place in each leaf.
These are determined by the model itself from the training data.
The second type is what humans decide in advance.
For example, how many trees to build, how deep the trees are allowed to grow, how many samples are required in a leaf, how much randomness to apply to feature selection, and what learning rate to use.
These are hyperparameters.
Put simply, hyperparameters are:
Control knobs that humans set in advance to determine how the model learns.
For LightGBM, examples include:
learning_rate
n_estimators
num_leaves
max_depth
min_child_samples
feature_fraction
bagging_fraction
lambda_l1
lambda_l2
min_gain_to_split
For Extra Trees, examples include:
n_estimators
max_features
max_depth
min_samples_leaf
min_samples_split
bootstrap
max_samples
These are not just minor settings.
They determine how complex a pattern the model is allowed to learn.
How much the model reacts to noise.
How stable its predictions are.
How much it depends on specific features.
How much randomness is introduced.
In other words, they are important elements that determine the model’s “personality.”
What the Paid Section Deeply Explores:
（Free Section)
(1) What Are Hyperparameters?
（Paid Section)
(2) Is Hyperparameter Tuning a Process for Improving Performance?
(3) In Financial ML, “Robustness” Matters More Than “Predictive Power”
(4) LightGBM and Extra Trees Require Different Tuning Philosophies
(5) LightGBM Is “Sharp, but Can Become Unstable”
(6) Extra Trees Is “Duller, but Easier to Stabilize”
(7) Even With the Same Features, Hyperparameters Can Change the Results
(8) RMSE Optimization and Investment Performance Optimization Are Not the Same
(9) Hyperparameters Are Not Tools for Finding the “Best Historical Performance”
(10) What to Look at First in LightGBM
(11) Summary
Pricing Plans
To continue learning highly specific and practical AI strategy construction methods, please consider a premium subscription.
Monthly Plan: $8 / Month, Flexible starter option
Annual Plan: $80 / Year, 2 months free (approximately $6.67/month)
Subscribe now and advance to the next level of AI Quant Strategy!
The following section is available to paid subscribers only.