Detecting Regimes in the Volatility Surface Using Clustering

Regime identification is important in portfolio and risk management. There are many ways to classify market regimes, for example, based on market direction, such as bullish, bearish, or sideways. Another common classification is based on volatility regimes, such as high or low volatility. Most existing methods for detecting volatility regimes rely on single-point data, such as implied volatility indices or realized volatility measures.

Reference [1] proposes a new regime classification approach based on the entire volatility surface. The method first calculates local gradients, defined as the partial derivatives of implied volatility with respect to moneyness and maturity. These gradient changes are then clustered using an unsupervised algorithm to identify recurring structural transformations of the volatility surface.

The author pointed out,

My goal was to analyze structural changes in the implied volatility surface and study how the surface evolves over time. In this study, I have successfully developed a methodology to represent and quantify daily structural changes in the IV surface using local gradients. I have also used unsupervised clustering algorithm to identify distinct types of surface transformations. I have also included interpretations for some of the clusters in the previous section.

My analysis revealed that there is a number of distinct types of surface transformations that can be identified and interpreted. As expected, the vast majority of daily IV surface changes were classified as noise because structural changes in the volatility surface do not occur often. Therefore, the identified clusters have sizes of up to 18 samples. Notably, several clusters represented specific skew or term structure dynamics for different levels of maturity and moneyness.

In summary, the paper demonstrated that the clusters correspond to specific structural changes in skew and term structure rather than random fluctuations. For example:

• Cluster 1: changes mainly affect short-term maturities (1–2 months) and alter the slope across moneyness, indicating localized movements in the front of the surface.
• Cluster 2: shows term-structure rotation, where longer-maturity implied volatilities fall relative to short maturities for some moneyness levels, making the term structure steeper.
• Cluster 3: reflects a flattening of the skew, where higher-moneyness volatilities decrease relative to lower-moneyness ones, interpreted as increased demand for downside protection (more bearish sentiment).

This paper is exploratory in nature, as it does not prove the economic benefits of using the proposed volatility surface clustering technique. However, it points to an important research direction: volatility regime detection may benefit from analyzing the full volatility surface rather than relying on single volatility indicators, and the paper proposes a practical framework for doing so.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Dzhafarov, Shakhin (2025). Detecting Structural Evolution of Implied Volatility Surface Using Gradient-Based Features: A Machine Learning Approach to Market Regime Detection. Master’s thesis, Aalto University.

Post Source Here: Detecting Regimes in the Volatility Surface Using Clustering

source https://harbourfronts.com/detecting-regimes-volatility-surface-using-clustering/

Retail Options Trading and Gambling Behavior

Options trading volume has risen sharply in recent years, and a significant portion of this increase is attributed to the growing participation of retail traders. We have discussed retail options trading behavior in previous posts. Reference [1] continues this line of research by examining how retail investors trade stock options and how their attention influences options market activity.

The study posits that retail investors often treat stock options similarly to gambling. The authors construct a Search Volume Index (SVI), which measures the intensity of Google searches for specific keywords across U.S. states, and use it as a proxy for retail investors’ attention to stock options, capturing surges in public interest around firm-specific news such as earnings announcements. They pointed out,

This paper examines how regional gambling propensity relates to retail participation in U.S. options markets. Using state-level Google search data, gambling measures, and regulatory and event-based shocks, we analyze whether gambling intensity is associated with option attention and speculative trading behavior.

We document that option attention is higher in gambling-prone states, particularly around salient events such as earnings announcements. We also show that option search intensity is positively associated with brokerage-related search activity, consistent with attention translating into trading-related behavior…

We further find that lottery-like option characteristics—including out-of-the-money contracts, short maturities, and high implied volatility—receive greater attention in high-gambling states. These results suggest that regional gambling propensity helps explain cross-state variation in speculative option demand.

Finally, we relate option attention to household credit outcomes. Elevated option attention in gambling-prone states is associated with higher short-term borrowing and increased delinquency rates. While these associations do not establish household-level causality, they indicate that gambling-motivated financial attention coincides with measures of financial vulnerability at the state level.

Overall, the findings highlight how regional behavioral traits interact with financial market structure to shape retail participation in derivative markets. Future research may examine whether similar dynamics arise in other highly leveraged products or emerging speculative venues.

In short, this paper shows that retail option trading is higher in U.S. states with a stronger gambling culture, especially around earnings announcements when uncertainty is high. It also finds that this gambling-motivated attention increases trading in short-dated out-of-the-money options and is associated with higher implied volatility and higher household debt.

This paper provides additional and interesting insights into retail options behavior. Let us know what you think in the comments below or in the discussion forum.

References

[1] Matthew Flynn, Yifan Liu, Ivilina Popova, Do retail traders gamble on stock options? Journal of Financial Markets, 2026

Originally Published Here: Retail Options Trading and Gambling Behavior

source https://harbourfronts.com/retail-options-trading-gambling-behavior/

Entropy-Based Regime Detection of Tail Risks

Identifying market regimes is particularly important in portfolio and risk management. Typically, markets are classified as bullish or bearish, or as being in high- or low-volatility regimes.

Reference [1] proposes an alternative classification by distinguishing between “normal” and heavy-tailed regimes. Specifically, the study develops a nonparametric method to detect financial market regimes using differential entropy rather than volatility alone. The underlying idea is that while volatility measures dispersion, entropy captures the full distributional uncertainty, including tail behavior, which becomes particularly important during crisis periods.

The authors estimate entropy using a kernel density estimator with a heavy-tailed kernel in rolling windows and compare entropy with variance. When markets behave approximately Gaussian, i.e., normally, entropy and variance move together; during turbulent periods, the relationship breaks down, revealing heavy-tailed regimes that volatility alone cannot identify.

They pointed out,

This study demonstrates that differential entropy estimation with a heavy-tailed kernel provides an effective, nonparametric framework for identifying financial market regimes beyond traditional variance-based measures. By integrating entropy and tail-index analysis within a moving-window kernel density approach, the method captures dynamic shifts in distributional behavior without relying on parametric assumptions, offering a flexible tool for regime detection…

Empirically, applying the method to four major stock indices (Ibovespa, S&P 500, Nikkei, and SSE Composite) revealed that heavy-tailed regimes align with well-known episodes of market turbulence, including the Dot-com Bubble, the Global Financial Crisis, the COVID-19 shock, and the recent tariff-related crisis. In contrast, Gaussian regimes correspond to periods of relative stability and market efficiency.

Importantly, we showed that variance and entropy do not need to move in tandem. While volatility quantifies dispersion, entropy captures broader uncertainty and tail risk, remaining well-defined even when higher-order moments fail to exist. This divergence underscores the limitations of moment-based measures and highlights the potential of entropy as a complementary indicator of systemic instability.

In short, the paper developed a regime detection method based on entropy, which provides an alternative regime indicator that captures tail risk and structural shifts that standard volatility measures may miss.

Applying the method to Ibovespa, S&P 500, Nikkei, and SSE shows that detected heavy-tailed regimes coincide with major crises such as the Dot-com crash, the Global Financial Crisis, COVID-19, and other stress events, while Gaussian regimes correspond to calmer market periods.

This represents an important contribution to the literature, particularly in the context of managing tail risks and risk management more broadly.

Let us know what you think in the comments below or in the discussion forum.

References

[1]  Raul Matsushit, Iuri Nobre, Sergio Da Silva, Beyond volatility: Using differential entropy to detect financial market regimes, Chaos, Solitons and Fractals 202 (2026) 117553

Post Source Here: Entropy-Based Regime Detection of Tail Risks

source https://harbourfronts.com/entropy-based-regime-detection-tail-risks/

Improving Pairs Trading with Cluster-Based Pair Selection

Pairs trading is a classic quantitative trading strategy. Despite its widespread use, it continues to attract research attention. A recent line of research focuses on grouping underlyings into clusters with similar characteristics. We recently discussed such grouping using fundamental metrics.

Reference [1] also attempts to improve the pair selection process, but this time using a purely quantitative, unsupervised machine learning approach. Specifically, it groups the underlyings using density-based methods applied to the correlation-distance matrix. The objective is to identify neighborhoods of underlying that exhibit similar residual return dynamics, that is, comparable co-movement patterns after removing broad market effects, so that cointegration tests and mean-reversion diagnostics are applied primarily within economically coherent groups.

The authors pointed out,

The performance we obtain is encouraging given the constraints of our design: we use only daily adjusted closes, transparent mean-reversion filters, and simple z-score entry/exit rules without any intraday microstructure signals. Across the four clustering specifications in Table 2, annualized Sharpe ratios range from 1.7228 to 2.4935. The best risk-adjusted variant is OPTICS with PC1 removed (Sharpe 2.4935), which also achieves the smallest maximum drawdown (-8.74%). At the same time, the highest terminal wealth is achieved by HDBSCAN with PC1 removed (cumulative return 1020.34%), albeit with materially larger downside risk (max drawdown -28.39%). Overall, the results suggest that a fully reproducible, daily-data pipeline can still recover substantial relative-value structure in a broad ETF universe.

A natural benchmark is a passive investor who buys and holds a broad equity index over the same window. Over 2014–2025, Table 2 shows that SPY (S&P 500) delivers a cumulative return of 302.03% (wealth multiplier 1 + 3.0203 = 4.0203) while QQQ (Nasdaq–100) delivers 542.53% (wealth multiplier 1 + 5.4253 = 6.4253). By contrast, our pairs trading specifications produce cumulative returns between 449.48% and 1020.34%, corresponding to wealth multipliers between 5.4948 and 11.2034…

In summary, this paper shows that a simple, transparent pairs trading framework applied to ETFs using daily data and clustering techniques can achieve strong risk-adjusted performance, with Sharpe ratios between 1.72 and 2.49. The strategy produces cumulative returns between 449% and 1020%, outperforming passive benchmarks such as SPY and QQQ over the 2014–2025 period.

We find the results noteworthy, particularly given that the method is applied to liquid ETFs. As is well known, compared with single-name stocks, ETFs typically carry lower risk and therefore lower expected returns. Nevertheless, the proposed method delivers strong risk-adjusted returns without the use of leverage.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Fuwen Gan and Ramy Mizrachi, Cluster-Based Pairs Trading: Combining Unsupervised Learning with Cointegration Filtering, 2025, Rice University

Post Source Here: Improving Pairs Trading with Cluster-Based Pair Selection

source https://harbourfronts.com/improving-pairs-trading-cluster-based-pair-selection/

Evaluating a Logistic Regression Trading Framework

Regression is one of the oldest predictive methods used in finance and remains widely applied today. Reference [1] revisits this “simple” approach by employing logistic regression, which is particularly suited for modeling binary outcomes, such as whether an asset’s price will increase or decrease.

The author uses cumulative returns over the past 20 days and the past 12 months as predictive variables, capturing short-term and long-term momentum effects. Logistic regression is then applied to classify whether a stock’s return in the upcoming month exceeds that month’s median return. The procedure is implemented on S&P 500 stocks from January 1985 to July 2024 using survivorship-bias-free data.

The author pointed out,

The LRST strategy’s annualized return of 24.61% over the historical period indicates its potential for generating excess returns compared to the S&P 500. However, the accompanying high volatility, as evidenced by an annualized standard deviation of 26.11%, suggests that the strategy is not without significant risk. The Sharpe ratio of 0.7738, while indicative of positive risk-adjusted returns, falls below the optimal threshold of 1, highlighting the need for further refinement to enhance the strategy’s risk-return profile…

Moreover, the recent struggles of the LRST strategy could be attributed to several factors, including the rise of algorithmic trading and macroeconomic shifts that it may not have been designed to capture. This observation resonates with the work of Liu et al. [40], which emphasizes the importance of adapting trading strategies to changing market conditions to maintain profitability. The inability of the LRST strategy to capitalize on market gains during a period of strong growth suggests potential structural weaknesses that [glossary_exclude]warrant [/glossary_exclude]further investigation.

In short, the paper shows that the logistic regression-based strategy delivers an annualized return of 24.61%, outperforming the S&P 500, but its high volatility and Sharpe ratio of 0.77 indicate substantial risk and room for improvement in its risk-return profile. Its recent underperformance may reflect structural weaknesses amid the rise of algorithmic trading and shifting macroeconomic conditions, underscoring the need for adaptation.

This article is insightful as it demonstrates that,

• Even a basic regression framework can serve as a useful predictive tool within a trading system, although further refinement is necessary.
• There might be structural changes in market dynamics, driven by the increasing prevalence of algorithmic trading and artificial intelligence, implying that traders must adapt accordingly.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Conrad O. Voigt, Logistic Regression-Based Systematic Trading: Performance on the S&P 500, 2026, github.io

Originally Published Here: Evaluating a Logistic Regression Trading Framework

source https://harbourfronts.com/evaluating-logistic-regression-trading-framework/