Detecting Regimes in the Volatility Surface Using Clustering

Regime identification is important in portfolio and risk management. There are many ways to classify market regimes, for example, based on market direction, such as bullish, bearish, or sideways. Another common classification is based on volatility regimes, such as high or low volatility. Most existing methods for detecting volatility regimes rely on single-point data, such as implied volatility indices or realized volatility measures.

Reference [1] proposes a new regime classification approach based on the entire volatility surface. The method first calculates local gradients, defined as the partial derivatives of implied volatility with respect to moneyness and maturity. These gradient changes are then clustered using an unsupervised algorithm to identify recurring structural transformations of the volatility surface.

The author pointed out,

My goal was to analyze structural changes in the implied volatility surface and study how the surface evolves over time. In this study, I have successfully developed a methodology to represent and quantify daily structural changes in the IV surface using local gradients. I have also used unsupervised clustering algorithm to identify distinct types of surface transformations. I have also included interpretations for some of the clusters in the previous section.

My analysis revealed that there is a number of distinct types of surface transformations that can be identified and interpreted. As expected, the vast majority of daily IV surface changes were classified as noise because structural changes in the volatility surface do not occur often. Therefore, the identified clusters have sizes of up to 18 samples. Notably, several clusters represented specific skew or term structure dynamics for different levels of maturity and moneyness.

In summary, the paper demonstrated that the clusters correspond to specific structural changes in skew and term structure rather than random fluctuations. For example:

• Cluster 1: changes mainly affect short-term maturities (1–2 months) and alter the slope across moneyness, indicating localized movements in the front of the surface.
• Cluster 2: shows term-structure rotation, where longer-maturity implied volatilities fall relative to short maturities for some moneyness levels, making the term structure steeper.
• Cluster 3: reflects a flattening of the skew, where higher-moneyness volatilities decrease relative to lower-moneyness ones, interpreted as increased demand for downside protection (more bearish sentiment).

This paper is exploratory in nature, as it does not prove the economic benefits of using the proposed volatility surface clustering technique. However, it points to an important research direction: volatility regime detection may benefit from analyzing the full volatility surface rather than relying on single volatility indicators, and the paper proposes a practical framework for doing so.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Dzhafarov, Shakhin (2025). Detecting Structural Evolution of Implied Volatility Surface Using Gradient-Based Features: A Machine Learning Approach to Market Regime Detection. Master’s thesis, Aalto University.

Post Source Here: Detecting Regimes in the Volatility Surface Using Clustering

source https://harbourfronts.com/detecting-regimes-volatility-surface-using-clustering/

Retail Options Trading and Gambling Behavior

Options trading volume has risen sharply in recent years, and a significant portion of this increase is attributed to the growing participation of retail traders. We have discussed retail options trading behavior in previous posts. Reference [1] continues this line of research by examining how retail investors trade stock options and how their attention influences options market activity.

The study posits that retail investors often treat stock options similarly to gambling. The authors construct a Search Volume Index (SVI), which measures the intensity of Google searches for specific keywords across U.S. states, and use it as a proxy for retail investors’ attention to stock options, capturing surges in public interest around firm-specific news such as earnings announcements. They pointed out,

This paper examines how regional gambling propensity relates to retail participation in U.S. options markets. Using state-level Google search data, gambling measures, and regulatory and event-based shocks, we analyze whether gambling intensity is associated with option attention and speculative trading behavior.

We document that option attention is higher in gambling-prone states, particularly around salient events such as earnings announcements. We also show that option search intensity is positively associated with brokerage-related search activity, consistent with attention translating into trading-related behavior…

We further find that lottery-like option characteristics—including out-of-the-money contracts, short maturities, and high implied volatility—receive greater attention in high-gambling states. These results suggest that regional gambling propensity helps explain cross-state variation in speculative option demand.

Finally, we relate option attention to household credit outcomes. Elevated option attention in gambling-prone states is associated with higher short-term borrowing and increased delinquency rates. While these associations do not establish household-level causality, they indicate that gambling-motivated financial attention coincides with measures of financial vulnerability at the state level.

Overall, the findings highlight how regional behavioral traits interact with financial market structure to shape retail participation in derivative markets. Future research may examine whether similar dynamics arise in other highly leveraged products or emerging speculative venues.

In short, this paper shows that retail option trading is higher in U.S. states with a stronger gambling culture, especially around earnings announcements when uncertainty is high. It also finds that this gambling-motivated attention increases trading in short-dated out-of-the-money options and is associated with higher implied volatility and higher household debt.

This paper provides additional and interesting insights into retail options behavior. Let us know what you think in the comments below or in the discussion forum.

References

[1] Matthew Flynn, Yifan Liu, Ivilina Popova, Do retail traders gamble on stock options? Journal of Financial Markets, 2026

Originally Published Here: Retail Options Trading and Gambling Behavior

source https://harbourfronts.com/retail-options-trading-gambling-behavior/

Entropy-Based Regime Detection of Tail Risks

Identifying market regimes is particularly important in portfolio and risk management. Typically, markets are classified as bullish or bearish, or as being in high- or low-volatility regimes.

Reference [1] proposes an alternative classification by distinguishing between “normal” and heavy-tailed regimes. Specifically, the study develops a nonparametric method to detect financial market regimes using differential entropy rather than volatility alone. The underlying idea is that while volatility measures dispersion, entropy captures the full distributional uncertainty, including tail behavior, which becomes particularly important during crisis periods.

The authors estimate entropy using a kernel density estimator with a heavy-tailed kernel in rolling windows and compare entropy with variance. When markets behave approximately Gaussian, i.e., normally, entropy and variance move together; during turbulent periods, the relationship breaks down, revealing heavy-tailed regimes that volatility alone cannot identify.

They pointed out,

This study demonstrates that differential entropy estimation with a heavy-tailed kernel provides an effective, nonparametric framework for identifying financial market regimes beyond traditional variance-based measures. By integrating entropy and tail-index analysis within a moving-window kernel density approach, the method captures dynamic shifts in distributional behavior without relying on parametric assumptions, offering a flexible tool for regime detection…

Empirically, applying the method to four major stock indices (Ibovespa, S&P 500, Nikkei, and SSE Composite) revealed that heavy-tailed regimes align with well-known episodes of market turbulence, including the Dot-com Bubble, the Global Financial Crisis, the COVID-19 shock, and the recent tariff-related crisis. In contrast, Gaussian regimes correspond to periods of relative stability and market efficiency.

Importantly, we showed that variance and entropy do not need to move in tandem. While volatility quantifies dispersion, entropy captures broader uncertainty and tail risk, remaining well-defined even when higher-order moments fail to exist. This divergence underscores the limitations of moment-based measures and highlights the potential of entropy as a complementary indicator of systemic instability.

In short, the paper developed a regime detection method based on entropy, which provides an alternative regime indicator that captures tail risk and structural shifts that standard volatility measures may miss.

Applying the method to Ibovespa, S&P 500, Nikkei, and SSE shows that detected heavy-tailed regimes coincide with major crises such as the Dot-com crash, the Global Financial Crisis, COVID-19, and other stress events, while Gaussian regimes correspond to calmer market periods.

This represents an important contribution to the literature, particularly in the context of managing tail risks and risk management more broadly.

Let us know what you think in the comments below or in the discussion forum.

References

[1]  Raul Matsushit, Iuri Nobre, Sergio Da Silva, Beyond volatility: Using differential entropy to detect financial market regimes, Chaos, Solitons and Fractals 202 (2026) 117553

Post Source Here: Entropy-Based Regime Detection of Tail Risks

source https://harbourfronts.com/entropy-based-regime-detection-tail-risks/

Improving Pairs Trading with Cluster-Based Pair Selection

Pairs trading is a classic quantitative trading strategy. Despite its widespread use, it continues to attract research attention. A recent line of research focuses on grouping underlyings into clusters with similar characteristics. We recently discussed such grouping using fundamental metrics.

Reference [1] also attempts to improve the pair selection process, but this time using a purely quantitative, unsupervised machine learning approach. Specifically, it groups the underlyings using density-based methods applied to the correlation-distance matrix. The objective is to identify neighborhoods of underlying that exhibit similar residual return dynamics, that is, comparable co-movement patterns after removing broad market effects, so that cointegration tests and mean-reversion diagnostics are applied primarily within economically coherent groups.

The authors pointed out,

The performance we obtain is encouraging given the constraints of our design: we use only daily adjusted closes, transparent mean-reversion filters, and simple z-score entry/exit rules without any intraday microstructure signals. Across the four clustering specifications in Table 2, annualized Sharpe ratios range from 1.7228 to 2.4935. The best risk-adjusted variant is OPTICS with PC1 removed (Sharpe 2.4935), which also achieves the smallest maximum drawdown (-8.74%). At the same time, the highest terminal wealth is achieved by HDBSCAN with PC1 removed (cumulative return 1020.34%), albeit with materially larger downside risk (max drawdown -28.39%). Overall, the results suggest that a fully reproducible, daily-data pipeline can still recover substantial relative-value structure in a broad ETF universe.

A natural benchmark is a passive investor who buys and holds a broad equity index over the same window. Over 2014–2025, Table 2 shows that SPY (S&P 500) delivers a cumulative return of 302.03% (wealth multiplier 1 + 3.0203 = 4.0203) while QQQ (Nasdaq–100) delivers 542.53% (wealth multiplier 1 + 5.4253 = 6.4253). By contrast, our pairs trading specifications produce cumulative returns between 449.48% and 1020.34%, corresponding to wealth multipliers between 5.4948 and 11.2034…

In summary, this paper shows that a simple, transparent pairs trading framework applied to ETFs using daily data and clustering techniques can achieve strong risk-adjusted performance, with Sharpe ratios between 1.72 and 2.49. The strategy produces cumulative returns between 449% and 1020%, outperforming passive benchmarks such as SPY and QQQ over the 2014–2025 period.

We find the results noteworthy, particularly given that the method is applied to liquid ETFs. As is well known, compared with single-name stocks, ETFs typically carry lower risk and therefore lower expected returns. Nevertheless, the proposed method delivers strong risk-adjusted returns without the use of leverage.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Fuwen Gan and Ramy Mizrachi, Cluster-Based Pairs Trading: Combining Unsupervised Learning with Cointegration Filtering, 2025, Rice University

Post Source Here: Improving Pairs Trading with Cluster-Based Pair Selection

source https://harbourfronts.com/improving-pairs-trading-cluster-based-pair-selection/

Evaluating a Logistic Regression Trading Framework

Regression is one of the oldest predictive methods used in finance and remains widely applied today. Reference [1] revisits this “simple” approach by employing logistic regression, which is particularly suited for modeling binary outcomes, such as whether an asset’s price will increase or decrease.

The author uses cumulative returns over the past 20 days and the past 12 months as predictive variables, capturing short-term and long-term momentum effects. Logistic regression is then applied to classify whether a stock’s return in the upcoming month exceeds that month’s median return. The procedure is implemented on S&P 500 stocks from January 1985 to July 2024 using survivorship-bias-free data.

The author pointed out,

The LRST strategy’s annualized return of 24.61% over the historical period indicates its potential for generating excess returns compared to the S&P 500. However, the accompanying high volatility, as evidenced by an annualized standard deviation of 26.11%, suggests that the strategy is not without significant risk. The Sharpe ratio of 0.7738, while indicative of positive risk-adjusted returns, falls below the optimal threshold of 1, highlighting the need for further refinement to enhance the strategy’s risk-return profile…

Moreover, the recent struggles of the LRST strategy could be attributed to several factors, including the rise of algorithmic trading and macroeconomic shifts that it may not have been designed to capture. This observation resonates with the work of Liu et al. [40], which emphasizes the importance of adapting trading strategies to changing market conditions to maintain profitability. The inability of the LRST strategy to capitalize on market gains during a period of strong growth suggests potential structural weaknesses that [glossary_exclude]warrant [/glossary_exclude]further investigation.

In short, the paper shows that the logistic regression-based strategy delivers an annualized return of 24.61%, outperforming the S&P 500, but its high volatility and Sharpe ratio of 0.77 indicate substantial risk and room for improvement in its risk-return profile. Its recent underperformance may reflect structural weaknesses amid the rise of algorithmic trading and shifting macroeconomic conditions, underscoring the need for adaptation.

This article is insightful as it demonstrates that,

• Even a basic regression framework can serve as a useful predictive tool within a trading system, although further refinement is necessary.
• There might be structural changes in market dynamics, driven by the increasing prevalence of algorithmic trading and artificial intelligence, implying that traders must adapt accordingly.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Conrad O. Voigt, Logistic Regression-Based Systematic Trading: Performance on the S&P 500, 2026, github.io

Originally Published Here: Evaluating a Logistic Regression Trading Framework

source https://harbourfronts.com/evaluating-logistic-regression-trading-framework/

Integrating Fundamental Metrics into Pairs Trading

Statistical arbitrage is a classic quantitative trading strategy. Its core premise relies on the assumption that two similar stocks should move broadly in tandem and, when their prices diverge, are expected to converge over time. To identify similar underlyings, statistical methods such as cointegration and correlation are commonly employed.

It is widely recognized by practitioners that similarity between two stocks is not purely statistical but also fundamental. However, relatively little research has focused on systematically incorporating fundamental similarity into pair selection. Reference [1] addresses this gap by developing a pairs trading selection framework that integrates fundamental characteristics alongside statistical measures.

To identify fundamentally similar pairs, the author utilizes variables such as return on equity, differences in log sales growth over the past five years, differences in the most recently available financial leverage ratios, geographic proximity (e.g., whether firms are headquartered in the same U.S. state), and industry alignment. Each fundamental metric is then assigned a score, and a composite indicator is constructed for pair selection.

The author pointed out,

This paper introduces a novel approach to stock pair selection in pairs trading, using a multi-factor scoring model based on significant variables identified through panel regression. These variables are weighted by their regression coefficients to construct a composite score. I benchmark this inter-pair characteristic-based method against the Gatev et al. (2006) SSD approach, analyzing both extended and updated trading strategies to account for repeated pair selections. The strategy’s robustness is tested with and without the COVID-19 crisis, and all results are reported net of transaction costs.

The key finding is that the multi-factor method consistently outperforms the benchmark with higher risk-adjusted returns. This performance edge persists beyond the COVID-19 period, with a narrower gap, highlighting the scoring model’s advantage in stressful markets. Though differences between updating and extending pairs are modest, the updated strategy delivers superior returns in volatile markets despite higher transaction costs. Nonetheless, the strategy underperforms relative to passive index investing. Even with rolling estimation windows, out-of-sample performance remains challenging.

In short, this paper introduces a novel stock pair selection framework that incorporates fundamental metrics. The results show that the proposed method delivers higher risk-adjusted returns than the benchmark, particularly in volatile markets, although it underperforms passive index investing and faces challenges in out-of-sample performance.

This article represents a meaningful step toward refining statistical arbitrage by incorporating fundamental characteristics into the pair selection process.

Let us know what you think in the comments below or in the discussion forum.

References

[1]  Lukas Reichmann, Pairs trading- Selection via scoring systems, Finance Research Letters Volume 93, March 2026, 109642

Post Source Here: Integrating Fundamental Metrics into Pairs Trading

source https://harbourfronts.com/integrating-fundamental-metrics-pairs-trading/

Extreme VIX: Regime Shifts and Return Predictability

The Volatility Index (VIX) is widely regarded as a forward-looking measure of market uncertainty and investor sentiment. Although it has been extensively studied, certain aspects remain insufficiently explored.

Reference [1] investigates the implications of extreme VIX values, defined as VIX > 45, and examines two key research questions in this context.

1. Extreme implied volatility (VIX > 45) is followed by significantly positive equity returns over three-month and one-year horizons, and
2. Negative returns, elevated volatility, and deteriorating sentiment increase the probability of a VIX spike (VIX > 45)

Using U.S. data from 2008 to 2025, the authors employ linear regression, logistic regression, and GARCH(1,1) models to conduct the analysis. They pointed out,

Results show that extreme VIX spikes offer contrarian signals, with significant positive returns over three-month horizons. Logistic regression confirms significance over both three-month and one-year periods. These findings remain robust after controlling for valuation, credit spreads, PMI, sentiment ratios, and interaction effects. While GARCH captures conditional variance, it lacks forward-looking predictive power in high-stress regimes. Overall, the evidence suggests that volatility timing merits consideration as part of a tactical allocation framework. Our findings also contribute to the market efficiency debate by integrating market expectations with economic indicators for risk-aware decision-making.

In short, the paper's conclusions are as follows,

• The evidence robustly supports the contrarian effect at the 3-month horizon. At the 1-year horizon, the predictive power comes primarily from continuous volatility measures rather than the binary VIX>45 indicator.
• The results clearly demonstrate that extreme volatility episodes are statistically predictable using valuation, sentiment, and medium-term volatility measures. The evidence shows regime transitions are preceded by identifiable market conditions rather than occurring randomly.

This study makes an important contribution by formally examining phenomena long observed by practitioners and by providing results with meaningful practical implications.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Fazio, P., and Spohn, D. (2026). Predictive Signals from VIX Spikes: A Comparative Study of Linear, Logistic, and GARCH-Based Return Forecasting Models. Preprints.org.

Article Source Here: Extreme VIX: Regime Shifts and Return Predictability

source https://harbourfronts.com/extreme-vix-regime-shifts-return-predictability/

Volume Effects in Pairs Trading Performance

Volume is an important factor that has not been sufficiently studied in the literature, although increasing attention is now being devoted to its role. For example, we recently discussed the link between intraday volume and volatility.

Continuing this line of inquiry, Reference [1] revisits the age-old quantitative strategy of pairs trading and incorporates volume analysis into the framework. The author studied pairs formed from the S&P500 constituent stocks using the conitegration method. The data spans from 2005 to 2024. They pointed out,

This study investigated pairs trading strategies across market segments, parameters, weighting methods, and cost structures using U.S. equity data (2005-2024). Volume emerged as the dominant performance factor, with high-volume pairs consistently outperforming low-volume counterparts across multiple metrics, including superior returns when converging towards the historical relationship. The success of volume-based segmentation may stem partly from grouping stocks with similar trading characteristics, creating more stable price relationships. Notably, optimal parameter configurations were remarkably similar between volume deciles, demonstrating consistent underlying market dynamics across liquidity environments.

Contrary to efficient market predictions, performance improved in the testing period (2015-2024), suggesting structural market changes or institutional constraints have preserved the strategy’s effectiveness despite widespread documentation. Different weighting methodologies produced similar results, with equal weighting combining simplicity and strong performance.

In short, the paper finds that trading volume is a dominant performance driver, with high-volume pairs consistently outperforming low-volume counterparts. Contrary to efficient market predictions, strategy performance improved in the later sample period, while results remained robust across parameter choices and weighting schemes. Overall, pairs trading still remains profitable.

This represents an important contribution that may reinvigorate research on pairs trading and trading volume.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Mustafa Hussein, Can Safety and Profitability Coexist? Performance Analysis of Pairs Trading among S&P 500 Stocks, 2025, University of Gothenburg

Post Source Here: Volume Effects in Pairs Trading Performance

source https://harbourfronts.com/volume-effects-pairs-trading-performance/

State-Dependent Correlation Between the S&P 500 and the VIX

It is well known that the correlation between the S&P 500 and the VIX index is negative. In fact, arbitrage and hedging strategies have been designed around this relationship, for example, hedging vega risks with delta.

However, practitioners also recognize that this correlation can break down and even turn positive. Reference [1] addresses this issue by examining the time-varying nature of the correlation. The paper first posits and demonstrates that volatility-of-volatility (VoV) risk is distinct from volatility (Vol) risk, and then classifies the market into four regimes:

1. Low VOL risk and low VOV risk (LVOL–LVOV)
2. High VOL risk and low VOV risk (HVOL–LVOV)
3. Low VOL risk and high VOV risk (LVOL–HVOV)
4. High VOL risk and high VOV risk (HVOL–HVOV).

The authors pointed out,

We develop theoretical hypotheses on the correlations between the S&P 500 and VIX based on the VFE and two types of risk in the S&P 500–VIX pair: (1) VOL risk in the S&P 500 (i.e., the underlying asset) and (2) VOV risk in VIX (i.e., its VOL product). According to our hypothesis, the correlations between the S&P 500 and VIX do not remain constant and change across various VOL and VOV risk states. We developed a regime‐switching model with state‐contingent correlations to test our hypotheses.

Our empirical results are consistent with the following notions. First, shocks that drive high VOL risk are not necessarily identical to those that drive VOV risk and thus VOL and VOV do not respond to shocks in the same way. These results imply a four‐state VOL system: high/low VOL risk and high/low VOV risk pairs. Second, the minimum inverse correlation between the S&P 500 and VIX is observed under a high VOL–low VOV risk state. In contrast, the maximum inverse correlation between the S&P 500 and VIX occurs under a high VOL–high VOV risk state. Third, the proposed state‐contingent correlations prove more effective in portfolio construction than conventional time‐dependent correlations.

In short, the paper develops a regime-switching model to show that the relationship between the S&P 500 and the VIX varies across distinct volatility (VOL) and volatility-of-volatility (VOV) risk states. It identifies a four-state system and demonstrates that correlation strength depends on the joint VOL–VOV regime.

This is an important contribution, as it quantifies the dynamic relationship between the S&P 500 and the VIX, thereby providing useful guidance for refining hedging and arbitrage strategies.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Leon Li, Carl R. Chen, Volatility Risk and Volatility-of-Volatility Risk: State-Dependent Correlations Between VIX and the S&P 500 Stock Index and Hedging Effectiveness, Journal of Futures Markets, 2025; 1–20

Post Source Here: State-Dependent Correlation Between the S&P 500 and the VIX

source https://harbourfronts.com/state-dependent-correlation-between-the-sp-500-and-the-vix/

Multifractality and Its Underlying Drivers in Cryptocurrency Markets

Cryptocurrencies, like other financial time series, can be analyzed using traditional time series and econometric methods. However, they present additional challenges due to their distinctive characteristics, including extreme volatility, heavy-tailed distributions, and long-range temporal correlations, which [glossary_exclude]warrant [/glossary_exclude]specialized examination.

Reference [1] analyzes the multifractality characteristics of major cryptocurrencies and investigates their underlying sources.  The authors analyzed data from January 1, 2018, to December 31, 2024, and pointed out,

Taken together, these results reinforce the importance of applying multifractal frameworks to digital markets. They offer practical insights for volatility forecasting, systemic risk monitoring, and understanding the maturation of blockchain-based financial ecosystems. For example, the strong correlations between BTC and ETH on various time scales can be used in optimal portfolio construction [80]. The decay of the autocorrelation function, which indicates the average size of the volatility cluster [37], may also be used in risk management. The consistency of the log-return tail distributions with the inverse cubic power-law allows for the selection of appropriate distributions in various risk mitigation methods, such as Value-at-Risk. Moreover, the greater hierarchical dependence of correlations at the level of larger fluctuations compared to smaller ones, documented by left-sided asymmetry of the multifractal spectra, may potentially allow for extending risk management strategies to include fluctuation-specific correlation patterns. The direct link between multifractality and risk management is an interesting subject for further studies.

Briefly, the article finds that digital asset markets, including BTC, ETH, DEX trading, and NFTs, exhibit clear multifractal scaling, but this multifractality is driven primarily by temporal correlations, not by heavy-tailed return distributions alone.  It also shows strong multifractal cross-correlations between BTC and ETH, especially during large fluctuations. Overall, the study argues that genuine market complexity in crypto stems from persistent correlation structure rather than from fat tails alone.

This study provides important insights into cryptocurrency dynamics, enabling portfolio and risk managers to design more appropriate trading and risk management strategies.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Drozdz, S.; Kluszczynski. R., Kwapien. J.: Watorek. M., Multifractality and its sources in the digital currency market. Future Internet, 2022, 1, 0.

Article Source Here: Multifractality and Its Underlying Drivers in Cryptocurrency Markets

source https://harbourfronts.com/multifractality-underlying-drivers-cryptocurrency-markets/