The Impact of Retail Options Trading on the Implied Volatility Surface

Retail options trading is rising rapidly, driven by factors such as the growth of retail brokers, the popularity of social media, and more flexible working hours. Alongside this trend, there has been an increased interest in research on retail options trading behavior. We recently discussed retail options trading and its link to gambling behavior.

Reference [1] continues this line of research by examining how retail trading reshapes the implied volatility (IV) surface dynamics. The authors utilize OPRA and Nasdaq data for this study. To isolate the effect of retail options trading, they apply a difference-in-differences approach around retail broker outages, 82 events from 2019 to 2021, comparing implied volatility between high-retail and low-retail stocks, during versus pre-outage periods.

The authors pointed out,

Option market volume has dramatically increased in recent years, driven by an influx of retail investors. Our analysis reveals that retail option trading is particularly concentrated in call, short-dated, and out-of-the-money call options. We examine the effects of retail trading on option market volume and implied volatility, using brokerage outages as exogenous shocks to retail option trading. Our findings indicate that net buying volume by retail investors significantly declines during outages for the types of options they prefer. In contrast, net buying volume rises during outages for long-dated options, consistent with retail investors tending to write rather than purchase longer maturity options.

The paper’s central finding is that retail investor demand pressure significantly impacts option implied volatility. We find that implied volatility significantly decreases during retail brokerage outages, and in particular for call, OTM, and short-dated options. In contrast, implied volatility significantly increases for long-dated options. The effect of retail brokerage outages on implied volatility aligns with the shifts in net retail volume and points towards a significant role for retail investors in shaping the IV surface. Additional robustness analysis indicates that the outage results are unique to the outage period, are not driven by a small number of the most actively traded options, and continue to hold for alternative option trading data.

In short, retail investors systematically buy short-dated, out-of-the-money (especially calls) and sell long-dated options, creating predictable pressure across the surface. When retail trading activity is reduced during brokerage outages, IV falls for short-dated and OTM options but rises for long-dated options.

This paper contributes to a better understanding of retail options trading and shows how retail traders can materially affect the implied volatility surface.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Eaton, Gregory W., T. Clifton Green, Brian S. Roseman, and Yanbin Wu (2025). Retail Option Traders and the Implied Volatility Surface. SSRN 4104788

Originally Published Here: The Impact of Retail Options Trading on the Implied Volatility Surface

source https://harbourfronts.com/impact-retail-options-trading-implied-volatility-surface/

Evaluating Machine Learning Models for S&P 500 Return Prediction

Machine learning (ML) is increasingly applied in finance. In trading in particular, researchers aim to determine whether ML can generate consistent alpha. Reference [1] revisits this question. The authors utilize a long sample dataset from 1970 to 2024 and evaluate a wide range of ML models, including linear, nonlinear, and ensemble methods, to assess whether they can predict one-day-ahead S&P 500 returns and generate trading profits.

In the models, the inputs are price, returns, volume, volatility, market breadth, and simple sentiment proxies. The models include linear methods, such as OLS and elastic net, and nonlinear methods, such as random forest, XGBoost, SVM, and kNN, along with ensemble approaches. They are trained on rolling 10-year windows with recalibration every 2 to 5 years. The trading strategy takes a long position when the forecast is positive and otherwise holds Treasury bills.

The authors pointed out,

This study examined the performance of machine learning architectures applied to financial forecasting within the framework of the adaptive market hypothesis. By comparing 12 models and three ensemble systems over 55 years of daily S&P 500 data, we demonstrated that machine learning with adaptive retraining can be used to obtain stock market profits over an extended period from 1970 to 2024. However, the extent of abnormal returns is limited by transaction costs and the fact that financial markets quickly adapt, reducing the profitability of any machine learning system. Three architectures—elastic nets, logit, and XGboost—consistently outperformed the others, while ensemble designs provided the most resilient returns and stability across changing market regimes. Nevertheless, system profitability declined after 1987, indicating that even sophisticated systems face diminishing arbitrage opportunities as markets learn and adapt. This dynamic is consistent with both adaptive market and adaptive systems perspectives.

In short, without transaction costs, many models generate positive abnormal returns. However, once realistic trading costs are incorporated, these profits largely disappear, particularly in the post-1987 period. There is also a strong regime effect, with high profitability of around 20% or more during 1970 to 1987, followed by a significant decline to approximately 5% to 9% from 1988 to 2024, often rendering the strategies unprofitable after costs.

This paper finds that while ML strategies can generate abnormal returns and remain adaptive over time, profitability declines in later periods and exhibits significant variability, raising concerns about real-time performance.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Olson, D., Nusair, S., & Galariotis, E. (2026). Profitability of Machine Learning Models in Forecasting the S&P 500 Index. Information Systems Frontiers.

Post Source Here: Evaluating Machine Learning Models for S&P 500 Return Prediction

source https://harbourfronts.com/evaluating-machine-learning-models-sp-500-return-prediction/

Detecting Regimes in the Volatility Surface Using Clustering

Regime identification is important in portfolio and risk management. There are many ways to classify market regimes, for example, based on market direction, such as bullish, bearish, or sideways. Another common classification is based on volatility regimes, such as high or low volatility. Most existing methods for detecting volatility regimes rely on single-point data, such as implied volatility indices or realized volatility measures.

Reference [1] proposes a new regime classification approach based on the entire volatility surface. The method first calculates local gradients, defined as the partial derivatives of implied volatility with respect to moneyness and maturity. These gradient changes are then clustered using an unsupervised algorithm to identify recurring structural transformations of the volatility surface.

The author pointed out,

My goal was to analyze structural changes in the implied volatility surface and study how the surface evolves over time. In this study, I have successfully developed a methodology to represent and quantify daily structural changes in the IV surface using local gradients. I have also used unsupervised clustering algorithm to identify distinct types of surface transformations. I have also included interpretations for some of the clusters in the previous section.

My analysis revealed that there is a number of distinct types of surface transformations that can be identified and interpreted. As expected, the vast majority of daily IV surface changes were classified as noise because structural changes in the volatility surface do not occur often. Therefore, the identified clusters have sizes of up to 18 samples. Notably, several clusters represented specific skew or term structure dynamics for different levels of maturity and moneyness.

In summary, the paper demonstrated that the clusters correspond to specific structural changes in skew and term structure rather than random fluctuations. For example:

• Cluster 1: changes mainly affect short-term maturities (1–2 months) and alter the slope across moneyness, indicating localized movements in the front of the surface.
• Cluster 2: shows term-structure rotation, where longer-maturity implied volatilities fall relative to short maturities for some moneyness levels, making the term structure steeper.
• Cluster 3: reflects a flattening of the skew, where higher-moneyness volatilities decrease relative to lower-moneyness ones, interpreted as increased demand for downside protection (more bearish sentiment).

This paper is exploratory in nature, as it does not prove the economic benefits of using the proposed volatility surface clustering technique. However, it points to an important research direction: volatility regime detection may benefit from analyzing the full volatility surface rather than relying on single volatility indicators, and the paper proposes a practical framework for doing so.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Dzhafarov, Shakhin (2025). Detecting Structural Evolution of Implied Volatility Surface Using Gradient-Based Features: A Machine Learning Approach to Market Regime Detection. Master’s thesis, Aalto University.

Post Source Here: Detecting Regimes in the Volatility Surface Using Clustering

source https://harbourfronts.com/detecting-regimes-volatility-surface-using-clustering/

Retail Options Trading and Gambling Behavior

Options trading volume has risen sharply in recent years, and a significant portion of this increase is attributed to the growing participation of retail traders. We have discussed retail options trading behavior in previous posts. Reference [1] continues this line of research by examining how retail investors trade stock options and how their attention influences options market activity.

The study posits that retail investors often treat stock options similarly to gambling. The authors construct a Search Volume Index (SVI), which measures the intensity of Google searches for specific keywords across U.S. states, and use it as a proxy for retail investors’ attention to stock options, capturing surges in public interest around firm-specific news such as earnings announcements. They pointed out,

This paper examines how regional gambling propensity relates to retail participation in U.S. options markets. Using state-level Google search data, gambling measures, and regulatory and event-based shocks, we analyze whether gambling intensity is associated with option attention and speculative trading behavior.

We document that option attention is higher in gambling-prone states, particularly around salient events such as earnings announcements. We also show that option search intensity is positively associated with brokerage-related search activity, consistent with attention translating into trading-related behavior…

We further find that lottery-like option characteristics—including out-of-the-money contracts, short maturities, and high implied volatility—receive greater attention in high-gambling states. These results suggest that regional gambling propensity helps explain cross-state variation in speculative option demand.

Finally, we relate option attention to household credit outcomes. Elevated option attention in gambling-prone states is associated with higher short-term borrowing and increased delinquency rates. While these associations do not establish household-level causality, they indicate that gambling-motivated financial attention coincides with measures of financial vulnerability at the state level.

Overall, the findings highlight how regional behavioral traits interact with financial market structure to shape retail participation in derivative markets. Future research may examine whether similar dynamics arise in other highly leveraged products or emerging speculative venues.

In short, this paper shows that retail option trading is higher in U.S. states with a stronger gambling culture, especially around earnings announcements when uncertainty is high. It also finds that this gambling-motivated attention increases trading in short-dated out-of-the-money options and is associated with higher implied volatility and higher household debt.

This paper provides additional and interesting insights into retail options behavior. Let us know what you think in the comments below or in the discussion forum.

References

[1] Matthew Flynn, Yifan Liu, Ivilina Popova, Do retail traders gamble on stock options? Journal of Financial Markets, 2026

Originally Published Here: Retail Options Trading and Gambling Behavior

source https://harbourfronts.com/retail-options-trading-gambling-behavior/

Entropy-Based Regime Detection of Tail Risks

Identifying market regimes is particularly important in portfolio and risk management. Typically, markets are classified as bullish or bearish, or as being in high- or low-volatility regimes.

Reference [1] proposes an alternative classification by distinguishing between “normal” and heavy-tailed regimes. Specifically, the study develops a nonparametric method to detect financial market regimes using differential entropy rather than volatility alone. The underlying idea is that while volatility measures dispersion, entropy captures the full distributional uncertainty, including tail behavior, which becomes particularly important during crisis periods.

The authors estimate entropy using a kernel density estimator with a heavy-tailed kernel in rolling windows and compare entropy with variance. When markets behave approximately Gaussian, i.e., normally, entropy and variance move together; during turbulent periods, the relationship breaks down, revealing heavy-tailed regimes that volatility alone cannot identify.

They pointed out,

This study demonstrates that differential entropy estimation with a heavy-tailed kernel provides an effective, nonparametric framework for identifying financial market regimes beyond traditional variance-based measures. By integrating entropy and tail-index analysis within a moving-window kernel density approach, the method captures dynamic shifts in distributional behavior without relying on parametric assumptions, offering a flexible tool for regime detection…

Empirically, applying the method to four major stock indices (Ibovespa, S&P 500, Nikkei, and SSE Composite) revealed that heavy-tailed regimes align with well-known episodes of market turbulence, including the Dot-com Bubble, the Global Financial Crisis, the COVID-19 shock, and the recent tariff-related crisis. In contrast, Gaussian regimes correspond to periods of relative stability and market efficiency.

Importantly, we showed that variance and entropy do not need to move in tandem. While volatility quantifies dispersion, entropy captures broader uncertainty and tail risk, remaining well-defined even when higher-order moments fail to exist. This divergence underscores the limitations of moment-based measures and highlights the potential of entropy as a complementary indicator of systemic instability.

In short, the paper developed a regime detection method based on entropy, which provides an alternative regime indicator that captures tail risk and structural shifts that standard volatility measures may miss.

Applying the method to Ibovespa, S&P 500, Nikkei, and SSE shows that detected heavy-tailed regimes coincide with major crises such as the Dot-com crash, the Global Financial Crisis, COVID-19, and other stress events, while Gaussian regimes correspond to calmer market periods.

This represents an important contribution to the literature, particularly in the context of managing tail risks and risk management more broadly.

Let us know what you think in the comments below or in the discussion forum.

References

[1]  Raul Matsushit, Iuri Nobre, Sergio Da Silva, Beyond volatility: Using differential entropy to detect financial market regimes, Chaos, Solitons and Fractals 202 (2026) 117553

Post Source Here: Entropy-Based Regime Detection of Tail Risks

source https://harbourfronts.com/entropy-based-regime-detection-tail-risks/

Improving Pairs Trading with Cluster-Based Pair Selection

Pairs trading is a classic quantitative trading strategy. Despite its widespread use, it continues to attract research attention. A recent line of research focuses on grouping underlyings into clusters with similar characteristics. We recently discussed such grouping using fundamental metrics.

Reference [1] also attempts to improve the pair selection process, but this time using a purely quantitative, unsupervised machine learning approach. Specifically, it groups the underlyings using density-based methods applied to the correlation-distance matrix. The objective is to identify neighborhoods of underlying that exhibit similar residual return dynamics, that is, comparable co-movement patterns after removing broad market effects, so that cointegration tests and mean-reversion diagnostics are applied primarily within economically coherent groups.

The authors pointed out,

The performance we obtain is encouraging given the constraints of our design: we use only daily adjusted closes, transparent mean-reversion filters, and simple z-score entry/exit rules without any intraday microstructure signals. Across the four clustering specifications in Table 2, annualized Sharpe ratios range from 1.7228 to 2.4935. The best risk-adjusted variant is OPTICS with PC1 removed (Sharpe 2.4935), which also achieves the smallest maximum drawdown (-8.74%). At the same time, the highest terminal wealth is achieved by HDBSCAN with PC1 removed (cumulative return 1020.34%), albeit with materially larger downside risk (max drawdown -28.39%). Overall, the results suggest that a fully reproducible, daily-data pipeline can still recover substantial relative-value structure in a broad ETF universe.

A natural benchmark is a passive investor who buys and holds a broad equity index over the same window. Over 2014–2025, Table 2 shows that SPY (S&P 500) delivers a cumulative return of 302.03% (wealth multiplier 1 + 3.0203 = 4.0203) while QQQ (Nasdaq–100) delivers 542.53% (wealth multiplier 1 + 5.4253 = 6.4253). By contrast, our pairs trading specifications produce cumulative returns between 449.48% and 1020.34%, corresponding to wealth multipliers between 5.4948 and 11.2034…

In summary, this paper shows that a simple, transparent pairs trading framework applied to ETFs using daily data and clustering techniques can achieve strong risk-adjusted performance, with Sharpe ratios between 1.72 and 2.49. The strategy produces cumulative returns between 449% and 1020%, outperforming passive benchmarks such as SPY and QQQ over the 2014–2025 period.

We find the results noteworthy, particularly given that the method is applied to liquid ETFs. As is well known, compared with single-name stocks, ETFs typically carry lower risk and therefore lower expected returns. Nevertheless, the proposed method delivers strong risk-adjusted returns without the use of leverage.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Fuwen Gan and Ramy Mizrachi, Cluster-Based Pairs Trading: Combining Unsupervised Learning with Cointegration Filtering, 2025, Rice University

Post Source Here: Improving Pairs Trading with Cluster-Based Pair Selection

source https://harbourfronts.com/improving-pairs-trading-cluster-based-pair-selection/

Evaluating a Logistic Regression Trading Framework

Regression is one of the oldest predictive methods used in finance and remains widely applied today. Reference [1] revisits this “simple” approach by employing logistic regression, which is particularly suited for modeling binary outcomes, such as whether an asset’s price will increase or decrease.

The author uses cumulative returns over the past 20 days and the past 12 months as predictive variables, capturing short-term and long-term momentum effects. Logistic regression is then applied to classify whether a stock’s return in the upcoming month exceeds that month’s median return. The procedure is implemented on S&P 500 stocks from January 1985 to July 2024 using survivorship-bias-free data.

The author pointed out,

The LRST strategy’s annualized return of 24.61% over the historical period indicates its potential for generating excess returns compared to the S&P 500. However, the accompanying high volatility, as evidenced by an annualized standard deviation of 26.11%, suggests that the strategy is not without significant risk. The Sharpe ratio of 0.7738, while indicative of positive risk-adjusted returns, falls below the optimal threshold of 1, highlighting the need for further refinement to enhance the strategy’s risk-return profile…

Moreover, the recent struggles of the LRST strategy could be attributed to several factors, including the rise of algorithmic trading and macroeconomic shifts that it may not have been designed to capture. This observation resonates with the work of Liu et al. [40], which emphasizes the importance of adapting trading strategies to changing market conditions to maintain profitability. The inability of the LRST strategy to capitalize on market gains during a period of strong growth suggests potential structural weaknesses that [glossary_exclude]warrant [/glossary_exclude]further investigation.

In short, the paper shows that the logistic regression-based strategy delivers an annualized return of 24.61%, outperforming the S&P 500, but its high volatility and Sharpe ratio of 0.77 indicate substantial risk and room for improvement in its risk-return profile. Its recent underperformance may reflect structural weaknesses amid the rise of algorithmic trading and shifting macroeconomic conditions, underscoring the need for adaptation.

This article is insightful as it demonstrates that,

• Even a basic regression framework can serve as a useful predictive tool within a trading system, although further refinement is necessary.
• There might be structural changes in market dynamics, driven by the increasing prevalence of algorithmic trading and artificial intelligence, implying that traders must adapt accordingly.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Conrad O. Voigt, Logistic Regression-Based Systematic Trading: Performance on the S&P 500, 2026, github.io

Originally Published Here: Evaluating a Logistic Regression Trading Framework

source https://harbourfronts.com/evaluating-logistic-regression-trading-framework/

Integrating Fundamental Metrics into Pairs Trading

Statistical arbitrage is a classic quantitative trading strategy. Its core premise relies on the assumption that two similar stocks should move broadly in tandem and, when their prices diverge, are expected to converge over time. To identify similar underlyings, statistical methods such as cointegration and correlation are commonly employed.

It is widely recognized by practitioners that similarity between two stocks is not purely statistical but also fundamental. However, relatively little research has focused on systematically incorporating fundamental similarity into pair selection. Reference [1] addresses this gap by developing a pairs trading selection framework that integrates fundamental characteristics alongside statistical measures.

To identify fundamentally similar pairs, the author utilizes variables such as return on equity, differences in log sales growth over the past five years, differences in the most recently available financial leverage ratios, geographic proximity (e.g., whether firms are headquartered in the same U.S. state), and industry alignment. Each fundamental metric is then assigned a score, and a composite indicator is constructed for pair selection.

The author pointed out,

This paper introduces a novel approach to stock pair selection in pairs trading, using a multi-factor scoring model based on significant variables identified through panel regression. These variables are weighted by their regression coefficients to construct a composite score. I benchmark this inter-pair characteristic-based method against the Gatev et al. (2006) SSD approach, analyzing both extended and updated trading strategies to account for repeated pair selections. The strategy’s robustness is tested with and without the COVID-19 crisis, and all results are reported net of transaction costs.

The key finding is that the multi-factor method consistently outperforms the benchmark with higher risk-adjusted returns. This performance edge persists beyond the COVID-19 period, with a narrower gap, highlighting the scoring model’s advantage in stressful markets. Though differences between updating and extending pairs are modest, the updated strategy delivers superior returns in volatile markets despite higher transaction costs. Nonetheless, the strategy underperforms relative to passive index investing. Even with rolling estimation windows, out-of-sample performance remains challenging.

In short, this paper introduces a novel stock pair selection framework that incorporates fundamental metrics. The results show that the proposed method delivers higher risk-adjusted returns than the benchmark, particularly in volatile markets, although it underperforms passive index investing and faces challenges in out-of-sample performance.

This article represents a meaningful step toward refining statistical arbitrage by incorporating fundamental characteristics into the pair selection process.

Let us know what you think in the comments below or in the discussion forum.

References

[1]  Lukas Reichmann, Pairs trading- Selection via scoring systems, Finance Research Letters Volume 93, March 2026, 109642

Post Source Here: Integrating Fundamental Metrics into Pairs Trading

source https://harbourfronts.com/integrating-fundamental-metrics-pairs-trading/

Extreme VIX: Regime Shifts and Return Predictability

The Volatility Index (VIX) is widely regarded as a forward-looking measure of market uncertainty and investor sentiment. Although it has been extensively studied, certain aspects remain insufficiently explored.

Reference [1] investigates the implications of extreme VIX values, defined as VIX > 45, and examines two key research questions in this context.

1. Extreme implied volatility (VIX > 45) is followed by significantly positive equity returns over three-month and one-year horizons, and
2. Negative returns, elevated volatility, and deteriorating sentiment increase the probability of a VIX spike (VIX > 45)

Using U.S. data from 2008 to 2025, the authors employ linear regression, logistic regression, and GARCH(1,1) models to conduct the analysis. They pointed out,

Results show that extreme VIX spikes offer contrarian signals, with significant positive returns over three-month horizons. Logistic regression confirms significance over both three-month and one-year periods. These findings remain robust after controlling for valuation, credit spreads, PMI, sentiment ratios, and interaction effects. While GARCH captures conditional variance, it lacks forward-looking predictive power in high-stress regimes. Overall, the evidence suggests that volatility timing merits consideration as part of a tactical allocation framework. Our findings also contribute to the market efficiency debate by integrating market expectations with economic indicators for risk-aware decision-making.

In short, the paper's conclusions are as follows,

• The evidence robustly supports the contrarian effect at the 3-month horizon. At the 1-year horizon, the predictive power comes primarily from continuous volatility measures rather than the binary VIX>45 indicator.
• The results clearly demonstrate that extreme volatility episodes are statistically predictable using valuation, sentiment, and medium-term volatility measures. The evidence shows regime transitions are preceded by identifiable market conditions rather than occurring randomly.

This study makes an important contribution by formally examining phenomena long observed by practitioners and by providing results with meaningful practical implications.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Fazio, P., and Spohn, D. (2026). Predictive Signals from VIX Spikes: A Comparative Study of Linear, Logistic, and GARCH-Based Return Forecasting Models. Preprints.org.

Article Source Here: Extreme VIX: Regime Shifts and Return Predictability

source https://harbourfronts.com/extreme-vix-regime-shifts-return-predictability/

Volume Effects in Pairs Trading Performance

Volume is an important factor that has not been sufficiently studied in the literature, although increasing attention is now being devoted to its role. For example, we recently discussed the link between intraday volume and volatility.

Continuing this line of inquiry, Reference [1] revisits the age-old quantitative strategy of pairs trading and incorporates volume analysis into the framework. The author studied pairs formed from the S&P500 constituent stocks using the conitegration method. The data spans from 2005 to 2024. They pointed out,

This study investigated pairs trading strategies across market segments, parameters, weighting methods, and cost structures using U.S. equity data (2005-2024). Volume emerged as the dominant performance factor, with high-volume pairs consistently outperforming low-volume counterparts across multiple metrics, including superior returns when converging towards the historical relationship. The success of volume-based segmentation may stem partly from grouping stocks with similar trading characteristics, creating more stable price relationships. Notably, optimal parameter configurations were remarkably similar between volume deciles, demonstrating consistent underlying market dynamics across liquidity environments.

Contrary to efficient market predictions, performance improved in the testing period (2015-2024), suggesting structural market changes or institutional constraints have preserved the strategy’s effectiveness despite widespread documentation. Different weighting methodologies produced similar results, with equal weighting combining simplicity and strong performance.

In short, the paper finds that trading volume is a dominant performance driver, with high-volume pairs consistently outperforming low-volume counterparts. Contrary to efficient market predictions, strategy performance improved in the later sample period, while results remained robust across parameter choices and weighting schemes. Overall, pairs trading still remains profitable.

This represents an important contribution that may reinvigorate research on pairs trading and trading volume.

Let us know what you think in the comments below or in the discussion forum.

References

[1] Mustafa Hussein, Can Safety and Profitability Coexist? Performance Analysis of Pairs Trading among S&P 500 Stocks, 2025, University of Gothenburg

Post Source Here: Volume Effects in Pairs Trading Performance

source https://harbourfronts.com/volume-effects-pairs-trading-performance/