Detecting Market Regimes with a Two-State Markov Switching Model in Python
Financial markets alternate between different statistical environments, often referred to as regimes. A common pattern is a shift between low-volatility periods (stable or bullish conditions) and high-volatility periods (turbulent or bearish conditions). Identifying these regimes allows traders, risk managers, and portfolio allocators to adapt strategies to changing risk profiles.
A statistical framework well suited for this purpose is the Markov Switching Model (MSM).
1. Theoretical Framework
Let \(y_t\) denote the return at time \(t\), and \(S_t\) be an unobserved state variable taking one of \(k\) discrete values. In a two-state specification (\(k=2\)):
\[ S_t \in \{0, 1\} \]
\[ y_t = \mu_{S_t} + \varepsilon_t, \quad \varepsilon_t \sim N(0, \sigma_{S_t}^2) \]
where:
- \(\mu_{S_t}\) is the mean return in state \(S_t\)
- \(\sigma_{S_t}^2\) is the variance in state \(S_t\)
The state process \(\{S_t\}\) follows a first-order Markov chain:
\[ P_{ij} = \Pr(S_t = j \mid S_{t-1} = i), \quad i,j \in \{0, 1\} \]
The transition matrix is:
\[ P = \begin{bmatrix} P_{00} & 1 - P_{00} \\ 1 - P_{11} & P_{11} \end{bmatrix} \]
Estimation provides:
- Regime-specific parameters \((\mu_0, \mu_1, \sigma_0^2, \sigma_1^2)\)
- Transition probabilities \(P_{00}, P_{11}\)
- Smoothed probabilities \(\Pr(S_t = s \mid \text{all data})\) for each regime
2. Practical Applications
- Risk management: reduce exposure when high-volatility probability exceeds a set threshold
- Strategy timing: adapt rules to regime-dependent statistics
- Forecasting: improve short-term return or volatility predictions by accounting for regime shifts
3. Python Implementation
The script below downloads daily prices from Yahoo Finance, fits a two-state MSM on log returns, labels high- and low-volatility regimes, and produces three key outputs: returns with high-volatility shading, smoothed probability of high-volatility, and an equity curve comparison between buy-and-hold and a probability-filtered strategy.
import warnings
warnings.filterwarnings("ignore")
import os
import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt
from statsmodels.tsa.regime_switching.markov_regression import MarkovRegression
# Configuration
TICKER = "SPY" # Change to QQQ, AAPL, BTC-USD, etc.
START = "2005-01-01"
PROB_THRESHOLD = 0.60
OUT_DIR = "msm_output"
os.makedirs(OUT_DIR, exist_ok=True)
# Data loader
def load_data(ticker, start):
df = yf.download(ticker, start=start, progress=False, auto_adjust=True)
px = df["Close"]
rets = np.log(px).diff().dropna()
rets.name = "ret"
return px, rets
# Model fitting
def fit_msm(returns):
model = MarkovRegression(
endog=returns,
k_regimes=2,
trend="c",
switching_variance=True
)
res = model.fit(em_iter=10, search_reps=20, disp=False)
return res
# Regime labeling
def label_regimes(returns, smoothed_probs):
hard_state = smoothed_probs.idxmax(axis=1)
stats = returns.groupby(hard_state).agg(["mean", "std"])
high_vol = stats["std"].idxmax()
low_vol = 1 - high_vol
return hard_state, stats, low_vol, high_vol
# Strategy
def regime_strategy(returns, smoothed_probs, low_vol_regime, threshold):
p_low = smoothed_probs[low_vol_regime]
signal = (p_low >= threshold).astype(int)
strat_rets = returns * signal
return strat_rets
# Plot returns with regimes
def plot_returns_with_regimes(returns, hard_state, high_vol):
fig, ax = plt.subplots(figsize=(12,5))
ax.plot(returns.index, returns.values, lw=0.7)
ax.set_title(f"{TICKER} Returns with High-Volatility Regimes Shaded")
ax.grid(True, alpha=0.3)
in_high = (hard_state == high_vol)
blocks, start = [], None
for t, flag in in_high.items():
if flag and start is None:
start = t
elif not flag and start is not None:
blocks.append((start, t))
start = None
if start is not None:
blocks.append((start, in_high.index[-1]))
for s, e in blocks:
ax.axvspan(s, e, alpha=0.15)
fig.savefig(os.path.join(OUT_DIR, "returns_regimes.png"), dpi=150)
# Plot smoothed probabilities
def plot_probs(smoothed_probs, high_vol):
fig, ax = plt.subplots(figsize=(12,4))
ax.plot(smoothed_probs.index, smoothed_probs[high_vol], lw=1.0)
ax.set_title("Probability of High-Volatility Regime")
ax.set_ylim(-0.02, 1.02)
ax.grid(True, alpha=0.3)
fig.savefig(os.path.join(OUT_DIR, "prob_highvol.png"), dpi=150)
# Plot equity curves
def plot_equity_curves(returns, strat_returns):
bh = (1 + returns).cumprod()
st = (1 + strat_returns).cumprod()
fig, ax = plt.subplots(figsize=(12,5))
ax.plot(bh.index, bh.values, label="Buy & Hold")
ax.plot(st.index, st.values, label="Regime-Filtered")
ax.set_title("Equity Curves")
ax.grid(True, alpha=0.3)
ax.legend()
fig.savefig(os.path.join(OUT_DIR, "equity_curves.png"), dpi=150)
# Main execution
def main():
px, rets = load_data(TICKER, START)
res = fit_msm(rets)
print(res.summary())
smoothed_probs = res.smoothed_marginal_probabilities
smoothed_probs.columns = [int(c) for c in smoothed_probs.columns]
hard_state, stats, low_vol, high_vol = label_regimes(rets, smoothed_probs)
print("\nRegime statistics (mean, std):\n", stats)
plot_returns_with_regimes(rets, hard_state, high_vol)
plot_probs(smoothed_probs, high_vol)
strat_rets = regime_strategy(rets, smoothed_probs, low_vol, PROB_THRESHOLD)
plot_equity_curves(rets, strat_rets)
summary = pd.DataFrame({
"Total Return": [(1 + rets).prod() - 1, (1 + strat_rets).prod() - 1],
"Annualized Volatility": [rets.std() * np.sqrt(252), strat_rets.std() * np.sqrt(252)]
}, index=["Buy & Hold", "Regime-Filtered"])
print("\nPerformance Summary:\n", summary)
if __name__ == "__main__":
main()4. Results
Markov Switching Model Results
==============================================================================
Dep. Variable: ret No. Observations: 5182
Model: MarkovRegression Log Likelihood 16777.492
Date: Sat, 09 Aug 2025 AIC -33542.985
Time: 02:34:34 BIC -33503.667
Sample: 0 HQIC -33529.229
- 5182
Covariance Type: approx
Regime 0 parameters
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const -0.0013 0.001 -2.238 0.025 -0.002 -0.000
sigma2 0.0004 2.19e-05 19.488 0.000 0.000 0.000
Regime 1 parameters
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
const 0.0010 0.000 8.573 0.000 0.001 0.001
sigma2 4.489e-05 1.71e-06 26.189 0.000 4.15e-05 4.82e-05
Regime transition parameters
==============================================================================
coef std err z P>|z| [0.025 0.975]
------------------------------------------------------------------------------
p[0->0] 0.9558 0.008 117.083 0.000 0.940 0.972
p[1->0] 0.0156 0.003 5.283 0.000 0.010 0.021
==============================================================================
Regime stats (realized on hard assignments):
mean std count
state
0 -0.001445 0.021012 1313
1 0.001020 0.006665 3869
Low-vol regime: 1 | High-vol regime: 0
Performance summary (no costs, illustrative):
Total Return Ann. Vol Min Drawdown
Buy & Hold 4.3090 0.1919 -0.5958
Regime-Filtered 36.6696 0.0897 -0.1015
Returns with high-volatility shading The shaded periods correspond to the state with the highest estimated variance, highlighting market stress phases.
Smoothed probability plot This shows the model’s inferred probability of being in the high-volatility regime. Values close to 1 indicate strong evidence of elevated volatility.
Equity curve comparison The regime-filtered strategy invests only when the probability of the low-volatility regime exceeds the threshold. This approach can help manage drawdowns but may reduce returns if regime shifts are misclassified.
5. Key Points
The two-state MSM is a practical tool for uncovering hidden market conditions from historical returns. It provides both classification of past regimes and estimates of regime persistence through transition probabilities. These probabilities can serve as inputs to risk controls or dynamic allocation rules. Proper validation, including out-of-sample testing and accounting for transaction costs, is essential before operational use.