# Professional Backtesting Layers

The application now routes historical data through `data.py`, transaction
costs and fills through `execution.py`, and multi-asset allocation through
`portfolio.py`. Yahoo Finance is not used by the application data path.

## Data providers

Select a provider in the desktop sidebar or use `DataRequest(provider=...)`.

| Provider | Provider key | Configuration |
| --- | --- | --- |
| Polygon | `polygon` | `POLYGON_API_KEY` |
| Tiingo | `tiingo` | `TIINGO_API_KEY` |
| Alpaca | `alpaca` | `ALPACA_API_KEY`, `ALPACA_SECRET_KEY` |
| Interactive Brokers | `interactive_brokers` or `ib` | TWS/IB Gateway plus optional `ib_insync` package |
| Binance | `binance` | Public market-data API; no key required |
| Coinbase | `coinbase` | Public Exchange candle API; no key required |
| Nasdaq Data Link / Quandl | `nasdaq` or `quandl` | `NASDAQ_DATA_LINK_API_KEY` |
| CSV | `csv` | `csv_path`; `{symbol}` templates are supported |

`auto` is the default. It uses Coinbase for crypto and the first configured
professional equity provider in this order: Polygon, Tiingo, Alpaca,
Interactive Brokers, Nasdaq Data Link. It does not fall back to Yahoo.

CSV files need a datetime column (`date`, `datetime`, `time`, or `timestamp`)
and `open`, `high`, `low`, `close`, and `volume`. Optional columns include
`adj_close`, `dividends`, `splits`, `bid`, `ask`, and `funding_rate`.

```python
from data import DataRequest, get_data_manager

bundle = get_data_manager().fetch(DataRequest(
    symbols=["AAPL", "MSFT"],
    start="2020-01-01",
    end="2025-01-01",
    provider="polygon",
    interval="1d",
    adjusted=True,
    strict_quality=True,
))
```

Every fetch is cached as Parquet under `BACKTESTER_CACHE_DIR` or, by default,
`~/.backtester/cache`. Reports check duplicate bars, possible missing candles,
timezone normalization, invalid OHLC bounds, missing values, possible splits,
dividend/split metadata, and adjusted/raw price status.

The quality report always warns when equity research is not explicitly marked
as point-in-time (`survivorship_safe=True`). A clean price series cannot by
itself remove survivorship bias.

## Execution

`ExecutionEngine` supports partial fills, volume participation limits,
bid/ask spread, fixed slippage, square-root market impact, shortability,
borrow fees, leverage and margin limits, futures contract multipliers,
crypto funding, and pluggable commissions.

```python
from execution import (
    CallableCommission, ExecutionConfig, InstrumentSpec, SquareRootImpact,
)

commission = CallableCommission(
    lambda quantity, price, instrument: max(0.25, abs(quantity) * 0.002)
)
config = ExecutionConfig(
    commission_model=commission,
    slippage_model=SquareRootImpact(coefficient_bps=12),
    max_volume_participation=0.05,
)
future = InstrumentSpec.future("ES", contract_multiplier=50, margin_rate=0.10)
```

The desktop strategy run applies a Backtrader adapter for commissions,
slippage, borrow interest, futures margin/multipliers, liquidity limits, and
exposure-aware sizing. Use the standalone engine when exact per-order dynamic
market impact or funding cash flows are required.

## Portfolio backtesting

Enter comma-separated symbols and click **Run Portfolio**. Portfolio setup is
split into selection/weighting, exposure, risk, and rebalance tabs. The engine
uses a two-stage process: first it filters and ranks the universe, then it
constructs constrained weights. Available weighting methods include equal
weight, inverse volatility, risk parity, minimum variance, maximum Sharpe,
fractional Kelly, and momentum.

The selected Backtrader strategy is run independently on every symbol first.
Its long/flat/short position state gates the portfolio universe. Strategy state
changes trigger a portfolio target update on the following bar, while the
allocation method decides how capital is divided among currently active
signals. This preserves compatibility with the existing strategy catalog,
whose strategies are predominantly single-feed implementations.

The **Run Strategy** action also treats comma-separated symbols independently:
each symbol receives its own strategy instance, broker, equity curve, trade
log, and price/position chart row. It does not attach multiple feeds to one
legacy strategy instance.

The allocation pipeline is:

1. Raw model weights and short rules.
2. Liquidity, volatility, trend, ranking, and top-N selection.
3. Position caps with redistribution toward target gross exposure.
4. Sector and correlation controls.
5. Cost checks and final constrained weights.
6. Optional volatility scaling, which may intentionally retain more cash.

Target gross and net exposure are independent. If constraints make a target
impossible, the allocator preserves the hard limits and records the shortfall
in `PortfolioResult.diagnostics`; it never silently violates a cap. For
example, two assets with a 25% position cap can legally reach only 50% gross.

Rebalances are submitted to `ExecutionEngine`, so volume participation,
partial fills, rejected orders, lot sizes, minimum notional, commissions,
slippage, market impact, cash reserves, margin, borrow fees, and crypto
funding affect actual positions and cash. Unfilled target quantities remain
pending and are retried on later bars.

Sector limits and fixed allocations are available through the Python API:

```python
from execution import InstrumentSpec
from portfolio import PortfolioBacktester, PortfolioConfig, PortfolioConstraints

instruments = {
    "AAPL": InstrumentSpec("AAPL", sector="technology"),
    "MSFT": InstrumentSpec("MSFT", sector="technology"),
    "XOM": InstrumentSpec("XOM", sector="energy"),
}
config = PortfolioConfig(
    allocation="risk_parity",
    ranking_method="momentum",
    top_n_assets=5,
    rebalance="M",
    rebalance_timing="next_open",
    rebalance_threshold=0.01,
    cash_buffer=0.05,
    benchmark="equal_weight",
    constraints=PortfolioConstraints(
        target_gross_exposure=0.95,
        net_exposure_target=0.95,
        max_position_weight=0.35,
        max_gross_exposure=1.0,
        max_net_exposure=1.0,
        sector_limits={"technology": 0.50},
        max_pairwise_correlation=0.90,
        volatility_target=0.15,
    ),
)
result = PortfolioBacktester(bundle.frames, instruments, config).run()
```

## Validation

Run the focused regression suite with:

```powershell
python -m pytest -q
```