back to home

projects

llm stock trader, fine-tuning & rl alignment

#deepseek 7b#unsloth#lora#trl#grpo#pytorch#hugging face

15 fine-tuning experiments (sft to grpo) on deepseek 7b with lora r=16. score improved from 0.300 to 0.537 (+79%). diagnosed 3 training failures: hold collapse, reward hacking, kl catastrophe.

/ training journey

  • 15 fine-tuning experiments (sft -> grpo) on deepseek 7b with lora r=16 (q/k/v/o), lr=5e-6
  • score: 0.300 (base) -> 0.417 (sft) -> 0.537 (grpo), +79% total improvement
  • sft trained on 12k reverse-distilled dataset, outperformed larger noisy sets
  • grpo trained against a 1.22m-param causal transformer world model as synthetic environment

/ training failures diagnosed

  • hold collapse: agent learned inaction is never penalized, 85% hold actions. fixed with data rebalancing
  • reward hacking: 84% of reward came from formatting, not trading quality. fixed with asymmetric reward decomposition
  • kl catastrophe: kl divergence hit 4.2, destroyed base model knowledge, score dropped to 0.301. fixed with kl coefficient tuning

/ dataset engineering

  • 12k reverse-distilled sft dataset from gpt-4 rollouts through the trading environment
  • outperformed larger noisy datasets — quality over quantity for domain-specific fine-tuning
  • asymmetric reward decomposition separates format compliance from trading quality to prevent spec gaming

/ how it works

01generate training data by running gpt-4 through the stock trading environment
02filter and curate 12k high-quality demonstrations for sft
03train sft checkpoint with lora r=16 on deepseek 7b (q/k/v/o targets)
04train grpo on top of sft checkpoint using neural world model as environment
05evaluate on held-out market episodes, diagnose failures, iterate on reward design

/ features

sft -> grpo pipeline
two-stage training: supervised fine-tuning on distilled demonstrations, then grpo reinforcement learning against a neural world model. each stage addresses different failure modes.
specification gaming diagnosis
documented 3 distinct training failures (hold collapse, reward hacking, kl catastrophe) with root cause analysis and fixes. each failure taught a lesson about reward design and training dynamics.
reverse distillation
12k training examples generated by running gpt-4 through the trading environment and filtering for high-scoring trajectories. smaller, cleaner dataset outperformed larger noisy alternatives.
asymmetric reward decomposition
separates format compliance reward from trading quality reward. prevents the model from gaming the reward by producing well-formatted but poor trading decisions.

stock trader rl environment

#python#fastapi#pydantic#docker#websocket#openenv#hugging face spaces

openenv-compliant reinforcement learning environment for evaluating llm trading agents on indian equity markets. meta pytorch openenv hackathon finalist (top 800 out of 32,000+ teams).

/ what it does

  • simulates daily stock trading on nifty stocks using ~5 years of real historical ohlcv data
  • agents connect via http/websocket, receive market observations with technical indicators, and respond with plain-text trade actions (buy, sell, hold)
  • three difficulty tiers: single stock (20 days), portfolio (30 days), full autonomous (40 days) with escalating constraints like transaction costs, slippage, position limits, and regime gates

/ how it works

  • market simulator replays historical price windows with a 50-day lookback buffer for indicator computation
  • feature engine computes rsi, macd, bollinger bands, volume spikes, trend, momentum, and volatility — served as human-readable text summaries for llm agents
  • step-level reward shaping: pnl reward, discipline bonus, regime gate penalty, trade limit violations
  • task-specific graders score the full trajectory on sharpe ratio, discipline, regime compliance, and risk management

/ world model

  • 1.22m-param causal transformer world model (mdn output with 3 gaussians) as synthetic training environment for grpo
  • volatility 0.94x reality, mae 0.0167 — generates realistic market episodes without replaying historical data
  • drop-in replacement for static csv replay, enabling unlimited training rollouts

/ why it matters — rlvr & grpo

  • the grading system is designed as a verifiable reward function (rlvr) — deterministic scores that replace traditional reward models
  • this enables grpo-based training: generate multiple rollouts through the environment, rank them by grader score, and update model weights to favor better trading trajectories
  • no separate reward model or critic needed — the environment's graders are the reward signal

/ how it works

01agent connects via http/websocket and resets environment with a task and seed
02environment selects a random market window and returns initial observation
03agent reads market summary with technical indicators and submits trade action
04environment executes trade with realistic costs/slippage, computes reward, advances to next day
05at episode end, grader scores the full trajectory on task-specific criteria

/ features

meta pytorch hackathon finalist
qualified for the finale (top 800 out of 32,000+ teams). presented the environment to meta engineers in bangalore, april 2026.
three difficulty tiers
single stock (easy, 20 days), portfolio (medium, 30 days), and full autonomous (hard, 40 days) with escalating constraints — transaction costs, slippage, position limits, trade caps, and regime gates.
1.22m-param causal transformer world model
mdn output with 3 gaussians generates synthetic market episodes. volatility 0.94x reality, mae 0.0167. drop-in replacement for csv replay, enabling unlimited grpo training rollouts.
verifiable reward design (rlvr)
deterministic grading functions that score agents on sharpe ratio, discipline, regime compliance, and risk management. designed to serve as verifiable rewards for grpo-based rl training — no separate reward model needed.
llm-native interface
plain-text action space (buy, sell, hold) and human-readable market summaries — any llm can act as an agent without special tooling. invalid actions gracefully default to hold.
seed-reproducible episodes
fully deterministic episodes for reproducible evaluation. same seed produces same market window and sequence.

eta prediction engine

#python#pytorch#lightgbm#pandas#mlflow#docker#hugging face hub

3-model ensemble (neural net + lightgbm + ft-transformer) for eta prediction. trained on 37m trips, achieves 253s mae — 28% better than xgboost baseline, inference under 4ms on cpu.

/ what it does

  • predicts taxi trip duration given pickup zone, dropoff zone, timestamp, and passenger count using 37 million real yellow taxi trips from 2023
  • learns all spatial relationships from trip data via zone embeddings — no external geography, shapefiles, or hardcoded coordinates. if zone ids mapped to a different city, the model would work equally well
  • serves predictions in under 4ms on cpu with a 3-model ensemble, packaged in a ~500mb docker container

/ the ensemble

  • model 1: dual-branch embedding network (560k params) — zone embeddings with hash-based pair embedding, 24 continuous features, residual blocks. best at smooth interpolation for common routes
  • model 2: lightgbm (81 trees) — gradient-boosted trees with zone ids as native categoricals. near-zero bias (-6s) on rare pairs where the neural net struggles (-106s bias)
  • model 3: ft-transformer (406k params) — implemented from scratch, each feature projected into a 128-dim token, 3-layer self-attention with [cls] aggregation. positive bias (+65s) offsets nn's negative bias
  • ensemble weights optimized via grid search on full dev set: 0.6 nn + 0.2 lgbm + 0.2 ft-transformer

/ feature engineering

  • 14 zone-pair statistics with bayesian shrinkage — smooths sparse pairs toward pickup-zone mean with a fallback hierarchy: pair -> pickup zone -> dropoff zone -> global mean
  • 6 traffic-regime time buckets (late night, early morning, am rush, midday, pm rush, evening) with per-regime pair statistics
  • 10 temporal features: cyclical hour/dow/month encoding, rush hour flags, night flags, normalized minute-of-day
  • zone-pair median alone (296.7s) beats xgboost (351s) with zero ml — the signal is in the feature engineering

/ results

  • 253s mae — 28% better than xgboost baseline (351s). ensemble reduced mae from 261s (best single model) to 253s
  • nn: 261.2s (precision on common routes) | lgbm: 261.7s (low bias on rare pairs) | ft: 284.7s (different error pattern, bias offset)
  • diagnostic-driven tuning identified rare-pair bias as the true bottleneck — no amount of nn tuning could fix it, lightgbm solved it
  • inference under 4ms per request via onnx runtime, total model weights 6.3mb (2.3 + 2.4 + 1.6)

/ how it works

01download and clean 37m taxi trips (2023), split temporally into train/dev
02compute zone-pair statistics with bayesian shrinkage across 6 traffic regimes
03train neural net (37m rows, huber loss), lightgbm (10m rows, mae), ft-transformer (10m rows, l1)
04optimize ensemble weights via grid search on full 1.23m dev set
05evaluate on held-out dev set, pick best checkpoint by mae (not training loss)

/ features

3-model ensemble
neural net + lightgbm + ft-transformer with complementary strengths. each model has a different inductive bias — embeddings vs tree splits vs self-attention. ensemble reduced mae from 261s to 253s.
ft-transformer from scratch
feature tokenizer transformer (gorishniy et al., neurips 2021). each feature projected into a 128-dim token, [cls] token aggregates via 3-layer self-attention. captures cross-feature interactions the mlp misses.
learned zone embeddings
50-dim embeddings for 266 zones learn spatial relationships purely from trip patterns. no external geography needed — model is transferable to any city with zone ids.
bayesian shrinkage for sparse pairs
handles rare and unseen zone pairs gracefully. shrinkage prior smooths toward pickup-zone mean; fallback hierarchy prevents cold-start failures.
diagnostic-driven tuning
deep diagnostics (parameter health, rare-pair analysis, regularization checks) revealed rare-pair bias as the true bottleneck. prevented wasted experiments on architecture changes.
onnx runtime inference
ft-transformer compressed from 406k to 169k params. onnx runtime inference from 11ms to 4ms on cpu.

autonomous trader agent

#python#fastapi#postgresql#docker#github actions#zerodha kite connect

autonomous trading system for indian equity markets using cross-sectional reversal scoring on nifty stocks. backtested 8.6% cagr with 60% win rate over 5.4 years.

/ the strategy

  • cross-sectional reversal — ranks nifty stocks by magnitude of decline over a 5-21 day lookback, buys the most oversold, holds for 5 trading days
  • the edge is behavioral: panic selling pushes stocks below fair value, creating a mean-reversion opportunity that algorithms can't easily arbitrage away
  • information coefficient: +0.020 (large-cap), +0.025 (midcap) — a small but consistent edge compounded over thousands of trades

/ research journey

  • tested 6 strategies systematically before finding the edge
  • 5 failed: intraday ml prediction, breakout detection, 5-min mean reversion, 30-min trend following, cross-sectional ml — indian large-cap stocks are too efficient at intraday resolution
  • daily reversal was the only signal that survived — driven by human psychology, not technical patterns
  • evolved through 4 versions of allocation logic, each improving capital efficiency — the underlying signal never changed

/ how it works

  • 3-state regime classifier (bull/neutral/weak) using nifty vs 50-dma, momentum, and market breadth with a 2-day persistence filter
  • adaptive confidence scoring: continuous 0-1 score combining ic, rolling win rate, momentum, and breadth for smooth capital allocation
  • risk controls: regime-based exposure gates, soft drawdown dampening, recovery boost, kill switches on declining win rates or negative ic, panic filters
  • a/b pipeline testing with independent scan intervals, capital pools, and paper broker instances for isolated comparison

/ results

  • backtested over 5.4 years (oct 2020 – jan 2025): 8.6% cagr, 42% total return, 60% win rate
  • survived the 2025-26 bear market with 6.5% cagr and 9-16% max drawdown
  • large-cap returns: +38% | midcap returns: +108% (2.8x higher)
  • ~52% average capital deployment — the rest held as a protective cash buffer

/ how it works

01regime classifier evaluates market conditions (bull/neutral/weak)
02confidence scorer computes allocation weight from ic, win rate, momentum, breadth
03reversal scanner ranks stocks by decline magnitude across lookback windows
04risk guardian validates exposure limits, drawdown gates, and kill switches
05trade executor places orders via zerodha kite connect (cnc for swing holding)

/ features

cross-sectional reversal scoring
ranks nifty stocks by decline magnitude. information coefficient: +0.020 (large-cap), +0.025 (midcap). exploits behavioral overreaction — a structural edge driven by psychology, not patterns algorithms can arbitrage away.
3-state regime classifier
classifies market as bull (65-85% exposure), neutral (50-75%), or weak (8-40%) using nifty vs 50-dma, momentum, and breadth. 2-day persistence filter prevents whipsawing.
adaptive confidence scoring
continuous 0-1 scoring combining information coefficient, rolling win rate, momentum, and market breadth. replaces hard thresholds for smoother capital allocation.
a/b pipeline testing
two independent pipelines with separate scan intervals and capital pools. each pipeline runs its own paper broker instance for isolated comparison.
risk management layers
regime-based exposure gates, soft drawdown dampening (gentle in bull, aggressive in weak), recovery boost when signal improves during drawdown recovery, and kill switches that pause trading on declining win rates or negative ic.
research-driven development
tested 6 strategies systematically before finding the edge. 5 failed (ml prediction, breakouts, intraday mean reversion, trend following, cross-sectional ml). every version improvement came from better capital allocation — the signal never changed.

rag system

#fastapi#qdrant#redis#openai#sentence-transformers#deberta#docker#prometheus

production rag system with zero-hallucination verification pipeline. nli entailment checking, hybrid search (vector + bm25 + cross-encoder reranking), multi-tenant architecture, and abstention decider.

/ verification pipeline

  • nli entailment checking using deberta v3 — verifies every answer is logically supported by source documents before returning to user
  • citation validation ensures each cited passage actually supports the claim it's attached to
  • multi-signal abstention decider with 6 detection signals — system refuses to answer rather than hallucinate
  • faithfulness threshold at 0.7, citation support threshold at 0.5 — tunable per deployment

/ hybrid search architecture

  • vector search (bge-small-en-v1.5 embeddings) for semantic similarity
  • bm25 keyword search for exact term matching
  • reciprocal rank fusion combines both retrieval methods
  • cross-encoder reranking (ms-marco-minilm-l-6-v2) refines top candidates

/ multi-tenant system

  • per-tenant document isolation with database-backed tenancy
  • tenant authentication and authorization via api keys
  • admin api for tenant management, document ingestion, and usage tracking
  • async background job processing with redis for document ingestion status tracking

/ evaluation framework

  • offline evaluation metrics: faithfulness, context utilization, citation precision
  • cli evaluation runner with configurable thresholds
  • 35+ integration tests covering retrieval, verification, and end-to-end query flows
  • prometheus metrics endpoint and structured logging for production observability

/ how it works

01documents ingested via api — parsed, chunked, deduplicated, embedded, stored in qdrant
02query arrives — rewritten with conversation history if multi-turn
03hybrid retrieval: vector search + bm25, fused via reciprocal rank fusion
04cross-encoder reranks top candidates for precision
05llm generates answer grounded in retrieved context with citations
06verification pipeline checks entailment, validates citations, decides whether to return or abstain

/ features

zero-hallucination verification
nli entailment checking verifies every answer against source documents. the system prefers abstention over hallucination — refuses uncertain answers rather than guessing.
hybrid retrieval + reranking
vector search + bm25 keyword search combined via reciprocal rank fusion, then refined by cross-encoder reranking. captures both semantic similarity and exact term matches.
abstention decider
6 detection signals determine whether the system should answer or refuse. prevents confident-sounding but unsupported responses — the core failure mode of naive rag.
conversational context
redis-backed session store with query rewriting using chat history. supports multi-turn conversations with context-aware retrieval.
multi-format ingestion
supports pdf, txt, docx, pptx via pymupdf. configurable chunking (512 chars with 64 overlap), content deduplication via hashing, async background processing with status tracking.
production observability
structured logging with context, prometheus metrics endpoint, per-request timing instrumentation, and health checks for api, qdrant, and database.