A few weeks ago I decided to see if I could build a trading bot that uses an AI vision model — not price data alone, but actual chart images — to confirm trade entries. Here's the architecture, what surprised me, and the honest early numbers.

The Core IdeaThe Core Idea

Most algo traders feed raw price data into models. I wanted to try something different: generate a real chart image (like what a human trader would look at), hand it to a vision model, and ask "is this a good setup?"

The hypothesis: vision models like Gemini have been trained on enough trading content to have absorbed some pattern recognition. Whether that's true is an empirical question. I decided to find out.

How the System WorksHow the System Works

The pipeline:

1. Strategy layer scans ~50 assets on Hyperliquid every 5 minutes looking for three setup types:
   • Level bounce (price testing a strong S/R level)
   • Breakout (price clearing a key level with volume surge)
   • Extension (price stretched 2.5+ ATR from mean, crowd on wrong side)
2. Pre-filter gates (fast, no AI cost):
   • R:R must be ≥ 1.95 or skip
   • Level must be STRONG or CRITICAL or skip
3. Chart generation builds a side-by-side 1H + 4H chart with candles, volume, key S/R levels annotated, entry zone, stop, target marked, funding rate and long/short ratio from exchange data.
4. Gemini vision analysis receives the chart image + a structured prompt:
   • Market structure context (where price is relative to swing highs/lows)
   • Recent candle pattern
   • Funding rate regime
   • My calibration lessons from recent closed trades
   • Returns a score 1–10 + reasoning
5. Gate threshold: score ≥ 7 fires the trade. Below 7, it's skipped and logged.
6. Feedback loop: after trades close, a calibration script analyzes patterns (winning sides, symbols, sizes) and injects those lessons back into the next Gemini prompt.

What Surprised MeWhat Surprised Me

The scoring is more nuanced than I expected. I thought Gemini would either spit out junk or agree with everything. Instead it gives specific reasons: "4H is in a downtrend, this bounce is likely a dead cat — score 4." That's actually useful signal.

The feedback loop matters immediately. After 3 closed trades, the calibration already identified something real: longs are losing, shorts are winning in the current market regime. The next Gemini prompt now explicitly carries that lesson.

Volume is expensive. The pre-filter gates cut ~70% of signals before they reach Gemini. Without them, API costs would eat the P&L.

The Lightning integration is clean. I'm trading on LNMarkets (Bitcoin perpetuals settled in sats). You deposit sats via a Lightning invoice, trade, withdraw to your Lightning wallet. The whole cycle is Bitcoin-native with no bank rails.

Early Numbers (Honest)Early Numbers (Honest)

4 live trades so far:

• 1 win: SOL short, +6.0%
• 2 losses: BTC long (-3.1%), LINK long (-3.0%)
• 1 still open: BTC long from March 3

Win rate: 33% over 3 closed. Small sample — nowhere near enough to conclude anything about whether the AI layer is actually adding alpha.

The real test is comparing the win rate on trades the AI confirmed vs. what would have happened on setups it rejected. I'm logging every gated-out signal — symbol, score, reasoning, what price did after. In 30 days I'll have something to actually analyze.

The Interesting QuestionThe Interesting Question

Does showing a vision model a chart actually work? My prior is that it probably has some signal above random — but far less than its confident tone suggests. The feedback loop (injecting lessons from recent trades) is the more interesting piece. If the model can adapt its priors from closed trade data, that's genuinely useful.

Has anyone here experimented with vision models for chart analysis? Curious what you've found.

Built with: Python, Gemini 1.5 Flash (vision), Hyperliquid API, LNMarkets API, NWC (Nostr Wallet Connect)