# Chart Formations Feature Implementation Plan
**Lightweight Charts Version**: v5.1.0 (upgraded March 2026)
- Uses series-based coordinate conversion: `series.priceToCoordinate()` / `series.coordinateToPrice()`
- Markers via `createSeriesMarkers()` API
- Series creation via `chart.addSeries(SeriesType, options)`
## Summary
Add chart formations feature with:
- **9 pattern types**: Line, Channel, Triangle, H&S, Double Bottom/Top, Adam & Eve, Triple Bottom/Top
- **Interactive drawing**: Click-to-place shape, drag anchors to adjust (SVG overlay)
- **HUD panel**: Manage formations like indicators/signals
- **Strategy integration**: Reference formations in Blockly (by tbl_key, not name)
- **Auto-targets**: Calculate and draw projected target levels
- **Infinite extension**: Lines extrapolate beyond drawn endpoints
- **Ownership model**: Users own formations; public strategies use creator's formations
- **Scope**: Formations tied to exchange/market/timeframe
**Rendering approach**: SVG overlay with requestAnimationFrame sync (NOT Lightweight Charts line series - avoids infinite loops with bound charts).
## Supported Patterns
| Pattern | Points | Description |
|---------|--------|-------------|
| Support/Resistance | 2 | Single line (horizontal or diagonal) |
| Channel | 3 | Two parallel trendlines |
| Triangle | 3 | Three connected lines (ascending/descending/symmetrical) |
| Head & Shoulders | 5 | Reversal pattern with neckline + 3 peaks |
| Double Bottom (W) | 3 | Two bottoms with middle peak (bullish reversal) |
| Double Top (M) | 3 | Two tops with middle trough (bearish reversal) |
| Adam and Eve | 3 | Sharp V bottom + rounded U bottom variant |
| Triple Bottom | 5 | Three bottoms with two peaks (strong bullish reversal) |
| Triple Top | 5 | Three tops with two troughs (strong bearish reversal) |
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Formations HUD Panel │
│ (Draw tools, formation cards, CRUD) │
├─────────────────────────────────────────────────────────────────┤
│ FormationOverlay │
│ (SVG layer positioned over chart, draggable anchors) │
├─────────────────────────────────────────────────────────────────┤
│ Formations.py (Backend) │
│ (CRUD, value calculation, target projection, scope) │
├─────────────────────────────────────────────────────────────────┤
│ SQLite (formations table) │
│ (id, user_id, name, type, scope, lines_json) │
└─────────────────────────────────────────────────────────────────┘
Strategy Integration:
Blockly Block → PythonGenerator → process_formation(tbl_key, property, timestamp) → price
Rendering Approach:
- SVG overlay sits absolutely positioned over the Lightweight Charts canvas
- Formations drawn as SVG elements (line, polygon, circle anchors)
- Chart sync via polling (requestAnimationFrame), NOT event subscriptions
- Avoids infinite loops from subscribeVisibleTimeRangeChange
```
## Key Design Decisions
1. **Ownership**: Users own their formations. Public strategies reference creator's formations (subscribers can't see/edit them but the strategy uses them). Uses `formation_owner_id` pattern (parallel to `indicator_owner_id`).
2. **Line Extension**: All lines extend infinitely (extrapolate beyond drawn endpoints). MVP has no extension options - all lines are infinite.
3. **Target Projection**: Auto-calculate target levels as horizontal lines (e.g., H&S target = neckline - pattern_height). Added in Phase B, not MVP.
4. **Scope**: Uses `exchange`, `market`, `timeframe` (matching existing patterns - `market` not `symbol`).
5. **References**: Blocks reference formations by `tbl_key` (stable UUID) not name (avoids collisions).
6. **Timestamps**: All timestamps in seconds UTC. Backtest uses bar timestamp, not wall clock. Out-of-range returns extrapolated value (infinite lines).
7. **MVP Scope**: Ship Line + Channel first. Complex patterns (Triangle, H&S, etc.) in Phase B.
---
## Phase 1: Backend Foundation
### 1.1 Database Schema
**In `src/Formations.py`** - Use `_ensure_table_exists()` pattern (like Signals/Trades):
```python
class Formations:
TABLE_NAME = 'formations'
def __init__(self, data_cache: DataCache, database: Database):
self.data_cache = data_cache
self.database = database
self._ensure_table_exists()
self._init_cache()
def _ensure_table_exists(self):
"""Create formations table if it doesn't exist (repo pattern)."""
self.database.execute_sql('''
CREATE TABLE IF NOT EXISTS formations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tbl_key TEXT UNIQUE NOT NULL,
user_id INTEGER NOT NULL,
name TEXT NOT NULL,
formation_type TEXT NOT NULL,
exchange TEXT NOT NULL,
market TEXT NOT NULL,
timeframe TEXT NOT NULL,
lines_json TEXT NOT NULL,
color TEXT DEFAULT '#667eea',
visible INTEGER DEFAULT 1,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL,
FOREIGN KEY (user_id) REFERENCES users(id),
UNIQUE(user_id, name, exchange, market, timeframe)
)
''')
self.database.execute_sql('''
CREATE INDEX IF NOT EXISTS idx_formations_scope
ON formations(user_id, exchange, market, timeframe)
''')
def _init_cache(self):
"""Initialize formations cache using DataCache pattern."""
self.data_cache.create_cache('formations', {})
```
**Lines JSON Structure (MVP - infinite lines only):**
```json
{
"lines": [
{"point1": {"time": 1709900000, "price": 65000.0},
"point2": {"time": 1710500000, "price": 66000.0}}
],
"targets": [
{"price": 68000.0, "label": "target_1"}
]
}
```
**Note**: MVP uses infinite line extension only. No `extend` field needed.
### 1.2 Backend Module
**New file:** `src/Formations.py`
```python
class Formations:
# Full signature - see 1.1 for __init__ implementation
# def __init__(self, data_cache: DataCache, database: Database)
# CRUD operations
def create(self, user_id: int, data: dict) -> dict
def update(self, user_id: int, data: dict) -> dict
def delete(self, user_id: int, tbl_key: str) -> dict
# Queries - always constrained by user_id
def get_for_scope(self, user_id: int, exchange: str, market: str, timeframe: str) -> list
def get_by_tbl_key(self, user_id: int, tbl_key: str) -> dict | None
def get_by_tbl_key_for_strategy(self, tbl_key: str, owner_user_id: int) -> dict | None
"""For strategy execution - uses strategy owner's formations"""
# Value calculation
def calculate_line_value(self, line: dict, timestamp: int) -> float
def calculate_target_value(self, formation: dict, target_name: str) -> float
def get_property_value(self, formation: dict, property: str, timestamp: int) -> float
```
**Value calculation (core algorithm - infinite extension):**
```python
def calculate_line_value(self, line: dict, timestamp: int) -> float:
"""Calculate price at timestamp using linear interpolation/extrapolation.
Args:
line: {"point1": {"time": int, "price": float}, "point2": {...}}
timestamp: Unix timestamp in seconds UTC
Returns:
Extrapolated price value (works for any timestamp, past or future)
Edge case: t1 == t2 is a vertical line (same time, different prices).
This shouldn't happen in normal drawing but we handle it gracefully.
"""
t1, p1 = line['point1']['time'], line['point1']['price']
t2, p2 = line['point2']['time'], line['point2']['price']
if t1 == t2:
# Vertical line in time (same timestamp) - invalid for price lookup
# Return average price as fallback
logger.warning(f"Vertical line detected (t1==t2={t1}), returning average price")
return (p1 + p2) / 2
slope = (p2 - p1) / (t2 - t1)
return p1 + slope * (timestamp - t1)
def calculate_targets(self, formation_type: str, lines: list, points: dict) -> list:
"""Auto-calculate target levels based on formation type.
Returns list of horizontal target lines:
- H&S: neckline - pattern_height
- Double bottom: neckline + pattern_height
- Triangle: apex projection
"""
targets = []
# Pattern-specific target calculation logic
return targets
```
### 1.3 Socket Events
**File:** `src/BrighterTrades.py` - Add to `process_incoming_message()` at line ~1548
Keep ALL socket handling in `BrighterTrades.process_incoming_message`, not app.py. Use existing reply envelope shape (`reply` + `data`):
```python
# In process_incoming_message():
elif message_type == 'request_formations':
formations = self.formations.get_for_scope(user_id, data)
return {'reply': 'formations', 'data': {'formations': formations}}
elif message_type == 'new_formation':
result = self.formations.create(user_id, data)
return {'reply': 'formation_created', 'data': result}
elif message_type == 'edit_formation':
result = self.formations.update(user_id, data)
return {'reply': 'formation_updated', 'data': result}
elif message_type == 'delete_formation':
result = self.formations.delete(user_id, data['tbl_key'])
return {'reply': 'formation_deleted', 'data': result}
```
**Frontend handler** (in `formations.js`, follows signals.js pattern):
```javascript
// In Formations.registerSocketHandlers() - use comms.on() pattern
registerSocketHandlers() {
this.comms.on('formations', this.handleFormationsResponse.bind(this));
this.comms.on('formation_created', this.handleFormationCreated.bind(this));
this.comms.on('formation_updated', this.handleFormationUpdated.bind(this));
this.comms.on('formation_deleted', this.handleFormationDeleted.bind(this));
this.comms.on('formation_error', this.handleFormationError.bind(this));
}
```
---
## Phase 2: Frontend HUD Panel
### 2.1 Template
**New file:** `src/templates/formations_hud.html`
```html
Draw Formation
Formations
```
### 2.2 JavaScript Controller
**New file:** `src/static/formations.js`
Follow the three-class pattern from `signals.js`:
| Class | Purpose |
|-------|---------|
| `FormationsUIManager` | DOM rendering, card creation |
| `FormationsDataManager` | In-memory formations array |
| `Formations` | Socket handlers, coordinates UI + Data |
**New file:** `src/static/formation_overlay.js`
| Class | Purpose |
|-------|---------|
| `FormationOverlay` | SVG rendering, anchor dragging, chart sync |
### 2.3 Integration Points
**Modify:** `src/templates/control_panel.html`
```html
{% include "formations_hud.html" %}
```
**Modify:** `src/static/general.js`
```javascript
this.formations = new Formations(this);
// After charts init - use dynamic chart ID, pass candleSeries for v5 coordinate conversion:
this.formations.initOverlay(this.data.chart1_id, this.charts.chart_1, this.charts.candleSeries);
```
---
## Phase 3: Chart Drawing (SVG Overlay Approach)
### 3.1 Why SVG Overlay Instead of addLineSeries
**Problem with addLineSeries approach:**
- Lightweight Charts `subscribeVisibleTimeRangeChange` triggers infinite loops when multiple charts are bound together
- Adding guard flags (`_syncing`) helped but still caused page freezing after a few seconds
- Line series participate in chart scaling, causing unwanted price scale changes
- No native support for draggable anchor points
**SVG Overlay benefits:**
- Completely decoupled from chart internals - no event subscription loops
- Full control over anchor points and drag behavior
- Lines don't affect price scale calculations
- Simpler mental model: SVG is a transparent layer on top of chart
### 3.2 FormationOverlay Class
**New file:** `src/static/formation_overlay.js`
```javascript
class FormationOverlay {
constructor(chartContainerId, chart, candleSeries) {
this.chart = chart;
this.candleSeries = candleSeries; // Required for v5 coordinate conversion
this.container = document.getElementById(chartContainerId);
this.svg = null;
this.formations = new Map(); // tbl_key -> {group, anchors, lines}
this.selectedFormation = null;
this.draggingAnchor = null;
this._createSvgLayer();
this._startSyncLoop(); // NOT event subscription
}
_createSvgLayer() {
// Create SVG absolutely positioned over chart canvas
this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
this.svg.style.cssText = `
position: absolute;
top: 0; left: 0;
width: 100%; height: 100%;
pointer-events: none; // Let clicks pass through to chart
z-index: 10;
`;
this.container.style.position = 'relative';
this.container.appendChild(this.svg);
}
_startSyncLoop() {
// CRITICAL: Use requestAnimationFrame polling, NOT subscribeVisibleTimeRangeChange
// This avoids the infinite loop problem with bound charts
this._loopRunning = true;
const sync = () => {
// Guard: only run if there's something to sync
if (!this._loopRunning) return;
if (this.formations.size > 0 || this.tempFormation) {
this._updateAllPositions();
}
this._animationFrameId = requestAnimationFrame(sync);
};
this._animationFrameId = requestAnimationFrame(sync);
}
stopSyncLoop() {
this._loopRunning = false;
if (this._animationFrameId) {
cancelAnimationFrame(this._animationFrameId);
this._animationFrameId = null;
}
}
// Call when component is destroyed
destroy() {
this.stopSyncLoop();
if (this.svg && this.svg.parentNode) {
this.svg.parentNode.removeChild(this.svg);
}
}
_updateAllPositions() {
// Reposition all SVG elements based on current chart coordinates
for (const [tblKey, formation] of this.formations) {
this._updateFormationPosition(tblKey, formation);
}
}
_updateFormationPosition(tblKey, formation) {
const lines = formation.data.lines;
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
// Get infinite line endpoints (extends to viewport edges)
const extended = this._getInfiniteLineEndpoints(line);
// Get anchor positions (at defined points, not extended)
const p1 = this._chartToPixel(line.point1.time, line.point1.price);
const p2 = this._chartToPixel(line.point2.time, line.point2.price);
// Update line element with infinite extension
const lineEl = formation.lineElements[i];
if (lineEl && extended) {
lineEl.setAttribute('x1', extended.x1);
lineEl.setAttribute('y1', extended.y1);
lineEl.setAttribute('x2', extended.x2);
lineEl.setAttribute('y2', extended.y2);
}
// Update anchor elements at defined points (not extended)
const anchor1 = formation.anchorElements[i * 2];
const anchor2 = formation.anchorElements[i * 2 + 1];
if (anchor1 && p1) {
anchor1.setAttribute('cx', p1.x);
anchor1.setAttribute('cy', p1.y);
}
if (anchor2 && p2) {
anchor2.setAttribute('cx', p2.x);
anchor2.setAttribute('cy', p2.y);
}
}
}
_chartToPixel(time, price) {
// Convert chart coordinates to SVG pixel coordinates
// NOTE: In Lightweight Charts v5, use series for price conversion
const timeScale = this.chart.timeScale();
const x = timeScale.timeToCoordinate(time);
const y = this.candleSeries.priceToCoordinate(price);
if (x === null || y === null) return null;
return { x, y };
}
_pixelToChart(x, y) {
// Convert SVG pixel coordinates to chart coordinates
// NOTE: In Lightweight Charts v5, use series for price conversion
const timeScale = this.chart.timeScale();
const time = timeScale.coordinateToTime(x);
const price = this.candleSeries.coordinateToPrice(y);
return { time, price };
}
_getInfiniteLineEndpoints(line) {
// Calculate extended line endpoints to viewport edges (infinite extension)
// Returns pixel coordinates that extend beyond the defined points
const p1 = line.point1;
const p2 = line.point2;
// Get viewport bounds in pixels
const svgRect = this.svg.getBoundingClientRect();
const viewportWidth = svgRect.width;
const viewportHeight = svgRect.height;
// Convert line points to pixels
const px1 = this._chartToPixel(p1.time, p1.price);
const px2 = this._chartToPixel(p2.time, p2.price);
if (!px1 || !px2) return null;
// Handle vertical line (same x)
if (Math.abs(px2.x - px1.x) < 0.001) {
return {
x1: px1.x, y1: 0,
x2: px1.x, y2: viewportHeight
};
}
// Calculate slope and y-intercept: y = mx + b
const m = (px2.y - px1.y) / (px2.x - px1.x);
const b = px1.y - m * px1.x;
// Find intersection with viewport edges
// Left edge (x=0): y = b
// Right edge (x=viewportWidth): y = m * viewportWidth + b
// Top edge (y=0): x = -b / m
// Bottom edge (y=viewportHeight): x = (viewportHeight - b) / m
const intersections = [];
// Left edge
const yAtLeft = b;
if (yAtLeft >= 0 && yAtLeft <= viewportHeight) {
intersections.push({ x: 0, y: yAtLeft });
}
// Right edge
const yAtRight = m * viewportWidth + b;
if (yAtRight >= 0 && yAtRight <= viewportHeight) {
intersections.push({ x: viewportWidth, y: yAtRight });
}
// Top edge (if not horizontal)
if (Math.abs(m) > 0.001) {
const xAtTop = -b / m;
if (xAtTop >= 0 && xAtTop <= viewportWidth) {
intersections.push({ x: xAtTop, y: 0 });
}
}
// Bottom edge (if not horizontal)
if (Math.abs(m) > 0.001) {
const xAtBottom = (viewportHeight - b) / m;
if (xAtBottom >= 0 && xAtBottom <= viewportWidth) {
intersections.push({ x: xAtBottom, y: viewportHeight });
}
}
// Return the two furthest-apart intersections
if (intersections.length >= 2) {
// Sort by x coordinate and take first and last
intersections.sort((a, b) => a.x - b.x);
return {
x1: intersections[0].x, y1: intersections[0].y,
x2: intersections[intersections.length - 1].x,
y2: intersections[intersections.length - 1].y
};
}
// Fallback to original endpoints
return { x1: px1.x, y1: px1.y, x2: px2.x, y2: px2.y };
}
}
```
### 3.3 Click-to-Draw UX
**User interaction flow:**
1. User clicks "Line" button in HUD panel
2. A default line shape appears at chart center with visible anchor points
3. User drags anchors to adjust position (no click-to-place step)
4. User clicks "Save" or hits Enter to persist
```javascript
// In FormationOverlay
startDrawing(type) {
this.drawingMode = type;
// Create default shape at chart center
const center = this._getChartCenter();
const defaultPoints = this._getDefaultPointsForType(type, center);
// Create temporary formation (not saved yet)
this.tempFormation = this._createFormationElements(
'temp_' + Date.now(),
defaultPoints,
{ color: '#667eea', isTemp: true }
);
// Enable anchor dragging
this._enableAnchors(this.tempFormation);
// Show save/cancel buttons
this._showDrawingControls();
}
_getDefaultPointsForType(type, center) {
const offset = 50; // pixels
switch (type) {
case 'support_resistance':
return {
lines: [{
point1: this._pixelToChart(center.x - offset, center.y),
point2: this._pixelToChart(center.x + offset, center.y)
}]
};
case 'channel':
return {
lines: [
{
point1: this._pixelToChart(center.x - offset, center.y - 20),
point2: this._pixelToChart(center.x + offset, center.y - 20)
},
{
point1: this._pixelToChart(center.x - offset, center.y + 20),
point2: this._pixelToChart(center.x + offset, center.y + 20)
}
]
};
// ... other types
}
}
completeDrawing() {
if (!this.tempFormation) return;
// Convert temp formation to permanent
const formationData = {
formation_type: this.drawingMode,
lines_json: JSON.stringify(this.tempFormation.data),
color: '#667eea'
};
// Prompt for name
const name = prompt('Formation name:');
if (!name) {
this.cancelDrawing();
return;
}
formationData.name = name;
// Save via socket
this.onSave(formationData); // Callback to formations.js
// Cleanup temp
this._removeFormationElements(this.tempFormation);
this.tempFormation = null;
this.drawingMode = null;
}
```
### 3.4 Draggable Anchors
```javascript
_createAnchor(x, y, lineIndex, pointIndex, formation) {
const anchor = document.createElementNS('http://www.w3.org/2000/svg', 'circle');
anchor.setAttribute('cx', x);
anchor.setAttribute('cy', y);
anchor.setAttribute('r', 6);
anchor.setAttribute('fill', '#667eea');
anchor.setAttribute('stroke', '#fff');
anchor.setAttribute('stroke-width', 2);
anchor.style.cursor = 'grab';
anchor.style.pointerEvents = 'all'; // Enable interaction
// Drag handlers
anchor.addEventListener('mousedown', (e) => {
e.stopPropagation();
this.draggingAnchor = { anchor, lineIndex, pointIndex, formation };
anchor.style.cursor = 'grabbing';
});
return anchor;
}
_setupDragListeners() {
document.addEventListener('mousemove', (e) => {
if (!this.draggingAnchor) return;
const rect = this.svg.getBoundingClientRect();
const x = e.clientX - rect.left;
const y = e.clientY - rect.top;
// Guard: skip if outside SVG bounds
if (x < 0 || y < 0 || x > rect.width || y > rect.height) return;
// Update anchor position
const { anchor, lineIndex, pointIndex, formation } = this.draggingAnchor;
anchor.setAttribute('cx', x);
anchor.setAttribute('cy', y);
// Update formation data with null guard
const chartCoords = this._pixelToChart(x, y);
if (!chartCoords || chartCoords.time === null || chartCoords.price === null) {
return; // Invalid coordinates - don't mutate formation data
}
const pointKey = pointIndex === 0 ? 'point1' : 'point2';
formation.data.lines[lineIndex][pointKey] = {
time: chartCoords.time,
price: chartCoords.price
};
// Update line element
this._updateFormationPosition(formation.tblKey, formation);
});
document.addEventListener('mouseup', () => {
if (this.draggingAnchor) {
this.draggingAnchor.anchor.style.cursor = 'grab';
this.draggingAnchor = null;
// If editing existing formation, auto-save
if (this.selectedFormation && !this.selectedFormation.isTemp) {
this._autoSaveFormation(this.selectedFormation);
}
}
});
}
```
### 3.5 Drawing Interaction Summary
| Formation Type | Default Shape | Anchor Points |
|----------------|---------------|---------------|
| Support/Resistance | Horizontal line at center | 2 (endpoints) |
| Channel | Two parallel lines | 4 (2 per line) |
| Triangle | Three connected lines | 3 (vertices) |
| Head & Shoulders | 5-point pattern | 5 (key points) |
| Double Bottom (W) | W-shape | 3 (bottoms + peak) |
| Double Top (M) | M-shape | 3 (tops + trough) |
**Key UX principles:**
- Shape appears instantly on button click (no multi-click placement)
- Drag anchors to adjust - intuitive and immediate feedback
- Save explicitly via button/Enter
- Cancel via Escape or cancel button
### 3.6 Lessons Learned (from First Attempt)
**DO NOT:**
- Use `subscribeVisibleTimeRangeChange` for chart sync - causes infinite loops with bound charts
- Add line series that participate in price scale calculations
- Rely on chart events for positioning updates
**DO:**
- Use `requestAnimationFrame` polling for smooth, loop-free updates
- Keep SVG layer completely separate from chart internals
- Use `pointer-events: none` on SVG container, `all` only on anchors
- Store chart coordinates (time, price) not pixel coordinates in data
---
## Phase 4: Blockly Integration
### 4.1 Block Definition
**New file:** `src/static/blocks/formation_blocks.js`
Dynamic blocks generated per formation (like signals/indicators):
```javascript
// Block type: formation_
// Message: "Formation: {name} {property} at time {timestamp}"
// Output: dynamic_value
// Properties vary by type (all return price at timestamp or constant):
```
**Properties by Pattern Type:**
| Pattern | Properties | Description |
|---------|------------|-------------|
| support_resistance | `line`, `target_1` | Line + auto-target (if applicable) |
| channel | `upper`, `lower`, `midline` | Upper/lower bounds + calculated midline |
| triangle | `side_1`, `side_2`, `side_3`, `apex`, `target_1` | Sides + apex price + breakout target |
| head_shoulders | `neckline`, `left_shoulder`, `head`, `right_shoulder`, `pattern_height`, `target_1` | + auto-calculated target |
| double_bottom | `neckline`, `left_bottom`, `right_bottom`, `pattern_height`, `target_1` | Neckline + bullish target |
| double_top | `neckline`, `left_top`, `right_top`, `pattern_height`, `target_1` | Neckline + bearish target |
| adam_eve | `neckline`, `adam`, `eve`, `pattern_height`, `target_1` | Same as double_bottom |
| triple_bottom | `neckline`, `bottom_1`, `bottom_2`, `bottom_3`, `pattern_height`, `target_1` | Connects peaks + target |
| triple_top | `neckline`, `top_1`, `top_2`, `top_3`, `pattern_height`, `target_1` | Connects troughs + target |
**Target Calculation:**
- **H&S**: `target_1 = neckline_price - pattern_height` (bearish target)
- **Double Bottom**: `target_1 = neckline_price + pattern_height` (bullish target)
- **Triangle**: `target_1 = apex_price + pattern_height` (breakout direction)
- Targets are drawn as horizontal lines on chart
**Strategy Usage Examples:**
```python
# Reference by tbl_key (UUID), not name - avoids collisions
# Check if price breaks above double bottom neckline
price > process_formation('f8a3b2c1-...', 'neckline', current_time)
# Get auto-calculated target level
target = process_formation('f8a3b2c1-...', 'target_1', current_time)
# Current candle time for live/backtest
process_formation('f8a3b2c1-...', 'line', get_current_candle_time())
```
### 4.2 Python Generator
**Modify:** `src/PythonGenerator.py`
```python
def handle_formation(self, node, indent_level):
"""Generate process_formation call using tbl_key (stable) not name."""
fields = node.get('fields', {})
inputs = node.get('inputs', {})
tbl_key = fields.get('TBL_KEY') # UUID reference, not name
property = fields.get('PROPERTY', 'line')
timestamp = self.generate_condition_code(inputs.get('TIMESTAMP', {}), indent_level)
# Track formation usage for dependency resolution
if not hasattr(self, 'formations_used'):
self.formations_used = []
self.formations_used.append({'tbl_key': tbl_key, 'property': property})
return f"process_formation('{tbl_key}', '{property}', {timestamp})"
def handle_current_candle_time(self, node, indent_level):
return "get_current_candle_time()"
```
### 4.4 Blockly Toolbox Refresh
**Issue**: Dynamic blocks are initialized once (`blocksDefined` flag in `Strategies.js:989`).
New formations won't appear until toolbox rebuild.
**Solution**: Add formation change listener to trigger toolbox refresh:
```javascript
// In formations.js - after formation CRUD
handleFormationCreated(data) {
this.dataManager.addFormation(data.formation);
this.uiManager.renderFormations(this.dataManager.formations);
// Trigger toolbox rebuild
UI.strats.workspaceManager.rebuildToolbox();
}
// In Strategies.js - add rebuildToolbox method
rebuildToolbox() {
// Clear and rebuild dynamic categories
defineFormationBlocks(); // Re-run block definitions
this.workspace.updateToolbox(document.getElementById('toolbox_advanced'));
}
```
### 4.3 Strategy Instance Integration
**Critical**: Must be wired into ALL strategy instances (paper/live/backtest).
**Modify:** `src/StrategyInstance.py` (at line ~76, exec context setup)
```python
class StrategyInstance:
def __init__(self, ..., formations_manager=None, formation_owner_id=None):
# ... existing init
self.formations_manager = formations_manager
# formation_owner_id = strategy creator for subscribed strategies
# Parallel to existing indicator_owner_id pattern
self.formation_owner_id = formation_owner_id or strategy.get('user_id')
def _build_exec_context(self) -> dict:
"""Add process_formation to exec context (line ~76)."""
context = {
# ... existing functions
'process_formation': self.process_formation,
'get_current_candle_time': self.get_current_candle_time,
}
return context
def process_formation(self, tbl_key: str, property: str, timestamp: int = None) -> float:
"""Returns price value of formation property at timestamp.
Uses formation_owner_id (not current user) for subscribed strategies.
Parallel to indicator_owner_id pattern.
"""
if not self.formations_manager:
logger.warning("Formations manager not initialized")
return None
# Default timestamp: current candle time (not wall clock)
if timestamp is None:
timestamp = self.get_current_candle_time()
formation = self.formations_manager.get_by_tbl_key_for_strategy(
tbl_key, self.formation_owner_id
)
if not formation:
return None
return self.formations_manager.get_property_value(formation, property, timestamp)
def get_current_candle_time(self) -> int:
"""Returns current candle timestamp in seconds UTC.
IMPORTANT: In backtest, returns bar timestamp (not wall clock).
In live/paper, returns current candle's open time.
NEVER falls back to wall clock - that would corrupt backtest results.
"""
if not self.current_candle or 'time' not in self.current_candle:
raise ValueError("current_candle.time not set - cannot use wall clock for formations")
return self.current_candle['time']
```
**Modify:** `src/Strategies.py` (strategy instance creation, line ~268)
```python
# When creating strategy instances, inject formations manager + owner_id:
instance = PaperStrategyInstance(
...,
formations_manager=self.brighter_trades.formations,
formation_owner_id=strategy.get('user_id') # Creator's formations
)
```
---
## Phase 5: Backtest Support
**Timestamp Rules:**
- **Backtest**: `get_current_candle_time()` returns the bar's timestamp being processed
- **Live/Paper**: Returns current candle's open time
- **Never use wall clock** for formation lookups in strategy code
**Modify:** `src/backtest_strategy_instance.py`
```python
class BacktestStrategyInstance(StrategyInstance):
def __init__(self, ..., formations_manager=None, formation_owner_id=None):
super().__init__(
...,
formations_manager=formations_manager,
formation_owner_id=formation_owner_id
)
def get_current_candle_time(self) -> int:
"""In backtest, return the bar timestamp being processed.
This is critical: backtest must use historical bar time,
not wall clock time, for accurate formation value lookups.
"""
return self.current_bar_time # Set by backtest loop
# process_formation inherited - uses get_current_candle_time()
```
**Modify:** `src/backtesting.py` (line ~854, strategy instance creation)
```python
# Inject formations manager + owner_id when creating backtest instance
instance = BacktestStrategyInstance(
...,
formations_manager=brighter_trades.formations,
formation_owner_id=strategy.get('user_id')
)
```
---
## Phase 6: Chart Scope Lifecycle
**Issue**: Chart scope changes via form submit/page reload (`price_chart.html:11`), not dynamic events.
**Solution**: Formation loading tied to page lifecycle:
```javascript
// In formations.js
initialize(targetId) {
this.uiManager.initUI(targetId);
this.registerSocketHandlers();
// Load formations for current chart scope on page load
const chartView = window.bt_data?.chart_view;
if (chartView) {
this.loadFormationsForScope(chartView.exchange, chartView.market, chartView.timeframe);
}
}
loadFormationsForScope(exchange, market, timeframe) {
this.currentScope = { exchange, market, timeframe };
this.comms.sendToApp('request_formations', { exchange, market, timeframe });
}
// On scope change (if we add dynamic switching later):
onChartScopeChanged(newScope) {
this.drawingManager.clearAllFormations();
this.loadFormationsForScope(newScope.exchange, newScope.market, newScope.timeframe);
}
```
---
## Files Summary
### New Files
| File | Purpose |
|------|---------|
| `src/Formations.py` | Backend CRUD + value calculation |
| `src/static/formations.js` | Frontend controller (UI, Data, Socket handlers) |
| `src/static/formation_overlay.js` | SVG overlay, anchor dragging, chart sync |
| `src/templates/formations_hud.html` | HUD panel template |
| `src/static/blocks/formation_blocks.js` | Blockly block definitions |
### Modified Files
| File | Changes |
|------|---------|
| `src/BrighterTrades.py` | Initialize Formations manager + add socket handlers in `process_incoming_message()` |
| `src/PythonGenerator.py` | Add handle_formation, handle_current_candle_time |
| `src/StrategyInstance.py` | Add process_formation, get_current_candle_time |
| `src/backtest_strategy_instance.py` | Override get_current_candle_time for bar time |
| `src/Strategies.py` | Inject formations_manager + formation_owner_id |
| `src/backtesting.py` | Inject formations_manager |
| `src/templates/control_panel.html` | Include formations_hud.html |
| `src/static/general.js` | Initialize UI.formations + overlay |
| `src/static/communication.js` | Add reply handlers for formations |
| `src/templates/index.html` | Include formations.js + formation_overlay.js |
**Note**: Table creation uses `_ensure_table_exists()` pattern in `Formations.py` (not Database.py).
Socket handling stays in `BrighterTrades.process_incoming_message()` (not app.py).
---
## Verification Plan
1. **Database**: Run app, verify `formations` table created
2. **CRUD**: Create/edit/delete formation via HUD panel, verify persistence
3. **Drawing**: Draw line on chart using SVG overlay, drag anchors to adjust, verify it renders and persists on page reload. Scroll/zoom chart while formation visible - verify no freezing.
4. **Scope filtering**: Switch exchange/symbol/timeframe, verify formations update
5. **Blockly**: Create strategy referencing formation, verify code generates correctly
6. **Backtest**: Run backtest with formation-based strategy, verify values calculated correctly
7. **Tests**: Add unit tests for `calculate_line_value()` and formation CRUD
---
## Implementation Order
**Phase A: MVP (ship first - Line + Channel only)**
1. `src/Formations.py` - CRUD, `_ensure_table_exists()`, DataCache, line value calc
2. `src/BrighterTrades.py` - Add to `process_incoming_message()` (~line 1548)
3. `src/templates/formations_hud.html` - HUD panel (Line + Channel buttons only)
4. `src/static/formations.js` - UIManager, DataManager, Controller
5. `src/static/formation_overlay.js` - SVG layer, anchor dragging, requestAnimationFrame sync
6. `src/static/communication.js` - Add reply handlers (~line 88)
7. `src/static/blocks/formation_blocks.js` - Dynamic blocks (tbl_key reference)
8. `src/PythonGenerator.py` - Add `handle_formation`, `handle_current_candle_time`
9. `src/StrategyInstance.py` - Add to exec context (~line 76), inject formations_manager
10. `src/Strategies.py` - Inject formations_manager + formation_owner_id (~line 268)
11. `src/backtest_strategy_instance.py` - Override `get_current_candle_time()` for bar time
12. `src/backtesting.py` - Inject formations_manager (~line 854)
13. Toolbox refresh logic (rebuild on formation create/delete)
14. `tests/test_formations.py` - CRUD, value calc, ownership tests
**Phase B: Complex Patterns (iterate after MVP ships)**
14. Triangle pattern (3-point drawing)
15. Double Bottom/Top (W/M patterns)
16. Adam and Eve variant
17. Head & Shoulders (5-point)
18. Triple Bottom/Top
**Phase C: Targets (after patterns stable)**
19. Auto-calculate target levels per pattern type
20. Render targets as horizontal lines on chart
21. Add `target_1` property to Blockly blocks
---
## Tests Required
### MVP Tests (Phase A - ship with Line + Channel)
**Backend (`tests/test_formations.py`):**
```python
def test_create_formation_unique_constraint():
"""Cannot create duplicate name in same scope"""
def test_calculate_line_value_interpolation():
"""Value at midpoint timestamp"""
def test_calculate_line_value_extrapolation():
"""Value beyond endpoints (infinite extension)"""
def test_get_by_tbl_key_for_strategy_owner():
"""Strategy uses owner's formations, not current user's"""
def test_delete_formation():
"""Formation deleted from DB and cache"""
def test_update_formation_lines():
"""Lines JSON updated correctly"""
```
**Generator (`tests/test_strategy_generation.py`):**
```python
def test_formation_block_generates_process_formation():
"""Block with tbl_key generates correct function call"""
```
### Phase C Tests (add when implementing targets)
**Backend (`tests/test_formations.py`):**
```python
def test_calculate_targets_head_shoulders():
"""Target = neckline - pattern_height"""
def test_calculate_targets_double_bottom():
"""Target = neckline + pattern_height"""
def test_calculate_targets_triangle():
"""Target = apex projection"""
```
**Integration (`tests/test_strategy_execution.py`):**
```python
def test_process_formation_in_paper_strategy():
"""Formation value accessible in paper trading"""
def test_process_formation_in_backtest():
"""Formation value correct at historical timestamp"""
def test_subscribed_strategy_uses_creator_formations():
"""Subscriber runs strategy with creator's formations"""
```
---
## Verification (End-to-End Test)
1. **Start app**: `cd src && python app.py`
2. **Create formation**:
- Open browser to `http://127.0.0.1:5002`
- Navigate to Formations panel
- Click "Line" button - line appears at chart center with visible anchors
- Drag anchors to desired position
- Click "Save" and enter a name
- Verify it appears in panel and persists on page reload
- **Scroll chart** - verify no freezing (critical regression test)
3. **Use in strategy**:
- Create new strategy in Blockly
- Add Formation block (should see your formation by name, references tbl_key internally)
- Set condition: `if price > formation.line at current_time`
- Save strategy
4. **Run backtest**:
- Run backtest with the strategy
- Verify formation value calculated correctly at historical timestamps
- Check logs for `process_formation` calls
5. **Run tests**: `pytest tests/test_formations.py -v`
---
## Appendix: Lessons Learned from First Implementation Attempt
The first implementation attempt used Lightweight Charts `addLineSeries()` to render formations. This caused page freezing issues. Here's what we learned:
### What Went Wrong
1. **Infinite Loop with Bound Charts**
- Used `subscribeVisibleTimeRangeChange()` to sync formation positions when chart scrolled
- BrighterTrading has multiple charts bound together (main chart + indicator charts)
- When one chart scrolled, the sync callback triggered, which updated formation positions, which triggered another scroll event on bound charts, creating an infinite loop
- Adding `_syncing` guard flags reduced but didn't eliminate the problem
2. **Line Series Side Effects**
- `addLineSeries()` adds data that participates in price scale calculations
- Formation lines affected the auto-scaling of the price axis
- Lines were not truly "overlay" - they were part of chart data
3. **Incremental Fixes Created Instability**
- Each bug fix introduced new edge cases
- Guard flags, debouncing, and throttling added complexity without solving root cause
- Codebase became fragile and hard to reason about
### The SVG Overlay Solution
1. **Complete Separation**
- SVG layer is absolutely positioned over chart canvas
- No interaction with Lightweight Charts internals
- No event subscriptions that could loop
2. **Polling Instead of Events**
- Use `requestAnimationFrame` to poll chart coordinates
- Smooth 60fps updates without triggering events
- No risk of infinite loops
3. **Simpler UX Model**
- Click-to-place with instant shape appearance
- Drag anchors to adjust (familiar interaction pattern)
- Clear separation between "drawing" and "adjusting"
### Key Takeaways
| Problem | Bad Approach | Good Approach |
|---------|--------------|---------------|
| Chart sync | `subscribeVisibleTimeRangeChange` | `requestAnimationFrame` polling |
| Rendering | `addLineSeries()` | SVG overlay |
| Placement | Multi-click point collection | Shape appears, drag to adjust |
| Event handling | Chart event subscriptions | Minimal DOM events on SVG only |
### Reference Implementation
The experimental SVG overlay code is preserved in branch `feature/formations-svg-overlay-wip` for reference. While that implementation still had issues (the page freeze wasn't fully resolved), the overall SVG overlay architecture is sound and should be used as a starting point.