1058 lines
39 KiB
Markdown
1058 lines
39 KiB
Markdown
# Chart Formations Feature Implementation Plan
|
|
|
|
## Summary
|
|
|
|
Add chart formations feature with:
|
|
- **9 pattern types**: Line, Channel, Triangle, H&S, Double Bottom/Top, Adam & Eve, Triple Bottom/Top
|
|
- **Interactive drawing**: Click-to-place shape, drag anchors to adjust (SVG overlay)
|
|
- **HUD panel**: Manage formations like indicators/signals
|
|
- **Strategy integration**: Reference formations in Blockly (by tbl_key, not name)
|
|
- **Auto-targets**: Calculate and draw projected target levels
|
|
- **Infinite extension**: Lines extrapolate beyond drawn endpoints
|
|
- **Ownership model**: Users own formations; public strategies use creator's formations
|
|
- **Scope**: Formations tied to exchange/market/timeframe
|
|
|
|
**Rendering approach**: SVG overlay with requestAnimationFrame sync (NOT Lightweight Charts line series - avoids infinite loops with bound charts).
|
|
|
|
## Supported Patterns
|
|
|
|
| Pattern | Points | Description |
|
|
|---------|--------|-------------|
|
|
| Support/Resistance | 2 | Single line (horizontal or diagonal) |
|
|
| Channel | 3 | Two parallel trendlines |
|
|
| Triangle | 3 | Three connected lines (ascending/descending/symmetrical) |
|
|
| Head & Shoulders | 5 | Reversal pattern with neckline + 3 peaks |
|
|
| Double Bottom (W) | 3 | Two bottoms with middle peak (bullish reversal) |
|
|
| Double Top (M) | 3 | Two tops with middle trough (bearish reversal) |
|
|
| Adam and Eve | 3 | Sharp V bottom + rounded U bottom variant |
|
|
| Triple Bottom | 5 | Three bottoms with two peaks (strong bullish reversal) |
|
|
| Triple Top | 5 | Three tops with two troughs (strong bearish reversal) |
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ Formations HUD Panel │
|
|
│ (Draw tools, formation cards, CRUD) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ FormationOverlay │
|
|
│ (SVG layer positioned over chart, draggable anchors) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ Formations.py (Backend) │
|
|
│ (CRUD, value calculation, target projection, scope) │
|
|
├─────────────────────────────────────────────────────────────────┤
|
|
│ SQLite (formations table) │
|
|
│ (id, user_id, name, type, scope, lines_json) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
|
|
Strategy Integration:
|
|
Blockly Block → PythonGenerator → process_formation(tbl_key, property, timestamp) → price
|
|
|
|
Rendering Approach:
|
|
- SVG overlay sits absolutely positioned over the Lightweight Charts canvas
|
|
- Formations drawn as SVG elements (line, polygon, circle anchors)
|
|
- Chart sync via polling (requestAnimationFrame), NOT event subscriptions
|
|
- Avoids infinite loops from subscribeVisibleTimeRangeChange
|
|
```
|
|
|
|
## Key Design Decisions
|
|
|
|
1. **Ownership**: Users own their formations. Public strategies reference creator's formations (subscribers can't see/edit them but the strategy uses them). Uses `formation_owner_id` pattern (parallel to `indicator_owner_id`).
|
|
|
|
2. **Line Extension**: All lines extend infinitely (extrapolate beyond drawn endpoints). MVP has no extension options - all lines are infinite.
|
|
|
|
3. **Target Projection**: Auto-calculate target levels as horizontal lines (e.g., H&S target = neckline - pattern_height). Added in Phase B, not MVP.
|
|
|
|
4. **Scope**: Uses `exchange`, `market`, `timeframe` (matching existing patterns - `market` not `symbol`).
|
|
|
|
5. **References**: Blocks reference formations by `tbl_key` (stable UUID) not name (avoids collisions).
|
|
|
|
6. **Timestamps**: All timestamps in seconds UTC. Backtest uses bar timestamp, not wall clock. Out-of-range returns extrapolated value (infinite lines).
|
|
|
|
7. **MVP Scope**: Ship Line + Channel first. Complex patterns (Triangle, H&S, etc.) in Phase B.
|
|
|
|
---
|
|
|
|
## Phase 1: Backend Foundation
|
|
|
|
### 1.1 Database Schema
|
|
|
|
**In `src/Formations.py`** - Use `_ensure_table_exists()` pattern (like Signals/Trades):
|
|
|
|
```python
|
|
class Formations:
|
|
TABLE_NAME = 'formations'
|
|
|
|
def __init__(self, data_cache: DataCache, database: Database):
|
|
self.data_cache = data_cache
|
|
self.database = database
|
|
self._ensure_table_exists()
|
|
self._init_cache()
|
|
|
|
def _ensure_table_exists(self):
|
|
"""Create formations table if it doesn't exist (repo pattern)."""
|
|
self.database.execute_sql('''
|
|
CREATE TABLE IF NOT EXISTS formations (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
tbl_key TEXT UNIQUE NOT NULL,
|
|
user_id INTEGER NOT NULL,
|
|
name TEXT NOT NULL,
|
|
formation_type TEXT NOT NULL,
|
|
exchange TEXT NOT NULL,
|
|
market TEXT NOT NULL,
|
|
timeframe TEXT NOT NULL,
|
|
lines_json TEXT NOT NULL,
|
|
color TEXT DEFAULT '#667eea',
|
|
visible INTEGER DEFAULT 1,
|
|
created_at INTEGER NOT NULL,
|
|
updated_at INTEGER NOT NULL,
|
|
FOREIGN KEY (user_id) REFERENCES users(id),
|
|
UNIQUE(user_id, name, exchange, market, timeframe)
|
|
)
|
|
''')
|
|
self.database.execute_sql('''
|
|
CREATE INDEX IF NOT EXISTS idx_formations_scope
|
|
ON formations(user_id, exchange, market, timeframe)
|
|
''')
|
|
|
|
def _init_cache(self):
|
|
"""Initialize formations cache using DataCache pattern."""
|
|
self.data_cache.create_cache('formations', {})
|
|
```
|
|
|
|
**Lines JSON Structure (MVP - infinite lines only):**
|
|
```json
|
|
{
|
|
"lines": [
|
|
{"point1": {"time": 1709900000, "price": 65000.0},
|
|
"point2": {"time": 1710500000, "price": 66000.0}}
|
|
],
|
|
"targets": [
|
|
{"price": 68000.0, "label": "target_1"}
|
|
]
|
|
}
|
|
```
|
|
|
|
**Note**: MVP uses infinite line extension only. No `extend` field needed.
|
|
|
|
### 1.2 Backend Module
|
|
|
|
**New file:** `src/Formations.py`
|
|
|
|
```python
|
|
class Formations:
|
|
def __init__(self, database: Database):
|
|
self.database = database
|
|
|
|
# CRUD operations
|
|
def create(self, user_id: int, data: dict) -> dict
|
|
def update(self, user_id: int, data: dict) -> dict
|
|
def delete(self, user_id: int, tbl_key: str) -> dict
|
|
|
|
# Queries - always constrained by user_id
|
|
def get_for_scope(self, user_id: int, exchange: str, market: str, timeframe: str) -> list
|
|
def get_by_tbl_key(self, user_id: int, tbl_key: str) -> dict | None
|
|
def get_by_tbl_key_for_strategy(self, tbl_key: str, owner_user_id: int) -> dict | None
|
|
"""For strategy execution - uses strategy owner's formations"""
|
|
|
|
# Value calculation
|
|
def calculate_line_value(self, line: dict, timestamp: int) -> float
|
|
def calculate_target_value(self, formation: dict, target_name: str) -> float
|
|
def get_property_value(self, formation: dict, property: str, timestamp: int) -> float
|
|
```
|
|
|
|
**Value calculation (core algorithm - infinite extension):**
|
|
```python
|
|
def calculate_line_value(self, line: dict, timestamp: int) -> float:
|
|
"""Calculate price at timestamp using linear interpolation/extrapolation.
|
|
|
|
Args:
|
|
line: {"point1": {"time": int, "price": float}, "point2": {...}}
|
|
timestamp: Unix timestamp in seconds UTC
|
|
|
|
Returns:
|
|
Extrapolated price value (works for any timestamp, past or future)
|
|
|
|
Edge case: t1 == t2 is a vertical line (same time, different prices).
|
|
This shouldn't happen in normal drawing but we handle it gracefully.
|
|
"""
|
|
t1, p1 = line['point1']['time'], line['point1']['price']
|
|
t2, p2 = line['point2']['time'], line['point2']['price']
|
|
|
|
if t1 == t2:
|
|
# Vertical line in time (same timestamp) - invalid for price lookup
|
|
# Return average price as fallback
|
|
logger.warning(f"Vertical line detected (t1==t2={t1}), returning average price")
|
|
return (p1 + p2) / 2
|
|
|
|
slope = (p2 - p1) / (t2 - t1)
|
|
return p1 + slope * (timestamp - t1)
|
|
|
|
def calculate_targets(self, formation_type: str, lines: list, points: dict) -> list:
|
|
"""Auto-calculate target levels based on formation type.
|
|
|
|
Returns list of horizontal target lines:
|
|
- H&S: neckline - pattern_height
|
|
- Double bottom: neckline + pattern_height
|
|
- Triangle: apex projection
|
|
"""
|
|
targets = []
|
|
# Pattern-specific target calculation logic
|
|
return targets
|
|
```
|
|
|
|
### 1.3 Socket Events
|
|
|
|
**File:** `src/BrighterTrades.py` - Add to `process_incoming_message()` at line ~1548
|
|
|
|
Keep ALL socket handling in `BrighterTrades.process_incoming_message`, not app.py. Use existing reply envelope shape (`reply` + `data`):
|
|
|
|
```python
|
|
# In process_incoming_message():
|
|
elif message_type == 'request_formations':
|
|
formations = self.formations.get_for_scope(user_id, data)
|
|
return {'reply': 'formations', 'data': {'formations': formations}}
|
|
|
|
elif message_type == 'new_formation':
|
|
result = self.formations.create(user_id, data)
|
|
return {'reply': 'formation_created', 'data': result}
|
|
|
|
elif message_type == 'edit_formation':
|
|
result = self.formations.update(user_id, data)
|
|
return {'reply': 'formation_updated', 'data': result}
|
|
|
|
elif message_type == 'delete_formation':
|
|
result = self.formations.delete(user_id, data['tbl_key'])
|
|
return {'reply': 'formation_deleted', 'data': result}
|
|
```
|
|
|
|
**Frontend handler** (in `communication.js`, follows existing pattern at line ~88):
|
|
```javascript
|
|
// Socket response handler
|
|
socket.on('message', (msg) => {
|
|
if (msg.reply === 'formations') {
|
|
UI.formations.handleFormations(msg.data);
|
|
} else if (msg.reply === 'formation_created') {
|
|
UI.formations.handleFormationCreated(msg.data);
|
|
}
|
|
// ... etc
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 2: Frontend HUD Panel
|
|
|
|
### 2.1 Template
|
|
|
|
**New file:** `src/templates/formations_hud.html`
|
|
|
|
```html
|
|
<!-- MVP: Line + Channel only. More patterns added in Phase B -->
|
|
<div class="content" id="formations_panel">
|
|
<h4 style="margin: 5px 0;">Draw Formation</h4>
|
|
<div style="display: grid; grid-template-columns: 1fr 1fr; gap: 5px;">
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('support_resistance')">
|
|
━ Line
|
|
</button>
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('channel')">
|
|
═ Channel
|
|
</button>
|
|
</div>
|
|
|
|
<!-- Phase B: Add more patterns here -->
|
|
<!--
|
|
<h4 style="margin: 10px 0 5px;">Patterns</h4>
|
|
<div style="display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 5px;">
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('triangle')">△ Triangle</button>
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('head_shoulders')">⌒ H&S</button>
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('double_bottom')">W Bottom</button>
|
|
<button class="btn btn-sm" onclick="UI.formations.startDrawing('double_top')">M Top</button>
|
|
</div>
|
|
-->
|
|
<hr>
|
|
<h3>Formations</h3>
|
|
<div class="formations-container" id="formations_list"></div>
|
|
</div>
|
|
```
|
|
|
|
### 2.2 JavaScript Controller
|
|
|
|
**New file:** `src/static/formations.js`
|
|
|
|
Follow the three-class pattern from `signals.js`:
|
|
|
|
| Class | Purpose |
|
|
|-------|---------|
|
|
| `FormationsUIManager` | DOM rendering, card creation |
|
|
| `FormationsDataManager` | In-memory formations array |
|
|
| `Formations` | Socket handlers, coordinates UI + Data |
|
|
|
|
**New file:** `src/static/formation_overlay.js`
|
|
|
|
| Class | Purpose |
|
|
|-------|---------|
|
|
| `FormationOverlay` | SVG rendering, anchor dragging, chart sync |
|
|
|
|
### 2.3 Integration Points
|
|
|
|
**Modify:** `src/templates/control_panel.html`
|
|
```html
|
|
<button class="collapsible bg_blue">Formations</button>
|
|
{% include "formations_hud.html" %}
|
|
```
|
|
|
|
**Modify:** `src/static/general.js`
|
|
```javascript
|
|
this.formations = new Formations(this);
|
|
// After charts init:
|
|
this.formations.initOverlay('chart1', this.charts.chart_1);
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 3: Chart Drawing (SVG Overlay Approach)
|
|
|
|
### 3.1 Why SVG Overlay Instead of addLineSeries
|
|
|
|
**Problem with addLineSeries approach:**
|
|
- Lightweight Charts `subscribeVisibleTimeRangeChange` triggers infinite loops when multiple charts are bound together
|
|
- Adding guard flags (`_syncing`) helped but still caused page freezing after a few seconds
|
|
- Line series participate in chart scaling, causing unwanted price scale changes
|
|
- No native support for draggable anchor points
|
|
|
|
**SVG Overlay benefits:**
|
|
- Completely decoupled from chart internals - no event subscription loops
|
|
- Full control over anchor points and drag behavior
|
|
- Lines don't affect price scale calculations
|
|
- Simpler mental model: SVG is a transparent layer on top of chart
|
|
|
|
### 3.2 FormationOverlay Class
|
|
|
|
**New file:** `src/static/formation_overlay.js`
|
|
|
|
```javascript
|
|
class FormationOverlay {
|
|
constructor(chartContainerId, chart) {
|
|
this.chart = chart;
|
|
this.container = document.getElementById(chartContainerId);
|
|
this.svg = null;
|
|
this.formations = new Map(); // tbl_key -> {group, anchors, lines}
|
|
this.selectedFormation = null;
|
|
this.draggingAnchor = null;
|
|
|
|
this._createSvgLayer();
|
|
this._startSyncLoop(); // NOT event subscription
|
|
}
|
|
|
|
_createSvgLayer() {
|
|
// Create SVG absolutely positioned over chart canvas
|
|
this.svg = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
|
|
this.svg.style.cssText = `
|
|
position: absolute;
|
|
top: 0; left: 0;
|
|
width: 100%; height: 100%;
|
|
pointer-events: none; // Let clicks pass through to chart
|
|
z-index: 10;
|
|
`;
|
|
this.container.style.position = 'relative';
|
|
this.container.appendChild(this.svg);
|
|
}
|
|
|
|
_startSyncLoop() {
|
|
// CRITICAL: Use requestAnimationFrame polling, NOT subscribeVisibleTimeRangeChange
|
|
// This avoids the infinite loop problem with bound charts
|
|
const sync = () => {
|
|
this._updateAllPositions();
|
|
this._animationFrameId = requestAnimationFrame(sync);
|
|
};
|
|
this._animationFrameId = requestAnimationFrame(sync);
|
|
}
|
|
|
|
stopSyncLoop() {
|
|
if (this._animationFrameId) {
|
|
cancelAnimationFrame(this._animationFrameId);
|
|
}
|
|
}
|
|
|
|
_updateAllPositions() {
|
|
// Reposition all SVG elements based on current chart coordinates
|
|
for (const [tblKey, formation] of this.formations) {
|
|
this._updateFormationPosition(tblKey, formation);
|
|
}
|
|
}
|
|
|
|
_updateFormationPosition(tblKey, formation) {
|
|
const lines = formation.data.lines;
|
|
for (let i = 0; i < lines.length; i++) {
|
|
const line = lines[i];
|
|
const p1 = this._chartToPixel(line.point1.time, line.point1.price);
|
|
const p2 = this._chartToPixel(line.point2.time, line.point2.price);
|
|
|
|
// Update line element
|
|
const lineEl = formation.lineElements[i];
|
|
if (lineEl && p1 && p2) {
|
|
lineEl.setAttribute('x1', p1.x);
|
|
lineEl.setAttribute('y1', p1.y);
|
|
lineEl.setAttribute('x2', p2.x);
|
|
lineEl.setAttribute('y2', p2.y);
|
|
}
|
|
|
|
// Update anchor elements
|
|
const anchor1 = formation.anchorElements[i * 2];
|
|
const anchor2 = formation.anchorElements[i * 2 + 1];
|
|
if (anchor1 && p1) {
|
|
anchor1.setAttribute('cx', p1.x);
|
|
anchor1.setAttribute('cy', p1.y);
|
|
}
|
|
if (anchor2 && p2) {
|
|
anchor2.setAttribute('cx', p2.x);
|
|
anchor2.setAttribute('cy', p2.y);
|
|
}
|
|
}
|
|
}
|
|
|
|
_chartToPixel(time, price) {
|
|
// Convert chart coordinates to SVG pixel coordinates
|
|
const timeScale = this.chart.timeScale();
|
|
const priceScale = this.chart.priceScale('right');
|
|
|
|
const x = timeScale.timeToCoordinate(time);
|
|
const y = priceScale.priceToCoordinate(price);
|
|
|
|
if (x === null || y === null) return null;
|
|
return { x, y };
|
|
}
|
|
|
|
_pixelToChart(x, y) {
|
|
// Convert SVG pixel coordinates to chart coordinates
|
|
const timeScale = this.chart.timeScale();
|
|
const priceScale = this.chart.priceScale('right');
|
|
|
|
const time = timeScale.coordinateToTime(x);
|
|
const price = priceScale.coordinateToPrice(y);
|
|
|
|
return { time, price };
|
|
}
|
|
}
|
|
```
|
|
|
|
### 3.3 Click-to-Place UX
|
|
|
|
**User interaction flow:**
|
|
1. User clicks "Line" button in HUD panel
|
|
2. A default line shape appears at chart center (not following mouse - simpler)
|
|
3. User clicks chart to place the shape
|
|
4. Shape becomes editable with visible anchor points
|
|
5. User drags anchors to adjust position
|
|
6. User clicks "Save" or hits Enter to persist
|
|
|
|
```javascript
|
|
// In FormationOverlay
|
|
startDrawing(type) {
|
|
this.drawingMode = type;
|
|
|
|
// Create default shape at chart center
|
|
const center = this._getChartCenter();
|
|
const defaultPoints = this._getDefaultPointsForType(type, center);
|
|
|
|
// Create temporary formation (not saved yet)
|
|
this.tempFormation = this._createFormationElements(
|
|
'temp_' + Date.now(),
|
|
defaultPoints,
|
|
{ color: '#667eea', isTemp: true }
|
|
);
|
|
|
|
// Enable anchor dragging
|
|
this._enableAnchors(this.tempFormation);
|
|
|
|
// Show save/cancel buttons
|
|
this._showDrawingControls();
|
|
}
|
|
|
|
_getDefaultPointsForType(type, center) {
|
|
const offset = 50; // pixels
|
|
|
|
switch (type) {
|
|
case 'support_resistance':
|
|
return {
|
|
lines: [{
|
|
point1: this._pixelToChart(center.x - offset, center.y),
|
|
point2: this._pixelToChart(center.x + offset, center.y)
|
|
}]
|
|
};
|
|
case 'channel':
|
|
return {
|
|
lines: [
|
|
{
|
|
point1: this._pixelToChart(center.x - offset, center.y - 20),
|
|
point2: this._pixelToChart(center.x + offset, center.y - 20)
|
|
},
|
|
{
|
|
point1: this._pixelToChart(center.x - offset, center.y + 20),
|
|
point2: this._pixelToChart(center.x + offset, center.y + 20)
|
|
}
|
|
]
|
|
};
|
|
// ... other types
|
|
}
|
|
}
|
|
|
|
completeDrawing() {
|
|
if (!this.tempFormation) return;
|
|
|
|
// Convert temp formation to permanent
|
|
const formationData = {
|
|
formation_type: this.drawingMode,
|
|
lines_json: JSON.stringify(this.tempFormation.data),
|
|
color: '#667eea'
|
|
};
|
|
|
|
// Prompt for name
|
|
const name = prompt('Formation name:');
|
|
if (!name) {
|
|
this.cancelDrawing();
|
|
return;
|
|
}
|
|
|
|
formationData.name = name;
|
|
|
|
// Save via socket
|
|
this.onSave(formationData); // Callback to formations.js
|
|
|
|
// Cleanup temp
|
|
this._removeFormationElements(this.tempFormation);
|
|
this.tempFormation = null;
|
|
this.drawingMode = null;
|
|
}
|
|
```
|
|
|
|
### 3.4 Draggable Anchors
|
|
|
|
```javascript
|
|
_createAnchor(x, y, lineIndex, pointIndex, formation) {
|
|
const anchor = document.createElementNS('http://www.w3.org/2000/svg', 'circle');
|
|
anchor.setAttribute('cx', x);
|
|
anchor.setAttribute('cy', y);
|
|
anchor.setAttribute('r', 6);
|
|
anchor.setAttribute('fill', '#667eea');
|
|
anchor.setAttribute('stroke', '#fff');
|
|
anchor.setAttribute('stroke-width', 2);
|
|
anchor.style.cursor = 'grab';
|
|
anchor.style.pointerEvents = 'all'; // Enable interaction
|
|
|
|
// Drag handlers
|
|
anchor.addEventListener('mousedown', (e) => {
|
|
e.stopPropagation();
|
|
this.draggingAnchor = { anchor, lineIndex, pointIndex, formation };
|
|
anchor.style.cursor = 'grabbing';
|
|
});
|
|
|
|
return anchor;
|
|
}
|
|
|
|
_setupDragListeners() {
|
|
document.addEventListener('mousemove', (e) => {
|
|
if (!this.draggingAnchor) return;
|
|
|
|
const rect = this.svg.getBoundingClientRect();
|
|
const x = e.clientX - rect.left;
|
|
const y = e.clientY - rect.top;
|
|
|
|
// Update anchor position
|
|
const { anchor, lineIndex, pointIndex, formation } = this.draggingAnchor;
|
|
anchor.setAttribute('cx', x);
|
|
anchor.setAttribute('cy', y);
|
|
|
|
// Update formation data
|
|
const chartCoords = this._pixelToChart(x, y);
|
|
const pointKey = pointIndex === 0 ? 'point1' : 'point2';
|
|
formation.data.lines[lineIndex][pointKey] = {
|
|
time: chartCoords.time,
|
|
price: chartCoords.price
|
|
};
|
|
|
|
// Update line element
|
|
this._updateFormationPosition(formation.tblKey, formation);
|
|
});
|
|
|
|
document.addEventListener('mouseup', () => {
|
|
if (this.draggingAnchor) {
|
|
this.draggingAnchor.anchor.style.cursor = 'grab';
|
|
this.draggingAnchor = null;
|
|
|
|
// If editing existing formation, auto-save
|
|
if (this.selectedFormation && !this.selectedFormation.isTemp) {
|
|
this._autoSaveFormation(this.selectedFormation);
|
|
}
|
|
}
|
|
});
|
|
}
|
|
```
|
|
|
|
### 3.5 Drawing Interaction Summary
|
|
|
|
| Formation Type | Default Shape | Anchor Points |
|
|
|----------------|---------------|---------------|
|
|
| Support/Resistance | Horizontal line at center | 2 (endpoints) |
|
|
| Channel | Two parallel lines | 4 (2 per line) |
|
|
| Triangle | Three connected lines | 3 (vertices) |
|
|
| Head & Shoulders | 5-point pattern | 5 (key points) |
|
|
| Double Bottom (W) | W-shape | 3 (bottoms + peak) |
|
|
| Double Top (M) | M-shape | 3 (tops + trough) |
|
|
|
|
**Key UX principles:**
|
|
- Shape appears instantly on button click (no multi-click placement)
|
|
- Drag anchors to adjust - intuitive and immediate feedback
|
|
- Save explicitly via button/Enter
|
|
- Cancel via Escape or cancel button
|
|
|
|
### 3.6 Lessons Learned (from First Attempt)
|
|
|
|
**DO NOT:**
|
|
- Use `subscribeVisibleTimeRangeChange` for chart sync - causes infinite loops with bound charts
|
|
- Add line series that participate in price scale calculations
|
|
- Rely on chart events for positioning updates
|
|
|
|
**DO:**
|
|
- Use `requestAnimationFrame` polling for smooth, loop-free updates
|
|
- Keep SVG layer completely separate from chart internals
|
|
- Use `pointer-events: none` on SVG container, `all` only on anchors
|
|
- Store chart coordinates (time, price) not pixel coordinates in data
|
|
|
|
---
|
|
|
|
## Phase 4: Blockly Integration
|
|
|
|
### 4.1 Block Definition
|
|
|
|
**New file:** `src/static/blocks/formation_blocks.js`
|
|
|
|
Dynamic blocks generated per formation (like signals/indicators):
|
|
|
|
```javascript
|
|
// Block type: formation_<sanitized_name>
|
|
// Message: "Formation: {name} {property} at time {timestamp}"
|
|
// Output: dynamic_value
|
|
// Properties vary by type (all return price at timestamp or constant):
|
|
```
|
|
|
|
**Properties by Pattern Type:**
|
|
|
|
| Pattern | Properties | Description |
|
|
|---------|------------|-------------|
|
|
| support_resistance | `line`, `target_1` | Line + auto-target (if applicable) |
|
|
| channel | `upper`, `lower`, `midline` | Upper/lower bounds + calculated midline |
|
|
| triangle | `side_1`, `side_2`, `side_3`, `apex`, `target_1` | Sides + apex price + breakout target |
|
|
| head_shoulders | `neckline`, `left_shoulder`, `head`, `right_shoulder`, `pattern_height`, `target_1` | + auto-calculated target |
|
|
| double_bottom | `neckline`, `left_bottom`, `right_bottom`, `pattern_height`, `target_1` | Neckline + bullish target |
|
|
| double_top | `neckline`, `left_top`, `right_top`, `pattern_height`, `target_1` | Neckline + bearish target |
|
|
| adam_eve | `neckline`, `adam`, `eve`, `pattern_height`, `target_1` | Same as double_bottom |
|
|
| triple_bottom | `neckline`, `bottom_1`, `bottom_2`, `bottom_3`, `pattern_height`, `target_1` | Connects peaks + target |
|
|
| triple_top | `neckline`, `top_1`, `top_2`, `top_3`, `pattern_height`, `target_1` | Connects troughs + target |
|
|
|
|
**Target Calculation:**
|
|
- **H&S**: `target_1 = neckline_price - pattern_height` (bearish target)
|
|
- **Double Bottom**: `target_1 = neckline_price + pattern_height` (bullish target)
|
|
- **Triangle**: `target_1 = apex_price + pattern_height` (breakout direction)
|
|
- Targets are drawn as horizontal lines on chart
|
|
|
|
**Strategy Usage Examples:**
|
|
```python
|
|
# Reference by tbl_key (UUID), not name - avoids collisions
|
|
# Check if price breaks above double bottom neckline
|
|
price > process_formation('f8a3b2c1-...', 'neckline', current_time)
|
|
|
|
# Get auto-calculated target level
|
|
target = process_formation('f8a3b2c1-...', 'target_1', current_time)
|
|
|
|
# Current candle time for live/backtest
|
|
process_formation('f8a3b2c1-...', 'line', get_current_candle_time())
|
|
```
|
|
|
|
### 4.2 Python Generator
|
|
|
|
**Modify:** `src/PythonGenerator.py`
|
|
|
|
```python
|
|
def handle_formation(self, node, indent_level):
|
|
"""Generate process_formation call using tbl_key (stable) not name."""
|
|
fields = node.get('fields', {})
|
|
inputs = node.get('inputs', {})
|
|
tbl_key = fields.get('TBL_KEY') # UUID reference, not name
|
|
property = fields.get('PROPERTY', 'line')
|
|
timestamp = self.generate_condition_code(inputs.get('TIMESTAMP', {}), indent_level)
|
|
|
|
# Track formation usage for dependency resolution
|
|
if not hasattr(self, 'formations_used'):
|
|
self.formations_used = []
|
|
self.formations_used.append({'tbl_key': tbl_key, 'property': property})
|
|
|
|
return f"process_formation('{tbl_key}', '{property}', {timestamp})"
|
|
|
|
def handle_current_candle_time(self, node, indent_level):
|
|
return "get_current_candle_time()"
|
|
```
|
|
|
|
### 4.4 Blockly Toolbox Refresh
|
|
|
|
**Issue**: Dynamic blocks are initialized once (`blocksDefined` flag in `Strategies.js:989`).
|
|
New formations won't appear until toolbox rebuild.
|
|
|
|
**Solution**: Add formation change listener to trigger toolbox refresh:
|
|
|
|
```javascript
|
|
// In formations.js - after formation CRUD
|
|
handleFormationCreated(data) {
|
|
this.dataManager.addFormation(data.formation);
|
|
this.uiManager.renderFormations(this.dataManager.formations);
|
|
// Trigger toolbox rebuild
|
|
UI.strats.workspaceManager.rebuildToolbox();
|
|
}
|
|
|
|
// In Strategies.js - add rebuildToolbox method
|
|
rebuildToolbox() {
|
|
// Clear and rebuild dynamic categories
|
|
defineFormationBlocks(); // Re-run block definitions
|
|
this.workspace.updateToolbox(document.getElementById('toolbox_advanced'));
|
|
}
|
|
```
|
|
|
|
### 4.3 Strategy Instance Integration
|
|
|
|
**Critical**: Must be wired into ALL strategy instances (paper/live/backtest).
|
|
|
|
**Modify:** `src/StrategyInstance.py` (at line ~76, exec context setup)
|
|
|
|
```python
|
|
class StrategyInstance:
|
|
def __init__(self, ..., formations_manager=None, formation_owner_id=None):
|
|
# ... existing init
|
|
self.formations_manager = formations_manager
|
|
# formation_owner_id = strategy creator for subscribed strategies
|
|
# Parallel to existing indicator_owner_id pattern
|
|
self.formation_owner_id = formation_owner_id or strategy.get('user_id')
|
|
|
|
def _build_exec_context(self) -> dict:
|
|
"""Add process_formation to exec context (line ~76)."""
|
|
context = {
|
|
# ... existing functions
|
|
'process_formation': self.process_formation,
|
|
'get_current_candle_time': self.get_current_candle_time,
|
|
}
|
|
return context
|
|
|
|
def process_formation(self, tbl_key: str, property: str, timestamp: int = None) -> float:
|
|
"""Returns price value of formation property at timestamp.
|
|
|
|
Uses formation_owner_id (not current user) for subscribed strategies.
|
|
Parallel to indicator_owner_id pattern.
|
|
"""
|
|
if not self.formations_manager:
|
|
logger.warning("Formations manager not initialized")
|
|
return None
|
|
|
|
# Default timestamp: current candle time (not wall clock)
|
|
if timestamp is None:
|
|
timestamp = self.get_current_candle_time()
|
|
|
|
formation = self.formations_manager.get_by_tbl_key_for_strategy(
|
|
tbl_key, self.formation_owner_id
|
|
)
|
|
if not formation:
|
|
return None
|
|
|
|
return self.formations_manager.get_property_value(formation, property, timestamp)
|
|
|
|
def get_current_candle_time(self) -> int:
|
|
"""Returns current candle timestamp in seconds UTC.
|
|
|
|
IMPORTANT: In backtest, returns bar timestamp (not wall clock).
|
|
In live/paper, returns current candle's open time.
|
|
"""
|
|
return self.current_candle.get('time', int(time.time()))
|
|
```
|
|
|
|
**Modify:** `src/Strategies.py` (strategy instance creation, line ~268)
|
|
|
|
```python
|
|
# When creating strategy instances, inject formations manager + owner_id:
|
|
instance = PaperStrategyInstance(
|
|
...,
|
|
formations_manager=self.brighter_trades.formations,
|
|
formation_owner_id=strategy.get('user_id') # Creator's formations
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 5: Backtest Support
|
|
|
|
**Timestamp Rules:**
|
|
- **Backtest**: `get_current_candle_time()` returns the bar's timestamp being processed
|
|
- **Live/Paper**: Returns current candle's open time
|
|
- **Never use wall clock** for formation lookups in strategy code
|
|
|
|
**Modify:** `src/backtest_strategy_instance.py`
|
|
|
|
```python
|
|
class BacktestStrategyInstance(StrategyInstance):
|
|
def __init__(self, ..., formations_manager=None, formation_owner_id=None):
|
|
super().__init__(
|
|
...,
|
|
formations_manager=formations_manager,
|
|
formation_owner_id=formation_owner_id
|
|
)
|
|
|
|
def get_current_candle_time(self) -> int:
|
|
"""In backtest, return the bar timestamp being processed.
|
|
|
|
This is critical: backtest must use historical bar time,
|
|
not wall clock time, for accurate formation value lookups.
|
|
"""
|
|
return self.current_bar_time # Set by backtest loop
|
|
|
|
# process_formation inherited - uses get_current_candle_time()
|
|
```
|
|
|
|
**Modify:** `src/backtesting.py` (line ~854, strategy instance creation)
|
|
|
|
```python
|
|
# Inject formations manager + owner_id when creating backtest instance
|
|
instance = BacktestStrategyInstance(
|
|
...,
|
|
formations_manager=brighter_trades.formations,
|
|
formation_owner_id=strategy.get('user_id')
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Phase 6: Chart Scope Lifecycle
|
|
|
|
**Issue**: Chart scope changes via form submit/page reload (`price_chart.html:11`), not dynamic events.
|
|
|
|
**Solution**: Formation loading tied to page lifecycle:
|
|
|
|
```javascript
|
|
// In formations.js
|
|
initialize(targetId) {
|
|
this.uiManager.initUI(targetId);
|
|
this.registerSocketHandlers();
|
|
|
|
// Load formations for current chart scope on page load
|
|
const chartView = window.bt_data?.chart_view;
|
|
if (chartView) {
|
|
this.loadFormationsForScope(chartView.exchange, chartView.market, chartView.timeframe);
|
|
}
|
|
}
|
|
|
|
loadFormationsForScope(exchange, market, timeframe) {
|
|
this.currentScope = { exchange, market, timeframe };
|
|
this.comms.emit('message', {
|
|
type: 'request_formations',
|
|
exchange, market, timeframe
|
|
});
|
|
}
|
|
|
|
// On scope change (if we add dynamic switching later):
|
|
onChartScopeChanged(newScope) {
|
|
this.drawingManager.clearAllFormations();
|
|
this.loadFormationsForScope(newScope.exchange, newScope.market, newScope.timeframe);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Files Summary
|
|
|
|
### New Files
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `src/Formations.py` | Backend CRUD + value calculation |
|
|
| `src/static/formations.js` | Frontend controller (UI, Data, Socket handlers) |
|
|
| `src/static/formation_overlay.js` | SVG overlay, anchor dragging, chart sync |
|
|
| `src/templates/formations_hud.html` | HUD panel template |
|
|
| `src/static/blocks/formation_blocks.js` | Blockly block definitions |
|
|
|
|
### Modified Files
|
|
| File | Changes |
|
|
|------|---------|
|
|
| `src/Database.py` | Add formations table |
|
|
| `src/BrighterTrades.py` | Initialize Formations manager |
|
|
| `src/app.py` | Add socket handlers |
|
|
| `src/PythonGenerator.py` | Add handle_formation, handle_current_candle_time |
|
|
| `src/StrategyInstance.py` | Add process_formation, get_current_candle_time |
|
|
| `src/backtest_strategy_instance.py` | Load formations for backtest scope |
|
|
| `src/templates/control_panel.html` | Include formations_hud.html |
|
|
| `src/static/general.js` | Initialize UI.formations |
|
|
| `src/templates/index.html` | Include formations.js + formation_overlay.js |
|
|
|
|
---
|
|
|
|
## Verification Plan
|
|
|
|
1. **Database**: Run app, verify `formations` table created
|
|
2. **CRUD**: Create/edit/delete formation via HUD panel, verify persistence
|
|
3. **Drawing**: Draw line on chart using SVG overlay, drag anchors to adjust, verify it renders and persists on page reload. Scroll/zoom chart while formation visible - verify no freezing.
|
|
4. **Scope filtering**: Switch exchange/symbol/timeframe, verify formations update
|
|
5. **Blockly**: Create strategy referencing formation, verify code generates correctly
|
|
6. **Backtest**: Run backtest with formation-based strategy, verify values calculated correctly
|
|
7. **Tests**: Add unit tests for `calculate_line_value()` and formation CRUD
|
|
|
|
---
|
|
|
|
## Implementation Order
|
|
|
|
**Phase A: MVP (ship first - Line + Channel only)**
|
|
1. `src/Formations.py` - CRUD, `_ensure_table_exists()`, DataCache, line value calc
|
|
2. `src/BrighterTrades.py` - Add to `process_incoming_message()` (~line 1548)
|
|
3. `src/templates/formations_hud.html` - HUD panel (Line + Channel buttons only)
|
|
4. `src/static/formations.js` - UIManager, DataManager, Controller
|
|
5. `src/static/formation_overlay.js` - SVG layer, anchor dragging, requestAnimationFrame sync
|
|
6. `src/static/communication.js` - Add reply handlers (~line 88)
|
|
7. `src/static/blocks/formation_blocks.js` - Dynamic blocks (tbl_key reference)
|
|
8. `src/PythonGenerator.py` - Add `handle_formation`, `handle_current_candle_time`
|
|
9. `src/StrategyInstance.py` - Add to exec context (~line 76), inject formations_manager
|
|
10. `src/Strategies.py` - Inject formations_manager + formation_owner_id (~line 268)
|
|
11. `src/backtest_strategy_instance.py` - Override `get_current_candle_time()` for bar time
|
|
12. `src/backtesting.py` - Inject formations_manager (~line 854)
|
|
13. Toolbox refresh logic (rebuild on formation create/delete)
|
|
14. `tests/test_formations.py` - CRUD, value calc, ownership tests
|
|
|
|
**Phase B: Complex Patterns (iterate after MVP ships)**
|
|
14. Triangle pattern (3-point drawing)
|
|
15. Double Bottom/Top (W/M patterns)
|
|
16. Adam and Eve variant
|
|
17. Head & Shoulders (5-point)
|
|
18. Triple Bottom/Top
|
|
|
|
**Phase C: Targets (after patterns stable)**
|
|
19. Auto-calculate target levels per pattern type
|
|
20. Render targets as horizontal lines on chart
|
|
21. Add `target_1` property to Blockly blocks
|
|
|
|
---
|
|
|
|
## Tests Required
|
|
|
|
**Backend (`tests/test_formations.py`):**
|
|
```python
|
|
def test_create_formation_unique_constraint():
|
|
"""Cannot create duplicate name in same scope"""
|
|
|
|
def test_calculate_line_value_interpolation():
|
|
"""Value at midpoint timestamp"""
|
|
|
|
def test_calculate_line_value_extrapolation():
|
|
"""Value beyond endpoints (infinite extension)"""
|
|
|
|
def test_get_by_tbl_key_for_strategy_owner():
|
|
"""Strategy uses owner's formations, not current user's"""
|
|
|
|
def test_calculate_targets_head_shoulders():
|
|
"""Target = neckline - pattern_height"""
|
|
|
|
def test_calculate_targets_double_bottom():
|
|
"""Target = neckline + pattern_height"""
|
|
```
|
|
|
|
**Generator (`tests/test_strategy_generation.py`):**
|
|
```python
|
|
def test_formation_block_generates_process_formation():
|
|
"""Block with tbl_key generates correct function call"""
|
|
```
|
|
|
|
**Integration (`tests/test_strategy_execution.py`):**
|
|
```python
|
|
def test_process_formation_in_paper_strategy():
|
|
"""Formation value accessible in paper trading"""
|
|
|
|
def test_process_formation_in_backtest():
|
|
"""Formation value correct at historical timestamp"""
|
|
|
|
def test_subscribed_strategy_uses_creator_formations():
|
|
"""Subscriber runs strategy with creator's formations"""
|
|
```
|
|
|
|
---
|
|
|
|
## Verification (End-to-End Test)
|
|
|
|
1. **Start app**: `cd src && python app.py`
|
|
2. **Create formation**:
|
|
- Open browser to `http://127.0.0.1:5002`
|
|
- Navigate to Formations panel
|
|
- Click "Line" button - line appears at chart center with visible anchors
|
|
- Drag anchors to desired position
|
|
- Click "Save" and enter a name
|
|
- Verify it appears in panel and persists on page reload
|
|
- **Scroll chart** - verify no freezing (critical regression test)
|
|
|
|
3. **Use in strategy**:
|
|
- Create new strategy in Blockly
|
|
- Add Formation block (should see your formation by name, references tbl_key internally)
|
|
- Set condition: `if price > formation.line at current_time`
|
|
- Save strategy
|
|
|
|
4. **Run backtest**:
|
|
- Run backtest with the strategy
|
|
- Verify formation value calculated correctly at historical timestamps
|
|
- Check logs for `process_formation` calls
|
|
|
|
5. **Run tests**: `pytest tests/test_formations.py -v`
|
|
|
|
---
|
|
|
|
## Appendix: Lessons Learned from First Implementation Attempt
|
|
|
|
The first implementation attempt used Lightweight Charts `addLineSeries()` to render formations. This caused page freezing issues. Here's what we learned:
|
|
|
|
### What Went Wrong
|
|
|
|
1. **Infinite Loop with Bound Charts**
|
|
- Used `subscribeVisibleTimeRangeChange()` to sync formation positions when chart scrolled
|
|
- BrighterTrading has multiple charts bound together (main chart + indicator charts)
|
|
- When one chart scrolled, the sync callback triggered, which updated formation positions, which triggered another scroll event on bound charts, creating an infinite loop
|
|
- Adding `_syncing` guard flags reduced but didn't eliminate the problem
|
|
|
|
2. **Line Series Side Effects**
|
|
- `addLineSeries()` adds data that participates in price scale calculations
|
|
- Formation lines affected the auto-scaling of the price axis
|
|
- Lines were not truly "overlay" - they were part of chart data
|
|
|
|
3. **Incremental Fixes Created Instability**
|
|
- Each bug fix introduced new edge cases
|
|
- Guard flags, debouncing, and throttling added complexity without solving root cause
|
|
- Codebase became fragile and hard to reason about
|
|
|
|
### The SVG Overlay Solution
|
|
|
|
1. **Complete Separation**
|
|
- SVG layer is absolutely positioned over chart canvas
|
|
- No interaction with Lightweight Charts internals
|
|
- No event subscriptions that could loop
|
|
|
|
2. **Polling Instead of Events**
|
|
- Use `requestAnimationFrame` to poll chart coordinates
|
|
- Smooth 60fps updates without triggering events
|
|
- No risk of infinite loops
|
|
|
|
3. **Simpler UX Model**
|
|
- Click-to-place with instant shape appearance
|
|
- Drag anchors to adjust (familiar interaction pattern)
|
|
- Clear separation between "drawing" and "adjusting"
|
|
|
|
### Key Takeaways
|
|
|
|
| Problem | Bad Approach | Good Approach |
|
|
|---------|--------------|---------------|
|
|
| Chart sync | `subscribeVisibleTimeRangeChange` | `requestAnimationFrame` polling |
|
|
| Rendering | `addLineSeries()` | SVG overlay |
|
|
| Placement | Multi-click point collection | Shape appears, drag to adjust |
|
|
| Event handling | Chart event subscriptions | Minimal DOM events on SVG only |
|
|
|
|
### Reference Implementation
|
|
|
|
The experimental SVG overlay code is preserved in branch `feature/formations-svg-overlay-wip` for reference. While that implementation still had issues (the page freeze wasn't fully resolved), the overall SVG overlay architecture is sound and should be used as a starting point.
|