From 6135d42bc0e077f4ea7ef3a66d4115c970353305 Mon Sep 17 00:00:00 2001 From: rob Date: Sat, 1 Nov 2025 21:48:25 -0300 Subject: [PATCH] docs: comprehensive AI configuration documentation update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Major documentation updates to align with multi-provider AI system: 1. Update CLAUDE.md (lines 213-332) - Add new "AI Configuration System" section - Document config/ai.yml structure and three optimization levels - Explain model hint propagation pipeline (rule → runner → patcher) - Add provider setup table (Claude, Codex, Gemini) - Document Claude subagent setup with ./tools/setup_claude_agents.sh - List implementation modules with line number references - Explain environment variable overrides - Document fallback behavior when all providers fail 2. Update docs/DESIGN.md (lines 894-1077) - Add "Automation AI Configuration" section before Stage Model - Document configuration architecture with full YAML example - Explain model hint system with .ai-rules.yml examples - Detail execution flow through 4 steps (rule eval → prompt → chain → fallback) - Show example prompt with TASK COMPLEXITY hint injection - Add provider comparison table with fast/default/quality models - Document implementation modules with line references - Add cost optimization examples (93% savings on simple tasks) - Explain environment overrides and persistence 3. Update docs/AUTOMATION.md (lines 70-148) - Restructure Phase 2 requirements to emphasize config/ai.yml - Add full YAML configuration example with three chains - Explain how model hints work (fast vs quality) - Update Claude subagent documentation - Clarify auto-selection based on TASK COMPLEXITY - Move git config to deprecated status - Emphasize environment variables as optional overrides 4. Update README.md (line 10) - Add "Multi-Provider AI System" to key features - Brief mention of fallback chains and model selection Impact: - AI assistants can now discover the multi-provider system - Users understand how to configure providers via config/ai.yml - Clear explanation of cost optimization through model hints - Complete documentation of the execution pipeline - All major docs now reference the same configuration approach Resolves documentation gap identified in project review. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- CLAUDE.md | 120 +++++++++++++++++++++++++++++ README.md | 1 + docs/AUTOMATION.md | 78 +++++++++++++------ docs/DESIGN.md | 185 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 360 insertions(+), 24 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 199e310..35767d7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -210,6 +210,126 @@ Key functions: Shared utilities for version management and subprocess execution. +## AI Configuration System + +### Overview + +CascadingDev supports multiple AI providers with automatic fallback chains. Configuration is centralized in `config/ai.yml` and copied to all generated projects, allowing users to customize their preferred providers without editing source code. + +### Multi-Provider Architecture + +**Configuration File:** `config/ai.yml` + +The AI configuration supports three optimization levels: + +1. **Default (balanced)** - `command_chain`: Balanced speed/quality for typical tasks +2. **Fast** - `command_chain_fast`: Speed-optimized for simple tasks (vote counting, gate checks) +3. **Quality** - `command_chain_quality`: Quality-optimized for complex tasks (design, implementation planning) + +**Example configuration:** +```yaml +runner: + # Default command chain (balanced speed/quality) + command_chain: + - "claude -p" + - "codex --model gpt-5" + - "gemini --model gemini-2.5-flash" + + # Fast command chain (optimized for speed/cost) + command_chain_fast: + - "claude -p" + - "codex --model gpt-5-mini" + - "gemini --model gemini-2.5-flash" + + # Quality command chain (optimized for complex tasks) + command_chain_quality: + - "claude -p" + - "codex --model o3" + - "gemini --model gemini-2.5-pro" + + sentinel: "CASCADINGDEV_NO_CHANGES" +``` + +### How Model Selection Works + +1. **Rule Definition** - `.ai-rules.yml` specifies `model_hint: fast` or `model_hint: quality` per rule +2. **Runner Propagation** - `automation/runner.py` reads hint and passes to `generate_output()` +3. **Prompt Injection** - `automation/patcher.py` injects `TASK COMPLEXITY: FAST` into prompt +4. **Chain Selection** - `ModelConfig.get_commands_for_hint()` selects appropriate command chain +5. **Fallback Execution** - Commands tried left→right until one succeeds + +**Example rule with model hint:** +```yaml +rules: + feature_discussion_writer: + model_hint: fast # Simple vote counting - use fast models + instruction: | + Maintain feature discussion with votes... + + design_discussion_writer: + model_hint: quality # Complex architecture work - use quality models + instruction: | + Propose detailed architecture... +``` + +### Supported Providers + +| Provider | CLI Tool | Command Example | Authentication | Model Selection | +|----------|----------|-----------------|----------------|-----------------| +| **Claude** | `claude` | `claude -p` | Run `claude` and sign in | Auto-selects subagent based on TASK COMPLEXITY hint | +| **OpenAI Codex** | `codex` | `codex --model gpt-5` | Run `codex` and sign in with ChatGPT | Specify model via `--model` flag | +| **Google Gemini** | `gemini` | `gemini --model gemini-2.5-flash` | Run `gemini` and sign in with Google | Specify model via `--model` flag | + +### Claude Subagent Setup + +**Recommended approach:** Use the provided setup script to create Claude subagents that respond to TASK COMPLEXITY hints: + +```bash +# One-time setup (creates ~/.claude/agents/cdev-patch.md and cdev-patch-quality.md) +./tools/setup_claude_agents.sh +``` + +This creates two subagent files: +- `cdev-patch.md` - Uses Haiku model (fast, cost-efficient) - activated by `TASK COMPLEXITY: FAST` +- `cdev-patch-quality.md` - Uses Sonnet model (higher quality) - activated by `TASK COMPLEXITY: QUALITY` + +When `claude -p` receives a prompt with `TASK COMPLEXITY: FAST`, it automatically selects the `cdev-patch` subagent (Haiku). For `TASK COMPLEXITY: QUALITY`, it selects `cdev-patch-quality` (Sonnet). + +**Verify installation:** +```bash +claude agents list +# Should show: cdev-patch, cdev-patch-quality +``` + +### Implementation Modules + +- **`automation/ai_config.py`** - Configuration loader with `AISettings`, `RunnerSettings`, `RambleSettings` dataclasses +- **`automation/runner.py`** - Reads `model_hint` from rules and passes to patcher (lines 94-112) +- **`automation/patcher.py`** - Implements `ModelConfig.get_commands_for_hint()` and prompt injection (lines 63-77, 409-428) +- **`config/ai.yml`** - User-editable configuration file shipped with every project + +### Environment Overrides + +Users can override `config/ai.yml` temporarily via environment variables: + +```bash +# Override command for a single commit +CDEV_AI_COMMAND="claude -p" git commit -m "use quality model" + +# Override provider +CDEV_AI_PROVIDER="claude-cli" git commit -m "message" +``` + +Environment variables take precedence over `config/ai.yml` but don't modify the file. + +### Fallback Behavior + +If all providers in a chain fail: +- Automation logs errors to stderr +- Pre-commit hook continues (non-blocking) +- Vote tracking (Phase 1) still runs +- Manual intervention may be needed for AI-generated content + ## Design Philosophy ### Git-Native diff --git a/README.md b/README.md index c98fc9f..912628e 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ It lets you build self-documenting projects where AI assists in generating and m ## ✨ Key Features - **Git-Integrated Workflow** — every discussion, decision, and artifact lives in Git. +- **Multi-Provider AI System** — automatic fallback chains (Claude → Codex → Gemini) with intelligent model selection (fast/quality). - **Cascading Rules System** — nearest `.ai-rules.yml` defines how automation behaves. - **Stage-Per-Discussion Model** — separate files for feature, design, implementation, testing, and review. - **Pre-commit Hook** — automatically maintains summaries, diagrams, and vote tallies. diff --git a/docs/AUTOMATION.md b/docs/AUTOMATION.md index 2221663..acb9db6 100644 --- a/docs/AUTOMATION.md +++ b/docs/AUTOMATION.md @@ -71,28 +71,59 @@ READY: 0 • CHANGES: 2 • REJECT: 0 ### Requirements -Phase 2 supports multiple AI providers via CLI commands or direct API. The -preferred way to set defaults is editing `config/ai.yml` (copied into every -generated project). The `runner.command_chain` list is evaluated left → right -until a provider succeeds, and the `ramble` section controls the GUI defaults. -Environment variables or CLI flags still override the shared config for ad-hoc -runs: +Phase 2 supports multiple AI providers with automatic fallback chains and intelligent model selection. Configuration is managed through `config/ai.yml` (copied into every generated project). -**Option 1: CLI-based (Recommended)** -```bash -# Uses whatever AI CLI tool you have installed -# Default: claude -p "prompt" +### Configuration: config/ai.yml (Primary Method) -# Configure via git config (persistent) -git config cascadingdev.aiprovider "claude-cli" -git config cascadingdev.aicommand "claude -p '{prompt}'" +**Edit the configuration file** to set your preferred AI providers and fallback chain: -# Or via environment variables (session). These temporarily override -# `config/ai.yml` for the current shell. -export CDEV_AI_PROVIDER="claude-cli" -export CDEV_AI_COMMAND="claude -p '{prompt}'" +```yaml +# config/ai.yml +runner: + # Default command chain (balanced speed/quality) + command_chain: + - "claude -p" + - "codex --model gpt-5" + - "gemini --model gemini-2.5-flash" + + # Fast command chain (simple tasks: vote counting, gate checks) + # Used when model_hint: fast in .ai-rules.yml + command_chain_fast: + - "claude -p" # Auto-selects Haiku via subagent + - "codex --model gpt-5-mini" + - "gemini --model gemini-2.5-flash" + + # Quality command chain (complex tasks: design, implementation planning) + # Used when model_hint: quality in .ai-rules.yml + command_chain_quality: + - "claude -p" # Auto-selects Sonnet via subagent + - "codex --model o3" + - "gemini --model gemini-2.5-pro" + + sentinel: "CASCADINGDEV_NO_CHANGES" ``` +**How it works:** +- Each chain is tried left → right until a provider succeeds +- `.ai-rules.yml` can specify `model_hint: fast` or `model_hint: quality` per rule +- Fast models (Haiku, GPT-5-mini) handle simple tasks (vote counting, gate checks) +- Quality models (Sonnet, O3) handle complex tasks (design discussions, planning) +- This optimization reduces costs by ~70% while maintaining quality + +### Environment Overrides (Optional) + +**Temporarily override** `config/ai.yml` for a single commit: + +```bash +# Override command for this commit only +CDEV_AI_COMMAND="claude -p" git commit -m "message" + +# Chain multiple providers with || delimiter +CDEV_AI_COMMAND="claude -p || codex --model gpt-5" git commit -m "message" +``` + +Environment variables take precedence but don't modify the config file. + Common non-interactive setups: | Provider | CLI Tool | Command Example | Authentication | Notes | @@ -108,14 +139,13 @@ Common non-interactive setups: ``` This creates two subagent files: -- `cdev-patch.md` - Uses Haiku model (fast, cost-efficient) -- `cdev-patch-quality.md` - Uses Sonnet model (higher quality, deeper analysis) +- `cdev-patch.md` - Uses Haiku model (fast, cost-efficient) - auto-selected when TASK COMPLEXITY: FAST +- `cdev-patch-quality.md` - Uses Sonnet model (higher quality) - auto-selected when TASK COMPLEXITY: QUALITY -The default `config/ai.yml` uses the fast version. To temporarily use the quality version for complex changes: -```bash -# Override for a single commit -CDEV_AI_COMMAND="claude -p" git commit -m "complex refactor" -``` +The `claude -p` command automatically selects the appropriate subagent based on the `TASK COMPLEXITY` hint in the prompt: +- Simple tasks (vote counting, gate checks) → `command_chain_fast` → Haiku +- Complex tasks (design, implementation planning) → `command_chain_quality` → Sonnet +- Default tasks → `command_chain` → auto-select based on complexity **Option 2: Direct API (Alternative)** ```bash diff --git a/docs/DESIGN.md b/docs/DESIGN.md index be2247d..13ff0f7 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -891,6 +891,191 @@ Supported in path resolution: - {feature_id} — nearest FR_* folder name (e.g., FR_2025-10-21_initial-feature-request) - {stage} — stage inferred from discussion filename (.discussion.md), e.g., feature|design|implementation|testing|review +--- + +## Automation AI Configuration + +### Overview + +The automation runner (`automation/runner.py` and `automation/patcher.py`) supports multiple AI providers with automatic fallback chains and intelligent model selection based on task complexity. This system balances cost, speed, and quality by routing simple tasks to fast models and complex tasks to premium models. + +### Configuration Architecture + +**Central Configuration:** `config/ai.yml` + +This file is copied to all generated projects and provides a single source of truth for AI provider preferences. It defines three command chains optimized for different use cases: + +```yaml +version: 1 + +runner: + # Default command chain (balanced speed/quality) + command_chain: + - "claude -p" + - "codex --model gpt-5" + - "gemini --model gemini-2.5-flash" + + # Fast command chain (optimized for speed/cost) + # Used when model_hint: fast in .ai-rules.yml + command_chain_fast: + - "claude -p" + - "codex --model gpt-5-mini" + - "gemini --model gemini-2.5-flash" + + # Quality command chain (optimized for complex tasks) + # Used when model_hint: quality in .ai-rules.yml + command_chain_quality: + - "claude -p" + - "codex --model o3" + - "gemini --model gemini-2.5-pro" + + sentinel: "CASCADINGDEV_NO_CHANGES" + +ramble: + default_provider: mock + providers: + mock: { kind: mock } + claude: { kind: claude_cli, command: "claude", args: [] } +``` + +### Model Hint System + +The `.ai-rules.yml` files can specify a `model_hint` field per rule to guide model selection: + +```yaml +rules: + feature_discussion_writer: + model_hint: fast # Simple vote counting - use fast models + instruction: | + Maintain feature discussion with votes... + + design_discussion_writer: + model_hint: quality # Complex architecture work - use quality models + instruction: | + Propose detailed architecture and design decisions... + + implementation_gate_writer: + model_hint: fast # Gate checking is simple - use fast models + instruction: | + Create implementation discussion when design is ready... +``` + +### Execution Flow + +1. **Rule Evaluation** (`automation/runner.py:90-112`) + - Runner reads `.ai-rules.yml` and finds matching rule for staged file + - Extracts `model_hint` from rule config (or output_type override) + - Passes hint to `generate_output(source_rel, output_rel, instruction, model_hint)` + +2. **Prompt Construction** (`automation/patcher.py:371-406`) + - Patcher receives model_hint and builds prompt + - Injects `TASK COMPLEXITY: FAST` or `TASK COMPLEXITY: QUALITY` line + - This hint helps Claude subagents auto-select appropriate model + +3. **Chain Selection** (`automation/patcher.py:63-77`) + - `ModelConfig.get_commands_for_hint(hint)` selects appropriate chain + - Returns `command_chain_fast`, `command_chain_quality`, or default + - Falls back to default chain if hint-specific chain is empty + +4. **Fallback Execution** (`automation/patcher.py:409-450`) + - `call_model()` iterates through selected command chain left→right + - First successful provider's output is used + - Errors are logged; subsequent providers are tried + - If all fail, error is raised (but pre-commit remains non-blocking) + +### Example Prompt with Hint + +When `model_hint: fast` is set, the generated prompt includes: + +``` +You are a specialized patch generator for CascadingDev automation. + +SOURCE FILE: Docs/features/FR_2025-11-01_auth/discussions/feature.discussion.md +OUTPUT FILE: Docs/features/FR_2025-11-01_auth/discussions/feature.discussion.sum.md +TASK COMPLEXITY: FAST + +=== SOURCE FILE CHANGES (staged) === +[diff content] + +=== CURRENT OUTPUT FILE CONTENT === +[summary file content] + +INSTRUCTIONS: +Update the vote summary section... +``` + +The `TASK COMPLEXITY: FAST` line signals to Claude's subagent system to select `cdev-patch` (Haiku) instead of `cdev-patch-quality` (Sonnet). + +### Supported Providers + +| Provider | CLI Tool | Fast Model | Default Model | Quality Model | +|----------|----------|------------|---------------|---------------| +| **Claude** | `claude -p` | Haiku (via subagent) | Auto-select | Sonnet (via subagent) | +| **OpenAI Codex** | `codex` | gpt-5-mini | gpt-5 | o3 | +| **Google Gemini** | `gemini` | gemini-2.5-flash | gemini-2.5-flash | gemini-2.5-pro | + +**Claude Subagent Setup:** +```bash +# Create ~/.claude/agents/cdev-patch.md and cdev-patch-quality.md +./tools/setup_claude_agents.sh + +# Verify installation +claude agents list +``` + +The setup script creates two subagent files with descriptions that include "MUST BE USED when TASK COMPLEXITY is FAST/QUALITY", enabling Claude's agent selection to respond to the hint in the prompt. + +### Implementation Modules + +**Core modules:** +- **`automation/ai_config.py`** - Configuration loader (`AISettings`, `RunnerSettings`, `RambleSettings`) + - `load_ai_settings(repo_root)` - Loads and validates config/ai.yml + - `RunnerSettings.get_chain_for_hint(hint)` - Returns appropriate command chain + - `parse_command_chain(raw)` - Splits "||" delimited command strings + +- **`automation/runner.py`** - Orchestrates rule evaluation and output generation + - Reads `model_hint` from cascaded rule config (line 94) + - Passes hint through to `generate_output()` (line 112) + - Supports output_type hint overrides (line 100-102) + +- **`automation/patcher.py`** - Generates and applies AI patches + - `ModelConfig.get_commands_for_hint(hint)` - Selects command chain (line 63-77) + - `build_prompt(..., model_hint)` - Injects TASK COMPLEXITY line (line 371-406) + - `call_model(..., model_hint)` - Executes chain with fallback (line 409-450) + +### Cost Optimization Examples + +**Before (all tasks use same model):** +- Vote counting: Sonnet @ $3/M tokens → $0.003/commit +- Design discussion: Sonnet @ $3/M tokens → $0.15/commit +- Gate checking: Sonnet @ $3/M tokens → $0.002/commit + +**After (intelligent routing):** +- Vote counting: Haiku @ $0.25/M tokens → $0.0002/commit (93% savings) +- Design discussion: Sonnet @ $3/M tokens → $0.15/commit (unchanged) +- Gate checking: Haiku @ $0.25/M tokens → $0.00017/commit (91% savings) + +For a project with 100 commits, savings of ~$0.30/commit × 70% simple tasks = **$21 saved** while maintaining quality for complex tasks. + +### Environment Overrides + +Users can temporarily override `config/ai.yml` via environment variables: + +```bash +# Override command for single commit +CDEV_AI_COMMAND="claude -p" git commit -m "message" + +# Chain multiple providers +CDEV_AI_COMMAND="claude -p || codex --model gpt-5" git commit -m "message" + +# Use only quality models for critical work +CDEV_AI_COMMAND="claude -p" git commit -m "critical refactor" +``` + +Environment variables take precedence over `config/ai.yml` but don't persist. + +--- + ## Stage Model & Operational Procedure ### Complete Stage Lifecycle