# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

CmdForge is a lightweight personal tool builder for AI-powered CLI commands. It lets users create custom terminal commands that call AI providers, chain prompts with Python code steps, and use them like any Unix pipe command.

## Development Commands

```bash
# Install for development
pip install -e ".[dev]"

# Run all unit tests (excluding integration tests that need a server)
pytest tests/ -m "not integration"

# Run a specific test file
pytest tests/test_runner.py -v

# Run a specific test class or method
pytest tests/test_runner.py::TestSubstituteVariables -v
pytest tests/test_runner.py::TestSubstituteVariables::test_simple_substitution -v

# Run with coverage
pytest tests/ --cov=cmdforge --cov-report=html

# Run integration tests (requires local registry server at localhost:5000)
python -m cmdforge.registry.app  # Start server first
pytest tests/test_registry_integration.py -v -m integration

# CLI entry points
cmdforge              # Main CLI / GUI launcher
cf                    # Interactive tool picker (fzf-style)
python -m cmdforge.cli  # Alternative CLI invocation
```

## Architecture

### Core Modules (`src/cmdforge/`)

- **cli/**: CLI commands entry points (`cmdforge` command). Routes subcommands: list, create, edit, delete, test, run, ui, docs, check, refresh, providers, registry, collections, deps, install, lock, verify, add, remove, init, config, settings, system-deps
- **tool.py**: Tool definition dataclasses (`Tool`, `ToolArgument`, `PromptStep`, `CodeStep`, `ToolStep`, `McpStep`), YAML config loading/saving, wrapper script generation
- **runner.py**: Execution engine. Runs tool steps sequentially, handles variable substitution (`{input}`, `{varname}`), executes Python code steps via `exec()`, handles nested tool calls with depth limit (MAX_TOOL_DEPTH=10)
- **resolver.py**: Tool resolution. `resolve_tool()` searches: project manifest → local tools → owner/name → registry. Returns `ResolvedTool` with path info
- **collection.py**: Collection management (`Collection` dataclass, `resolve_tool_references()`, `classify_tool_reference()`), local collection storage in `~/.cmdforge/collections/`
- **providers.py**: Provider abstraction. Supports subprocess CLI tools, OpenAI-compatible HTTP APIs, experimental PTY wrappers, fallback chains, and tool/MCP-server allowlists. Auto-discovers installed providers on first run. Config in `~/.cmdforge/providers.yaml` with versioned migration.
- **skills.py**: Validated Agent Skills loader for per-provider `SKILL.md` context under `~/.cmdforge/providers/<name>/skills/`
- **mcp_client.py / mcp_server.py**: Stdio MCP client/server integration, configuration, schema discovery, and exposure policy
- **profiles.py**: AI persona profiles with system prompts, stored in `~/.cmdforge/profiles/`
- **manifest.py**: Project manifest (`cmdforge.yaml`) for declaring tool dependencies with version constraints
- **lockfile.py**: Lock file support for reproducible installs (`cmdforge.lock`)
- **registry_client.py**: Client for registry API (search, publish, download, authentication)
- **config.py**: Global configuration management (registry URL, auth tokens)
- **system_deps.py**: System package dependency management (apt/dnf/pacman detection and installation)
- **dependency_graph.py**: Dependency graph resolution for meta-tools
- **hash_utils.py**: Hash verification utilities for tool integrity
- **gui/**: PySide6 desktop GUI
  - **main_window.py**: Main application window with sidebar navigation
  - **pages/**: Welcome page, Tools page, Tool Builder, Registry browser, Collections page, Providers management, Profiles
  - **dialogs/**: Step editors (prompt/code/tool), Argument editor, Provider dialog, Provider Install dialog, Connect/Publish dialogs, Test Step dialog, Settings dialog, System Dep dialog, Help dialogs, Review/Issue dialogs
  - **widgets/**: Flow graph visualization (`flow_graph.py`), Icon utilities (`icons.py`)

### Key Paths

- **Tools storage**: `~/.cmdforge/<toolname>/config.yaml`
- **Tool defaults**: `~/.cmdforge/<toolname>/defaults.yaml` (optional, published with tool)
- **Tool settings**: `~/.cmdforge/<toolname>/settings.yaml` (user overrides, auto-created from defaults)
- **Wrapper scripts**: `~/.local/bin/<toolname>` (auto-generated bash scripts)
- **Provider config**: `~/.cmdforge/providers.yaml`
- **Collections storage**: `~/.cmdforge/collections/<name>.yaml`

### Tool Structure

Tools are YAML configs with:
- `name`, `description`, `category`
- `arguments`: Custom flags with defaults (e.g., `--max` → `{max}`)
- `steps`: Ordered list of `prompt`, `code`, or `tool` steps
- `output`: Template for final output (e.g., `"{response}"`)
- `input_schema`, `output_schema`: Optional tool-level JSON Schema contracts;
  schemas are validated when tools are loaded or created

Run `cmdforge inspect <tool> [--registry]` for the shared local preflight
report. `cmdforge registry publish <path> --dry-run` runs local checks first
and, when authenticated, the registry's publish-time checks without publishing.

### Step Types

1. **Prompt Step**: Calls AI provider with template, stores result in `output_var`
   - `profile`: AI persona (system prompt prefix)
   - `strip_fences`: Remove markdown code fences from output
   - `output_schema`: JSON schema for validated structured responses
   - `max_tokens`: Max output tokens (provider-dependent, e.g., 4096 for haiku)
   - `plain_text`: Bypass structured output enforcement
2. **Code Step**: Executes Python code via `exec()`, captures specified variables (comma-separated for multiple outputs)
3. **Tool Step**: Calls another tool (meta-tools), supports `input_template`, `args`, and recursive agent context (`provider`, `profile`, `skills`, `tools`). Provider and delegated allowlists are both enforced before resolution.
4. **MCP Step**: Calls a tool on a configured MCP server. Provider `mcp_servers` allowlists are enforced before connecting.

### Variable Flow

Variables are passed between steps:
- `{input}` - always available (stdin/file content)
- `{argname}` - from tool arguments
- `{step_output_var}` - from previous step's `output_var`
- `{settings.key}` - from tool's settings.yaml (top-level scalars only in templates)
- `settings['key']` - full dict access in code steps

Variable substitution is handled in `runner.py:substitute_variables()`. Settings are loaded from `settings.yaml` if it exists, otherwise an empty dict is used.

## Provider System

Providers wrap AI CLIs or compatible HTTP APIs. Defined in `~/.cmdforge/providers.yaml`:

```yaml
version: 2
providers:
  - name: claude
    command: "claude -p"
    description: "Anthropic Claude"
    type: subprocess
  - name: openrouter
    command: "https://openrouter.ai/api/v1"
    description: "OpenRouter API"
    type: api
    model: openrouter/auto-beta
    api_key_env: OPENROUTER_API_KEY
  - name: mock
    command: "mock"
    description: "Mock provider for testing"
```

Provider fields:
- `name`: Provider identifier used in tool configs
- `command`: Shell command (subprocess/pty) or endpoint URL (api)
- `type`: `subprocess` (default), `api` (OpenAI-compatible HTTP), or `pty` (interactive CLI, experimental)
- `model`: Model ID for api-type providers
- `api_key_env`: Environment variable holding the API key for api-type providers
- `description`: Optional human-readable description
- `fallback`: Optional provider to try if this one fails
- `fallback_chain`: Ordered list of providers for multi-step fallback
- `tags`: List of strings (e.g. `["free", "code", "local"]`)
- `pty_config`: Dict with `prompt_pattern`, `response_pattern`, `exit_command` for pty providers
- `install`: Optional structured install metadata dict
- `tools`: Optional CmdForge tool allowlist; `null` allows all and `[]` denies all
- `mcp_servers`: Optional MCP server allowlist; `null` allows all and `[]` denies all

The `mock` provider is built-in for testing without API calls. Use `--provider mock` or `--dry-run` flags when testing tools.

Provider CLI commands:
- `cmdforge providers list` - List all providers and their status
- `cmdforge providers check` - Check which providers are available
- `cmdforge providers add <name> <command>` - Add/update a provider (supports `--type`, `--model`, `--api-key-env`, `--tag`, `--fallback`, `--fallback-chain`)
- `cmdforge providers remove <name>` - Remove a provider
- `cmdforge providers test <name>` - Test a provider
- `cmdforge providers discover [--add]` - Scan system for installed CLIs and API keys
- `cmdforge providers install` - Interactive guide to install AI providers
- `cmdforge providers for-tools <tool> [tools...]` - List providers used by tools (with `--warm` to pre-load local models)

## Web UI & Registry

CmdForge includes a web interface and tool registry:

### Web Modules (`src/cmdforge/web/`)

- **app.py**: Flask app factory, registers blueprints
- **routes.py**: Main web routes (docs, tutorials, tools, etc.)
- **auth.py**: Authentication middleware and decorators
- **sessions.py**: Session management
- **email.py**: Email utilities (password reset, notifications)
- **seo.py**: SEO utilities (sitemap, meta tags)
- **filters.py**: Jinja2 filters (timeago, markdown, etc.)
- **docs_content.py**: Documentation and tutorial content
- **forum/**: Community forum blueprint
  - **models.py**: Forum database schema (categories, topics, replies)
  - **routes.py**: Forum routes (/forum, /forum/c/<slug>, /forum/t/<id>)

### Registry Modules (`src/cmdforge/registry/`)

- **app.py**: Flask-based Registry API (tool publishing, search, downloads, authentication, rate limiting)
- **db.py**: SQLite schema and queries (`connect_db()`, `query_one()`, `query_all()`)
- **embeddings.py**: Semantic search with vector embeddings for tool discovery
- **sync.py**: Git-based tool sync from Gitea repository
- **categorize.py**: Auto-categorization of tools based on content
- **rate_limit.py**: Rate limiting for API endpoints
- **scrutiny.py**: Tool vetting (honesty, transparency, scope, efficiency checks)
- **similarity.py**: Duplicate detection via embedding similarity
- **stats.py**: Registry statistics and metrics
- **settings.py**: Server-side settings management

### Key URLs

- `/forum` - Community forum
- `/docs` - Documentation
- `/tutorials` - Tutorial guides
- `/tools` - Tool registry browser

### Running the Web UI

```bash
# Development
python -m cmdforge.web.app

# Production (example)
CMDFORGE_REGISTRY_DB=/path/to/db PORT=5050 python -m cmdforge.web.app
```

## Testing Conventions

Tests use `pytest` without a shared conftest.py. Common patterns:

**Mocking strategy:**
- **File system**: Use `tmp_path` fixture and patch module-level paths (`TOOLS_DIR`, `BIN_DIR`, `PROVIDERS_FILE`)
- **Subprocess calls**: Mock `subprocess.run` and `shutil.which`
- **Provider calls**: Mock `call_provider` to return `ProviderResult(text="...", success=True)`
- **Registry calls**: Mock `requests` for API tests

**Example fixture pattern:**
```python
@pytest.fixture
def temp_providers_file(tmp_path):
    providers_file = tmp_path / "providers.yaml"
    with patch('cmdforge.providers.PROVIDERS_FILE', providers_file):
        yield providers_file
```

Integration tests are marked with `@pytest.mark.integration` and require a running registry server.

## Infrastructure Documentation

For deployment, server details, and operations, see the `docs/` folder:

- **docs/servers.md** - Server IPs (192.168.0.162), SSH access, paths, service commands
- **docs/deployment.md** - Architecture diagram, deploy process, systemd service config
- **docs/maintenance.md** - Backups, updates, troubleshooting
- **docs/architecture.md** - Module structure, data flow diagrams

### Production Server Quick Reference

| Property | Value |
|----------|-------|
| Server | 192.168.0.162 (OpenMediaVault) |
| SSH | `ssh rob@192.168.0.162` |
| App Path | `/srv/mergerfs/data_pool/home/rob/cmdforge-registry/` |
| Service | `systemctl --user status cmdforge-web` |
| Public URL | https://cmdforge.brrd.tech |
| Port | 5050 (via Cloudflare tunnel) |