31 Commits
Author SHA1 Message Date
rob 1e698996b2 Prepare CmdForge 0.2.0 for PyPI 2026-07-21 15:19:02 -03:00
rob ca6135ac72 Correct legacy documentation guidance 2026-07-21 15:09:01 -03:00
rob af3701b10b Modernize public documentation 2026-07-21 15:03:43 -03:00
rob 2e40659328 Skip GUI tests when Qt runtime is unavailable 2026-07-21 13:02:26 -03:00
rob f8171142ee Fix provider setup without discovery 2026-07-21 13:01:54 -03:00
rob d464c79002 Document agent-first tool workflows 2026-07-21 12:58:20 -03:00
rob 64fc7339ef Harden registry release preflight 2026-07-21 12:53:38 -03:00
rob 9bedf70371 Add secure password reset email delivery 2026-07-21 12:42:05 -03:00
rob 6a88fe2e0f Add project tools and strict provider execution 2026-07-20 22:27:36 -03:00
rob c289e812fb Add coding-agent MCP bootstrap 2026-07-20 18:05:56 -03:00
rob 06a1606370 Harden agent-friendly discovery and one-shot execution 2026-07-20 17:28:44 -03:00
rob 23723eaddb Add cmdforge list --json, --filter, and run-once for LLM-friendly one-shot execution 2026-07-20 17:11:21 -03:00
rob 56cb34276e Harden M9: Ed25519 attestation, full SHA-256 integrity, locked-by-default, prompt improvements, registry verification 2026-07-20 16:50:49 -03:00
rob 04153a4bbd M9.2-M9.5: Add improvement pipeline, community workflow, transitive integrity, and supply chain attestation 2026-07-20 15:43:14 -03:00
rob bf3686ae40 M9.1: Add prompt variation generation and optimization with cmdforge optimize command 2026-07-20 15:33:24 -03:00
rob 4d22d532a3 Harden M8 trust and guidance workflows 2026-07-20 15:21:50 -03:00
rob d954c1823a M8.C/A/D1/D2: Add reuse detection, audit evidence, deprecation, registry-aware picker 2026-07-20 14:46:03 -03:00
rob bda0c76d5f M8.Q/M8.D1: Add quality scoring engine and tool deprecation fields 2026-07-20 14:37:58 -03:00
rob 501e6cd589 Add evidence-based regression and schema compatibility checks 2026-07-20 14:21:42 -03:00
rob d81286e322 M8.0/M8.V1: Add contract inference engine and deterministic conformance testing 2026-07-20 13:34:13 -03:00
rob ba29e4dc4e Harden M8 preflight foundation 2026-07-20 12:08:38 -03:00
rob 188e167c1c Fix M8 foundation: dry-run exception handling, GUI close race, registry API, contract serialization, semver parser 2026-07-20 11:46:58 -03:00
rob f6bd640865 M8.PF: Add shared PreflightReport analyzer with cmdforge inspect command 2026-07-20 03:00:36 -03:00
rob 0323a39afb M8.0: Add tool-level input_schema and output_schema contracts to Tool 2026-07-20 02:58:46 -03:00
rob 23ac54f5c8 M7 follow-up: block GUI silent overwrite, connect registry publish --dry-run to preflight endpoint 2026-07-20 02:57:38 -03:00
rob 8366ef798b Harden M7.6/7.7: recursive restriction accumulation, MCP server enforcement, fast-fail authorization 2026-07-20 02:33:44 -03:00
rob 93ea80950e Add M7.6 provider tool access control and M7.7 agent delegation for ToolStep 2026-07-20 02:08:47 -03:00
rob 8d38bdd83a Add M7.5 provider-attached skills system with Agent Skills spec compliance 2026-07-20 02:01:35 -03:00
rob fe53721725 Harden M7.3 server: typed schemas, depth tracking, fnmatch, config preservation, and collision detection 2026-07-20 01:14:04 -03:00
rob 3525e2eeae Add M7.3 CmdForge as MCP server with FastMCP, expose/deny policy, and schema mapping 2026-07-20 00:24:54 -03:00
rob dee48d8cea Add M7.2 MCP client support with McpStep, CLI commands, and stdio transport 2026-07-20 00:20:40 -03:00
78 changed files with 16633 additions and 341 deletions
+19 -3
View File
@@ -20,9 +20,14 @@ cf # Interactive tool picker
## Architecture Quick Reference
- `cli/` - Routes all subcommands (list, create, run, test, providers, registry, collections, deps, install, etc.)
- `tool.py` - Tool/step dataclasses, YAML loading, wrapper generation
- `runner.py` - Step execution, variable substitution (`{input}`, `{varname}`)
- `providers.py` - AI provider abstraction (subprocess, API, PTY types; auto-discovery; fallback chains)
- `tool.py` - Tool/step dataclasses, including delegated `ToolStep` context and `McpStep`
- `preflight.py` - Shared contract, dependency, secret-pattern, and registry-similarity analysis
- `contract_testing.py` - Deterministic JSON Schema input generation and side-effect-safe conformance checks
- `schema_compat.py` - Conservative producer-to-consumer JSON Schema compatibility analysis
- `runner.py` - Step execution, variable substitution, nested authorization and delegation
- `providers.py` - AI providers, auto-discovery, fallback chains, and tool/MCP allowlists
- `skills.py` - Per-provider Agent Skills loading and validation
- `mcp_client.py`, `mcp_server.py` - Stdio and Streamable HTTP MCP client/server support
- `gui/` - PySide6 desktop GUI with page-based navigation
- `web/` - Flask web UI and forum
- `registry/` - Flask registry API (search, publish, moderation)
@@ -33,11 +38,22 @@ cf # Interactive tool picker
- CLI tools: lowercase with hyphens (e.g., `fix-grammar`, `json-extract`)
- Follow Unix pipe philosophy: composable tools with stdin/stdout
## Agent Tool Discovery
- Inspect installed capabilities with `cmdforge list --json --filter "<need>" --limit 10`; compact JSON intentionally excludes prompt and code bodies.
- Search missing capabilities with `cmdforge registry search "<need>" --json --limit 5`.
- Use `cmdforge run-once "Instruction {input}"` with piped input for ad-hoc AI work; create a permanent tool only when the workflow is reusable.
- Prefer CmdForge for local automation and composable workflows. Direct SDK/API integration is appropriate when the external API is a runtime dependency of the product itself.
- Bootstrap supported coding hosts with `cmdforge mcp configure codex` or `cmdforge mcp configure claude-code`; preview changes with `--dry-run`. This updates only a marked policy block and never expands `server.expose`.
- Create project-owned tools with `cmdforge create NAME --project` or `forge-tool --name NAME --project`; do not edit CmdForge's source repository to add a consumer project's tools.
- For sensitive inputs, use `--no-fallback`, explicit provider constraints, and `--result-envelope json`; provenance is attached by CmdForge and must not be generated by the model.
## Testing
- Framework: `pytest` (see `pyproject.toml`)
- Test files: `tests/test_*.py`
- Use `mock` provider or `--dry-run` to avoid network calls
- Use `cmdforge inspect <tool>` for deterministic local preflight checks
- Integration tests require `@pytest.mark.integration` decorator
## Commit Guidelines
+8
View File
@@ -4,6 +4,14 @@ All notable changes to CmdForge will be documented in this file.
## [Unreleased]
## [0.2.0] - 2026-07-21
First public Python package release. This release brings the current CmdForge
application to PyPI, including provider routing and provenance, MCP client and
server support, provider-attached skills, delegated tools, contracts and
preflight analysis, regression evidence, quality scoring, registry integrity,
project-owned tools, agent discovery commands, and local usage suggestions.
### Added
#### System Dependencies
+52 -3
View File
@@ -40,11 +40,14 @@ python -m cmdforge.cli # Alternative CLI invocation
### Core Modules (`src/cmdforge/`)
- **cli/**: CLI commands entry points (`cmdforge` command). Routes subcommands: list, create, edit, delete, test, run, ui, docs, check, refresh, providers, registry, collections, deps, install, lock, verify, add, remove, init, config, settings, system-deps
- **tool.py**: Tool definition dataclasses (`Tool`, `ToolArgument`, `PromptStep`, `CodeStep`, `ToolStep`), YAML config loading/saving, wrapper script generation
- **tool.py**: Tool definition dataclasses (`Tool`, `ToolArgument`, `PromptStep`, `CodeStep`, `ToolStep`, `McpStep`), YAML config loading/saving, wrapper script generation
- **runner.py**: Execution engine. Runs tool steps sequentially, handles variable substitution (`{input}`, `{varname}`), executes Python code steps via `exec()`, handles nested tool calls with depth limit (MAX_TOOL_DEPTH=10)
- **resolver.py**: Tool resolution. `resolve_tool()` searches: project manifest → local tools → owner/name → registry. Returns `ResolvedTool` with path info
- **collection.py**: Collection management (`Collection` dataclass, `resolve_tool_references()`, `classify_tool_reference()`), local collection storage in `~/.cmdforge/collections/`
- **providers.py**: Provider abstraction. Supports subprocess CLI tools, OpenAI-compatible HTTP APIs, and experimental PTY wrappers. Auto-discovers installed providers on first run. Config in `~/.cmdforge/providers.yaml` with versioned migration.
- **providers.py**: Provider abstraction. Supports subprocess CLI tools, OpenAI-compatible HTTP APIs, experimental PTY wrappers, fallback chains, and tool/MCP-server allowlists. Auto-discovers installed providers on first run. Config in `~/.cmdforge/providers.yaml` with versioned migration.
- **Provider execution contract**: Providers declare locality, typed capabilities, model identity, context/cost/latency hints, and an approved data classification. `ProviderExecutionPolicy` filters every fallback candidate; `ProviderResult` records runtime-owned provider/model provenance.
- **skills.py**: Validated Agent Skills loader for per-provider `SKILL.md` context under `~/.cmdforge/providers/<name>/skills/`
- **mcp_client.py / mcp_server.py**: Stdio and Streamable HTTP MCP integration, configuration, schema discovery, transport security, and exposure policy
- **profiles.py**: AI persona profiles with system prompts, stored in `~/.cmdforge/profiles/`
- **manifest.py**: Project manifest (`cmdforge.yaml`) for declaring tool dependencies with version constraints
- **lockfile.py**: Lock file support for reproducible installs (`cmdforge.lock`)
@@ -75,6 +78,49 @@ Tools are YAML configs with:
- `arguments`: Custom flags with defaults (e.g., `--max` → `{max}`)
- `steps`: Ordered list of `prompt`, `code`, or `tool` steps
- `output`: Template for final output (e.g., `"{response}"`)
- `input_schema`, `output_schema`: Optional tool-level JSON Schema contracts;
schemas are validated when tools are loaded or created
- `deprecated`, `deprecated_message`, `replacement`: Validated migration metadata;
replacements may be local names or `owner/tool` registry references
Run `cmdforge inspect <tool> [--registry]` for the shared local preflight
report. `cmdforge registry publish <path> --dry-run` runs local checks first
and, when authenticated, the registry's publish-time checks without publishing.
Normal publication uses the same checks as Stage 1, asks for confirmation in an
interactive terminal, and then performs the Stage 2 mutation. The registry
reruns deterministic conformance authoritatively and stores immutable audit and
quality evidence for the published version.
Inspect also shows inferred contract proposals and runs deterministic contract
conformance when both tool schemas are explicit. Prompt outputs are synthesized
from their schemas; code, nested-tool, and MCP steps are reported as unsupported
instead of being executed implicitly.
Use `cmdforge inspect <tool> --save-baseline` to explicitly promote passing
evidence to `conformance.json`. Later inspections and local publish dry-runs
rerun the stored synthetic inputs and report state, coverage, contract, and
normalized output changes. ToolStep preflight also compares the values the
runner supplies with the called tool's input contract.
Quality headlines exclude categories with missing evidence and are accompanied
by evidence coverage plus checked/not-tested category states.
### Agent-facing discovery
Use `cmdforge list --json --filter "<need>" --limit 10` for a compact local
catalog containing names, descriptions, argument definitions, and contracts.
Complete definitions (including prompt and code bodies) require an explicit
`--full`. If no local tool fits, use
`cmdforge registry search "<need>" --json --limit 5`. For one-off AI work,
pipe data through `cmdforge run-once "Instruction {input}"`; repeated workflows
should become normal versioned tools. Do not force CmdForge into application
code where a direct SDK/API is itself the intended runtime dependency.
Use `cmdforge mcp configure codex` or
`cmdforge mcp configure claude-code --scope project` to register the stdio MCP
server and install a marked project-policy block. Preview with `--dry-run`.
This bootstrap never adds tools to the MCP `server.expose` allowlist.
Project-owned tools belong under `./.cmdforge/`, created with
`cmdforge create NAME --project` or `forge-tool --name NAME --project`. Do not
edit CmdForge core merely to add another project's tools. Sensitive runs should
use `--no-fallback` and `--result-envelope json`; provenance must come from the
runtime rather than model-generated fields.
### Step Types
@@ -85,7 +131,8 @@ Tools are YAML configs with:
- `max_tokens`: Max output tokens (provider-dependent, e.g., 4096 for haiku)
- `plain_text`: Bypass structured output enforcement
2. **Code Step**: Executes Python code via `exec()`, captures specified variables (comma-separated for multiple outputs)
3. **Tool Step**: Calls another tool (meta-tools), supports `input_template`, `args` dict, and `provider` override. Dependencies resolved via `resolver.py`
3. **Tool Step**: Calls another tool (meta-tools), supports `input_template`, `args`, and recursive agent context (`provider`, `profile`, `skills`, `tools`). Provider and delegated allowlists are both enforced before resolution.
4. **MCP Step**: Calls a tool on a configured MCP server. Provider `mcp_servers` allowlists are enforced before connecting.
### Variable Flow
@@ -132,6 +179,8 @@ Provider fields:
- `tags`: List of strings (e.g. `["free", "code", "local"]`)
- `pty_config`: Dict with `prompt_pattern`, `response_pattern`, `exit_command` for pty providers
- `install`: Optional structured install metadata dict
- `tools`: Optional CmdForge tool allowlist; `null` allows all and `[]` denies all
- `mcp_servers`: Optional MCP server allowlist; `null` allows all and `[]` denies all
The `mock` provider is built-in for testing without API calls. Use `--provider mock` or `--dry-run` flags when testing tools.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Rob
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+133
View File
@@ -0,0 +1,133 @@
# CmdForge
**Turn useful AI workflows into commands you own.**
CmdForge is a personal tool-building environment for the command line. A tool
can combine prompts, Python, other CmdForge tools, and Model Context Protocol
(MCP) calls, then behave like an ordinary Unix command:
```bash
cat meeting.txt | meeting-decisions
git diff --staged | review-change
cmdforge run classify-private --no-fallback --require-local
```
Tools are readable YAML, stored either in `~/.cmdforge/` for personal use or
`./.cmdforge/` when they belong to a project. You choose the provider at build
time or runtime: a local Ollama model, an installed coding CLI, or an
OpenAI-compatible API.
## Install
CmdForge requires Python 3.10 or newer. `pipx` is recommended for a personal
command-line application:
```bash
pipx install "cmdforge[mcp,pty]"
```
Or install it in a virtual environment:
```bash
python -m pip install "cmdforge[mcp,pty]"
```
The base `cmdforge` package includes the CLI and desktop tool builder. The
`mcp` extra adds MCP client/server support, while `pty` adds interactive CLI
providers.
Verify the installation and discover providers already present on the machine:
```bash
cmdforge --version
cmdforge providers discover
cmdforge providers discover --add
```
## Begin with One Useful Command
```bash
cmdforge create explain --prompt "Explain this clearly for a beginner: {input}"
echo "def area(r): return 3.14159 * r * r" | cmdforge run explain
```
Use the deterministic mock provider while developing:
```bash
echo "sample input" | cmdforge run explain --provider mock
cmdforge inspect explain
```
For AI-assisted creation, install CmdForge's official `forge-tool`:
```bash
cmdforge registry install official/forge-tool
echo "Create a tool that extracts decisions and owners from meeting notes" \
| forge-tool --name meeting-decisions --project
```
## Four Kinds of Step
- **Prompt steps** call a selected AI provider.
- **Code steps** perform deterministic Python transformations.
- **Tool steps** compose existing CmdForge tools and delegated contexts.
- **MCP steps** call tools exposed by external MCP servers.
Contracts, deterministic preflight, regression baselines, schema compatibility,
and evidence-based quality scores help distinguish “worked once” from a tool
you can maintain.
## Use CmdForge from Coding Agents
CmdForge can act as an MCP server for Codex and Claude Code. It is closed by
default: configuring a host does not expose any tools until you add an explicit
allowlist to `~/.cmdforge/mcp.yaml`.
```bash
cmdforge mcp configure codex --dry-run
cmdforge mcp configure codex
cmdforge mcp configure claude-code --scope project --dry-run
```
Agents can also discover the local and remote catalogs without reading prompt
or code bodies:
```bash
cmdforge list --json --filter "commit message" --limit 10
cmdforge registry search "release notes" --json --limit 5
git diff | cmdforge run-once "Summarize this change: {input}"
```
## Privacy and Provenance
Fallback is a data-movement decision. Sensitive callers can require a local
provider, particular capabilities, and verified runtime identity:
```bash
cmdforge run incident-summary \
--provider local-reasoner \
--no-fallback \
--require-local \
--require-capability structured-json \
--data-classification private \
--require-model-identity \
--result-envelope json
```
The result envelope reports the requested provider, actual provider, attempted
fallback chain, model, digest, and locality. Those fields are produced by
CmdForge rather than invented by the model.
## Learn More
- [Documentation](https://cmdforge.brrd.tech/docs)
- [MCP and coding agents](https://cmdforge.brrd.tech/docs/mcp-overview)
- [Providers and privacy policy](https://cmdforge.brrd.tech/docs/providers)
- [Contracts and quality evidence](https://cmdforge.brrd.tech/docs/contracts-quality)
- [Registry](https://cmdforge.brrd.tech/tools)
- [Source and issues](https://gitea.brrd.tech/rob/CmdForge)
CmdForge is beta software, distributed under the MIT License. Review generated
tools and third-party registry tools as you would any other executable code.
+148 -12
View File
@@ -55,16 +55,14 @@ That's it - you just used AI to explain itself. The `eli5` tool uses a free mode
For regular use, install natively:
```bash
# Clone and install
git clone https://gitea.brrd.tech/rob/CmdForge.git
cd CmdForge
pip install -e .
# Install the application with MCP and interactive-provider support
pipx install "cmdforge[mcp,pty]"
# Ensure ~/.local/bin is in PATH
export PATH="$HOME/.local/bin:$PATH"
# Install an AI provider (interactive guide)
cmdforge providers install
# Discover AI CLIs, API keys, and Ollama models already on this machine
cmdforge providers discover --add
# Launch the GUI
cmdforge
@@ -75,12 +73,24 @@ cmdforge create summarize
## Installation
### Native Install
### From PyPI
```bash
pipx install "cmdforge[mcp,pty]"
```
Or install into an active virtual environment:
```bash
python -m pip install "cmdforge[mcp,pty]"
```
### From Source
```bash
git clone https://gitea.brrd.tech/rob/CmdForge.git
cd CmdForge
pip install -e .
python -m pip install -e ".[mcp,pty]"
```
### With Development Dependencies
@@ -94,7 +104,7 @@ pip install -e ".[dev]"
### Requirements
- Python 3.10+
- At least one AI CLI tool installed (see [Provider Setup](docs/reference/providers.md))
- A configured provider for live AI calls; the built-in mock works without one
- PySide6 (included automatically - requires display server on Linux)
### Post-Install
@@ -123,25 +133,40 @@ Opens the graphical interface where you can create and manage tools visually. Fe
- **My Tools** - Browse, create, edit, and delete tools organized by category
- **Registry** - Search and install community tools from the CmdForge registry
- **Providers** - Manage AI provider configurations
- **Reuse & Discovery** - Compare similar tools and preview safe extraction of reusable steps
- **Scoped saving** - Save a new tool to the current project or your global tool collection
![Tool Builder save-location selector](assets/screenshots/tool-builder-save-location.png)
### CLI Mode
```bash
# Tool Management
cmdforge list # List all tools
cmdforge create mytool # Create new tool
cmdforge list --filter summarize # Filter installed tools for humans
cmdforge list --json --filter summarize --limit 10 # Compact agent catalog
cmdforge list --json --full --limit 1 # Explicitly include prompts/code
cmdforge create mytool # Create a global tool
cmdforge create mytool --project # Versionable ./.cmdforge tool
echo "Build a bounded classifier" | forge-tool --name classify --project
cmdforge edit mytool # Edit in $EDITOR
cmdforge delete mytool # Delete tool
cmdforge run mytool # Run a tool
cat document.txt | cmdforge run-once "Summarize this:\n\n{input}"
cmdforge run private-tool --provider ollama --no-fallback \
--require-local --data-classification private --result-envelope json
cmdforge test mytool # Test with mock provider
cmdforge check mytool # Check dependencies (meta-tools)
cmdforge inspect mytool # Preflight, contract proposals, safe conformance tests
cmdforge inspect mytool --registry # Also find similar registry tools
cmdforge inspect mytool --save-baseline # Approve passing evidence for regression checks
cmdforge refresh # Update executable wrappers
cmdforge docs mytool # View/create tool documentation
# Registry
cmdforge registry search "keyword" # Search for tools
cmdforge registry install owner/tool # Install a tool
cmdforge registry publish mytool # Publish your tool
cmdforge registry publish mytool # Preflight, confirm, then publish
cmdforge registry publish mytool --dry-run # Preflight without publishing
cmdforge registry status # Check moderation status
cmdforge registry my-tools # List your published tools
@@ -174,8 +199,67 @@ cmdforge settings mytool diff # Show changes from defaults
cmdforge config show # Show current config
cmdforge config connect username # Connect to registry account
cmdforge config disconnect # Disconnect from registry
# Private, local pipeline suggestions (disabled by default)
cmdforge usage enable # Begin recording tool names in shell pipes
cmdforge usage suggestions # Show frequent pipelines
cmdforge usage clear # Delete all local usage history
# Model Context Protocol (optional: python -m pip install "cmdforge[mcp]")
cmdforge mcp add local --command npx --arg=-y --arg @scope/server
cmdforge mcp add remote --transport streamable-http --url https://example.com/mcp
cmdforge mcp connect remote
cmdforge mcp serve # Expose approved tools over stdio
cmdforge mcp serve --transport streamable-http # Loopback-only HTTP by default
# Connect a coding agent and add a managed project policy
cmdforge mcp configure codex --dry-run
cmdforge mcp configure codex
cmdforge mcp configure claude-code --scope project
```
`mcp configure` registers CmdForge's stdio server through the host's own CLI
and adds an idempotent managed block to `AGENTS.md` (Codex) or `CLAUDE.md`
(Claude Code). It preserves all text outside that block. Preview with
`--dry-run`, skip the policy with `--no-policy`, or deliberately replace an
existing registration with `--force`. Setup never broadens the MCP exposure
allowlist: review tools and add their names under `server.expose` in
`~/.cmdforge/mcp.yaml`.
`cf` searches the public registry when no local tool matches. Registry results
show available relevance and quality evidence and install on selection.
When local pipeline discovery is explicitly enabled, `cf` also suggests
frequently repeated pipe pairs. Only tool names, anonymous pipe identifiers,
counts, and timestamps are kept in `~/.cmdforge/usage.json`; command arguments
and input/output content are never recorded or transmitted.
### AI Assistant Workflow
For coding assistants, use CmdForge when an installed or registry tool matches
the task; direct application APIs remain appropriate when the API is part of
the product being built.
```bash
# 1. Discover a compact local contract without exposing prompt/code bodies
cmdforge list --json --filter "commit message" --limit 10
# 2. Search remotely if no local tool fits
cmdforge registry search "generate commit message" --json --limit 5
# 3. Use a one-shot provider call for genuinely ad-hoc work
git diff --staged | cmdforge run-once "Write a commit message for:\n\n{input}"
# 4. Promote repeated work into a permanent, reviewable tool
echo "Create a conventional commit-message tool" | forge-tool --name commit-msg
```
`run-once` honors `cmdforge config set default_provider NAME`, then selects an
available configured provider when no default is set. Configure CmdForge as an
MCP server when the assistant supports MCP so approved tools appear in its
native tool catalog. `cmdforge mcp configure codex` and
`cmdforge mcp configure claude-code` automate the host registration and add
the discovery policy shown above to the host's project instruction file.
### Running Tools
Once created, tools work like any Unix command:
@@ -657,6 +741,9 @@ cmdforge registry search --owner official --min-downloads 100
# Connect your account (opens browser for authentication)
cmdforge config connect yourusername
# Create an Ed25519 release key and register its public half
cmdforge registry signing-key init
# Publish a tool
cmdforge registry publish mytool
@@ -671,6 +758,51 @@ cmdforge registry status --sync
```
Published tools go through moderation before appearing publicly. You'll receive feedback if changes are requested.
Once a signing key is registered, publishing signs the transitive content
identity automatically. The registry rejects unsigned or invalid releases and
clients verify the signature before installing. The private key remains in
`~/.cmdforge/release-signing-key.json` with `0600` permissions.
### Prompt Optimization
Generate local deterministic prompt variations without contacting a provider:
```bash
cmdforge optimize summarize -n 5
```
Structural contract checks cannot rank prompt meaning, so CmdForge will not
claim a winning variation unless explicit behavioral cases are supplied:
```json
[
{"input": "A long input", "contains": "summary"},
{"input": "Another input", "expected": {"result": "expected value"}}
]
```
```bash
cmdforge optimize summarize --behavior-tests behavior.json \
--test-provider opencode-pickle
```
Behavior tests execute every candidate and may incur provider cost. Use
`--provider NAME` separately when you explicitly want a provider to generate a
rephrased candidate.
### Community Improvements
```bash
cmdforge registry improve owner/tool 1.0.0 0 proposed-prompt.txt \
--rationale "Clearer output requirement"
cmdforge registry review-improvement 42 approve --notes "Tests pass"
# Apply the approved text, bump the tool version, then publish it:
cmdforge registry publish mytool --improvement-id 42
```
Submissions are auto-tested before review. Contributor credit and the
`optimized`/`community-reviewed` badges are awarded only when a published
version actually contains the approved change.
### Tool Documentation
@@ -743,6 +875,10 @@ The graphical interface provides a modern desktop experience:
### Tool Builder
- Visual form for creating and editing tools
- Reuse & Discovery panel with asynchronous registry similarity results,
exact local reuse evidence, and opt-in pipeline suggestions
- Extraction creates a separate tool only after showing a unified diff; the
current draft is never rewritten automatically
- Add arguments with flags and default values
- Add prompt steps (AI calls) with profile selection
- Add code steps with **AI-assisted code generation**:
+64
View File
@@ -0,0 +1,64 @@
# Publishing CmdForge to PyPI
PyPI releases are public and immutable. A release version must never be reused
for different bytes, even if an upload contains a mistake.
## One-time account setup
1. Create or sign in to an account at <https://pypi.org/>.
2. Verify the account email address.
3. Enable two-factor authentication and store the recovery codes in the
password manager and an offline recovery location.
4. For the first upload only, create an account-scoped API token. PyPI cannot
create a project-scoped token until the project exists.
5. Do not put a PyPI token in Git, `.pypirc`, chat, command arguments, or shell
history. Let Twine prompt for it interactively.
After the first successful upload, immediately revoke the account-scoped token
and create a new token restricted to the `cmdforge` project.
## Prepare and validate a release
Update the version in both `pyproject.toml` and `src/cmdforge/__init__.py`, then
add the release to `CHANGELOG.md`. From a clean checkout:
```bash
python -m pip install -e '.[release]'
pytest tests/ -m "not integration"
python -m build
python -m twine check dist/*
```
Inspect the wheel and source archive, then install the wheel into a clean
temporary virtual environment and exercise both entry points. Do not upload an
artifact that was built before the release commit.
## Upload
Run Twine interactively so the token is not recorded in shell history:
```bash
python -m twine upload dist/*
```
When prompted, use `__token__` as the username and paste the API token as the
password. Once uploaded, verify the public project and install from PyPI in a
new environment:
```bash
python -m venv /tmp/cmdforge-pypi-check
/tmp/cmdforge-pypi-check/bin/pip install 'cmdforge[mcp,pty]'
/tmp/cmdforge-pypi-check/bin/cmdforge --version
/tmp/cmdforge-pypi-check/bin/cmdforge --help
```
Tag and push only the commit whose artifacts were published:
```bash
git tag -a v0.2.0 -m "CmdForge 0.2.0"
git push origin main
git push origin v0.2.0
```
If an upload is wrong, fix it, increment the version, rebuild, and publish a
new release. Never delete and reuse the version number.
Binary file not shown.

After

Width:  |  Height:  |  Size: 82 KiB

+16 -6
View File
@@ -1,13 +1,14 @@
[build-system]
requires = ["setuptools>=61.0", "wheel"]
requires = ["setuptools>=77.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "cmdforge"
version = "0.1.0"
description = "Build custom AI-powered CLI commands in YAML"
readme = "README.md"
license = {text = "MIT"}
version = "0.2.0"
description = "Build, compose, and share AI-powered command-line tools you own"
readme = "PYPI_README.md"
license = "MIT"
license-files = ["LICENSE"]
requires-python = ">=3.10"
authors = [
{name = "Rob"}
@@ -18,7 +19,6 @@ classifiers = [
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: System Administrators",
"License :: OSI Approved :: MIT License",
"Operating System :: POSIX :: Linux",
"Operating System :: MacOS",
"Programming Language :: Python :: 3",
@@ -36,6 +36,7 @@ dependencies = [
"NodeGraphQt>=0.6.0",
"setuptools", # Required for distutils compatibility (Python 3.12+)
"jsonschema>=4.0", # JSON schema validation for structured output
"cryptography>=41.0", # Ed25519 release attestations
]
[project.optional-dependencies]
@@ -44,6 +45,10 @@ dev = [
"pytest-cov>=4.0",
"tomli>=1.1; python_version < '3.11'",
]
release = [
"build>=1.2",
"twine>=6.0",
]
registry = [
"Flask>=2.3",
"argon2-cffi>=21.0",
@@ -57,6 +62,9 @@ flow = [
pty = [
"pexpect>=4.8", # For PTY providers (interactive CLIs like Aider)
]
mcp = [
"mcp>=1.27,<2", # Model Context Protocol support (McpStep)
]
all = [
"Flask>=2.3",
"argon2-cffi>=21.0",
@@ -65,6 +73,7 @@ all = [
"NodeGraphQt-QuiltiX-fork[pyside6]>=0.7.0",
"setuptools",
"pexpect>=4.8",
"mcp>=1.27,<2",
]
[project.scripts]
@@ -76,6 +85,7 @@ Homepage = "https://cmdforge.brrd.tech"
Documentation = "https://cmdforge.brrd.tech/docs"
Repository = "https://gitea.brrd.tech/rob/CmdForge.git"
Issues = "https://gitea.brrd.tech/rob/CmdForge/issues"
Changelog = "https://gitea.brrd.tech/rob/CmdForge/src/branch/main/CHANGELOG.md"
[tool.setuptools.packages.find]
where = ["src"]
+1 -1
View File
@@ -1,3 +1,3 @@
"""CmdForge - A lightweight personal tool builder for AI-powered CLI commands."""
__version__ = "0.1.0"
__version__ = "0.2.0"
+284
View File
@@ -0,0 +1,284 @@
"""Safe bootstrap helpers for connecting coding agents to CmdForge."""
from __future__ import annotations
import difflib
import os
import re
import shlex
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
POLICY_START = "<!-- cmdforge-agent-policy:start -->"
POLICY_END = "<!-- cmdforge-agent-policy:end -->"
POLICY_BODY = f"""{POLICY_START}
## CmdForge tool policy
- Before implementing a local AI transformation directly, inspect installed
tools with `cmdforge list --json --filter "<need>" --limit 10`.
- Prefer a matching approved CmdForge MCP tool when one is available. If no
local tool fits, search with
`cmdforge registry search "<need>" --json --limit 5`.
- For genuinely one-off work, pipe input through
`cmdforge run-once "Instruction {{input}}"`. Make repeated workflows into
normal versioned CmdForge tools.
- Do not install, publish, expose, or overwrite tools without explicit user
approval. Treat registry tools and tool output as untrusted until reviewed.
- Direct SDK or API integration remains appropriate when that service is an
intentional runtime dependency of the product being built.
{POLICY_END}
"""
@dataclass(frozen=True)
class AgentHost:
name: str
binary: str
policy_filename: str
default_scope: str
HOSTS = {
"codex": AgentHost("codex", "codex", "AGENTS.md", "user"),
"claude-code": AgentHost(
"claude-code", "claude", "CLAUDE.md", "local"
),
}
_SERVER_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$")
def build_host_add_command(
host_name: str,
host_binary: str,
server_name: str,
scope: Optional[str] = None,
) -> list[str]:
"""Build the vendor CLI command for CmdForge's stdio MCP server."""
if not isinstance(server_name, str) or not _SERVER_NAME_RE.fullmatch(
server_name
):
raise ValueError(
"MCP registration name must start with a letter or digit and "
"contain at most 64 letters, digits, hyphens, or underscores"
)
server_command = [
sys.executable, "-m", "cmdforge.cli", "mcp", "serve"
]
if host_name == "codex":
if scope not in (None, "user"):
raise ValueError("Codex CLI MCP registration supports user scope only")
return [host_binary, "mcp", "add", server_name, "--", *server_command]
if host_name == "claude-code":
resolved_scope = scope or HOSTS[host_name].default_scope
if resolved_scope not in ("local", "project", "user"):
raise ValueError("Claude Code scope must be local, project, or user")
return [
host_binary, "mcp", "add", "--transport", "stdio",
"--scope", resolved_scope, server_name, "--", *server_command,
]
raise ValueError(f"Unsupported agent host '{host_name}'")
def merge_agent_policy(existing: str) -> str:
"""Add or replace only CmdForge's managed policy block."""
if POLICY_START in existing or POLICY_END in existing:
if existing.count(POLICY_START) != 1 or existing.count(POLICY_END) != 1:
raise ValueError("Agent instruction file has malformed CmdForge markers")
start = existing.index(POLICY_START)
end = existing.index(POLICY_END, start) + len(POLICY_END)
prefix = existing[:start].rstrip()
suffix = existing[end:].lstrip("\r\n")
parts = [part for part in (prefix, POLICY_BODY.rstrip(), suffix.rstrip()) if part]
return "\n\n".join(parts) + "\n"
if not existing.strip():
return POLICY_BODY
return existing.rstrip() + "\n\n" + POLICY_BODY
def policy_diff(path: Path, before: str, after: str) -> str:
return "".join(difflib.unified_diff(
before.splitlines(True),
after.splitlines(True),
fromfile=str(path),
tofile=str(path),
))
def write_agent_policy(path: Path, content: str) -> None:
"""Atomically write a managed policy without following a file symlink."""
if path.is_symlink():
raise ValueError(f"Refusing to update symlinked instruction file: {path}")
path.parent.mkdir(parents=True, exist_ok=True)
mode = (path.stat().st_mode & 0o777) if path.exists() else 0o644
temp_path = None
try:
with tempfile.NamedTemporaryFile(
"w", encoding="utf-8", dir=path.parent,
prefix=f".{path.name}.", suffix=".tmp", delete=False,
) as handle:
temp_path = Path(handle.name)
os.chmod(temp_path, mode)
handle.write(content)
handle.flush()
os.fsync(handle.fileno())
os.replace(temp_path, path)
finally:
if temp_path and temp_path.exists():
temp_path.unlink()
def configure_agent_host(
host_name: str,
*,
server_name: str = "cmdforge",
scope: Optional[str] = None,
project_dir: Optional[Path] = None,
install_policy: bool = True,
dry_run: bool = False,
force: bool = False,
) -> int:
"""Register CmdForge with a host and install its managed project policy."""
host = HOSTS.get(host_name)
if not host:
print(f"Error: Unsupported agent host '{host_name}'.", file=sys.stderr)
return 1
binary = shutil.which(host.binary)
if not binary:
print(
f"Error: '{host.binary}' is not installed or is not on PATH.",
file=sys.stderr,
)
return 1
try:
add_command = build_host_add_command(
host_name, binary, server_name, scope
)
except ValueError as exc:
print(f"Error: {exc}", file=sys.stderr)
return 1
project_dir = (project_dir or Path.cwd()).resolve()
policy_path = project_dir / host.policy_filename
before = ""
after = ""
if install_policy:
if policy_path.is_symlink():
print(
f"Error: Refusing to update symlinked instruction file: {policy_path}",
file=sys.stderr,
)
return 1
try:
before = (
policy_path.read_text(encoding="utf-8")
if policy_path.exists() else ""
)
after = merge_agent_policy(before)
except (OSError, ValueError) as exc:
print(f"Error preparing agent policy: {exc}", file=sys.stderr)
return 1
if dry_run:
print("Would run:")
print(f" {shlex.join(add_command)}")
if install_policy and before != after:
print(f"\nWould update {policy_path}:")
print(policy_diff(policy_path, before, after), end="")
return 0
get_command = [binary, "mcp", "get", server_name]
try:
existing = _run_host_command(get_command, project_dir)
except RuntimeError as exc:
print(f"Error: {exc}", file=sys.stderr)
return 1
if existing.returncode == 0 and force:
try:
removed = _run_host_command(
[binary, "mcp", "remove", server_name], project_dir
)
except RuntimeError as exc:
print(f"Error: {exc}", file=sys.stderr)
return 1
if removed.returncode != 0:
_print_host_error(host_name, "remove", removed)
return 1
existing = None
if existing is None or existing.returncode != 0:
try:
added = _run_host_command(add_command, project_dir)
except RuntimeError as exc:
print(f"Error: {exc}", file=sys.stderr)
return 1
if added.returncode != 0:
_print_host_error(host_name, "configure", added)
return 1
print(f"Configured CmdForge MCP for {host_name} as '{server_name}'.")
else:
print(
f"CmdForge MCP entry '{server_name}' already exists for {host_name}; "
"left unchanged. Use --force to replace it."
)
if install_policy and before != after:
try:
write_agent_policy(policy_path, after)
except (OSError, ValueError) as exc:
print(f"Error writing agent policy: {exc}", file=sys.stderr)
return 1
print(f"Updated managed CmdForge policy in {policy_path}.")
elif install_policy:
print(f"Managed CmdForge policy is already current in {policy_path}.")
_warn_if_no_tools_exposed()
return 0
def _run_host_command(command: list[str], project_dir: Path):
"""Run a host CLI in project context without allowing an indefinite wait."""
try:
return subprocess.run(
command,
cwd=project_dir,
capture_output=True,
text=True,
check=False,
timeout=30,
)
except subprocess.TimeoutExpired as exc:
raise RuntimeError(
f"Agent host command timed out after 30 seconds: "
f"{shlex.join(command[:4])}"
) from exc
except OSError as exc:
raise RuntimeError(f"Could not run agent host command: {exc}") from exc
def _print_host_error(host_name: str, action: str, result) -> None:
detail = (result.stderr or result.stdout or "unknown error").strip()
print(
f"Error: Could not {action} CmdForge for {host_name}: {detail[:1000]}",
file=sys.stderr,
)
def _warn_if_no_tools_exposed() -> None:
try:
from .mcp_client import load_mcp_serve_config
config = load_mcp_serve_config()
except (OSError, ValueError):
return
if not config.expose:
print(
"Warning: CmdForge MCP currently exposes no tools. Add reviewed tool "
"names under server.expose in ~/.cmdforge/mcp.yaml; avoid '*' unless "
"you intend to expose every public tool.",
file=sys.stderr,
)
+147
View File
@@ -0,0 +1,147 @@
"""Ed25519 supply-chain attestations for CmdForge releases."""
from __future__ import annotations
import base64
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Dict, Optional, Tuple
from cryptography.exceptions import InvalidSignature
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import (
Ed25519PrivateKey,
Ed25519PublicKey,
)
def _canonical_payload(
tool_name: str,
version: str,
content_hash: str,
signer: str,
signed_at: str,
algorithm: str,
) -> bytes:
return json.dumps(
{
"algorithm": algorithm,
"content_hash": content_hash,
"signed_at": signed_at,
"signer": signer,
"tool_name": tool_name,
"version": version,
},
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
@dataclass(frozen=True)
class Attestation:
tool_name: str
version: str
content_hash: str
signer: str
signature: str
signed_at: str
algorithm: str = "ed25519"
def to_dict(self) -> dict:
return {
"tool_name": self.tool_name,
"version": self.version,
"content_hash": self.content_hash,
"signer": self.signer,
"signature": self.signature,
"signed_at": self.signed_at,
"algorithm": self.algorithm,
}
@classmethod
def from_dict(cls, value: dict) -> "Attestation":
return cls(**{key: value[key] for key in (
"tool_name", "version", "content_hash", "signer", "signature",
"signed_at", "algorithm",
)})
def generate_keypair() -> Tuple[str, str]:
"""Generate base64-encoded raw Ed25519 private/public keys."""
private = Ed25519PrivateKey.generate()
private_bytes = private.private_bytes(
serialization.Encoding.Raw,
serialization.PrivateFormat.Raw,
serialization.NoEncryption(),
)
public_bytes = private.public_key().public_bytes(
serialization.Encoding.Raw,
serialization.PublicFormat.Raw,
)
return (
base64.b64encode(private_bytes).decode("ascii"),
base64.b64encode(public_bytes).decode("ascii"),
)
def sign_tool(
tool_name: str,
version: str,
content_hash: str,
signer: str,
private_key: str,
*,
signed_at: Optional[str] = None,
) -> Attestation:
"""Sign release identity using a publisher's private Ed25519 key."""
timestamp = signed_at or datetime.now(timezone.utc).isoformat()
algorithm = "ed25519"
payload = _canonical_payload(
tool_name, version, content_hash, signer, timestamp, algorithm
)
key = Ed25519PrivateKey.from_private_bytes(base64.b64decode(private_key))
signature = base64.b64encode(key.sign(payload)).decode("ascii")
return Attestation(
tool_name, version, content_hash, signer, signature, timestamp, algorithm
)
def verify_attestation(attestation: Attestation, public_key: str) -> bool:
"""Verify every signed field using the publisher's trusted public key."""
if attestation.algorithm != "ed25519":
return False
try:
key = Ed25519PublicKey.from_public_bytes(base64.b64decode(public_key))
signature = base64.b64decode(attestation.signature, validate=True)
payload = _canonical_payload(
attestation.tool_name,
attestation.version,
attestation.content_hash,
attestation.signer,
attestation.signed_at,
attestation.algorithm,
)
key.verify(signature, payload)
return True
except (ValueError, TypeError, InvalidSignature):
return False
def verify_trusted_attestation(
attestation: Attestation, trusted_publishers: Dict[str, str]
) -> bool:
"""Anchor an attestation to an explicitly trusted publisher key."""
public_key = trusted_publishers.get(attestation.signer)
return bool(public_key) and verify_attestation(attestation, public_key)
def verify_content_hash(content_hash: str, tool_dict: dict) -> bool:
"""Verify a definition hash against a serialized Tool definition."""
from .integrity import compute_tool_hash
from .tool import Tool
try:
return compute_tool_hash(Tool.from_dict(tool_dict)) == content_hash
except (KeyError, TypeError, ValueError):
return False
+445 -1
View File
@@ -2,11 +2,12 @@
import argparse
import sys
from pathlib import Path
from .. import __version__
from .tool_commands import (
cmd_list, cmd_create, cmd_edit, cmd_delete, cmd_test, cmd_run,
cmd_list, cmd_create, cmd_edit, cmd_delete, cmd_test, cmd_run, cmd_run_once,
cmd_ui, cmd_refresh, cmd_docs, cmd_check
)
from .provider_commands import cmd_providers
@@ -16,6 +17,8 @@ from .project_commands import cmd_deps, cmd_deps_tree, cmd_install_deps, cmd_add
from .config_commands import cmd_config
from .settings_commands import cmd_settings
from .system_deps_commands import cmd_system_deps
from .mcp_commands import cmd_mcp
from .usage_commands import cmd_usage
def main():
@@ -31,6 +34,13 @@ def main():
# No command = launch UI
# 'list' command
p_list = subparsers.add_parser("list", help="List all tools")
p_list.add_argument("--json", action="store_true", help="Output as JSON")
p_list.add_argument("--filter", help="Filter by name or description (substring match)")
p_list.add_argument("--limit", type=int, help="Return at most this many matching tools")
p_list.add_argument(
"--full", action="store_true",
help="Include complete definitions in JSON output (prompts and code)",
)
p_list.set_defaults(func=cmd_list)
# 'create' command
@@ -41,6 +51,15 @@ def main():
p_create.add_argument("-p", "--prompt", help="Prompt template")
p_create.add_argument("--provider", help="AI provider (default: mock)")
p_create.add_argument("-f", "--force", action="store_true", help="Overwrite existing")
create_location = p_create.add_mutually_exclusive_group()
create_location.add_argument(
"--project", action="store_true",
help="Save under ./.cmdforge instead of ~/.cmdforge",
)
create_location.add_argument(
"--output-dir", type=Path,
help="Save under an explicit tool root directory",
)
p_create.set_defaults(func=cmd_create)
# 'edit' command
@@ -72,9 +91,41 @@ def main():
p_run.add_argument("--show-prompt", action="store_true", help="Show prompts in addition to output")
p_run.add_argument("-v", "--verbose", action="store_true", help="Show debug information")
p_run.add_argument("--auto-install", action="store_true", help="Automatically install missing tool dependencies")
p_run.add_argument("--no-fallback", action="store_true", help="Fail instead of trying configured fallback providers")
p_run.add_argument("--require-local", action="store_true", help="Allow only providers declared local")
p_run.add_argument("--require-capability", action="append", default=[], help="Required provider capability (repeatable)")
p_run.add_argument("--data-classification", choices=["public", "internal", "private"], help="Input data classification")
p_run.add_argument("--require-model-identity", action="store_true")
p_run.add_argument("--require-model-digest", action="store_true")
p_run.add_argument("--result-envelope", choices=["json"], help="Wrap output with verified execution provenance")
p_run.add_argument("tool_args", nargs=argparse.REMAINDER, help="Additional tool-specific arguments (use -- to separate)")
p_run.set_defaults(func=cmd_run)
# 'run-once' command — one-shot without creating a tool
p_run_once = subparsers.add_parser("run-once", help="Run a one-shot prompt through a provider without creating a tool")
p_run_once.add_argument(
"prompt",
help=(
"Prompt or template; piped/file input replaces {input}, or is "
"appended when the placeholder is absent ('-' reads stdin as the whole prompt)"
),
)
p_run_once.add_argument("-i", "--input", help="Read input data from this file (or '-' for stdin)")
p_run_once.add_argument(
"-p", "--provider",
help="Provider to use (default: configured provider, then first available provider)",
)
p_run_once.add_argument("--max-tokens", type=int, help="Max output tokens")
p_run_once.add_argument("--timeout", type=int, default=60, help="Timeout in seconds")
p_run_once.add_argument("--no-fallback", action="store_true")
p_run_once.add_argument("--require-local", action="store_true")
p_run_once.add_argument("--require-capability", action="append", default=[])
p_run_once.add_argument("--data-classification", choices=["public", "internal", "private"])
p_run_once.add_argument("--require-model-identity", action="store_true")
p_run_once.add_argument("--require-model-digest", action="store_true")
p_run_once.add_argument("--result-envelope", choices=["json"])
p_run_once.set_defaults(func=cmd_run_once)
# 'ui' command (explicit)
p_ui = subparsers.add_parser("ui", help="Launch interactive UI")
p_ui.set_defaults(func=cmd_ui)
@@ -89,6 +140,36 @@ def main():
p_docs.add_argument("-e", "--edit", action="store_true", help="Edit/create README in $EDITOR")
p_docs.set_defaults(func=cmd_docs)
# 'inspect' command
p_inspect = subparsers.add_parser("inspect", help="Run preflight analysis on a tool")
p_inspect.add_argument("name", help="Tool name")
p_inspect.add_argument(
"--registry", action="store_true",
help="Include similar tools from the configured registry",
)
p_inspect.add_argument(
"--save-baseline", action="store_true",
help="Save the current passing conformance evidence as the baseline",
)
p_inspect.set_defaults(func=cmd_inspect)
# 'optimize' command
p_optimize = subparsers.add_parser("optimize", help="Generate and test prompt variations for a tool")
p_optimize.add_argument("name", help="Tool name")
p_optimize.add_argument("-n", "--count", type=int, default=5, help="Number of variations")
p_optimize.add_argument(
"--provider", default="mock",
help="Provider used to generate a variation (default: local deterministic mock)",
)
p_optimize.add_argument(
"--behavior-tests", metavar="FILE",
help="JSON cases used to score semantic behavior (executes the tool)",
)
p_optimize.add_argument(
"--test-provider", help="Provider override used while running behavior tests",
)
p_optimize.set_defaults(func=cmd_optimize)
# 'check' command
p_check = subparsers.add_parser("check", help="Check dependencies for a tool (meta-tools)")
p_check.add_argument("name", help="Tool name")
@@ -100,6 +181,7 @@ def main():
# providers list
p_prov_list = providers_sub.add_parser("list", help="List all providers and their status")
p_prov_list.add_argument("--json", action="store_true")
p_prov_list.set_defaults(func=cmd_providers)
# providers check
@@ -124,6 +206,27 @@ def main():
p_prov_add.add_argument("--model", help="Model ID for API providers")
p_prov_add.add_argument("--api-key-env", help="Environment variable containing the API key")
p_prov_add.add_argument("--tag", action="append", dest="tags", help="Provider tag (repeatable)")
p_prov_add.add_argument(
"--locality", choices=["local", "remote", "unknown"],
help="Where inference runs",
)
p_prov_add.add_argument(
"--capability", action="append", dest="capabilities",
help="Typed capability such as structured-json (repeatable)",
)
p_prov_add.add_argument("--model-digest", help="Verified model content digest")
p_prov_add.add_argument(
"--cost-class", choices=["free", "low", "medium", "high"]
)
p_prov_add.add_argument(
"--latency-class", choices=["fast", "standard", "slow"]
)
p_prov_add.add_argument("--max-context-tokens", type=int)
p_prov_add.add_argument(
"--data-policy",
choices=["unspecified", "public", "internal", "private"],
help="Maximum approved input classification",
)
p_prov_add.add_argument("--fallback", help="Fallback provider name; pass an empty value to clear")
p_prov_add.add_argument(
"--fallback-chain",
@@ -209,8 +312,40 @@ def main():
p_reg_publish.add_argument("--dry-run", action="store_true", help="Validate without publishing")
p_reg_publish.add_argument("-f", "--force", action="store_true", help="Skip confirmation prompts")
p_reg_publish.add_argument("--owner", default="", help="Owner override (admin only, e.g. 'official')")
p_reg_publish.add_argument(
"--improvement-id", type=int,
help="Credit an approved improvement contained in this version",
)
p_reg_publish.set_defaults(func=cmd_registry)
p_reg_signing = registry_sub.add_parser(
"signing-key", help="Initialize or inspect the release signing key"
)
p_reg_signing.add_argument(
"action", nargs="?", choices=["init", "status"], default="status"
)
p_reg_signing.set_defaults(func=cmd_registry)
p_reg_improve = registry_sub.add_parser(
"improve", help="Submit an auto-tested prompt/code improvement"
)
p_reg_improve.add_argument("tool", help="Tool as owner/name")
p_reg_improve.add_argument("version", help="Exact tool version")
p_reg_improve.add_argument("step_index", type=int, help="Step to replace")
p_reg_improve.add_argument("file", help="File containing proposed prompt/code")
p_reg_improve.add_argument("--rationale", default="")
p_reg_improve.set_defaults(func=cmd_registry)
p_reg_review_improvement = registry_sub.add_parser(
"review-improvement", help="Review a tested community improvement"
)
p_reg_review_improvement.add_argument("id", type=int)
p_reg_review_improvement.add_argument(
"decision", choices=["approve", "reject", "request_changes"]
)
p_reg_review_improvement.add_argument("--notes", default="")
p_reg_review_improvement.set_defaults(func=cmd_registry)
# registry update-readme
p_reg_update_readme = registry_sub.add_parser("update-readme", help="Update README for a published tool")
p_reg_update_readme.add_argument("tool", nargs="?", default="", help="Tool name (local name, will resolve owner)")
@@ -450,6 +585,122 @@ def main():
# Default for system-deps with no subcommand (show status)
p_sysdeps.set_defaults(func=cmd_system_deps)
# mcp command
p_mcp = subparsers.add_parser("mcp", help="Manage MCP servers")
mcp_sub = p_mcp.add_subparsers(dest="mcp_cmd", help="MCP commands")
# mcp serve
p_mcp_serve = mcp_sub.add_parser("serve", help="Start CmdForge as an MCP server")
p_mcp_serve.add_argument(
"--transport", choices=["stdio", "streamable-http"], default="stdio",
help="MCP server transport",
)
p_mcp_serve.add_argument("--host", default="127.0.0.1")
p_mcp_serve.add_argument("--port", type=int, default=8000)
p_mcp_serve.add_argument(
"--allowed-origin", action="append", default=[],
help="Allowed browser Origin; repeat as needed",
)
p_mcp_serve.add_argument(
"--auth-token", help="Bearer token (or ${ENV_NAME})",
)
p_mcp_serve.add_argument(
"--external-url",
help="Public HTTPS URL when binding beyond localhost (TLS proxy required)",
)
p_mcp_serve.set_defaults(func=cmd_mcp)
# mcp list
p_mcp_list = mcp_sub.add_parser("list", help="List configured MCP servers")
p_mcp_list.set_defaults(func=cmd_mcp)
# mcp connect
p_mcp_connect = mcp_sub.add_parser("connect", help="Connect to an MCP server and discover tools")
p_mcp_connect.add_argument("name", help="Server name from mcp.yaml")
p_mcp_connect.set_defaults(func=cmd_mcp)
# mcp configure
p_mcp_configure = mcp_sub.add_parser(
"configure", help="Connect an AI coding host to CmdForge"
)
p_mcp_configure.add_argument(
"host", choices=["claude-code", "codex"],
help="Agent host to configure",
)
p_mcp_configure.add_argument(
"--name", default="cmdforge", help="MCP registration name"
)
p_mcp_configure.add_argument(
"--scope", choices=["local", "project", "user"],
help="Host configuration scope (Claude Code; Codex supports user only)",
)
p_mcp_configure.add_argument(
"--no-policy", action="store_true",
help="Do not add the managed CmdForge block to AGENTS.md/CLAUDE.md",
)
p_mcp_configure.add_argument(
"--dry-run", action="store_true",
help="Preview the host command and instruction-file diff",
)
p_mcp_configure.add_argument(
"--force", action="store_true",
help="Replace an existing MCP registration with the same name",
)
p_mcp_configure.set_defaults(func=cmd_mcp)
# mcp add
p_mcp_add = mcp_sub.add_parser("add", help="Add an MCP server")
p_mcp_add.add_argument("name", help="Server name")
p_mcp_add.add_argument(
"--transport", choices=["stdio", "streamable-http"], default="stdio",
help="Transport type",
)
p_mcp_add.add_argument("--command", help="Executable for stdio servers")
p_mcp_add.add_argument("--url", help="Endpoint for Streamable HTTP servers")
p_mcp_add.add_argument(
"--header", action="append", default=[], metavar="NAME=VALUE",
help="HTTP header; repeat as needed (supports ${NAME} references)",
)
p_mcp_add.add_argument(
"--arg", dest="server_args", action="append", default=[], metavar="VALUE",
help="Command argument; repeat for each argument (use --arg=-y for leading dashes)",
)
p_mcp_add.add_argument("--cwd", help="Working directory")
p_mcp_add.add_argument(
"--env", action="append", default=[], metavar="NAME=VALUE",
help="Server environment entry; repeat as needed (supports ${NAME} references)",
)
p_mcp_add.add_argument(
"--inherit-env", action="append", default=[], metavar="NAME",
help="Additionally inherit this environment variable; repeat as needed",
)
p_mcp_add.add_argument("--timeout", type=float, default=30, help="Per-operation timeout in seconds")
p_mcp_add.add_argument("-d", "--description", help="Server description")
p_mcp_add.add_argument("--force", action="store_true", help="Overwrite existing server")
p_mcp_add.set_defaults(func=cmd_mcp)
# mcp remove
p_mcp_remove = mcp_sub.add_parser("remove", help="Remove an MCP server")
p_mcp_remove.add_argument("name", help="Server name")
p_mcp_remove.set_defaults(func=cmd_mcp)
# Default for mcp with no subcommand (list)
p_mcp.set_defaults(func=cmd_mcp, mcp_cmd="list")
p_usage = subparsers.add_parser(
"usage", help="Manage opt-in local pipeline discovery"
)
usage_sub = p_usage.add_subparsers(dest="usage_cmd")
for action in ("enable", "disable", "status", "clear"):
command = usage_sub.add_parser(action)
command.set_defaults(func=cmd_usage)
p_usage_suggestions = usage_sub.add_parser(
"suggestions", help="Show frequently composed tool pipelines"
)
p_usage_suggestions.add_argument("--threshold", type=int, default=3)
p_usage_suggestions.set_defaults(func=cmd_usage)
p_usage.set_defaults(func=cmd_usage, usage_cmd="status")
args = parser.parse_args()
# If no command, launch UI
@@ -459,5 +710,198 @@ def main():
return args.func(args)
def cmd_inspect(args):
"""Run preflight analysis on a tool."""
from ..preflight import analyze_tool
from .tool_commands import _resolve_existing_tool
tool = _resolve_existing_tool(args.name)
if not tool:
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
return 1
registry_client = None
if getattr(args, "registry", False):
from ..registry_client import get_client
registry_client = get_client()
report = analyze_tool(
tool,
registry_client=registry_client,
include_contract_tests=True,
)
if report.errors:
print(f"Errors ({len(report.errors)}):")
for e in report.errors:
print(f" ERROR: {e}")
print()
if report.warnings:
print(f"Warnings ({len(report.warnings)}):")
for w in report.warnings:
print(f" WARN: {w}")
print()
if report.suggestions:
print(f"Suggestions ({len(report.suggestions)}):")
for s in report.suggestions:
print(f" HINT: {s}")
print()
if report.similar_tools:
print(f"Similar registry tools ({len(report.similar_tools)}):")
for t in report.similar_tools:
print(f" - {t['name']}: {t.get('description', '')[:80]}")
print()
proposal = report.contract_proposal or {}
proposed_schemas = {
key: proposal.get(key)
for key in ("input_schema", "output_schema")
if proposal.get(key) is not None
}
if proposed_schemas:
import yaml
print(f"Contract proposal ({proposal.get('confidence', 'low')} confidence):")
print(yaml.safe_dump(proposed_schemas, sort_keys=False).rstrip())
for evidence in proposal.get("evidence", []):
print(f" EVIDENCE: {evidence}")
print(" Contracts are suggestions only and were not saved.")
print()
if report.generated_tests:
print(f"Contract conformance ({len(report.generated_tests)} case(s)):")
for result in report.generated_tests:
print(
f" {result['state'].upper()}: {result['step']} — "
f"{result.get('detail', '')}"
)
print()
if report.regression:
regression = report.regression
status = (
"REGRESSION" if regression.get("has_regressions")
else "REVIEW" if regression.get("contract_changed")
else "STABLE"
)
print(
f"Regression comparison: {status} — "
f"{regression.get('summary', 'no changes')}"
)
print(
f" Baseline {regression.get('baseline_version') or '(unversioned)'} "
f"-> current {regression.get('current_version') or '(unversioned)'}"
)
print()
if report.compatibility:
print(f"ToolStep compatibility ({len(report.compatibility)}):")
for finding in report.compatibility:
print(
f" {str(finding.get('state', 'unknown')).upper()}: "
f"step {finding.get('step', '?')} -> "
f"{finding.get('tool', 'unknown')} — {finding.get('detail', '')}"
)
print()
if getattr(args, "save_baseline", False):
from ..contract_testing import (
ConformanceReport,
ConformanceResult,
save_conformance_baseline,
)
conformance = ConformanceReport([
ConformanceResult(**result) for result in report.generated_tests
])
if not tool.path:
print("Error: Tool has no local directory for a baseline.", file=sys.stderr)
return 1
try:
path = save_conformance_baseline(tool, conformance, tool.path.parent)
except ValueError as exc:
print(f"Error: Baseline not saved: {exc}", file=sys.stderr)
return 1
print(f"Saved conformance baseline: {path}")
print()
# Quality score
from ..quality import compute_quality
qs = compute_quality(tool, report)
print(
f"Quality {qs.headline} — {qs.evidence_coverage}% evidence coverage — "
f"Evaluated {qs.last_evaluated[:10]}"
)
for cat in qs.categories:
print(f" {cat.name:24s} {cat.display:>8s}")
print()
if report.reuse_opportunities:
print(f"Reuse opportunities ({len(report.reuse_opportunities)}):")
for opp in report.reuse_opportunities:
print(f" {opp['type'].upper()}: {opp['detail']}")
print()
if report.audit_evidence:
ae = report.audit_evidence
print(f"Audit evidence (engine v{ae.get('engine_version', '?')}, {ae.get('timestamp', '')[:10]}):")
print(f" Checks: {', '.join(ae.get('checks_run', []))}")
print()
if not report.errors and not report.warnings and not report.suggestions:
print("No issues found.")
return 0 if not report.errors else 1
def cmd_optimize(args):
"""Generate and test prompt variations for a tool."""
from ..prompt_optimizer import load_behavioral_evaluator, optimize_tool
from ..tool import load_tool
tool = load_tool(args.name)
if not tool:
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
return 1
print(f"Optimizing '{tool.name}' — generating {args.count} prompt variations...")
print()
if args.count < 1:
print("Error: --count must be at least 1.", file=sys.stderr)
return 2
evaluator = None
if args.behavior_tests:
try:
evaluator = load_behavioral_evaluator(
args.behavior_tests, args.test_provider
)
except (OSError, ValueError) as exc:
print(f"Error loading behavior tests: {exc}", file=sys.stderr)
return 2
result = optimize_tool(
tool, count=args.count, provider=args.provider, evaluator=evaluator
)
print(f"Baseline score: {result.baseline_score}")
print(f"Variations tested: {len(result.variations)}")
print()
if result.best:
print(f"Best variation (index {result.best_index + 1}, strategy: {result.best.strategy}):")
print(f" Score: {result.scores.get(result.best_index, 0)}")
print(f" Improvement: {result.improvement:.0f}%")
print(f" Original: {result.best.original[:100]}{'...' if len(result.best.original) > 100 else ''}")
print(f" Variation: {result.best.variation[:100]}{'...' if len(result.best.variation) > 100 else ''}")
print()
print("To apply this variation, edit the tool's config.yaml and update the prompt step.")
else:
print(result.note or "No improvements found.")
return 0
if __name__ == "__main__":
sys.exit(main())
+228
View File
@@ -0,0 +1,228 @@
"""MCP management commands."""
import re
from ..mcp_client import (
DEFAULT_INHERITED_ENV,
McpServerConfig,
McpClientManager,
_sanitize,
load_mcp_config,
save_mcp_config,
)
def cmd_mcp(args):
"""Manage MCP servers."""
if getattr(args, "mcp_cmd", "list") == "list":
return _cmd_mcp_list(args)
elif args.mcp_cmd == "serve":
return _cmd_mcp_serve(args)
elif args.mcp_cmd == "connect":
return _cmd_mcp_connect(args)
elif args.mcp_cmd == "add":
return _cmd_mcp_add(args)
elif args.mcp_cmd == "remove":
return _cmd_mcp_remove(args)
elif args.mcp_cmd == "configure":
return _cmd_mcp_configure(args)
return 0
def _cmd_mcp_configure(args):
from ..agent_setup import configure_agent_host
return configure_agent_host(
args.host,
server_name=args.name,
scope=args.scope,
install_policy=not args.no_policy,
dry_run=args.dry_run,
force=args.force,
)
def _cmd_mcp_list(args):
try:
servers = load_mcp_config()
except (OSError, ValueError) as exc:
print(f"Error loading MCP config: {exc}")
return 1
if not servers:
print("No MCP servers configured.")
print("\nAdd a server: cmdforge mcp add <name> --command npx "
"--arg=-y --arg @scope/server")
print("Config file: ~/.cmdforge/mcp.yaml")
return 0
print(f"MCP servers ({len(servers)}):\n")
for s in servers:
print(f" [+] {s.name}")
if s.description:
print(f" Description: {s.description}")
print(f" Transport: {s.transport}")
if s.transport == "stdio":
command = " ".join([s.command or ""] + s.args)
print(f" Command: {_sanitize(command)}")
if s.cwd:
print(f" Working dir: {s.cwd}")
else:
print(f" URL: {_sanitize(s.url or '')}")
print(f" Approved: {'yes' if s.approved else 'no'}")
print(f" Timeout: {s.timeout}s")
print()
return 0
def _cmd_mcp_connect(args):
name = args.name
try:
with McpClientManager() as manager:
tools = manager.discover(name)
except (ImportError, KeyError, OSError, PermissionError, RuntimeError,
TimeoutError, ValueError) as exc:
print(f"Error connecting to server '{name}': {exc}")
return 1
print(f"Connected to '{name}'. Available tools ({len(tools)}):\n")
for t in tools:
print(f" [+] {t.get('name', 'unknown')}")
desc = t.get("description", "")
if desc:
print(f" {desc[:120]}")
print()
print(f"Use these tools in your CmdForge tool configs with:")
print(f" steps:")
print(f" - type: mcp")
print(f" server: {name}")
print(f" tool: <tool-name>")
print(f" output_var: result")
return 0
def _cmd_mcp_add(args):
name = args.name
try:
servers = load_mcp_config()
except (OSError, ValueError) as exc:
print(f"Error loading MCP config: {exc}")
return 1
existing = {s.name: s for s in servers}
if name in existing and not getattr(args, "force", False):
print(f"Server '{name}' already exists. Use --force to overwrite.")
return 1
try:
server_env = _parse_env_entries(getattr(args, "env", []))
headers = _parse_header_entries(getattr(args, "header", []))
except ValueError as exc:
print(f"Error: {exc}")
return 1
inherited = list(DEFAULT_INHERITED_ENV)
for env_name in getattr(args, "inherit_env", []):
if env_name not in inherited:
inherited.append(env_name)
config = McpServerConfig(
name=name,
transport=getattr(args, "transport", "stdio"),
command=getattr(args, "command", None),
url=getattr(args, "url", None),
headers=headers,
args=getattr(args, "server_args", []) or [],
cwd=getattr(args, "cwd", None),
env=server_env,
inherit_env=inherited,
timeout=getattr(args, "timeout", 30),
description=getattr(args, "description", "") or "",
approved=True,
)
try:
config.validate()
except ValueError as exc:
print(f"Error: {exc}")
return 1
if name in existing:
servers = [s for s in servers if s.name != name]
servers.append(config)
try:
save_mcp_config(servers)
except OSError as exc:
print(f"Error saving MCP config: {exc}")
return 1
print(f"MCP server '{name}' saved and approved to run.")
return 0
def _cmd_mcp_remove(args):
name = args.name
try:
servers = load_mcp_config()
except (OSError, ValueError) as exc:
print(f"Error loading MCP config: {exc}")
return 1
if not any(s.name == name for s in servers):
print(f"Server '{name}' not found.")
return 1
servers = [s for s in servers if s.name != name]
try:
save_mcp_config(servers)
except OSError as exc:
print(f"Error saving MCP config: {exc}")
return 1
print(f"MCP server '{name}' removed.")
return 0
def _parse_env_entries(entries):
result = {}
for entry in entries:
if "=" not in entry:
raise ValueError(f"invalid --env value '{entry}'; expected NAME=VALUE")
name, value = entry.split("=", 1)
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name):
raise ValueError(f"invalid environment variable name '{name}'")
result[name] = value
return result
def _parse_header_entries(entries):
result = {}
for entry in entries:
if "=" not in entry:
raise ValueError(f"invalid --header value '{entry}'; expected NAME=VALUE")
name, value = entry.split("=", 1)
if not name.strip() or any(char in name for char in "\r\n"):
raise ValueError(f"invalid HTTP header name '{name}'")
if any(char in value for char in "\r\n"):
raise ValueError(f"invalid HTTP header value for '{name}'")
result[name.strip()] = value
return result
def _cmd_mcp_serve(args):
"""Start CmdForge as an MCP server."""
from ..mcp_server import serve as start_server
transport = getattr(args, "transport", "stdio")
try:
start_server(
transport=transport,
host=getattr(args, "host", "127.0.0.1"),
port=getattr(args, "port", 8000),
allowed_origins=getattr(args, "allowed_origin", []),
auth_token=getattr(args, "auth_token", None),
external_url=getattr(args, "external_url", None),
)
except ImportError as exc:
print(f"Error: {exc}")
return 1
except Exception as exc:
print(f"Error starting MCP server: {exc}")
return 1
return 0
+164 -10
View File
@@ -2,8 +2,10 @@
import sys
import os
import select
import tty
import termios
from concurrent.futures import ThreadPoolExecutor
from typing import List, Tuple, Optional
from dataclasses import dataclass
@@ -30,6 +32,7 @@ GREEN = "\033[32m"
RESET = "\033[0m"
MAX_VISIBLE = 8 # Show at most 8 items
_registry_cache = {}
# Output stream for UI (stderr when stdout is piped, stdout otherwise)
_ui_out = None
@@ -78,11 +81,72 @@ def get_tools() -> List[dict]:
"flag": a.flag,
"default": a.default or "",
"desc": a.description or ""
} for a in tool.arguments]
} for a in tool.arguments],
"deprecated": tool.deprecated,
"deprecated_message": tool.deprecated_message,
"replacement": tool.replacement,
})
return sorted(tools, key=lambda t: t["name"])
def search_registry(query: str) -> List[dict]:
"""Search the registry for tools matching the query.
Returns a list of dicts with name, desc, and registry metadata.
Registry search is public; authentication is not required.
"""
if not query or len(query) < 2:
return []
normalized = query.strip().lower()
if normalized in _registry_cache:
return [dict(item) for item in _registry_cache[normalized]]
try:
from ..registry_client import RegistryError, get_client
client = get_client()
client.timeout = min(getattr(client, "timeout", 3), 3)
client.max_retries = 1
semantic = client.semantic_search(query, limit=5)
if semantic.get("available") and semantic.get("data"):
items = semantic["data"]
else:
results = client.search_tools(query, per_page=5)
items = results.data if hasattr(results, "data") else results
found = [
{
"name": f"{item.get('owner', '')}/{item.get('name', '')}",
"desc": (item.get("description") or "")[:50],
"args": [],
"registry": True,
"downloads": item.get("downloads", 0),
"relevance": item.get("similarity", item.get("score")),
"quality_score": item.get("quality_score"),
"quality_coverage": item.get("quality_coverage"),
}
for item in items
]
_registry_cache[normalized] = found
return [dict(item) for item in found]
except RegistryError:
return []
def _install_registry_selection(tool: dict) -> PickerResult:
"""Install a registry result and retain its qualified identity for execution."""
from ..resolver import install_from_registry
install_from_registry(tool["name"])
return PickerResult(tool["name"], {})
def _local_selection(tool: dict, arguments: Optional[dict] = None) -> PickerResult:
if tool.get("deprecated"):
guidance = tool.get("deprecated_message") or "This tool is no longer maintained."
if tool.get("replacement"):
guidance += f" Use '{tool['replacement']}' instead."
_write(f"{YELLOW}Warning: {guidance}{RESET}\n")
return PickerResult(tool["name"], arguments or {})
class TTYInput:
"""Read from /dev/tty for keyboard input, even when stdin is piped."""
@@ -103,10 +167,14 @@ class TTYInput:
if self.tty:
self.tty.close()
def getch(self):
def getch(self, timeout: Optional[float] = None):
"""Read a single character."""
tty.setraw(self.fd)
try:
if timeout is not None:
ready, _, _ = select.select([self.tty], [], [], timeout)
if not ready:
return None
ch = self.tty.read(1)
if ch == '\x1b':
# Read escape sequence - bytes arrive together from terminal
@@ -132,19 +200,25 @@ def clear_dropdown(n_lines: int):
def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
"""Run inline picker."""
tools = get_tools()
if not tools:
_write("No tools. Create one: cmdforge\n")
return None
query = ""
selected = 0
scroll = 0
last_drawn = 0
registry_results = {}
registry_future = None
registry_future_query = None
registry_executor = ThreadPoolExecutor(max_workers=1)
_write(HIDE_CURSOR)
try:
while True:
if registry_future is not None and registry_future.done():
registry_results[registry_future_query] = registry_future.result()
registry_future = None
registry_future_query = None
# Filter
matches = []
for t in tools:
@@ -157,6 +231,19 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
matches.sort(key=lambda x: -x[1])
filtered = [m[0] for m in matches]
# If no local matches, search registry
if query and len(query) >= 2 and not filtered:
normalized_query = query.strip().lower()
found = registry_results.get(normalized_query, [])
if registry_future is None and normalized_query not in registry_results:
registry_future_query = normalized_query
registry_future = registry_executor.submit(
search_registry, query
)
for rt in found:
rt["_registry"] = True
filtered = found
if selected >= len(filtered):
selected = max(0, len(filtered) - 1)
@@ -181,13 +268,29 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
# Items
for i, t in enumerate(visible):
actual_idx = scroll + i
is_registry = t.get("_registry", False) or t.get("registry", False)
is_deprecated = t.get("deprecated", False)
if actual_idx == selected:
prefix = f"{CYAN}{BOLD}▸ {t['name']}{RESET}"
else:
prefix = f" {t['name']}"
# Add registry marker
if is_registry:
prefix += f" {YELLOW}[registry]{RESET}"
if t.get("quality_score") is not None:
prefix += f" Q{t['quality_score']}"
if t.get("quality_coverage") is not None:
prefix += f"/{t['quality_coverage']}%"
if t.get("relevance") is not None:
prefix += f" R{float(t['relevance']):.2f}"
# Add deprecation marker
if is_deprecated:
prefix += f" {YELLOW}[deprecated]{RESET}"
# Add arg indicator
if t['args']:
if t.get('args'):
prefix += f" {GREEN}⚙{RESET}"
# Add description
@@ -201,24 +304,46 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
last_drawn = len(lines)
# Input
ch = tty_input.getch()
ch = tty_input.getch(
timeout=0.1 if registry_future is not None else None
)
if ch is None:
continue
if ch in ('\r', '\n'): # Enter - run
if filtered:
selected_tool = filtered[selected]
clear_dropdown(last_drawn)
_write(SHOW_CURSOR)
return PickerResult(filtered[selected]["name"], {})
# If it's a registry tool, install it first
if selected_tool.get("_registry") or selected_tool.get("registry"):
_write(f"{YELLOW}Installing {selected_tool['name']}...{RESET}\n")
try:
return _install_registry_selection(selected_tool)
except Exception as exc:
_write(f"Install failed: {exc}\n")
return None
return _local_selection(selected_tool)
elif ch == '\t': # Tab - configure args or run
if filtered:
tool = filtered[selected]
clear_dropdown(last_drawn)
if tool.get("_registry") or tool.get("registry"):
_write(f"{YELLOW}Installing {tool['name']}...{RESET}\n")
try:
_write(SHOW_CURSOR)
return _install_registry_selection(tool)
except Exception as exc:
_write(f"Install failed: {exc}\n")
_write(SHOW_CURSOR)
return None
if tool['args']:
args = pick_args(tty_input, tool)
_write(SHOW_CURSOR)
return PickerResult(tool["name"], args) if args is not None else None
return _local_selection(tool, args) if args is not None else None
_write(SHOW_CURSOR)
return PickerResult(tool["name"], {})
return _local_selection(tool)
elif ch == '\x1b' or ch == '\x03': # Esc or Ctrl+C
clear_dropdown(last_drawn)
@@ -243,6 +368,8 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
except Exception:
_write(SHOW_CURSOR)
raise
finally:
registry_executor.shutdown(wait=False, cancel_futures=True)
def pick_args(tty_input: TTYInput, tool: dict) -> Optional[dict]:
@@ -324,9 +451,23 @@ def pick_args(tty_input: TTYInput, tool: dict) -> Optional[dict]:
def main():
"""Entry point for cf command."""
global _ui_out
import argparse
import subprocess
import signal
# Parse metadata flags before touching stdin or /dev/tty. This keeps
# ``cf --help`` and ``cf --version`` usable in packaging checks, CI, and
# other non-interactive environments.
parser = argparse.ArgumentParser(
prog="cf",
description="Interactively find and run local or registry CmdForge tools",
)
from .. import __version__
parser.add_argument(
"--version", action="version", version=f"%(prog)s {__version__}"
)
parser.parse_args()
# Handle Ctrl+C gracefully
def handle_sigint(sig, frame):
# Restore cursor and exit cleanly
@@ -352,6 +493,19 @@ def main():
# Use stderr for UI if stdout is piped, so tool output stays clean
_ui_out = sys.stderr if not sys.stdout.isatty() else sys.stdout
try:
from ..usage import get_suggestions
suggestions = get_suggestions()[:3]
if suggestions:
_write(f"{GREEN}Frequent pipelines you could save as tools:{RESET}\n")
for item in suggestions:
_write(
f" {' | '.join(item['tools'])} "
f"{DIM}({item['count']} uses){RESET}\n"
)
except (OSError, ValueError):
pass
try:
with TTYInput() as tty_input:
result = run_picker(tty_input)
+40
View File
@@ -353,6 +353,17 @@ def _cmd_providers_discover(args):
def _cmd_providers_list(args):
"""List all providers and their status."""
providers = load_providers()
if getattr(args, "json", False):
import json
payload = []
for provider in providers:
available, status = _provider_status(provider)
item = provider.to_dict()
item["available"] = available
item["status"] = status
payload.append(item)
print(json.dumps(payload, indent=2))
return 0
print(f"Configured providers ({len(providers)}):\n")
for p in providers:
exists, status = _provider_status(p)
@@ -365,6 +376,15 @@ def _cmd_providers_list(args):
print(f" Status: {status}")
if p.description:
print(f" Info: {p.description}")
print(f" Locality: {p.locality}")
print(f" Data policy: {p.data_policy}")
if p.capabilities:
print(f" Capabilities: {', '.join(p.capabilities)}")
if p.model:
identity = p.model
if p.model_digest:
identity += f" ({p.model_digest})"
print(f" Model: {identity}")
print()
return 0
@@ -379,6 +399,17 @@ def _cmd_providers_add(args):
model = args.model if args.model is not None else (existing.model if existing else None)
api_key_env = args.api_key_env if args.api_key_env is not None else (existing.api_key_env if existing else None)
tags = args.tags if args.tags is not None else (existing.tags if existing else [])
locality = args.locality if args.locality is not None else (existing.locality if existing else "unknown")
capabilities = args.capabilities if args.capabilities is not None else (existing.capabilities if existing else [])
model_digest = args.model_digest if args.model_digest is not None else (existing.model_digest if existing else None)
cost_class = args.cost_class if args.cost_class is not None else (existing.cost_class if existing else None)
latency_class = args.latency_class if args.latency_class is not None else (existing.latency_class if existing else None)
max_context_tokens = (
args.max_context_tokens
if args.max_context_tokens is not None
else existing.max_context_tokens if existing else None
)
data_policy = args.data_policy if args.data_policy is not None else (existing.data_policy if existing else "unspecified")
fallback = args.fallback if args.fallback is not None else (existing.fallback if existing else None)
fallback = fallback or None
@@ -406,6 +437,15 @@ def _cmd_providers_add(args):
fallback_chain=fallback_chain,
api_key_env=api_key_env,
pty_config=existing.pty_config if existing else None,
tools=existing.tools if existing else None,
mcp_servers=existing.mcp_servers if existing else None,
locality=locality,
capabilities=capabilities,
model_digest=model_digest,
cost_class=cost_class,
latency_class=latency_class,
max_context_tokens=max_context_tokens,
data_policy=data_policy,
)
add_provider(provider)
print(f"Provider '{name}' added/updated.")
+287 -4
View File
@@ -29,6 +29,12 @@ def cmd_registry(args):
return _cmd_registry_update(args)
elif args.registry_cmd == "publish":
return _cmd_registry_publish(args)
elif args.registry_cmd == "signing-key":
return _cmd_registry_signing_key(args)
elif args.registry_cmd == "improve":
return _cmd_registry_improve(args)
elif args.registry_cmd == "review-improvement":
return _cmd_registry_review_improvement(args)
elif args.registry_cmd == "update-readme":
return _cmd_registry_update_readme(args)
elif args.registry_cmd == "my-tools":
@@ -51,6 +57,8 @@ def cmd_registry(args):
print(" info <tool> Show tool information")
print(" update Update local index cache")
print(" publish [path] Publish a tool")
print(" signing-key Manage Ed25519 release signing")
print(" improve Submit a community improvement")
print(" update-readme Update README for published tool(s)")
print(" my-tools List your published tools")
print(" status <tool> Check moderation status of a tool")
@@ -351,6 +359,108 @@ def _cmd_registry_update(args):
return 0
def _print_preflight_sections(report: dict, prefix: str = "") -> None:
"""Render the common portions of a structured preflight report."""
labels = (
("errors", "ERROR"),
("warnings", "WARN"),
("suggestions", "HINT"),
)
for key, marker in labels:
values = report.get(key) or []
if values:
print(f"{prefix}{key.title()} ({len(values)}):")
for value in values:
print(f" {marker}: {value}")
proposal = report.get("contract_proposal") or {}
proposed = {
key: proposal.get(key)
for key in ("input_schema", "output_schema")
if proposal.get(key) is not None
}
if proposed:
print(
f"{prefix}Contract proposal "
f"({proposal.get('confidence', 'low')} confidence; not saved):"
)
print(yaml.safe_dump(proposed, sort_keys=False).rstrip())
generated = report.get("generated_tests") or []
if generated:
print(f"{prefix}Contract conformance ({len(generated)} case(s)):")
for result in generated:
print(
f" {str(result.get('state', 'not_run')).upper()}: "
f"{result.get('step', 'unknown')} — {result.get('detail', '')}"
)
regression = report.get("regression") or {}
if regression:
marker = (
"REGRESSION" if regression.get("has_regressions")
else "REVIEW" if regression.get("contract_changed")
else "STABLE"
)
print(
f"{prefix}Regression comparison: {marker} — "
f"{regression.get('summary', 'no changes')}"
)
compatibility = report.get("compatibility") or []
if compatibility:
print(f"{prefix}ToolStep compatibility ({len(compatibility)}):")
for finding in compatibility:
print(
f" {str(finding.get('state', 'unknown')).upper()}: "
f"step {finding.get('step', '?')} -> "
f"{finding.get('tool', 'unknown')} — {finding.get('detail', '')}"
)
def _print_quality_summary(quality: dict | None) -> None:
if not quality:
return
print(
f"Quality {quality.get('score', 0)} — "
f"{quality.get('evidence_coverage', 0)}% evidence coverage"
)
for category in quality.get("categories", []):
state = category.get("state", "not_tested")
value = (
f"{category.get('earned', 0)}/{category.get('available', 0)}"
if state == "checked" else state.replace("_", " ")
)
print(f" {category.get('name', 'Unknown'):24s} {value:>12s}")
def _print_registry_suggestions(suggestions: dict) -> None:
"""Render registry-specific category, similarity, and scrutiny evidence."""
category = suggestions.get("category") or {}
if category.get("suggested"):
print(
f"Suggested category: {category['suggested']} "
f"({category.get('confidence', 0):.0%} confidence)"
)
similar = suggestions.get("similar_tools") or []
if similar:
print(f"Similar registry tools ({len(similar)}):")
for item in similar:
print(
f" - {item.get('name', 'unknown')} "
f"({item.get('similarity', 0):.0%} similar)"
)
scrutiny = suggestions.get("scrutiny") or {}
findings = scrutiny.get("findings") or []
if findings:
print(f"Scrutiny findings ({len(findings)}):")
for finding in findings:
marker = str(finding.get("result", "info")).upper()
print(f" {marker}: {finding.get('message', '')}")
def _cmd_registry_publish(args):
"""Publish a tool to the registry."""
from ..registry_client import RegistryError, get_client
@@ -404,24 +514,80 @@ def _cmd_registry_publish(args):
# Validate
try:
data = yaml.safe_load(config_yaml)
if not isinstance(data, dict):
print("Error: config.yaml must contain a YAML mapping", file=sys.stderr)
return 1
name = data.get("name", "")
version = data.get("version", "")
if not name or not version:
print("Error: config.yaml must have 'name' and 'version' fields", file=sys.stderr)
return 1
# Registry fields are local installation/status metadata. Including a
# previous release's hash or owner in a new payload makes content
# identity depend on stale registry state.
for key in (
"registry_hash",
"registry_status",
"registry_owner",
"registry_feedback",
):
data.pop(key, None)
config_yaml = yaml.dump(data, default_flow_style=False, sort_keys=False)
except yaml.YAMLError as e:
print(f"Error: Invalid YAML in config.yaml: {e}", file=sys.stderr)
return 1
if args.dry_run:
print("Dry run - validating only")
print("Dry run — validating locally and via registry preflight")
print()
print(f"Would publish:")
# Local validation
print(f" Name: {name}")
print(f" Version: {version}")
print(f" Config: {len(config_yaml)} bytes")
print(f" README: {len(readme)} bytes")
return 0
print()
from ..preflight import analyze_tool
from ..tool import Tool
try:
local_tool = Tool.from_dict(data)
local_tool.path = config_path
local_report = analyze_tool(
local_tool, include_contract_tests=True
)
except (KeyError, TypeError, ValueError) as exc:
print(f"Local preflight error: {exc}", file=sys.stderr)
return 1
_print_preflight_sections(local_report.to_dict(), prefix="Local ")
if not local_report.ok:
return 1
# Attempt registry preflight if token is configured
config = load_config()
if not config.registry.token:
print("No registry token configured — local preflight only.")
return 0
try:
client = get_client()
result = client.publish_tool(
config_yaml, readme=readme, defaults=defaults,
dry_run=True,
)
except RegistryError as exc:
print(f"Registry preflight failed: {exc}", file=sys.stderr)
return 1
remote_report = result.get("preflight") or {}
_print_preflight_sections(remote_report, prefix="Registry ")
_print_quality_summary(result.get("quality"))
_print_registry_suggestions(result.get("suggestions") or {})
if not remote_report.get("errors"):
print("Registry preflight passed.")
return 0
return 1
# Check for token
config = load_config()
@@ -451,6 +617,7 @@ def _cmd_registry_publish(args):
from ..tool import load_tool, Tool, ToolStep
dep_result = None
my_owner = ""
try:
client = get_client()
@@ -561,12 +728,70 @@ def _cmd_registry_publish(args):
except Exception:
pass # Continue with publish
# Stage 1 runs after dependency publication and any automatic version bump,
# so the evidence describes the exact version Stage 2 will mutate.
from ..preflight import analyze_tool
try:
preflight_tool = Tool.from_dict(data)
preflight_tool.path = config_path
local_report = analyze_tool(preflight_tool, include_contract_tests=True)
except (KeyError, TypeError, ValueError) as exc:
print(f"Local preflight error: {exc}", file=sys.stderr)
return 1
print("Preflight stage:")
_print_preflight_sections(local_report.to_dict(), prefix="Local ")
if not local_report.ok:
return 1
try:
preflight_client = get_client()
preflight_result = preflight_client.publish_tool(
config_yaml, readme=readme, defaults=defaults,
owner=getattr(args, "owner", ""), dry_run=True,
)
except RegistryError as exc:
print(f"Registry preflight failed: {exc}", file=sys.stderr)
return 1
remote_report = preflight_result.get("preflight") or {}
_print_preflight_sections(remote_report, prefix="Registry ")
_print_quality_summary(preflight_result.get("quality"))
if remote_report.get("errors"):
return 1
release_content_hash = preflight_result.get("content_hash", "")
if sys.stdin.isatty() and not getattr(args, "force", False):
try:
if input("Publish this validated version? [y/N] ").strip().lower() != "y":
print("Cancelled.")
return 1
except (EOFError, KeyboardInterrupt):
print("\nCancelled.")
return 1
print(f"Publishing {name}@{version}...")
try:
client = get_client()
owner = getattr(args, "owner", "")
result = client.publish_tool(config_yaml, readme, defaults, owner=owner)
attestation = None
from ..signing import load_signing_key
signing_key = load_signing_key()
if signing_key:
if not release_content_hash:
print(
"Registry did not return a content identity; refusing to sign.",
file=sys.stderr,
)
return 1
from ..attestation import sign_tool
signer = owner or my_owner
attestation = sign_tool(
name, version, release_content_hash, signer, signing_key[0]
).to_dict()
result = client.publish_tool(
config_yaml, readme, defaults, owner=owner,
attestation=attestation,
improvement_id=getattr(args, "improvement_id", None),
)
pr_url = result.get("pr_url", "")
status = result.get("status", "")
@@ -641,6 +866,64 @@ def _cmd_registry_publish(args):
return 0
def _cmd_registry_signing_key(args):
"""Initialize a local key and register its public half with the registry."""
from ..signing import SIGNING_KEY_FILE, initialize_signing_key, load_signing_key
from ..registry_client import RegistryError, get_client
if args.action == "status":
key = load_signing_key()
print(
f"Signing key: {SIGNING_KEY_FILE}"
if key else "No release signing key configured."
)
return 0 if key else 1
try:
_, public_key = initialize_signing_key()
get_client().set_signing_public_key(public_key)
except (OSError, ValueError, RegistryError) as exc:
print(f"Could not initialize signing key: {exc}", file=sys.stderr)
return 1
print(f"Release signing key initialized: {SIGNING_KEY_FILE}")
print("The private key is stored locally with mode 0600; back it up securely.")
return 0
def _cmd_registry_improve(args):
from ..registry_client import RegistryError, get_client
if "/" not in args.tool:
print("Tool must be specified as owner/name.", file=sys.stderr)
return 2
owner, name = args.tool.split("/", 1)
try:
proposed = Path(args.file).read_text(encoding="utf-8")
result = get_client().submit_improvement(
owner, name, args.version, args.step_index, proposed,
args.rationale,
)
except (OSError, RegistryError) as exc:
print(f"Could not submit improvement: {exc}", file=sys.stderr)
return 1
print(f"Improvement {result.get('id')} tested and submitted for review.")
return 0
def _cmd_registry_review_improvement(args):
from ..registry_client import RegistryError, get_client
try:
result = get_client().review_improvement(args.id, args.decision, args.notes)
except RegistryError as exc:
print(f"Could not review improvement: {exc}", file=sys.stderr)
return 1
print(f"Improvement {args.id}: {result.get('status')}")
if result.get("ready_to_apply"):
print(
"Publish the updated version with "
f"--improvement-id {args.id} to apply credit and badges."
)
return 0
def _cmd_registry_update_readme(args):
"""Update README for published tool(s) on the registry."""
from ..registry_client import RegistryError, get_client
+362 -46
View File
@@ -5,56 +5,336 @@ from pathlib import Path
from ..tool import (
list_tools, load_tool, save_tool, delete_tool, get_tools_dir,
Tool, ToolArgument, PromptStep, CodeStep, ToolStep, validate_tool_name
get_project_tools_dir, Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
validate_tool_name
)
from ..gui import run_gui
def _resolve_existing_tool(name: str):
"""Resolve project-local before global without registry auto-fetch."""
from ..resolver import find_tool
project_candidate = any(
(Path.cwd() / root / name / "config.yaml").exists()
for root in (".cmdforge", "cmdforge")
)
if project_candidate:
resolved = find_tool(name)
if resolved:
return resolved.tool
from .. import tool as tool_module
return tool_module.load_tool(name)
def _project_tool_names(project_dir=None) -> list[str]:
"""List project-local references using the same two resolver roots."""
project_dir = (project_dir or Path.cwd()).resolve()
names = []
for root_name in (".cmdforge", "cmdforge"):
root = project_dir / root_name
if not root.is_dir():
continue
for item in sorted(root.iterdir()):
if not item.is_dir() or item.name.startswith("."):
continue
if (item / "config.yaml").is_file():
names.append(item.name)
continue
for child in sorted(item.iterdir()):
if child.is_dir() and (child / "config.yaml").is_file():
names.append(f"{item.name}/{child.name}")
return list(dict.fromkeys(names))
def _execution_policy_from_args(*namespaces):
"""Combine top-level and wrapper-compatible execution policy flags."""
from ..providers import ProviderExecutionPolicy
capabilities = []
for namespace in namespaces:
capabilities.extend(
getattr(namespace, "require_capability", None) or []
)
data_classification = next((
getattr(namespace, "data_classification", None)
for namespace in namespaces
if getattr(namespace, "data_classification", None)
), "public")
return ProviderExecutionPolicy(
allow_fallback=not any(
getattr(namespace, "no_fallback", False) for namespace in namespaces
),
required_locality=(
"local" if any(
getattr(namespace, "require_local", False)
for namespace in namespaces
) else None
),
required_capabilities=tuple(dict.fromkeys(capabilities)),
data_classification=data_classification,
require_model_identity=any(
getattr(namespace, "require_model_identity", False)
for namespace in namespaces
),
require_model_digest=any(
getattr(namespace, "require_model_digest", False)
for namespace in namespaces
),
)
def _result_envelope(
output: str, provider_calls: list[dict], exit_code: int = 0
) -> str:
"""Build a stable machine-readable envelope without parsing tool output."""
import json
execution = {
"exit_code": exit_code,
"provider_calls": provider_calls,
"fallback_used": any(
call.get("fallback_used", False) for call in provider_calls
),
"actual_providers": list(dict.fromkeys(
call["actual_provider"] for call in provider_calls
if call.get("actual_provider")
)),
}
return json.dumps({"output": output, "execution": execution}, indent=2)
def cmd_list(args):
"""List all tools."""
tools = list_tools()
import json
if not tools:
print("No tools found.")
print("Create your first tool with: cmdforge ui")
filter_text = (getattr(args, "filter", "") or "").casefold()
limit = getattr(args, "limit", None)
if getattr(args, "full", False) and not getattr(args, "json", False):
print("Error: --full requires --json.", file=sys.stderr)
return 1
if limit is not None and limit < 1:
print("Error: --limit must be at least 1.", file=sys.stderr)
return 1
matches = []
project_names = _project_tool_names()
all_names = list(dict.fromkeys([*project_names, *sorted(list_tools())]))
for name in all_names:
tool = _resolve_existing_tool(name)
if not tool:
continue
if filter_text and (
filter_text not in name.casefold()
and filter_text not in (tool.description or "").casefold()
):
continue
matches.append((name, tool))
if limit is not None and len(matches) >= limit:
break
if getattr(args, "json", False):
full = getattr(args, "full", False)
result = [
_full_tool_entry(name, tool) if full else _catalog_tool_entry(name, tool)
for name, tool in matches
]
json.dump(result, sys.stdout, indent=2)
print()
return 0
print(f"Available tools ({len(tools)}):\n")
for name in tools:
tool = load_tool(name)
if tool:
# Show source indicator for imported tools
source_marker = ""
if tool.source and tool.source.type == "imported":
source_marker = " [imported]"
elif tool.source and tool.source.type == "forked":
source_marker = " [forked]"
if not matches:
if filter_text:
print(f"No tools match filter '{getattr(args, 'filter', '')}'.")
else:
print("No tools found.")
print("Create your first tool with: cmdforge ui")
return 0
print(f" {name}{source_marker}")
print(f" {tool.description or 'No description'}")
print(f"Available tools ({len(matches)}):\n")
for name, tool in matches:
# Show source indicator for imported tools
source_marker = ""
if tool.source and tool.source.type == "imported":
source_marker = " [imported]"
elif tool.source and tool.source.type == "forked":
source_marker = " [forked]"
# Show arguments
if tool.arguments:
args_str = ", ".join(arg.flag for arg in tool.arguments)
print(f" Arguments: {args_str}")
print(f" {name}{source_marker}")
print(f" {tool.description or 'No description'}")
# Show steps
if tool.steps:
step_info = []
for step in tool.steps:
if isinstance(step, PromptStep):
step_info.append(f"PROMPT[{step.provider}]")
elif isinstance(step, CodeStep):
step_info.append("CODE")
elif isinstance(step, ToolStep):
step_info.append(f"TOOL[{step.tool}]")
print(f" Steps: {' -> '.join(step_info)}")
# Show arguments
if tool.arguments:
args_str = ", ".join(arg.flag for arg in tool.arguments)
print(f" Arguments: {args_str}")
print()
# Show steps
if tool.steps:
step_info = []
for step in tool.steps:
if isinstance(step, PromptStep):
step_info.append(f"PROMPT[{step.provider}]")
elif isinstance(step, CodeStep):
step_info.append("CODE")
elif isinstance(step, ToolStep):
step_info.append(f"TOOL[{step.tool}]")
print(f" Steps: {' -> '.join(step_info)}")
print()
return 0
def _catalog_tool_entry(name, tool) -> dict:
"""Return the stable, compact discovery contract intended for agents."""
entry = {
"name": name,
"description": tool.description or "",
"version": tool.version or "",
"category": tool.category or "Other",
"arguments": [argument.to_dict() for argument in tool.arguments],
"input_schema": tool.input_schema,
"output_schema": tool.output_schema,
"deprecated": bool(tool.deprecated),
}
if tool.replacement:
entry["replacement"] = tool.replacement
return entry
def _full_tool_entry(name, tool) -> dict:
"""Return an explicitly requested complete definition."""
entry = tool.to_dict()
# list_tools() owns the canonical local/qualified reference. A config's
# internal name may be only the short name.
entry["name"] = name
return entry
def cmd_run_once(args):
"""Run a one-shot prompt through a provider without creating a tool."""
from ..providers import call_provider
timeout = getattr(args, "timeout", 60)
max_tokens = getattr(args, "max_tokens", None)
if timeout < 1 or timeout > 3600:
print("Error: --timeout must be between 1 and 3600 seconds.", file=sys.stderr)
return 1
if max_tokens is not None and not 1 <= max_tokens <= 1_000_000:
print("Error: --max-tokens must be between 1 and 1000000.", file=sys.stderr)
return 1
try:
prompt = _build_one_shot_prompt(
args.prompt, getattr(args, "input", None), sys.stdin
)
except (OSError, ValueError) as exc:
print(f"Error reading input: {exc}", file=sys.stderr)
return 1
if not prompt.strip():
print("Error: No prompt provided.", file=sys.stderr)
return 1
provider = _resolve_one_shot_provider(getattr(args, "provider", None))
if not provider:
print(
"Error: No usable provider found. Pass --provider, set "
"'cmdforge config set default_provider NAME', or run "
"'cmdforge providers install'.",
file=sys.stderr,
)
return 1
print(f"[run-once] Provider: {provider}", file=sys.stderr)
policy = _execution_policy_from_args(args)
result = call_provider(
provider, prompt, timeout=timeout, max_tokens=max_tokens,
execution_policy=policy,
)
if not result.success:
if getattr(args, "result_envelope", None) == "json":
print(_result_envelope("", [result.provenance()], exit_code=1))
print(f"Error: {result.error}", file=sys.stderr)
return 1
if getattr(args, "result_envelope", None) == "json":
print(_result_envelope(result.text, [result.provenance()]))
else:
print(result.text)
return 0
def _build_one_shot_prompt(template: str, input_path, stdin) -> str:
"""Combine a one-shot instruction with explicit or piped input."""
if template == "-":
if input_path and input_path != "-":
raise ValueError("prompt '-' cannot be combined with --input")
return stdin.read()
input_text = None
if input_path:
input_text = (
stdin.read()
if input_path == "-"
else Path(input_path).read_text(encoding="utf-8")
)
elif not stdin.isatty():
input_text = stdin.read()
if not input_text:
return template
if "{input}" in template:
return template.replace("{input}", input_text)
return f"{template.rstrip()}\n\n{input_text}"
def _resolve_one_shot_provider(explicit: str = None):
"""Honor the configured default, otherwise choose an available provider."""
if explicit:
return explicit
from ..config import load_config
from ..providers import PRESET_CHAINS, load_providers
configured = load_config().default_provider
if configured:
return configured
providers = {provider.name: provider for provider in load_providers()}
ordered_names = list(PRESET_CHAINS["balanced"][:-1])
ordered_names.extend(
name for name in providers if name not in ordered_names and name != "mock"
)
for name in ordered_names:
provider = providers.get(name)
if provider and _provider_is_available(provider):
return name
return None
def _provider_is_available(provider) -> bool:
"""Perform a non-invasive availability check without contacting a model."""
import os
import shlex
import shutil
if provider.type == "api":
key_name = provider.api_key_env or (
f"{provider.name.upper().replace('-', '_')}_API_KEY"
)
return bool(os.environ.get(key_name))
try:
parts = shlex.split(os.path.expandvars(provider.command))
except ValueError:
return False
if not parts:
return False
executable = Path(os.path.expanduser(parts[0]))
return bool(shutil.which(str(executable)) or executable.is_file())
def cmd_create(args):
"""Create a new tool (basic CLI creation - use 'ui' for full builder)."""
name = args.name
@@ -65,10 +345,22 @@ def cmd_create(args):
print(f"Error: Invalid tool name '{name}': {error_msg}")
return 1
# Check if already exists
existing = load_tool(name)
if existing and not args.force:
print(f"Error: Tool '{name}' already exists. Use --force to overwrite.")
output_dir = getattr(args, "output_dir", None)
project = getattr(args, "project", False)
target_root = (
Path(output_dir).resolve()
if output_dir is not None
else get_project_tools_dir() if project else get_tools_dir()
)
# Check only the requested scope. A project tool may intentionally shadow
# a global tool with the same name.
config_path = target_root / name / "config.yaml"
if config_path.exists() and not args.force:
print(
f"Error: Tool '{name}' already exists at {config_path}. "
"Use --force to overwrite."
)
return 1
# Create a tool with a single prompt step
@@ -89,11 +381,19 @@ def cmd_create(args):
output="{response}" if steps else "{input}"
)
path = save_tool(tool)
path = save_tool(tool, tools_dir=target_root)
print(f"Created tool '{name}'")
print(f"Config: {path}")
print(f"\nUse 'cmdforge ui' to add arguments, steps, and customize.")
print(f"Or run: {name} < input.txt")
print("\nUse 'cmdforge ui' to add arguments, steps, and customize.")
if project:
print(f"Run from its project: cmdforge run {name} < input.txt")
elif output_dir is not None:
print(
"Saved under the requested tool root. CmdForge automatically "
"discovers roots named .cmdforge or cmdforge."
)
else:
print(f"Or run: {name} < input.txt")
return 0
@@ -146,7 +446,7 @@ def cmd_delete(args):
def cmd_test(args):
"""Test a tool with mock provider."""
tool = load_tool(args.name)
tool = _resolve_existing_tool(args.name)
if not tool:
print(f"Error: Tool '{args.name}' not found.")
return 1
@@ -183,10 +483,14 @@ def cmd_run(args):
"""Run a tool."""
from ..runner import collect_custom_args, create_argument_parser, run_tool
tool = load_tool(args.name)
tool = _resolve_existing_tool(args.name)
if not tool:
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
return 1
from ..tool import deprecation_warning
warning = deprecation_warning(tool)
if warning:
print(warning, file=sys.stderr)
tool_args = list(args.tool_args or [])
if tool_args and tool_args[0] == '--':
@@ -223,6 +527,12 @@ def cmd_run(args):
show_prompt = args.show_prompt or parsed_tool_args.show_prompt
verbose = args.verbose or parsed_tool_args.verbose
auto_install = args.auto_install or parsed_tool_args.auto_install
execution_policy = _execution_policy_from_args(args, parsed_tool_args)
result_envelope = (
getattr(args, "result_envelope", None)
or getattr(parsed_tool_args, "result_envelope", None)
)
provider_calls = []
# Run tool
output, code = run_tool(
@@ -233,16 +543,22 @@ def cmd_run(args):
dry_run=dry_run,
show_prompt=show_prompt,
verbose=verbose,
auto_install=auto_install
auto_install=auto_install,
execution_policy=execution_policy,
provenance_log=provider_calls,
)
# Write output
if code == 0 and output:
if result_envelope == "json" or (code == 0 and output):
rendered_output = (
_result_envelope(output, provider_calls, exit_code=code)
if result_envelope == "json" else output
)
output_file = args.output or parsed_tool_args.output_file
if output_file:
Path(output_file).write_text(output)
Path(output_file).write_text(rendered_output)
else:
print(output)
print(rendered_output)
return code
+29
View File
@@ -0,0 +1,29 @@
"""CLI for opt-in local usage discovery."""
from ..usage import clear_usage, get_suggestions, is_enabled, set_enabled
def cmd_usage(args):
action = getattr(args, "usage_cmd", None) or "status"
if action == "enable":
set_enabled(True)
print("Local pipeline discovery enabled. No input, output, arguments, or telemetry is recorded.")
return 0
if action == "disable":
set_enabled(False)
print("Local pipeline discovery disabled.")
return 0
if action == "clear":
clear_usage()
print("Local usage history cleared.")
return 0
if action == "suggestions":
suggestions = get_suggestions(getattr(args, "threshold", 3))
if not suggestions:
print("No frequent pipeline suggestions yet.")
return 0
for index, item in enumerate(suggestions, start=1):
print(f"{index}. {' | '.join(item['tools'])} ({item['count']} uses)")
return 0
print(f"Local pipeline discovery: {'enabled' if is_enabled() else 'disabled'}")
return 0
+194
View File
@@ -0,0 +1,194 @@
"""Community improvement workflow (M9.3).
Allows community members to submit suggested prompt/code improvements.
Submissions are auto-tested before review. Successful improvements
credit both original and improving author.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional
from .tool import Tool, PromptStep, CodeStep
@dataclass
class ImprovementSubmission:
"""A community-submitted improvement for a tool."""
tool_name: str
tool_version: str
submitter: str
step_index: int
step_type: str # "prompt" | "code"
original: str
proposed: str
rationale: str = ""
submitted_at: str = ""
status: str = "pending" # pending | tested | approved | rejected
test_result: Optional[dict] = None
def to_dict(self) -> dict:
return {
"tool_name": self.tool_name,
"tool_version": self.tool_version,
"submitter": self.submitter,
"step_index": self.step_index,
"step_type": self.step_type,
"original": self.original[:200],
"proposed": self.proposed[:200],
"rationale": self.rationale,
"submitted_at": self.submitted_at,
"status": self.status,
"test_result": self.test_result,
}
@dataclass
class SubmissionReview:
"""Review result for an improvement submission."""
submission: ImprovementSubmission
decision: str # "approve" | "reject" | "request_changes"
reviewer: str = ""
notes: str = ""
reviewed_at: str = ""
def create_submission(
tool: Tool,
step_index: int,
proposed: str,
submitter: str,
rationale: str = "",
) -> ImprovementSubmission:
"""Create a new improvement submission."""
if step_index < 0 or step_index >= len(tool.steps):
raise ValueError(f"Step index {step_index} out of range (0-{len(tool.steps)-1})")
step = tool.steps[step_index]
if isinstance(step, PromptStep):
original = step.prompt
step_type = "prompt"
elif isinstance(step, CodeStep):
original = step.code
step_type = "code"
else:
raise ValueError(f"Step {step_index} is not a prompt or code step")
return ImprovementSubmission(
tool_name=tool.name,
tool_version=tool.version or "",
submitter=submitter,
step_index=step_index,
step_type=step_type,
original=original,
proposed=proposed,
rationale=rationale,
submitted_at=datetime.now(timezone.utc).isoformat(),
)
def run_submission_tests(submission: ImprovementSubmission, tool: Tool) -> dict:
"""Run contract tests on the tool with the proposed change applied.
Returns a dict with test results comparing baseline vs proposed.
"""
import copy
from .contract_testing import run_contract_tests
if submission.tool_name != tool.name or submission.tool_version != (tool.version or ""):
raise ValueError("Submission targets a different tool or version")
# Baseline
baseline = run_contract_tests(tool)
baseline_passed = sum(
1 for r in baseline.results if r.state == "passed"
)
# Apply proposed change
tool_copy = copy.deepcopy(tool)
if submission.step_index >= len(tool_copy.steps):
return {"error": "Step index out of range", "baseline_passed": baseline_passed}
step = tool_copy.steps[submission.step_index]
current = step.prompt if isinstance(step, PromptStep) else (
step.code if isinstance(step, CodeStep) else None
)
if current != submission.original:
return {
"error": "Original step has changed since submission",
"baseline_passed": baseline_passed,
}
if isinstance(step, PromptStep) and submission.step_type == "prompt":
step.prompt = submission.proposed
elif isinstance(step, CodeStep) and submission.step_type == "code":
try:
compile(submission.proposed, f"<{tool.name}:step-{submission.step_index}>", "exec")
except SyntaxError as exc:
result = {
"error": f"Proposed code has invalid syntax: {exc.msg}",
"baseline_passed": baseline_passed,
"passed_for_review": False,
}
submission.test_result = result
return result
step.code = submission.proposed
else:
return {"error": "Step type mismatch", "baseline_passed": baseline_passed}
# Test with proposed change
proposed_result = run_contract_tests(tool_copy)
proposed_passed = sum(
1 for r in proposed_result.results if r.state == "passed"
)
result = {
"baseline_passed": baseline_passed,
"proposed_passed": proposed_passed,
"improvement": proposed_passed - baseline_passed,
"regressed": proposed_passed < baseline_passed,
"baseline_outcome": baseline.outcome,
"proposed_outcome": proposed_result.outcome,
"passed_for_review": (
(
proposed_result.outcome == "passed"
and proposed_passed >= baseline_passed
)
or (
submission.step_type == "code"
and proposed_result.outcome == "unsupported"
)
),
"details": [r.to_dict() for r in proposed_result.results],
}
submission.status = "tested"
submission.test_result = result
return result
def review_submission(
submission: ImprovementSubmission,
decision: str,
reviewer: str,
notes: str = "",
) -> SubmissionReview:
"""Review a tested submission."""
if decision not in ("approve", "reject", "request_changes"):
raise ValueError("Decision must be: approve, reject, or request_changes")
if submission.status != "tested" or not (
submission.test_result or {}
).get("passed_for_review"):
raise ValueError("Submission must be tested before review")
submission.status = "approved" if decision == "approve" else decision
return SubmissionReview(
submission=submission,
decision=decision,
reviewer=reviewer,
notes=notes,
reviewed_at=datetime.now(timezone.utc).isoformat(),
)
+906
View File
@@ -0,0 +1,906 @@
"""Deterministic, side-effect-safe contract conformance verification.
The verifier generates inputs that are validated against ``input_schema``,
replaces prompt steps with schema-derived deterministic values, executes the
remaining safe pipeline, and validates its final result. It never calls an AI
provider. Steps that could execute user code or contact another tool/server are
reported as unsupported instead of being run implicitly.
"""
from __future__ import annotations
import copy
import hashlib
import json
import math
import os
import re
import tempfile
from dataclasses import asdict, dataclass, field
from datetime import date, datetime, timezone
from pathlib import Path
from typing import Any, Dict, List, Literal, Optional
from uuid import UUID
from .tool import CodeStep, McpStep, PromptStep, Tool, ToolStep
ConformanceState = Literal["passed", "failed", "unsupported", "not_run"]
_VALID_STATES = ("passed", "failed", "unsupported", "not_run")
class UnsupportedContract(ValueError):
"""Raised when a valid deterministic value cannot be produced safely."""
@dataclass
class ConformanceResult:
step: str
state: ConformanceState
detail: str = ""
case_id: str = ""
input_value: Any = None
input_fingerprint: str = ""
output_shape: Optional[Dict[str, Any]] = None
output_fingerprint: str = ""
def __post_init__(self) -> None:
if self.state not in _VALID_STATES:
raise ValueError(f"Unknown conformance state: {self.state}")
def is_pass(self) -> bool:
return self.state == "passed"
def to_dict(self) -> Dict[str, Any]:
return asdict(self)
@dataclass
class ConformanceReport:
results: List[ConformanceResult] = field(default_factory=list)
@property
def outcome(self) -> ConformanceState:
states = {result.state for result in self.results}
if "failed" in states:
return "failed"
if "passed" in states:
return "passed"
if "unsupported" in states:
return "unsupported"
return "not_run"
@property
def passed(self) -> bool:
return self.outcome == "passed"
@property
def failed(self) -> bool:
return self.outcome == "failed"
@property
def summary(self) -> Dict[str, int]:
counts = {state: 0 for state in _VALID_STATES}
for result in self.results:
counts[result.state] += 1
return counts
def to_dict(self) -> Dict[str, Any]:
return {
"outcome": self.outcome,
"summary": self.summary,
"results": [result.to_dict() for result in self.results],
}
def run_contract_tests(
tool: Tool, test_inputs: Optional[List[Any]] = None
) -> ConformanceReport:
"""Run safe structural conformance tests for a tool.
This is deliberately not a semantic quality test. Original CodeStep,
ToolStep, and McpStep instances are not executed because they may have side
effects, make network calls, or invoke real providers.
"""
if tool.input_schema is None or tool.output_schema is None:
return _single_result(
"preflight",
"not_run",
"Both input_schema and output_schema are required",
)
unsafe = next(
(
type(step).__name__
for step in tool.steps
if isinstance(step, (CodeStep, ToolStep, McpStep))
),
None,
)
if unsafe:
return _single_result(
"preflight",
"unsupported",
f"Safe conformance mode does not execute {unsafe}",
)
generation_issue = None
try:
generated_cases = _generate_inputs(tool.input_schema)
except UnsupportedContract as exc:
if not test_inputs:
return _single_result("input-generation", "unsupported", str(exc))
generated_cases = []
generation_issue = str(exc)
test_cases = list(copy.deepcopy(test_inputs or [])) + generated_cases
deduplicated = {}
for test_input in test_cases:
deduplicated[_fingerprint(test_input)] = test_input
test_cases = list(deduplicated.values())
if not test_cases:
return _single_result(
"input-generation", "not_run", "No deterministic inputs generated"
)
report = ConformanceReport()
if generation_issue:
report.results.append(ConformanceResult(
"input-generation", "unsupported", generation_issue
))
for test_input in test_cases:
input_fingerprint = _fingerprint(test_input)
case_id = f"case-{input_fingerprint[:12]}"
evidence = {
"case_id": case_id,
"input_value": copy.deepcopy(test_input),
"input_fingerprint": input_fingerprint,
}
try:
_validate_instance(test_input, tool.input_schema, "input")
except UnsupportedContract as exc:
report.results.append(ConformanceResult(
case_id, "unsupported", str(exc), **evidence
))
continue
except ValueError as exc:
report.results.append(ConformanceResult(
case_id, "failed", f"Stored {exc}", **evidence
))
continue
try:
result = _run_with_stub(tool, test_input)
except UnsupportedContract as exc:
report.results.append(
ConformanceResult(case_id, "unsupported", str(exc), **evidence)
)
except Exception as exc:
report.results.append(
ConformanceResult(
case_id, "failed", f"Execution error: {exc}", **evidence
)
)
else:
output_evidence = {
"output_shape": _json_shape(result),
"output_fingerprint": _fingerprint(result),
}
try:
_validate_instance(result, tool.output_schema, "output")
except UnsupportedContract as exc:
report.results.append(
ConformanceResult(
case_id, "unsupported", str(exc),
**evidence, **output_evidence,
)
)
except ValueError as exc:
report.results.append(ConformanceResult(
case_id, "failed", str(exc), **evidence, **output_evidence
))
else:
report.results.append(
ConformanceResult(
case_id, "passed", "Output conforms to output_schema",
**evidence, **output_evidence,
)
)
return report
def _single_result(
step: str, state: ConformanceState, detail: str
) -> ConformanceReport:
return ConformanceReport([ConformanceResult(step, state, detail)])
def _canonical_json(value: Any) -> str:
return json.dumps(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False,
default=str,
)
def _fingerprint(value: Any) -> str:
return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest()
def _json_shape(value: Any) -> Dict[str, Any]:
if value is None:
return {"type": "null"}
if isinstance(value, bool):
return {"type": "boolean"}
if isinstance(value, int):
return {"type": "integer"}
if isinstance(value, float):
return {"type": "number"}
if isinstance(value, str):
return {"type": "string"}
if isinstance(value, dict):
return {
"type": "object",
"properties": {
str(key): _json_shape(item)
for key, item in sorted(value.items(), key=lambda pair: str(pair[0]))
},
}
if isinstance(value, list):
shapes = {
_canonical_json(_json_shape(item)): _json_shape(item) for item in value
}
return {
"type": "array",
"items": [shapes[key] for key in sorted(shapes)],
}
return {"type": type(value).__name__}
def _generate_inputs(schema: dict) -> List[Any]:
"""Generate deterministic values and validate every returned candidate."""
candidates: List[Any] = []
schema_type = _schema_type(schema)
if schema_type == "object":
enum_properties = {
name: spec["enum"]
for name, spec in schema.get("properties", {}).items()
if isinstance(spec, dict) and spec.get("enum")
}
variant_count = min(max((len(v) for v in enum_properties.values()), default=1), 3)
for index in range(variant_count):
overrides = {
name: values[index % len(values)]
for name, values in enum_properties.items()
}
candidates.append(_object_value(schema, overrides))
else:
candidates.append(_value_for_schema(schema, "input"))
unique: List[Any] = []
seen = set()
for candidate in candidates:
try:
_validate_instance(candidate, schema, "generated input")
except ValueError as exc:
raise UnsupportedContract(
f"Could not generate a valid input: {exc}"
) from exc
marker = json.dumps(candidate, sort_keys=True, default=str)
if marker not in seen:
seen.add(marker)
unique.append(candidate)
return unique
def _schema_type(schema: dict) -> str | None:
schema_type = schema.get("type")
if isinstance(schema_type, list):
return next((value for value in schema_type if value != "null"), "null")
if schema_type is None and ("properties" in schema or "required" in schema):
return "object"
return schema_type
def _value_for_schema(schema: dict, name: str) -> Any:
for keyword in ("const", "default"):
if keyword in schema:
return copy.deepcopy(schema[keyword])
if schema.get("examples"):
return copy.deepcopy(schema["examples"][0])
if schema.get("enum"):
return copy.deepcopy(schema["enum"][0])
alternatives = schema.get("oneOf") or schema.get("anyOf")
if alternatives:
for alternative in alternatives:
try:
candidate = _value_for_schema(alternative, name)
_validate_instance(candidate, schema, name)
return candidate
except (UnsupportedContract, ValueError):
continue
raise UnsupportedContract(f"Cannot synthesize {name} from alternatives")
if schema.get("allOf"):
merged = _merge_all_of(schema)
candidate = _value_for_schema(merged, name)
_validate_instance(candidate, schema, name)
return candidate
schema_type = _schema_type(schema)
if schema_type == "object":
return _object_value(schema, {})
if schema_type == "array":
item_schema = schema.get("items", {})
count = max(0, schema.get("minItems", 0))
if count == 0 and schema.get("contains"):
return [_value_for_schema(schema["contains"], f"{name}-item")]
return [
_vary_value(_value_for_schema(item_schema, f"{name}-item"), index)
for index in range(count)
]
if schema_type == "string":
return _string_value(schema, name)
if schema_type in ("integer", "number"):
return _number_value(schema, integer=schema_type == "integer")
if schema_type == "boolean":
return True
if schema_type == "null":
return None
if schema_type is None and not schema:
return {}
if schema_type is None:
candidate = {}
try:
_validate_instance(candidate, schema, name)
return candidate
except ValueError as exc:
raise UnsupportedContract(str(exc)) from exc
raise UnsupportedContract(f"Unsupported JSON Schema type for {name}: {schema_type}")
def _object_value(schema: dict, overrides: Dict[str, Any]) -> Dict[str, Any]:
properties = schema.get("properties", {})
required = list(schema.get("required", []))
result: Dict[str, Any] = {}
names = list(required)
for name, spec in properties.items():
if name in overrides or any(
keyword in spec for keyword in ("const", "default", "examples", "enum")
):
if name not in names:
names.append(name)
minimum = schema.get("minProperties", 0)
for name in properties:
if len(names) >= minimum:
break
if name not in names:
names.append(name)
for name in names:
if name in overrides:
result[name] = copy.deepcopy(overrides[name])
else:
result[name] = _value_for_schema(properties.get(name, {}), name)
for trigger, dependents in schema.get("dependentRequired", {}).items():
if trigger in result:
for dependent in dependents:
if dependent not in result:
result[dependent] = _value_for_schema(
properties.get(dependent, {}), dependent
)
return result
def _string_value(schema: dict, name: str) -> str:
formats = {
"date": date(2020, 1, 2).isoformat(),
"date-time": datetime(2020, 1, 2, 3, 4, 5, tzinfo=timezone.utc).isoformat(),
"email": "[email protected]",
"hostname": "example.com",
"ipv4": "192.0.2.1",
"uri": "https://example.com/",
"uuid": str(UUID(int=0)),
}
minimum = max(0, schema.get("minLength", 0))
maximum = schema.get("maxLength")
seeds = [formats.get(schema.get("format")), f"test-{name}", "a", "0", ""]
pattern = schema.get("pattern")
for seed in seeds:
if seed is None:
continue
candidate = seed
if len(candidate) < minimum:
candidate += "a" * (minimum - len(candidate))
if maximum is not None:
candidate = candidate[:maximum]
if pattern and re.search(pattern, candidate) is None:
continue
return candidate
raise UnsupportedContract(f"Cannot synthesize string '{name}' for pattern/bounds")
def _number_value(schema: dict, *, integer: bool) -> int | float:
lower = schema.get("minimum")
if "exclusiveMinimum" in schema:
exclusive_lower = schema["exclusiveMinimum"] + (1 if integer else 0.5)
lower = exclusive_lower if lower is None else max(lower, exclusive_lower)
upper = schema.get("maximum")
if "exclusiveMaximum" in schema:
exclusive_upper = schema["exclusiveMaximum"] - (1 if integer else 0.5)
upper = exclusive_upper if upper is None else min(upper, exclusive_upper)
value = 0
if lower is not None:
value = max(value, lower)
if upper is not None:
value = min(value, upper)
multiple = schema.get("multipleOf")
if multiple:
if lower is not None:
value = math.ceil(value / multiple) * multiple
else:
value = math.floor(value / multiple) * multiple
if (lower is not None and value < lower) or (upper is not None and value > upper):
raise UnsupportedContract("Numeric bounds do not admit a deterministic value")
return int(value) if integer else float(value)
def _vary_value(value: Any, index: int) -> Any:
if index == 0:
return value
if isinstance(value, bool) or value is None:
return value
if isinstance(value, int):
return value + index
if isinstance(value, float):
return value + float(index)
if isinstance(value, str):
return f"{value}{index}"
return value
def _merge_all_of(schema: dict) -> dict:
merged = {key: copy.deepcopy(value) for key, value in schema.items() if key != "allOf"}
properties = copy.deepcopy(merged.get("properties", {}))
required = list(merged.get("required", []))
for part in schema["allOf"]:
properties.update(copy.deepcopy(part.get("properties", {})))
for name in part.get("required", []):
if name not in required:
required.append(name)
for key, value in part.items():
if key not in ("properties", "required"):
merged.setdefault(key, copy.deepcopy(value))
if properties:
merged["properties"] = properties
if required:
merged["required"] = required
return merged
def _validate_instance(instance: Any, schema: dict, label: str) -> None:
try:
from jsonschema import FormatChecker
from jsonschema.exceptions import ValidationError
from jsonschema.validators import validator_for
except ImportError as exc:
raise UnsupportedContract("jsonschema is required for conformance testing") from exc
validator = validator_for(schema)(schema, format_checker=FormatChecker())
try:
validator.validate(instance)
except ValidationError as exc:
path = ".".join(str(part) for part in exc.absolute_path) or "root"
raise ValueError(f"{label} does not conform at {path}: {exc.message}") from exc
def _run_with_stub(tool: Tool, inputs: Any) -> Any:
"""Execute only a prompt/no-step pipeline with generated prompt outputs."""
from .runner import DEFAULT_OUTPUT_SCHEMA, run_tool
tool_copy = copy.deepcopy(tool)
transformed = []
final_variable = _single_output_variable(tool.output)
for step in tool_copy.steps:
if not isinstance(step, PromptStep):
raise UnsupportedContract(
f"Safe conformance mode does not execute {type(step).__name__}"
)
schema = step.output_schema
if schema is None and step.output_var == final_variable:
schema = tool.output_schema
if schema is None:
schema = DEFAULT_OUTPUT_SCHEMA
value = _value_for_schema(schema, step.output_var)
try:
_validate_instance(value, schema, f"stub output '{step.output_var}'")
except ValueError as exc:
raise UnsupportedContract(
f"Could not synthesize '{step.output_var}': {exc}"
) from exc
transformed.append(
CodeStep(
code=f"globals()[{step.output_var!r}] = {value!r}",
output_var=step.output_var,
name=f"Contract stub for {step.name or step.output_var}",
)
)
tool_copy.steps = transformed
if isinstance(inputs, dict):
custom_args = {key: value for key, value in inputs.items() if key != "input"}
input_text = inputs.get("input", "")
else:
custom_args = {}
input_text = inputs
output, exit_code = run_tool(
tool_copy,
input_text="" if input_text is None else str(input_text),
custom_args=custom_args,
provider_override=None,
dry_run=False,
verbose=False,
)
if exit_code != 0:
raise RuntimeError(f"Tool execution failed with exit code {exit_code}")
try:
return json.loads(output)
except (json.JSONDecodeError, TypeError):
return output
def _single_output_variable(output: str) -> str | None:
match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", output.strip())
return match.group(1) if match else None
def _fallback_value(prop: dict, name: str, required: list | None = None) -> Any:
"""Backward-compatible helper used by callers and focused tests."""
return _value_for_schema(prop, name)
def _validate_output(output: Any, schema: dict) -> bool:
"""Backward-compatible boolean output validator."""
try:
_validate_instance(output, schema, "output")
return True
except ValueError:
return False
# ---------------------------------------------------------------------------
# M8.V2: Regression comparison
# ---------------------------------------------------------------------------
CONFORMANCE_FILE = "conformance.json"
BASELINE_FORMAT_VERSION = 1
CONFORMANCE_ENGINE_VERSION = 1
class BaselineFormatError(ValueError):
"""Raised when a stored conformance baseline is corrupt or incompatible."""
@dataclass
class RegressionChange:
case_id: str
kind: str
baseline_state: str = "missing"
current_state: str = "missing"
detail: str = ""
def to_dict(self) -> Dict[str, str]:
return asdict(self)
@dataclass
class RegressionDelta:
"""Difference between baseline and current conformance results."""
tool: str
baseline_version: str
current_version: str
baseline_timestamp: str
current_timestamp: str
baseline_contract_hash: str
current_contract_hash: str
regressed: List[RegressionChange] = field(default_factory=list)
improved: List[RegressionChange] = field(default_factory=list)
new_failures: List[RegressionChange] = field(default_factory=list)
new_cases: List[RegressionChange] = field(default_factory=list)
removed_cases: List[RegressionChange] = field(default_factory=list)
coverage_losses: List[RegressionChange] = field(default_factory=list)
output_changes: List[RegressionChange] = field(default_factory=list)
unchanged: List[RegressionChange] = field(default_factory=list)
@property
def has_regressions(self) -> bool:
return any((
self.regressed,
self.new_failures,
self.removed_cases,
self.coverage_losses,
self.output_changes,
))
@property
def contract_changed(self) -> bool:
return self.baseline_contract_hash != self.current_contract_hash
@property
def summary(self) -> str:
parts = []
if self.contract_changed:
parts.append("contract changed")
if self.regressed:
parts.append(f"{len(self.regressed)} regression(s)")
if self.new_failures:
parts.append(f"{len(self.new_failures)} new failure(s)")
if self.improved:
parts.append(f"{len(self.improved)} improvement(s)")
if self.coverage_losses:
parts.append(f"{len(self.coverage_losses)} coverage loss(es)")
if self.removed_cases:
parts.append(f"{len(self.removed_cases)} removed case(s)")
if self.output_changes:
parts.append(f"{len(self.output_changes)} output change(s)")
if self.new_cases:
parts.append(f"{len(self.new_cases)} new case(s)")
if self.unchanged:
parts.append(f"{len(self.unchanged)} unchanged case(s)")
return ", ".join(parts) if parts else "no changes"
def to_dict(self) -> Dict[str, Any]:
return {
"tool": self.tool,
"baseline_version": self.baseline_version,
"current_version": self.current_version,
"baseline_timestamp": self.baseline_timestamp,
"current_timestamp": self.current_timestamp,
"baseline_contract_hash": self.baseline_contract_hash,
"current_contract_hash": self.current_contract_hash,
"contract_changed": self.contract_changed,
"has_regressions": self.has_regressions,
"summary": self.summary,
**{
name: [change.to_dict() for change in getattr(self, name)]
for name in (
"regressed", "improved", "new_failures", "new_cases",
"removed_cases", "coverage_losses", "output_changes",
"unchanged",
)
},
}
def save_conformance_baseline(tool: Tool, report: ConformanceReport, tool_dir: Path) -> Path:
"""Atomically persist a successful, reproducible conformance baseline."""
if not report.results or any(
result.state != "passed" for result in report.results
):
raise ValueError("Only a passing conformance report can become a baseline")
if any(not result.case_id or not result.input_fingerprint for result in report.results):
raise ValueError("Baseline results must include stable case evidence")
tool_dir = Path(tool_dir)
tool_dir.mkdir(parents=True, exist_ok=True)
path = tool_dir / CONFORMANCE_FILE
data = {
"format_version": BASELINE_FORMAT_VERSION,
"engine_version": CONFORMANCE_ENGINE_VERSION,
"tool": {"name": tool.name, "version": tool.version},
"timestamp": datetime.now(timezone.utc).isoformat(),
"contract_hash": _contract_hash(tool),
"contracts": {
"input_schema": copy.deepcopy(tool.input_schema),
"output_schema": copy.deepcopy(tool.output_schema),
},
"results": [r.to_dict() for r in report.results],
}
_validate_baseline_data(data)
descriptor, temporary_name = tempfile.mkstemp(
prefix=f".{CONFORMANCE_FILE}.", dir=tool_dir
)
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
json.dump(data, handle, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.chmod(temporary_name, 0o600)
os.replace(temporary_name, path)
finally:
if os.path.exists(temporary_name):
os.unlink(temporary_name)
return path
def load_conformance_baseline(tool_dir: Path) -> Optional[dict]:
"""Load and validate a previously saved conformance baseline."""
path = Path(tool_dir) / CONFORMANCE_FILE
if not path.exists():
return None
try:
data = json.loads(path.read_text(encoding="utf-8"))
except OSError as exc:
raise BaselineFormatError(f"Could not read baseline: {exc}") from exc
except json.JSONDecodeError as exc:
raise BaselineFormatError(f"Baseline is not valid JSON: {exc}") from exc
_validate_baseline_data(data)
return data
def compare_conformance(
baseline: dict,
current: ConformanceReport,
tool: Tool,
) -> RegressionDelta:
"""Compare stable cases, states, coverage, and normalized output evidence."""
_validate_baseline_data(baseline)
if baseline["tool"]["name"] != tool.name:
raise BaselineFormatError(
f"Baseline belongs to '{baseline['tool']['name']}', not '{tool.name}'"
)
base_map = {result["case_id"]: result for result in baseline["results"]}
current_map = {result.case_id: result for result in current.results if result.case_id}
delta = RegressionDelta(
tool=tool.name,
baseline_version=baseline["tool"]["version"],
current_version=tool.version,
baseline_timestamp=baseline["timestamp"],
current_timestamp=datetime.now(timezone.utc).isoformat(),
baseline_contract_hash=baseline["contract_hash"],
current_contract_hash=_contract_hash(tool),
)
for case_id, current_result in current_map.items():
base_result = base_map.get(case_id)
if base_result is None:
if current_result.state == "failed":
delta.new_failures.append(_change(
case_id, "new_failure", None, current_result
))
else:
delta.new_cases.append(_change(
case_id, "new_case", None, current_result
))
continue
baseline_state = base_result["state"]
if baseline_state == "passed" and current_result.state == "failed":
delta.regressed.append(_change(
case_id, "state_regression", base_result, current_result
))
elif baseline_state == "passed" and current_result.state in (
"unsupported", "not_run"
):
delta.coverage_losses.append(_change(
case_id, "coverage_loss", base_result, current_result
))
elif baseline_state != "passed" and current_result.state == "passed":
delta.improved.append(_change(
case_id, "improved", base_result, current_result
))
elif (
baseline_state == "passed"
and current_result.state == "passed"
and (
base_result.get("output_shape") != current_result.output_shape
or base_result.get("output_fingerprint")
!= current_result.output_fingerprint
)
):
delta.output_changes.append(_change(
case_id, "output_changed", base_result, current_result
))
else:
delta.unchanged.append(_change(
case_id, "unchanged", base_result, current_result
))
for case_id, baseline_result in base_map.items():
if case_id not in current_map:
delta.removed_cases.append(_change(
case_id, "case_removed", baseline_result, None
))
return delta
def _change(
case_id: str,
kind: str,
baseline: Optional[dict],
current: Optional[ConformanceResult],
) -> RegressionChange:
baseline_state = baseline.get("state", "missing") if baseline else "missing"
current_state = current.state if current else "missing"
detail = current.detail if current else "Previously recorded case is absent"
return RegressionChange(
case_id=case_id,
kind=kind,
baseline_state=baseline_state,
current_state=current_state,
detail=detail,
)
def _contract_hash(tool: Tool) -> str:
return _fingerprint({
"input_schema": tool.input_schema,
"output_schema": tool.output_schema,
})
def _validate_baseline_data(data: Any) -> None:
if not isinstance(data, dict):
raise BaselineFormatError("Baseline root must be an object")
if data.get("format_version") != BASELINE_FORMAT_VERSION:
raise BaselineFormatError("Unsupported baseline format version")
if data.get("engine_version") != CONFORMANCE_ENGINE_VERSION:
raise BaselineFormatError("Unsupported conformance engine version")
tool = data.get("tool")
if not isinstance(tool, dict) or not isinstance(tool.get("name"), str):
raise BaselineFormatError("Baseline tool metadata is invalid")
if not isinstance(tool.get("version"), str):
raise BaselineFormatError("Baseline tool version is invalid")
if not isinstance(data.get("timestamp"), str):
raise BaselineFormatError("Baseline timestamp is invalid")
if not isinstance(data.get("contract_hash"), str) or not re.fullmatch(
r"[0-9a-f]{64}", data["contract_hash"]
):
raise BaselineFormatError("Baseline contract hash is invalid")
if not isinstance(data.get("contracts"), dict):
raise BaselineFormatError("Baseline contracts are invalid")
if _fingerprint(data["contracts"]) != data["contract_hash"]:
raise BaselineFormatError("Baseline contract hash does not match contracts")
results = data.get("results")
if not isinstance(results, list):
raise BaselineFormatError("Baseline results must be an array")
seen = set()
for result in results:
if not isinstance(result, dict):
raise BaselineFormatError("Baseline result must be an object")
case_id = result.get("case_id")
if not isinstance(case_id, str) or not case_id or case_id in seen:
raise BaselineFormatError("Baseline case IDs must be unique strings")
seen.add(case_id)
if result.get("state") not in _VALID_STATES:
raise BaselineFormatError(f"Invalid state for baseline case '{case_id}'")
if not isinstance(result.get("input_fingerprint"), str) or not re.fullmatch(
r"[0-9a-f]{64}", result["input_fingerprint"]
):
raise BaselineFormatError(
f"Missing input fingerprint for baseline case '{case_id}'"
)
if "input_value" not in result:
raise BaselineFormatError(
f"Missing input value for baseline case '{case_id}'"
)
if _fingerprint(result["input_value"]) != result["input_fingerprint"]:
raise BaselineFormatError(
f"Input fingerprint mismatch for baseline case '{case_id}'"
)
shape = result.get("output_shape")
if shape is not None and not isinstance(shape, dict):
raise BaselineFormatError(
f"Invalid output shape for baseline case '{case_id}'"
)
if result["state"] == "passed":
if not isinstance(shape, dict):
raise BaselineFormatError(
f"Passing baseline case '{case_id}' lacks output shape"
)
fingerprint = result.get("output_fingerprint")
if not isinstance(fingerprint, str) or not re.fullmatch(
r"[0-9a-f]{64}", fingerprint
):
raise BaselineFormatError(
f"Passing baseline case '{case_id}' lacks output fingerprint"
)
+81 -1
View File
@@ -44,6 +44,44 @@ class ProviderDialog(QDialog):
self.desc_input.setPlaceholderText("Claude AI via claude-cli")
form.addRow("Description:", self.desc_input)
self.locality_combo = QComboBox()
for value in ("unknown", "local", "remote"):
self.locality_combo.addItem(value.title(), value)
form.addRow("Locality:", self.locality_combo)
self.data_policy_combo = QComboBox()
for value in ("unspecified", "public", "internal", "private"):
self.data_policy_combo.addItem(value.title(), value)
form.addRow("Data policy:", self.data_policy_combo)
self.capabilities_input = QLineEdit()
self.capabilities_input.setPlaceholderText("text, structured-json, reasoning")
form.addRow("Capabilities:", self.capabilities_input)
self.model_input = QLineEdit()
self.model_input.setPlaceholderText("Model ID, if known")
form.addRow("Model:", self.model_input)
self.model_digest_input = QLineEdit()
self.model_digest_input.setPlaceholderText("sha256:... (verified local models)")
form.addRow("Model digest:", self.model_digest_input)
self.max_context_input = QLineEdit()
self.max_context_input.setPlaceholderText("Maximum context tokens")
form.addRow("Context tokens:", self.max_context_input)
self.cost_class_combo = QComboBox()
self.cost_class_combo.addItem("Unspecified", None)
for value in ("free", "low", "medium", "high"):
self.cost_class_combo.addItem(value.title(), value)
form.addRow("Cost class:", self.cost_class_combo)
self.latency_class_combo = QComboBox()
self.latency_class_combo.addItem("Unspecified", None)
for value in ("fast", "standard", "slow"):
self.latency_class_combo.addItem(value.title(), value)
form.addRow("Latency class:", self.latency_class_combo)
self.fallback_combo = QComboBox()
self.fallback_combo.addItem("(none)", None)
# Populate with available providers
@@ -86,6 +124,24 @@ class ProviderDialog(QDialog):
self.name_input.setEnabled(False) # Can't rename
self.cmd_input.setText(provider.command)
self.desc_input.setText(provider.description or "")
self.locality_combo.setCurrentIndex(
max(self.locality_combo.findData(provider.locality), 0)
)
self.data_policy_combo.setCurrentIndex(
max(self.data_policy_combo.findData(provider.data_policy), 0)
)
self.capabilities_input.setText(", ".join(provider.capabilities))
self.model_input.setText(provider.model or "")
self.model_digest_input.setText(provider.model_digest or "")
self.max_context_input.setText(
str(provider.max_context_tokens) if provider.max_context_tokens else ""
)
self.cost_class_combo.setCurrentIndex(
max(self.cost_class_combo.findData(provider.cost_class), 0)
)
self.latency_class_combo.setCurrentIndex(
max(self.latency_class_combo.findData(provider.latency_class), 0)
)
# Set fallback selection
if provider.fallback:
idx = self.fallback_combo.findData(provider.fallback)
@@ -107,6 +163,21 @@ class ProviderDialog(QDialog):
description = self.desc_input.text().strip()
fallback = self.fallback_combo.currentData()
capabilities = [
value.strip() for value in self.capabilities_input.text().split(",")
if value.strip()
]
try:
max_context_tokens = (
int(self.max_context_input.text().strip())
if self.max_context_input.text().strip() else None
)
except ValueError:
from PySide6.QtWidgets import QMessageBox
QMessageBox.warning(
self, "Invalid", "Context tokens must be a positive integer."
)
return
# Prevent self-referential fallback
if fallback == name:
@@ -122,12 +193,21 @@ class ProviderDialog(QDialog):
description=description,
fallback=fallback,
type=existing.type if existing else "subprocess",
model=existing.model if existing else None,
model=self.model_input.text().strip() or None,
tags=existing.tags if existing else [],
install=existing.install if existing else None,
fallback_chain=existing.fallback_chain if existing else None,
api_key_env=existing.api_key_env if existing else None,
pty_config=existing.pty_config if existing else None,
tools=existing.tools if existing else None,
mcp_servers=existing.mcp_servers if existing else None,
locality=self.locality_combo.currentData(),
capabilities=capabilities,
model_digest=self.model_digest_input.text().strip() or None,
cost_class=self.cost_class_combo.currentData(),
latency_class=self.latency_class_combo.currentData(),
max_context_tokens=max_context_tokens,
data_policy=self.data_policy_combo.currentData(),
))
self.accept()
except Exception as e:
+14 -1
View File
@@ -321,6 +321,15 @@ class PromptStepDialog(QDialog):
self.strip_fences_check = QCheckBox("Strip markdown code fences from output")
form.addRow("", self.strip_fences_check)
self.allow_fallback_check = QCheckBox(
"Allow configured fallback providers if this provider fails"
)
self.allow_fallback_check.setChecked(True)
self.allow_fallback_check.setToolTip(
"Disable for privacy-sensitive or reproducible steps that must fail closed."
)
form.addRow("", self.allow_fallback_check)
# Structured output options
form.addRow(QLabel("")) # Spacer
structured_label = QLabel("<b>Structured Output</b>")
@@ -455,6 +464,7 @@ class PromptStepDialog(QDialog):
self.output_input.setText(step.output_var)
self.prompt_input.setPlainText(step.prompt)
self.strip_fences_check.setChecked(step.strip_fences)
self.allow_fallback_check.setChecked(step.fallback_policy != "deny")
# Structured output fields
self.plain_text_check.setChecked(step.plain_text)
@@ -497,7 +507,10 @@ class PromptStepDialog(QDialog):
strip_fences=self.strip_fences_check.isChecked(),
output_schema=self._output_schema,
plain_text=self.plain_text_check.isChecked(),
max_retries=self.retries_spin.value()
max_retries=self.retries_spin.value(),
fallback_policy=(
"allow" if self.allow_fallback_check.isChecked() else "deny"
),
)
+11
View File
@@ -1,5 +1,6 @@
"""Registry page - browse and install tools from registry."""
import html
from PySide6.QtWidgets import (
QWidget, QVBoxLayout, QHBoxLayout, QLineEdit,
QPushButton, QTableWidget, QTableWidgetItem, QLabel,
@@ -877,6 +878,16 @@ class RegistryPage(QWidget):
if tool.get("description"):
lines.append(f"<p style='color: #4a5568;'>{tool.get('description')}</p>")
if tool.get("deprecated"):
guidance = html.escape(
tool.get("deprecated_message") or "This tool is no longer maintained."
)
if tool.get("replacement"):
guidance += f" Use <strong>{html.escape(tool['replacement'])}</strong> instead."
lines.append(
"<p style='background: #fef3c7; color: #92400e; padding: 8px 10px; "
f"border-radius: 4px;'><strong>Deprecated:</strong> {guidance}</p>"
)
# Rating display
rating = tool.get("average_rating", 0) or 0
+480 -6
View File
@@ -1,5 +1,10 @@
"""Tool builder page - create and edit tools."""
import copy
import difflib
import re
from pathlib import Path
import yaml
from PySide6.QtWidgets import (
@@ -7,18 +12,117 @@ from PySide6.QtWidgets import (
QLineEdit, QTextEdit, QPlainTextEdit, QComboBox, QPushButton,
QGroupBox, QListWidget, QListWidgetItem, QLabel,
QMessageBox, QSplitter, QFrame, QStackedWidget,
QButtonGroup
QButtonGroup, QInputDialog
)
from PySide6.QtCore import Qt
from PySide6.QtCore import Qt, QThread, Signal, QTimer
from ...tool import (
Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
load_tool, save_tool, validate_tool_name, get_all_categories,
ensure_settings
ensure_settings, get_project_tools_dir, get_tools_dir
)
from ..widgets.icons import get_prompt_icon, get_code_icon, get_tool_icon
class SimilarToolsWorker(QThread):
"""Search the registry without blocking the tool builder."""
results_ready = Signal(str, list)
def __init__(self, query: str, parent=None):
super().__init__(parent)
self.query = query
def run(self):
try:
from ...registry_client import RegistryClient, RegistryError
try:
result = RegistryClient().search_tools(self.query, per_page=5)
except RegistryError:
result = None
self.results_ready.emit(self.query, list(result.data) if result else [])
except Exception:
# Guidance is advisory; an unavailable registry must never prevent
# local authoring.
self.results_ready.emit(self.query, [])
def build_extracted_tool(tool: Tool, opportunity: dict, name: str) -> Tool:
"""Build, but do not save, a reusable tool from an evidenced sequence."""
if opportunity.get("type") != "repeated_sequence":
raise ValueError("Only an internal repeated sequence can be extracted")
locations = opportunity.get("locations") or []
length = int(opportunity.get("length", 0))
if not locations or length < 2:
raise ValueError("Reuse evidence does not identify a valid sequence")
start = int(locations[0]) - 1
selected = tool.steps[start:start + length]
if len(selected) != length:
raise ValueError("Reuse evidence falls outside the current draft")
available = {"input", "settings"} | {
argument.variable for argument in tool.arguments
}
for step in selected:
if not isinstance(step, PromptStep):
raise ValueError("Only prompt-step sequences can be extracted safely")
references = set(re.findall(r"(?<!\{)\{([A-Za-z_]\w*)\}(?!\})", step.prompt or ""))
missing = references - available
if missing:
raise ValueError(
"The sequence depends on values produced outside its boundary: "
+ ", ".join(sorted(missing))
)
if step.output_var:
available.update(
value.strip() for value in step.output_var.split(",") if value.strip()
)
output_var = getattr(selected[-1], "output_var", "result") or "result"
output_schema = getattr(selected[-1], "output_schema", None)
return Tool(
name=name,
description=f"Extracted reusable sequence from {tool.name}",
category=tool.category,
arguments=copy.deepcopy(tool.arguments),
steps=copy.deepcopy(selected),
output=f"{{{output_var}}}",
dependencies=[],
system_dependencies=copy.deepcopy(tool.system_dependencies),
visibility=tool.visibility,
input_schema=copy.deepcopy(tool.input_schema),
output_schema=copy.deepcopy(output_schema),
)
def extraction_diff(tool: Tool) -> str:
"""Return a unified-diff preview for a proposed new tool."""
rendered = yaml.safe_dump(tool.to_dict(), sort_keys=False).splitlines(True)
return "".join(difflib.unified_diff(
[], rendered, fromfile="/dev/null", tofile=f"{tool.name}/config.yaml",
))
def _switch_to_existing_tool(main_window, name: str) -> None:
"""Remove the creation page before opening an existing tool editor."""
main_window.close_tool_builder()
main_window.open_tool_builder(name)
def default_save_scope(project_dir: Path) -> str:
"""Prefer project scope only when the cwd is recognizably a project."""
project_dir = Path(project_dir)
if any((project_dir / marker).exists() for marker in (
"cmdforge.yaml", ".cmdforge", ".git", "pyproject.toml", "package.json",
)):
return "project"
return "global"
def existing_tool_root(config_path: Path) -> Path:
"""Return the tool-root directory containing an existing named tool."""
return Path(config_path).resolve().parent.parent
class ToolBuilderPage(QWidget):
"""Tool builder/editor page."""
@@ -29,6 +133,7 @@ class ToolBuilderPage(QWidget):
self.original_name = tool_name
self._tool = None
self._flow_widget = None # Lazy-loaded
self._guidance_worker = None
self._setup_ui()
@@ -96,8 +201,63 @@ class ToolBuilderPage(QWidget):
self.category_combo.addItem(cat)
info_layout.addRow("Category:", self.category_combo)
self.save_location_combo = QComboBox()
project_root = get_project_tools_dir()
self.save_location_combo.addItem(
f"This project ({project_root})", "project"
)
self.save_location_combo.addItem(
f"My tools ({get_tools_dir()})", "global"
)
preferred_scope = default_save_scope(Path.cwd())
preferred_index = self.save_location_combo.findData(preferred_scope)
self.save_location_combo.setCurrentIndex(max(preferred_index, 0))
self.save_location_combo.setToolTip(
"Project tools are versionable and resolve only from this project; "
"My tools are available globally and receive a shell wrapper."
)
info_layout.addRow("Save tool:", self.save_location_combo)
left_layout.addWidget(info_box)
# Reuse guidance is advisory and never changes a draft automatically.
guidance_box = QGroupBox("Reuse & Discovery")
guidance_layout = QVBoxLayout(guidance_box)
self.guidance_summary = QLabel(
"CmdForge can check for similar tools, reusable step sequences, "
"and frequent local pipelines."
)
self.guidance_summary.setWordWrap(True)
guidance_layout.addWidget(self.guidance_summary)
self.guidance_list = QListWidget()
self.guidance_list.setMaximumHeight(130)
self.guidance_list.currentItemChanged.connect(
self._guidance_selection_changed
)
guidance_layout.addWidget(self.guidance_list)
guidance_buttons = QHBoxLayout()
self.btn_refresh_guidance = QPushButton("Refresh")
self.btn_refresh_guidance.clicked.connect(self._refresh_guidance)
guidance_buttons.addWidget(self.btn_refresh_guidance)
self.btn_open_guidance = QPushButton("Open Existing")
self.btn_open_guidance.setObjectName("secondary")
self.btn_open_guidance.clicked.connect(self._open_guidance_tool)
self.btn_open_guidance.setEnabled(False)
guidance_buttons.addWidget(self.btn_open_guidance)
self.btn_extract_guidance = QPushButton("Extract as New Tool")
self.btn_extract_guidance.clicked.connect(self._extract_guidance)
self.btn_extract_guidance.setEnabled(False)
guidance_buttons.addWidget(self.btn_extract_guidance)
guidance_layout.addLayout(guidance_buttons)
left_layout.addWidget(guidance_box)
self._guidance_timer = QTimer(self)
self._guidance_timer.setSingleShot(True)
self._guidance_timer.setInterval(700)
self._guidance_timer.timeout.connect(self._refresh_guidance)
self.name_input.textChanged.connect(self._schedule_guidance)
self.desc_input.textChanged.connect(self._schedule_guidance)
# Arguments group
args_box = QGroupBox()
args_layout = QVBoxLayout(args_box)
@@ -398,6 +558,176 @@ class ToolBuilderPage(QWidget):
layout.addWidget(splitter, 1)
def _current_draft(self) -> Tool:
"""Materialize the current form without saving it."""
current = self._tool
return Tool(
name=self.name_input.text().strip() or "untitled-tool",
description=self.desc_input.text().strip(),
category=self.category_combo.currentText() or "Other",
arguments=copy.deepcopy(current.arguments) if current else [],
steps=copy.deepcopy(current.steps) if current else [],
output=self.output_input.toPlainText().strip() or "{response}",
dependencies=list(current.dependencies) if current else [],
system_dependencies=(
copy.deepcopy(current.system_dependencies) if current else []
),
visibility=current.visibility if current else "public",
input_schema=copy.deepcopy(current.input_schema) if current else None,
output_schema=copy.deepcopy(current.output_schema) if current else None,
)
def _schedule_guidance(self):
self._guidance_timer.start()
def _add_guidance_item(self, text: str, data: dict) -> None:
item = QListWidgetItem(text)
item.setData(Qt.UserRole, data)
self.guidance_list.addItem(item)
def _refresh_guidance(self):
"""Refresh deterministic local guidance, then search remotely."""
from ...preflight import analyze_tool
from ...usage import get_suggestions
draft = self._current_draft()
self.guidance_list.clear()
report = analyze_tool(
draft,
check_local_dependencies=False,
include_contract_tests=False,
check_reuse=True,
)
for opportunity in report.reuse_opportunities:
self._add_guidance_item(
f"Reusable: {opportunity.get('detail', 'exact sequence')}",
opportunity,
)
for suggestion in get_suggestions():
tools = suggestion["tools"]
self._add_guidance_item(
f"Frequent pipeline ({suggestion['count']}x): " + " | ".join(tools),
{"type": "usage_pipeline", **suggestion},
)
query = (draft.description or draft.name).strip()
if not query or query == "untitled-tool":
self._finish_guidance(query, [])
return
if self._guidance_worker and self._guidance_worker.isRunning():
# The pending result is ignored if the query has since changed.
self.guidance_summary.setText("Checking registry…")
return
self.guidance_summary.setText("Checking registry…")
self.btn_refresh_guidance.setEnabled(False)
self._guidance_worker = SimilarToolsWorker(query, self)
self._guidance_worker.results_ready.connect(self._finish_guidance)
self._guidance_worker.start()
def _finish_guidance(self, query: str, results: list):
current_query = (
self.desc_input.text().strip() or self.name_input.text().strip()
)
if query == current_query:
seen = set()
for result in results:
name = result.get("name", "")
owner = result.get("owner", "")
reference = f"{owner}/{name}" if owner else name
if reference and reference not in seen:
seen.add(reference)
score = result.get("quality_score")
quality = f" [{score}/100]" if score is not None else ""
self._add_guidance_item(
f"Registry: {reference}{quality} — {result.get('description', '')}",
{"type": "registry_similar", "tool": reference},
)
self.btn_refresh_guidance.setEnabled(True)
count = self.guidance_list.count()
self.guidance_summary.setText(
f"{count} advisory suggestion{'s' if count != 1 else ''}. "
"Nothing changes until you confirm an action."
if count else "No evidenced reuse opportunities found."
)
self._guidance_selection_changed(self.guidance_list.currentItem())
if query != current_query:
self._guidance_timer.start(0)
def _guidance_selection_changed(self, current, previous=None):
data = current.data(Qt.UserRole) if current else {}
kind = data.get("type")
local_name = data.get("tool", "")
self.btn_open_guidance.setEnabled(
kind == "duplicate_sequence" and bool(local_name)
)
self.btn_extract_guidance.setEnabled(
kind in {"repeated_sequence", "usage_pipeline"}
)
def _open_guidance_tool(self):
item = self.guidance_list.currentItem()
data = item.data(Qt.UserRole) if item else {}
if data.get("type") == "duplicate_sequence" and data.get("tool"):
_switch_to_existing_tool(self.main_window, data["tool"])
def _extract_guidance(self):
"""Preview and explicitly save an evidenced reusable tool."""
item = self.guidance_list.currentItem()
data = item.data(Qt.UserRole) if item else {}
kind = data.get("type")
if kind not in {"repeated_sequence", "usage_pipeline"}:
return
suggested = (
"-".join(data.get("tools", []))
if kind == "usage_pipeline" else f"{self.name_input.text().strip()}-shared"
)
name, accepted = QInputDialog.getText(
self, "Extract Reusable Tool", "New tool name:", text=suggested
)
name = name.strip()
if not accepted or not name:
return
valid, error = validate_tool_name(name)
if not valid:
QMessageBox.warning(self, "Validation", error)
return
extraction_root = self._selected_tools_dir()
if (extraction_root / name / "config.yaml").exists():
QMessageBox.warning(self, "Tool Already Exists", f"'{name}' already exists.")
return
try:
if kind == "usage_pipeline":
from ...usage import build_composite_tool
extracted = build_composite_tool(name, data["tools"])
else:
extracted = build_extracted_tool(self._current_draft(), data, name)
except (KeyError, TypeError, ValueError) as exc:
QMessageBox.warning(self, "Cannot Extract", str(exc))
return
preview = QMessageBox(self)
preview.setIcon(QMessageBox.Information)
preview.setWindowTitle("Review Extraction")
preview.setText(
f"Create '{name}' as a separate tool? Your current draft will not be modified."
)
preview.setDetailedText(extraction_diff(extracted))
create_button = preview.addButton("Create Tool", QMessageBox.AcceptRole)
preview.addButton(QMessageBox.Cancel)
preview.exec()
if preview.clickedButton() is not create_button:
return
try:
save_tool(extracted, tools_dir=extraction_root)
except (OSError, ValueError) as exc:
QMessageBox.critical(self, "Extraction Failed", str(exc))
return
QMessageBox.information(
self, "Tool Created", f"Created '{name}'. The current draft is unchanged."
)
self._populate_deps_combo()
self._refresh_guidance()
def _set_view_mode(self, mode: int):
"""Switch between list (0) and flow (1) views."""
if mode == 1 and self._flow_widget is None:
@@ -511,6 +841,17 @@ class ToolBuilderPage(QWidget):
self._tool = tool
self.name_input.setText(tool.name)
self.name_input.setEnabled(False) # Can't rename
if tool.path:
project_root = get_project_tools_dir()
scope = (
"project"
if existing_tool_root(tool.path) == project_root.resolve()
else "global"
)
index = self.save_location_combo.findData(scope)
if index >= 0:
self.save_location_combo.setCurrentIndex(index)
self.save_location_combo.setEnabled(False)
self.desc_input.setText(tool.description or "")
# Set category
@@ -1075,6 +1416,44 @@ class ToolBuilderPage(QWidget):
QMessageBox.warning(self, "Validation", error)
return
target_root = self._selected_tools_dir()
# Block silent overwrite in the selected scope. A project-local tool
# may intentionally shadow a global tool with the same name.
target_config = target_root / name / "config.yaml"
if (not self.editing or name != self.original_name) and target_config.exists():
existing = load_tool(name) if self.save_location_combo.currentData() == "global" else None
existing_desc = f" ({existing.description})" if existing and existing.description else ""
msg = QMessageBox(self)
msg.setIcon(QMessageBox.Warning)
msg.setWindowTitle("Tool Already Exists")
msg.setText(f"'{name}' already exists.{existing_desc}")
msg.setInformativeText("What would you like to do?")
btn_open = msg.addButton("Open Existing", QMessageBox.ActionRole)
btn_copy = msg.addButton("Create Copy", QMessageBox.ActionRole)
btn_rename = msg.addButton("Choose Another Name", QMessageBox.ActionRole)
msg.setStandardButtons(QMessageBox.Cancel)
msg.setDefaultButton(btn_rename)
msg.exec()
clicked = msg.clickedButton()
if clicked is btn_open:
_switch_to_existing_tool(self.main_window, name)
return
elif clicked is btn_copy:
suffix = 2
while (target_root / f"{name}-{suffix}" / "config.yaml").exists():
suffix += 1
copy_name = f"{name}-{suffix}"
self.name_input.setText(copy_name)
self.name_input.setFocus()
return
elif clicked is btn_rename:
self.name_input.setFocus()
return
else:
return
description = self.desc_input.text().strip()
category = self.category_combo.currentText()
output = self.output_input.toPlainText().strip() or "{response}"
@@ -1088,8 +1467,92 @@ class ToolBuilderPage(QWidget):
steps=self._tool.steps if self._tool else [],
output=output,
dependencies=self._tool.dependencies if self._tool else [],
system_dependencies=self._tool.system_dependencies if self._tool else []
system_dependencies=self._tool.system_dependencies if self._tool else [],
visibility=self._tool.visibility if self._tool else "public",
input_schema=self._tool.input_schema if self._tool else None,
output_schema=self._tool.output_schema if self._tool else None,
)
if self._tool and self._tool.path:
tool.path = self._tool.path
from ...preflight import analyze_tool
preflight = analyze_tool(tool, include_contract_tests=True)
if preflight.errors:
QMessageBox.warning(
self,
"Preflight Failed",
"\n".join(preflight.errors),
)
return
proposal = preflight.contract_proposal or {}
proposed_schemas = {
key: proposal.get(key)
for key in ("input_schema", "output_schema")
if proposal.get(key) is not None
}
if proposed_schemas:
msg = QMessageBox(self)
msg.setIcon(QMessageBox.Information)
msg.setWindowTitle("Contract Proposal")
msg.setText(
"CmdForge inferred a possible tool contract. "
"It will not be applied automatically."
)
msg.setInformativeText(
"Review the proposal below. Applying it changes only this "
"in-memory draft and remains reversible until you save."
)
msg.setDetailedText(
yaml.safe_dump(proposed_schemas, sort_keys=False).rstrip()
)
apply_button = msg.addButton(
"Apply Proposal", QMessageBox.AcceptRole
)
save_button = msg.addButton(
"Save Without Applying", QMessageBox.AcceptRole
)
msg.addButton(QMessageBox.Cancel)
msg.exec()
if msg.clickedButton() is apply_button:
if proposed_schemas.get("input_schema") is not None:
tool.input_schema = proposed_schemas["input_schema"]
if proposed_schemas.get("output_schema") is not None:
tool.output_schema = proposed_schemas["output_schema"]
elif msg.clickedButton() is not save_button:
return
if preflight.reuse_opportunities:
reuse = QMessageBox(self)
reuse.setIcon(QMessageBox.Information)
reuse.setWindowTitle("Reuse Opportunities")
reuse.setText(
"CmdForge found exact, contract-backed reuse evidence. "
"This is advisory and will not change your draft."
)
reuse.setDetailedText("\n".join(
f"- {item['detail']}\n Evidence: {item.get('evidence', 'exact match')}"
for item in preflight.reuse_opportunities
))
continue_button = reuse.addButton(
"Continue With Draft", QMessageBox.AcceptRole
)
matching = next(
(item.get("tool") for item in preflight.reuse_opportunities if item.get("tool")),
None,
)
open_button = (
reuse.addButton("Open Existing", QMessageBox.ActionRole)
if matching else None
)
reuse.addButton(QMessageBox.Cancel)
reuse.exec()
if open_button and reuse.clickedButton() is open_button:
_switch_to_existing_tool(self.main_window, matching)
return
if reuse.clickedButton() is not continue_button:
return
advisory_count = len(preflight.warnings) + len(preflight.suggestions)
# Preserve source if editing
if self._tool and self._tool.source:
@@ -1100,7 +1563,7 @@ class ToolBuilderPage(QWidget):
tool.version = self._tool.version
try:
config_path = save_tool(tool)
config_path = save_tool(tool, tools_dir=target_root)
tool_dir = config_path.parent
# Save defaults if provided
@@ -1125,7 +1588,10 @@ class ToolBuilderPage(QWidget):
if settings_path.exists():
settings_path.unlink()
self.main_window.show_status(f"Saved tool '{name}'")
status = f"Saved tool '{name}'"
if advisory_count:
status += f" with {advisory_count} preflight suggestion(s)"
self.main_window.show_status(status)
self.main_window.close_tool_builder()
except Exception as e:
QMessageBox.critical(self, "Error", f"Failed to save tool:\n{e}")
@@ -1134,6 +1600,14 @@ class ToolBuilderPage(QWidget):
"""Cancel and return to tools page."""
self.main_window.close_tool_builder()
def _selected_tools_dir(self) -> Path:
"""Return the save root selected by the author."""
if self.editing and self._tool and self._tool.path:
return existing_tool_root(self._tool.path)
if self.save_location_combo.currentData() == "project":
return get_project_tools_dir()
return get_tools_dir().resolve()
def save_tool(self):
"""Public method for keyboard shortcut to save the tool."""
self._save()
+11
View File
@@ -1,5 +1,6 @@
"""Tools page - main view for managing tools."""
import html
from collections import defaultdict
from pathlib import Path
from typing import Optional, Tuple, Dict, Any
@@ -852,6 +853,16 @@ class ToolsPage(QWidget):
lines.append(f"<h2 style='margin: 0 0 8px 0; color: #2d3748;'>{tool.name}</h2>")
if tool.description:
lines.append(f"<p style='color: #4a5568; margin-bottom: 16px;'>{tool.description}</p>")
if tool.deprecated:
guidance = html.escape(
tool.deprecated_message or "This tool is no longer maintained."
)
if tool.replacement:
guidance += f" Use <strong>{html.escape(tool.replacement)}</strong> instead."
lines.append(
"<p style='background: #fef3c7; color: #92400e; padding: 8px 10px; "
f"border-radius: 4px;'><strong>Deprecated:</strong> {guidance}</p>"
)
# Publish state
state, registry_hash = get_tool_publish_state(qname)
+243
View File
@@ -0,0 +1,243 @@
"""Scrutiny-driven improvement pipeline (M9.2).
Takes a tool with low scrutiny scores and generates actionable improvement
suggestions for honesty, efficiency, and transparency.
"""
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
from .tool import Tool, PromptStep, CodeStep
@dataclass
class ImprovementSuggestion:
"""One actionable improvement suggestion."""
category: str # "honesty" | "efficiency" | "transparency"
severity: str # "high" | "medium" | "low"
title: str
description: str
location: str = "" # step index or "tool"
current: str = "" # current code/prompt text
suggested: str = "" # suggested replacement
auto_applicable: bool = False
def to_dict(self) -> dict:
return {
"category": self.category,
"severity": self.severity,
"title": self.title,
"description": self.description,
"location": self.location,
"current": self.current[:200],
"suggested": self.suggested[:200],
"auto_applicable": self.auto_applicable,
}
@dataclass
class ImprovementReport:
"""Collection of improvement suggestions for a tool."""
tool_name: str
suggestions: List[ImprovementSuggestion] = field(default_factory=list)
@property
def has_high_severity(self) -> bool:
return any(s.severity == "high" for s in self.suggestions)
@property
def by_category(self) -> Dict[str, List[ImprovementSuggestion]]:
result: Dict[str, List[ImprovementSuggestion]] = {}
for s in self.suggestions:
result.setdefault(s.category, []).append(s)
return result
def to_dict(self) -> dict:
return {
"tool": self.tool_name,
"suggestion_count": len(self.suggestions),
"has_high_severity": self.has_high_severity,
"suggestions": [s.to_dict() for s in self.suggestions],
}
def generate_improvements(
tool: Tool, scrutiny_report: Optional[dict] = None
) -> ImprovementReport:
"""Analyze a tool and generate improvement suggestions.
Categories:
- Honesty: description doesn't match behavior
- Efficiency: unnecessary AI calls, redundant steps
- Transparency: obfuscated code, unclear prompts
"""
report = ImprovementReport(tool_name=tool.name)
_check_honesty(tool, report)
_check_efficiency(tool, report)
_check_transparency(tool, report)
# Preserve the registry scrutiny engine's evidence rather than running a
# disconnected approximation. Only actionable warnings/failures become
# suggestions; passing checks are deliberately omitted.
for finding in (scrutiny_report or {}).get("findings", []):
result = finding.get("result")
if result not in ("warning", "fail"):
continue
check = str(finding.get("check", "scrutiny")).lower()
category = next(
(value for value in ("honesty", "efficiency", "transparency")
if value in check),
"transparency",
)
suggestion = finding.get("suggestion") or (
"Review this finding and make the behavior explicit."
)
candidate = ImprovementSuggestion(
category=category,
severity="high" if result == "fail" else "medium",
title=f"Scrutiny: {finding.get('check', 'review required')}",
description=finding.get("message", "Scrutiny identified an issue."),
location=finding.get("location") or "tool",
suggested=suggestion,
)
marker = (candidate.category, candidate.title, candidate.location)
existing = {(s.category, s.title, s.location) for s in report.suggestions}
if marker not in existing:
report.suggestions.append(candidate)
return report
def _check_honesty(tool: Tool, report: ImprovementReport):
"""Check description-to-behavior alignment."""
if not tool.description:
report.suggestions.append(ImprovementSuggestion(
category="honesty",
severity="high",
title="Missing description",
description="Tool has no description. Users cannot know what it does without reading the config.",
location="tool",
auto_applicable=False,
))
return
# Check if description mentions capabilities not in steps
desc_lower = tool.description.lower()
has_prompt = any(isinstance(s, PromptStep) for s in tool.steps)
has_code = any(isinstance(s, CodeStep) for s in tool.steps)
if "ai" in desc_lower or "generate" in desc_lower or "analyze" in desc_lower:
if not has_prompt:
report.suggestions.append(ImprovementSuggestion(
category="honesty",
severity="medium",
title="Description implies AI but no prompt step found",
description="The description mentions AI/generation/analysis but the tool has no PromptStep.",
location="tool",
auto_applicable=False,
))
# Check if description is too vague
if len(tool.description) < 20:
report.suggestions.append(ImprovementSuggestion(
category="honesty",
severity="low",
title="Description is very short",
description="A more detailed description helps users find and trust this tool.",
location="tool",
auto_applicable=False,
))
def _check_efficiency(tool: Tool, report: ImprovementReport):
"""Check for unnecessary AI calls and redundant steps."""
prompt_steps = [s for s in tool.steps if isinstance(s, PromptStep)]
# Multiple prompt steps with same provider — could be combined
if len(prompt_steps) > 1:
providers = [s.provider for s in prompt_steps]
if len(set(providers)) == 1:
report.suggestions.append(ImprovementSuggestion(
category="efficiency",
severity="medium",
title="Multiple prompt steps with same provider",
description=f"Found {len(prompt_steps)} prompt steps all using '{providers[0]}'. "
"Consider combining them into a single step to reduce API calls.",
location=f"steps 0-{len(prompt_steps)-1}",
auto_applicable=False,
))
# Prompt step that just passes input through
for i, step in enumerate(tool.steps):
if isinstance(step, PromptStep):
if step.prompt.strip() in ("{input}", "Repeat: {input}", "Echo: {input}"):
report.suggestions.append(ImprovementSuggestion(
category="efficiency",
severity="high",
title="Prompt step just echoes input",
description="This prompt step appears to just pass input through without transformation. "
"Remove it and use the input directly.",
location=f"step {i}",
current=step.prompt,
suggested="(remove this step — use {input} directly)",
auto_applicable=False,
))
# No steps at all — output is just a template
if not tool.steps and "{input}" in tool.output:
report.suggestions.append(ImprovementSuggestion(
category="efficiency",
severity="low",
title="Tool has no steps",
description="This tool only does variable substitution. Consider if it needs to be a CmdForge tool "
"or if a simple shell alias would suffice.",
location="tool",
auto_applicable=False,
))
def _check_transparency(tool: Tool, report: ImprovementReport):
"""Check for obfuscated code and unclear prompts."""
for i, step in enumerate(tool.steps):
if isinstance(step, CodeStep):
# Check for overly complex code
lines = step.code.strip().split("\n")
if len(lines) > 20:
report.suggestions.append(ImprovementSuggestion(
category="transparency",
severity="medium",
title=f"Code step {i} is complex ({len(lines)} lines)",
description="Long code steps are harder to audit. Consider extracting into a separate "
"tool or adding comments.",
location=f"step {i}",
auto_applicable=False,
))
# Check for exec/eval
if "exec(" in step.code or "eval(" in step.code:
report.suggestions.append(ImprovementSuggestion(
category="transparency",
severity="high",
title=f"Code step {i} uses exec/eval",
description="Using exec() or eval() in code steps makes the tool harder to audit and "
"poses security risks. Consider restructuring.",
location=f"step {i}",
auto_applicable=False,
))
if isinstance(step, PromptStep):
# Check for very short prompts
if len(step.prompt.strip()) < 20:
report.suggestions.append(ImprovementSuggestion(
category="transparency",
severity="medium",
title=f"Prompt step {i} is very short",
description="Short prompts give the AI model less context, leading to unpredictable output. "
"Add more detail about the expected format and content.",
location=f"step {i}",
current=step.prompt,
auto_applicable=False,
))
+180
View File
@@ -0,0 +1,180 @@
"""Content-addressable, transitive integrity verification for CmdForge tools."""
from __future__ import annotations
import hashlib
import json
from dataclasses import dataclass, field
from typing import Dict, List, Optional, Set
from .tool import Tool, ToolStep, load_tool
def _sha256(value: object) -> str:
encoded = json.dumps(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
).encode("utf-8")
return f"sha256:{hashlib.sha256(encoded).hexdigest()}"
@dataclass
class IntegrityNode:
name: str
version: str = ""
definition_hash: str = ""
content_hash: str = ""
dependencies: List[str] = field(default_factory=list)
dependency_hashes: Dict[str, str] = field(default_factory=dict)
def to_dict(self) -> dict:
return {
"name": self.name,
"version": self.version,
"definition_hash": self.definition_hash,
"hash": self.content_hash,
"dependencies": self.dependencies,
"dependency_hashes": self.dependency_hashes,
}
@dataclass
class IntegrityChain:
root: IntegrityNode
nodes: Dict[str, IntegrityNode] = field(default_factory=dict)
errors: List[str] = field(default_factory=list)
@property
def is_valid(self) -> bool:
if self.errors or self.root.name not in self.nodes:
return False
for name, node in self.nodes.items():
if not node.definition_hash or not node.content_hash:
return False
if sorted(node.dependencies) != sorted(node.dependency_hashes):
return False
for dep_name in node.dependencies:
dep = self.nodes.get(dep_name)
if dep is None or node.dependency_hashes[dep_name] != dep.content_hash:
return False
expected = _identity_hash(node.definition_hash, node.dependency_hashes)
if not _constant_time_equal(expected, node.content_hash):
return False
return True
def to_dict(self) -> dict:
return {
"root": self.root.to_dict(),
"nodes": {key: value.to_dict() for key, value in self.nodes.items()},
"errors": list(self.errors),
"valid": self.is_valid,
}
def _constant_time_equal(left: str, right: str) -> bool:
import hmac
return hmac.compare_digest(left, right)
def compute_tool_hash(tool: Tool) -> str:
"""Return the full SHA-256 hash of the canonical tool definition."""
data = tool.to_dict()
data.pop("path", None)
# Registry bookkeeping is not executable tool content.
for key in ("registry_hash", "registry_status", "registry_feedback"):
data.pop(key, None)
return _sha256(data)
def _identity_hash(definition_hash: str, dependency_hashes: Dict[str, str]) -> str:
"""Hash a definition together with the identities of every direct dep."""
return _sha256({
"definition": definition_hash,
"dependencies": dict(sorted(dependency_hashes.items())),
})
def compute_content_identity(
definition_hash: str, dependency_hashes: Dict[str, str]
) -> str:
"""Return the canonical identity used by integrity chains and lockfiles."""
return _identity_hash(definition_hash, dependency_hashes)
def _dependency_names(tool: Tool) -> List[str]:
names = list(tool.dependencies)
names.extend(step.tool for step in tool.steps if isinstance(step, ToolStep))
return sorted(set(names))
def build_integrity_chain(tool: Tool, max_depth: int = 50) -> IntegrityChain:
"""Resolve dependencies bottom-up and compute a Merkle-style identity."""
placeholder = IntegrityNode(tool.name, tool.version or "")
chain = IntegrityChain(root=placeholder)
visiting: Set[str] = set()
def visit(current: Tool, depth: int) -> Optional[IntegrityNode]:
name = current.name
if name in visiting:
chain.errors.append(f"Dependency cycle detected at {name}")
return None
if name in chain.nodes:
return chain.nodes[name]
if depth > max_depth:
chain.errors.append(f"Maximum dependency depth exceeded at {name}")
return None
visiting.add(name)
dep_hashes: Dict[str, str] = {}
dependencies = _dependency_names(current)
for dep_name in dependencies:
from .resolver import ToolSpec
from .semver import matches_constraint
dep_spec = ToolSpec.parse(dep_name)
dep_tool = load_tool(dep_spec.full_name)
if dep_tool is None:
chain.errors.append(f"Unresolved dependency: {dep_name}")
continue
if dep_spec.version and not matches_constraint(
dep_tool.version or "", dep_spec.version
):
chain.errors.append(
f"Dependency version mismatch: {dep_name} "
f"resolved to {dep_tool.version or 'unknown'}"
)
continue
dep_node = visit(dep_tool, depth + 1)
if dep_node is not None:
dep_hashes[dep_name] = dep_node.content_hash
visiting.remove(name)
definition_hash = compute_tool_hash(current)
node = IntegrityNode(
name=name,
version=current.version or "",
definition_hash=definition_hash,
content_hash=_identity_hash(definition_hash, dep_hashes),
dependencies=dependencies,
dependency_hashes=dep_hashes,
)
chain.nodes[name] = node
return node
root = visit(tool, 0)
if root is not None:
chain.root = root
return chain
def verify_integrity(chain: IntegrityChain, *, verify_installed: bool = False) -> bool:
"""Verify chain structure, optionally rebuilding it from installed tools."""
if not chain.is_valid:
return False
if not verify_installed:
return True
root_tool = load_tool(chain.root.name)
if root_tool is None:
return False
rebuilt = build_integrity_chain(root_tool)
return rebuilt.is_valid and _constant_time_equal(
rebuilt.root.content_hash, chain.root.content_hash
)
+63 -1
View File
@@ -26,6 +26,8 @@ class LockedPackage:
direct: bool # True if in manifest
required_by: List[str] = field(default_factory=list) # Parent packages
path: Optional[str] = None # Relative path for local tools
content_hash: str = "" # Definition plus dependency identities
dependency_hashes: Dict[str, str] = field(default_factory=dict)
@property
def owner(self) -> str:
@@ -145,7 +147,9 @@ class Lockfile:
source=pkg_data.get("source", "registry"),
direct=pkg_data.get("direct", False),
required_by=pkg_data.get("required_by", []),
path=pkg_data.get("path")
path=pkg_data.get("path"),
content_hash=pkg_data.get("content_hash", ""),
dependency_hashes=pkg_data.get("dependency_hashes", {})
)
return cls(
@@ -180,6 +184,10 @@ class Lockfile:
pkg_dict["required_by"] = pkg.required_by
if pkg.path:
pkg_dict["path"] = pkg.path
if pkg.content_hash:
pkg_dict["content_hash"] = pkg.content_hash
if pkg.dependency_hashes:
pkg_dict["dependency_hashes"] = dict(sorted(pkg.dependency_hashes.items()))
d["packages"][name] = pkg_dict
return d
@@ -259,6 +267,34 @@ def generate_lockfile(
)
lock.packages[qualified_name] = pkg
from .integrity import compute_content_identity
visiting = set()
def identity_for(package_name: str) -> str:
pkg = lock.packages.get(package_name)
node = graph.nodes.get(package_name)
if pkg is None or node is None or not pkg.integrity:
return ""
if pkg.content_hash:
return pkg.content_hash
if package_name in visiting:
return ""
visiting.add(package_name)
dependencies = {}
for child_name in sorted(node.children):
child_hash = identity_for(child_name)
if not child_hash:
visiting.remove(package_name)
return ""
dependencies[child_name] = child_hash
visiting.remove(package_name)
pkg.dependency_hashes = dependencies
pkg.content_hash = compute_content_identity(pkg.integrity, dependencies)
return pkg.content_hash
for package_name in sorted(lock.packages):
identity_for(package_name)
return lock
@@ -368,4 +404,30 @@ def verify_lockfile(
except Exception as e:
errors.append(f"{name}: could not verify integrity ({e})")
if locked.content_hash:
from .integrity import compute_content_identity
actual_dependencies = {}
missing_dependencies = []
for dep_name, expected_hash in locked.dependency_hashes.items():
dep = lock.packages.get(dep_name)
if dep is None or not dep.content_hash:
missing_dependencies.append(dep_name)
elif dep.content_hash != expected_hash:
errors.append(
f"{name}: dependency identity mismatch ({dep_name})"
)
else:
actual_dependencies[dep_name] = dep.content_hash
if missing_dependencies:
errors.append(
f"{name}: unresolved integrity dependencies: "
+ ", ".join(missing_dependencies)
)
elif locked.integrity:
actual_identity = compute_content_identity(
locked.integrity, actual_dependencies
)
if actual_identity != locked.content_hash:
errors.append(f"{name}: transitive content identity mismatch")
return errors
+668
View File
@@ -0,0 +1,668 @@
"""MCP client support built on the official MCP Python SDK."""
import asyncio
import contextvars
import fnmatch
import hashlib
import json
import os
import re
import tempfile
import time
from dataclasses import dataclass, field
from datetime import timedelta
from pathlib import Path
from typing import Any, Awaitable, Callable, Dict, List, Optional, TypeVar
from urllib.parse import urlparse
import yaml
MCP_CONFIG_FILE = Path.home() / ".cmdforge" / "mcp.yaml"
MCP_CONFIG_VERSION = 1
MCP_DEPTH_ENV = "CMDFORGE_MCP_DEPTH"
RESULT_MODES = ("auto", "structured", "content", "text")
SUPPORTED_TRANSPORTS = ("stdio", "streamable-http")
DEFAULT_INHERITED_ENV = (
"PATH",
"HOME",
"USER",
"LOGNAME",
"TMPDIR",
"TMP",
"TEMP",
"SYSTEMROOT",
"WINDIR",
"COMSPEC",
"PATHEXT",
)
_ENV_REFERENCE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
_ENV_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
_SECRET_PATTERNS = (
re.compile(r"(?i)bearer\s+\S+"),
re.compile(r"(?i)\b(?:sk|ghp|github_pat)-[A-Za-z0-9_-]+"),
re.compile(r"(?i)(?:api[_-]?key|token|secret|password)\s*[:=]\s*\S+"),
)
T = TypeVar("T")
_MCP_CALL_DEPTH: contextvars.ContextVar[int] = contextvars.ContextVar(
"cmdforge_mcp_call_depth", default=0
)
@dataclass
class McpServeConfig:
"""Server-mode configuration: which CmdForge tools to expose via MCP."""
expose: List[str] = field(default_factory=list)
deny: List[str] = field(default_factory=list)
def __post_init__(self) -> None:
for field_name, patterns in (("expose", self.expose), ("deny", self.deny)):
if not isinstance(patterns, list) or not all(
isinstance(pattern, str) and pattern for pattern in patterns
):
raise ValueError(
f"MCP server '{field_name}' must be a list of non-empty strings"
)
def is_exposed(self, tool_name: str) -> bool:
"""Check whether a tool should be exposed to MCP clients."""
if self._matches_deny(tool_name):
return False
if not self.expose:
return False
return self._matches_expose(tool_name)
def _matches_deny(self, name: str) -> bool:
for pattern in self.deny:
if _glob_match(pattern, name):
return True
return False
def _matches_expose(self, name: str) -> bool:
for pattern in self.expose:
if _glob_match(pattern, name) or pattern == "*":
return True
return False
def _glob_match(pattern: str, name: str) -> bool:
"""Match a policy pattern using shell-style wildcards."""
return fnmatch.fnmatchcase(name, pattern)
@dataclass
class McpServerConfig:
"""Configuration for one MCP server.
``approved`` records explicit user consent to execute a local command or
connect to the configured HTTP endpoint.
"""
name: str
transport: str = "stdio"
command: Optional[str] = None
url: Optional[str] = None
headers: Dict[str, str] = field(default_factory=dict)
args: List[str] = field(default_factory=list)
cwd: Optional[str] = None
env: Dict[str, str] = field(default_factory=dict)
inherit_env: List[str] = field(default_factory=lambda: list(DEFAULT_INHERITED_ENV))
timeout: float = 30
description: str = ""
approved: bool = False
def validate(self) -> None:
if not isinstance(self.name, str) or not self.name.strip():
raise ValueError("MCP server name must be a non-empty string")
if self.transport not in SUPPORTED_TRANSPORTS:
raise ValueError(
f"Unsupported MCP transport '{self.transport}'. "
f"Supported transports: {', '.join(SUPPORTED_TRANSPORTS)}"
)
if self.transport == "stdio":
if not isinstance(self.command, str) or not self.command.strip():
raise ValueError(f"MCP server '{self.name}' requires a command")
if self.url is not None:
raise ValueError(f"MCP stdio server '{self.name}' cannot define url")
if self.headers:
raise ValueError(f"MCP stdio server '{self.name}' cannot define headers")
else:
if self.command is not None:
raise ValueError(
f"MCP streamable-http server '{self.name}' cannot define command"
)
_validate_remote_url(self.name, self.url)
if self.args or self.cwd or self.env:
raise ValueError(
f"MCP streamable-http server '{self.name}' cannot define args, cwd, or env"
)
if not isinstance(self.args, list) or not all(isinstance(arg, str) for arg in self.args):
raise ValueError(f"MCP server '{self.name}' args must be a list of strings")
if not isinstance(self.env, dict) or not all(
isinstance(key, str)
and _ENV_NAME.fullmatch(key)
and isinstance(value, str)
for key, value in self.env.items()
):
raise ValueError(
f"MCP server '{self.name}' env must map valid variable names to strings"
)
if not isinstance(self.headers, dict) or not all(
isinstance(key, str) and key.strip()
and "\n" not in key and "\r" not in key
and isinstance(value, str) and "\n" not in value and "\r" not in value
for key, value in self.headers.items()
):
raise ValueError(
f"MCP server '{self.name}' headers must map safe names to strings"
)
if not isinstance(self.inherit_env, list) or not all(
isinstance(name, str) and _ENV_NAME.fullmatch(name)
for name in self.inherit_env
):
raise ValueError(
f"MCP server '{self.name}' inherit_env must contain valid variable names"
)
if isinstance(self.timeout, bool) or not isinstance(self.timeout, (int, float)):
raise ValueError(f"MCP server '{self.name}' timeout must be a number")
if self.timeout <= 0 or self.timeout > 3600:
raise ValueError(
f"MCP server '{self.name}' timeout must be greater than 0 and at most 3600 seconds"
)
if self.cwd is not None and not isinstance(self.cwd, str):
raise ValueError(f"MCP server '{self.name}' cwd must be a string")
if not isinstance(self.approved, bool):
raise ValueError(f"MCP server '{self.name}' approved must be a boolean")
def _fingerprint(cfg: McpServerConfig) -> str:
payload = json.dumps(
[
cfg.transport,
cfg.command,
cfg.url,
cfg.headers,
cfg.args,
cfg.cwd,
cfg.env,
cfg.inherit_env,
cfg.timeout,
cfg.approved,
],
sort_keys=True,
)
return hashlib.sha256(payload.encode()).hexdigest()[:16]
def _sanitize(value: str, secrets: Optional[List[str]] = None) -> str:
"""Redact common credential forms and explicit resolved secret values."""
sanitized = str(value)
for secret in secrets or []:
if secret:
sanitized = sanitized.replace(secret, "[redacted]")
for pattern in _SECRET_PATTERNS:
sanitized = pattern.sub("[redacted]", sanitized)
return sanitized
def _expand_env_value(value: str) -> str:
def replace(match: re.Match) -> str:
name = match.group(1)
if name not in os.environ:
raise ValueError(f"Environment variable '{name}' referenced by MCP config is not set")
return os.environ[name]
return _ENV_REFERENCE.sub(replace, value)
def _build_server_env(cfg: McpServerConfig) -> Dict[str, str]:
"""Build a minimal subprocess environment without leaking all credentials."""
environment = {
name: os.environ[name]
for name in cfg.inherit_env
if name in os.environ
}
for name, value in cfg.env.items():
environment[name] = _expand_env_value(value)
call_depth = _MCP_CALL_DEPTH.get()
if call_depth > 0:
environment[MCP_DEPTH_ENV] = str(call_depth)
return environment
def _is_loopback_host(hostname: Optional[str]) -> bool:
if not hostname:
return False
if hostname.lower() == "localhost":
return True
try:
import ipaddress
return ipaddress.ip_address(hostname).is_loopback
except ValueError:
return False
def _validate_remote_url(name: str, url: Optional[str]) -> None:
if not isinstance(url, str) or not url.strip():
raise ValueError(f"MCP streamable-http server '{name}' requires a url")
parsed = urlparse(url)
if parsed.scheme not in ("http", "https") or not parsed.hostname:
raise ValueError(f"MCP server '{name}' has an invalid HTTP URL")
if parsed.username or parsed.password:
raise ValueError("MCP URLs must not contain credentials; use headers")
if parsed.scheme != "https" and not _is_loopback_host(parsed.hostname):
raise ValueError(
f"Remote MCP server '{name}' must use HTTPS; HTTP is allowed only for localhost"
)
def _build_http_headers(cfg: McpServerConfig) -> Dict[str, str]:
return {name: _expand_env_value(value) for name, value in cfg.headers.items()}
def _load_mcp_document() -> dict:
if not MCP_CONFIG_FILE.exists():
return {}
data = yaml.safe_load(MCP_CONFIG_FILE.read_text()) or {}
if not isinstance(data, dict):
raise ValueError("MCP config must be a YAML mapping")
version = data.get("version", MCP_CONFIG_VERSION)
if version != MCP_CONFIG_VERSION:
raise ValueError(f"Unsupported MCP config version: {version}")
return data
def load_mcp_config() -> List[McpServerConfig]:
data = _load_mcp_document()
raw_servers = data.get("servers") or {}
if not isinstance(raw_servers, dict):
raise ValueError("MCP config 'servers' must be a mapping")
servers = []
for name, raw in raw_servers.items():
if not isinstance(raw, dict):
raise ValueError(f"MCP server '{name}' config must be a mapping")
cfg = McpServerConfig(
name=name,
transport=raw.get("transport", "stdio"),
command=raw.get("command"),
url=raw.get("url"),
headers=raw.get("headers", {}),
args=raw.get("args", []),
cwd=raw.get("cwd"),
env=raw.get("env", {}),
inherit_env=raw.get("inherit_env", list(DEFAULT_INHERITED_ENV)),
timeout=raw.get("timeout", 30),
description=raw.get("description", ""),
approved=raw.get("approved", False),
)
cfg.validate()
servers.append(cfg)
return servers
def save_mcp_config(servers: List[McpServerConfig]) -> None:
for server in servers:
server.validate()
MCP_CONFIG_FILE.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
body = _load_mcp_document()
body["version"] = MCP_CONFIG_VERSION
body["servers"] = {server.name: _server_to_dict(server) for server in servers}
serialized = yaml.safe_dump(body, default_flow_style=False, sort_keys=False)
temp_path = None
try:
MCP_CONFIG_FILE.parent.chmod(0o700)
with tempfile.NamedTemporaryFile(
mode="w",
encoding="utf-8",
dir=MCP_CONFIG_FILE.parent,
prefix=".mcp-",
suffix=".yaml.tmp",
delete=False,
) as temp_file:
temp_path = Path(temp_file.name)
temp_path.chmod(0o600)
temp_file.write(serialized)
temp_file.flush()
os.fsync(temp_file.fileno())
os.replace(temp_path, MCP_CONFIG_FILE)
MCP_CONFIG_FILE.chmod(0o600)
finally:
if temp_path is not None and temp_path.exists():
temp_path.unlink()
def _server_to_dict(server: McpServerConfig) -> dict:
data: dict = {
"transport": server.transport,
"approved": server.approved,
}
if server.command:
data["command"] = server.command
if server.url:
data["url"] = server.url
if server.headers:
data["headers"] = server.headers
if server.args:
data["args"] = server.args
if server.cwd:
data["cwd"] = server.cwd
if server.env:
data["env"] = server.env
if server.inherit_env != list(DEFAULT_INHERITED_ENV):
data["inherit_env"] = server.inherit_env
if server.timeout != 30:
data["timeout"] = server.timeout
if server.description:
data["description"] = server.description
return data
def _require_mcp_sdk() -> None:
try:
import mcp # noqa: F401
except ImportError as exc:
raise ImportError(
"MCP support requires the mcp package. Install with: pip install 'cmdforge[mcp]'"
) from exc
def load_mcp_serve_config() -> McpServeConfig:
data = _load_mcp_document()
server_raw = data.get("server") or {}
if not isinstance(server_raw, dict):
raise ValueError("MCP config 'server' must be a mapping")
return McpServeConfig(
expose=server_raw.get("expose", []),
deny=server_raw.get("deny", []),
)
def _serialize_tool(tool: Any) -> Dict[str, Any]:
if hasattr(tool, "model_dump"):
return tool.model_dump(mode="json", by_alias=True, exclude_none=True)
if isinstance(tool, dict):
return tool
return {
key: getattr(tool, key)
for key in ("name", "description", "inputSchema", "outputSchema")
if hasattr(tool, key)
}
class McpClientManager:
"""Invocation-scoped MCP configuration and schema manager.
SDK sessions are intentionally scoped to each discovery or tool operation.
This guarantees transport cleanup while the manager reuses loaded
configuration and discovered schemas across all MCP steps in one tool run.
"""
def __init__(self):
self._configs: Dict[str, McpServerConfig] = {}
self._schemas: Dict[str, Dict[str, Any]] = {}
self._schema_fingerprints: Dict[str, str] = {}
self._loaded = False
def __enter__(self) -> "McpClientManager":
return self
def __exit__(self, exc_type, exc_value, traceback) -> None:
self.shutdown()
def _ensure_loaded(self) -> None:
if self._loaded:
return
for cfg in load_mcp_config():
self._configs[cfg.name] = cfg
self._loaded = True
def shutdown(self) -> None:
self._schemas.clear()
self._schema_fingerprints.clear()
def list_servers(self) -> List[McpServerConfig]:
self._ensure_loaded()
return list(self._configs.values())
def _get_approved_config(self, server_name: str) -> McpServerConfig:
self._ensure_loaded()
cfg = self._configs.get(server_name)
if not cfg:
raise KeyError(f"MCP server '{server_name}' not configured. Add it to {MCP_CONFIG_FILE}")
cfg.validate()
if not cfg.approved:
command = (
" ".join([cfg.command or ""] + cfg.args)
if cfg.transport == "stdio" else cfg.url or ""
)
raise PermissionError(
f"MCP server '{server_name}' is not approved to execute: {_sanitize(command)}. "
"Re-add it with 'cmdforge mcp add ...' or set approved: true after review."
)
return cfg
def discover(self, server_name: str, refresh: bool = False) -> List[Dict[str, Any]]:
_require_mcp_sdk()
cfg = self._get_approved_config(server_name)
fingerprint = _fingerprint(cfg)
if (
not refresh
and server_name in self._schemas
and self._schema_fingerprints.get(server_name) == fingerprint
):
return list(self._schemas[server_name].values())
async def list_all_tools(session: Any) -> List[Dict[str, Any]]:
tools: List[Dict[str, Any]] = []
cursor = None
seen_cursors = set()
while True:
page = await session.list_tools(cursor=cursor)
tools.extend(_serialize_tool(tool) for tool in page.tools)
cursor = getattr(page, "nextCursor", None)
if not cursor:
break
if cursor in seen_cursors:
raise RuntimeError("MCP server repeated a tools/list cursor")
seen_cursors.add(cursor)
return tools
tools = _run_operation(cfg, list_all_tools, "discover tools")
self._schemas[server_name] = {tool["name"]: tool for tool in tools}
self._schema_fingerprints[server_name] = fingerprint
return tools
def get_tool_schema(self, server_name: str, tool_name: str) -> Dict[str, Any]:
if server_name not in self._schemas:
self.discover(server_name)
server_schemas = self._schemas.get(server_name, {})
if tool_name not in server_schemas:
raise KeyError(
f"Tool '{tool_name}' not found on MCP server '{server_name}'. "
f"Available: {list(server_schemas.keys())}"
)
return server_schemas[tool_name]
def call_tool(
self,
server_name: str,
tool_name: str,
arguments: Dict[str, Any],
result_mode: str = "auto",
) -> Any:
if result_mode not in RESULT_MODES:
raise ValueError(f"result_mode must be one of {RESULT_MODES}")
if not isinstance(arguments, dict):
raise ValueError("MCP tool arguments must be a mapping")
_require_mcp_sdk()
cfg = self._get_approved_config(server_name)
async def call(session: Any) -> Any:
result = await session.call_tool(
tool_name,
arguments,
read_timeout_seconds=timedelta(seconds=cfg.timeout),
)
return _normalize_result(result, result_mode)
return _run_operation(cfg, call, f"call {tool_name}")
def _run_operation(
cfg: McpServerConfig,
operation: Callable[[Any], Awaitable[T]],
operation_name: str,
) -> T:
if cfg.transport == "stdio":
return _run_stdio_operation(cfg, operation, operation_name)
return _run_http_operation(cfg, operation, operation_name)
def _run_http_operation(
cfg: McpServerConfig,
operation: Callable[[Any], Awaitable[T]],
operation_name: str,
) -> T:
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client
import httpx
resolved_headers = _build_http_headers(cfg)
async def run() -> T:
timeout = httpx.Timeout(cfg.timeout)
async with httpx.AsyncClient(
headers=resolved_headers, timeout=timeout, follow_redirects=False
) as client:
async with streamable_http_client(
cfg.url, http_client=client
) as (read, write, _):
async with ClientSession(
read, write,
read_timeout_seconds=timedelta(seconds=cfg.timeout),
) as session:
await session.initialize()
return await operation(session)
try:
return asyncio.run(asyncio.wait_for(run(), timeout=cfg.timeout))
except (asyncio.TimeoutError, TimeoutError) as exc:
raise TimeoutError(
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s"
) from exc
except Exception as exc:
secrets = list(resolved_headers.values())
raise RuntimeError(
f"MCP {operation_name} on '{cfg.name}' failed: "
f"{_sanitize(str(exc), secrets)}"
) from exc
def _run_stdio_operation(
cfg: McpServerConfig,
operation: Callable[[Any], Awaitable[T]],
operation_name: str,
) -> T:
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
resolved_env = _build_server_env(cfg)
server_params = StdioServerParameters(
command=cfg.command,
args=cfg.args,
cwd=cfg.cwd,
env=resolved_env,
)
async def run(stderr_file: Any) -> T:
async with stdio_client(server_params, errlog=stderr_file) as (read, write):
async with ClientSession(
read,
write,
read_timeout_seconds=timedelta(seconds=cfg.timeout),
) as session:
await session.initialize()
return await operation(session)
# The SDK passes errlog directly to subprocess.Popen, so it must expose a
# real file descriptor (StringIO is insufficient).
with tempfile.TemporaryFile(mode="w+", encoding="utf-8") as stderr_file:
started = time.monotonic()
try:
return asyncio.run(asyncio.wait_for(run(stderr_file), timeout=cfg.timeout))
except (asyncio.TimeoutError, TimeoutError) as exc:
stderr_file.seek(0)
detail = _sanitize(
stderr_file.read().strip(),
[resolved_env[name] for name in cfg.env if name in resolved_env],
)
suffix = f" (server stderr: {detail[:500]})" if detail else ""
raise TimeoutError(
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s{suffix}"
) from exc
except Exception as exc:
# Some SDK/AnyIO cleanup paths replace asyncio's cancellation with
# an ExceptionGroup (for example BrokenResourceError). The elapsed
# deadline remains authoritative: the requested operation timed out.
if time.monotonic() - started >= cfg.timeout:
raise TimeoutError(
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s"
) from exc
# Only configured values are secrets. Redacting inherited PATH/HOME
# values would make ordinary diagnostics unreadable.
secrets = [resolved_env[name] for name in cfg.env if name in resolved_env]
stderr_file.seek(0)
detail = _sanitize(stderr_file.read().strip(), secrets)
message = _sanitize(str(exc), secrets)
if detail:
message = f"{message} (server stderr: {detail[:500]})"
raise RuntimeError(
f"MCP {operation_name} on '{cfg.name}' failed: {message}"
) from exc
def _normalize_result(result: Any, mode: str) -> Any:
if mode not in RESULT_MODES:
raise ValueError(f"result_mode must be one of {RESULT_MODES}")
content = getattr(result, "content", []) or []
if getattr(result, "isError", False):
error_content = [_serialize_content_block(block) for block in content]
raise RuntimeError(f"MCP tool returned an error: {error_content}")
structured = getattr(result, "structuredContent", None)
if mode in ("auto", "structured") and structured is not None:
return structured
if mode == "structured":
raise ValueError("MCP tool did not return structuredContent")
text_parts = [
block.text
for block in content
if getattr(block, "type", None) == "text" and hasattr(block, "text")
]
if mode == "text":
return "\n".join(text_parts)
serialized = [_serialize_content_block(block) for block in content]
if mode == "content":
return serialized
if content and len(text_parts) == len(content):
return "\n".join(text_parts)
return serialized
def _serialize_content_block(block: Any) -> Dict[str, Any]:
if hasattr(block, "model_dump"):
return block.model_dump(mode="json", by_alias=True, exclude_none=True)
if isinstance(block, dict):
return block
result = {}
for name in ("type", "text", "mimeType", "data", "uri", "resource"):
if hasattr(block, name):
result[name] = getattr(block, name)
return result
+395
View File
@@ -0,0 +1,395 @@
"""MCP server support: expose CmdForge tools as MCP tools."""
import inspect
import hmac
import os
import sys
from typing import Annotated, Any, Literal, Optional
from urllib.parse import urlparse
from .mcp_client import (
MCP_DEPTH_ENV,
McpServeConfig,
_MCP_CALL_DEPTH,
_expand_env_value,
_is_loopback_host,
_require_mcp_sdk,
load_mcp_serve_config,
)
from .runner import MAX_TOOL_DEPTH
from .tool import ToolArgument
def _cmdforge_tool_name(name: str) -> str:
"""Map a CmdForge qualified tool name to an MCP-safe name.
MCP tool names must be simple identifiers. We map owner/name to owner__name.
"""
return name.replace("/", "__")
def _argument_to_json_schema(arg: ToolArgument) -> dict:
"""Map a CmdForge ToolArgument to a JSON Schema property."""
prop: dict = {}
arg_type = arg.type
type_map = {
"string": "string",
"integer": "integer",
"number": "number",
"boolean": "boolean",
}
prop["type"] = type_map.get(arg_type, "string")
if arg.description:
prop["description"] = arg.description
if arg.default is not None:
prop["default"] = _coerce_argument_value(arg.default, arg.type)
if arg.enum:
prop["enum"] = [
_coerce_argument_value(value, arg.type) for value in arg.enum
]
return prop
def _build_tool_schema(tool) -> dict:
"""Build the MCP inputSchema for a CmdForge tool.
Returns a dict with name, description, and inputSchema suitable for
registering with FastMCP.
"""
properties = {}
required = []
for arg in tool.arguments:
prop = _argument_to_json_schema(arg)
properties[arg.variable] = prop
if arg.required:
required.append(arg.variable)
if "input" not in properties:
properties["input"] = {
"type": "string",
"description": "Text passed to the CmdForge tool as standard input.",
"default": "",
}
input_schema = {
"type": "object",
"properties": properties,
}
if required:
input_schema["required"] = required
return {
"name": _cmdforge_tool_name(tool.name),
"description": tool.description or f"CmdForge tool: {tool.name}",
"inputSchema": input_schema,
}
class _StaticTokenVerifier:
def __init__(self, token: str):
self._token = token
async def verify_token(self, token: str):
if not hmac.compare_digest(token, self._token):
return None
from mcp.server.auth.provider import AccessToken
return AccessToken(
token=token, client_id="cmdforge-mcp-client", scopes=[],
subject="cmdforge-mcp",
)
def _validate_http_server_options(
host: str,
port: int,
allowed_origins: list[str],
auth_token: Optional[str],
external_url: Optional[str],
) -> tuple[list[str], Optional[str]]:
if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
raise ValueError("MCP server port must be between 1 and 65535")
local = _is_loopback_host(host)
resolved_token = _expand_env_value(auth_token) if auth_token else None
if not local:
parsed = urlparse(external_url or "")
if parsed.scheme != "https" or not parsed.hostname:
raise ValueError(
"Non-local MCP serving requires --external-url https://... "
"behind a TLS-terminating proxy"
)
if not resolved_token:
raise ValueError("Non-local MCP serving requires --auth-token")
origins = allowed_origins or (
[f"http://localhost:{port}", f"http://127.0.0.1:{port}"]
if local else [f"{urlparse(external_url).scheme}://{urlparse(external_url).netloc}"]
)
for origin in origins:
parsed = urlparse(origin)
if parsed.scheme not in ("http", "https") or not parsed.netloc or parsed.path not in ("", "/"):
raise ValueError(f"Invalid allowed origin: {origin}")
if not local and parsed.scheme != "https":
raise ValueError("Non-local MCP origins must use HTTPS")
return origins, resolved_token
def serve(
transport: str = "stdio",
*,
host: str = "127.0.0.1",
port: int = 8000,
allowed_origins: Optional[list[str]] = None,
auth_token: Optional[str] = None,
external_url: Optional[str] = None,
) -> None:
"""Start CmdForge as an MCP server on the given transport.
HTTP defaults to loopback and enables SDK DNS-rebinding/origin protection.
Non-local binding requires bearer auth and an explicit public HTTPS URL;
TLS is expected to terminate at the reverse proxy represented by that URL.
"""
if transport not in ("stdio", "streamable-http"):
raise ValueError(f"Unsupported MCP server transport '{transport}'")
_require_mcp_sdk()
from mcp.server.fastmcp import FastMCP
serve_config = load_mcp_serve_config()
from .tool import list_tools, load_tool
server_options = {}
if transport == "streamable-http":
origins, resolved_token = _validate_http_server_options(
host, port, allowed_origins or [], auth_token, external_url
)
from mcp.server.transport_security import TransportSecuritySettings
allowed_hosts = [host, f"{host}:{port}", "localhost", f"localhost:{port}"]
if external_url:
external_host = urlparse(external_url).netloc
allowed_hosts.extend([external_host, urlparse(external_url).hostname or ""])
server_options.update({
"host": host,
"port": port,
"transport_security": TransportSecuritySettings(
enable_dns_rebinding_protection=True,
allowed_hosts=[value for value in allowed_hosts if value],
allowed_origins=origins,
),
})
if resolved_token:
from mcp.server.auth.settings import AuthSettings
issuer = external_url or f"http://{host}:{port}"
server_options.update({
"token_verifier": _StaticTokenVerifier(resolved_token),
"auth": AuthSettings(
issuer_url=issuer,
resource_server_url=f"{issuer.rstrip('/')}/mcp",
required_scopes=[],
),
})
server = FastMCP("CmdForge", **server_options)
tool_names = list_tools()
exposed_count = 0
exposed_names = {}
for name in sorted(tool_names):
# Skip private/unlisted tools unless explicitly exposed
tool_obj = load_tool(name)
if not tool_obj:
continue
if not _is_exposable(tool_obj, serve_config, qualified_name=name):
continue
mcp_name = _cmdforge_tool_name(name)
previous = exposed_names.get(mcp_name)
if previous is not None:
raise RuntimeError(
f"MCP tool name collision: '{previous}' and '{name}' both map to "
f"'{mcp_name}'"
)
exposed_names[mcp_name] = name
schema = _build_tool_schema(tool_obj)
# Capture tool reference for the handler
_register_tool(server, name, mcp_name, schema, tool_obj)
exposed_count += 1
print(f"[mcp] CmdForge MCP server starting on {transport}", file=sys.stderr)
print(f"[mcp] Exposing {exposed_count} tool(s)", file=sys.stderr)
server.run(transport=transport)
def _is_exposable(
tool, config: McpServeConfig, qualified_name: str | None = None
) -> bool:
"""Check whether a tool should be exposed per the server policy.
Private/unlisted tools must be explicitly listed in expose. Public tools
are exposed only if the expose list includes them or '*'.
"""
policy_name = qualified_name or tool.name
if config._matches_deny(policy_name):
return False
if tool.visibility not in ("public", None, ""):
# private or unlisted — must be explicitly named
return policy_name in config.expose
return config.is_exposed(policy_name)
def _register_tool(server, cmdforge_name: str, mcp_name: str, schema: dict, tool_obj) -> None:
"""Register a CmdForge tool handler with the FastMCP server."""
tool_desc = schema.get("description", "")
argument_names = {arg.variable for arg in tool_obj.arguments}
def handler(**kwargs: Any) -> str:
from .runner import run_tool
inherited_depth = _read_mcp_depth()
if inherited_depth >= MAX_TOOL_DEPTH:
raise RuntimeError(
f"Maximum MCP nesting depth ({MAX_TOOL_DEPTH}) exceeded"
)
if "input" in argument_names:
input_text = kwargs.get("input", "") or ""
else:
input_text = kwargs.pop("input", "") or ""
token = _MCP_CALL_DEPTH.set(inherited_depth + 1)
try:
output, exit_code = run_tool(
tool_obj,
input_text=input_text,
custom_args=kwargs,
provider_override=None,
verbose=False,
dry_run=False,
_depth=inherited_depth,
)
finally:
_MCP_CALL_DEPTH.reset(token)
if exit_code != 0:
raise RuntimeError(f"Tool '{cmdforge_name}' exited with code {exit_code}")
return output
handler.__name__ = f"cmdforge_{mcp_name}"
handler.__signature__ = _build_handler_signature(tool_obj)
server.add_tool(
handler,
name=mcp_name,
description=tool_desc,
structured_output=False,
)
def _build_handler_signature(tool) -> inspect.Signature:
"""Create the flat typed signature FastMCP uses for schema and validation."""
required_parameters = []
optional_parameters = []
seen = set()
for arg in tool.arguments:
if not arg.variable.isidentifier():
raise ValueError(
f"Tool '{tool.name}' has invalid argument variable '{arg.variable}'"
)
if arg.variable in seen:
raise ValueError(
f"Tool '{tool.name}' defines argument '{arg.variable}' more than once"
)
seen.add(arg.variable)
annotation = _argument_annotation(arg)
default = inspect.Parameter.empty
if not arg.required:
default = (
_coerce_argument_value(arg.default, arg.type)
if arg.default is not None
else None
)
parameter = inspect.Parameter(
arg.variable,
kind=inspect.Parameter.KEYWORD_ONLY,
default=default,
annotation=annotation,
)
(required_parameters if arg.required else optional_parameters).append(parameter)
if "input" not in seen:
optional_parameters.append(
inspect.Parameter(
"input",
kind=inspect.Parameter.KEYWORD_ONLY,
default="",
annotation=Annotated[
str,
_pydantic_field(
"Text passed to the CmdForge tool as standard input."
),
],
)
)
return inspect.Signature(
required_parameters + optional_parameters,
return_annotation=str,
)
def _argument_annotation(arg: ToolArgument):
python_type = {
"string": str,
"integer": int,
"number": float,
"boolean": bool,
}[arg.type]
if arg.enum:
values = tuple(_coerce_argument_value(value, arg.type) for value in arg.enum)
python_type = Literal[values]
if arg.description:
return Annotated[python_type, _pydantic_field(arg.description)]
return python_type
def _pydantic_field(description: str):
from pydantic import Field
return Field(description=description)
def _coerce_argument_value(value: Any, arg_type: str) -> Any:
if value is None:
return None
if arg_type == "string":
return str(value)
if arg_type == "integer":
return int(value)
if arg_type == "number":
return float(value)
if arg_type == "boolean":
if isinstance(value, str):
normalized = value.strip().lower()
if normalized in ("true", "1", "yes", "on"):
return True
if normalized in ("false", "0", "no", "off"):
return False
raise ValueError(f"Invalid boolean value: {value}")
return bool(value)
raise ValueError(f"Unsupported argument type: {arg_type}")
def _read_mcp_depth() -> int:
raw_depth = os.environ.get(MCP_DEPTH_ENV, "0")
try:
depth = int(raw_depth)
except ValueError as exc:
raise RuntimeError(f"Invalid {MCP_DEPTH_ENV} value: {raw_depth}") from exc
if depth < 0:
raise RuntimeError(f"Invalid {MCP_DEPTH_ENV} value: {raw_depth}")
return depth
+626
View File
@@ -0,0 +1,626 @@
"""Shared preflight analysis engine.
Produces a PreflightReport used by the GUI, CLI, and registry workflow.
All checks are deterministic and evidence-based.
"""
import copy
import hashlib
import json
import re
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
from .tool import Tool, validate_json_schema
@dataclass
class PreflightReport:
"""Structured analysis of a tool before creation, save, or publish."""
errors: List[str] = field(default_factory=list)
warnings: List[str] = field(default_factory=list)
suggestions: List[str] = field(default_factory=list)
similar_tools: List[Dict[str, Any]] = field(default_factory=list)
generated_tests: List[Dict[str, Any]] = field(default_factory=list)
compatibility: List[Dict[str, Any]] = field(default_factory=list)
contract_proposal: Optional[Dict[str, Any]] = None
regression: Optional[Dict[str, Any]] = None
reuse_opportunities: List[Dict[str, Any]] = field(default_factory=list)
audit_evidence: Optional[Dict[str, Any]] = None
@property
def ok(self) -> bool:
return not self.errors
def merge(self, other: "PreflightReport") -> "PreflightReport":
return PreflightReport(
errors=self.errors + other.errors,
warnings=self.warnings + other.warnings,
suggestions=self.suggestions + other.suggestions,
similar_tools=self.similar_tools + other.similar_tools,
generated_tests=self.generated_tests + other.generated_tests,
compatibility=self.compatibility + other.compatibility,
contract_proposal=self.contract_proposal or other.contract_proposal,
regression=self.regression or other.regression,
reuse_opportunities=self.reuse_opportunities + other.reuse_opportunities,
audit_evidence=self.audit_evidence or other.audit_evidence,
)
def to_dict(self) -> Dict[str, Any]:
return {
"errors": list(self.errors),
"warnings": list(self.warnings),
"suggestions": list(self.suggestions),
"similar_tools": list(self.similar_tools),
"generated_tests": list(self.generated_tests),
"compatibility": list(self.compatibility),
"contract_proposal": copy.deepcopy(self.contract_proposal),
"regression": copy.deepcopy(self.regression),
"reuse_opportunities": list(self.reuse_opportunities),
"audit_evidence": copy.deepcopy(self.audit_evidence) if self.audit_evidence else None,
}
def analyze_tool(
tool: Tool,
registry_client=None,
*,
check_local_dependencies: bool = True,
include_contract_tests: bool = False,
check_reuse: Optional[bool] = None,
) -> PreflightReport:
"""Run all preflight checks on a tool.
Args:
tool: The Tool to analyze.
registry_client: Optional registry client for registry-side checks.
Returns:
A PreflightReport with structured findings.
"""
report = PreflightReport()
checks_run = ["config_integrity", "contracts"]
_check_config_integrity(tool, report)
_check_contracts(tool, report)
_add_contract_guidance(tool, report, include_contract_tests)
if include_contract_tests:
checks_run.append("contract_conformance")
_check_secrets(tool, report)
checks_run.append("secrets")
if check_local_dependencies:
_check_toolstep_compatibility(tool, report)
_check_dependencies(tool, report)
checks_run.extend(["toolstep_compatibility", "dependencies"])
if registry_client:
_check_similar_tools(tool, report, registry_client)
checks_run.append("registry_similarity")
if check_reuse is None:
check_reuse = check_local_dependencies
if check_reuse:
_check_reuse_opportunities(tool, report)
checks_run.append("reuse_opportunities")
_add_audit_evidence(tool, report, checks_run)
return report
def _add_contract_guidance(
tool: Tool, report: PreflightReport, include_contract_tests: bool
) -> None:
proposal = infer_contracts(tool)
report.contract_proposal = proposal
inferred = [
label
for label in ("input", "output")
if proposal[f"{label}_source"] == "inferred"
]
if inferred:
report.suggestions.append(
"Review inferred " + " and ".join(inferred) + " contract proposal"
)
if not include_contract_tests:
return
if tool.input_schema is None or tool.output_schema is None:
return
from .contract_testing import (
BaselineFormatError,
compare_conformance,
load_conformance_baseline,
run_contract_tests,
)
baseline = None
if tool.path:
try:
baseline = load_conformance_baseline(tool.path.parent)
except BaselineFormatError as exc:
report.warnings.append(f"Regression baseline is invalid: {exc}")
baseline_inputs = (
[result["input_value"] for result in baseline["results"]]
if baseline is not None else None
)
conformance = run_contract_tests(tool, test_inputs=baseline_inputs)
report.generated_tests = [result.to_dict() for result in conformance.results]
states = {result.state for result in conformance.results}
if "failed" in states:
details = "; ".join(
result.detail for result in conformance.results if result.state == "failed"
)
report.errors.append(f"Contract conformance failed: {details}")
if "unsupported" in states:
details = "; ".join(
result.detail
for result in conformance.results
if result.state == "unsupported"
)
report.warnings.append(f"Contract conformance unsupported: {details}")
if baseline is not None:
try:
delta = compare_conformance(baseline, conformance, tool)
except BaselineFormatError as exc:
report.warnings.append(f"Regression baseline is invalid: {exc}")
else:
report.regression = delta.to_dict()
if delta.has_regressions:
report.warnings.append(f"Regression comparison: {delta.summary}")
elif delta.contract_changed:
report.suggestions.append(
"Tool contracts changed since the saved conformance baseline"
)
def _check_config_integrity(tool: Tool, report: PreflightReport):
if not tool.name:
report.errors.append("Tool name is required")
if tool.version and not _is_semver(tool.version):
report.errors.append(f"Version '{tool.version}' is not valid semver")
if not tool.steps and not tool.arguments and tool.output == "{input}":
report.warnings.append(
"Tool has no steps and no arguments — output template is passthrough"
)
def _check_contracts(tool: Tool, report: PreflightReport):
if tool.output_schema is not None:
_validate_schema(tool.output_schema, "output_schema", report)
if tool.input_schema is not None:
_validate_schema(tool.input_schema, "input_schema", report)
if tool.output_schema is None and tool.input_schema is None:
report.suggestions.append(
"Add input_schema and output_schema to enable automated verification"
)
def _validate_schema(schema: Any, label: str, report: PreflightReport):
try:
validate_json_schema(schema, label)
except ValueError as exc:
report.errors.append(str(exc))
def _check_secrets(tool: Tool, report: PreflightReport):
secret_patterns = ["api_key", "api_secret", "password", "token", "secret"]
for step in tool.steps:
if hasattr(step, "prompt"):
prompt_lower = step.prompt.lower()
for pat in secret_patterns:
# Do not flag schema/field names such as max_tokens or
# token_count. Underscores are identifier characters here,
# even though regex \b would treat them inconsistently around
# compound names.
if re.search(
rf"(?<![a-z0-9_]){re.escape(pat)}(?![a-z0-9_])",
prompt_lower,
):
report.warnings.append(
f"Prompt step contains potential secret pattern '{pat}'"
)
break
def _check_toolstep_compatibility(tool: Tool, report: PreflightReport):
from .schema_compat import compare_json_schemas
from .tool import ToolStep, load_tool
for index, step in enumerate(tool.steps):
if not isinstance(step, ToolStep):
continue
nested = load_tool(step.tool)
if not nested:
report.compatibility.append({
"tool": step.tool,
"step": index + 1,
"state": "unresolved",
"detail": f"Tool '{step.tool}' not found locally",
"issues": ["Called tool is not installed locally"],
})
report.suggestions.append(
f"Could not check ToolStep {index + 1} -> '{step.tool}': "
"tool is not installed locally"
)
continue
if nested.input_schema is None:
finding = {
"tool": step.tool,
"step": index + 1,
"state": "unknown",
"detail": f"Tool '{step.tool}' has no input_schema",
"issues": ["Downstream input contract is missing"],
}
report.compatibility.append(finding)
report.suggestions.append(finding["detail"])
continue
supplied_schema, source_schema = _toolstep_supplied_schema(
tool, step, index, nested
)
result = compare_json_schemas(supplied_schema, nested.input_schema)
finding = {
"tool": step.tool,
"step": index + 1,
"state": result.state,
"detail": result.detail,
"issues": result.issues,
"upstream_schema": source_schema,
"supplied_schema": supplied_schema,
"downstream_schema": copy.deepcopy(nested.input_schema),
}
report.compatibility.append(finding)
if result.state == "incompatible":
report.warnings.append(
f"ToolStep {index + 1} -> '{step.tool}' is incompatible: "
f"{result.detail}"
)
elif result.state == "unknown":
report.suggestions.append(
f"Could not prove ToolStep {index + 1} -> '{step.tool}' "
f"compatibility: {result.detail}"
)
def _toolstep_supplied_schema(tool: Tool, step, index: int, nested: Tool):
"""Describe values the current runner actually supplies to a nested tool."""
from .schema_compat import schema_for_value
input_schema = (
schema_for_value(step.input_template)
if "{" not in step.input_template
else {"type": "string"}
)
properties: Dict[str, Any] = {"input": input_schema}
required = ["input"]
source_schema = _template_source_schema(tool, step.input_template, index)
for name, value in step.args.items():
# execute_tool_step substitutes every explicit argument through str().
if isinstance(value, str) and "{" not in value:
properties[name] = schema_for_value(value)
else:
properties[name] = {"type": "string"}
required.append(name)
for argument in nested.arguments:
if argument.variable in properties or argument.default is None:
continue
properties[argument.variable] = schema_for_value(argument.default)
required.append(argument.variable)
supplied = {
"type": "object",
"properties": properties,
"required": required,
"additionalProperties": False,
}
if _schema_type_name(nested.input_schema) != "object":
# Scalar contracts describe stdin directly when no argument object is used.
supplied = properties["input"]
return supplied, source_schema
def _template_source_schema(tool: Tool, template: str, before_index: int):
match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", template.strip())
if not match:
return {"type": "string"}
variable = match.group(1)
if tool.input_schema is not None:
if _schema_type_name(tool.input_schema) == "object":
schema = tool.input_schema.get("properties", {}).get(variable)
if schema is not None:
return copy.deepcopy(schema)
elif variable == "input":
return copy.deepcopy(tool.input_schema)
from .tool import PromptStep, ToolStep, load_tool
for previous in reversed(tool.steps[:before_index]):
if getattr(previous, "output_var", None) != variable:
continue
if isinstance(previous, PromptStep):
return copy.deepcopy(previous.output_schema)
if isinstance(previous, ToolStep):
nested = load_tool(previous.tool)
return copy.deepcopy(nested.output_schema) if nested else None
return None
return None
def _schema_type_name(schema: Optional[dict]) -> Optional[str]:
if not schema:
return None
value = schema.get("type")
if isinstance(value, str):
return value
if "properties" in schema or "required" in schema:
return "object"
return None
def _check_dependencies(tool: Tool, report: PreflightReport):
from .tool import tool_exists
for dep in tool.dependencies:
ref = dep.get("name", dep) if isinstance(dep, dict) else dep
if not tool_exists(ref):
report.warnings.append(f"Dependency '{ref}' is not installed locally")
def _check_similar_tools(tool: Tool, report: PreflightReport, client):
from .registry_client import RegistryError
try:
results = client.search_tools(tool.name, per_page=5)
items = results.data if hasattr(results, "data") else results
for item in items:
report.similar_tools.append({
"name": (item.get("owner") or "") + "/" + item.get("name", ""),
"description": item.get("description", ""),
"downloads": item.get("downloads", 0),
})
if not items:
report.suggestions.append("No similar tools found on the registry")
except RegistryError:
report.warnings.append("Could not search registry for similar tools")
def _is_semver(version: str) -> bool:
from .semver import Version
return Version.parse(version) is not None
def _check_reuse_opportunities(tool: Tool, report: PreflightReport):
"""Report only exact repeated sequences with contract-boundary evidence."""
from .tool import list_tools, load_tool
if len(tool.steps) < 2:
return
step_signatures = [_step_signature(step) for step in tool.steps]
_find_internal_reuse(tool, step_signatures, report)
# Check if this step sequence appears in other local tools
our_sequence = tuple(step_signatures)
for name in list_tools():
if name == tool.name:
continue
other = load_tool(name)
if not other or len(other.steps) < 2:
continue
other_sigs = [_step_signature(s) for s in other.steps]
other_sigs = [s for s in other_sigs if s]
if (
our_sequence == tuple(other_sigs)
and _has_tool_contracts(tool)
and _has_tool_contracts(other)
):
report.reuse_opportunities.append({
"type": "duplicate_sequence",
"tool": name,
"length": len(our_sequence),
"evidence": "exact normalized steps and explicit tool contracts",
"detail": f"Exact contracted step sequence also exists in '{name}'",
})
def _step_signature(step) -> str:
"""Build an exact normalized signature without incidental labels/outputs."""
data = step.to_dict()
data.pop("name", None)
data.pop("output_var", None)
encoded = json.dumps(data, sort_keys=True, separators=(",", ":"), default=str)
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
def _has_tool_contracts(tool: Tool) -> bool:
return tool.input_schema is not None and tool.output_schema is not None
def _find_internal_reuse(
tool: Tool, signatures: List[str], report: PreflightReport
) -> None:
"""Find non-overlapping repeated sequences with per-step output evidence."""
from .tool import PromptStep
count = len(signatures)
for length in range(count // 2, 1, -1):
for first in range(0, count - (2 * length) + 1):
candidate = signatures[first:first + length]
for second in range(first + length, count - length + 1):
if candidate != signatures[second:second + length]:
continue
steps = tool.steps[first:first + length]
if not all(
isinstance(step, PromptStep) and step.output_schema is not None
for step in steps
):
continue
report.reuse_opportunities.append({
"type": "repeated_sequence",
"length": length,
"locations": [first + 1, second + 1],
"evidence": "exact normalized prompt steps with output schemas",
"detail": (
f"Steps {first + 1}-{first + length} repeat at "
f"steps {second + 1}-{second + length}"
),
})
return
def _add_audit_evidence(
tool: Tool, report: PreflightReport, checks_run: List[str]
):
"""Attach audit evidence metadata to the report."""
from datetime import datetime, timezone
findings = {
"errors": report.errors,
"warnings": report.warnings,
"suggestions": report.suggestions,
"generated_tests": report.generated_tests,
"regression": report.regression,
"compatibility": report.compatibility,
"similar_tools": report.similar_tools,
"reuse_opportunities": report.reuse_opportunities,
}
findings_hash = hashlib.sha256(
json.dumps(findings, sort_keys=True, separators=(",", ":"), default=str).encode()
).hexdigest()
report.audit_evidence = {
"tool_version": tool.version or "",
"engine_version": "1.0",
"timestamp": datetime.now(timezone.utc).isoformat(),
"checks_run": checks_run,
"findings_hash": findings_hash,
"findings": copy.deepcopy(findings),
"outcome": "failed" if report.errors else "advisory" if report.warnings else "passed",
}
def infer_contracts(tool: Tool) -> Dict[str, Any]:
"""Propose input/output schemas without mutating the tool.
Returns a dict with:
- input_schema: proposed JSON Schema (or None)
- output_schema: proposed JSON Schema (or None)
- evidence: list of human-readable provenance strings
- confidence: "high" / "medium" / "low"
Never overwrites an explicit schema. ``{}`` counts as explicit.
"""
evidence = []
proposed_input = None
proposed_output = None
if tool.input_schema is None:
proposed_input, input_evidence = _infer_input_schema(tool)
evidence.extend(input_evidence)
if tool.output_schema is None:
proposed_output, output_evidence = _infer_output_schema(tool)
evidence.extend(output_evidence)
effective_input = tool.input_schema if tool.input_schema is not None else proposed_input
effective_output = tool.output_schema if tool.output_schema is not None else proposed_output
confidence = _confidence(effective_input, effective_output)
return {
"input_schema": proposed_input,
"output_schema": proposed_output,
"input_source": "explicit" if tool.input_schema is not None else (
"inferred" if proposed_input is not None else "unavailable"
),
"output_source": "explicit" if tool.output_schema is not None else (
"inferred" if proposed_output is not None else "unavailable"
),
"evidence": evidence,
"confidence": confidence,
}
def _infer_input_schema(tool: Tool) -> tuple:
props = {}
required = []
evidence = []
has_stdin = _contains_input_reference(tool.output) or any(
_contains_input_reference(vars(step)) for step in tool.steps
)
if has_stdin:
props["input"] = {"type": "string", "description": "Standard input text"}
evidence.append("stdin detected from {input} usage")
for arg in tool.arguments:
prop = {"type": arg.type or "string"}
if arg.description:
prop["description"] = arg.description
if arg.default is not None:
prop["default"] = arg.default
if arg.enum:
prop["enum"] = arg.enum
props[arg.variable] = prop
if arg.required:
required.append(arg.variable)
evidence.append(f"argument {arg.flag} ({arg.type or 'string'})")
if not props:
return None, []
schema = {"type": "object", "properties": props}
if required:
schema["required"] = required
return schema, evidence
def _infer_output_schema(tool: Tool) -> tuple:
evidence = []
output = tool.output.strip()
if not output or output == "{input}":
return None, evidence
var_match = None
m = re.match(r"^\{([a-zA-Z_][a-zA-Z0-9_]*)\}$", output)
if m:
var_match = m.group(1)
if not var_match:
evidence.append("output template is a composite expression — schema must be defined manually")
return None, evidence
# Look for the step that produces this variable
for step in reversed(tool.steps):
if hasattr(step, "output_var") and step.output_var == var_match:
schema = getattr(step, "output_schema", None)
if schema is not None:
evidence.append(
f"output variable '{var_match}' has an explicit step schema — using as tool output schema"
)
return copy.deepcopy(schema), evidence
elif hasattr(step, "provider"):
evidence.append(
f"output variable '{var_match}' comes from a provider step with no schema — define output_schema on the step"
)
return None, evidence
break
evidence.append(f"output variable '{var_match}' source not found")
return None, evidence
def _contains_input_reference(value: Any) -> bool:
if isinstance(value, str):
return "{input}" in value or "{input." in value
if isinstance(value, dict):
return any(_contains_input_reference(item) for item in value.values())
if isinstance(value, (list, tuple)):
return any(_contains_input_reference(item) for item in value)
return False
def _confidence(input_schema, output_schema) -> str:
if input_schema is not None and output_schema is not None:
return "high"
if input_schema is not None or output_schema is not None:
return "medium"
return "low"
+300
View File
@@ -0,0 +1,300 @@
"""Prompt variation generation and optimization (M9.1).
Generates N variations of a tool's prompt steps, runs each through M8.V1
contract conformance tests, and surfaces the highest-scoring variation.
"""
from dataclasses import dataclass, field
from typing import Callable, Dict, List, Optional
from .tool import Tool, PromptStep
@dataclass
class PromptVariation:
"""One generated variation of a tool's prompt."""
step_index: int
original: str
variation: str
strategy: str # how this variation was created
def to_dict(self) -> dict:
return {
"step_index": self.step_index,
"original": self.original,
"variation": self.variation,
"strategy": self.strategy,
}
@dataclass
class OptimizationResult:
"""Results of running all variations through contract tests."""
tool_name: str
variations: List[PromptVariation] = field(default_factory=list)
scores: Dict[int, int] = field(default_factory=dict) # variation_index -> score
best_index: Optional[int] = None
baseline_score: int = 0
evaluation_kind: str = "structural"
note: str = ""
@property
def best(self) -> Optional[PromptVariation]:
if self.best_index is not None and self.best_index < len(self.variations):
return self.variations[self.best_index]
return None
@property
def improvement(self) -> float:
if self.best is None or self.baseline_score == 0:
return 0.0
best = self.scores.get(self.best_index, 0)
return (best - self.baseline_score) / max(self.baseline_score, 1) * 100
def to_dict(self) -> dict:
return {
"tool": self.tool_name,
"baseline_score": self.baseline_score,
"evaluation_kind": self.evaluation_kind,
"note": self.note,
"best_index": self.best_index,
"improvement_pct": round(self.improvement, 1),
"best": self.best.to_dict() if self.best else None,
"variations": [v.to_dict() for v in self.variations],
"scores": {str(k): v for k, v in self.scores.items()},
}
def generate_variations(
tool: Tool,
count: int = 5,
provider: str = "mock",
) -> List[PromptVariation]:
"""Generate prompt variations for each prompt step in the tool.
Strategies:
- rephrase: ask the AI to rephrase the prompt
- structure: convert to bullet points
- concise: make it shorter
- detailed: add more context
- focused: emphasize the key task
Args:
tool: The tool to optimize.
count: Number of variations per prompt step.
provider: Provider to use for variation generation.
Returns:
List of PromptVariation objects.
"""
variations: List[PromptVariation] = []
for i, step in enumerate(tool.steps):
if not isinstance(step, PromptStep):
continue
original = step.prompt
strategies = _get_strategies(original, count, provider=provider)
for strategy, variation_text in strategies:
if variation_text and variation_text != original:
variations.append(PromptVariation(
step_index=i,
original=original,
variation=variation_text,
strategy=strategy,
))
return variations
def _get_strategies(
original: str, count: int, provider: str = "mock"
) -> List[tuple]:
"""Generate variations using different strategies."""
strategies: List[tuple] = []
if count <= 0:
return []
# Provider calls are explicit. The default/mock path is deterministic and
# never sends a user's prompt to an external process or service.
if provider != "mock":
rephrased = _call_provider_for_variation(
provider,
"Rephrase this instruction while keeping the same meaning. "
f"Output only the rephrased text:\n\n{original}",
)
if rephrased and rephrased != original:
strategies.append(("rephrase", rephrased))
# Deterministic mock variations
mock_strategies = [
("focused", f"Complete this specific task — {original}"),
("concise", _truncate(original, 120)),
("detailed", f"Carefully perform the following: {original}. Ensure correctness and completeness."),
("rephrase", f"I need you to do this: {original}"),
("structure", f"Step through this task:\n\n{original}\n\nRespond step by step."),
]
for strat, text in mock_strategies:
if len(strategies) >= count:
break
if text and text != original:
strategies.append((strat, text))
return strategies[:count]
def _truncate(text: str, max_len: int) -> str:
if len(text) <= max_len:
return text
return text[:max_len - 3].rsplit(" ", 1)[0] + "..."
def _call_provider_for_variation(provider: str, prompt: str) -> Optional[str]:
"""Call the provider explicitly selected by the user."""
from .providers import call_provider
result = call_provider(provider, prompt, timeout=15)
if result.success and result.text:
text = result.text.strip()
if text and len(text) >= 10:
return text
return None
def optimize_tool(
tool: Tool,
count: int = 5,
provider: str = "mock",
evaluator: Optional[Callable[[Tool], float]] = None,
) -> OptimizationResult:
"""Run prompt optimization for a tool.
Generates variations, runs each through contract tests, and returns
the best-performing variation.
Args:
tool: The tool to optimize.
count: Number of variations per prompt step.
provider: Provider for variation generation.
Returns:
OptimizationResult with scores and best variation.
"""
from .contract_testing import run_contract_tests
result = OptimizationResult(
tool_name=tool.name,
evaluation_kind="behavioral" if evaluator else "structural",
)
# Baseline score
if evaluator:
result.baseline_score = evaluator(tool)
else:
baseline = run_contract_tests(tool)
result.baseline_score = _count_passed(baseline)
# Generate variations
result.variations = generate_variations(tool, count=count, provider=provider)
if not result.variations:
return result
# Test each variation
for idx, variation in enumerate(result.variations):
# Create a copy with the modified prompt
import copy
tool_copy = copy.deepcopy(tool)
if 0 <= variation.step_index < len(tool_copy.steps):
step = tool_copy.steps[variation.step_index]
if isinstance(step, PromptStep):
step.prompt = variation.variation
try:
if evaluator:
result.scores[idx] = evaluator(tool_copy)
else:
test_result = run_contract_tests(tool_copy)
result.scores[idx] = _count_passed(test_result)
except Exception:
result.scores[idx] = 0
# Find best
if evaluator and result.scores:
candidate = max(result.scores, key=lambda k: result.scores[k])
if result.scores[candidate] > result.baseline_score:
result.best_index = candidate
else:
result.note = "No variation improved on the behavioral baseline."
elif result.scores:
result.note = (
"Structural conformance cannot compare prompt semantics; "
"no best variation was selected."
)
return result
def _count_passed(test_result) -> int:
"""Count passed tests in a ConformanceReport."""
return sum(1 for r in getattr(test_result, "results", [])
if getattr(r, "state", "") == "passed")
def load_behavioral_evaluator(path, provider_override: Optional[str] = None):
"""Load explicit behavioral cases and return a semantic score function.
The JSON file must contain a list of cases. Each case accepts ``input`` and
optional ``args``, plus exactly one of ``expected`` (exact value) or
``contains`` (substring). Supplying this file is explicit authorization to
execute the candidate tools and their configured providers.
"""
import json
from pathlib import Path
cases = json.loads(Path(path).read_text(encoding="utf-8"))
if not isinstance(cases, list) or not cases:
raise ValueError("Behavior test file must contain a non-empty JSON list")
for index, case in enumerate(cases):
if not isinstance(case, dict):
raise ValueError(f"Behavior case {index} must be an object")
assertions = [key for key in ("expected", "contains") if key in case]
if len(assertions) != 1:
raise ValueError(
f"Behavior case {index} needs exactly one of expected or contains"
)
if not isinstance(case.get("args", {}), dict):
raise ValueError(f"Behavior case {index} args must be an object")
def evaluate(tool: Tool) -> float:
from .runner import run_tool
passed = 0
for case in cases:
output, exit_code = run_tool(
tool,
input_text=str(case.get("input", "")),
custom_args=case.get("args", {}),
provider_override=provider_override,
dry_run=False,
verbose=False,
)
if exit_code != 0:
continue
if "expected" in case:
try:
actual = json.loads(output)
except (TypeError, json.JSONDecodeError):
actual = output
if actual == case["expected"]:
passed += 1
elif str(case["contains"]) in output:
passed += 1
return passed
return evaluate
+384 -32
View File
@@ -5,6 +5,7 @@ import re
import shlex
import subprocess
import shutil
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional, List
@@ -54,6 +55,9 @@ KNOWN_PROVIDER_CLIS = {
"command": "opencode run --model opencode/big-pickle",
"description": "OpenCode - Big Pickle (free general model)",
"tags": ["free", "code", "general"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
"install_group": "opencode",
},
"agy": {
@@ -61,6 +65,9 @@ KNOWN_PROVIDER_CLIS = {
"command": "agy -p",
"description": "Antigravity - Google free tier, Gemini models",
"tags": ["free-tier", "code", "large-context"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
"install_group": "agy",
},
"codex": {
@@ -68,6 +75,9 @@ KNOWN_PROVIDER_CLIS = {
"command": "codex exec -",
"description": "Codex CLI - OpenAI free tier available",
"tags": ["free-tier", "code", "general"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
"install_group": "codex",
},
"claude": {
@@ -75,6 +85,9 @@ KNOWN_PROVIDER_CLIS = {
"command": "claude -p",
"description": "Claude Code - auto-routes to best model",
"tags": ["paid", "subscription", "code"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
"install_group": "claude",
},
"crush": {
@@ -82,6 +95,9 @@ KNOWN_PROVIDER_CLIS = {
"command": "crush run --quiet",
"description": "Crush - multi-model via Hyper credits or API keys",
"tags": ["free-tier", "multi", "code"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
"install_group": "crush",
},
"ollama": {
@@ -89,6 +105,10 @@ KNOWN_PROVIDER_CLIS = {
"command": "ollama run llama3.2",
"description": "Ollama - local, private, free",
"tags": ["free", "local", "private"],
"model": "llama3.2",
"locality": "local",
"capabilities": ["text", "structured-json"],
"data_policy": "private",
"install_group": "ollama",
},
}
@@ -101,6 +121,9 @@ KNOWN_API_KEYS = {
"model": "openrouter/auto-beta",
"description": "OpenRouter - 300+ models, one API key",
"tags": ["api", "per-token", "multi"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
},
"DEEPSEEK_API_KEY": {
"name": "deepseek-api",
@@ -108,6 +131,9 @@ KNOWN_API_KEYS = {
"model": "deepseek-chat",
"description": "DeepSeek API - inexpensive per-token access",
"tags": ["api", "per-token", "cheap"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
},
"OPENAI_API_KEY": {
"name": "openai-api",
@@ -115,6 +141,9 @@ KNOWN_API_KEYS = {
"model": "gpt-4o",
"description": "OpenAI API - direct GPT access",
"tags": ["api", "per-token", "code"],
"locality": "remote",
"capabilities": ["text", "structured-json", "reasoning"],
"data_policy": "public",
},
}
@@ -180,6 +209,10 @@ def discover_installed_providers() -> List[dict]:
"command": f"ollama run {model_name}",
"description": f"Ollama local model: {model_name}",
"tags": ["free", "local", "private"],
"model": model_name,
"locality": "local",
"capabilities": ["text", "structured-json"],
"data_policy": "private",
})
except (subprocess.TimeoutExpired, FileNotFoundError):
pass
@@ -187,6 +220,26 @@ def discover_installed_providers() -> List[dict]:
return found
def _provider_from_discovery(data: dict) -> "Provider":
"""Convert trusted discovery facts without persisting scanner-only keys."""
return Provider(
name=data["name"],
command=data["command"],
description=data.get("description", ""),
type=data.get("type", "subprocess"),
model=data.get("model"),
tags=data.get("tags", []),
api_key_env=data.get("env_var"),
locality=data.get("locality", "unknown"),
capabilities=data.get("capabilities", []),
model_digest=data.get("model_digest"),
cost_class=data.get("cost_class"),
latency_class=data.get("latency_class"),
max_context_tokens=data.get("max_context_tokens"),
data_policy=data.get("data_policy", "unspecified"),
)
@dataclass
class Provider:
"""Definition of an AI provider.
@@ -212,6 +265,52 @@ class Provider:
fallback_chain: Optional[List[str]] = None # Ordered multi-step fallback (new)
api_key_env: Optional[str] = None # Env var name for api-type providers
pty_config: Optional[dict] = None # Patterns for pty-type providers
tools: Optional[List[str]] = None # Allowed CmdForge tools (None = all)
mcp_servers: Optional[List[str]] = None # MCP servers available to this provider
locality: str = "unknown" # "local" | "remote" | "unknown"
capabilities: List[str] = field(default_factory=list)
model_digest: Optional[str] = None
cost_class: Optional[str] = None # "free" | "low" | "medium" | "high"
latency_class: Optional[str] = None # "fast" | "standard" | "slow"
max_context_tokens: Optional[int] = None
data_policy: str = "unspecified" # max: public | internal | private
def __post_init__(self) -> None:
for field_name, values in (
("tools", self.tools),
("mcp_servers", self.mcp_servers),
):
if values is not None and (
not isinstance(values, list)
or not all(isinstance(value, str) and value for value in values)
):
raise ValueError(
f"Provider {field_name} must be a list of non-empty strings or null"
)
if self.locality not in {"local", "remote", "unknown"}:
raise ValueError("Provider locality must be local, remote, or unknown")
if (
not isinstance(self.capabilities, list)
or not all(isinstance(value, str) and value for value in self.capabilities)
):
raise ValueError("Provider capabilities must be a list of non-empty strings")
if self.cost_class not in {None, "free", "low", "medium", "high"}:
raise ValueError("Provider cost_class must be free, low, medium, or high")
if self.latency_class not in {None, "fast", "standard", "slow"}:
raise ValueError("Provider latency_class must be fast, standard, or slow")
if self.data_policy not in {"unspecified", "public", "internal", "private"}:
raise ValueError(
"Provider data_policy must be unspecified, public, internal, or private"
)
if (
self.max_context_tokens is not None
and (
not isinstance(self.max_context_tokens, int)
or isinstance(self.max_context_tokens, bool)
or self.max_context_tokens <= 0
)
):
raise ValueError("Provider max_context_tokens must be a positive integer")
def to_dict(self) -> dict:
d = {
@@ -236,6 +335,24 @@ class Provider:
d["api_key_env"] = self.api_key_env
if self.pty_config:
d["pty_config"] = self.pty_config
if self.tools is not None:
d["tools"] = self.tools
if self.mcp_servers is not None:
d["mcp_servers"] = self.mcp_servers
if self.locality != "unknown":
d["locality"] = self.locality
if self.capabilities:
d["capabilities"] = self.capabilities
if self.model_digest:
d["model_digest"] = self.model_digest
if self.cost_class:
d["cost_class"] = self.cost_class
if self.latency_class:
d["latency_class"] = self.latency_class
if self.max_context_tokens is not None:
d["max_context_tokens"] = self.max_context_tokens
if self.data_policy != "unspecified":
d["data_policy"] = self.data_policy
return d
@classmethod
@@ -245,13 +362,24 @@ class Provider:
command=data["command"],
description=data.get("description", ""),
type=data.get("type", "subprocess"),
model=data.get("model"),
model=data.get("model", data.get("model_id")),
fallback=data.get("fallback"),
tags=data.get("tags", []) or [],
install=data.get("install"),
fallback_chain=data.get("fallback_chain"),
api_key_env=data.get("api_key_env"),
pty_config=data.get("pty_config"),
tools=data.get("tools"),
mcp_servers=data.get("mcp_servers"),
locality=data.get("locality", "unknown"),
capabilities=data.get("capabilities", []) or [],
model_digest=data.get("model_digest"),
cost_class=data.get("cost_class"),
latency_class=data.get("latency_class"),
max_context_tokens=data.get(
"max_context_tokens", data.get("max_input_tokens")
),
data_policy=data.get("data_policy", "unspecified"),
)
@@ -261,6 +389,164 @@ class ProviderResult:
text: str
success: bool
error: Optional[str] = None
requested_provider: Optional[str] = None
actual_provider: Optional[str] = None
model: Optional[str] = None
model_digest: Optional[str] = None
attempted_providers: List[str] = field(default_factory=list)
fallback_used: bool = False
locality: str = "unknown"
model_identity_source: str = "unknown"
def provenance(self) -> dict:
"""Return runtime-owned execution facts safe for result envelopes."""
data = {
"success": self.success,
"requested_provider": self.requested_provider,
"actual_provider": self.actual_provider,
"attempted_providers": list(self.attempted_providers),
"fallback_used": self.fallback_used,
"model": self.model,
"model_digest": self.model_digest,
"locality": self.locality,
"model_identity_source": self.model_identity_source,
}
if self.error:
data["error"] = self.error
return data
@dataclass(frozen=True)
class ProviderExecutionPolicy:
"""Caller-supplied, fail-closed constraints for provider execution."""
allow_fallback: bool = True
required_locality: Optional[str] = None
required_capabilities: tuple[str, ...] = ()
data_classification: str = "public"
require_model_identity: bool = False
require_model_digest: bool = False
def __post_init__(self) -> None:
if self.required_locality not in {None, "local", "remote"}:
raise ValueError("required_locality must be local, remote, or null")
if self.data_classification not in {"public", "internal", "private"}:
raise ValueError(
"data_classification must be public, internal, or private"
)
_DATA_CLASS_LEVEL = {"public": 0, "internal": 1, "private": 2}
def effective_provider_locality(provider: Provider) -> str:
"""Account for Ollama clients redirected to a non-loopback host."""
try:
executable = shlex.split(os.path.expandvars(provider.command))[0]
except (IndexError, ValueError):
executable = ""
if Path(executable).name != "ollama":
return provider.locality
ollama_host = os.environ.get("OLLAMA_HOST", "").strip()
if not ollama_host:
return provider.locality
from urllib.parse import urlparse
parsed = urlparse(
ollama_host if "://" in ollama_host else f"//{ollama_host}"
)
hostname = (parsed.hostname or "").lower()
if hostname in {"localhost", "127.0.0.1", "::1"}:
return "local"
return "remote"
def runtime_provider_identity(provider: Provider) -> tuple[Optional[str], Optional[str], str]:
"""Return observed model identity when supported, else configured facts."""
configured = (
provider.model,
provider.model_digest,
"provider-config" if provider.model or provider.model_digest else "unknown",
)
try:
parts = shlex.split(os.path.expandvars(provider.command))
except ValueError:
return configured
if not parts or Path(parts[0]).name != "ollama":
return configured
if effective_provider_locality(provider) != "local":
return configured
try:
run_index = parts.index("run")
model = provider.model or parts[run_index + 1]
except (ValueError, IndexError):
return configured
host = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434").strip()
if "://" not in host:
host = f"http://{host}"
from urllib.request import Request, urlopen
import json
request = Request(host.rstrip("/") + "/api/tags")
api_key = os.environ.get("OLLAMA_API_KEY")
if api_key:
request.add_header("Authorization", f"Bearer {api_key}")
try:
with urlopen(request, timeout=2) as response:
payload = json.loads(response.read().decode("utf-8"))
except Exception:
return configured
wanted = {model, f"{model}:latest"}
for item in payload.get("models", []):
names = {item.get("name"), item.get("model")}
if names & wanted or any(
value and value.removesuffix(":latest") == model.removesuffix(":latest")
for value in names
):
digest = item.get("digest") or provider.model_digest
if digest and not digest.startswith("sha256:"):
digest = f"sha256:{digest}"
return item.get("model") or item.get("name") or model, digest, "ollama-api"
return configured
def provider_policy_error(
provider: Provider,
policy: ProviderExecutionPolicy,
runtime_identity: Optional[tuple[Optional[str], Optional[str], str]] = None,
) -> Optional[str]:
"""Return why a provider is ineligible, or None when it is allowed."""
if (
policy.required_locality is not None
and effective_provider_locality(provider) != policy.required_locality
):
return (
f"Provider '{provider.name}' locality is "
f"{effective_provider_locality(provider)}; "
f"{policy.required_locality} is required"
)
missing = sorted(set(policy.required_capabilities) - set(provider.capabilities))
if missing:
return (
f"Provider '{provider.name}' lacks required capabilities: "
+ ", ".join(missing)
)
if policy.data_classification != "public":
allowed = _DATA_CLASS_LEVEL.get(provider.data_policy, -1)
needed = _DATA_CLASS_LEVEL[policy.data_classification]
if allowed < needed:
return (
f"Provider '{provider.name}' data_policy is {provider.data_policy}; "
f"{policy.data_classification} data requires an explicit compatible policy"
)
model, digest, _ = runtime_identity or (
provider.model, provider.model_digest, "provider-config"
)
if policy.require_model_identity and not model:
return f"Provider '{provider.name}' does not declare a model identity"
if policy.require_model_digest and not digest:
return f"Provider '{provider.name}' does not declare a model digest"
return None
# Default providers that come pre-configured
@@ -312,7 +598,12 @@ DEFAULT_PROVIDERS = [
# LOCAL MODELS
Provider("ollama", "ollama run llama3.2",
"Ollama - local, private, free (GPU recommended)",
tags=["free", "local", "private"]),
model="llama3.2",
tags=["free", "local", "private"],
locality="local",
capabilities=["text", "structured-json"],
cost_class="free",
data_policy="private"),
# API-TYPE PROVIDERS (pay-per-token, fallback when no CLI covers model)
Provider("openrouter",
@@ -321,18 +612,26 @@ DEFAULT_PROVIDERS = [
type="api",
model="openrouter/auto-beta",
api_key_env="OPENROUTER_API_KEY",
tags=["api", "per-token", "multi", "fallback"]),
tags=["api", "per-token", "multi", "fallback"],
locality="remote",
capabilities=["text", "structured-json", "reasoning"],
data_policy="public"),
Provider("deepseek-api",
"https://api.deepseek.com/v1",
"DeepSeek API - inexpensive per-token access",
type="api",
model="deepseek-chat",
api_key_env="DEEPSEEK_API_KEY",
tags=["api", "per-token", "cheap", "reasoning"]),
tags=["api", "per-token", "cheap", "reasoning"],
locality="remote",
capabilities=["text", "structured-json", "reasoning"],
data_policy="public"),
# Mock for testing
Provider("mock", "mock", "Mock provider for testing",
tags=["testing"]),
model="mock", tags=["testing"], locality="local",
capabilities=["text", "structured-json"],
data_policy="private"),
]
@@ -343,7 +642,6 @@ def get_providers_file() -> Path:
discovered = discover_installed_providers()
selected = []
if discovered:
import sys
print("=" * 60, file=sys.stderr)
print("CmdForge First-Run Provider Setup", file=sys.stderr)
print("=" * 60, file=sys.stderr)
@@ -357,10 +655,7 @@ def get_providers_file() -> Path:
for d in cli_found:
name = d["name"]
desc = d.get("description", "")
selected.append(Provider(
name=name, command=d["command"], description=desc,
tags=d.get("tags", []),
))
selected.append(_provider_from_discovery(d))
print(f" [+] {name:20s} {desc}", file=sys.stderr)
if api_found:
@@ -370,11 +665,7 @@ def get_providers_file() -> Path:
desc = d.get("description", "")
model = d.get("model", "auto")
env_var = d.get("env_var", "")
selected.append(Provider(
name=name, command=d["command"], description=desc,
type="api", model=model, api_key_env=env_var,
tags=d.get("tags", []),
))
selected.append(_provider_from_discovery(d))
print(f" [+] {name:20s} ({model})", file=sys.stderr)
if ollama_found:
@@ -382,10 +673,7 @@ def get_providers_file() -> Path:
for d in ollama_found[:10]:
name = d["name"]
desc = d.get("description", "")
selected.append(Provider(
name=name, command=d["command"], description=desc,
tags=d.get("tags", []),
))
selected.append(_provider_from_discovery(d))
print(f" [+] {name:30s} {d['command']}", file=sys.stderr)
if len(ollama_found) > 10:
print(f" ... and {len(ollama_found) - 10} more", file=sys.stderr)
@@ -411,8 +699,11 @@ def load_providers() -> List[Provider]:
if not data or "providers" not in data:
return DEFAULT_PROVIDERS.copy()
providers = [Provider.from_dict(p) for p in data["providers"]]
except Exception:
return DEFAULT_PROVIDERS.copy()
except Exception as exc:
# A present but malformed configuration must not silently fall back to
# unrestricted defaults, particularly when access policies are invalid.
print(f"Warning: Failed to load provider configuration: {exc}", file=sys.stderr)
return []
config_version = data.get("version", 1)
if not isinstance(config_version, int):
@@ -489,7 +780,17 @@ def delete_provider(name: str) -> bool:
return False
def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_tokens: Optional[int] = None, _tried: Optional[set] = None) -> ProviderResult:
def call_provider(
provider_name: str,
prompt: str,
timeout: int = 300,
max_tokens: Optional[int] = None,
_tried: Optional[set] = None,
*,
execution_policy: Optional[ProviderExecutionPolicy] = None,
_requested_provider: Optional[str] = None,
_attempted_providers: Optional[List[str]] = None,
) -> ProviderResult:
"""
Call an AI provider with the given prompt.
@@ -503,59 +804,110 @@ def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_token
timeout: Maximum execution time in seconds
max_tokens: Optional max output tokens (appends provider-specific flag)
_tried: Internal set of already-tried providers (prevents infinite loops)
execution_policy: Optional fallback and eligibility constraints
Returns:
ProviderResult with the response text or error
"""
policy = execution_policy or ProviderExecutionPolicy()
requested_provider = _requested_provider or provider_name
attempted = _attempted_providers if _attempted_providers is not None else []
if provider_name not in attempted:
attempted.append(provider_name)
observed_identity = ("mock", None, "built-in")
# Track which providers we've tried to prevent infinite fallback loops
if _tried is None:
_tried = set()
_tried.add(provider_name)
def finish(
result: ProviderResult,
provider: Optional[Provider] = None,
) -> ProviderResult:
"""Attach facts observed by CmdForge, never values generated by the model."""
result.requested_provider = requested_provider
result.attempted_providers = list(attempted)
if result.success:
result.actual_provider = provider_name
(
result.model,
result.model_digest,
result.model_identity_source,
) = observed_identity
result.locality = (
effective_provider_locality(provider) if provider else "local"
)
result.fallback_used = provider_name != requested_provider
return result
# Handle mock provider specially
if provider_name.lower() == "mock":
return mock_provider(prompt)
return finish(mock_provider(prompt))
# Look up provider
provider = get_provider(provider_name)
if not provider:
return ProviderResult(
return finish(ProviderResult(
text="",
success=False,
error=f"Provider '{provider_name}' not found. Use 'cmdforge providers' to manage providers."
)
))
observed_identity = runtime_provider_identity(provider)
if max_tokens is not None:
try:
max_tokens = int(max_tokens)
except (TypeError, ValueError):
return ProviderResult(text="", success=False, error="max_tokens must be an integer")
return finish(ProviderResult(text="", success=False, error="max_tokens must be an integer"), provider)
if max_tokens <= 0 or max_tokens > 1_000_000:
return ProviderResult(
return finish(ProviderResult(
text="", success=False, error="max_tokens must be between 1 and 1000000"
)
), provider)
# Helper to try fallback provider(s) if available
def try_fallback(error_msg: str) -> ProviderResult:
import sys
last_error = error_msg
if not policy.allow_fallback:
return finish(
ProviderResult(text="", success=False, error=last_error), provider
)
# Walk fallback_chain first (ordered, multi-step)
if provider.fallback_chain:
for fb in provider.fallback_chain:
if fb not in _tried:
print(f"[fallback] {provider_name} failed, trying {fb}...", file=sys.stderr)
result = call_provider(fb, prompt, timeout, max_tokens, _tried)
result = call_provider(
fb, prompt, timeout, max_tokens, _tried,
execution_policy=policy,
_requested_provider=requested_provider,
_attempted_providers=attempted,
)
if result.success:
return result
last_error = result.error or last_error
# Fall back to single fallback (backward-compat)
if provider.fallback and provider.fallback not in _tried:
print(f"[fallback] {provider_name} failed, trying {provider.fallback}...", file=sys.stderr)
result = call_provider(provider.fallback, prompt, timeout, max_tokens, _tried)
result = call_provider(
provider.fallback, prompt, timeout, max_tokens, _tried,
execution_policy=policy,
_requested_provider=requested_provider,
_attempted_providers=attempted,
)
if result.success:
return result
last_error = result.error or last_error
return ProviderResult(text="", success=False, error=last_error)
return finish(
ProviderResult(text="", success=False, error=last_error), provider
)
eligibility_error = provider_policy_error(
provider, policy, observed_identity
)
if eligibility_error:
return try_fallback(eligibility_error)
# Dispatch by provider type
ptype = getattr(provider, "type", None) or "subprocess"
@@ -570,7 +922,7 @@ def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_token
return try_fallback(f"Unknown provider type: {ptype}")
if result.success:
return result
return finish(result, provider)
return try_fallback(result.error or "Provider call failed")
except Exception as e:
return try_fallback(f"Provider error: {str(e)}")
+248
View File
@@ -0,0 +1,248 @@
"""Explainable quality scores for CmdForge tools.
Produces a headline score with a per-category breakdown. Each category shows
earned/available points, and missing evidence is shown as an opportunity
rather than a penalty.
"""
from dataclasses import dataclass, field
from typing import Dict, List, Optional
from .tool import Tool
from .preflight import PreflightReport
@dataclass
class CategoryScore:
"""One quality category."""
name: str
earned: int
available: int
state: str = "checked" # checked | not_tested | not_applicable
possible: Optional[int] = None # Category maximum when only partial evidence ran
@property
def display(self) -> str:
if self.state == "not_tested":
return "not tested"
if self.state == "not_applicable":
return "n/a"
return f"{self.earned}/{self.available}"
@property
def percentage(self) -> float:
return (self.earned / self.available * 100) if self.available > 0 else 0.0
@dataclass
class QualityScore:
"""Headline score with category breakdown."""
tool_name: str
version: str
headline: int # 0-100
categories: List[CategoryScore] = field(default_factory=list)
last_evaluated: str = ""
evidence_coverage: int = 0
def to_dict(self) -> dict:
return {
"tool": self.tool_name,
"version": self.version,
"score": self.headline,
"categories": [
{
"name": c.name,
"earned": c.earned,
"available": c.available,
"state": c.state,
"possible": c.possible if c.possible is not None else c.available,
}
for c in self.categories
],
"last_evaluated": self.last_evaluated,
"evidence_coverage": self.evidence_coverage,
}
def __str__(self) -> str:
lines = [f"Quality {self.headline} — {self.evidence_coverage}% evidence coverage"]
if self.last_evaluated:
lines[0] += f" — Evaluated {self.last_evaluated}"
for c in self.categories:
lines.append(f" {c.name:24s} {c.display:>8s}")
return "\n".join(lines)
def compute_quality(
tool: Tool,
report: PreflightReport,
registry_data: Optional[dict] = None,
) -> QualityScore:
"""Compute an explainable quality score from preflight evidence.
Categories:
- Contracts (15 pts): input_schema and output_schema present and valid
- Deterministic tests (30 pts): conformance test pass rate
- Regression history (20 pts): no regressions in baseline comparison
- Security scrutiny (20 pts): no secrets, no unresolved dependencies
- Community evidence (15 pts): registry reviews, downloads
Missing evidence shows as "not_tested" and does not penalize.
"""
from datetime import datetime, timezone
categories = []
categories.append(_score_contracts(tool, report))
categories.append(_score_tests(report))
categories.append(_score_regression(report))
categories.append(_score_security(report, tool))
categories.append(_score_community(registry_data))
checked = [c for c in categories if c.state == "checked"]
earned = sum(c.earned for c in checked)
available = sum(c.available for c in checked)
possible = sum(
c.possible if c.possible is not None else c.available
for c in categories if c.state != "not_applicable"
)
headline = int((earned / available * 100)) if available > 0 else 0
return QualityScore(
tool_name=tool.name,
version=tool.version or "",
headline=headline,
categories=categories,
last_evaluated=datetime.now(timezone.utc).isoformat(),
evidence_coverage=int((available / possible * 100)) if possible else 0,
)
def _score_contracts(tool: Tool, report: PreflightReport) -> CategoryScore:
available = 15
earned = 0
if tool.input_schema is not None:
earned += 5
if tool.output_schema is not None:
earned += 5
if tool.input_schema is None and tool.output_schema is None:
return CategoryScore("Contracts", 0, available, "not_tested")
# No contract-related errors in the preflight
contract_errors = [
e for e in report.errors if "schema" in e.lower() or "contract" in e.lower()
]
if not contract_errors:
earned += 5
else:
earned = max(0, earned - len(contract_errors))
return CategoryScore("Contracts", earned, available, "checked")
def _score_tests(report: PreflightReport) -> CategoryScore:
maximum = 30
tests = report.generated_tests or []
if not tests:
return CategoryScore("Deterministic tests", 0, maximum, "not_tested")
executed = [t for t in tests if t.get("state") in ("passed", "failed")]
if not executed:
return CategoryScore("Deterministic tests", 0, maximum, "not_tested")
passed = sum(1 for t in executed if t.get("state") == "passed")
total = len(executed)
available = max(1, int(maximum * total / len(tests)))
earned = int((passed / total) * available)
return CategoryScore(
"Deterministic tests", earned, available, "checked", possible=maximum
)
def _score_regression(report: PreflightReport) -> CategoryScore:
available = 20
regression = report.regression
if not regression:
return CategoryScore("Regression history", 0, available, "not_tested")
if regression.get("has_regressions"):
return CategoryScore("Regression history", 0, available, "checked")
if regression.get("summary") and "no changes" in regression.get("summary", "").lower():
return CategoryScore("Regression history", available, available, "checked")
# Improvements or new tests — partial credit
return CategoryScore("Regression history", available, available, "checked")
def _score_security(report: PreflightReport, tool: Tool) -> CategoryScore:
maximum = 20
checks = set((report.audit_evidence or {}).get("checks_run", []))
if "security_scrutiny" in checks:
available = maximum
elif checks:
available = (10 if "secrets" in checks else 0) + (
10 if checks.intersection({"dependencies", "registry_dependencies"}) else 0
)
elif report.errors or report.warnings:
# Backward-compatible evidence supplied by callers without audit metadata.
available = maximum
else:
available = 0
if not available:
return CategoryScore(
"Security scrutiny", 0, maximum, "not_tested", possible=maximum
)
earned = available
# Deduct for secret patterns found
secret_warnings = [
w for w in report.warnings if "secret" in w.lower()
]
earned -= len(secret_warnings) * 5
# Deduct for unresolved dependencies
dependency_findings = [
finding for finding in report.warnings + report.errors
if ("dependency" in finding.lower() or "dependencies" in finding.lower())
and any(
marker in finding.lower()
for marker in ("not installed", "missing", "deprecated")
)
]
earned -= len(dependency_findings) * 3
earned = max(0, earned)
return CategoryScore(
"Security scrutiny", earned, available, "checked", possible=maximum
)
def _score_community(registry_data: Optional[dict]) -> CategoryScore:
available = 15
if not registry_data:
return CategoryScore("Community evidence", 0, available, "not_tested")
earned = 0
reviews = registry_data.get("reviews", [])
rating_count = registry_data.get("rating_count", len(reviews))
avg_rating = registry_data.get("average_rating")
if avg_rating is None and reviews:
avg_rating = sum(r.get("rating", 0) for r in reviews) / len(reviews)
if avg_rating is not None and rating_count:
earned += int((avg_rating / 5) * 8)
downloads = registry_data.get("downloads", 0)
if downloads > 100:
earned += 4
elif downloads > 10:
earned += 2
if registry_data.get("featured"):
earned += 3
earned = min(earned, available)
return CategoryScore("Community evidence", earned, available, "checked")
+569 -18
View File
@@ -9,10 +9,11 @@ import os
import re
import secrets
from dataclasses import dataclass
from datetime import date, datetime, timedelta
from datetime import date, datetime, timedelta, timezone
from typing import Any, Dict, Iterable, List, Optional, Tuple
from urllib.parse import urlsplit
from flask import Flask, Response, g, jsonify, request
from flask import Flask, Response, current_app, g, jsonify, request
import yaml
from functools import wraps
from argon2 import PasswordHasher
@@ -67,6 +68,11 @@ rate_limiter = RateLimiter()
password_hasher = PasswordHasher(memory_cost=65536, time_cost=3, parallelism=4)
def _naive_utc_now() -> datetime:
"""Return UTC without tzinfo for compatibility with existing DB values."""
return datetime.now(timezone.utc).replace(tzinfo=None)
@dataclass(frozen=True)
class Semver:
major: int
@@ -275,6 +281,14 @@ def run_ai_scrutiny_review(scrutiny_report: dict, config: dict, tool_name: str,
def create_app() -> Flask:
app = Flask(__name__)
app.config["MAX_CONTENT_LENGTH"] = MAX_BODY_BYTES
app.config.update(
CMDFORGE_ENV=os.environ.get("CMDFORGE_ENV", "development"),
PUBLIC_BASE_URL=os.environ.get("CMDFORGE_PUBLIC_URL", "").rstrip("/"),
MAIL_TRANSPORT=os.environ.get("MAIL_TRANSPORT", "disabled"),
MAIL_FROM=os.environ.get("MAIL_FROM", ""),
MAIL_TIMEOUT=os.environ.get("MAIL_TIMEOUT", "10"),
RESEND_API_KEY=os.environ.get("RESEND_API_KEY", ""),
)
# Initialize database schema once at startup
with connect_db() as init_conn:
@@ -418,7 +432,7 @@ def create_app() -> Flask:
g.db.execute(
"UPDATE api_tokens SET last_used_at = ? WHERE id = ?",
[datetime.utcnow().isoformat(), row["id"]],
[datetime.now(timezone.utc).isoformat(), row["id"]],
)
g.current_publisher = {
"id": row["publisher_id"],
@@ -870,6 +884,14 @@ def create_app() -> Flask:
order_sql = f"{sort} {order_dir}, published_at DESC, f.id DESC"
stats_join = "LEFT JOIN tool_stats ts ON ts.tool_id = f.id"
audit_join = """
LEFT JOIN tool_audits ta ON ta.id = (
SELECT latest_audit.id FROM tool_audits latest_audit
WHERE latest_audit.tool_id = f.id
ORDER BY latest_audit.evaluated_at DESC, latest_audit.id DESC
LIMIT 1
)
"""
rows = query_all(
g.db,
@@ -897,9 +919,13 @@ def create_app() -> Flask:
GROUP BY owner, name
)
SELECT f.*, COALESCE(ts.average_rating, 0) AS average_rating,
COALESCE(ts.rating_count, 0) AS rating_count
COALESCE(ts.rating_count, 0) AS rating_count,
json_extract(ta.quality_json, '$.score') AS quality_score,
json_extract(ta.quality_json, '$.evidence_coverage') AS quality_coverage,
ta.evaluated_at AS quality_evaluated_at
FROM filtered f
{stats_join}
{audit_join}
JOIN (
SELECT a.owner, a.name, COALESCE(s.max_id, a.max_id) AS max_id
FROM latest_any a
@@ -948,6 +974,12 @@ def create_app() -> Flask:
"average_rating": row["average_rating"],
"rating_count": row["rating_count"],
"score": score,
"quality_score": row["quality_score"],
"quality_coverage": row["quality_coverage"],
"quality_evaluated_at": row["quality_evaluated_at"],
"deprecated": bool(row["deprecated"]),
"deprecated_message": row["deprecated_message"],
"replacement": row["replacement"],
})
result: dict = {"data": data, "meta": paginate(page, per_page, total)}
@@ -1089,6 +1121,58 @@ def create_app() -> Flask:
if fork_row:
fork_count = fork_row["cnt"]
replacement_chain = []
if row["replacement"]:
try:
replacement_chain = _get_replacement_chain(
row["owner"], row["name"], row["replacement"]
)
except ValueError:
# Preserve access to legacy rows while refusing new invalid chains.
replacement_chain = []
latest_audit = query_one(
g.db,
"""
SELECT engine_version, evidence_json, quality_json, evaluated_at
FROM tool_audits
WHERE tool_id = ?
ORDER BY evaluated_at DESC, id DESC
LIMIT 1
""",
[row["id"]],
)
audit_stale = True
if latest_audit:
try:
evaluated = datetime.fromisoformat(
latest_audit["evaluated_at"].replace("Z", "+00:00")
)
if evaluated.tzinfo is None:
evaluated = evaluated.replace(tzinfo=timezone.utc)
audit_stale = datetime.now(timezone.utc) - evaluated > timedelta(days=30)
except (AttributeError, TypeError, ValueError):
audit_stale = True
contributor_rows = query_all(
g.db,
"""SELECT p.slug, c.contribution FROM tool_contributors c
JOIN publishers p ON p.id = c.publisher_id
WHERE c.tool_id = ? ORDER BY p.slug""",
[row["id"]],
)
approved_improvement = query_one(
g.db,
"""SELECT id FROM improvement_submissions
WHERE applied_tool_id = ? LIMIT 1""",
[row["id"]],
)
badges = []
if row.get("attestation_json"):
badges.append("verified")
if approved_improvement:
badges.extend(["optimized", "community-reviewed"])
payload = {
"owner": row["owner"],
"name": row["name"],
@@ -1100,7 +1184,8 @@ def create_app() -> Flask:
"published_at": row["published_at"],
"deprecated": bool(row["deprecated"]),
"deprecated_message": row["deprecated_message"],
"replacement": row["replacement"],
"replacement": row["replacement"] if replacement_chain else None,
"replacement_chain": replacement_chain,
"config": row["config_yaml"],
"readme": row["readme"],
"defaults": row.get("defaults") or "",
@@ -1108,6 +1193,15 @@ def create_app() -> Flask:
"forked_from": row.get("forked_from"),
"forked_version": row.get("forked_version"),
"fork_count": fork_count,
"audit": json.loads(latest_audit["evidence_json"]) if latest_audit else None,
"quality": json.loads(latest_audit["quality_json"]) if latest_audit else None,
"audit_evaluated_at": latest_audit["evaluated_at"] if latest_audit else None,
"audit_stale": audit_stale,
"contributors": [
{"name": item["slug"], "contribution": item["contribution"]}
for item in contributor_rows
],
"badges": badges,
}
response = jsonify({"data": payload})
response.headers["Cache-Control"] = "max-age=60"
@@ -1257,6 +1351,18 @@ def create_app() -> Flask:
except Exception:
g.db.rollback()
publisher = query_one(
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
[row["publisher_id"]],
)
attestation = None
if row.get("attestation_json"):
try:
attestation = json.loads(row["attestation_json"])
except (TypeError, json.JSONDecodeError):
return error_response(
"INVALID_ATTESTATION", "Stored release attestation is corrupt", 500
)
response = jsonify({
"data": {
"owner": row["owner"],
@@ -1265,12 +1371,172 @@ def create_app() -> Flask:
"config": row["config_yaml"],
"readme": row["readme"] or "",
"config_hash": row.get("config_hash") or "",
"content_hash": row.get("content_hash") or "",
"dependency_hashes": json.loads(
row.get("dependency_hashes_json") or "{}"
),
"defaults": row.get("defaults") or "",
"attestation": attestation,
"signing_public_key": (
publisher["signing_public_key"] if publisher else ""
) or "",
}
})
response.headers["Cache-Control"] = "max-age=3600, immutable"
return response
@app.route("/api/v1/tools/by-content-hash/<path:content_hash>", methods=["GET"])
def tool_by_content_hash(content_hash: str) -> Response:
"""Resolve an approved public release by its transitive identity."""
if not re.fullmatch(r"sha256:[0-9a-f]{64}", content_hash):
return error_response("VALIDATION_ERROR", "Invalid content hash", 400)
row = query_one(
g.db,
"""SELECT owner, name, version, description, category, tags,
downloads, published_at
FROM tools WHERE content_hash = ? AND visibility = 'public'
AND moderation_status = 'approved'
ORDER BY published_at DESC LIMIT 1""",
[content_hash],
)
if not row:
return error_response("TOOL_NOT_FOUND", "Content hash not found", 404)
result = dict(row)
result["tags"] = json.loads(result.get("tags") or "[]")
return jsonify({"data": result})
@app.route(
"/api/v1/tools/<owner>/<name>/<version>/improvements", methods=["POST"]
)
@require_token
def submit_improvement(owner: str, name: str, version: str) -> Response:
"""Submit and automatically test a prompt/code improvement."""
row = query_one(
g.db,
"""SELECT id, config_yaml, visibility, owner FROM tools
WHERE owner = ? AND name = ? AND version = ?
AND moderation_status = 'approved'""",
[owner, name, version],
)
if not row:
return error_response("TOOL_NOT_FOUND", "Tool release not found", 404)
if (
row["visibility"] != "public"
and g.current_publisher["slug"] != row["owner"]
and g.current_publisher.get("role") not in ("moderator", "admin")
):
return error_response("TOOL_NOT_FOUND", "Tool release not found", 404)
payload = request.get_json(silent=True) or {}
try:
from ..community import create_submission, run_submission_tests
from ..tool import Tool
tool = Tool.from_dict(yaml.safe_load(row["config_yaml"]) or {})
submission = create_submission(
tool,
int(payload.get("step_index", -1)),
str(payload.get("proposed", "")),
g.current_publisher["slug"],
str(payload.get("rationale", "")),
)
if not submission.proposed.strip():
raise ValueError("Proposed content must not be empty")
test_result = run_submission_tests(submission, tool)
except (KeyError, TypeError, ValueError) as exc:
return error_response("VALIDATION_ERROR", str(exc), 400)
if not test_result.get("passed_for_review"):
return error_response(
"IMPROVEMENT_TEST_FAILED",
test_result.get("error") or "Proposed change did not pass automated tests",
400,
details={"test_result": test_result},
)
cursor = g.db.execute(
"""INSERT INTO improvement_submissions (
tool_id, submitter_id, step_index, step_type, original,
proposed, rationale, status, test_result_json, submitted_at
) VALUES (?, ?, ?, ?, ?, ?, ?, 'tested', ?, ?)""",
[
row["id"], g.current_publisher["id"], submission.step_index,
submission.step_type, submission.original, submission.proposed,
submission.rationale, json.dumps(test_result, sort_keys=True),
submission.submitted_at,
],
)
g.db.commit()
return jsonify({
"data": {"id": cursor.lastrowid, **submission.to_dict()}
}), 201
@app.route("/api/v1/improvements/<int:submission_id>", methods=["PATCH"])
@require_token
def review_improvement(submission_id: int) -> Response:
"""Allow the tool owner/moderator to review a tested submission."""
row = query_one(
g.db,
"""SELECT s.*, t.owner FROM improvement_submissions s
JOIN tools t ON t.id = s.tool_id WHERE s.id = ?""",
[submission_id],
)
if not row:
return error_response("NOT_FOUND", "Improvement not found", 404)
if (
g.current_publisher["slug"] != row["owner"]
and g.current_publisher.get("role") not in ("moderator", "admin")
):
return error_response("FORBIDDEN", "Owner or moderator required", 403)
decision = (request.get_json(silent=True) or {}).get("decision")
if decision not in ("approve", "reject", "request_changes"):
return error_response("VALIDATION_ERROR", "Invalid decision", 400)
status = "approved" if decision == "approve" else decision
now = datetime.now(timezone.utc).isoformat()
notes = (request.get_json(silent=True) or {}).get("notes", "")
g.db.execute(
"""UPDATE improvement_submissions SET status = ?, reviewer_id = ?,
review_notes = ?, reviewed_at = ? WHERE id = ?""",
[status, g.current_publisher["id"], notes, now, submission_id],
)
g.db.commit()
return jsonify({
"data": {
"id": submission_id, "status": status,
"ready_to_apply": status == "approved",
}
})
@app.route("/api/v1/me/signing-key", methods=["PUT"])
@require_token
def set_signing_key() -> Response:
"""Register an Ed25519 public key as the publisher trust anchor."""
public_key = (request.get_json(silent=True) or {}).get("public_key", "")
try:
import base64
raw = base64.b64decode(public_key, validate=True)
if len(raw) != 32:
raise ValueError
except (ValueError, TypeError):
return error_response(
"VALIDATION_ERROR", "public_key must be a base64 Ed25519 key", 400
)
existing_key = query_one(
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
[g.current_publisher["id"]],
)
current_key = (
existing_key["signing_public_key"] if existing_key else ""
) or ""
if current_key and current_key != public_key:
return error_response(
"SIGNING_KEY_EXISTS",
"Signing key rotation requires administrator recovery",
409,
)
g.db.execute(
"UPDATE publishers SET signing_public_key = ? WHERE id = ?",
[public_key, g.current_publisher["id"]],
)
g.db.commit()
return jsonify({"data": {"signing_public_key": public_key}})
@app.route("/api/v1/categories", methods=["GET"])
def list_categories() -> Response:
page, per_page, sort, order, error = parse_pagination("/categories", "name")
@@ -2048,7 +2314,7 @@ def create_app() -> Flask:
# Generate a secure token
token, token_hash = generate_token()
expires_at = datetime.utcnow() + timedelta(hours=1)
expires_at = _naive_utc_now() + timedelta(hours=1)
g.db.execute(
"""
@@ -2059,10 +2325,44 @@ def create_app() -> Flask:
)
g.db.commit()
# Send email (logs to console in dev mode)
# Use a configured canonical URL in production. Falling back to the
# request URL is development-only because Host headers are untrusted.
from cmdforge.web.email import send_password_reset_email
base_url = request.url_root.rstrip("/")
send_password_reset_email(publisher["email"], token, base_url)
base_url = str(current_app.config.get("PUBLIC_BASE_URL", "")).rstrip("/")
environment = str(current_app.config.get("CMDFORGE_ENV", "development")).lower()
if not base_url and environment != "production":
base_url = request.url_root.rstrip("/")
parsed_base = urlsplit(base_url)
base_url_is_safe = bool(
base_url
and parsed_base.scheme in {"http", "https"}
and parsed_base.netloc
and parsed_base.username is None
and parsed_base.password is None
and not parsed_base.query
and not parsed_base.fragment
and (environment != "production" or parsed_base.scheme == "https")
)
try:
delivered = base_url_is_safe and send_password_reset_email(
publisher["email"], token, base_url
)
except Exception:
# Preserve the non-enumerating response while ensuring the token is
# invalidated below. Flask logs the exception without email/token
# values because they are not part of this message.
current_app.logger.exception("Password reset delivery raised unexpectedly")
delivered = False
if not delivered:
# Never leave a usable reset token in the database when its only
# intended recipient could not receive it.
g.db.execute(
"UPDATE password_reset_tokens SET used_at = CURRENT_TIMESTAMP WHERE token_hash = ?",
[token_hash],
)
g.db.commit()
return success_response
@@ -2094,7 +2394,7 @@ def create_app() -> Flask:
try:
expires_at = datetime.fromisoformat(row["expires_at"])
if datetime.utcnow() > expires_at:
if _naive_utc_now() > expires_at:
return error_response("TOKEN_EXPIRED", "This reset token has expired", 400)
except ValueError:
return error_response("INVALID_TOKEN", "Invalid token data", 400)
@@ -2136,7 +2436,7 @@ def create_app() -> Flask:
try:
expires_at = datetime.fromisoformat(row["expires_at"])
if datetime.utcnow() > expires_at:
if _naive_utc_now() > expires_at:
return error_response("TOKEN_EXPIRED", "This reset token has expired", 400)
except ValueError:
return error_response("INVALID_TOKEN", "Invalid token data", 400)
@@ -2414,6 +2714,8 @@ def create_app() -> Flask:
readme = payload.get("readme") or ""
defaults = payload.get("defaults") or ""
dry_run = bool(payload.get("dry_run"))
attestation_data = payload.get("attestation")
improvement_id = payload.get("improvement_id")
size_resp = validate_payload_size("config", config_text, MAX_CONFIG_BYTES)
if size_resp:
@@ -2436,6 +2738,10 @@ def create_app() -> Flask:
data = yaml.safe_load(config_text) or {}
except yaml.YAMLError:
return error_response("VALIDATION_ERROR", "Invalid YAML in config")
if not isinstance(data, dict):
return error_response(
"VALIDATION_ERROR", "Tool config must contain a YAML mapping"
)
name = (data.get("name") or "").strip()
version = (data.get("version") or "").strip()
@@ -2485,6 +2791,30 @@ def create_app() -> Flask:
if len(str(tag)) > MAX_TAG_LEN:
return error_response("VALIDATION_ERROR", "Tag exceeds 32 characters")
# Run the same deterministic contract/configuration checks used by the
# CLI and builder. Registry hosts intentionally skip local-install
# dependency checks because publisher dependencies are resolved by name.
try:
from ..preflight import analyze_tool as run_preflight
from ..tool import Tool
published_tool = Tool.from_dict(data)
preflight_report = run_preflight(
published_tool,
check_local_dependencies=False,
include_contract_tests=True,
check_reuse=False,
)
except (KeyError, TypeError, ValueError) as exc:
return error_response("INVALID_CONFIG", str(exc), 400)
if not preflight_report.ok:
return error_response(
"PREFLIGHT_FAILED",
preflight_report.errors[0],
400,
details={"preflight": preflight_report.to_dict()},
)
# Determine owner - admins can publish as "official" or other owners
owner = g.current_publisher["slug"]
requested_owner = payload.get("owner", "").strip()
@@ -2514,12 +2844,104 @@ def create_app() -> Flask:
)
owner = requested_owner
from .auditing import registry_dependency_findings
dependency_findings = registry_dependency_findings(
g.db, published_tool, owner
)
missing_dependencies = dependency_findings["missing"]
preflight_report.warnings.extend(dependency_findings["deprecated"])
from ..preflight import _add_audit_evidence
audit_checks = list(
(preflight_report.audit_evidence or {}).get("checks_run", [])
)
_add_audit_evidence(
published_tool, preflight_report,
audit_checks + ["registry_dependencies"],
)
from ..quality import compute_quality
quality_report = compute_quality(
published_tool, preflight_report
).to_dict()
if missing_dependencies:
message = "Missing registry dependencies: " + ", ".join(
missing_dependencies
)
preflight_report.errors.append(message)
_add_audit_evidence(
published_tool, preflight_report,
audit_checks + ["registry_dependencies"],
)
return error_response(
"PREFLIGHT_FAILED", message, 400,
details={"preflight": preflight_report.to_dict()},
)
if data.get("deprecated") and data.get("replacement"):
try:
_get_replacement_chain(owner, name, data["replacement"])
except ValueError as exc:
return error_response("VALIDATION_ERROR", str(exc), 400)
# Compute config hash early for idempotency check
config_hash = compute_yaml_hash(config_text)
from ..integrity import compute_content_identity
from ..tool import ToolStep
dependency_names = set(published_tool.dependencies)
dependency_names.update(
step.tool for step in published_tool.steps if isinstance(step, ToolStep)
)
dependency_hashes = {}
for dependency_name in sorted(dependency_names):
from ..resolver import ToolSpec
dep_spec = ToolSpec.parse(dependency_name)
dep_owner = dep_spec.owner or owner
dep_row = resolve_tool(dep_owner, dep_spec.name, dep_spec.version)
if dep_row:
dep_identity = dep_row.get("content_hash") or dep_row.get("config_hash")
if dep_identity:
dependency_hashes[dependency_name] = dep_identity
content_hash = compute_content_identity(config_hash, dependency_hashes)
publisher_record = query_one(
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
[g.current_publisher["id"]],
)
signing_public_key = (
publisher_record["signing_public_key"] if publisher_record else ""
) or ""
attestation_json = None
if signing_public_key and not dry_run and not attestation_data:
return error_response(
"ATTESTATION_REQUIRED",
"This publisher has a signing key; releases must be signed",
400,
)
if attestation_data:
try:
from ..attestation import Attestation, verify_attestation
attestation = Attestation.from_dict(attestation_data)
except (KeyError, TypeError, ValueError):
return error_response(
"INVALID_ATTESTATION", "Malformed release attestation", 400
)
if (
attestation.tool_name != name
or attestation.version != version
or attestation.content_hash != content_hash
or attestation.signer != owner
or not signing_public_key
or not verify_attestation(attestation, signing_public_key)
):
return error_response(
"INVALID_ATTESTATION",
"Release attestation does not match this publisher and content",
400,
)
attestation_json = json.dumps(attestation.to_dict(), sort_keys=True)
existing = query_one(
g.db,
"SELECT published_at, config_hash, moderation_status, visibility FROM tools WHERE owner = ? AND name = ? AND version = ?",
"SELECT published_at, config_hash, content_hash, moderation_status, visibility FROM tools WHERE owner = ? AND name = ? AND version = ?",
[owner, name, version],
)
if existing:
@@ -2531,6 +2953,7 @@ def create_app() -> Flask:
"name": name,
"version": version,
"config_hash": config_hash,
"content_hash": existing.get("content_hash") or content_hash,
"pr_url": "",
"status": existing["moderation_status"],
"visibility": existing["visibility"],
@@ -2545,6 +2968,43 @@ def create_app() -> Flask:
details={"published_at": existing["published_at"]},
)
applied_improvement = None
if improvement_id is not None:
try:
improvement_id = int(improvement_id)
except (TypeError, ValueError):
return error_response("VALIDATION_ERROR", "Invalid improvement_id", 400)
applied_improvement = query_one(
g.db,
"""SELECT s.*, t.owner, t.name FROM improvement_submissions s
JOIN tools t ON t.id = s.tool_id
WHERE s.id = ? AND s.status = 'approved'
AND s.applied_tool_id IS NULL""",
[improvement_id],
)
if (
not applied_improvement
or applied_improvement["owner"] != owner
or applied_improvement["name"] != name
or applied_improvement["step_index"] >= len(published_tool.steps)
):
return error_response(
"VALIDATION_ERROR", "Improvement cannot be applied to this release", 400
)
candidate_step = published_tool.steps[applied_improvement["step_index"]]
candidate_value = (
candidate_step.prompt if applied_improvement["step_type"] == "prompt"
and hasattr(candidate_step, "prompt") else
candidate_step.code if applied_improvement["step_type"] == "code"
and hasattr(candidate_step, "code") else None
)
if candidate_value != applied_improvement["proposed"]:
return error_response(
"VALIDATION_ERROR",
"Published config does not contain the approved improvement",
400,
)
suggestions = {"category": None, "similar_tools": []}
try:
from .categorize import suggest_categories
@@ -2592,6 +3052,10 @@ def create_app() -> Flask:
# Check scrutiny decision
if scrutiny_report:
suggestions["scrutiny"] = scrutiny_report
from ..improvement import generate_improvements
suggestions["improvements"] = generate_improvements(
published_tool, scrutiny_report
).to_dict()
if scrutiny_report.get("decision") == "reject":
# Find the failing check for error message
fail_findings = [f for f in scrutiny_report.get("findings", []) if f.get("result") == "fail"]
@@ -2625,6 +3089,9 @@ def create_app() -> Flask:
"version": version,
"status": "validated",
"suggestions": suggestions,
"preflight": preflight_report.to_dict(),
"quality": quality_report,
"content_hash": content_hash,
}
})
@@ -2650,15 +3117,16 @@ def create_app() -> Flask:
# Public tools need moderation approval
moderation_status = "pending"
g.db.execute(
insert_cursor = g.db.execute(
"""
INSERT INTO tools (
owner, name, version, description, category, tags, config_yaml, readme,
defaults, publisher_id, deprecated, deprecated_message, replacement, downloads,
scrutiny_status, scrutiny_report, source, source_url, source_json,
config_hash, visibility, moderation_status, forked_from, forked_version,
published_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
config_hash, content_hash, dependency_hashes_json, visibility,
moderation_status, forked_from, forked_version, published_at,
attestation_json
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
[
owner,
@@ -2681,11 +3149,43 @@ def create_app() -> Flask:
source_url,
source_json,
config_hash,
content_hash,
json.dumps(dependency_hashes, sort_keys=True),
visibility,
moderation_status,
forked_from,
forked_version,
datetime.utcnow().isoformat(),
datetime.now(timezone.utc).isoformat(),
attestation_json,
],
)
tool_id = insert_cursor.lastrowid
if applied_improvement:
g.db.execute(
"UPDATE improvement_submissions SET applied_tool_id = ? WHERE id = ?",
[tool_id, applied_improvement["id"]],
)
g.db.execute(
"""INSERT OR IGNORE INTO tool_contributors
(tool_id, publisher_id, contribution)
VALUES (?, ?, 'improvement')""",
[tool_id, applied_improvement["submitter_id"]],
)
audit = preflight_report.audit_evidence or {}
g.db.execute(
"""
INSERT INTO tool_audits (
tool_id, engine_version, trigger, evidence_json, quality_json,
findings_hash, evaluated_at
) VALUES (?, ?, 'publish', ?, ?, ?, ?)
""",
[
tool_id,
audit.get("engine_version", "1.0"),
json.dumps(audit, sort_keys=True),
json.dumps(quality_report, sort_keys=True),
audit.get("findings_hash", ""),
audit.get("timestamp") or datetime.now(timezone.utc).isoformat(),
],
)
g.db.commit()
@@ -2724,12 +3224,15 @@ def create_app() -> Flask:
"name": name,
"version": version,
"config_hash": config_hash,
"content_hash": content_hash,
"pr_url": "",
"status": moderation_status,
"visibility": visibility,
"forked_from": forked_from,
"forked_version": forked_version,
"suggestions": suggestions,
"preflight": preflight_report.to_dict(),
"quality": quality_report,
}
})
response.status_code = 201
@@ -2844,6 +3347,47 @@ def create_app() -> Flask:
return jsonify({"data": results})
def _get_replacement_chain(
original_owner: str, original_name: str, replacement: str
) -> List[str]:
"""Validate and resolve a deprecation chain without allowing cycles."""
chain: List[str] = []
visited = {(original_owner, original_name)}
current = replacement
default_owner = original_owner
for _ in range(20):
if "/" in current:
next_owner, next_name = current.split("/", 1)
else:
next_owner, next_name = default_owner, current
if not OWNER_RE.fullmatch(next_owner) or not TOOL_NAME_RE.fullmatch(next_name):
raise ValueError(
"replacement must be a valid tool name or owner/tool reference"
)
identity = (next_owner, next_name)
if identity in visited:
raise ValueError("replacement creates a deprecation cycle")
row = query_one(
g.db,
"""
SELECT deprecated, replacement FROM tools
WHERE owner = ? AND name = ?
ORDER BY id DESC LIMIT 1
""",
[next_owner, next_name],
)
if row is None:
raise ValueError(
f"replacement tool '{next_owner}/{next_name}' does not exist"
)
visited.add(identity)
chain.append(f"{next_owner}/{next_name}")
if not row["deprecated"] or not row["replacement"]:
return chain
current = row["replacement"]
default_owner = next_owner
raise ValueError("replacement chain exceeds 20 tools")
@app.route("/api/v1/tools/<owner>/<name>/deprecate", methods=["POST"])
@require_token
def deprecate_tool(owner: str, name: str) -> Response:
@@ -2857,6 +3401,10 @@ def create_app() -> Flask:
if message and len(message) > 500:
return error_response("VALIDATION_ERROR", "Message too long (max 500)", 400)
try:
chain = _get_replacement_chain(owner, name, replacement) if replacement else []
except ValueError as exc:
return error_response("VALIDATION_ERROR", str(exc), 400)
# Update all versions of the tool
result = g.db.execute(
@@ -2870,7 +3418,10 @@ def create_app() -> Flask:
return error_response("TOOL_NOT_FOUND", f"Tool {owner}/{name} not found", 404)
g.db.commit()
return jsonify({"data": {"status": "deprecated", "owner": owner, "name": name}})
return jsonify({"data": {
"status": "deprecated", "owner": owner, "name": name,
"replacement_chain": chain,
}})
@app.route("/api/v1/tools/<owner>/<name>/undeprecate", methods=["POST"])
@require_token
+220
View File
@@ -0,0 +1,220 @@
"""Immutable publish-time and refresh audits for registry tool versions."""
from __future__ import annotations
import argparse
import json
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List
import yaml
from ..preflight import analyze_tool
from ..quality import compute_quality
from ..tool import Tool
from .db import connect_db, get_db_path, init_db
def registry_dependency_findings(conn, tool: Tool, owner: str) -> Dict[str, List[str]]:
"""Resolve registry dependencies and report missing/deprecated references."""
from ..tool import ToolStep
references = set(tool.dependencies)
references.update(
step.tool for step in tool.steps if isinstance(step, ToolStep)
)
missing: List[str] = []
deprecated: List[str] = []
for reference in sorted(references):
from ..resolver import ToolSpec
from ..semver import matches_constraint
spec = ToolSpec.parse(reference)
if spec.name == tool.name and (spec.owner in (None, owner)):
continue
if spec.owner:
candidates = [(spec.owner, spec.name)]
else:
candidates = [(owner, spec.name), ("official", spec.name)]
row = None
for dep_owner, dep_name in candidates:
rows = conn.execute(
"""
SELECT version, deprecated, replacement FROM tools
WHERE owner = ? AND name = ? ORDER BY id DESC
""",
[dep_owner, dep_name],
).fetchall()
row = next(
(
candidate for candidate in rows
if not spec.version
or matches_constraint(candidate["version"], spec.version)
),
None,
)
if row is not None:
break
if row is None:
missing.append(reference)
elif row["deprecated"]:
guidance = f"{reference} is deprecated"
if row["replacement"]:
guidance += f"; use {row['replacement']}"
deprecated.append(guidance)
return {"missing": missing, "deprecated": deprecated}
def audit_tool_version(conn, tool_id: int, trigger: str = "background") -> Dict[str, Any]:
"""Evaluate one immutable tool version and append an audit record."""
row = conn.execute(
"""
SELECT tools.*, COALESCE(tool_stats.average_rating, 0) AS average_rating,
COALESCE(tool_stats.rating_count, 0) AS rating_count
FROM tools
LEFT JOIN tool_stats ON tool_stats.tool_id = tools.id
WHERE tools.id = ?
""",
[tool_id],
).fetchone()
if row is None:
raise ValueError(f"Unknown tool version id {tool_id}")
previous = conn.execute(
"""
SELECT evidence_json, quality_json FROM tool_audits
WHERE tool_id = ? ORDER BY evaluated_at DESC, id DESC LIMIT 1
""",
[tool_id],
).fetchone()
config = yaml.safe_load(row["config_yaml"])
tool = Tool.from_dict(config)
report = analyze_tool(
tool,
check_local_dependencies=False,
include_contract_tests=True,
check_reuse=False,
)
dependency_findings = registry_dependency_findings(conn, tool, row["owner"])
if dependency_findings["missing"]:
report.errors.append(
"Missing registry dependencies: "
+ ", ".join(dependency_findings["missing"])
)
report.warnings.extend(dependency_findings["deprecated"])
from ..preflight import _add_audit_evidence
checks = list((report.audit_evidence or {}).get("checks_run", []))
_add_audit_evidence(tool, report, checks + ["registry_dependencies"])
registry_data = {
"average_rating": row["average_rating"],
"rating_count": row["rating_count"],
"downloads": row["downloads"],
}
quality = compute_quality(tool, report, registry_data=registry_data).to_dict()
evidence = report.audit_evidence or {}
evaluated_at = datetime.now(timezone.utc).isoformat()
conn.execute(
"""
INSERT INTO tool_audits (
tool_id, engine_version, trigger, evidence_json, quality_json,
findings_hash, evaluated_at
) VALUES (?, ?, ?, ?, ?, ?, ?)
""",
[
tool_id,
evidence.get("engine_version", "1.0"),
trigger,
json.dumps(evidence, sort_keys=True),
json.dumps(quality, sort_keys=True),
evidence.get("findings_hash", ""),
evaluated_at,
],
)
quality_change = None
if previous:
previous_quality = json.loads(previous["quality_json"])
quality_change = quality["score"] - previous_quality.get("score", 0)
previous_evidence = json.loads(previous["evidence_json"])
if quality_change < 0 or previous_evidence.get("outcome") != evidence.get("outcome"):
conn.execute(
"""
INSERT INTO audit_log (
action, target_type, target_id, actor_id, details
) VALUES ('automated_audit_changed', 'tool', ?, 'system', ?)
""",
[
str(tool_id),
json.dumps({
"quality_change": quality_change,
"previous_outcome": previous_evidence.get("outcome"),
"current_outcome": evidence.get("outcome"),
}, sort_keys=True),
],
)
conn.commit()
return {
"tool_id": tool_id, "evidence": evidence, "quality": quality,
"quality_change": quality_change,
}
def audit_stale_tools(
conn, *, max_age_days: int = 30, limit: int = 100
) -> List[Dict[str, Any]]:
"""Append refreshed evidence for versions with missing or stale audits."""
if max_age_days < 0:
raise ValueError("max_age_days must be non-negative")
if limit < 1:
raise ValueError("limit must be positive")
cutoff = datetime.now(timezone.utc) - timedelta(days=max_age_days)
rows = conn.execute(
"""
SELECT tools.id, MAX(tool_audits.evaluated_at) AS last_evaluated
FROM tools
LEFT JOIN tool_audits ON tool_audits.tool_id = tools.id
GROUP BY tools.id
ORDER BY last_evaluated IS NOT NULL, last_evaluated
"""
).fetchall()
stale_ids = []
for row in rows:
value = row["last_evaluated"]
if value:
try:
evaluated = datetime.fromisoformat(value.replace("Z", "+00:00"))
if evaluated.tzinfo is None:
evaluated = evaluated.replace(tzinfo=timezone.utc)
except ValueError:
evaluated = datetime.min.replace(tzinfo=timezone.utc)
if evaluated >= cutoff:
continue
stale_ids.append(row["id"])
if len(stale_ids) >= limit:
break
results = []
for tool_id in stale_ids:
try:
results.append(audit_tool_version(conn, tool_id))
except Exception as exc:
results.append({"tool_id": tool_id, "error": str(exc)})
return results
def main() -> int:
parser = argparse.ArgumentParser(description="Refresh stale CmdForge registry audits")
parser.add_argument("--max-age-days", type=int, default=30)
parser.add_argument("--limit", type=int, default=100)
args = parser.parse_args()
conn = connect_db(get_db_path())
try:
init_db(conn)
results = audit_stale_tools(
conn, max_age_days=args.max_age_days, limit=args.limit
)
finally:
conn.close()
print(f"Audited {len(results)} tool version(s)")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+60
View File
@@ -27,6 +27,7 @@ CREATE TABLE IF NOT EXISTS publishers (
ban_reason TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
,signing_public_key TEXT
);
CREATE TABLE IF NOT EXISTS api_tokens (
@@ -61,6 +62,8 @@ CREATE TABLE IF NOT EXISTS tools (
source_url TEXT,
source_json TEXT,
config_hash TEXT,
content_hash TEXT,
dependency_hashes_json TEXT,
visibility TEXT DEFAULT 'public',
moderation_status TEXT DEFAULT 'pending',
moderation_note TEXT,
@@ -69,6 +72,7 @@ CREATE TABLE IF NOT EXISTS tools (
forked_from TEXT,
forked_version TEXT,
published_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
attestation_json TEXT,
UNIQUE(owner, name, version)
);
@@ -80,6 +84,45 @@ CREATE TABLE IF NOT EXISTS download_stats (
UNIQUE(tool_id, client_id, downloaded_at)
);
CREATE TABLE IF NOT EXISTS tool_audits (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
engine_version TEXT NOT NULL,
trigger TEXT NOT NULL,
evidence_json TEXT NOT NULL,
quality_json TEXT NOT NULL,
findings_hash TEXT NOT NULL,
evaluated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_tool_audits_tool_time
ON tool_audits(tool_id, evaluated_at DESC, id DESC);
CREATE TABLE IF NOT EXISTS improvement_submissions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
submitter_id INTEGER NOT NULL REFERENCES publishers(id),
step_index INTEGER NOT NULL,
step_type TEXT NOT NULL,
original TEXT NOT NULL,
proposed TEXT NOT NULL,
rationale TEXT,
status TEXT NOT NULL DEFAULT 'tested',
test_result_json TEXT NOT NULL,
reviewer_id INTEGER REFERENCES publishers(id),
review_notes TEXT,
submitted_at TIMESTAMP NOT NULL,
reviewed_at TIMESTAMP
,applied_tool_id INTEGER REFERENCES tools(id)
);
CREATE TABLE IF NOT EXISTS tool_contributors (
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
publisher_id INTEGER NOT NULL REFERENCES publishers(id),
contribution TEXT NOT NULL,
PRIMARY KEY (tool_id, publisher_id, contribution)
);
CREATE VIRTUAL TABLE IF NOT EXISTS tools_fts USING fts5(
name, description, tags, readme,
content='tools',
@@ -475,6 +518,8 @@ def migrate_db(conn: sqlite3.Connection) -> None:
("source_url", "TEXT", "NULL"),
("source_json", "TEXT", "NULL"),
("config_hash", "TEXT", "NULL"),
("content_hash", "TEXT", "NULL"),
("dependency_hashes_json", "TEXT", "NULL"),
("visibility", "TEXT", "'public'"),
("moderation_status", "TEXT", "'pending'"),
("moderation_note", "TEXT", "NULL"),
@@ -483,6 +528,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
("forked_from", "TEXT", "NULL"),
("forked_version", "TEXT", "NULL"),
("defaults", "TEXT", "NULL"),
("attestation_json", "TEXT", "NULL"),
]
for col_name, col_type, default in tools_migrations:
@@ -500,6 +546,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
("banned_at", "TIMESTAMP", "NULL"),
("banned_by", "TEXT", "NULL"),
("ban_reason", "TEXT", "NULL"),
("signing_public_key", "TEXT", "NULL"),
]
for col_name, col_type, default in publishers_migrations:
@@ -526,6 +573,18 @@ def migrate_db(conn: sqlite3.Connection) -> None:
except sqlite3.OperationalError:
pass
cursor = conn.execute("PRAGMA table_info(improvement_submissions)")
improvement_cols = {row[1] for row in cursor.fetchall()}
if improvement_cols and "applied_tool_id" not in improvement_cols:
try:
conn.execute(
"ALTER TABLE improvement_submissions "
"ADD COLUMN applied_tool_id INTEGER REFERENCES tools(id)"
)
conn.commit()
except sqlite3.OperationalError:
pass
# Grandfather existing tools: set moderation_status to 'approved' for tools that have NULL
# This ensures existing tools remain visible after migration (one-time migration)
# Note: Only applies to NULL, NOT to 'pending' - pending tools need manual review
@@ -559,6 +618,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_owner ON tools(owner)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_moderation ON tools(moderation_status, visibility)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_hash ON tools(config_hash)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_content_hash ON tools(content_hash)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_publishers_role ON publishers(role)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_publishers_banned ON publishers(banned)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_audit_log_target ON audit_log(target_type, target_id)")
+12 -1
View File
@@ -157,10 +157,19 @@ def semantic_search(
"""
SELECT te.tool_id, te.embedding, te.dimensions, te.model,
t.owner, t.name, t.version, t.description, t.category,
t.tags, t.downloads
t.tags, t.downloads,
json_extract(ta.quality_json, '$.score') AS quality_score,
json_extract(ta.quality_json, '$.evidence_coverage') AS quality_coverage
FROM tool_embeddings te
JOIN tools t ON t.id = te.tool_id
LEFT JOIN tool_audits ta ON ta.id = (
SELECT latest_audit.id FROM tool_audits latest_audit
WHERE latest_audit.tool_id = t.id
ORDER BY latest_audit.evaluated_at DESC, latest_audit.id DESC
LIMIT 1
)
WHERE t.visibility = 'public' AND t.moderation_status = 'approved'
AND t.deprecated = 0
""",
).fetchall()
@@ -192,6 +201,8 @@ def semantic_search(
"tags": tags_list,
"downloads": row["downloads"] or 0,
"similarity": round(score, 4),
"quality_score": row["quality_score"],
"quality_coverage": row["quality_coverage"],
})
# Sort by similarity descending
+59 -1
View File
@@ -116,6 +116,10 @@ class DownloadResult:
readme: str = ""
config_hash: str = "" # Registry hash for integrity verification
defaults: str = "" # Default settings YAML content
attestation: Optional[Dict[str, Any]] = None
signing_public_key: str = ""
content_hash: str = ""
dependency_hashes: Dict[str, str] = field(default_factory=dict)
class RegistryClient:
@@ -562,8 +566,56 @@ class RegistryClient:
readme=data.get("readme", ""),
config_hash=data.get("config_hash", ""),
defaults=data.get("defaults", "")
,attestation=data.get("attestation")
,signing_public_key=data.get("signing_public_key", "")
,content_hash=data.get("content_hash", "")
,dependency_hashes=data.get("dependency_hashes", {})
)
def get_tool_by_content_hash(self, content_hash: str) -> ToolInfo:
response = self._request("GET", f"/tools/by-content-hash/{content_hash}")
if response.status_code != 200:
self._handle_error_response(response)
return ToolInfo.from_dict(response.json().get("data", {}))
def set_signing_public_key(self, public_key: str) -> Dict[str, Any]:
"""Register the public key used to verify this publisher's releases."""
response = self._request(
"PUT", "/me/signing-key",
json_data={"public_key": public_key}, require_auth=True,
)
if response.status_code != 200:
self._handle_error_response(response)
return response.json().get("data", {})
def submit_improvement(
self, owner: str, name: str, version: str, step_index: int,
proposed: str, rationale: str = "",
) -> Dict[str, Any]:
response = self._request(
"POST", f"/tools/{owner}/{name}/{version}/improvements",
json_data={
"step_index": step_index, "proposed": proposed,
"rationale": rationale,
},
require_auth=True,
)
if response.status_code != 201:
self._handle_error_response(response)
return response.json().get("data", {})
def review_improvement(
self, submission_id: int, decision: str, notes: str = ""
) -> Dict[str, Any]:
response = self._request(
"PATCH", f"/improvements/{submission_id}",
json_data={"decision": decision, "notes": notes},
require_auth=True,
)
if response.status_code != 200:
self._handle_error_response(response)
return response.json().get("data", {})
def get_categories(self) -> List[Dict[str, Any]]:
"""
Get list of tool categories.
@@ -623,7 +675,9 @@ class RegistryClient:
defaults: str = "",
dry_run: bool = False,
visibility: str = "public",
owner: str = ""
owner: str = "",
attestation: Optional[Dict[str, Any]] = None,
improvement_id: Optional[int] = None,
) -> Dict[str, Any]:
"""
Publish a tool to the registry.
@@ -649,6 +703,10 @@ class RegistryClient:
payload["defaults"] = defaults
if owner:
payload["owner"] = owner
if attestation:
payload["attestation"] = attestation
if improvement_id is not None:
payload["improvement_id"] = improvement_id
response = self._request(
"POST",
+32 -5
View File
@@ -16,7 +16,7 @@ from typing import Optional, Tuple
import yaml
from .tool import Tool, TOOLS_DIR, get_bin_dir, BIN_DIR
from .tool import Tool, TOOLS_DIR, get_tools_dir, get_bin_dir, BIN_DIR
from .config import is_auto_fetch_enabled, load_config
from .manifest import load_manifest
@@ -243,7 +243,7 @@ class ToolResolver:
searched_paths: list
) -> Optional[ResolvedTool]:
"""Search for tool in global user directory."""
global_dir = TOOLS_DIR
global_dir = get_tools_dir()
if not global_dir.exists():
return None
@@ -354,7 +354,10 @@ class ToolResolver:
config_yaml=result.config_yaml,
readme=result.readme,
config_hash=result.config_hash,
defaults=result.defaults
defaults=result.defaults,
attestation=result.attestation,
signing_public_key=result.signing_public_key,
content_hash=result.content_hash,
)
if self.verbose:
@@ -383,7 +386,10 @@ class ToolResolver:
config_yaml: str,
readme: str = "",
config_hash: str = "",
defaults: str = ""
defaults: str = "",
attestation: Optional[dict] = None,
signing_public_key: str = "",
content_hash: str = "",
) -> ResolvedTool:
"""Install a tool fetched from registry to global directory."""
# Verify hash if provided
@@ -396,6 +402,24 @@ class ToolResolver:
f"got {computed_hash[:20]}... - content may have been tampered with"
)
if attestation:
from .attestation import Attestation, verify_attestation
try:
signed = Attestation.from_dict(attestation)
except (KeyError, TypeError, ValueError) as exc:
raise RuntimeError("Malformed release attestation") from exc
if (
not signing_public_key
or signed.tool_name != name
or signed.version != version
or signed.content_hash != content_hash
or signed.signer != owner
or not verify_attestation(signed, signing_public_key)
):
raise RuntimeError(
f"Invalid release attestation for {owner}/{name}@{version}"
)
# Create directory structure
tool_dir = TOOLS_DIR / owner / name
tool_dir.mkdir(parents=True, exist_ok=True)
@@ -628,7 +652,10 @@ def install_from_registry(spec: str, version: Optional[str] = None) -> ResolvedT
config_yaml=result.config_yaml,
readme=result.readme,
config_hash=result.config_hash,
defaults=result.defaults
defaults=result.defaults,
attestation=result.attestation,
signing_public_key=result.signing_public_key,
content_hash=result.content_hash,
)
+372 -29
View File
@@ -1,19 +1,24 @@
"""Tool execution engine."""
import argparse
import fnmatch
import json
import re
import sys
from dataclasses import replace
from pathlib import Path
from typing import Optional
from typing import Optional, List
import yaml
from .tool import Tool, PromptStep, CodeStep, ToolStep
from .providers import call_provider, mock_provider
from .tool import Tool, PromptStep, CodeStep, ToolStep, McpStep
from .providers import (
call_provider, mock_provider, ProviderExecutionPolicy,
)
from .resolver import resolve_tool, ToolNotFoundError, ToolSpec, install_from_registry
from .manifest import load_manifest
from .profiles import load_profile
from .mcp_client import McpClientManager
# Maximum recursion depth for nested tool calls
MAX_TOOL_DEPTH = 10
@@ -352,7 +357,9 @@ def substitute_variables(template: str, variables: dict, warn_non_scalar: bool =
if base_name == "settings":
# Settings dict - leave unchanged (handled specially)
return match.group(0)
# Replace with empty string for None, str() for other values
if isinstance(base_value, (dict, list)):
return json.dumps(base_value)
# Replace with empty string for None, str() for other scalar values
return "" if base_value is None else str(base_value)
# Handle nested access
@@ -503,7 +510,9 @@ def execute_prompt_step(
variables: dict,
provider_override: str = None,
verbose: bool = False,
base_dir: Optional[Path] = None
base_dir: Optional[Path] = None,
execution_policy: Optional[ProviderExecutionPolicy] = None,
provenance_log: Optional[list[dict]] = None,
) -> tuple[str, bool]:
"""
Execute a prompt step.
@@ -520,6 +529,24 @@ def execute_prompt_step(
Otherwise, enforces structured JSON output with schema validation.
"""
import re
from dataclasses import replace
effective_policy = execution_policy or ProviderExecutionPolicy()
if step.fallback_policy == "deny" and effective_policy.allow_fallback:
effective_policy = replace(effective_policy, allow_fallback=False)
def record_result(result, selected_provider: str) -> None:
# Mock calls bypass call_provider for deterministic tests, so attach
# the same runtime-owned identity here.
if selected_provider.lower() == "mock" and not result.actual_provider:
result.requested_provider = "mock"
result.actual_provider = "mock"
result.attempted_providers = ["mock"]
result.model = "mock"
result.locality = "local"
result.model_identity_source = "built-in"
if provenance_log is not None:
provenance_log.append(result.provenance())
# Build prompt with variable substitution
try:
@@ -530,22 +557,34 @@ def execute_prompt_step(
prompt = substitute_variables(prompt_template, variables, warn_non_scalar=verbose)
# Inject profile system prompt if specified
# Determine provider
provider = provider_override or substitute_variables(step.provider, variables, warn_non_scalar=verbose)
# Build context from the inside out so the final order is:
# profile system prompt -> skills -> user prompt.
if step.skills is not None:
from .skills import inject_skills
skill_names = None if "*" in step.skills else step.skills
prompt = inject_skills(prompt, provider, skill_names)
if step.profile:
profile = load_profile(step.profile)
if profile and profile.system_prompt:
# Prepend system prompt to user prompt
prompt = f"{profile.system_prompt}\n\n---\n\n{prompt}"
# Determine provider
provider = provider_override or substitute_variables(step.provider, variables, warn_non_scalar=verbose)
# Structured or plain-text
# Plain text mode - bypass structured output
if step.plain_text:
if provider.lower() == "mock":
result = mock_provider(prompt)
else:
result = call_provider(provider, prompt, max_tokens=step.max_tokens)
result = call_provider(
provider, prompt, max_tokens=step.max_tokens,
execution_policy=effective_policy,
)
record_result(result, provider)
if not result.success:
print(f"Error in prompt step: {result.error}", file=sys.stderr)
@@ -589,7 +628,12 @@ Please try again with valid JSON matching the schema exactly."""
if provider.lower() == "mock":
result = mock_provider(current_prompt)
else:
result = call_provider(provider, current_prompt, max_tokens=step.max_tokens)
result = call_provider(
provider, current_prompt, max_tokens=step.max_tokens,
execution_policy=effective_policy,
)
record_result(result, provider)
if not result.success:
print(f"Error in prompt step: {result.error}", file=sys.stderr)
@@ -727,7 +771,12 @@ def execute_tool_step(
provider_override: Optional[str] = None,
dry_run: bool = False,
verbose: bool = False,
call_stack: Optional[list] = None
call_stack: Optional[list] = None,
agent_profile: Optional[str] = None,
agent_skills: Optional[List[str]] = None,
agent_tool_policies: Optional[List[List[str]]] = None,
execution_policy: Optional[ProviderExecutionPolicy] = None,
provenance_log: Optional[list[dict]] = None,
) -> tuple[str, bool]:
"""
Execute a tool step by calling another tool.
@@ -751,7 +800,15 @@ def execute_tool_step(
_print_call_stack(call_stack, f"Maximum tool nesting depth ({MAX_TOOL_DEPTH}) exceeded")
return "", False
# Resolve the tool reference
# Authorization must happen before resolution because resolution may fetch a
# missing tool from the registry.
effective_provider = step.provider or provider_override
if not _authorize_tool_call(
step.tool, effective_provider, agent_tool_policies, call_stack
):
return "", False
# Resolve the tool reference only after authorization succeeds.
try:
resolved = resolve_tool(step.tool)
nested_tool = resolved.tool
@@ -768,9 +825,6 @@ def execute_tool_step(
for key, value in step.args.items():
custom_args[key] = substitute_variables(str(value), variables, warn_non_scalar=verbose)
# Determine effective provider (step override > parent override)
effective_provider = step.provider or provider_override
if verbose:
print(f"[verbose] Tool step: calling {step.tool}", file=sys.stderr)
print(f"[verbose] Input length: {len(input_text)} chars", file=sys.stderr)
@@ -789,7 +843,14 @@ def execute_tool_step(
show_prompt=False,
verbose=verbose,
_depth=depth + 1,
_call_stack=call_stack
_call_stack=call_stack,
agent_profile=step.profile if step.profile is not None else agent_profile,
agent_skills=step.skills if step.skills is not None else agent_skills,
agent_tool_policies=_extend_agent_tool_policies(
agent_tool_policies, step.tools
),
execution_policy=execution_policy,
provenance_log=provenance_log,
)
return output, exit_code == 0
@@ -805,7 +866,12 @@ def run_tool(
verbose: bool = False,
auto_install: bool = False,
_depth: int = 0,
_call_stack: Optional[list] = None
_call_stack: Optional[list] = None,
agent_profile: Optional[str] = None,
agent_skills: Optional[List[str]] = None,
agent_tool_policies: Optional[List[List[str]]] = None,
execution_policy: Optional[ProviderExecutionPolicy] = None,
provenance_log: Optional[list[dict]] = None,
) -> tuple[str, int]:
"""
Execute a tool.
@@ -901,6 +967,9 @@ def run_tool(
output = substitute_variables(tool.output, variables, warn_non_scalar=verbose)
return output, 0
# Reuse MCP configuration and schema caches across all MCP steps in this run.
mcp_manager = None
# Execute each step
for i, step in enumerate(tool.steps):
if verbose:
@@ -910,6 +979,8 @@ def run_tool(
step_type = "CODE"
elif isinstance(step, ToolStep):
step_type = f"TOOL({step.tool})"
elif isinstance(step, McpStep):
step_type = f"MCP({step.server}/{step.tool})"
else:
step_type = "UNKNOWN"
print(f"[verbose] Step {i+1}: {step_type} -> {{{step.output_var}}}", file=sys.stderr)
@@ -925,12 +996,20 @@ def run_tool(
if dry_run:
variables[step.output_var] = f"[DRY RUN - would call {step.provider}]"
else:
# Apply agent-level defaults if step doesn't set its own
effective_step = step
if agent_profile and not step.profile:
effective_step = _step_with_profile(step, agent_profile)
if agent_skills is not None and step.skills is None:
effective_step = _step_with_skills(effective_step, agent_skills)
output, success = execute_prompt_step(
step,
effective_step,
variables,
provider_override,
verbose=verbose,
base_dir=tool_base_dir
base_dir=tool_base_dir,
execution_policy=execution_policy,
provenance_log=provenance_log,
)
if not success:
return "", 2
@@ -975,18 +1054,95 @@ def run_tool(
provider_override=provider_override,
dry_run=dry_run,
verbose=verbose,
call_stack=step_stack
call_stack=step_stack,
agent_profile=agent_profile,
agent_skills=agent_skills,
agent_tool_policies=agent_tool_policies,
execution_policy=execution_policy,
provenance_log=provenance_log,
)
if not success:
return "", 3
variables[step.output_var] = output
elif isinstance(step, McpStep):
if verbose or dry_run:
print(f"=== MCP (Step {i+1}) -> {{{step.output_var}}} ===", file=sys.stderr)
print(f" Server: {step.server} Tool: {step.tool}", file=sys.stderr)
print(f" Args: {step.arguments}", file=sys.stderr)
print("=== END MCP ===", file=sys.stderr)
if dry_run:
variables[step.output_var] = f"[DRY RUN - would call mcp:{step.server}/{step.tool}]"
else:
step_stack = _call_stack + [(tool.name, i + 1)]
if not _authorize_mcp_call(
step.server, provider_override, step_stack
):
return "", 4
args = _substitute_mcp_args(step.arguments, variables)
try:
if mcp_manager is None:
mcp_manager = McpClientManager()
result = mcp_manager.call_tool(
step.server, step.tool, args,
result_mode=step.result_mode,
)
variables[step.output_var] = result
except ImportError as e:
print(f"{e}", file=sys.stderr)
return "", 4
except Exception as e:
print(f"MCP step failed (server={step.server}, tool={step.tool}): {e}", file=sys.stderr)
return "", 4
# Generate final output
output = substitute_variables(tool.output, variables, warn_non_scalar=verbose)
return output, 0
def _substitute_mcp_args(arguments: dict, variables: dict) -> dict:
"""Substitute variables in MCP arguments, preserving types."""
result = {}
for key, value in arguments.items():
result[key] = _deep_substitute(value, variables)
return result
def _deep_substitute(value, variables: dict):
if isinstance(value, str):
import re
exact_reference = re.fullmatch(
r"\s*\{([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)\}\s*",
value,
)
if exact_reference:
path = exact_reference.group(1).split(".")
if path[0] in variables:
resolved = variables[path[0]]
found = True
for key in path[1:]:
if isinstance(resolved, dict) and key in resolved:
resolved = resolved[key]
elif hasattr(resolved, key):
resolved = getattr(resolved, key)
else:
found = False
break
if found:
return resolved
substituted = substitute_variables(value, variables)
return substituted if substituted != value else value
elif isinstance(value, dict):
return {k: _deep_substitute(v, variables) for k, v in value.items()}
elif isinstance(value, list):
return [_deep_substitute(v, variables) for v in value]
return value
def create_argument_parser(tool: Tool) -> argparse.ArgumentParser:
"""
Create an argument parser for a tool.
@@ -1019,14 +1175,65 @@ def create_argument_parser(tool: Tool) -> argparse.ArgumentParser:
help="Show debug information")
parser.add_argument("--auto-install", action="store_true",
help="Automatically install missing tool dependencies")
parser.add_argument("--no-fallback", action="store_true",
help="Fail instead of trying configured fallbacks")
parser.add_argument("--require-local", action="store_true",
help="Allow only providers declared local")
parser.add_argument("--require-capability", action="append", default=[],
help="Required provider capability (repeatable)")
parser.add_argument("--data-classification",
choices=["public", "internal", "private"])
parser.add_argument("--require-model-identity", action="store_true")
parser.add_argument("--require-model-digest", action="store_true")
parser.add_argument("--result-envelope", choices=["json"])
# Tool-specific flags from arguments
for arg in tool.arguments:
existing_action = parser._option_string_actions.get(arg.flag)
if existing_action is not None:
# Preserve the legacy and useful case where a boolean tool option
# intentionally shares a universal flag (for example --verbose).
# Both consumers then read the same parsed boolean.
if arg.type == "boolean" and existing_action.dest == arg.variable:
continue
raise ValueError(
f"Tool argument {arg.flag!r} conflicts with CmdForge's "
"universal run options"
)
def parse_boolean(value):
if isinstance(value, bool):
return value
normalized = str(value).strip().lower()
if normalized in {"1", "true", "yes", "on"}:
return True
if normalized in {"0", "false", "no", "off"}:
return False
raise argparse.ArgumentTypeError(
"expected true/false, yes/no, on/off, or 1/0"
)
value_type = {
"string": str,
"integer": int,
"number": float,
"boolean": parse_boolean,
}[arg.type]
options = {
"dest": arg.variable,
"default": arg.default,
"help": arg.description or f"{arg.variable} (default: {arg.default})",
"required": arg.required,
"type": value_type,
}
if arg.enum is not None:
options["choices"] = [value_type(value) for value in arg.enum]
if arg.type == "boolean":
options["nargs"] = "?"
options["const"] = True
parser.add_argument(
arg.flag,
dest=arg.variable,
default=arg.default,
help=arg.description or f"{arg.variable} (default: {arg.default})"
**options,
)
return parser
@@ -1059,6 +1266,11 @@ def main():
print(f"Searched: {', '.join(e.searched_paths[:3])}", file=sys.stderr)
sys.exit(1)
from .tool import deprecation_warning
warning = deprecation_warning(tool)
if warning:
print(warning, file=sys.stderr)
# Check for manifest overrides
manifest = load_manifest()
provider_override_from_manifest = None
@@ -1096,6 +1308,16 @@ def main():
# Determine provider override (CLI flag takes precedence over manifest)
effective_provider = args.provider or provider_override_from_manifest
execution_policy = ProviderExecutionPolicy(
allow_fallback=not args.no_fallback,
required_locality="local" if args.require_local else None,
required_capabilities=tuple(args.require_capability or []),
data_classification=args.data_classification or "public",
require_model_identity=args.require_model_identity,
require_model_digest=args.require_model_digest,
)
provider_calls = []
# Run tool
output, exit_code = run_tool(
tool=tool,
@@ -1105,18 +1327,139 @@ def main():
dry_run=args.dry_run,
show_prompt=args.show_prompt,
verbose=args.verbose,
auto_install=args.auto_install
auto_install=args.auto_install,
execution_policy=execution_policy,
provenance_log=provider_calls,
)
# Write output
if exit_code == 0 and output:
if args.output_file:
Path(args.output_file).write_text(output)
if args.result_envelope == "json" or (exit_code == 0 and output):
if args.result_envelope == "json":
import json
rendered_output = json.dumps({
"output": output,
"execution": {
"exit_code": exit_code,
"provider_calls": provider_calls,
"fallback_used": any(
item.get("fallback_used", False)
for item in provider_calls
),
"actual_providers": list(dict.fromkeys(
item["actual_provider"] for item in provider_calls
if item.get("actual_provider")
)),
},
}, indent=2)
else:
print(output)
rendered_output = output
if args.output_file:
Path(args.output_file).write_text(rendered_output)
else:
print(rendered_output)
if exit_code == 0 and not args.dry_run:
try:
from .usage import record_invocation
record_invocation(resolved.full_name)
except (OSError, ValueError):
# Usage discovery is optional and must never break tool execution.
pass
sys.exit(exit_code)
def _matches_allowlist(name: str, allowed: List[str]) -> bool:
return any(fnmatch.fnmatchcase(name, pattern) for pattern in allowed)
def _authorize_tool_call(
name: str,
provider_name: Optional[str],
agent_tool_policies: Optional[List[List[str]]],
call_stack: list,
) -> bool:
"""Enforce provider and delegated-agent tool policies."""
if provider_name:
from .providers import get_provider
provider = get_provider(provider_name)
if provider is None:
_print_call_stack(
call_stack,
f"Provider '{provider_name}' is not configured; refusing tool call '{name}'",
)
return False
if provider.tools is not None and not _matches_allowlist(name, provider.tools):
_print_call_stack(
call_stack,
f"Provider '{provider_name}' is not allowed to call tool '{name}'",
)
print(
f"Provider '{provider_name}' tools allowlist: {provider.tools}",
file=sys.stderr,
)
return False
for policy in agent_tool_policies or []:
if not _matches_allowlist(name, policy):
_print_call_stack(
call_stack,
f"Delegated agent is not allowed to call tool '{name}'",
)
print(f"Delegated agent tools allowlist: {policy}", file=sys.stderr)
return False
return True
def _extend_agent_tool_policies(
policies: Optional[List[List[str]]],
additional: Optional[List[str]],
) -> Optional[List[List[str]]]:
"""Add a restriction without allowing descendants to broaden ancestors."""
if additional is None:
return policies
return [*(policies or []), additional]
def _authorize_mcp_call(
server: str,
provider_name: Optional[str],
call_stack: list,
) -> bool:
"""Enforce the effective provider's MCP-server policy."""
if not provider_name:
return True
from .providers import get_provider
provider = get_provider(provider_name)
if provider is None:
_print_call_stack(
call_stack,
f"Provider '{provider_name}' is not configured; refusing MCP server '{server}'",
)
return False
if provider.mcp_servers is not None and not _matches_allowlist(
server, provider.mcp_servers
):
_print_call_stack(
call_stack,
f"Provider '{provider_name}' is not allowed to use MCP server '{server}'",
)
print(
f"Provider '{provider_name}' MCP server allowlist: {provider.mcp_servers}",
file=sys.stderr,
)
return False
return True
def _step_with_profile(step, profile_name):
return replace(step, profile=profile_name)
def _step_with_skills(step, skill_names):
return replace(step, skills=skill_names)
if __name__ == "__main__":
main()
+348
View File
@@ -0,0 +1,348 @@
"""Conservative JSON Schema compatibility checks for tool composition."""
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from typing import Any, Dict, List, Literal, Optional, Set
CompatibilityState = Literal["compatible", "incompatible", "unknown"]
@dataclass
class SchemaCompatibility:
state: CompatibilityState
detail: str
issues: List[str] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return asdict(self)
def compare_json_schemas(
producer: Optional[dict], consumer: Optional[dict], path: str = "$"
) -> SchemaCompatibility:
"""Check whether every value allowed by producer is accepted by consumer.
This intentionally implements a useful, conservative subset of JSON Schema
subsumption. Unsupported combinators and constraints return ``unknown``
rather than claiming compatibility.
"""
if producer is None or consumer is None:
return _unknown(path, "schema evidence is missing")
if consumer == {}:
return _compatible("consumer accepts any JSON value")
if producer == {}:
return _unknown(path, "producer schema is unconstrained")
concrete = _concrete_values(producer)
if concrete is not None:
failures = [value for value in concrete if not _value_matches(value, consumer)]
if failures:
return _incompatible(
path, f"producer can emit value not accepted by consumer: {failures[0]!r}"
)
return _compatible("all producer const/enum values satisfy consumer")
if _concrete_values(consumer) is not None:
return _incompatible(
path, "producer is not restricted to the consumer's const/enum values"
)
if any(key in producer or key in consumer for key in ("oneOf", "anyOf", "allOf", "not", "$ref")):
return _unknown(path, "schema combinators or references require deeper analysis")
advanced_constraints = (
"contains", "minContains", "maxContains", "patternProperties",
"dependentSchemas", "propertyNames", "unevaluatedProperties",
"unevaluatedItems", "if", "then", "else",
)
differing = [
keyword for keyword in advanced_constraints
if keyword in consumer and producer.get(keyword) != consumer.get(keyword)
]
if differing:
return _unknown(
path, "cannot prove constraint(s): " + ", ".join(differing)
)
producer_types = _types(producer)
consumer_types = _types(consumer)
if producer_types and consumer_types:
rejected = {
item for item in producer_types
if not any(_type_accepted(item, target) for target in consumer_types)
}
if rejected:
return _incompatible(
path,
"producer type(s) " + ", ".join(sorted(rejected))
+ " are not accepted by consumer type(s) "
+ ", ".join(sorted(consumer_types)),
)
elif not producer_types or not consumer_types:
return _unknown(path, "type constraints are incomplete")
states: List[SchemaCompatibility] = []
effective_types = producer_types or set()
if "object" in effective_types:
states.append(_compare_objects(producer, consumer, path))
if "array" in effective_types:
states.append(_compare_arrays(producer, consumer, path))
if "string" in effective_types:
states.append(_compare_strings(producer, consumer, path))
if effective_types & {"integer", "number"}:
states.append(_compare_numbers(producer, consumer, path))
incompatible = [state for state in states if state.state == "incompatible"]
if incompatible:
issues = [issue for state in incompatible for issue in state.issues]
return SchemaCompatibility("incompatible", incompatible[0].detail, issues)
unknown = [state for state in states if state.state == "unknown"]
if unknown:
issues = [issue for state in unknown for issue in state.issues]
return SchemaCompatibility("unknown", unknown[0].detail, issues)
return _compatible("producer schema is accepted by consumer schema")
def schema_for_value(value: Any) -> dict:
"""Return an exact schema for a literal value."""
return {"const": value}
def _compare_objects(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
producer_properties = producer.get("properties", {})
consumer_properties = consumer.get("properties", {})
producer_required = set(producer.get("required", []))
consumer_required = set(consumer.get("required", []))
issues: List[str] = []
unknown = False
incompatible = False
if (
consumer.get("additionalProperties") is False
and producer.get("additionalProperties") is not False
):
issues.append(f"{path}: producer permits unspecified object fields")
incompatible = True
producer_min = max(producer.get("minProperties", 0), len(producer_required))
consumer_min = consumer.get("minProperties", 0)
if producer_min < consumer_min:
issues.append(f"{path}: producer permits too few object properties")
incompatible = True
producer_max = producer.get("maxProperties")
if producer.get("additionalProperties") is False:
closed_max = len(producer_properties)
producer_max = closed_max if producer_max is None else min(producer_max, closed_max)
consumer_max = consumer.get("maxProperties")
if consumer_max is not None and (
producer_max is None or producer_max > consumer_max
):
issues.append(f"{path}: producer permits too many object properties")
incompatible = True
for name in sorted(consumer_required - producer_required):
issues.append(f"{path}.{name}: required consumer field is not guaranteed")
incompatible = True
producer_additional = producer.get("additionalProperties", True)
for name, consumer_schema in consumer_properties.items():
if name in producer_properties or producer_additional is False:
continue
if producer_additional is True:
issues.append(
f"{path}.{name}: producer may emit an unconstrained value"
)
incompatible = True
continue
if isinstance(producer_additional, dict):
result = compare_json_schemas(
producer_additional, consumer_schema, f"{path}.{name}"
)
issues.extend(result.issues)
incompatible = incompatible or result.state == "incompatible"
unknown = unknown or result.state == "unknown"
if consumer.get("additionalProperties") is False:
for name in sorted(producer_required - set(consumer_properties)):
issues.append(f"{path}.{name}: consumer rejects this supplied field")
incompatible = True
for name, producer_schema in producer_properties.items():
consumer_schema = consumer_properties.get(name)
if consumer_schema is None:
additional = consumer.get("additionalProperties", True)
if additional is False and name in producer_required:
issues.append(f"{path}.{name}: consumer does not allow this field")
incompatible = True
elif isinstance(additional, dict):
result = compare_json_schemas(
producer_schema, additional, f"{path}.{name}"
)
issues.extend(result.issues)
unknown = unknown or result.state == "unknown"
continue
result = compare_json_schemas(
producer_schema, consumer_schema, f"{path}.{name}"
)
if result.state == "incompatible":
issues.extend(result.issues)
incompatible = True
elif result.state == "unknown":
unknown = True
issues.extend(result.issues)
if incompatible:
return SchemaCompatibility("incompatible", issues[0], issues)
if unknown:
return SchemaCompatibility("unknown", issues[0] if issues else "object constraints are incomplete", issues)
return _compatible("object fields and required properties are compatible")
def _compare_arrays(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
if consumer.get("uniqueItems") is True and producer.get("uniqueItems") is not True:
return _unknown(path, "producer does not guarantee unique array items")
producer_min = producer.get("minItems", 0)
consumer_min = consumer.get("minItems", 0)
if producer_min < consumer_min:
return _incompatible(path, "producer permits fewer array items than consumer")
producer_max = producer.get("maxItems")
consumer_max = consumer.get("maxItems")
if consumer_max is not None and (
producer_max is None or producer_max > consumer_max
):
return _incompatible(path, "producer permits more array items than consumer")
if "items" in consumer:
if "items" not in producer:
return _unknown(path, "producer array item schema is missing")
return compare_json_schemas(
producer["items"], consumer["items"], f"{path}[]"
)
return _compatible("array bounds are compatible")
def _compare_strings(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
if producer.get("minLength", 0) < consumer.get("minLength", 0):
return _incompatible(path, "producer permits strings shorter than consumer")
consumer_max = consumer.get("maxLength")
producer_max = producer.get("maxLength")
if consumer_max is not None and (
producer_max is None or producer_max > consumer_max
):
return _incompatible(path, "producer permits strings longer than consumer")
for keyword in ("pattern", "format"):
if keyword in consumer and producer.get(keyword) != consumer.get(keyword):
return _unknown(path, f"cannot prove differing {keyword} constraints")
return _compatible("string bounds are compatible")
def _compare_numbers(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
producer_min = _lower_bound(producer)
consumer_min = _lower_bound(consumer)
if consumer_min is not None and (
producer_min is None or producer_min < consumer_min
):
return _incompatible(path, "producer permits numbers below consumer minimum")
if "exclusiveMinimum" in consumer and producer_min == consumer_min:
if producer.get("exclusiveMinimum") != consumer.get("exclusiveMinimum"):
return _incompatible(
path, "producer includes the consumer's exclusive minimum"
)
producer_max = _upper_bound(producer)
consumer_max = _upper_bound(consumer)
if consumer_max is not None and (
producer_max is None or producer_max > consumer_max
):
return _incompatible(path, "producer permits numbers above consumer maximum")
if "exclusiveMaximum" in consumer and producer_max == consumer_max:
if producer.get("exclusiveMaximum") != consumer.get("exclusiveMaximum"):
return _incompatible(
path, "producer includes the consumer's exclusive maximum"
)
if "multipleOf" in consumer and producer.get("multipleOf") != consumer.get("multipleOf"):
return _unknown(path, "cannot prove differing multipleOf constraints")
return _compatible("numeric bounds are compatible")
def _types(schema: dict) -> Set[str]:
value = schema.get("type")
if isinstance(value, str):
return {value}
if isinstance(value, list):
return {item for item in value if isinstance(item, str)}
if "properties" in schema or "required" in schema:
return {"object"}
if "items" in schema:
return {"array"}
concrete = _concrete_values(schema)
if concrete:
return {_json_type(value) for value in concrete}
return set()
def _concrete_values(schema: dict) -> Optional[List[Any]]:
if "const" in schema:
return [schema["const"]]
if isinstance(schema.get("enum"), list) and schema["enum"]:
return list(schema["enum"])
return None
def _value_matches(value: Any, schema: dict) -> bool:
try:
from jsonschema import validate
from jsonschema.exceptions import ValidationError
validate(instance=value, schema=schema)
return True
except ValidationError:
return False
def _json_type(value: Any) -> str:
if value is None:
return "null"
if isinstance(value, bool):
return "boolean"
if isinstance(value, int):
return "integer"
if isinstance(value, float):
return "number"
if isinstance(value, str):
return "string"
if isinstance(value, list):
return "array"
return "object"
def _type_accepted(producer: str, consumer: str) -> bool:
return producer == consumer or (producer == "integer" and consumer == "number")
def _lower_bound(schema: dict) -> Optional[float]:
if "exclusiveMinimum" in schema:
return float(schema["exclusiveMinimum"])
if "minimum" in schema:
return float(schema["minimum"])
return None
def _upper_bound(schema: dict) -> Optional[float]:
if "exclusiveMaximum" in schema:
return float(schema["exclusiveMaximum"])
if "maximum" in schema:
return float(schema["maximum"])
return None
def _compatible(detail: str) -> SchemaCompatibility:
return SchemaCompatibility("compatible", detail)
def _unknown(path: str, detail: str) -> SchemaCompatibility:
issue = f"{path}: {detail}"
return SchemaCompatibility("unknown", issue, [issue])
def _incompatible(path: str, detail: str) -> SchemaCompatibility:
issue = f"{path}: {detail}"
return SchemaCompatibility("incompatible", issue, [issue])
+43
View File
@@ -0,0 +1,43 @@
"""Local publisher signing-key storage."""
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Optional, Tuple
from .attestation import generate_keypair
from .config import CONFIG_DIR
SIGNING_KEY_FILE = CONFIG_DIR / "release-signing-key.json"
def load_signing_key(path: Path = SIGNING_KEY_FILE) -> Optional[Tuple[str, str]]:
if not path.exists():
return None
data = json.loads(path.read_text(encoding="utf-8"))
private_key = data.get("private_key", "")
public_key = data.get("public_key", "")
if not private_key or not public_key:
raise ValueError("Signing key file is incomplete")
return private_key, public_key
def initialize_signing_key(
path: Path = SIGNING_KEY_FILE, *, overwrite: bool = False
) -> Tuple[str, str]:
if path.exists() and not overwrite:
existing = load_signing_key(path)
if existing is None:
raise ValueError("Signing key file could not be loaded")
return existing
private_key, public_key = generate_keypair()
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
path.write_text(json.dumps({
"algorithm": "ed25519",
"private_key": private_key,
"public_key": public_key,
}, indent=2) + "\n", encoding="utf-8")
os.chmod(path, 0o600)
return private_key, public_key
+221
View File
@@ -0,0 +1,221 @@
"""Provider-attached skills in the Agent Skills open standard.
Each provider can have skills at::
~/.cmdforge/providers/<provider>/skills/<skill-name>/SKILL.md
Every ``SKILL.md`` must contain the required Agent Skills frontmatter. The
directory name is the canonical skill identifier and must match ``name``.
"""
import re
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import List, Optional
import yaml
PROVIDERS_DIR = Path.home() / ".cmdforge" / "providers"
_PROVIDER_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$")
_SKILL_NAME_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
_MISSING_POLICIES = {"warn", "error", "ignore"}
@dataclass(frozen=True)
class Skill:
"""A single loaded skill."""
name: str
directory: str
description: str
content: str = ""
def __str__(self) -> str:
return self.content
@dataclass
class SkillSet:
"""Collection of skills for one provider, deterministically ordered."""
provider_name: str
skills: List[Skill] = field(default_factory=list)
def get(
self,
requested: Optional[List[str]] = None,
on_missing: str = "warn",
) -> List[Skill]:
"""Return enabled skills, reporting missing ones."""
if on_missing not in _MISSING_POLICIES:
raise ValueError(
f"Invalid missing-skill policy '{on_missing}'; expected one of "
f"{sorted(_MISSING_POLICIES)}"
)
available = {skill.name: skill for skill in self.skills}
if requested is None:
return list(self.skills)
enabled: List[Skill] = []
for name in requested:
if name in available:
enabled.append(available[name])
continue
message = (
f"Skill '{name}' requested for provider '{self.provider_name}' "
"but not found in skills directory"
)
if on_missing == "error":
raise KeyError(message)
if on_missing == "warn":
print(f"[skills] {message}", file=sys.stderr)
return enabled
def _validate_provider_name(name: str) -> None:
"""Validate a provider name before using it as a path component."""
if not isinstance(name, str) or not _PROVIDER_NAME_RE.fullmatch(name):
raise ValueError(
f"Invalid provider name '{name}'. Must match {_PROVIDER_NAME_RE.pattern}"
)
def _validate_skill_name(name: str, provider: str = "") -> None:
"""Validate the Agent Skills canonical name rules."""
context = f" under provider '{provider}'" if provider else ""
if (
not isinstance(name, str)
or not 1 <= len(name) <= 64
or not _SKILL_NAME_RE.fullmatch(name)
):
raise ValueError(
f"Invalid skill name '{name}'{context}. Skill names must be 1-64 "
"lowercase letters, numbers, or single hyphens, and may not start "
"or end with a hyphen"
)
def _parse_frontmatter(text: str) -> tuple[dict, str]:
"""Parse the mandatory YAML frontmatter from a ``SKILL.md`` file."""
lines = text.splitlines(keepends=True)
if not lines or lines[0].rstrip("\r\n") != "---":
raise ValueError("SKILL.md must begin with YAML frontmatter")
closing_index = next(
(index for index, line in enumerate(lines[1:], start=1)
if line.rstrip("\r\n") == "---"),
None,
)
if closing_index is None:
raise ValueError("SKILL.md frontmatter is not terminated")
frontmatter_raw = "".join(lines[1:closing_index]).strip()
if not frontmatter_raw:
raise ValueError("SKILL.md frontmatter is empty; metadata is required")
try:
metadata = yaml.safe_load(frontmatter_raw)
except yaml.YAMLError as exc:
raise ValueError(f"Invalid YAML in SKILL.md frontmatter: {exc}") from exc
if not isinstance(metadata, dict):
raise ValueError("SKILL.md frontmatter must be a YAML mapping")
content = "".join(lines[closing_index + 1:]).lstrip("\r\n").rstrip()
return metadata, content
def _validate_metadata(metadata: dict, directory_name: str, provider: str) -> tuple[str, str]:
"""Validate required Agent Skills metadata and return normalized values."""
if "name" not in metadata:
raise ValueError(f"SKILL.md for '{directory_name}' is missing required 'name'")
name = metadata["name"]
_validate_skill_name(name, provider)
if name != directory_name:
raise ValueError(
f"SKILL.md frontmatter name '{name}' does not match skill directory "
f"name '{directory_name}' under provider '{provider}'"
)
if "description" not in metadata:
raise ValueError(
f"SKILL.md for '{directory_name}' is missing required 'description'"
)
description = metadata["description"]
if not isinstance(description, str) or not description.strip():
raise ValueError(
f"SKILL.md description for '{directory_name}' must be a non-empty string"
)
if len(description) > 1024:
raise ValueError(
f"SKILL.md description for '{directory_name}' must not exceed 1024 characters"
)
return name, description.strip()
def _reject_symlink(path: Path, label: str) -> None:
if path.is_symlink():
raise ValueError(f"Refusing symlinked {label}: {path}")
def load_provider_skills(provider_name: str) -> SkillSet:
"""Load and validate all skills configured for a provider."""
_validate_provider_name(provider_name)
provider_dir = PROVIDERS_DIR / provider_name
skills_dir = provider_dir / "skills"
_reject_symlink(provider_dir, "provider directory")
_reject_symlink(skills_dir, "skills directory")
if not skills_dir.is_dir():
return SkillSet(provider_name=provider_name)
skills: List[Skill] = []
for entry in sorted(skills_dir.iterdir(), key=lambda path: path.name):
_reject_symlink(entry, "skill directory")
if not entry.is_dir():
continue
_validate_skill_name(entry.name, provider_name)
skill_md = entry / "SKILL.md"
_reject_symlink(skill_md, "SKILL.md")
if not skill_md.is_file():
continue
try:
text = skill_md.read_text(encoding="utf-8")
except OSError as exc:
raise OSError(f"Cannot read {skill_md}: {exc}") from exc
metadata, content = _parse_frontmatter(text)
name, description = _validate_metadata(
metadata, entry.name, provider_name
)
skills.append(
Skill(
name=name,
directory=str(entry),
description=description,
content=content,
)
)
return SkillSet(provider_name=provider_name, skills=skills)
def inject_skills(
prompt: str,
provider_name: str,
skill_names: Optional[List[str]] = None,
on_missing: str = "warn",
) -> str:
"""Prepend enabled provider skills to a prompt."""
skillset = load_provider_skills(provider_name)
enabled = skillset.get(skill_names, on_missing=on_missing)
if not enabled:
return prompt
skill_context = "\n\n".join(
f"## Skill: {skill.name}\n\n{skill.content}" for skill in enabled
)
return f"{skill_context}\n\n---\n\n{prompt}"
+263 -12
View File
@@ -1,11 +1,12 @@
"""Tool loading, saving, and management."""
import os
import re
import shutil
import stat
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional, List, Literal
from typing import Any, Optional, List, Literal
import yaml
@@ -15,6 +16,52 @@ TOOLS_DIR = Path.home() / ".cmdforge"
# Default bin directory for wrapper scripts
BIN_DIR = Path.home() / ".local" / "bin"
_TOOL_REF_RE = re.compile(
r"^(?:[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?/)?[A-Za-z0-9_-]{1,64}$"
)
def _validate_skill_selection(skills: Optional[List[str]]) -> None:
"""Validate a step's optional Agent Skills selection."""
if skills is None:
return
if not isinstance(skills, list):
raise ValueError("skills must be a list of skill names")
if any(not isinstance(name, str) for name in skills):
raise ValueError("skills entries must be strings")
if "*" in skills and skills != ["*"]:
raise ValueError("'*' must be the only entry when enabling all skills")
if len(skills) != len(set(skills)):
raise ValueError("skills must not contain duplicate names")
from .skills import _validate_skill_name
for name in skills:
if name != "*":
_validate_skill_name(name)
def _validate_optional_patterns(patterns: Optional[List[str]], field_name: str) -> None:
if patterns is not None and (
not isinstance(patterns, list)
or not all(isinstance(pattern, str) and pattern for pattern in patterns)
):
raise ValueError(f"{field_name} must be a list of non-empty strings or null")
def validate_json_schema(schema: Optional[dict], field_name: str) -> None:
"""Validate an optional tool contract as a real JSON Schema."""
if schema is None:
return
if not isinstance(schema, dict):
raise ValueError(f"{field_name} must be a JSON Schema object")
from jsonschema.exceptions import SchemaError
from jsonschema.validators import validator_for
try:
validator_for(schema).check_schema(schema)
except SchemaError as exc:
raise ValueError(f"Invalid {field_name}: {exc.message}") from exc
@dataclass
@@ -70,15 +117,45 @@ class ToolArgument:
"""Definition of a custom input argument."""
flag: str # e.g., "--max-size"
variable: str # e.g., "max_size"
default: Optional[str] = None
default: Optional[Any] = None
description: str = ""
type: str = "string"
enum: Optional[List[Any]] = None
required: bool = False
def __post_init__(self) -> None:
# Older CmdForge tools used Python-style scalar names. Normalize them
# on load so typed parsing does not break existing user tools.
legacy_type_aliases = {
"str": "string",
"int": "integer",
"float": "number",
"bool": "boolean",
}
self.type = legacy_type_aliases.get(self.type, self.type)
if self.type not in ("string", "integer", "number", "boolean"):
raise ValueError(
"ToolArgument type must be string, integer, number, or boolean"
)
if self.enum is not None and (
not isinstance(self.enum, list) or not self.enum
):
raise ValueError("ToolArgument enum must be a non-empty list")
if not isinstance(self.required, bool):
raise ValueError("ToolArgument required must be a boolean")
def to_dict(self) -> dict:
d = {"flag": self.flag, "variable": self.variable}
if self.default:
if self.default is not None:
d["default"] = self.default
if self.description:
d["description"] = self.description
if self.type != "string":
d["type"] = self.type
if self.enum is not None:
d["enum"] = self.enum
if self.required:
d["required"] = True
return d
@classmethod
@@ -87,7 +164,10 @@ class ToolArgument:
flag=data["flag"],
variable=data["variable"],
default=data.get("default"),
description=data.get("description", "")
description=data.get("description", ""),
type=data.get("type", "string"),
enum=data.get("enum"),
required=data.get("required", False),
)
@@ -106,6 +186,13 @@ class PromptStep:
max_retries: int = 1 # Retry count on validation failure
plain_text: bool = False # Bypass structured output enforcement
max_tokens: Optional[int] = None # Max output tokens (provider-dependent)
skills: Optional[List[str]] = None # Skill names to enable for this step
fallback_policy: str = "allow" # "allow" | "deny"
def __post_init__(self) -> None:
_validate_skill_selection(self.skills)
if self.fallback_policy not in {"allow", "deny"}:
raise ValueError("PromptStep fallback_policy must be allow or deny")
def to_dict(self) -> dict:
d = {
@@ -130,6 +217,10 @@ class PromptStep:
d["plain_text"] = self.plain_text
if self.max_tokens:
d["max_tokens"] = self.max_tokens
if self.skills is not None:
d["skills"] = self.skills
if self.fallback_policy != "allow":
d["fallback_policy"] = self.fallback_policy
return d
@classmethod
@@ -154,7 +245,9 @@ class PromptStep:
output_schema=data.get("output_schema"),
max_retries=data.get("max_retries", 1),
plain_text=data.get("plain_text", False),
max_tokens=max_tokens
max_tokens=max_tokens,
skills=data.get("skills"),
fallback_policy=data.get("fallback_policy", "allow"),
)
@@ -196,7 +289,18 @@ class ToolStep:
input_template: str = "{input}" # Input template (supports variable substitution)
args: dict = field(default_factory=dict) # Arguments to pass to the tool
provider: Optional[str] = None # Provider override for the called tool
profile: Optional[str] = None # AI persona for the nested tool
skills: Optional[List[str]] = None # Skills to enable in the nested tool
name: Optional[str] = None # Optional display name for the step
tools: Optional[List[str]] = None # Tools the delegated agent may call
def __post_init__(self) -> None:
_validate_skill_selection(self.skills)
_validate_optional_patterns(self.tools, "tools")
if self.profile is not None and (
not isinstance(self.profile, str) or not self.profile
):
raise ValueError("profile must be a non-empty string or null")
def to_dict(self) -> dict:
d = {
@@ -210,6 +314,12 @@ class ToolStep:
d["args"] = self.args
if self.provider:
d["provider"] = self.provider
if self.profile:
d["profile"] = self.profile
if self.skills is not None:
d["skills"] = self.skills
if self.tools is not None:
d["tools"] = self.tools
if self.name:
d["name"] = self.name
return d
@@ -222,12 +332,66 @@ class ToolStep:
input_template=data.get("input", "{input}"),
args=data.get("args", {}),
provider=data.get("provider"),
profile=data.get("profile"),
skills=data.get("skills"),
tools=data.get("tools"),
name=data.get("name")
)
Step = PromptStep | CodeStep | ToolStep
@dataclass
class McpStep:
"""A step that calls an MCP server tool."""
server: str # Server name in ~/.cmdforge/mcp.yaml
tool: str # Tool name on that server
arguments: dict = field(default_factory=dict)
output_var: str = "mcp_result"
result_mode: str = "auto" # auto | structured | content | text
name: Optional[str] = None
def to_dict(self) -> dict:
d = {
"type": "mcp",
"server": self.server,
"tool": self.tool,
"output_var": self.output_var,
}
if self.arguments:
d["arguments"] = self.arguments
if self.result_mode != "auto":
d["result_mode"] = self.result_mode
if self.name:
d["name"] = self.name
return d
@classmethod
def from_dict(cls, data: dict) -> "McpStep":
server = data.get("server")
tool = data.get("tool")
arguments = data.get("arguments", {})
output_var = data.get("output_var", "mcp_result")
result_mode = data.get("result_mode", "auto")
if not isinstance(server, str) or not server.strip():
raise ValueError("MCP step server must be a non-empty string")
if not isinstance(tool, str) or not tool.strip():
raise ValueError("MCP step tool must be a non-empty string")
if not isinstance(arguments, dict):
raise ValueError("MCP step arguments must be a mapping")
if not isinstance(output_var, str) or not output_var.strip():
raise ValueError("MCP step output_var must be a non-empty string")
if result_mode not in ("auto", "structured", "content", "text"):
raise ValueError("MCP step result_mode must be auto, structured, content, or text")
return cls(
server=server,
tool=tool,
arguments=arguments,
output_var=output_var,
result_mode=result_mode,
name=data.get("name"),
)
Step = PromptStep | CodeStep | ToolStep | McpStep
@dataclass
class ToolSource:
@@ -302,10 +466,33 @@ class Tool:
source: Optional[ToolSource] = None # Attribution for imported/external tools
version: str = "" # Tool version
visibility: str = "public" # "public", "private", or "unlisted"
input_schema: Optional[dict] = None # JSON Schema for tool input contract
output_schema: Optional[dict] = None # JSON Schema for tool output contract
path: Optional[Path] = None # Path to config.yaml (set by load_tool)
deprecated: bool = False # Tool is deprecated
deprecated_message: str = "" # Migration guidance for deprecated tools
replacement: Optional[str] = None # Suggested replacement tool name
def __post_init__(self) -> None:
validate_json_schema(self.input_schema, "input_schema")
validate_json_schema(self.output_schema, "output_schema")
if not isinstance(self.deprecated, bool):
raise ValueError("deprecated must be true or false")
if not isinstance(self.deprecated_message, str):
raise ValueError("deprecated_message must be a string")
if len(self.deprecated_message) > 500:
raise ValueError("deprecated_message must be at most 500 characters")
if self.replacement is not None and (
not isinstance(self.replacement, str)
or not _TOOL_REF_RE.fullmatch(self.replacement)
):
raise ValueError("replacement must be a tool name or owner/tool reference")
@classmethod
def from_dict(cls, data: dict) -> "Tool":
if not isinstance(data, dict):
raise ValueError("Tool configuration must be a YAML mapping")
arguments = []
for arg in data.get("arguments", []):
arguments.append(ToolArgument.from_dict(arg))
@@ -318,11 +505,21 @@ class Tool:
steps.append(CodeStep.from_dict(step))
elif step.get("type") == "tool":
steps.append(ToolStep.from_dict(step))
elif step.get("type") == "mcp":
steps.append(McpStep.from_dict(step))
# Parse source attribution if present
source = None
if "source" in data:
source = ToolSource.from_dict(data["source"])
source_data = data["source"]
if isinstance(source_data, dict):
source = ToolSource.from_dict(source_data)
elif isinstance(source_data, str) and source_data.strip():
source = ToolSource(
type="imported", original_tool=source_data.strip()
)
elif source_data is not None:
raise ValueError("source must be an attribution object or string")
# Normalize dependencies - can be strings or dicts with name/version
raw_deps = data.get("dependencies", [])
@@ -352,6 +549,11 @@ class Tool:
source=source,
version=data.get("version", ""),
visibility=data.get("visibility", "public"),
deprecated=data.get("deprecated", False),
deprecated_message=data.get("deprecated_message", ""),
replacement=data.get("replacement"),
input_schema=data.get("input_schema"),
output_schema=data.get("output_schema"),
)
def to_dict(self) -> dict:
@@ -367,6 +569,16 @@ class Tool:
# Only include visibility if it's not the default
if self.visibility and self.visibility != "public":
d["visibility"] = self.visibility
if self.deprecated:
d["deprecated"] = True
if self.deprecated_message:
d["deprecated_message"] = self.deprecated_message
if self.replacement:
d["replacement"] = self.replacement
if self.input_schema is not None:
d["input_schema"] = self.input_schema
if self.output_schema is not None:
d["output_schema"] = self.output_schema
# Include source attribution if present
if self.source:
d["source"] = self.source.to_dict()
@@ -401,6 +613,11 @@ def get_tools_dir() -> Path:
return TOOLS_DIR
def get_project_tools_dir(project_dir: Optional[Path] = None) -> Path:
"""Return the canonical project-local tool root without creating it."""
return (project_dir or Path.cwd()).resolve() / ".cmdforge"
def ensure_settings(tool_dir: Path) -> Optional[Path]:
"""Ensure settings.yaml exists if defaults.yaml exists.
@@ -550,12 +767,32 @@ def load_tool(name: str) -> Optional[Tool]:
_REGISTRY_FIELDS = ("registry_hash", "registry_status", "registry_owner", "registry_feedback")
def save_tool(tool: Tool) -> Path:
"""Save a tool to disk, preserving registry metadata from existing config."""
tool_dir = get_tools_dir() / tool.name
def save_tool(
tool: Tool,
*,
tools_dir: Optional[Path] = None,
create_wrapper: Optional[bool] = None,
) -> Path:
"""Save a tool globally or under an explicit project-local tool root.
Global saves keep the historical wrapper behavior. Project-local saves are
invoked through ``cmdforge run`` so a global wrapper cannot accidentally
resolve a different project tool with the same name.
"""
valid, error = validate_tool_name(tool.name)
if not valid:
raise ValueError(error)
global_tools_dir = get_tools_dir()
target_root = Path(tools_dir).resolve() if tools_dir is not None else global_tools_dir
target_root.mkdir(parents=True, exist_ok=True)
tool_dir = target_root / tool.name
if tool_dir.is_symlink():
raise ValueError(f"Refusing to save through symlinked tool directory: {tool_dir}")
tool_dir.mkdir(parents=True, exist_ok=True)
config_path = tool_dir / "config.yaml"
if config_path.is_symlink():
raise ValueError(f"Refusing to overwrite symlinked tool config: {config_path}")
# Preserve registry fields from existing config (not part of Tool model)
preserved = {}
@@ -572,8 +809,12 @@ def save_tool(tool: Tool) -> Path:
new_data.update(preserved)
config_path.write_text(yaml.dump(new_data, default_flow_style=False, sort_keys=False))
# Create wrapper script
create_wrapper_script(tool.name)
# Global tools retain convenient shell wrappers. Project tools are
# intentionally cwd-scoped and must be resolved by ``cmdforge run``.
if create_wrapper is None:
create_wrapper = target_root == global_tools_dir.resolve()
if create_wrapper:
create_wrapper_script(tool.name)
# Ensure settings.yaml exists if defaults.yaml exists
ensure_settings(tool_dir)
@@ -627,6 +868,16 @@ def tool_exists(name: str) -> bool:
return (get_tools_dir() / name / "config.yaml").exists()
def deprecation_warning(tool: Tool) -> Optional[str]:
"""Return consistent migration guidance for a deprecated tool."""
if not tool.deprecated:
return None
message = tool.deprecated_message or "This tool is no longer maintained."
if tool.replacement:
message += f" Use '{tool.replacement}' instead."
return f"Warning: '{tool.name}' is deprecated. {message}"
def validate_tool_name(name: str) -> tuple[bool, str]:
"""
Validate a tool name.
+202
View File
@@ -0,0 +1,202 @@
"""Opt-in, local-only pipeline usage discovery.
Only tool names, anonymous pipe inode numbers, counts, and timestamps are
stored. Inputs, outputs, arguments, environment, and working directories are
never recorded.
"""
from __future__ import annotations
import json
import os
import stat
import tempfile
from contextlib import contextmanager
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, Iterator, List, Optional
from .config import CONFIG_DIR
USAGE_FILE = CONFIG_DIR / "usage.json"
USAGE_LOCK_FILE = CONFIG_DIR / ".usage.lock"
USAGE_VERSION = 1
EVENT_TTL_SECONDS = 120
MAX_EVENTS = 200
DEFAULT_THRESHOLD = 3
def _empty_document(enabled: bool = False) -> dict:
return {
"version": USAGE_VERSION,
"enabled": enabled,
"events": [],
"pipelines": {},
}
@contextmanager
def _locked() -> Iterator[None]:
CONFIG_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
with open(USAGE_LOCK_FILE, "a+", encoding="utf-8") as lock:
os.chmod(USAGE_LOCK_FILE, 0o600)
try:
import fcntl
fcntl.flock(lock.fileno(), fcntl.LOCK_EX)
except ImportError:
pass
try:
yield
finally:
try:
import fcntl
fcntl.flock(lock.fileno(), fcntl.LOCK_UN)
except ImportError:
pass
def _load_unlocked() -> dict:
if not USAGE_FILE.exists():
return _empty_document()
try:
data = json.loads(USAGE_FILE.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return _empty_document()
if not isinstance(data, dict) or data.get("version") != USAGE_VERSION:
return _empty_document()
return data
def _save_unlocked(data: dict) -> None:
CONFIG_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
temp_path = None
try:
with tempfile.NamedTemporaryFile(
"w", encoding="utf-8", dir=CONFIG_DIR,
prefix=".usage-", suffix=".tmp", delete=False,
) as handle:
temp_path = Path(handle.name)
os.chmod(temp_path, 0o600)
json.dump(data, handle, indent=2, sort_keys=True)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temp_path, USAGE_FILE)
os.chmod(USAGE_FILE, 0o600)
finally:
if temp_path and temp_path.exists():
temp_path.unlink()
def is_enabled() -> bool:
with _locked():
return bool(_load_unlocked().get("enabled", False))
def set_enabled(enabled: bool) -> None:
with _locked():
data = _load_unlocked()
data["enabled"] = bool(enabled)
if not enabled:
data["events"] = []
_save_unlocked(data)
def clear_usage() -> None:
with _locked():
enabled = bool(_load_unlocked().get("enabled", False))
_save_unlocked(_empty_document(enabled=enabled))
def _pipe_inode(stream) -> Optional[int]:
try:
info = os.fstat(stream.fileno())
except (AttributeError, OSError, ValueError):
return None
return info.st_ino if stat.S_ISFIFO(info.st_mode) else None
def record_invocation(tool_name: str, stdin=None, stdout=None) -> None:
"""Record a successful invocation when it participates in a shell pipe."""
import sys
stdin = stdin or sys.stdin
stdout = stdout or sys.stdout
input_pipe = _pipe_inode(stdin)
output_pipe = _pipe_inode(stdout)
if input_pipe is None and output_pipe is None:
return
now = datetime.now(timezone.utc)
timestamp = now.timestamp()
with _locked():
data = _load_unlocked()
if not data.get("enabled", False):
return
events = [
event for event in data.get("events", [])
if timestamp - float(event.get("time", 0)) <= EVENT_TTL_SECONDS
]
discovered = set()
for event in events:
if input_pipe is not None and event.get("stdout_pipe") == input_pipe:
discovered.add((event["tool"], tool_name))
if output_pipe is not None and event.get("stdin_pipe") == output_pipe:
discovered.add((tool_name, event["tool"]))
pipelines = data.setdefault("pipelines", {})
for first, second in discovered:
if first == second:
continue
key = json.dumps([first, second], separators=(",", ":"))
current = pipelines.setdefault(key, {"tools": [first, second], "count": 0})
current["count"] = int(current.get("count", 0)) + 1
current["last_seen"] = now.isoformat()
events.append({
"tool": tool_name,
"time": timestamp,
"stdin_pipe": input_pipe,
"stdout_pipe": output_pipe,
})
data["events"] = events[-MAX_EVENTS:]
_save_unlocked(data)
def get_suggestions(threshold: int = DEFAULT_THRESHOLD) -> List[Dict]:
with _locked():
data = _load_unlocked()
if not data.get("enabled", False):
return []
suggestions = [
{
"tools": list(item.get("tools", [])),
"count": int(item.get("count", 0)),
"last_seen": item.get("last_seen", ""),
}
for item in data.get("pipelines", {}).values()
if int(item.get("count", 0)) >= threshold
and len(item.get("tools", [])) >= 2
]
return sorted(suggestions, key=lambda item: (-item["count"], item["tools"]))
def build_composite_tool(name: str, tools: List[str]):
"""Build, but do not save, a Tool that composes the suggested pipeline."""
from .tool import Tool, ToolStep
if len(tools) < 2:
raise ValueError("A composite requires at least two tools")
steps = []
previous = "input"
for index, tool_name in enumerate(tools, start=1):
output_var = f"pipeline_{index}"
steps.append(ToolStep(
tool=tool_name,
input_template="{input}" if previous == "input" else f"{{{previous}}}",
output_var=output_var,
))
previous = output_var
return Tool(
name=name,
description="Composite pipeline: " + " | ".join(tools),
dependencies=list(dict.fromkeys(tools)),
steps=steps,
output=f"{{{previous}}}",
)
+82 -26
View File
@@ -34,6 +34,13 @@ cmdforge create # CLI wizard
# Or install a tool from the registry
cmdforge registry install official/summarize
# Discover tools from scripts or an AI coding assistant
cmdforge list --json --filter "summarize" --limit 10
cmdforge registry search "summarize text" --json --limit 5
# Run a one-off prompt without creating a tool
cat article.txt | cmdforge run-once "Summarize this:\n\n{input}"
# Use it!
cat article.txt | summarize
@@ -46,6 +53,26 @@ cf</code></pre>
<code>cmdforge create</code> uses a command-line wizard. Both create the same YAML config files.</p>
</div>
<h2 id="coding-agents">Using CmdForge with Coding Agents</h2>
<p>CmdForge can expose installed tools to supported coding hosts over MCP. Preview the
configuration first; CmdForge updates only its marked policy block and does not expose
tools that you have not explicitly allowed.</p>
<pre><code class="language-bash">cmdforge mcp configure codex --dry-run
cmdforge mcp configure codex
# Claude Code is supported too
cmdforge mcp configure claude-code --dry-run</code></pre>
<p>For a reusable workflow owned by an application, create it inside that application's
<code>.cmdforge/</code> directory. Do not modify the CmdForge source repository merely to
add a tool for another project.</p>
<pre><code class="language-bash">cd your-project
cmdforge create classify-docs --project
# Or use the official AI-assisted creator
cmdforge registry install official/forge-tool
echo "Create a bounded document classifier" \
| forge-tool --name classify-docs --project</code></pre>
<h2 id="how-it-works">How It Works</h2>
<p>Each tool is a YAML file that defines:</p>
<ol>
@@ -96,6 +123,7 @@ output: "{summary}"</code></pre>
"headings": [
("what-is-cmdforge", "What is CmdForge?"),
("quick-start", "Quick Start"),
("coding-agents", "Using CmdForge with Coding Agents"),
("how-it-works", "How It Works"),
("next-steps", "Next Steps"),
("get-help", "Get Help"),
@@ -160,8 +188,13 @@ pip install -e ".[dev]"</code></pre>
<p class="lead">Let's create a simple tool that explains code. You'll learn the basics of tool configuration.</p>
<h2 id="create-tool">Create the Tool</h2>
<p>Run the interactive creator:</p>
<pre><code class="language-bash">cmdforge create</code></pre>
<p>Run the interactive creator. Use <code>--project</code> when the tool belongs to the
current application and should be versioned with it:</p>
<pre><code class="language-bash"># Personal tool in ~/.cmdforge/
cmdforge create explain
# Project-owned tool in ./.cmdforge/
cmdforge create explain --project</code></pre>
<p>Or create the file manually at <code>~/.cmdforge/explain/config.yaml</code>:</p>
<pre><code class="language-yaml">name: explain
@@ -550,7 +583,7 @@ from multiple AI providers simultaneously.</p>
from concurrent.futures import ThreadPoolExecutor, as_completed
def run_tool(tool_name: str, input_text: str) -> dict:
\"\"\"Run a SmartTool and return its output.\"\"\"
\"\"\"Run a CmdForge tool and return its output.\"\"\"
result = subprocess.run(
[tool_name],
input=input_text,
@@ -600,7 +633,7 @@ import subprocess
import json
from concurrent.futures import ThreadPoolExecutor, as_completed
# Define your perspective tools (each is a SmartTool)
# Define your perspective tools (each is a CmdForge tool)
PERSPECTIVES = [
"perspective-optimist", # Focuses on opportunities
"perspective-critic", # Identifies problems
@@ -758,10 +791,10 @@ project. It implements:</p>
},
"yaml-config": {
"title": "The Anatomy of a SmartTool",
"description": "Master the YAML configuration that powers every SmartTool",
"title": "The Anatomy of a CmdForge Tool",
"description": "Master the YAML configuration that powers every CmdForge tool",
"content": """
<p class="lead">Every SmartTool is just a YAML file with a secret superpower: it turns plain English
<p class="lead">Every CmdForge tool is just a YAML file with a secret superpower: it turns plain English
instructions into Unix commands. In the next 10 minutes, you'll understand exactly how that magic works.</p>
<div class="bg-indigo-50 border-l-4 border-indigo-500 p-4 my-6">
@@ -797,7 +830,7 @@ HELLO WORLD!!!</code></pre>
<h2 id="the-five-parts">The Five Parts of Every Tool</h2>
<p>Think of a SmartTool config like a recipe card:</p>
<p>Think of a CmdForge tool config like a recipe card:</p>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4 my-6">
<div class="bg-white border rounded-lg p-4">
@@ -4004,12 +4037,12 @@ cat file.txt | summarize --provider mock</code></pre>
<td class="px-4 py-2">Launch the Visual Builder (desktop GUI)</td>
</tr>
<tr class="border-b">
<td class="px-4 py-2"><code>cmdforge list</code></td>
<td class="px-4 py-2">List all installed tools</td>
<td class="px-4 py-2"><code>cmdforge list [--json] [--filter QUERY]</code></td>
<td class="px-4 py-2">List or machine-search installed tools</td>
</tr>
<tr class="border-b">
<td class="px-4 py-2"><code>cmdforge create</code></td>
<td class="px-4 py-2">Create a new tool (interactive wizard)</td>
<td class="px-4 py-2"><code>cmdforge create NAME [--project]</code></td>
<td class="px-4 py-2">Create a personal or project-owned tool</td>
</tr>
<tr class="border-b">
<td class="px-4 py-2"><code>cmdforge edit &lt;tool&gt;</code></td>
@@ -4027,6 +4060,10 @@ cat file.txt | summarize --provider mock</code></pre>
<td class="px-4 py-2"><code>cmdforge run &lt;tool&gt;</code></td>
<td class="px-4 py-2">Run a tool directly (without wrapper)</td>
</tr>
<tr class="border-b">
<td class="px-4 py-2"><code>cmdforge run-once "Instruction {input}"</code></td>
<td class="px-4 py-2">Run an ad-hoc prompt without creating a tool</td>
</tr>
<tr class="border-b">
<td class="px-4 py-2"><code>cmdforge refresh</code></td>
<td class="px-4 py-2">Regenerate all wrapper scripts</td>
@@ -4301,6 +4338,14 @@ cf | cf | cf</code></pre>
},
}
# The original manual remains useful for the fundamentals. Newer architecture
# chapters are maintained together so post-M6 features do not become scattered
# one-off additions to otherwise unrelated pages. Entries in MODERN_DOCS may
# deliberately replace an older page (currently the CLI reference).
from .docs_modern import MODERN_DOCS
DOCS.update(MODERN_DOCS)
def get_doc(path: str) -> dict:
"""Get documentation content by path."""
@@ -4310,32 +4355,43 @@ def get_doc(path: str) -> dict:
def get_toc():
"""Get table of contents structure."""
"""Return the documentation as a small, deliberately ordered book."""
from types import SimpleNamespace
return [
SimpleNamespace(slug="getting-started", title="Getting Started", children=[
SimpleNamespace(slug="getting-started", title="Start Here", children=[
SimpleNamespace(slug="installation", title="Installation"),
SimpleNamespace(slug="first-tool", title="Your First Tool"),
SimpleNamespace(slug="forge-tool", title="Forge a Tool with AI"),
SimpleNamespace(slug="interactive-picker", title="Interactive Picker (cf)"),
SimpleNamespace(slug="visual-builder", title="Visual Builder"),
SimpleNamespace(slug="yaml-config", title="YAML Config"),
]),
SimpleNamespace(slug="registry-usage", title="Using the Registry", children=[
SimpleNamespace(slug="collections", title="Tool Collections"),
]),
SimpleNamespace(slug="arguments", title="Custom Arguments", children=[]),
SimpleNamespace(slug="multi-step", title="Multi-Step Workflows", children=[
SimpleNamespace(slug="yaml-config", title="Building Tools", children=[
SimpleNamespace(slug="arguments", title="Arguments and Inputs"),
SimpleNamespace(slug="multi-step", title="Multi-Step Workflows"),
SimpleNamespace(slug="code-steps", title="Code Steps"),
SimpleNamespace(slug="tool-steps", title="Tools Within Tools"),
SimpleNamespace(slug="contracts-quality", title="Contracts and Quality"),
]),
SimpleNamespace(slug="testing-steps", title="Testing Sandbox", children=[]),
SimpleNamespace(slug="providers", title="Providers", children=[
SimpleNamespace(slug="providers", title="Providers and Delegation", children=[
SimpleNamespace(slug="provider-setup", title="Provider Setup"),
SimpleNamespace(slug="provider-policy", title="Privacy and Routing Policy"),
SimpleNamespace(slug="skills-delegation", title="Skills and Delegation"),
]),
SimpleNamespace(slug="project-deps", title="Project Dependencies", children=[]),
SimpleNamespace(slug="publishing", title="Publishing", children=[]),
SimpleNamespace(slug="advanced-workflows", title="Advanced Workflows", children=[
SimpleNamespace(slug="mcp-overview", title="MCP and Coding Agents", children=[
SimpleNamespace(slug="mcp-client", title="Calling MCP Servers"),
SimpleNamespace(slug="mcp-server", title="Serving CmdForge Tools"),
SimpleNamespace(slug="agent-integration", title="Coding Agent Integration"),
]),
SimpleNamespace(slug="registry-usage", title="Discover, Trust, and Share", children=[
SimpleNamespace(slug="collections", title="Tool Collections"),
SimpleNamespace(slug="publishing", title="Publishing"),
SimpleNamespace(slug="trust-publishing", title="Trust and Provenance"),
]),
SimpleNamespace(slug="project-deps", title="Projects and Advanced Use", children=[
SimpleNamespace(slug="testing-steps", title="Testing Sandbox"),
SimpleNamespace(slug="optimization-usage", title="Optimization and Usage"),
SimpleNamespace(slug="advanced-workflows", title="Advanced Workflows"),
SimpleNamespace(slug="parallel-orchestration", title="Parallel Orchestration"),
]),
SimpleNamespace(slug="cli-reference", title="CLI Reference", children=[]),
SimpleNamespace(slug="cli-reference", title="The Command Atlas", children=[]),
]
File diff suppressed because it is too large Load Diff
+81 -73
View File
@@ -1,72 +1,103 @@
"""Email sending utilities for CmdForge web.
"""Transactional email utilities for the CmdForge web application.
In development mode, emails are logged to the console instead of being sent.
To enable real email sending, set MAIL_ENABLED=true and configure SMTP settings.
Production delivery uses Resend's HTTPS API. Email bodies and reset URLs are
never written to logs. Console delivery is an explicit development-only mode.
"""
from __future__ import annotations
import logging
from html import escape
from typing import Optional
import requests
from flask import current_app
logger = logging.getLogger(__name__)
RESEND_ENDPOINT = "https://api.resend.com/emails"
SUPPORTED_TRANSPORTS = {"disabled", "console", "resend"}
def _transport() -> str:
transport = str(current_app.config.get("MAIL_TRANSPORT", "disabled")).strip().lower()
if transport not in SUPPORTED_TRANSPORTS:
logger.error("Unsupported email transport configured; delivery disabled")
return "disabled"
return transport
def send_email(to: str, subject: str, html_body: str, text_body: Optional[str] = None) -> bool:
"""Send an email.
"""Deliver an email without logging its recipient or contents.
In dev mode (MAIL_ENABLED=false or unset), logs the email to console.
In production (MAIL_ENABLED=true), sends via SMTP.
Args:
to: Recipient email address
subject: Email subject
html_body: HTML content of the email
text_body: Plain text content (optional)
Returns:
True if email was sent/logged successfully, False otherwise
``MAIL_TRANSPORT`` must be one of ``disabled``, ``console``, or ``resend``.
Console mode records only that a message was suppressed and reports no
delivery, so reset tokens are invalidated. It is also rejected in
production. Resend mode requires ``RESEND_API_KEY`` and ``MAIL_FROM``.
"""
mail_enabled = current_app.config.get("MAIL_ENABLED", False)
transport = _transport()
environment = str(current_app.config.get("CMDFORGE_ENV", "development")).lower()
if mail_enabled:
# Future: implement real SMTP sending
# smtp_host = current_app.config.get("MAIL_SERVER", "localhost")
# smtp_port = current_app.config.get("MAIL_PORT", 587)
# smtp_user = current_app.config.get("MAIL_USERNAME")
# smtp_pass = current_app.config.get("MAIL_PASSWORD")
# smtp_tls = current_app.config.get("MAIL_USE_TLS", True)
logger.warning("MAIL_ENABLED is true but SMTP is not implemented yet. Falling back to console logging.")
if transport == "disabled":
logger.warning("Email delivery is disabled")
return False
# Dev mode: log to console
logger.info("=" * 60)
logger.info("[EMAIL] To: %s", to)
logger.info("[EMAIL] Subject: %s", subject)
logger.info("[EMAIL] Body:")
if transport == "console":
if environment == "production":
logger.error("Console email transport is not permitted in production")
return False
logger.info("Development email suppressed (subject length=%d)", len(subject))
return False
api_key = str(current_app.config.get("RESEND_API_KEY") or "").strip()
sender = str(current_app.config.get("MAIL_FROM") or "").strip()
if not api_key or not sender:
logger.error("Resend email transport is missing required configuration")
return False
try:
timeout = float(current_app.config.get("MAIL_TIMEOUT", 10.0))
except (TypeError, ValueError):
logger.error("Invalid email timeout configured")
return False
if timeout <= 0 or timeout > 60:
logger.error("Email timeout must be greater than zero and at most 60 seconds")
return False
payload = {
"from": sender,
"to": [to],
"subject": subject,
"html": html_body,
}
if text_body:
logger.info(text_body)
else:
logger.info(html_body)
logger.info("=" * 60)
payload["text"] = text_body
try:
response = requests.post(
RESEND_ENDPOINT,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
json=payload,
timeout=timeout,
)
response.raise_for_status()
except requests.RequestException as exc:
# Do not log the response body, recipient, request headers, or message;
# providers can echo sensitive request data in error responses.
logger.error("Transactional email delivery failed (%s)", type(exc).__name__)
return False
logger.info("Transactional email accepted by provider")
return True
def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
"""Send a password reset email.
Args:
to: Recipient email address
token: The password reset token
base_url: Base URL of the application (e.g., https://cmdforge.brrd.tech)
Returns:
True if email was sent/logged successfully
"""
"""Send a one-hour password reset link using the configured transport."""
reset_url = f"{base_url.rstrip('/')}/reset-password?token={token}"
html_reset_url = escape(reset_url, quote=True)
subject = "Reset your CmdForge password"
html_body = f"""
@@ -86,28 +117,15 @@ def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
</head>
<body>
<div class="container">
<div class="header">
<div class="logo">CmdForge</div>
</div>
<div class="header"><div class="logo">CmdForge</div></div>
<p>Hello,</p>
<p>You requested to reset your password for your CmdForge account. Click the button below to set a new password:</p>
<p style="text-align: center; margin: 30px 0;">
<a href="{reset_url}" class="button">Reset Password</a>
</p>
<p style="text-align: center; margin: 30px 0;"><a href="{html_reset_url}" class="button">Reset Password</a></p>
<p>Or copy and paste this link into your browser:</p>
<p class="link">{reset_url}</p>
<p class="link">{html_reset_url}</p>
<p><strong>This link will expire in 1 hour.</strong></p>
<p>If you didn't request this password reset, you can safely ignore this email. Your password will remain unchanged.</p>
<div class="footer">
<p>This email was sent by CmdForge. If you have questions, visit our website or contact support.</p>
</div>
<div class="footer"><p>This email was sent by CmdForge.</p></div>
</div>
</body>
</html>
@@ -115,20 +133,10 @@ def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
text_body = f"""Reset your CmdForge password
Hello,
You requested to reset your password for your CmdForge account.
Reset your password by visiting this link:
Visit this link to set a new password:
{reset_url}
This link will expire in 1 hour.
If you didn't request this password reset, you can safely ignore this email.
Your password will remain unchanged.
---
CmdForge
This link will expire in 1 hour. If you did not request this reset, ignore this email.
"""
return send_email(to, subject, html_body, text_body)
+4 -4
View File
@@ -3,7 +3,7 @@
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}CmdForge{% endblock %} - Build Custom AI Commands</title>
<title>{% block title %}CmdForge{% endblock %} - Build AI Tools You Own</title>
<!-- Meta tags -->
<meta name="description" content="{% block meta_description %}Create Unix-style pipeable tools that work with any AI provider. Provider-agnostic, composable, and community-driven.{% endblock %}">
@@ -11,7 +11,7 @@
<!-- Open Graph -->
<meta property="og:title" content="{% block og_title %}CmdForge{% endblock %}">
<meta property="og:description" content="{% block og_description %}Build custom AI commands in YAML{% endblock %}">
<meta property="og:description" content="{% block og_description %}Build, compose, test, and share AI tools that run on your computer{% endblock %}">
<meta property="og:type" content="website">
<meta property="og:url" content="{{ request.url }}">
{% block og_extra %}{% endblock %}
@@ -19,7 +19,7 @@
<!-- Twitter Card -->
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="{% block twitter_title %}CmdForge{% endblock %}">
<meta name="twitter:description" content="{% block twitter_description %}Build custom AI commands in YAML{% endblock %}">
<meta name="twitter:description" content="{% block twitter_description %}Build, compose, test, and share AI tools that run on your computer{% endblock %}">
{% block twitter_extra %}{% endblock %}
<!-- Canonical URL -->
@@ -42,7 +42,7 @@
"@type": "Organization",
"name": "CmdForge",
"url": "{{ request.host_url }}",
"description": "Build custom AI commands in YAML"
"description": "Build, compose, test, and share AI tools that run on your computer"
}
</script>
{% endblock %}
+29 -28
View File
@@ -3,31 +3,31 @@
{% from "components/tutorial_card.html" import tutorial_card %}
{% from "components/contributor_card.html" import contributor_card %}
{% block title %}CmdForge - Compose AI Capabilities{% endblock %}
{% block title %}CmdForge - Build AI Tools You Own{% endblock %}
{% block meta_description %}Stop searching for libraries. Ask for capabilities. CmdForge lets you describe what you need and compose AI-powered tools that work together.{% endblock %}
{% block meta_description %}Turn useful AI workflows into commands you own: local or cloud providers, tested pipelines, Unix pipes, registry sharing, and MCP for coding agents.{% endblock %}
{% block content %}
<!-- Hero Section -->
<section class="bg-white py-16 md:py-24">
<div class="max-w-5xl mx-auto px-4 sm:px-6 lg:px-8 text-center">
<p class="text-indigo-600 font-medium mb-4">From Libraries to Capabilities</p>
<p class="text-indigo-600 font-medium mb-4">Your Models. Your Workflows. Your Computer.</p>
<h1 class="text-4xl md:text-5xl font-bold text-gray-900 leading-tight">
Ask for Capabilities.<br>Compose Solutions.
Turn AI Workflows<br>Into Tools You Own.
</h1>
<p class="mt-6 text-xl text-gray-600 max-w-3xl mx-auto">
Stop searching through documentation. Describe what you need in plain English
and discover AI-powered tools that work together like Unix pipes.
Build one useful command, connect it to another, and keep the result on your machine.
CmdForge makes AI capabilities composable like Unix pipes—and callable from coding agents through MCP.
</p>
<!-- Semantic Search Demo -->
<div class="mt-10 max-w-2xl mx-auto">
<div class="bg-gray-900 rounded-lg p-4 text-left font-mono text-sm">
<p class="text-gray-400 mb-2"># Ask for what you need:</p>
<p class="text-green-400">$ cmdforge registry describe "something that can summarize long documents"</p>
<p class="text-gray-300 mt-2">Found 3 matching tools:</p>
<p class="text-cyan-400"> rob/summarize</p>
<p class="text-gray-500 ml-4">Condenses text while preserving key points</p>
<p class="text-gray-400 mb-2"># Improvise once. Keep what becomes useful.</p>
<p class="text-green-400">$ cat meeting.txt | cmdforge run-once \</p>
<p class="text-green-400 ml-4">"Extract decisions and owners from {input}"</p>
<p class="text-gray-300 mt-3"># Reusable tomorrow—from your shell or your coding agent:</p>
<p class="text-cyan-400">$ cat meeting.txt | meeting-decisions</p>
</div>
</div>
@@ -62,11 +62,10 @@
<section class="py-16 bg-gray-50">
<div class="max-w-6xl mx-auto px-4 sm:px-6 lg:px-8">
<h2 class="text-3xl font-bold text-gray-900 text-center mb-4">
A New Way to Build
Small Tools, Serious Leverage
</h2>
<p class="text-center text-gray-600 mb-12 max-w-2xl mx-auto">
Traditional development: search docs, install libraries, write glue code.<br>
CmdForge: describe what you need, compose existing capabilities, ship.
Start with a command that solves one problem. Add contracts, composition, and sharing only when the work earns it.
</p>
<div class="grid grid-cols-1 md:grid-cols-3 gap-8">
@@ -77,10 +76,10 @@
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 10h.01M12 10h.01M16 10h.01M9 16H5a2 2 0 01-2-2V6a2 2 0 012-2h14a2 2 0 012 2v8a2 2 0 01-2 2h-5l-5 5v-5z"/>
</svg>
</div>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Ask for Capabilities</h3>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Discover or Create</h3>
<p class="text-gray-600">
Semantic search finds tools by what they <em>do</em>, not what they're named.
"I need something that extracts key points" just works.
Search local tools and the registry by what they do. For a new idea, forge a
personal or project-owned tool from plain language.
</p>
</div>
@@ -91,10 +90,10 @@
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 5a1 1 0 011-1h14a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1V5zM4 13a1 1 0 011-1h6a1 1 0 011 1v6a1 1 0 01-1 1H5a1 1 0 01-1-1v-6zM16 13a1 1 0 011-1h2a1 1 0 011 1v6a1 1 0 01-1 1h-2a1 1 0 01-1-1v-6z"/>
</svg>
</div>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Compose Solutions</h3>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Compose and Verify</h3>
<p class="text-gray-600">
Tools call other tools. Chain capabilities together like Unix pipes.
Build complex workflows from simple, tested components.
Join prompts, Python, CmdForge tools, and MCP calls. Contracts, preflight,
regression evidence, and quality coverage keep the result inspectable.
</p>
</div>
@@ -105,10 +104,10 @@
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8.684 13.342C8.886 12.938 9 12.482 9 12c0-.482-.114-.938-.316-1.342m0 2.684a3 3 0 110-2.684m0 2.684l6.632 3.316m-6.632-6l6.632-3.316m0 0a3 3 0 105.367-2.684 3 3 0 00-5.367 2.684zm0 9.316a3 3 0 105.368 2.684 3 3 0 00-5.368-2.684z"/>
</svg>
</div>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Share & Discover</h3>
<h3 class="text-xl font-semibold text-gray-900 mb-3">Use It Everywhere</h3>
<p class="text-gray-600">
Every tool you publish becomes a capability others can use.
Collaboration over competition. Build on each other's progress.
Run tools as ordinary commands, compose them in projects, publish immutable
versions, or expose a deliberate allowlist to Codex and Claude Code over MCP.
</p>
</div>
</div>
@@ -156,10 +155,12 @@
<div class="bg-gray-900 rounded-lg p-4 font-mono text-sm">
<p class="text-gray-400"># Tools can call other tools:</p>
<p class="text-yellow-400">steps:</p>
<p class="text-gray-300"> - tool: official/code-review</p>
<p class="text-gray-300"> - type: tool</p>
<p class="text-gray-300"> tool: official/code-review</p>
<p class="text-gray-300"> output_var: review</p>
<p class="text-gray-300"> - tool: rob/summarize</p>
<p class="text-gray-300"> input_template: "{review}"</p>
<p class="text-gray-300"> - type: tool</p>
<p class="text-gray-300"> tool: rob/summarize</p>
<p class="text-gray-300"> input: "{review}"</p>
<p class="text-gray-300"> output_var: summary</p>
<p class="text-gray-400 mt-2"># Use like any Unix command:</p>
<p class="text-green-400">$ cat app.py | my-review-tool</p>
@@ -180,8 +181,8 @@
</svg>
</div>
<div>
<h3 class="text-lg font-semibold text-gray-900">Works With Any AI</h3>
<p class="text-sm text-gray-600">Claude, GPT, Ollama, or any CLI-accessible model. Switch providers without changing tools.</p>
<h3 class="text-lg font-semibold text-gray-900">Choose the Engine at Runtime</h3>
<p class="text-sm text-gray-600">Local models, coding CLIs, and OpenAI-compatible APIs—with explicit privacy, capability, fallback, and provenance controls.</p>
</div>
</div>
<a href="{{ url_for('web.docs', path='providers') }}" class="text-indigo-600 hover:text-indigo-800 font-medium flex items-center whitespace-nowrap">
+44
View File
@@ -0,0 +1,44 @@
"""Small real MCP stdio server used by the client integration tests."""
import os
import time
from mcp.server.fastmcp import FastMCP
server = FastMCP("CmdForge MCP test server")
@server.tool()
def echo(message: str) -> str:
"""Return a message unchanged."""
return message
@server.tool()
def add(a: int, b: int) -> dict[str, int]:
"""Add two integers."""
return {"sum": a + b}
@server.tool()
def read_env(name: str) -> str:
"""Return an environment value, or an empty string if absent."""
return os.environ.get(name, "")
@server.tool()
def current_directory() -> str:
"""Return the server process working directory."""
return os.getcwd()
@server.tool()
def wait(seconds: float) -> str:
"""Block for the requested duration."""
time.sleep(seconds)
return "done"
if __name__ == "__main__":
server.run(transport="stdio")
+71
View File
@@ -0,0 +1,71 @@
"""Tests for M9.5 supply chain attestation."""
from cmdforge.attestation import (
Attestation,
generate_keypair,
sign_tool,
verify_attestation,
verify_content_hash,
verify_trusted_attestation,
)
from cmdforge.tool import Tool
class TestSignTool:
def test_creates_attestation(self):
private, public = generate_keypair()
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
assert att.tool_name == "mytool"
assert att.version == "1.0.0"
assert att.content_hash == "abc123"
assert att.signer == "alice"
assert len(att.signature) == 88
assert att.algorithm == "ed25519"
assert verify_attestation(att, public)
def test_different_keys_different_signatures(self):
key1, _ = generate_keypair()
key2, _ = generate_keypair()
att1 = sign_tool("tool", "1.0.0", "hash", "alice", key1)
att2 = sign_tool("tool", "1.0.0", "hash", "alice", key2)
assert att1.signature != att2.signature
class TestVerifyAttestation:
def test_valid_signature(self):
private, public = generate_keypair()
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
assert verify_attestation(att, public)
def test_wrong_key_fails(self):
private, _ = generate_keypair()
_, wrong_public = generate_keypair()
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
assert not verify_attestation(att, wrong_public)
def test_tampered_content_fails(self):
from dataclasses import replace
private, public = generate_keypair()
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
att = replace(att, content_hash="tampered")
assert not verify_attestation(att, public)
def test_tampered_signer_fails(self):
from dataclasses import replace
private, public = generate_keypair()
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
att = replace(att, signer="eve")
assert not verify_attestation(att, public)
def test_timestamp_and_algorithm_are_signed(self):
from dataclasses import replace
private, public = generate_keypair()
att = sign_tool("tool", "1.0.0", "hash", "alice", private)
assert not verify_attestation(replace(att, signed_at="tomorrow"), public)
assert not verify_attestation(replace(att, algorithm="hmac-sha256"), public)
def test_requires_trusted_publisher_mapping(self):
private, public = generate_keypair()
att = sign_tool("tool", "1.0.0", "hash", "alice", private)
assert verify_trusted_attestation(att, {"alice": public})
assert not verify_trusted_attestation(att, {"mallory": public})
+710 -1
View File
@@ -1,8 +1,10 @@
"""Tests for CLI commands."""
import json
import tempfile
from pathlib import Path
from unittest.mock import patch, MagicMock
from types import SimpleNamespace
from unittest.mock import call, patch, MagicMock
from io import StringIO
import pytest
@@ -32,6 +34,28 @@ class TestCLIBasics:
assert exc_info.value.code == 0
def test_mcp_configure_dispatches_host_options(self):
with (
patch(
"cmdforge.agent_setup.configure_agent_host",
return_value=0,
) as configure,
patch("sys.argv", [
"cmdforge", "mcp", "configure", "claude-code",
"--scope", "project", "--dry-run",
]),
):
assert main() == 0
configure.assert_called_once_with(
"claude-code",
server_name="cmdforge",
scope="project",
install_policy=True,
dry_run=True,
force=False,
)
class TestListCommand:
"""Tests for 'cmdforge list' command."""
@@ -66,6 +90,209 @@ class TestListCommand:
assert 'test-tool' in captured.out
assert 'another-tool' in captured.out
def test_json_is_compact_catalog_unless_full_requested(
self, temp_tools_dir, capsys
):
from cmdforge.tool import save_tool
save_tool(Tool(
name="catalog-tool",
description="Catalog entry",
input_schema={"type": "string"},
output_schema={"type": "string"},
steps=[PromptStep(
prompt="Private implementation {input}",
provider="mock",
output_var="result",
plain_text=True,
)],
output="{result}",
))
with patch('sys.argv', ['cmdforge', 'list', '--json']):
assert main() == 0
compact = json.loads(capsys.readouterr().out)
assert compact[0]["name"] == "catalog-tool"
assert compact[0]["input_schema"] == {"type": "string"}
assert "steps" not in compact[0]
with patch('sys.argv', ['cmdforge', 'list', '--json', '--full']):
assert main() == 0
full = json.loads(capsys.readouterr().out)
assert full[0]["steps"][0]["prompt"] == "Private implementation {input}"
def test_filter_reports_filtered_count_and_empty_result(
self, temp_tools_dir, capsys
):
from cmdforge.tool import save_tool
save_tool(Tool(name="alpha", description="First"))
save_tool(Tool(name="beta", description="Second"))
with patch('sys.argv', ['cmdforge', 'list', '--filter', 'alpha']):
assert main() == 0
output = capsys.readouterr().out
assert "Available tools (1)" in output
assert "alpha" in output
assert "beta" not in output
with patch('sys.argv', ['cmdforge', 'list', '--filter', 'missing']):
assert main() == 0
assert "No tools match filter 'missing'" in capsys.readouterr().out
def test_json_limit_and_invalid_limit(self, temp_tools_dir, capsys):
from cmdforge.tool import save_tool
save_tool(Tool(name="alpha"))
save_tool(Tool(name="beta"))
with patch('sys.argv', ['cmdforge', 'list', '--json', '--limit', '1']):
assert main() == 0
assert len(json.loads(capsys.readouterr().out)) == 1
with patch('sys.argv', ['cmdforge', 'list', '--limit', '0']):
assert main() == 1
assert "at least 1" in capsys.readouterr().err
with patch('sys.argv', ['cmdforge', 'list', '--full']):
assert main() == 1
assert "--full requires --json" in capsys.readouterr().err
def test_list_includes_project_tools_and_prefers_project_shadow(
self, temp_tools_dir, tmp_path, monkeypatch, capsys
):
from cmdforge.tool import save_tool
save_tool(Tool(name="shadowed", description="global"))
project = tmp_path / "project"
project.mkdir()
save_tool(
Tool(name="shadowed", description="project"),
tools_dir=project / ".cmdforge",
)
save_tool(
Tool(name="project-only", description="local"),
tools_dir=project / ".cmdforge",
)
monkeypatch.chdir(project)
with patch("sys.argv", ["cmdforge", "list", "--json"]):
assert main() == 0
payload = json.loads(capsys.readouterr().out)
by_name = {item["name"]: item for item in payload}
assert by_name["shadowed"]["description"] == "project"
assert "project-only" in by_name
assert [item["name"] for item in payload].count("shadowed") == 1
class TestRunOnceCommand:
def test_combines_instruction_with_piped_input(self, capsys):
result = SimpleNamespace(success=True, text="summary", error=None)
with (
patch('sys.argv', [
'cmdforge', 'run-once', 'Summarize:\n{input}',
'--provider', 'mock',
]),
patch('sys.stdin', StringIO('document body')),
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
):
assert main() == 0
assert call_provider.call_args.args[:2] == (
'mock', 'Summarize:\ndocument body'
)
assert capsys.readouterr().out == "summary\n"
def test_appends_piped_input_when_template_has_no_placeholder(self):
result = SimpleNamespace(success=True, text="ok", error=None)
with (
patch('sys.argv', [
'cmdforge', 'run-once', 'Summarize this', '--provider', 'mock',
]),
patch('sys.stdin', StringIO('document body')),
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
):
assert main() == 0
assert call_provider.call_args.args[1] == "Summarize this\n\ndocument body"
def test_uses_configured_default_provider(self):
result = SimpleNamespace(success=True, text="ok", error=None)
config = SimpleNamespace(default_provider="configured-provider")
stdin = MagicMock()
stdin.isatty.return_value = True
with (
patch('sys.argv', ['cmdforge', 'run-once', 'hello']),
patch('sys.stdin', stdin),
patch('cmdforge.config.load_config', return_value=config),
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
):
assert main() == 0
assert call_provider.call_args.args[0] == "configured-provider"
def test_selects_first_available_balanced_provider(self):
from cmdforge.cli.tool_commands import _resolve_one_shot_provider
from cmdforge.providers import Provider
config = SimpleNamespace(default_provider=None)
providers = [
Provider("opencode-pickle", "missing-opencode"),
Provider("codex", "codex exec -"),
]
with (
patch('cmdforge.config.load_config', return_value=config),
patch('cmdforge.providers.load_providers', return_value=providers),
patch(
'cmdforge.cli.tool_commands._provider_is_available',
side_effect=lambda provider: provider.name == "codex",
),
):
assert _resolve_one_shot_provider() == "codex"
@pytest.mark.parametrize(
"arguments,message",
[
(["--timeout", "0"], "--timeout"),
(["--max-tokens", "-1"], "--max-tokens"),
],
)
def test_rejects_invalid_limits_before_provider_call(
self, arguments, message, capsys
):
with (
patch('sys.argv', [
'cmdforge', 'run-once', 'hello', '--provider', 'mock', *arguments,
]),
patch('sys.stdin', StringIO()),
patch('cmdforge.providers.call_provider') as call_provider,
):
assert main() == 1
call_provider.assert_not_called()
assert message in capsys.readouterr().err
def test_json_envelope_contains_runtime_provenance(self, capsys):
from cmdforge.providers import ProviderResult
result = ProviderResult(
text="answer", success=True,
requested_provider="local", actual_provider="local",
attempted_providers=["local"], locality="local",
)
with (
patch("sys.argv", [
"cmdforge", "run-once", "hello", "--provider", "local",
"--no-fallback", "--result-envelope", "json",
]),
patch("sys.stdin", StringIO()),
patch("cmdforge.providers.call_provider", return_value=result),
):
assert main() == 0
payload = json.loads(capsys.readouterr().out)
call = payload["execution"]["provider_calls"][0]
assert call["actual_provider"] == "local"
assert call["fallback_used"] is False
class TestCreateCommand:
"""Tests for 'cmdforge create' command."""
@@ -119,6 +346,36 @@ class TestCreateCommand:
captured = capsys.readouterr()
assert 'invalid' in captured.out.lower() or 'invalid' in captured.err.lower()
def test_create_project_tool_does_not_create_global_wrapper(
self, temp_tools_dir, tmp_path, monkeypatch, capsys
):
project = tmp_path / "project"
project.mkdir()
monkeypatch.chdir(project)
with patch("sys.argv", [
"cmdforge", "create", "project-echo", "--project",
"--prompt", "Echo {input}", "--provider", "mock",
]):
assert main() == 0
config = project / ".cmdforge" / "project-echo" / "config.yaml"
assert config.exists()
assert not (tmp_path / ".cmdforge" / "project-echo" / "config.yaml").exists()
assert not (tmp_path / ".local" / "bin" / "project-echo").exists()
assert "cmdforge run project-echo" in capsys.readouterr().out
def test_create_output_dir_uses_explicit_tool_root(
self, temp_tools_dir, tmp_path
):
target = tmp_path / "portable-tools"
with patch("sys.argv", [
"cmdforge", "create", "portable", "--output-dir", str(target),
]):
assert main() == 0
assert (target / "portable" / "config.yaml").exists()
class TestDeleteCommand:
"""Tests for 'cmdforge delete' command."""
@@ -228,6 +485,91 @@ class TestRunCommand:
assert result != 0
def test_run_prefers_project_local_tool(
self, temp_tools_dir, tmp_path, monkeypatch, capsys
):
from cmdforge.tool import save_tool
save_tool(Tool(name="shadowed", output="global"))
project = tmp_path / "project"
project.mkdir()
save_tool(
Tool(name="shadowed", output="project"),
tools_dir=project / ".cmdforge",
)
monkeypatch.chdir(project)
with (
patch("sys.argv", ["cmdforge", "run", "shadowed"]),
patch("sys.stdin", StringIO("")),
patch("sys.stdin.isatty", return_value=True),
):
assert main() == 0
assert capsys.readouterr().out.strip() == "project"
def test_run_json_envelope_preserves_output_and_provenance(
self, temp_tools_dir, capsys
):
from cmdforge.tool import save_tool
save_tool(Tool(
name="audited",
steps=[PromptStep(
prompt="Echo {input}", provider="mock",
output_var="answer", plain_text=True,
)],
output="{answer}",
))
with (
patch("sys.argv", [
"cmdforge", "run", "audited", "--result-envelope", "json",
]),
patch("sys.stdin", StringIO("hello")),
patch("sys.stdin.isatty", return_value=False),
):
assert main() == 0
payload = json.loads(capsys.readouterr().out)
assert "MOCK" in payload["output"]
assert payload["execution"]["actual_providers"] == ["mock"]
assert payload["execution"]["provider_calls"][0]["locality"] == "local"
def test_failed_run_still_emits_audit_envelope(
self, temp_tools_dir, capsys
):
from cmdforge.providers import ProviderResult
from cmdforge.tool import save_tool
save_tool(Tool(
name="strict",
steps=[PromptStep(
prompt="Private", provider="primary",
output_var="answer", plain_text=True,
)],
output="{answer}",
))
failure = ProviderResult(
text="", success=False, error="offline",
requested_provider="primary", attempted_providers=["primary"],
)
with (
patch("sys.argv", [
"cmdforge", "run", "strict", "--no-fallback",
"--result-envelope", "json",
]),
patch("sys.stdin", StringIO("private")),
patch("sys.stdin.isatty", return_value=False),
patch("cmdforge.runner.call_provider", return_value=failure),
):
assert main() == 2
payload = json.loads(capsys.readouterr().out)
assert payload["output"] == ""
assert payload["execution"]["exit_code"] == 2
assert payload["execution"]["provider_calls"][0]["error"] == "offline"
class TestTestCommand:
"""Tests for 'cmdforge test' command."""
@@ -281,6 +623,29 @@ class TestProvidersCommand:
# Should show some default providers
assert 'mock' in captured.out.lower() or 'claude' in captured.out.lower()
def test_providers_list_json_exposes_routing_metadata(
self, temp_providers_file, capsys
):
from cmdforge.providers import Provider, save_providers
save_providers([Provider(
"safe", "safe-cli", locality="local",
capabilities=["structured-json"], data_policy="private",
max_context_tokens=8192,
)])
with (
patch("sys.argv", ["cmdforge", "providers", "list", "--json"]),
patch("cmdforge.cli.provider_commands._provider_status", return_value=(True, "Ready")),
):
assert main() == 0
payload = json.loads(capsys.readouterr().out)
assert payload[0]["locality"] == "local"
assert payload[0]["capabilities"] == ["structured-json"]
assert payload[0]["data_policy"] == "private"
assert payload[0]["max_context_tokens"] == 8192
assert payload[0]["available"] is True
def test_providers_add(self, temp_providers_file, capsys):
"""Add a custom provider."""
with patch('sys.argv', ['cmdforge', 'providers', 'add',
@@ -302,6 +667,13 @@ class TestProvidersCommand:
'--model', 'example/model',
'--api-key-env', 'CUSTOM_API_KEY',
'--tag', 'api',
'--locality', 'remote',
'--capability', 'structured-json',
'--model-digest', 'sha256:abc',
'--cost-class', 'low',
'--latency-class', 'fast',
'--max-context-tokens', '128000',
'--data-policy', 'internal',
'--fallback-chain', 'free',
]):
result = main()
@@ -314,6 +686,13 @@ class TestProvidersCommand:
assert provider.api_key_env == 'CUSTOM_API_KEY'
assert provider.tags == ['api']
assert provider.fallback_chain == PRESET_CHAINS['free']
assert provider.locality == 'remote'
assert provider.capabilities == ['structured-json']
assert provider.model_digest == 'sha256:abc'
assert provider.cost_class == 'low'
assert provider.latency_class == 'fast'
assert provider.max_context_tokens == 128000
assert provider.data_policy == 'internal'
def test_providers_list_reports_missing_api_key(self, temp_providers_file, capsys):
from cmdforge.providers import Provider, save_providers
@@ -441,3 +820,333 @@ class TestDocsCommand:
assert result == 1 # Returns 1 when no README
captured = capsys.readouterr()
assert 'No documentation' in captured.out or '--edit' in captured.out
class TestRegistryPublishDryRun:
@pytest.fixture
def tool_dir(self, tmp_path):
directory = tmp_path / "dry-run-tool"
directory.mkdir()
(directory / "config.yaml").write_text(
"name: dry-run-tool\n"
"version: 1.0.0\n"
"description: Test dry-run behavior\n"
"output: constant\n"
)
return directory
@staticmethod
def args(tool_dir):
return SimpleNamespace(
path=str(tool_dir), dry_run=True, force=False, owner=""
)
def test_without_token_stops_after_local_preflight(
self, tool_dir, monkeypatch, capsys
):
from cmdforge.cli.registry_commands import _cmd_registry_publish
get_client = MagicMock()
monkeypatch.setattr(
"cmdforge.cli.registry_commands.load_config",
lambda: SimpleNamespace(registry=SimpleNamespace(token="")),
)
monkeypatch.setattr("cmdforge.registry_client.get_client", get_client)
assert _cmd_registry_publish(self.args(tool_dir)) == 0
get_client.assert_not_called()
assert "local preflight only" in capsys.readouterr().out
def test_renders_structured_registry_evidence(
self, tool_dir, monkeypatch, capsys
):
from cmdforge.cli.registry_commands import _cmd_registry_publish
client = MagicMock()
client.publish_tool.return_value = {
"preflight": {
"errors": [],
"warnings": ["remote warning"],
"suggestions": ["remote hint"],
},
"suggestions": {
"similar_tools": [{
"name": "official/similar",
"similarity": 0.8,
}],
"scrutiny": {
"findings": [{"result": "warning", "message": "review me"}]
},
},
}
monkeypatch.setattr(
"cmdforge.cli.registry_commands.load_config",
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
)
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
assert _cmd_registry_publish(self.args(tool_dir)) == 0
output = capsys.readouterr().out
assert "remote warning" in output
assert "official/similar" in output
assert "review me" in output
client.publish_tool.assert_called_once()
def test_registry_rejection_returns_failure(
self, tool_dir, monkeypatch, capsys
):
from cmdforge.cli.registry_commands import _cmd_registry_publish
from cmdforge.registry_client import RegistryError
client = MagicMock()
client.publish_tool.side_effect = RegistryError(
"SCRUTINY_FAILED", "Tool rejected"
)
monkeypatch.setattr(
"cmdforge.cli.registry_commands.load_config",
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
)
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
assert _cmd_registry_publish(self.args(tool_dir)) == 1
assert "Tool rejected" in capsys.readouterr().err
def test_strips_local_registry_metadata_from_payload(
self, tool_dir, monkeypatch
):
import yaml
from cmdforge.cli.registry_commands import _cmd_registry_publish
config_path = tool_dir / "config.yaml"
config_path.write_text(
config_path.read_text()
+ "registry_hash: sha256:stale\n"
+ "registry_status: approved\n"
+ "registry_owner: previous-owner\n"
+ "registry_feedback: old feedback\n"
)
client = MagicMock()
client.publish_tool.return_value = {"preflight": {"errors": []}}
monkeypatch.setattr(
"cmdforge.cli.registry_commands.load_config",
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
)
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
assert _cmd_registry_publish(self.args(tool_dir)) == 0
payload = yaml.safe_load(client.publish_tool.call_args.args[0])
assert not {
"registry_hash",
"registry_status",
"registry_owner",
"registry_feedback",
}.intersection(payload)
def test_non_mapping_config_returns_failure(self, tool_dir, capsys):
from cmdforge.cli.registry_commands import _cmd_registry_publish
(tool_dir / "config.yaml").write_text("- not\n- a\n- tool\n")
assert _cmd_registry_publish(self.args(tool_dir)) == 1
assert "YAML mapping" in capsys.readouterr().err
def test_normal_publish_runs_remote_preflight_before_mutation(
self, tool_dir, monkeypatch
):
from cmdforge.cli.registry_commands import _cmd_registry_publish
client = MagicMock()
client.get_me.return_value = {"slug": "testuser"}
client.get_my_tool_status.return_value = {"status": "pending"}
client.publish_tool.side_effect = [
{
"preflight": {"errors": [], "warnings": [], "suggestions": []},
"quality": {"score": 80, "evidence_coverage": 60, "categories": []},
},
{
"owner": "testuser", "name": "dry-run-tool",
"version": "1.0.0", "status": "pending",
},
]
monkeypatch.setattr(
"cmdforge.cli.registry_commands.load_config",
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
)
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
args = SimpleNamespace(
path=str(tool_dir), dry_run=False, force=True, owner=""
)
assert _cmd_registry_publish(args) == 0
assert client.publish_tool.call_count == 2
assert client.publish_tool.call_args_list[0].kwargs["dry_run"] is True
assert "dry_run" not in client.publish_tool.call_args_list[1].kwargs
def test_inspect_with_registry_uses_similarity_results(monkeypatch, capsys):
from cmdforge.cli import cmd_inspect
from cmdforge.registry_client import PaginatedResponse
tool = Tool(name="summary", output="constant")
client = MagicMock()
client.search_tools.return_value = PaginatedResponse(data=[{
"owner": "official",
"name": "summarize",
"description": "Summarize text",
}])
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
assert cmd_inspect(SimpleNamespace(name="summary", registry=True)) == 0
assert "official/summarize" in capsys.readouterr().out
def test_inspect_shows_contract_proposal_and_passing_conformance(
monkeypatch, capsys
):
from cmdforge.cli import cmd_inspect
step_schema = {
"type": "object",
"properties": {"answer": {"type": "string"}},
"required": ["answer"],
}
tool = Tool(
name="contracted",
arguments=[],
input_schema={},
output_schema=step_schema,
steps=[PromptStep(
prompt="Answer", provider="paid-provider", output_var="answer",
output_schema=step_schema,
)],
output="{answer}",
)
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
assert cmd_inspect(SimpleNamespace(name="contracted", registry=False)) == 0
output = capsys.readouterr().out
assert "Contract conformance" in output
assert "PASSED" in output
def test_inspect_can_save_passing_baseline(tmp_path, monkeypatch, capsys):
from cmdforge.cli import cmd_inspect
from cmdforge.contract_testing import CONFORMANCE_FILE
tool = Tool(
name="baseline", version="1.0.0", input_schema={},
output_schema={"type": "string"}, output="stable",
path=tmp_path / "config.yaml",
)
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
args = SimpleNamespace(
name="baseline", registry=False, save_baseline=True
)
assert cmd_inspect(args) == 0
assert (tmp_path / CONFORMANCE_FILE).exists()
assert "Saved conformance baseline" in capsys.readouterr().out
def test_switch_to_existing_tool_closes_creation_page_first():
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
from cmdforge.gui.pages.tool_builder_page import _switch_to_existing_tool
main_window = MagicMock()
_switch_to_existing_tool(main_window, "existing")
assert main_window.method_calls == [
call.close_tool_builder(),
call.open_tool_builder("existing"),
]
def test_tool_builder_save_scope_prefers_recognized_projects(tmp_path):
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
from cmdforge.gui.pages.tool_builder_page import default_save_scope
assert default_save_scope(tmp_path) == "global"
(tmp_path / "pyproject.toml").write_text("[project]\nname='demo'\n")
assert default_save_scope(tmp_path) == "project"
def test_tool_builder_existing_tool_root_does_not_nest_tool_name(tmp_path):
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
from cmdforge.gui.pages.tool_builder_page import existing_tool_root
config = tmp_path / ".cmdforge" / "demo" / "config.yaml"
assert existing_tool_root(config) == (tmp_path / ".cmdforge").resolve()
def test_guided_extraction_builds_new_tool_without_mutating_draft():
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
from cmdforge.gui.pages.tool_builder_page import (
build_extracted_tool,
extraction_diff,
)
schema = {"type": "string"}
repeated = [
PromptStep(
prompt="First {input}", provider="mock", output_var="first",
output_schema=schema,
),
PromptStep(
prompt="Second {first}", provider="mock", output_var="second",
output_schema=schema,
),
]
draft = Tool(
name="draft", steps=repeated + [
PromptStep(
prompt="First {input}", provider="mock", output_var="first_copy",
output_schema=schema,
),
PromptStep(
prompt="Second {first}", provider="mock", output_var="second_copy",
output_schema=schema,
),
], input_schema=schema,
)
before = draft.to_dict()
extracted = build_extracted_tool(
draft,
{"type": "repeated_sequence", "locations": [1, 3], "length": 2},
"draft-shared",
)
assert len(extracted.steps) == 2
assert extracted.output == "{second}"
assert extracted.output_schema == schema
assert draft.to_dict() == before
assert "+++ draft-shared/config.yaml" in extraction_diff(extracted)
def test_guided_extraction_rejects_hidden_external_step_dependency():
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
from cmdforge.gui.pages.tool_builder_page import build_extracted_tool
schema = {"type": "string"}
draft = Tool(name="draft", steps=[
PromptStep(
prompt="Produce", provider="mock", output_var="earlier",
output_schema=schema,
),
PromptStep(
prompt="Use {earlier}", provider="mock", output_var="one",
output_schema=schema,
),
PromptStep(
prompt="Finish {one}", provider="mock", output_var="two",
output_schema=schema,
),
])
with pytest.raises(ValueError, match="outside its boundary"):
build_extracted_tool(
draft,
{"type": "repeated_sequence", "locations": [2, 4], "length": 2},
"draft-shared",
)
+328 -4
View File
@@ -37,7 +37,10 @@ def app():
app = create_app()
app.config["TESTING"] = True
yield app
# Keep the environment override active for the entire test. Registry
# request handlers open fresh connections rather than reusing the one
# created during app initialization.
yield app
# Cleanup
Path(db_path).unlink(missing_ok=True)
@@ -53,7 +56,7 @@ def client(app):
def auth_headers(app):
"""Create auth headers with a valid token."""
import hashlib
from datetime import datetime
from datetime import datetime, timezone
from cmdforge.registry.db import connect_db
token = "test-token"
@@ -91,7 +94,7 @@ def auth_headers(app):
INSERT INTO api_tokens (publisher_id, token_hash, name, created_at)
VALUES (?, ?, ?, ?)
""",
[publisher_id, token_hash, "test-token", datetime.utcnow().isoformat()],
[publisher_id, token_hash, "test-token", datetime.now(timezone.utc).isoformat()],
)
# Insert a public approved tool for tests if missing
@@ -109,7 +112,7 @@ def auth_headers(app):
[
"testuser", "tool1", "1.0.0", "Test tool", "Other", "[]",
"name: tool1\nversion: 1.0.0\n", "", publisher_id,
"public", "approved", datetime.utcnow().isoformat(),
"public", "approved", datetime.now(timezone.utc).isoformat(),
],
)
@@ -155,6 +158,327 @@ class TestToolApprovedEndpoint:
assert data['data']['has_approved_public_version'] is False
@flask_required
class TestDeprecationChains:
def test_replacement_chain_is_validated_and_returned(self, client, auth_headers):
target = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": "name: replacement\nversion: 1.0.0\noutput: ok\n"},
)
assert target.status_code == 201
response = client.post(
"/api/v1/tools/testuser/tool1/deprecate",
headers=auth_headers,
json={"deprecated_message": "Moved", "replacement": "replacement"},
)
assert response.status_code == 200
assert response.get_json()["data"]["replacement_chain"] == [
"testuser/replacement"
]
def test_replacement_cycle_is_rejected(self, client, auth_headers):
target = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": "name: cycle-target\nversion: 1.0.0\noutput: ok\n"},
)
assert target.status_code == 201
assert client.post(
"/api/v1/tools/testuser/cycle-target/deprecate",
headers=auth_headers,
json={"replacement": "tool1"},
).status_code == 200
response = client.post(
"/api/v1/tools/testuser/tool1/deprecate",
headers=auth_headers,
json={"replacement": "cycle-target"},
)
assert response.status_code == 400
assert "cycle" in response.get_json()["error"]["message"]
@flask_required
class TestPublishPreflightEndpoint:
def test_dry_run_includes_shared_preflight(self, client, auth_headers):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"dry_run": True,
"config": (
"name: preflight-tool\n"
"version: 1.0.0\n"
"description: Constant output\n"
"output: constant\n"
"input_schema:\n type: string\n"
"output_schema:\n type: string\n"
),
},
)
assert response.status_code == 200
report = response.get_json()["data"]["preflight"]
assert report["errors"] == []
assert report["generated_tests"][0]["state"] == "passed"
assert response.get_json()["data"]["quality"]["evidence_coverage"] > 0
def test_real_publish_persists_immutable_audit(self, client, auth_headers):
from cmdforge.registry.db import connect_db
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"config": (
"name: audited-tool\nversion: 1.0.0\noutput: stable\n"
"input_schema: {}\noutput_schema:\n type: string\n"
),
},
)
assert response.status_code == 201
conn = connect_db()
try:
rows = conn.execute(
"""
SELECT tool_audits.trigger, tool_audits.evidence_json
FROM tool_audits JOIN tools ON tools.id = tool_audits.tool_id
WHERE tools.owner = 'testuser' AND tools.name = 'audited-tool'
"""
).fetchall()
conn.execute(
"""
UPDATE tools SET moderation_status = 'approved'
WHERE owner = 'testuser' AND name = 'audited-tool'
"""
)
conn.commit()
finally:
conn.close()
assert len(rows) == 1
assert rows[0]["trigger"] == "publish"
assert "contract_conformance" in rows[0]["evidence_json"]
search = client.get("/api/v1/tools/search?q=audited")
result = search.get_json()["data"][0]
assert isinstance(result["quality_score"], int)
assert isinstance(result["quality_coverage"], int)
assert result["quality_evaluated_at"]
def test_real_publish_blocks_deterministic_contract_failure(
self, client, auth_headers
):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"config": (
"name: broken-contract\nversion: 1.0.0\noutput: text\n"
"input_schema: {}\noutput_schema:\n type: integer\n"
),
},
)
assert response.status_code == 400
assert response.get_json()["error"]["code"] == "PREFLIGHT_FAILED"
def test_real_publish_blocks_missing_registry_dependency(
self, client, auth_headers
):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"config": (
"name: missing-dependency\nversion: 1.0.0\n"
"dependencies:\n - absent-tool\n"
),
},
)
assert response.status_code == 400
assert "absent-tool" in response.get_json()["error"]["message"]
def test_invalid_contract_is_rejected(self, client, auth_headers):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"dry_run": True,
"config": (
"name: invalid-contract\n"
"version: 1.0.0\n"
"output_schema:\n type: nonsense\n"
),
},
)
assert response.status_code == 400
assert response.get_json()["error"]["code"] == "INVALID_CONFIG"
def test_non_mapping_config_is_rejected(self, client, auth_headers):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={"dry_run": True, "config": "- not\n- a\n- tool\n"},
)
assert response.status_code == 400
assert response.get_json()["error"]["code"] == "VALIDATION_ERROR"
def test_legacy_source_string_remains_publishable(self, client, auth_headers):
response = client.post(
"/api/v1/tools",
headers=auth_headers,
json={
"dry_run": True,
"config": (
"name: legacy-source\n"
"version: 1.0.0\n"
"source: old/tool\n"
"output: constant\n"
),
},
)
assert response.status_code == 200
@flask_required
class TestM9RegistryTrustAndCommunity:
def test_signed_publish_download_and_content_lookup(
self, client, auth_headers
):
from cmdforge.attestation import (
Attestation, generate_keypair, sign_tool, verify_attestation,
)
from cmdforge.registry.db import connect_db
private_key, public_key = generate_keypair()
response = client.put(
"/api/v1/me/signing-key", headers=auth_headers,
json={"public_key": public_key},
)
assert response.status_code == 200
config = "name: signed-tool\nversion: 1.0.0\noutput: stable\n"
preflight = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": config, "dry_run": True},
)
assert preflight.status_code == 200
content_hash = preflight.get_json()["data"]["content_hash"]
attestation = sign_tool(
"signed-tool", "1.0.0", content_hash, "testuser", private_key
)
published = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": config, "attestation": attestation.to_dict()},
)
assert published.status_code == 201
assert published.get_json()["data"]["content_hash"] == content_hash
conn = connect_db()
try:
conn.execute(
"UPDATE tools SET moderation_status = 'approved' "
"WHERE owner = 'testuser' AND name = 'signed-tool'"
)
conn.commit()
finally:
conn.close()
downloaded = client.get(
"/api/v1/tools/testuser/signed-tool/download?install=false"
)
assert downloaded.status_code == 200
data = downloaded.get_json()["data"]
assert data["content_hash"] == content_hash
assert verify_attestation(Attestation.from_dict(data["attestation"]), public_key)
lookup = client.get(f"/api/v1/tools/by-content-hash/{content_hash}")
assert lookup.status_code == 200
assert lookup.get_json()["data"]["name"] == "signed-tool"
def test_registered_key_requires_valid_signature(self, client, auth_headers):
from cmdforge.attestation import generate_keypair
_, public_key = generate_keypair()
assert client.put(
"/api/v1/me/signing-key", headers=auth_headers,
json={"public_key": public_key},
).status_code == 200
response = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": "name: unsigned-tool\nversion: 1.0.0\n"},
)
assert response.status_code == 400
assert response.get_json()["error"]["code"] == "ATTESTATION_REQUIRED"
def test_improvement_is_tested_reviewed_and_credited(
self, client, auth_headers
):
from cmdforge.registry.db import connect_db
config = (
"name: improvable\nversion: 1.0.0\noutput: '{result}'\n"
"input_schema:\n type: string\noutput_schema:\n type: string\n"
"steps:\n - type: prompt\n prompt: 'Summarize: {input}'\n"
" provider: mock\n output_var: result\n"
)
published = client.post(
"/api/v1/tools", headers=auth_headers, json={"config": config}
)
assert published.status_code == 201
conn = connect_db()
try:
conn.execute(
"UPDATE tools SET moderation_status = 'approved' "
"WHERE owner = 'testuser' AND name = 'improvable'"
)
conn.commit()
finally:
conn.close()
submitted = client.post(
"/api/v1/tools/testuser/improvable/1.0.0/improvements",
headers=auth_headers,
json={
"step_index": 0,
"proposed": "Summarize the input accurately and concisely: {input}",
"rationale": "Clearer expected behavior",
},
)
assert submitted.status_code == 201
submission_id = submitted.get_json()["data"]["id"]
reviewed = client.patch(
f"/api/v1/improvements/{submission_id}", headers=auth_headers,
json={"decision": "approve", "notes": "Validated"},
)
assert reviewed.status_code == 200
assert reviewed.get_json()["data"]["ready_to_apply"] is True
improved_config = config.replace("version: 1.0.0", "version: 1.0.1").replace(
"Summarize: {input}",
"Summarize the input accurately and concisely: {input}",
)
applied = client.post(
"/api/v1/tools", headers=auth_headers,
json={"config": improved_config, "improvement_id": submission_id},
)
assert applied.status_code == 201
conn = connect_db()
try:
conn.execute(
"UPDATE tools SET moderation_status = 'approved' "
"WHERE owner = 'testuser' AND name = 'improvable' "
"AND version = '1.0.1'"
)
conn.commit()
credited = conn.execute(
"SELECT COUNT(*) AS count FROM tool_contributors"
).fetchone()["count"]
finally:
conn.close()
assert credited == 1
detail = client.get("/api/v1/tools/testuser/improvable?version=1.0.1")
assert detail.status_code == 200
assert detail.get_json()["data"]["badges"] == [
"optimized", "community-reviewed"
]
@flask_required
class TestPostCollectionsEndpoint:
"""Tests for POST /api/v1/collections endpoint."""
+80
View File
@@ -0,0 +1,80 @@
"""Tests for M9.3 community improvement workflow."""
from cmdforge.community import (
ImprovementSubmission,
create_submission,
run_submission_tests,
review_submission,
)
from cmdforge.tool import Tool, PromptStep, CodeStep
class TestCreateSubmission:
def test_create_for_prompt_step(self):
tool = Tool(
name="test",
version="1.0.0",
steps=[PromptStep(prompt="Summarize", provider="mock", output_var="x")],
output="{x}",
)
sub = create_submission(tool, 0, "Summarize the text thoroughly", "alice", "More detail")
assert sub.tool_name == "test"
assert sub.step_type == "prompt"
assert sub.original == "Summarize"
assert sub.proposed == "Summarize the text thoroughly"
assert sub.submitter == "alice"
assert sub.status == "pending"
def test_create_for_code_step(self):
tool = Tool(
name="test",
steps=[CodeStep(code="x = 1", output_var="x")],
output="{x}",
)
sub = create_submission(tool, 0, "x = 42", "bob")
assert sub.step_type == "code"
assert sub.original == "x = 1"
def test_invalid_step_index(self):
tool = Tool(name="test")
try:
create_submission(tool, 5, "new", "alice")
assert False, "Should have raised"
except ValueError:
pass
class TestReviewSubmission:
def test_review_approved(self):
sub = ImprovementSubmission(
tool_name="test", tool_version="1.0.0", submitter="alice",
step_index=0, step_type="prompt", original="a", proposed="b",
status="tested", test_result={"passed_for_review": True},
)
review = review_submission(sub, "approve", "admin", "Good improvement")
assert review.decision == "approve"
assert sub.status == "approved"
def test_review_pending_rejected(self):
sub = ImprovementSubmission(
tool_name="test", tool_version="1.0.0", submitter="alice",
step_index=0, step_type="prompt", original="a", proposed="b",
status="pending",
)
try:
review_submission(sub, "approve", "admin")
assert False, "Should have raised"
except ValueError:
pass
def test_invalid_decision(self):
sub = ImprovementSubmission(
tool_name="test", tool_version="1.0.0", submitter="alice",
step_index=0, step_type="prompt", original="a", proposed="b",
status="tested",
)
try:
review_submission(sub, "maybe", "admin")
assert False, "Should have raised"
except ValueError:
pass
+368
View File
@@ -0,0 +1,368 @@
"""Tests for deterministic contract conformance (M8.V1)."""
from pathlib import Path
import pytest
from jsonschema import validate
from cmdforge.contract_testing import (
ConformanceResult,
ConformanceReport,
UnsupportedContract,
_generate_inputs,
_fallback_value,
run_contract_tests,
)
from cmdforge.tool import Tool
class TestInputGeneration:
def test_enum_generates_variants(self):
schema = {
"type": "object",
"properties": {
"mode": {"type": "string", "enum": ["fast", "accurate", "balanced"]},
"limit": {"type": "integer", "default": 10},
},
"required": ["mode"],
}
cases = _generate_inputs(schema)
assert len(cases) >= 1
assert cases[0]["mode"] in ("fast", "accurate", "balanced")
def test_fallback_string(self):
cases = _generate_inputs({
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
})
assert "test-name" in cases[0]["name"]
def test_fallback_integer(self):
cases = _generate_inputs({
"type": "object",
"properties": {"count": {"type": "integer"}},
"required": ["count"],
})
assert cases[0]["count"] == 0
def test_default_used(self):
cases = _generate_inputs({
"type": "object",
"properties": {"name": {"type": "string", "default": "World"}},
"required": ["name"],
})
assert cases[0]["name"] == "World"
def test_scalar_schema_generates_scalar(self):
cases = _generate_inputs({"type": "string"})
assert cases == ["test-input"]
def test_generated_value_respects_numeric_bounds(self):
schema = {
"type": "object",
"properties": {
"count": {"type": "integer", "minimum": 1, "maximum": 3}
},
"required": ["count"],
}
case = _generate_inputs(schema)[0]
validate(case, schema)
assert 1 <= case["count"] <= 3
def test_array_and_nested_object_are_supported(self):
schema = {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
},
}
value = _generate_inputs(schema)[0]
validate(value, schema)
def test_unsynthesizable_pattern_is_unsupported(self):
with pytest.raises(UnsupportedContract):
_generate_inputs({"type": "string", "pattern": "^z{20}$"})
def test_invalid_default_is_unsupported_not_returned(self):
with pytest.raises(UnsupportedContract, match="valid input"):
_generate_inputs({"type": "integer", "default": "wrong"})
class TestFallbackValue:
def test_explicit_default(self):
assert _fallback_value({"default": "hi"}, "x", []) == "hi"
def test_examples(self):
assert _fallback_value({"examples": ["a", "b"]}, "x", []) == "a"
def test_enum(self):
assert _fallback_value({"enum": ["yes", "no"]}, "x", []) == "yes"
def test_type_fallback(self):
assert _fallback_value({"type": "boolean"}, "flag", []) is True
class TestConformanceReport:
def test_empty_is_not_run_not_pass(self):
report = ConformanceReport()
assert report.outcome == "not_run"
assert not report.passed
def test_invalid_state_is_rejected(self):
with pytest.raises(ValueError, match="Unknown conformance state"):
ConformanceResult("case", "unknown")
def test_summary_counts(self):
report = ConformanceReport(results=[
ConformanceResult("a", "passed"),
ConformanceResult("b", "passed"),
ConformanceResult("c", "failed", "error"),
])
assert report.summary == {"passed": 2, "failed": 1, "unsupported": 0, "not_run": 0}
assert not report.passed
class TestRunContractTests:
def test_missing_schemas_returns_not_run(self, tmp_path):
from unittest.mock import patch
from cmdforge.tool import Tool
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(name="bare")
report = run_contract_tests(tool)
assert report.results[0].state == "not_run"
def test_simple_tool_passes(self, tmp_path):
from unittest.mock import patch
from cmdforge.tool import Tool, PromptStep
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(
name="greet",
input_schema={
"type": "object",
"properties": {
"name": {"type": "string", "default": "World"},
},
},
output_schema={
"type": "object",
"properties": {"result": {"type": "string"}},
"required": ["result"],
},
steps=[
PromptStep(
prompt="Greet {name}",
provider="real-provider-that-must-not-run",
output_var="response",
output_schema={
"type": "object",
"properties": {"result": {"type": "string"}},
"required": ["result"],
},
),
],
output="{response}",
)
report = run_contract_tests(tool)
assert report.outcome == "passed"
assert report.results[0].state == "passed"
def test_empty_schemas_are_explicit_and_runnable(self):
from cmdforge.tool import Tool
report = run_contract_tests(
Tool(name="anything", input_schema={}, output_schema={}, output="ok")
)
assert report.outcome == "passed"
@pytest.mark.parametrize("step_type", ["code", "tool", "mcp"])
def test_side_effecting_steps_are_unsupported(self, step_type):
from cmdforge.tool import CodeStep, McpStep, Tool, ToolStep
steps = {
"code": CodeStep(code="raise AssertionError('must not run')", output_var="x"),
"tool": ToolStep(tool="external", output_var="x"),
"mcp": McpStep(server="remote", tool="external", output_var="x"),
}
tool = Tool(
name="unsafe",
input_schema={},
output_schema={},
steps=[steps[step_type]],
output="{x}",
)
report = run_contract_tests(tool)
assert report.outcome == "unsupported"
def test_schema_mismatch_is_a_failure(self):
from cmdforge.tool import Tool
report = run_contract_tests(
Tool(
name="mismatch",
input_schema={},
output_schema={"type": "integer"},
output="not-an-integer",
)
)
assert report.outcome == "failed"
class TestRegressionComparison:
@staticmethod
def contracted_tool(version="1.0.0", output_schema=None):
from cmdforge.tool import Tool
return Tool(
name="test",
version=version,
input_schema={},
output_schema=output_schema or {"type": "string"},
output="stable",
)
def passing_report(self, tool=None):
return run_contract_tests(tool or self.contracted_tool())
def test_save_and_load_baseline(self, tmp_path):
from cmdforge.contract_testing import (
save_conformance_baseline, load_conformance_baseline,
)
import stat
tool = self.contracted_tool()
report = self.passing_report(tool)
path = save_conformance_baseline(tool, report, tmp_path)
assert path.exists()
assert stat.S_IMODE(path.stat().st_mode) == 0o600
loaded = load_conformance_baseline(tmp_path)
assert loaded["format_version"] == 1
assert loaded["tool"]["version"] == "1.0.0"
assert loaded["contract_hash"]
assert len(loaded["results"]) == 1
assert loaded["results"][0]["case_id"].startswith("case-")
assert loaded["results"][0]["output_shape"] == {"type": "string"}
def test_missing_baseline(self, tmp_path):
from cmdforge.contract_testing import load_conformance_baseline
assert load_conformance_baseline(tmp_path) is None
def test_corrupt_baseline_is_rejected(self, tmp_path):
from cmdforge.contract_testing import (
BaselineFormatError, CONFORMANCE_FILE, load_conformance_baseline,
)
(tmp_path / CONFORMANCE_FILE).write_text("[]")
with pytest.raises(BaselineFormatError, match="root"):
load_conformance_baseline(tmp_path)
def test_tampered_baseline_evidence_is_rejected(self, tmp_path):
from cmdforge.contract_testing import (
BaselineFormatError, CONFORMANCE_FILE, load_conformance_baseline,
save_conformance_baseline,
)
import json
tool = self.contracted_tool()
save_conformance_baseline(tool, self.passing_report(tool), tmp_path)
path = tmp_path / CONFORMANCE_FILE
data = json.loads(path.read_text())
data["results"][0]["input_value"] = {"tampered": True}
path.write_text(json.dumps(data))
with pytest.raises(BaselineFormatError, match="fingerprint mismatch"):
load_conformance_baseline(tmp_path)
def test_failed_report_cannot_replace_baseline(self, tmp_path):
from cmdforge.contract_testing import save_conformance_baseline
report = run_contract_tests(Tool(
name="failed", input_schema={}, output_schema={"type": "integer"},
output="text",
))
with pytest.raises(ValueError, match="passing"):
save_conformance_baseline(self.contracted_tool(), report, tmp_path)
def baseline_and_report(self, tmp_path):
from cmdforge.contract_testing import (
load_conformance_baseline, save_conformance_baseline,
)
tool = self.contracted_tool()
report = self.passing_report(tool)
save_conformance_baseline(tool, report, tmp_path)
return load_conformance_baseline(tmp_path), report
def test_detects_regression(self):
from dataclasses import replace
from cmdforge.contract_testing import (
ConformanceReport, compare_conformance,
)
import tempfile
with tempfile.TemporaryDirectory() as directory:
baseline, report = self.baseline_and_report(Path(directory))
current = ConformanceReport([
replace(report.results[0], state="failed", detail="now broken")
])
delta = compare_conformance(
baseline, current, self.contracted_tool(version="2.0.0")
)
assert len(delta.regressed) == 1
assert delta.has_regressions
assert delta.current_version == "2.0.0"
def test_lost_coverage_is_regression(self, tmp_path):
from dataclasses import replace
from cmdforge.contract_testing import (
ConformanceReport, compare_conformance,
)
baseline, report = self.baseline_and_report(tmp_path)
current = ConformanceReport([
replace(report.results[0], state="unsupported", detail="lost")
])
delta = compare_conformance(baseline, current, self.contracted_tool())
assert len(delta.coverage_losses) == 1
assert delta.has_regressions
def test_removed_case_is_regression(self, tmp_path):
from cmdforge.contract_testing import (
ConformanceReport, compare_conformance,
)
baseline, _ = self.baseline_and_report(tmp_path)
delta = compare_conformance(
baseline, ConformanceReport(), self.contracted_tool()
)
assert len(delta.removed_cases) == 1
assert delta.has_regressions
def test_output_shape_or_value_change_is_flagged(self, tmp_path):
from dataclasses import replace
from cmdforge.contract_testing import (
ConformanceReport, compare_conformance,
)
baseline, report = self.baseline_and_report(tmp_path)
current = ConformanceReport([
replace(
report.results[0],
output_shape={"type": "object", "properties": {}},
output_fingerprint="changed",
)
])
delta = compare_conformance(baseline, current, self.contracted_tool())
assert len(delta.output_changes) == 1
assert delta.has_regressions
def test_contract_change_is_recorded(self, tmp_path):
from cmdforge.contract_testing import compare_conformance
baseline, report = self.baseline_and_report(tmp_path)
changed = self.contracted_tool(output_schema={"type": ["string", "null"]})
delta = compare_conformance(baseline, report, changed)
assert delta.contract_changed
+173
View File
@@ -0,0 +1,173 @@
"""Tests for transactional email and password-reset delivery safety."""
from __future__ import annotations
import hashlib
import logging
from unittest.mock import patch
import requests
from flask import Flask
from cmdforge.registry.app import create_app, password_hasher
from cmdforge.registry.db import connect_db
from cmdforge.web.email import send_email
def _email_app(**config) -> Flask:
app = Flask(__name__)
defaults = {
"CMDFORGE_ENV": "development",
"MAIL_TRANSPORT": "disabled",
"MAIL_FROM": "CmdForge <[email protected]>",
"MAIL_TIMEOUT": 10,
"RESEND_API_KEY": "",
}
defaults.update(config)
app.config.update(defaults)
return app
def test_disabled_delivery_does_not_log_message_or_recipient(caplog):
app = _email_app()
with app.app_context(), caplog.at_level(logging.INFO):
assert not send_email(
"[email protected]", "Reset subject", "secret reset body", "secret text"
)
assert "[email protected]" not in caplog.text
assert "secret reset body" not in caplog.text
assert "secret text" not in caplog.text
def test_console_transport_is_rejected_in_production(caplog):
app = _email_app(CMDFORGE_ENV="production", MAIL_TRANSPORT="console")
with app.app_context(), caplog.at_level(logging.INFO):
assert not send_email("[email protected]", "subject", "body")
assert "[email protected]" not in caplog.text
assert "body" not in caplog.text
def test_resend_transport_sends_expected_request(monkeypatch):
app = _email_app(MAIL_TRANSPORT="resend", RESEND_API_KEY="test-api-key")
captured = {}
class Response:
def raise_for_status(self):
return None
def fake_post(url, **kwargs):
captured["url"] = url
captured.update(kwargs)
return Response()
monkeypatch.setattr(requests, "post", fake_post)
with app.app_context():
assert send_email("[email protected]", "subject", "<p>body</p>", "body")
assert captured["url"] == "https://api.resend.com/emails"
assert captured["headers"]["Authorization"] == "Bearer test-api-key"
assert captured["json"]["to"] == ["[email protected]"]
assert captured["timeout"] == 10.0
def test_resend_failure_does_not_log_sensitive_data(monkeypatch, caplog):
app = _email_app(MAIL_TRANSPORT="resend", RESEND_API_KEY="test-api-key")
def fail_post(*args, **kwargs):
raise requests.ConnectionError("provider unavailable")
monkeypatch.setattr(requests, "post", fail_post)
with app.app_context(), caplog.at_level(logging.INFO):
assert not send_email("[email protected]", "subject", "reset-secret")
assert "[email protected]" not in caplog.text
assert "reset-secret" not in caplog.text
assert "test-api-key" not in caplog.text
def _registry_client(tmp_path, monkeypatch):
db_path = tmp_path / "registry.db"
monkeypatch.setenv("CMDFORGE_REGISTRY_DB", str(db_path))
monkeypatch.setenv("CMDFORGE_ENV", "production")
monkeypatch.setenv("CMDFORGE_PUBLIC_URL", "https://cmdforge.example")
app = create_app()
app.config["TESTING"] = True
with connect_db() as db:
db.execute(
"""
INSERT INTO publishers (email, password_hash, slug, display_name)
VALUES (?, ?, ?, ?)
""",
["[email protected]", password_hasher.hash("initial-password"), "owner", "Owner"],
)
db.commit()
return app.test_client(), db_path
def test_reset_uses_canonical_url_and_leaves_delivered_token_valid(tmp_path, monkeypatch):
client, db_path = _registry_client(tmp_path, monkeypatch)
captured = {}
def delivered(to, token, base_url):
captured.update(to=to, token=token, base_url=base_url)
return True
with patch("cmdforge.web.email.send_password_reset_email", delivered):
response = client.post(
"/api/v1/password-reset/request",
json={"email": "[email protected]"},
headers={"Host": "attacker.example"},
)
assert response.status_code == 200
assert captured["base_url"] == "https://cmdforge.example"
with connect_db() as db:
row = db.execute(
"SELECT token_hash, used_at FROM password_reset_tokens"
).fetchone()
assert row["token_hash"] == hashlib.sha256(captured["token"].encode()).hexdigest()
assert row["used_at"] is None
def test_reset_invalidates_token_when_delivery_fails(tmp_path, monkeypatch):
client, db_path = _registry_client(tmp_path, monkeypatch)
with patch("cmdforge.web.email.send_password_reset_email", return_value=False):
response = client.post(
"/api/v1/password-reset/request", json={"email": "[email protected]"}
)
assert response.status_code == 200
with connect_db() as db:
row = db.execute("SELECT used_at FROM password_reset_tokens").fetchone()
assert row["used_at"] is not None
def test_reset_invalidates_token_without_safe_production_base_url(tmp_path, monkeypatch):
client, db_path = _registry_client(tmp_path, monkeypatch)
client.application.config["PUBLIC_BASE_URL"] = "http://insecure.example"
with patch("cmdforge.web.email.send_password_reset_email") as sender:
response = client.post(
"/api/v1/password-reset/request", json={"email": "[email protected]"}
)
assert response.status_code == 200
sender.assert_not_called()
with connect_db() as db:
row = db.execute("SELECT used_at FROM password_reset_tokens").fetchone()
assert row["used_at"] is not None
def test_unknown_email_creates_no_token_and_sends_nothing(tmp_path, monkeypatch):
client, db_path = _registry_client(tmp_path, monkeypatch)
with patch("cmdforge.web.email.send_password_reset_email") as sender:
response = client.post(
"/api/v1/password-reset/request", json={"email": "[email protected]"}
)
assert response.status_code == 200
sender.assert_not_called()
with connect_db() as db:
count = db.execute("SELECT COUNT(*) FROM password_reset_tokens").fetchone()[0]
assert count == 0
+109
View File
@@ -0,0 +1,109 @@
"""Tests for scrutiny-driven improvement pipeline (M9.2)."""
from cmdforge.improvement import (
ImprovementSuggestion,
ImprovementReport,
generate_improvements,
)
from cmdforge.tool import Tool, PromptStep, CodeStep
class TestImprovementReport:
def test_empty_report(self):
report = ImprovementReport(tool_name="test")
assert not report.has_high_severity
assert report.by_category == {}
def test_has_high_severity(self):
report = ImprovementReport(
tool_name="test",
suggestions=[
ImprovementSuggestion("honesty", "high", "Bad", "desc"),
ImprovementSuggestion("efficiency", "low", "Minor", "desc"),
],
)
assert report.has_high_severity
def test_by_category(self):
report = ImprovementReport(
tool_name="test",
suggestions=[
ImprovementSuggestion("honesty", "high", "A", "d"),
ImprovementSuggestion("honesty", "low", "B", "d"),
ImprovementSuggestion("efficiency", "medium", "C", "d"),
],
)
assert len(report.by_category["honesty"]) == 2
assert len(report.by_category["efficiency"]) == 1
class TestGenerateImprovements:
def test_missing_description(self):
tool = Tool(name="nodesc")
report = generate_improvements(tool)
honesty = [s for s in report.suggestions if s.category == "honesty"]
assert any("Missing description" in s.title for s in honesty)
def test_short_description(self):
tool = Tool(name="short", description="Does stuff")
report = generate_improvements(tool)
honesty = [s for s in report.suggestions if s.category == "honesty"]
assert any("short" in s.title.lower() for s in honesty)
def test_echo_prompt_detected(self):
tool = Tool(
name="echo",
steps=[PromptStep(prompt="{input}", provider="mock", output_var="out")],
output="{out}",
)
report = generate_improvements(tool)
efficiency = [s for s in report.suggestions if s.category == "efficiency"]
assert any("echo" in s.title.lower() for s in efficiency)
def test_multiple_same_provider_prompts(self):
tool = Tool(
name="multi",
steps=[
PromptStep(prompt="Step 1: {input}", provider="claude", output_var="a"),
PromptStep(prompt="Step 2: {a}", provider="claude", output_var="b"),
],
output="{b}",
)
report = generate_improvements(tool)
efficiency = [s for s in report.suggestions if s.category == "efficiency"]
assert any("same provider" in s.title.lower() for s in efficiency)
def test_exec_detected(self):
tool = Tool(
name="dangerous",
steps=[CodeStep(code="exec('print(1)')", output_var="x")],
output="{x}",
)
report = generate_improvements(tool)
transparency = [s for s in report.suggestions if s.category == "transparency"]
assert any("exec" in s.title.lower() for s in transparency)
def test_short_prompt_detected(self):
tool = Tool(
name="vague",
steps=[PromptStep(prompt="Do it", provider="mock", output_var="x")],
output="{x}",
)
report = generate_improvements(tool)
transparency = [s for s in report.suggestions if s.category == "transparency"]
assert any("short" in s.title.lower() for s in transparency)
def test_clean_tool_no_suggestions(self):
tool = Tool(
name="clean",
description="A well-documented tool that summarizes text input concisely",
steps=[PromptStep(
prompt="Summarize the following text in 2-3 sentences. Focus on key points.\n\nText: {input}",
provider="mock",
output_var="summary",
)],
output="{summary}",
)
report = generate_improvements(tool)
high = [s for s in report.suggestions if s.severity == "high"]
assert len(high) == 0
+121
View File
@@ -0,0 +1,121 @@
"""Tests for M9.4 transitive integrity verification."""
from unittest.mock import patch
from cmdforge.integrity import (
IntegrityNode,
IntegrityChain,
compute_tool_hash,
build_integrity_chain,
verify_integrity,
)
from cmdforge.tool import Tool, PromptStep, ToolStep
class TestComputeToolHash:
def test_same_tool_same_hash(self):
tool = Tool(name="test", version="1.0.0")
assert compute_tool_hash(tool) == compute_tool_hash(tool)
def test_different_tools_different_hash(self):
a = Tool(name="a", version="1.0.0")
b = Tool(name="b", version="1.0.0")
assert compute_tool_hash(a) != compute_tool_hash(b)
def test_version_change_changes_hash(self):
v1 = Tool(name="test", version="1.0.0")
v2 = Tool(name="test", version="2.0.0")
assert compute_tool_hash(v1) != compute_tool_hash(v2)
class TestIntegrityChain:
def test_single_tool_chain(self):
tool = Tool(name="solo", version="1.0.0")
chain = build_integrity_chain(tool)
assert chain.root.name == "solo"
assert chain.is_valid
def test_chain_with_unresolved_dep(self):
tool = Tool(
name="parent",
version="1.0.0",
steps=[ToolStep(tool="missing-dep", output_var="x")],
output="{x}",
)
chain = build_integrity_chain(tool)
assert "missing-dep" not in chain.nodes
assert not chain.is_valid
assert chain.errors == ["Unresolved dependency: missing-dep"]
def test_chain_valid_with_resolved_dep(self, tmp_path):
with (
patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"),
patch("cmdforge.tool.BIN_DIR", tmp_path / "bin"),
):
from cmdforge.tool import save_tool
child = Tool(name="child", version="1.0.0")
save_tool(child)
parent = Tool(
name="parent",
version="1.0.0",
steps=[ToolStep(tool="child", output_var="x")],
output="{x}",
)
save_tool(parent)
chain = build_integrity_chain(parent)
assert "child" in chain.nodes
assert chain.is_valid
def test_child_change_changes_root_identity(self, tmp_path):
with (
patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"),
patch("cmdforge.tool.BIN_DIR", tmp_path / "bin"),
):
from cmdforge.tool import save_tool
child = Tool(name="child", version="1.0.0", output="one")
save_tool(child)
parent = Tool(
name="parent", version="1.0.0",
steps=[ToolStep(tool="child", output_var="x")], output="{x}",
)
first = build_integrity_chain(parent).root.content_hash
child.output = "two"
save_tool(child)
second = build_integrity_chain(parent).root.content_hash
assert first != second
class TestVerifyIntegrity:
def test_valid_chain(self):
from cmdforge.integrity import compute_content_identity
definition_hash = "sha256:" + "a" * 64
content_hash = compute_content_identity(definition_hash, {})
node = IntegrityNode(
name="root", definition_hash=definition_hash,
content_hash=content_hash,
)
chain = IntegrityChain(root=node, nodes={"root": node})
assert verify_integrity(chain)
def test_tampered_chain(self):
chain = IntegrityChain(
root=IntegrityNode(
name="root",
content_hash="abc",
dependencies=["dep"],
dependency_hashes={"dep": "expected_hash"},
),
nodes={
"root": IntegrityNode(
name="root",
content_hash="abc",
dependencies=["dep"],
dependency_hashes={"dep": "expected_hash"},
),
"dep": IntegrityNode(name="dep", content_hash="different_hash"),
},
)
assert not chain.is_valid
+13 -1
View File
@@ -259,6 +259,12 @@ class TestGenerateLockfile:
assert "official/text-utils" in lock.packages
assert lock.packages["official/summarize"].direct is True
assert lock.packages["official/text-utils"].direct is False
child_identity = lock.packages["official/text-utils"].content_hash
assert child_identity.startswith("sha256:")
assert lock.packages["official/summarize"].dependency_hashes == {
"official/text-utils": child_identity
}
assert lock.packages["official/summarize"].content_hash.startswith("sha256:")
class TestVerifyLockfile:
@@ -319,7 +325,9 @@ class TestLockfileRoundTrip:
integrity="sha256:abc123",
source="registry",
direct=True,
required_by=[]
required_by=[],
content_hash="sha256:root",
dependency_hashes={"official/text-utils": "sha256:child"},
),
"official/text-utils": LockedPackage(
name="official/text-utils",
@@ -364,6 +372,10 @@ class TestLockfileRoundTrip:
assert summarize.integrity == "sha256:abc123"
assert summarize.source == "registry"
assert summarize.direct is True
assert summarize.content_hash == "sha256:root"
assert summarize.dependency_hashes == {
"official/text-utils": "sha256:child"
}
text_utils = loaded.packages["official/text-utils"]
assert text_utils.direct is False
+988
View File
@@ -0,0 +1,988 @@
"""Tests for MCP client and McpStep execution."""
import sys
import subprocess
import time
from pathlib import Path
import pytest
import yaml
from unittest.mock import patch
from cmdforge.mcp_client import (
DEFAULT_INHERITED_ENV,
McpServerConfig,
McpClientManager,
_build_server_env,
_build_http_headers,
_normalize_result,
_sanitize,
_fingerprint,
load_mcp_config,
save_mcp_config,
)
from cmdforge.runner import _substitute_mcp_args, _deep_substitute, run_tool
from cmdforge.tool import McpStep, Tool
class TestAgentHostSetup:
def test_builds_current_vendor_cli_commands(self):
from cmdforge.agent_setup import build_host_add_command
codex = build_host_add_command(
"codex", "/usr/bin/codex", "cmdforge", "user"
)
assert codex[:5] == [
"/usr/bin/codex", "mcp", "add", "cmdforge", "--"
]
assert codex[-4:] == ["-m", "cmdforge.cli", "mcp", "serve"]
claude = build_host_add_command(
"claude-code", "/usr/bin/claude", "cmdforge", "project"
)
assert claude[:9] == [
"/usr/bin/claude", "mcp", "add", "--transport", "stdio",
"--scope", "project", "cmdforge", "--",
]
def test_codex_rejects_unsupported_scope(self):
from cmdforge.agent_setup import build_host_add_command
with pytest.raises(ValueError, match="user scope only"):
build_host_add_command(
"codex", "/usr/bin/codex", "cmdforge", "project"
)
def test_rejects_unsafe_registration_name(self):
from cmdforge.agent_setup import build_host_add_command
with pytest.raises(ValueError, match="registration name"):
build_host_add_command(
"codex", "/usr/bin/codex", "--config", "user"
)
def test_policy_merge_is_idempotent_and_preserves_user_text(self):
from cmdforge.agent_setup import POLICY_BODY, merge_agent_policy
original = "# Existing guidance\n\nKeep this.\n"
merged = merge_agent_policy(original)
assert merged.startswith(original)
assert POLICY_BODY.strip() in merged
assert merge_agent_policy(merged) == merged
def test_policy_merge_rejects_malformed_markers(self):
from cmdforge.agent_setup import POLICY_START, merge_agent_policy
with pytest.raises(ValueError, match="malformed"):
merge_agent_policy(f"text\n{POLICY_START}\n")
def test_policy_writer_refuses_symlink(self, tmp_path):
from cmdforge.agent_setup import write_agent_policy
target = tmp_path / "target.md"
target.write_text("unchanged")
link = tmp_path / "AGENTS.md"
link.symlink_to(target)
with pytest.raises(ValueError, match="symlinked"):
write_agent_policy(link, "replacement")
assert target.read_text() == "unchanged"
def test_dry_run_changes_nothing(self, tmp_path, capsys):
from cmdforge.agent_setup import configure_agent_host
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
patch("cmdforge.agent_setup.subprocess.run") as run,
):
assert configure_agent_host(
"codex", project_dir=tmp_path, dry_run=True
) == 0
run.assert_not_called()
assert not (tmp_path / "AGENTS.md").exists()
output = capsys.readouterr().out
assert "Would run:" in output
assert "cmdforge-agent-policy:start" in output
def test_configures_host_and_managed_policy(self, tmp_path, capsys):
from cmdforge.agent_setup import configure_agent_host
results = [
subprocess.CompletedProcess([], 1, "", "not found"),
subprocess.CompletedProcess([], 0, "added", ""),
]
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
patch("cmdforge.agent_setup.subprocess.run", side_effect=results) as run,
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
):
assert configure_agent_host("codex", project_dir=tmp_path) == 0
assert run.call_count == 2
assert run.call_args_list[0].args[0] == [
"/usr/bin/codex", "mcp", "get", "cmdforge"
]
assert run.call_args_list[1].args[0][:5] == [
"/usr/bin/codex", "mcp", "add", "cmdforge", "--"
]
policy = (tmp_path / "AGENTS.md").read_text()
assert "cmdforge-agent-policy:start" in policy
assert "Do not install, publish, expose" in policy
assert "Configured CmdForge MCP" in capsys.readouterr().out
def test_existing_registration_is_not_overwritten(self, tmp_path):
from cmdforge.agent_setup import configure_agent_host
result = subprocess.CompletedProcess([], 0, "exists", "")
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/claude"),
patch("cmdforge.agent_setup.subprocess.run", return_value=result) as run,
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
):
assert configure_agent_host(
"claude-code", project_dir=tmp_path
) == 0
assert run.call_count == 1
assert (tmp_path / "CLAUDE.md").exists()
def test_force_removes_then_readds_registration(self, tmp_path):
from cmdforge.agent_setup import configure_agent_host
results = [
subprocess.CompletedProcess([], 0, "exists", ""),
subprocess.CompletedProcess([], 0, "removed", ""),
subprocess.CompletedProcess([], 0, "added", ""),
]
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/claude"),
patch("cmdforge.agent_setup.subprocess.run", side_effect=results) as run,
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
):
assert configure_agent_host(
"claude-code", project_dir=tmp_path,
install_policy=False, force=True,
) == 0
assert run.call_args_list[1].args[0] == [
"/usr/bin/claude", "mcp", "remove", "cmdforge"
]
assert run.call_args_list[2].args[0][:4] == [
"/usr/bin/claude", "mcp", "add", "--transport"
]
def test_failed_host_add_does_not_write_policy(self, tmp_path):
from cmdforge.agent_setup import configure_agent_host
results = [
subprocess.CompletedProcess([], 1, "", "missing"),
subprocess.CompletedProcess([], 1, "", "bad config"),
]
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
patch("cmdforge.agent_setup.subprocess.run", side_effect=results),
):
assert configure_agent_host(
"codex", project_dir=tmp_path
) == 1
assert not (tmp_path / "AGENTS.md").exists()
def test_host_command_timeout_is_reported(self, tmp_path, capsys):
from cmdforge.agent_setup import configure_agent_host
with (
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
patch(
"cmdforge.agent_setup.subprocess.run",
side_effect=subprocess.TimeoutExpired("codex", 30),
),
):
assert configure_agent_host(
"codex", project_dir=tmp_path, install_policy=False
) == 1
assert "timed out after 30 seconds" in capsys.readouterr().err
class TestMcpServerConfig:
def test_defaults(self):
cfg = McpServerConfig(name="test")
assert cfg.transport == "stdio"
assert cfg.timeout == 30
assert cfg.args == []
assert cfg.env == {}
assert cfg.approved is False
@pytest.mark.parametrize(
"change, message",
[
({"transport": "streamable-http"}, "requires a url"),
({"args": "-y"}, "list of strings"),
({"timeout": 0}, "greater than 0"),
({"timeout": True}, "must be a number"),
],
)
def test_validation_rejects_invalid_config(self, change, message):
command = None if change.get("transport") == "streamable-http" else "server"
cfg = McpServerConfig(name="test", command=command, **change)
with pytest.raises(ValueError, match=message):
cfg.validate()
def test_fingerprint_changes_with_command(self):
a = McpServerConfig(name="a", command="cmd-a")
b = McpServerConfig(name="b", command="cmd-b")
assert _fingerprint(a) != _fingerprint(b)
def test_fingerprint_same_for_same_config(self):
a = McpServerConfig(name="a", command="cmd", args=["-v"])
b = McpServerConfig(name="b", command="cmd", args=["-v"])
assert _fingerprint(a) == _fingerprint(b)
def test_streamable_http_accepts_https(self):
cfg = McpServerConfig(
name="remote", transport="streamable-http",
url="https://example.com/mcp", approved=True,
)
cfg.validate()
def test_streamable_http_allows_loopback_http_only(self):
McpServerConfig(
name="local", transport="streamable-http",
url="http://127.0.0.1:8000/mcp",
).validate()
with pytest.raises(ValueError, match="must use HTTPS"):
McpServerConfig(
name="remote", transport="streamable-http",
url="http://example.com/mcp",
).validate()
def test_streamable_http_rejects_url_credentials(self):
with pytest.raises(ValueError, match="must not contain credentials"):
McpServerConfig(
name="remote", transport="streamable-http",
url="https://user:[email protected]/mcp",
).validate()
class TestSanitize:
def test_redacts_bearer(self):
assert "redacted" in _sanitize("Bearer abcdefghijklmnopqrstuvwxyz")
def test_redacts_sk_prefix(self):
assert "redacted" in _sanitize("sk-abcdefghijklmnopqrstuvwxyz")
def test_preserves_normal_text(self):
assert _sanitize("hello world") == "hello world"
def test_redacts_explicit_secret(self):
assert _sanitize("failed with hunter2", ["hunter2"]) == "failed with [redacted]"
class TestEnvironmentIsolation:
def test_only_allowlisted_environment_is_inherited(self, monkeypatch):
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "do-not-leak")
cfg = McpServerConfig(name="test", command="server")
environment = _build_server_env(cfg)
assert "CMDFORGE_TEST_SECRET" not in environment
assert set(environment).issubset(set(DEFAULT_INHERITED_ENV))
def test_explicit_environment_reference_is_resolved(self, monkeypatch):
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "allowed")
cfg = McpServerConfig(
name="test", command="server", env={"SERVER_TOKEN": "${CMDFORGE_TEST_SECRET}"}
)
assert _build_server_env(cfg)["SERVER_TOKEN"] == "allowed"
def test_http_header_reference_is_resolved(self, monkeypatch):
monkeypatch.setenv("MCP_TOKEN", "secret")
cfg = McpServerConfig(
name="remote", transport="streamable-http",
url="https://example.com/mcp",
headers={"Authorization": "Bearer ${MCP_TOKEN}"},
)
assert _build_http_headers(cfg)["Authorization"] == "Bearer secret"
def test_missing_environment_reference_fails(self, monkeypatch):
monkeypatch.delenv("CMDFORGE_MISSING", raising=False)
cfg = McpServerConfig(
name="test", command="server", env={"SERVER_TOKEN": "${CMDFORGE_MISSING}"}
)
with pytest.raises(ValueError, match="is not set"):
_build_server_env(cfg)
class TestMcpConfigPersistence:
@pytest.fixture
def temp_mcp_file(self, tmp_path, monkeypatch):
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
yield config_file
def test_save_and_load(self, temp_mcp_file):
servers = [
McpServerConfig(
name="filesystem",
command="npx",
args=["-y", "@scope/server", "/tmp"],
timeout=30,
approved=True,
)
]
save_mcp_config(servers)
loaded = load_mcp_config()
assert len(loaded) == 1
assert loaded[0].name == "filesystem"
assert loaded[0].command == "npx"
assert loaded[0].args == ["-y", "@scope/server", "/tmp"]
assert loaded[0].approved is True
def test_load_no_file(self, temp_mcp_file):
assert load_mcp_config() == []
def test_streamable_http_round_trip(self, temp_mcp_file):
save_mcp_config([McpServerConfig(
name="remote", transport="streamable-http",
url="https://example.com/mcp",
headers={"Authorization": "Bearer ${MCP_TOKEN}"},
approved=True,
)])
loaded = load_mcp_config()[0]
assert loaded.transport == "streamable-http"
assert loaded.command is None
assert loaded.url == "https://example.com/mcp"
assert loaded.headers == {"Authorization": "Bearer ${MCP_TOKEN}"}
def test_save_creates_0600_permissions(self, temp_mcp_file):
save_mcp_config([McpServerConfig(name="test", command="echo")])
perms = oct(temp_mcp_file.stat().st_mode & 0o777)
assert perms == "0o600"
def test_save_preserves_server_exposure_policy(self, temp_mcp_file):
temp_mcp_file.write_text(
"version: 1\nserver:\n expose: [summarize]\n deny: [dangerous]\n"
)
save_mcp_config([McpServerConfig(name="test", command="echo")])
data = yaml.safe_load(temp_mcp_file.read_text())
assert data["server"] == {
"expose": ["summarize"],
"deny": ["dangerous"],
}
class TestResultNormalization:
def test_auto_prefers_structured_content(self):
class FakeResult:
isError = False
structuredContent = {"sum": 3}
content = [type("Block", (), {"text": "plain text"})()]
result = _normalize_result(FakeResult(), "auto")
assert result == {"sum": 3}
def test_auto_falls_back_to_text(self):
class FakeResult:
isError = False
structuredContent = None
content = [type("Block", (), {"type": "text", "text": "hello"})()]
result = _normalize_result(FakeResult(), "auto")
assert result == "hello"
def test_auto_preserves_mixed_content(self):
class FakeResult:
isError = False
structuredContent = None
content = [
{"type": "text", "text": "hello"},
{"type": "image", "mimeType": "image/png", "data": "AA=="},
]
result = _normalize_result(FakeResult(), "auto")
assert result == FakeResult.content
def test_content_mode_always_returns_blocks(self):
class FakeResult:
isError = False
structuredContent = None
content = [{"type": "text", "text": "hello"}]
assert _normalize_result(FakeResult(), "content") == FakeResult.content
def test_structured_fails_without_structured_content(self):
class FakeResult:
isError = False
structuredContent = None
content = [type("Block", (), {"type": "text", "text": "hello"})()]
with pytest.raises(ValueError, match="structuredContent"):
_normalize_result(FakeResult(), "structured")
def test_text_mode(self):
class FakeResult:
isError = False
structuredContent = None
content = [
type("Block", (), {"type": "text", "text": "line1"})(),
type("Block", (), {"type": "text", "text": "line2"})(),
]
result = _normalize_result(FakeResult(), "text")
assert result == "line1\nline2"
def test_iserror_causes_failure(self):
class FakeResult:
isError = True
structuredContent = None
content = [type("Block", (), {"type": "text", "text": "error message"})()]
with pytest.raises(RuntimeError, match="error"):
_normalize_result(FakeResult(), "auto")
class TestMcpClientManager:
@pytest.fixture
def manager(self, tmp_path, monkeypatch):
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
return McpClientManager()
def test_list_servers_empty(self, manager):
assert manager.list_servers() == []
def test_list_servers_with_config(self, manager, tmp_path, monkeypatch):
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
save_mcp_config([McpServerConfig(name="test", command="echo")])
mgr = McpClientManager()
servers = mgr.list_servers()
assert len(servers) == 1
assert servers[0].name == "test"
def test_call_tool_unknown_server(self, manager):
with pytest.raises(KeyError, match="not configured"):
manager.call_tool("unknown", "tool", {})
def test_unapproved_server_is_not_executed(self, tmp_path, monkeypatch):
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
save_mcp_config([McpServerConfig(name="test", command="echo")])
with pytest.raises(PermissionError, match="not approved"):
McpClientManager().discover("test")
def test_missing_sdk_has_actionable_error(self, tmp_path, monkeypatch):
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
save_mcp_config([
McpServerConfig(name="test", command="echo", approved=True)
])
monkeypatch.setitem(sys.modules, "mcp", None)
with pytest.raises(ImportError, match=r"cmdforge\[mcp\]"):
McpClientManager().discover("test")
class TestArgumentSubstitution:
def test_single_variable_preserves_integer(self):
result = _deep_substitute("42", {"limit": 42})
assert result == "42" # no braces, so no substitution
def test_braced_variable_preserves_type(self):
variables = {"settings": {"limit": 50}}
result = _substitute_mcp_args(
{"options": {"limit": "{settings.limit}"}}, variables
)
assert result["options"]["limit"] == 50
def test_nested_dict_substitution(self):
variables = {"path": "/tmp/data.csv"}
result = _substitute_mcp_args(
{"files": [{"source": "{path}"}]}, variables
)
assert result["files"][0]["source"] == "/tmp/data.csv"
def test_string_with_variable_inside_text(self):
variables = {"name": "report"}
result = _deep_substitute("/tmp/{name}.csv", variables)
assert result == "/tmp/report.csv"
@pytest.mark.parametrize(
"value",
[
{"a": 1},
[1, 2],
-5,
"001",
"true",
False,
None,
],
)
def test_exact_reference_preserves_original_type(self, value):
assert _deep_substitute("{value}", {"value": value}) == value
class TestMcpStep:
def test_round_trip(self):
original = McpStep(
server="fixture", tool="add", arguments={"a": 1, "b": 2},
output_var="sum", result_mode="structured",
)
assert McpStep.from_dict(original.to_dict()) == original
@pytest.mark.parametrize(
"field, value",
[("server", ""), ("tool", None), ("arguments", []), ("result_mode", "raw")],
)
def test_rejects_invalid_fields(self, field, value):
data = {"server": "fixture", "tool": "echo", field: value}
with pytest.raises(ValueError):
McpStep.from_dict(data)
def test_runner_preserves_structured_result(self, monkeypatch):
calls = []
class FakeManager:
def call_tool(self, server, tool, arguments, result_mode="auto"):
calls.append((server, tool, arguments, result_mode))
return {"sum": 3}
monkeypatch.setattr("cmdforge.runner.McpClientManager", FakeManager)
tool = Tool(
name="mcp-add",
steps=[
McpStep(
server="fixture",
tool="add",
arguments={"a": "{input}", "b": 2},
output_var="result",
)
],
output="{result.sum}",
)
output, exit_code = run_tool(tool, 1, {})
assert (output, exit_code) == ("3", 0)
assert calls == [("fixture", "add", {"a": 1, "b": 2}, "auto")]
class TestRealMcpSdk:
@pytest.fixture
def manager(self, tmp_path, monkeypatch):
pytest.importorskip("mcp")
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
fixture = Path(__file__).parent / "fixtures" / "mcp_test_server.py"
save_mcp_config([
McpServerConfig(
name="fixture",
command=sys.executable,
args=[str(fixture)],
timeout=10,
approved=True,
)
])
return McpClientManager()
def test_discovery_uses_mcp_handshake_and_tools_list(self, manager):
tools = manager.discover("fixture")
schemas = {tool["name"]: tool for tool in tools}
assert {"echo", "add", "read_env", "current_directory", "wait"}.issubset(schemas)
assert schemas["add"]["inputSchema"]["required"] == ["a", "b"]
def test_calls_text_and_structured_tools(self, manager):
assert manager.call_tool("fixture", "echo", {"message": "hello"}, "text") == "hello"
assert manager.call_tool("fixture", "add", {"a": 1, "b": 2}) == {"sum": 3}
def test_parent_secret_is_not_leaked(self, manager, monkeypatch):
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "do-not-leak")
result = manager.call_tool(
"fixture", "read_env", {"name": "CMDFORGE_TEST_SECRET"}, "text"
)
assert result == ""
def test_configured_working_directory_is_used(self, manager, tmp_path):
manager.list_servers()
manager._configs["fixture"].cwd = str(tmp_path)
assert manager.call_tool("fixture", "current_directory", {}, "text") == str(tmp_path)
def test_timeout_is_enforced(self, manager):
manager.list_servers()
manager._configs["fixture"].timeout = 0.5
started = time.monotonic()
with pytest.raises(TimeoutError, match="timed out"):
manager.call_tool("fixture", "wait", {"seconds": 10}, "text")
assert time.monotonic() - started < 4
class TestMcpCli:
def test_add_accepts_leading_dash_argument(self, tmp_path, monkeypatch, capsys):
from cmdforge.cli import main
config_file = tmp_path / "mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
monkeypatch.setattr(
sys,
"argv",
[
"cmdforge", "mcp", "add", "fixture", "--command", "npx",
"--arg=-y", "--arg", "@scope/server",
],
)
assert main() == 0
loaded = load_mcp_config()
assert loaded[0].args == ["-y", "@scope/server"]
assert loaded[0].approved is True
assert "saved and approved" in capsys.readouterr().out
def test_bare_mcp_lists_servers(self, tmp_path, monkeypatch, capsys):
from cmdforge.cli import main
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", tmp_path / "mcp.yaml")
monkeypatch.setattr(sys, "argv", ["cmdforge", "mcp"])
assert main() == 0
assert "No MCP servers configured" in capsys.readouterr().out
class TestMcpServeConfig:
def test_empty_expose_exposes_nothing(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=[], deny=[])
assert not config.is_exposed("summarize")
def test_wildcard_exposes_all_public(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=["*"], deny=[])
assert config.is_exposed("summarize")
assert config.is_exposed("any-tool")
def test_deny_overrides_expose(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=["*"], deny=["dangerous"])
assert config.is_exposed("summarize")
assert not config.is_exposed("dangerous")
def test_prefix_deny(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=["*"], deny=["internal-*"])
assert not config.is_exposed("internal-cleanup")
assert not config.is_exposed("internal-")
assert config.is_exposed("public-tool")
def test_explicit_expose(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=["summarize", "extract"])
assert config.is_exposed("summarize")
assert config.is_exposed("extract")
assert not config.is_exposed("other")
@pytest.mark.parametrize(
"field, value",
[("expose", "*"), ("deny", "dangerous"), ("expose", [""])],
)
def test_rejects_malformed_policy_lists(self, field, value):
from cmdforge.mcp_client import McpServeConfig
values = {"expose": [], "deny": [], field: value}
with pytest.raises(ValueError, match="list of non-empty strings"):
McpServeConfig(**values)
def test_general_wildcard_matching(self):
from cmdforge.mcp_client import McpServeConfig
config = McpServeConfig(expose=["official/*"], deny=["*/dangerous"])
assert config.is_exposed("official/summarize")
assert not config.is_exposed("official/dangerous")
def test_private_tool_still_honors_deny(self):
from cmdforge.mcp_client import McpServeConfig
from cmdforge.mcp_server import _is_exposable
from cmdforge.tool import Tool
tool = Tool(name="dangerous", visibility="private")
config = McpServeConfig(expose=["official/dangerous"], deny=["official/*"])
assert not _is_exposable(tool, config, qualified_name="official/dangerous")
def test_private_qualified_tool_requires_qualified_expose(self):
from cmdforge.mcp_client import McpServeConfig
from cmdforge.mcp_server import _is_exposable
from cmdforge.tool import Tool
tool = Tool(name="extract", visibility="private")
assert not _is_exposable(
tool,
McpServeConfig(expose=["extract"]),
qualified_name="official/extract",
)
assert _is_exposable(
tool,
McpServeConfig(expose=["official/extract"]),
qualified_name="official/extract",
)
class TestToolNameMapping:
def test_simple_name_passes_through(self):
from cmdforge.mcp_server import _cmdforge_tool_name
assert _cmdforge_tool_name("summarize") == "summarize"
def test_qualified_name_uses_double_underscore(self):
from cmdforge.mcp_server import _cmdforge_tool_name
assert _cmdforge_tool_name("owner/tool") == "owner__tool"
class TestArgumentToJsonSchema:
def test_string_arg(self):
from cmdforge.mcp_server import _argument_to_json_schema
from cmdforge.tool import ToolArgument
arg = ToolArgument(flag="--name", variable="name", description="The name")
schema = _argument_to_json_schema(arg)
assert schema["type"] == "string"
assert schema["description"] == "The name"
def test_integer_arg_with_default(self):
from cmdforge.tool import ToolArgument
from cmdforge.mcp_server import _argument_to_json_schema
arg = ToolArgument(
flag="--limit", variable="limit", default="10", type="integer"
)
schema = _argument_to_json_schema(arg)
assert schema["type"] == "integer"
assert schema["default"] == 10
def test_arg_with_enum(self):
from cmdforge.tool import ToolArgument
from cmdforge.mcp_server import _argument_to_json_schema
arg = ToolArgument(
flag="--mode", variable="mode", enum=["fast", "accurate"]
)
schema = _argument_to_json_schema(arg)
assert schema["enum"] == ["fast", "accurate"]
class TestBuildToolSchema:
def test_builds_input_schema(self, tmp_path):
from cmdforge.tool import Tool, ToolArgument
from cmdforge.mcp_server import _build_tool_schema
with patch('cmdforge.tool.TOOLS_DIR', tmp_path / ".cmdforge"):
tool = Tool(
name="greet",
description="Greet someone",
arguments=[
ToolArgument(flag="--name", variable="name",
description="Who to greet", default="World"),
ToolArgument(flag="--count", variable="count", default="1"),
],
output="Hello {name} x{count}\n",
)
schema = _build_tool_schema(tool)
assert schema["name"] == "greet"
assert "Greet" in schema["description"]
assert schema["inputSchema"]["type"] == "object"
assert "name" in schema["inputSchema"]["properties"]
assert "count" in schema["inputSchema"]["properties"]
assert "input" in schema["inputSchema"]["properties"]
class TestRegisteredMcpTool:
def test_fastmcp_uses_flat_typed_schema_and_invokes_tool(self):
import asyncio
from mcp.server.fastmcp import FastMCP
from cmdforge.mcp_server import _build_tool_schema, _register_tool
from cmdforge.tool import Tool, ToolArgument
server = FastMCP("test")
tool = Tool(
name="greet",
arguments=[
ToolArgument(
flag="--name",
variable="name",
type="string",
required=True,
description="Who to greet",
),
ToolArgument(
flag="--count",
variable="count",
type="integer",
default=1,
enum=[1, 2, 3],
),
],
output="Hello {name} x{count}: {input}",
)
_register_tool(server, "greet", "greet", _build_tool_schema(tool), tool)
async def exercise():
listed = (await server.list_tools())[0]
result = await server.call_tool(
"greet", {"name": "Alice", "count": 2, "input": "welcome"}
)
return listed, result
listed, result = asyncio.run(exercise())
assert listed.inputSchema["required"] == ["name"]
assert listed.inputSchema["properties"]["count"]["type"] == "integer"
assert listed.inputSchema["properties"]["count"]["enum"] == [1, 2, 3]
assert "kwargs" not in listed.inputSchema["properties"]
assert result[0].text == "Hello Alice x2: welcome"
def test_handler_starts_at_inherited_depth(self):
import asyncio
from mcp.server.fastmcp import FastMCP
from cmdforge.mcp_server import _build_tool_schema, _register_tool
from cmdforge.tool import Tool
server = FastMCP("test")
tool = Tool(name="depth", output="ok")
with patch("cmdforge.runner.run_tool", return_value=("ok", 0)) as run:
_register_tool(server, "depth", "depth", _build_tool_schema(tool), tool)
asyncio.run(server.call_tool("depth", {}))
assert run.call_args.kwargs["_depth"] == 0
def test_handler_rejects_exhausted_mcp_depth(self, monkeypatch):
import asyncio
from mcp.server.fastmcp import FastMCP
from cmdforge.mcp_server import (
MAX_TOOL_DEPTH,
_build_tool_schema,
_register_tool,
)
from cmdforge.tool import Tool
monkeypatch.setenv("CMDFORGE_MCP_DEPTH", str(MAX_TOOL_DEPTH))
server = FastMCP("test")
tool = Tool(name="depth", output="ok")
_register_tool(server, "depth", "depth", _build_tool_schema(tool), tool)
with pytest.raises(Exception, match="Maximum MCP nesting depth"):
asyncio.run(server.call_tool("depth", {}))
def test_serve_fails_closed_on_mapped_name_collision(self):
from cmdforge.mcp_client import McpServeConfig
from cmdforge.mcp_server import serve
from cmdforge.tool import Tool
tools = {
"owner/tool": Tool(name="tool"),
"owner__tool": Tool(name="owner__tool"),
}
with (
patch(
"cmdforge.mcp_server.load_mcp_serve_config",
return_value=McpServeConfig(expose=["*"]),
),
patch("cmdforge.tool.list_tools", return_value=list(tools)),
patch("cmdforge.tool.load_tool", side_effect=tools.get),
patch("mcp.server.fastmcp.FastMCP.run"),
):
with pytest.raises(RuntimeError, match="name collision"):
serve()
class TestStreamableHttpServerSafety:
def test_defaults_are_loopback_with_origin_allowlist(self):
from cmdforge.mcp_server import _validate_http_server_options
origins, token = _validate_http_server_options(
"127.0.0.1", 8000, [], None, None
)
assert token is None
assert origins == [
"http://localhost:8000", "http://127.0.0.1:8000"
]
def test_nonlocal_requires_https_and_auth(self):
from cmdforge.mcp_server import _validate_http_server_options
with pytest.raises(ValueError, match="external-url"):
_validate_http_server_options("0.0.0.0", 8000, [], "secret", None)
with pytest.raises(ValueError, match="auth-token"):
_validate_http_server_options(
"0.0.0.0", 8000, [], None, "https://mcp.example.com"
)
def test_nonlocal_infers_https_origin_and_expands_token(self, monkeypatch):
from cmdforge.mcp_server import _validate_http_server_options
monkeypatch.setenv("MCP_AUTH", "secret-value")
origins, token = _validate_http_server_options(
"0.0.0.0", 8000, [], "${MCP_AUTH}",
"https://mcp.example.com",
)
assert origins == ["https://mcp.example.com"]
assert token == "secret-value"
def test_nonlocal_rejects_insecure_origin(self):
from cmdforge.mcp_server import _validate_http_server_options
with pytest.raises(ValueError, match="origins must use HTTPS"):
_validate_http_server_options(
"0.0.0.0", 8000, ["http://example.com"], "secret",
"https://mcp.example.com",
)
class TestCmdForgeMcpServerEndToEnd:
def test_stdio_discovery_and_invocation(self, tmp_path, monkeypatch):
pytest.importorskip("mcp")
home = tmp_path / "home"
cmdforge_dir = home / ".cmdforge"
cmdforge_dir.mkdir(parents=True)
(cmdforge_dir / "mcp.yaml").write_text(
yaml.safe_dump({
"version": 1,
"server": {
"expose": ["greet", "echo-input", "hidden"],
"deny": ["hidden"],
},
})
)
tools = {
"greet": {
"name": "greet",
"arguments": [{
"flag": "--name",
"variable": "name",
"type": "string",
"required": True,
}],
"steps": [],
"output": "Hello {name}",
},
"echo-input": {
"name": "echo-input",
"steps": [],
"output": "{input}",
},
"hidden": {
"name": "hidden",
"visibility": "private",
"steps": [],
"output": "secret",
},
}
for name, data in tools.items():
tool_dir = cmdforge_dir / name
tool_dir.mkdir()
(tool_dir / "config.yaml").write_text(yaml.safe_dump(data))
client_config = tmp_path / "client-mcp.yaml"
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", client_config)
save_mcp_config([
McpServerConfig(
name="cmdforge",
command=sys.executable,
args=["-m", "cmdforge.cli", "mcp", "serve"],
env={"HOME": str(home)},
timeout=10,
approved=True,
)
])
manager = McpClientManager()
discovered = {tool["name"]: tool for tool in manager.discover("cmdforge")}
assert set(discovered) == {"greet", "echo-input"}
assert discovered["greet"]["inputSchema"]["required"] == ["name"]
assert manager.call_tool(
"cmdforge", "greet", {"name": "Alice"}, "text"
) == "Hello Alice"
assert manager.call_tool(
"cmdforge", "echo-input", {"input": "through stdin"}, "text"
) == "through stdin"
+46 -4
View File
@@ -1,5 +1,6 @@
"""Tests for packaging metadata."""
"""Release metadata checks for the public Python distribution."""
from pathlib import Path
import sys
import pytest
@@ -9,12 +10,53 @@ if sys.version_info >= (3, 11):
else:
tomllib = pytest.importorskip("tomli")
from cmdforge import __version__
ROOT = Path(__file__).resolve().parents[1]
def _project_metadata() -> dict:
with (ROOT / "pyproject.toml").open("rb") as handle:
return tomllib.load(handle)["project"]
def test_runtime_and_distribution_versions_match():
assert _project_metadata()["version"] == __version__
def test_public_distribution_metadata_is_complete():
project = _project_metadata()
assert project["name"] == "cmdforge"
assert project["readme"] == "PYPI_README.md"
assert project["license"] == "MIT"
assert project["license-files"] == ["LICENSE"]
assert project["requires-python"] == ">=3.10"
assert {"Homepage", "Documentation", "Repository", "Issues", "Changelog"} <= set(
project["urls"]
)
def test_public_readme_describes_installation_and_current_capabilities():
readme = (ROOT / "PYPI_README.md").read_text(encoding="utf-8")
for term in (
'pipx install "cmdforge[mcp,pty]"',
"cmdforge providers discover",
"cmdforge mcp configure codex",
"--no-fallback",
"--result-envelope json",
):
assert term in readme
assert (ROOT / "LICENSE").is_file()
def test_web_templates_and_static_are_declared_as_package_data():
with open("pyproject.toml", "rb") as f:
data = tomllib.load(f)
with (ROOT / "pyproject.toml").open("rb") as handle:
data = tomllib.load(handle)
package_data = data["tool"]["setuptools"]["package-data"]["cmdforge.web"]
assert "templates/**/*.html" in package_data
assert "static/**/*" in package_data
+110
View File
@@ -0,0 +1,110 @@
"""Behavioral coverage for registry-aware picker and deprecation UX."""
from io import StringIO
import sys
import time
from unittest.mock import patch
import cmdforge.cli.picker as picker
import pytest
from cmdforge import __version__
from cmdforge.cli.picker import PickerResult
class FakeTTY:
def __init__(self, *characters):
self.characters = iter(characters)
def getch(self, timeout=None):
if timeout is not None:
time.sleep(0.01)
return None
return next(self.characters)
def setup_function():
picker._registry_cache.clear()
picker._ui_out = StringIO()
def test_public_registry_search_does_not_require_token():
class Client:
token = ""
def semantic_search(self, query, limit):
return {"available": False, "data": [], "error": None}
def search_tools(self, query, per_page):
return [{
"owner": "official", "name": "weather",
"description": "Weather", "score": 0.8,
"quality_score": 91, "quality_coverage": 70,
}]
with patch("cmdforge.registry_client.get_client", return_value=Client()):
results = picker.search_registry("weather")
assert results[0]["name"] == "official/weather"
assert results[0]["quality_score"] == 91
assert results[0]["quality_coverage"] == 70
def test_picker_can_install_when_no_local_tools():
registry_tool = {
"name": "official/weather", "desc": "Weather", "args": [],
"registry": True, "quality_score": 91, "quality_coverage": 70,
"relevance": 0.8,
}
with patch.object(picker, "get_tools", return_value=[]), patch.object(
picker, "search_registry", return_value=[registry_tool]
), patch.object(
picker, "_install_registry_selection",
return_value=PickerResult("official/weather", {}),
) as install:
result = picker.run_picker(FakeTTY("w", "e", "\n"))
install.assert_called_once()
assert result.tool_name == "official/weather"
assert "Q91/70%" in picker._ui_out.getvalue()
def test_tab_installs_registry_selection_too():
registry_tool = {
"name": "official/weather", "desc": "Weather", "args": [],
"registry": True,
}
with patch.object(picker, "get_tools", return_value=[]), patch.object(
picker, "search_registry", return_value=[registry_tool]
), patch.object(
picker, "_install_registry_selection",
return_value=PickerResult("official/weather", {}),
) as install:
result = picker.run_picker(FakeTTY("w", "e", "\t"))
install.assert_called_once()
assert result.tool_name == "official/weather"
def test_local_deprecation_selection_prints_migration_guidance():
tool = {
"name": "old", "desc": "", "args": [], "deprecated": True,
"deprecated_message": "Moved.", "replacement": "official/new",
}
result = picker._local_selection(tool)
assert result.tool_name == "old"
output = picker._ui_out.getvalue()
assert "Moved." in output
assert "official/new" in output
def test_picker_help_does_not_require_a_terminal(capsys):
with patch.object(sys, "argv", ["cf", "--help"]), pytest.raises(SystemExit) as exc:
picker.main()
assert exc.value.code == 0
assert "Interactively find and run" in capsys.readouterr().out
def test_picker_version_does_not_require_a_terminal(capsys):
with patch.object(sys, "argv", ["cf", "--version"]), pytest.raises(SystemExit) as exc:
picker.main()
assert exc.value.code == 0
assert f"cf {__version__}" in capsys.readouterr().out
+509
View File
@@ -0,0 +1,509 @@
"""Tests for preflight analysis engine."""
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from cmdforge.preflight import (
PreflightReport,
analyze_tool,
_check_config_integrity,
_check_similar_tools,
_is_semver,
)
from cmdforge.registry_client import PaginatedResponse, RegistryError
class TestPreflightReport:
def test_empty_report_is_ok(self):
report = PreflightReport()
assert report.ok
def test_errors_make_not_ok(self):
report = PreflightReport(errors=["bad"])
assert not report.ok
def test_merge(self):
a = PreflightReport(errors=["e1"], warnings=["w1"])
b = PreflightReport(errors=["e2"], suggestions=["s1"])
c = a.merge(b)
assert c.errors == ["e1", "e2"]
assert c.warnings == ["w1"]
assert c.suggestions == ["s1"]
def test_to_dict_returns_independent_lists(self):
report = PreflightReport(errors=["bad"])
serialized = report.to_dict()
serialized["errors"].append("another")
assert report.errors == ["bad"]
def test_to_dict_copies_contract_proposal(self):
report = PreflightReport(
contract_proposal={"input_schema": {"type": "string"}}
)
serialized = report.to_dict()
serialized["contract_proposal"]["input_schema"]["type"] = "integer"
assert report.contract_proposal["input_schema"]["type"] == "string"
class TestAnalyzeTool:
def test_no_steps_warns(self, tmp_path):
from cmdforge.tool import Tool, save_tool
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"), patch(
"cmdforge.tool.BIN_DIR", tmp_path / "bin"
):
tool = Tool(name="empty-tool")
save_tool(tool)
report = analyze_tool(tool)
assert report.ok
assert any("no steps" in w.lower() for w in report.warnings)
def test_missing_name(self, tmp_path):
from cmdforge.tool import Tool
report = PreflightReport()
tool = Tool(name="")
_check_config_integrity(tool, report)
assert not report.ok
def test_invalid_version(self, tmp_path):
from cmdforge.tool import Tool
report = PreflightReport()
tool = Tool(name="test", version="not-semver")
_check_config_integrity(tool, report)
assert not report.ok
def test_ok_tool(self, tmp_path):
from cmdforge.tool import Tool
report = PreflightReport()
tool = Tool(name="ok-tool", version="1.0.0")
_check_config_integrity(tool, report)
assert report.ok
def test_constant_tool_without_steps_is_not_passthrough_warning(self):
from cmdforge.tool import Tool
report = analyze_tool(Tool(name="constant", output="hello"))
assert not any("passthrough" in warning for warning in report.warnings)
def test_secret_pattern_in_prompt(self, tmp_path):
from cmdforge.tool import Tool, PromptStep
tool = Tool(
name="leaky",
steps=[PromptStep(
prompt="Use api_key: abc123 to auth",
provider="mock",
output_var="out",
)],
)
report = PreflightReport()
from cmdforge.preflight import _check_secrets
_check_secrets(tool, report)
assert any("secret" in w.lower() for w in report.warnings)
def test_secret_pattern_does_not_match_compound_schema_fields(self):
from cmdforge.tool import PromptStep, Tool
tool = Tool(
name="schema-docs",
steps=[PromptStep(
prompt="Document max_tokens and token_count fields",
provider="mock",
output_var="out",
)],
)
report = PreflightReport()
from cmdforge.preflight import _check_secrets
_check_secrets(tool, report)
assert report.warnings == []
@pytest.mark.parametrize("version", ["1.2.3garbage", "1.2.3.4", "1.2"])
def test_semver_rejects_trailing_or_incomplete_values(self, version):
assert _is_semver(version) is False
def test_registry_similarity_uses_paginated_data(self):
class Client:
def search_tools(self, query, per_page):
assert (query, per_page) == ("summarize", 5)
return PaginatedResponse(data=[{
"owner": "official",
"name": "summary",
"description": "Summarize text",
"downloads": 10,
}])
report = PreflightReport()
_check_similar_tools(SimpleNamespace(name="summarize"), report, Client())
assert report.similar_tools[0]["name"] == "official/summary"
def test_server_audit_does_not_claim_skipped_local_checks(self):
from cmdforge.tool import Tool, ToolStep
report = analyze_tool(
Tool(name="server", steps=[ToolStep(tool="child", output_var="out")]),
check_local_dependencies=False,
)
checks = report.audit_evidence["checks_run"]
assert "dependencies" not in checks
assert "toolstep_compatibility" not in checks
assert "reuse_opportunities" not in checks
assert report.compatibility == []
def test_unrelated_code_steps_are_not_reuse_evidence(self):
from cmdforge.tool import CodeStep, Tool
current = Tool(
name="current", input_schema={}, output_schema={},
steps=[
CodeStep(code="x = 1", output_var="x"),
CodeStep(code="y = 2", output_var="y"),
],
)
other = Tool(
name="other", input_schema={}, output_schema={},
steps=[
CodeStep(code="a = 99", output_var="a"),
CodeStep(code="b = 100", output_var="b"),
],
)
with patch("cmdforge.tool.list_tools", return_value=["other"]), patch(
"cmdforge.tool.load_tool", return_value=other
):
report = analyze_tool(current)
assert report.reuse_opportunities == []
def test_exact_contracted_duplicate_has_evidence(self):
from cmdforge.tool import PromptStep, Tool
def make(name):
return Tool(
name=name, input_schema={}, output_schema={"type": "string"},
steps=[
PromptStep(
prompt="first", provider="mock", output_var="one",
output_schema={"type": "string"},
),
PromptStep(
prompt="second", provider="mock", output_var="two",
output_schema={"type": "string"},
),
],
)
with patch("cmdforge.tool.list_tools", return_value=["other"]), patch(
"cmdforge.tool.load_tool", return_value=make("other")
):
report = analyze_tool(make("current"))
duplicate = [
item for item in report.reuse_opportunities
if item["type"] == "duplicate_sequence"
]
assert duplicate
assert "explicit tool contracts" in duplicate[0]["evidence"]
def test_registry_error_becomes_warning(self):
class Client:
def search_tools(self, query, per_page):
raise RegistryError("CONNECTION_ERROR", "offline")
report = PreflightReport()
_check_similar_tools(SimpleNamespace(name="summarize"), report, Client())
assert report.warnings == ["Could not search registry for similar tools"]
def test_programming_error_is_not_swallowed(self):
class Client:
def search_tools(self, query, per_page):
raise TypeError("broken adapter")
with pytest.raises(TypeError, match="broken adapter"):
_check_similar_tools(
SimpleNamespace(name="summarize"), PreflightReport(), Client()
)
class TestContractInference:
def test_no_arguments_no_input(self):
from cmdforge.tool import Tool
from cmdforge.preflight import infer_contracts
tool = Tool(name="empty", output="{response}")
result = infer_contracts(tool)
assert result["input_schema"] is None
assert result["output_schema"] is None
assert result["confidence"] == "low"
def test_argument_becomes_property(self):
from cmdforge.tool import Tool, ToolArgument
from cmdforge.preflight import infer_contracts
tool = Tool(
name="greet",
arguments=[ToolArgument(flag="--name", variable="name", description="Who to greet")],
output="Hello",
)
result = infer_contracts(tool)
assert result["input_schema"] is not None
assert "name" in result["input_schema"]["properties"]
assert result["input_schema"]["properties"]["name"]["description"] == "Who to greet"
def test_stdin_detected_when_input_used(self):
from cmdforge.tool import Tool, PromptStep
from cmdforge.preflight import infer_contracts
tool = Tool(
name="echo",
steps=[PromptStep(prompt="Repeat: {input}", provider="mock", output_var="response")],
output="{response}",
)
result = infer_contracts(tool)
assert result["input_schema"] is not None
assert "input" in result["input_schema"]["properties"]
assert any("stdin" in e.lower() for e in result["evidence"])
def test_enum_propagates(self):
from cmdforge.tool import Tool, ToolArgument
from cmdforge.preflight import infer_contracts
arg = ToolArgument(flag="--mode", variable="mode")
arg.enum = ["fast", "accurate"]
tool = Tool(name="tool", arguments=[arg], output="done")
result = infer_contracts(tool)
assert result["input_schema"]["properties"]["mode"]["enum"] == ["fast", "accurate"]
def test_explicit_empty_schema_not_overwritten(self):
from cmdforge.tool import Tool
from cmdforge.preflight import infer_contracts
tool = Tool(name="tool", input_schema={}, output_schema={})
result = infer_contracts(tool)
assert result["input_schema"] is None
assert result["output_schema"] is None
def test_output_propagated_from_step_schema(self):
from cmdforge.tool import Tool, PromptStep
from cmdforge.preflight import infer_contracts
step = PromptStep(
prompt="Analyze: {input}",
provider="mock",
output_var="analysis",
)
step.output_schema = {
"type": "object",
"properties": {"summary": {"type": "string"}},
}
tool = Tool(name="analyze", steps=[step], output="{analysis}")
result = infer_contracts(tool)
assert result["output_schema"] is not None
assert result["output_schema"]["type"] == "object"
assert "summary" in result["output_schema"]["properties"]
assert result["confidence"] == "high"
@pytest.mark.parametrize("schema_type", ["string", "number", "array", "boolean"])
def test_non_object_output_schema_is_propagated(self, schema_type):
from cmdforge.tool import PromptStep, Tool
from cmdforge.preflight import infer_contracts
schema = {"type": schema_type}
step = PromptStep(
prompt="Generate", provider="mock", output_var="result",
output_schema=schema,
)
proposal = infer_contracts(
Tool(name="typed", steps=[step], output="{result}")
)
assert proposal["output_schema"] == schema
def test_proposed_schema_does_not_alias_step_schema(self):
from cmdforge.tool import PromptStep, Tool
from cmdforge.preflight import infer_contracts
step = PromptStep(
prompt="Generate", provider="mock", output_var="result",
output_schema={
"type": "object",
"properties": {"value": {"type": "string"}},
},
)
proposal = infer_contracts(
Tool(name="copy", steps=[step], output="{result}")
)
proposal["output_schema"]["properties"]["value"]["type"] = "integer"
assert step.output_schema["properties"]["value"]["type"] == "string"
def test_stdin_detected_in_nested_tool_input(self):
from cmdforge.tool import Tool, ToolStep
from cmdforge.preflight import infer_contracts
proposal = infer_contracts(Tool(
name="nested",
steps=[ToolStep(tool="child", output_var="result")],
output="{result}",
))
assert "input" in proposal["input_schema"]["properties"]
def test_evidence_uses_real_argument_flag(self):
from cmdforge.tool import Tool, ToolArgument
from cmdforge.preflight import infer_contracts
proposal = infer_contracts(Tool(
name="flags",
arguments=[ToolArgument(flag="-n", variable="name")],
output="constant",
))
assert any("argument -n" in item for item in proposal["evidence"])
def test_explicit_contracts_produce_high_confidence(self):
from cmdforge.tool import Tool
from cmdforge.preflight import infer_contracts
proposal = infer_contracts(
Tool(name="explicit", input_schema={}, output_schema={})
)
assert proposal["confidence"] == "high"
def test_bare_variable_output_stays_bare(self):
from cmdforge.tool import Tool
from cmdforge.preflight import infer_contracts
tool = Tool(name="passthrough", steps=[], output="{response}")
result = infer_contracts(tool)
assert result["output_schema"] is None
assert any("source not found" in e.lower() for e in result["evidence"])
class TestConformanceIntegration:
def test_contract_tests_are_opt_in_for_server_side_preflight(self):
from cmdforge.tool import Tool
tool = Tool(name="server-safe", input_schema={}, output_schema={})
with patch("cmdforge.contract_testing.run_contract_tests") as runner:
analyze_tool(tool)
runner.assert_not_called()
def test_analyze_tool_records_passing_contract_case(self):
from cmdforge.tool import PromptStep, Tool
schema = {
"type": "object",
"properties": {"answer": {"type": "string"}},
"required": ["answer"],
}
tool = Tool(
name="contracted",
input_schema={},
output_schema=schema,
steps=[PromptStep(
prompt="Answer", provider="paid-provider", output_var="answer",
output_schema=schema,
)],
output="{answer}",
)
report = analyze_tool(tool, include_contract_tests=True)
assert report.ok
assert report.generated_tests[0]["state"] == "passed"
def test_unsupported_execution_is_advisory_not_failure(self):
from cmdforge.tool import CodeStep, Tool
tool = Tool(
name="code-tool",
input_schema={},
output_schema={},
steps=[CodeStep(code="result = 1", output_var="result")],
output="{result}",
)
report = analyze_tool(tool, include_contract_tests=True)
assert report.ok
assert report.generated_tests[0]["state"] == "unsupported"
assert any("unsupported" in warning.lower() for warning in report.warnings)
def test_saved_baseline_is_compared_during_local_preflight(self, tmp_path):
from cmdforge.contract_testing import (
run_contract_tests, save_conformance_baseline,
)
from cmdforge.tool import Tool
baseline_tool = Tool(
name="regression", version="1.0.0", input_schema={},
output_schema={"type": "string"}, output="before",
path=tmp_path / "config.yaml",
)
save_conformance_baseline(
baseline_tool, run_contract_tests(baseline_tool), tmp_path
)
current = Tool(
name="regression", version="2.0.0", input_schema={},
output_schema={"type": "string"}, output="after",
path=tmp_path / "config.yaml",
)
report = analyze_tool(current, include_contract_tests=True)
assert report.regression["has_regressions"] is True
assert report.regression["current_version"] == "2.0.0"
assert any("Regression comparison" in warning for warning in report.warnings)
def test_baseline_inputs_are_rerun_after_input_schema_change(self, tmp_path):
from cmdforge.contract_testing import (
run_contract_tests, save_conformance_baseline,
)
from cmdforge.tool import Tool
baseline_tool = Tool(
name="input-change", version="1.0.0", input_schema={},
output_schema={"type": "string"}, output="stable",
path=tmp_path / "config.yaml",
)
save_conformance_baseline(
baseline_tool, run_contract_tests(baseline_tool), tmp_path
)
current = Tool(
name="input-change", version="2.0.0",
input_schema={
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
},
output_schema={"type": "string"}, output="stable",
path=tmp_path / "config.yaml",
)
report = analyze_tool(current, include_contract_tests=True)
assert report.regression["has_regressions"] is True
assert len(report.regression["regressed"]) == 1
assert any(
result["input_value"] == {} and result["state"] == "failed"
for result in report.generated_tests
)
def test_invalid_baseline_is_an_advisory(self, tmp_path):
from cmdforge.contract_testing import CONFORMANCE_FILE
from cmdforge.tool import Tool
(tmp_path / CONFORMANCE_FILE).write_text("not json")
tool = Tool(
name="invalid-baseline", input_schema={}, output_schema={},
output="ok", path=tmp_path / "config.yaml",
)
report = analyze_tool(tool, include_contract_tests=True)
assert report.ok
assert any("baseline is invalid" in warning.lower() for warning in report.warnings)
def test_partial_unsupported_conformance_is_not_hidden_by_a_pass(self, monkeypatch):
from cmdforge.contract_testing import ConformanceReport, ConformanceResult
from cmdforge.tool import Tool
monkeypatch.setattr(
"cmdforge.contract_testing.run_contract_tests",
lambda tool, test_inputs=None: ConformanceReport([
ConformanceResult("case-a", "passed", "valid"),
ConformanceResult("input-generation", "unsupported", "unsupported keyword"),
]),
)
tool = Tool(name="mixed", input_schema={}, output_schema={})
report = analyze_tool(tool, include_contract_tests=True)
assert any("unsupported keyword" in warning for warning in report.warnings)
+186
View File
@@ -0,0 +1,186 @@
"""Tests for prompt optimization (M9.1)."""
from unittest.mock import patch
from cmdforge.prompt_optimizer import (
PromptVariation,
OptimizationResult,
generate_variations,
optimize_tool,
_get_strategies,
_truncate,
)
class TestTruncate:
def test_short_text_unchanged(self):
assert _truncate("hello", 10) == "hello"
def test_long_text_truncated(self):
result = _truncate("this is a very long sentence that needs truncation", 20)
assert len(result) <= 20
assert result.endswith("...")
class TestGenerateVariations:
def test_mock_provider_never_calls_external_provider(self):
with patch("cmdforge.providers.call_provider") as call:
_get_strategies("Summarize the text", 2, provider="mock")
call.assert_not_called()
def test_selected_provider_is_honored(self):
from cmdforge.providers import ProviderResult
with patch(
"cmdforge.providers.call_provider",
return_value=ProviderResult(text="A materially clearer instruction", success=True),
) as call:
result = _get_strategies("Summarize the text", 1, provider="chosen")
assert result[0][1] == "A materially clearer instruction"
assert call.call_args.args[0] == "chosen"
def test_generates_for_prompt_steps(self, tmp_path):
from cmdforge.tool import Tool, PromptStep
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(
name="test-tool",
steps=[
PromptStep(
prompt="Analyze the input and produce a summary",
provider="mock",
output_var="summary",
),
],
output="{summary}",
)
variations = generate_variations(tool, count=3)
assert len(variations) >= 1
for v in variations:
assert v.step_index == 0
assert v.original == "Analyze the input and produce a summary"
assert v.variation != v.original
def test_skips_non_prompt_steps(self, tmp_path):
from cmdforge.tool import Tool, CodeStep
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(
name="test-tool",
steps=[CodeStep(code="print('hello')", output_var="out")],
output="{out}",
)
variations = generate_variations(tool)
assert len(variations) == 0
class TestGetStrategies:
def test_produces_variations(self):
strategies = _get_strategies("Summarize the text", 3)
assert len(strategies) == 3
for strat, text in strategies:
assert len(text) > 0
assert strat in ("focused", "concise", "detailed", "rephrase", "structure")
def test_truncates_concise(self):
original = "A" * 200
strategies = _get_strategies(original, 3)
concise = [t for s, t in strategies if s == "concise"]
if concise:
assert len(concise[0]) <= 125 # 120 + "..."
class TestOptimizeTool:
def test_structural_results_do_not_claim_a_best_prompt(self):
from cmdforge.tool import Tool, PromptStep
tool = Tool(
name="structural", steps=[PromptStep("Summarize", "mock", "out")],
output="{out}", input_schema={"type": "string"},
output_schema={"type": "string"},
)
result = optimize_tool(tool, count=2)
assert result.best is None
assert "cannot compare prompt semantics" in result.note
def test_behavioral_evaluator_selects_only_an_improvement(self):
from cmdforge.tool import Tool, PromptStep
tool = Tool(
name="behavioral", steps=[PromptStep("Do it", "mock", "out")],
output="{out}",
)
result = optimize_tool(
tool, count=3,
evaluator=lambda candidate: len(candidate.steps[0].prompt),
)
assert result.best is not None
assert result.scores[result.best_index] > result.baseline_score
def test_baseline_score_recorded(self, tmp_path):
from cmdforge.tool import Tool, PromptStep
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(
name="opt-test",
steps=[
PromptStep(
prompt="Summarize: {input}",
provider="mock",
output_var="result",
),
],
output="{result}",
)
result = optimize_tool(tool, count=2)
assert result.tool_name == "opt-test"
assert result.baseline_score >= 0
assert len(result.variations) >= 1
assert len(result.scores) == len(result.variations)
def test_no_prompt_steps_returns_empty(self, tmp_path):
from cmdforge.tool import Tool, CodeStep
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
tool = Tool(
name="code-only",
steps=[CodeStep(code="pass", output_var="x")],
output="{x}",
)
result = optimize_tool(tool)
assert len(result.variations) == 0
assert result.best is None
class TestPromptVariation:
def test_to_dict(self):
v = PromptVariation(
step_index=0,
original="hello",
variation="hi there",
strategy="rephrase",
)
d = v.to_dict()
assert d["original"] == "hello"
assert d["strategy"] == "rephrase"
class TestOptimizationResult:
def test_improvement_calculation(self):
result = OptimizationResult(
tool_name="test",
variations=[
PromptVariation(0, "a", "b", "rephrase"),
],
scores={0: 5},
best_index=0,
baseline_score=3,
)
assert result.best is not None
assert result.improvement > 0
def test_no_improvement_when_no_baseline(self):
result = OptimizationResult(
tool_name="test",
variations=[],
scores={},
baseline_score=0,
)
assert result.best is None
assert result.improvement == 0.0
+293 -2
View File
@@ -8,8 +8,8 @@ import pytest
import yaml
from cmdforge.providers import (
Provider, ProviderResult,
load_providers, save_providers, get_provider,
Provider, ProviderResult, ProviderExecutionPolicy,
load_providers, save_providers, get_provider, get_providers_file,
add_provider, delete_provider,
call_provider, discover_installed_providers, mock_provider,
DEFAULT_PROVIDERS, strip_ansi
@@ -60,6 +60,19 @@ class TestProvider:
provider = Provider.from_dict(data)
assert provider.description == ""
def test_legacy_policy_field_aliases_are_canonicalized(self):
provider = Provider.from_dict({
"name": "legacy-policy",
"command": "provider",
"model_id": "model-a",
"max_input_tokens": 8192,
})
assert provider.model == "model-a"
assert provider.max_context_tokens == 8192
assert provider.to_dict()["model"] == "model-a"
assert provider.to_dict()["max_context_tokens"] == 8192
def test_roundtrip(self):
original = Provider(
name="custom",
@@ -90,6 +103,13 @@ class TestProvider:
fallback_chain=["backup", "mock"],
api_key_env="TEST_API_KEY",
pty_config={"prompt_pattern": ">"},
locality="remote",
capabilities=["text", "structured-json"],
model_digest="sha256:abc",
cost_class="low",
latency_class="fast",
max_context_tokens=131072,
data_policy="internal",
)
assert Provider.from_dict(original.to_dict()) == original
@@ -172,6 +192,32 @@ class TestProviderPersistence:
assert oct(temp_providers_file.stat().st_mode & 0o777) == "0o600"
def test_first_run_without_discovered_providers_writes_defaults(
self, temp_providers_file, capsys
):
with patch(
"cmdforge.providers.discover_installed_providers", return_value=[]
):
result = get_providers_file()
assert result == temp_providers_file
assert temp_providers_file.is_file()
assert "No AI providers detected" in capsys.readouterr().err
def test_malformed_access_policy_fails_closed(self, temp_providers_file, capsys):
temp_providers_file.parent.mkdir(parents=True)
temp_providers_file.write_text(yaml.safe_dump({
"version": 2,
"providers": [{
"name": "claude",
"command": "claude -p",
"tools": "should-have-been-a-list",
}],
}))
assert load_providers() == []
assert "Failed to load provider configuration" in capsys.readouterr().err
def test_legacy_config_gains_missing_defaults_without_losing_custom_provider(self, temp_providers_file):
temp_providers_file.parent.mkdir(parents=True)
temp_providers_file.write_text(yaml.safe_dump({
@@ -502,6 +548,200 @@ class TestProviderFallback:
# Fallback to mock should succeed
assert result.success is True
assert "[MOCK]" in result.text
assert result.requested_provider == "primary"
assert result.actual_provider == "mock"
assert result.attempted_providers == ["primary", "mock"]
assert result.fallback_used is True
def test_no_fallback_fails_closed(self, temp_providers_file):
save_providers([
Provider("primary", "nonexistent-cmd", fallback="mock"),
Provider("mock", "mock"),
])
result = call_provider(
"primary", "Test prompt",
execution_policy=ProviderExecutionPolicy(allow_fallback=False),
)
assert result.success is False
assert result.requested_provider == "primary"
assert result.actual_provider is None
assert result.attempted_providers == ["primary"]
assert result.fallback_used is False
def test_private_data_requires_explicit_compatible_policy(
self, temp_providers_file
):
save_providers([
Provider(
"local", "unused", locality="local",
capabilities=["structured-json"],
),
])
result = call_provider(
"local", "private packet",
execution_policy=ProviderExecutionPolicy(
allow_fallback=False,
required_locality="local",
required_capabilities=("structured-json",),
data_classification="private",
),
)
assert result.success is False
assert "data_policy is unspecified" in result.error
def test_require_local_rejects_ollama_redirected_to_remote_host(
self, temp_providers_file, monkeypatch
):
save_providers([
Provider(
"ollama-private", "ollama run safe", locality="local",
data_policy="private",
),
])
monkeypatch.setenv("OLLAMA_HOST", "gpu.example.test:11434")
result = call_provider(
"ollama-private", "private packet",
execution_policy=ProviderExecutionPolicy(
allow_fallback=False,
required_locality="local",
data_classification="private",
),
)
assert result.success is False
assert "locality is remote" in result.error
def test_ollama_digest_is_observed_from_local_api(
self, temp_providers_file, monkeypatch
):
save_providers([
Provider(
"ollama-safe", "ollama run safe:latest", model="safe:latest",
locality="local", data_policy="private",
),
])
monkeypatch.delenv("OLLAMA_HOST", raising=False)
class Response:
def __enter__(self):
return self
def __exit__(self, *args):
return False
def read(self):
return (
b'{"models":[{"name":"safe:latest","model":"safe:latest",'
b'"digest":"abcdef"}]}'
)
with (
patch("urllib.request.urlopen", return_value=Response()),
patch(
"cmdforge.providers.call_provider_subprocess",
return_value=ProviderResult(text="safe", success=True),
),
):
result = call_provider(
"ollama-safe", "private",
execution_policy=ProviderExecutionPolicy(
allow_fallback=False,
required_locality="local",
data_classification="private",
require_model_identity=True,
require_model_digest=True,
),
)
assert result.model == "safe:latest"
assert result.model_digest == "sha256:abcdef"
assert result.model_identity_source == "ollama-api"
def test_policy_eligible_provider_reports_verified_metadata(
self, temp_providers_file
):
provider = Provider(
"local", "local-ai", model="model-a",
model_digest="sha256:123", locality="local",
capabilities=["structured-json"], data_policy="private",
)
save_providers([provider])
low_level = ProviderResult(text='{"ok": true}', success=True)
with patch(
"cmdforge.providers.call_provider_subprocess",
return_value=low_level,
):
result = call_provider(
"local", "private packet",
execution_policy=ProviderExecutionPolicy(
allow_fallback=False,
required_locality="local",
required_capabilities=("structured-json",),
data_classification="private",
require_model_identity=True,
require_model_digest=True,
),
)
assert result.success is True
assert result.provenance() == {
"success": True,
"requested_provider": "local",
"actual_provider": "local",
"attempted_providers": ["local"],
"fallback_used": False,
"model": "model-a",
"model_digest": "sha256:123",
"locality": "local",
"model_identity_source": "provider-config",
}
def test_fallback_chain_skips_policy_ineligible_provider(
self, temp_providers_file
):
save_providers([
Provider(
"primary", "primary", fallback_chain=["remote", "local"],
locality="local", data_policy="private",
),
Provider(
"remote", "remote", locality="remote", data_policy="private",
),
Provider(
"local", "local", locality="local", data_policy="private",
model="safe-model",
),
])
def invoke(provider, prompt, timeout, max_tokens):
if provider.name == "primary":
return ProviderResult(text="", success=False, error="offline")
assert provider.name == "local"
return ProviderResult(text="safe", success=True)
with patch(
"cmdforge.providers.call_provider_subprocess", side_effect=invoke
) as low_level:
result = call_provider(
"primary", "private packet",
execution_policy=ProviderExecutionPolicy(
required_locality="local",
data_classification="private",
),
)
assert result.success is True
assert result.actual_provider == "local"
assert result.attempted_providers == ["primary", "remote", "local"]
assert [call.args[0].name for call in low_level.call_args_list] == [
"primary", "local"
]
def test_fallback_chain(self, temp_providers_file):
"""Fallback can chain to another provider with fallback."""
@@ -784,3 +1024,54 @@ class TestProviderDiscovery:
assert len({item["name"] for item in models}) == 2
assert models[0]["name"] == "ollama-hermes4-3-latest"
assert models[1]["name"] == "ollama-hermes4-3-q4-k-m"
class TestProviderAccessControl:
def test_tools_allowlist_defaults_to_none(self):
provider = Provider("test", "cmd")
assert provider.tools is None
assert provider.mcp_servers is None
def test_tools_allowlist_roundtrip(self):
provider = Provider("test", "cmd", tools=["tool-a", "tool-b"])
d = provider.to_dict()
assert d["tools"] == ["tool-a", "tool-b"]
restored = Provider.from_dict(d)
assert restored.tools == ["tool-a", "tool-b"]
def test_mcp_servers_roundtrip(self):
provider = Provider("test", "cmd", mcp_servers=["filesystem"])
d = provider.to_dict()
assert d["mcp_servers"] == ["filesystem"]
restored = Provider.from_dict(d)
assert restored.mcp_servers == ["filesystem"]
def test_tools_not_in_dict_when_none(self):
provider = Provider("test", "cmd")
assert "tools" not in provider.to_dict()
assert "mcp_servers" not in provider.to_dict()
def test_empty_allowlists_survive_roundtrip(self):
provider = Provider("locked", "cmd", tools=[], mcp_servers=[])
serialized = provider.to_dict()
assert serialized["tools"] == []
assert serialized["mcp_servers"] == []
restored = Provider.from_dict(serialized)
assert restored.tools == []
assert restored.mcp_servers == []
@pytest.mark.parametrize(
("field", "value"),
[
("tools", "tool-a"),
("tools", [""]),
("tools", [1]),
("mcp_servers", "filesystem"),
("mcp_servers", [""]),
("mcp_servers", [1]),
],
)
def test_rejects_invalid_access_policies(self, field, value):
with pytest.raises(ValueError, match=field):
Provider("test", "cmd", **{field: value})
+199
View File
@@ -0,0 +1,199 @@
"""Tests for explainable quality scoring (M8.Q)."""
from cmdforge.quality import (
CategoryScore,
QualityScore,
compute_quality,
)
from cmdforge.preflight import PreflightReport
from cmdforge.tool import Tool
class TestCategoryScore:
def test_display(self):
cs = CategoryScore("Tests", 25, 30)
assert cs.display == "25/30"
def test_percentage(self):
cs = CategoryScore("Tests", 15, 30)
assert cs.percentage == 50.0
def test_zero_available(self):
cs = CategoryScore("X", 0, 0)
assert cs.percentage == 0.0
def test_display_distinguishes_missing_evidence(self):
assert CategoryScore("Tests", 0, 30, "not_tested").display == "not tested"
assert CategoryScore("Tests", 0, 30, "not_applicable").display == "n/a"
class TestQualityScore:
def test_to_dict(self):
qs = QualityScore(
tool_name="test",
version="1.0.0",
headline=85,
categories=[CategoryScore("Contracts", 15, 15)],
)
d = qs.to_dict()
assert d["tool"] == "test"
assert d["score"] == 85
assert d["categories"][0]["name"] == "Contracts"
def test_str_has_headline(self):
qs = QualityScore(
tool_name="test",
version="1.0.0",
headline=87,
categories=[CategoryScore("Contracts", 15, 15)],
)
assert "Quality 87" in str(qs)
class TestComputeQuality:
def test_empty_tool_low_score(self):
tool = Tool(name="bare")
report = PreflightReport()
qs = compute_quality(tool, report)
assert qs.headline < 50
# Contracts should be not_tested
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
assert contracts.state == "not_tested"
assert qs.headline == 0
assert qs.evidence_coverage == 0
def test_missing_categories_do_not_reduce_checked_score(self):
tool = Tool(name="contracted", input_schema={}, output_schema={})
report = PreflightReport()
qs = compute_quality(tool, report)
assert qs.headline == 100
assert qs.evidence_coverage == 15
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
assert contracts.state == "checked"
def test_unsupported_tests_are_not_scored_as_failures(self):
report = PreflightReport(generated_tests=[{
"step": "preflight", "state": "unsupported", "detail": "safe mode"
}])
tests = [
c for c in compute_quality(Tool(name="tool"), report).categories
if c.name == "Deterministic tests"
][0]
assert tests.state == "not_tested"
assert tests.display == "not tested"
def test_partial_unsupported_tests_reduce_coverage_not_score(self):
report = PreflightReport(generated_tests=[
{"step": "case", "state": "passed", "detail": "ok"},
{"step": "generation", "state": "unsupported", "detail": "unknown"},
])
tests = [
c for c in compute_quality(Tool(name="tool"), report).categories
if c.name == "Deterministic tests"
][0]
assert tests.earned == tests.available == 15
assert tests.possible == 30
def test_with_contracts_scores_higher(self):
tool = Tool(
name="contracted",
input_schema={"type": "object", "properties": {"input": {"type": "string"}}},
output_schema={"type": "object", "properties": {"result": {"type": "string"}}},
)
report = PreflightReport()
qs = compute_quality(tool, report)
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
assert contracts.earned == 15
assert contracts.state == "checked"
def test_with_tests_scores_higher(self):
tool = Tool(name="tested")
report = PreflightReport(
generated_tests=[
{"step": "case-1", "state": "passed", "detail": "ok"},
{"step": "case-2", "state": "passed", "detail": "ok"},
]
)
qs = compute_quality(tool, report)
tests = [c for c in qs.categories if c.name == "Deterministic tests"][0]
assert tests.earned == 30
assert tests.state == "checked"
def test_failed_tests_reduce_score(self):
tool = Tool(name="broken")
report = PreflightReport(
generated_tests=[
{"step": "case-1", "state": "passed", "detail": "ok"},
{"step": "case-2", "state": "failed", "detail": "broken"},
]
)
qs = compute_quality(tool, report)
tests = [c for c in qs.categories if c.name == "Deterministic tests"][0]
assert tests.earned == 15
def test_regression_no_baseline_is_not_tested(self):
tool = Tool(name="test")
report = PreflightReport()
qs = compute_quality(tool, report)
regression = [c for c in qs.categories if c.name == "Regression history"][0]
assert regression.state == "not_tested"
def test_regression_stable_full_score(self):
tool = Tool(name="test")
report = PreflightReport(
regression={"has_regressions": False, "summary": "no changes"}
)
qs = compute_quality(tool, report)
regression = [c for c in qs.categories if c.name == "Regression history"][0]
assert regression.earned == 20
def test_regression_regressions_zero(self):
tool = Tool(name="test")
report = PreflightReport(
regression={"has_regressions": True, "summary": "1 regression(s)"}
)
qs = compute_quality(tool, report)
regression = [c for c in qs.categories if c.name == "Regression history"][0]
assert regression.earned == 0
def test_secret_warnings_reduce_security(self):
tool = Tool(name="leaky")
report = PreflightReport(
warnings=["Prompt step contains potential secret pattern 'api_key'"]
)
qs = compute_quality(tool, report)
security = [c for c in qs.categories if c.name == "Security scrutiny"][0]
assert security.earned == 15 # 20 - 5
def test_skipped_dependency_check_is_not_claimed_as_security_evidence(self):
tool = Tool(name="server")
report = PreflightReport(audit_evidence={"checks_run": ["secrets"]})
security = [
c for c in compute_quality(tool, report).categories
if c.name == "Security scrutiny"
][0]
assert security.earned == 10
assert security.available == 10
assert security.possible == 20
def test_community_not_tested_without_data(self):
tool = Tool(name="new")
report = PreflightReport()
qs = compute_quality(tool, report)
community = [c for c in qs.categories if c.name == "Community evidence"][0]
assert community.state == "not_tested"
def test_community_with_data(self):
tool = Tool(name="popular")
report = PreflightReport()
qs = compute_quality(
tool, report,
registry_data={
"reviews": [{"rating": 4}, {"rating": 5}],
"downloads": 500,
"featured": True,
}
)
community = [c for c in qs.categories if c.name == "Community evidence"][0]
assert community.earned == 14 # 7 (reviews) + 4 (downloads) + 3 (featured)
assert community.state == "checked"
+76
View File
@@ -0,0 +1,76 @@
"""Tests for immutable registry audit refreshes."""
from datetime import datetime, timedelta, timezone
import pytest
from cmdforge.registry.auditing import audit_stale_tools, audit_tool_version
from cmdforge.registry.db import connect_db, init_db
@pytest.fixture
def audit_db(tmp_path):
conn = connect_db(tmp_path / "registry.db")
init_db(conn)
cursor = conn.execute(
"""
INSERT INTO publishers (email, password_hash, slug, display_name)
VALUES ('[email protected]', 'x', 'auditor', 'Auditor')
"""
)
publisher_id = cursor.lastrowid
tool = conn.execute(
"""
INSERT INTO tools (
owner, name, version, config_yaml, publisher_id,
visibility, moderation_status
) VALUES ('auditor', 'checked', '1.0.0', ?, ?, 'public', 'approved')
""",
[
"name: checked\nversion: 1.0.0\noutput: stable\n"
"input_schema: {}\noutput_schema:\n type: string\n",
publisher_id,
],
)
conn.commit()
yield conn, tool.lastrowid
conn.close()
def test_background_audits_append_instead_of_overwriting(audit_db):
conn, tool_id = audit_db
first = audit_tool_version(conn, tool_id)
second = audit_tool_version(conn, tool_id)
count = conn.execute(
"SELECT COUNT(*) FROM tool_audits WHERE tool_id = ?", [tool_id]
).fetchone()[0]
assert count == 2
assert first["evidence"]["findings_hash"] == second["evidence"]["findings_hash"]
assert "generated_tests" in first["evidence"]["findings"]
assert "dependencies" not in first["evidence"]["checks_run"]
def test_stale_audit_refresh_adds_new_evidence(audit_db):
conn, tool_id = audit_db
audit_tool_version(conn, tool_id)
old = (datetime.now(timezone.utc) - timedelta(days=60)).isoformat()
conn.execute("UPDATE tool_audits SET evaluated_at = ?", [old])
conn.commit()
results = audit_stale_tools(conn, max_age_days=30)
assert [item["tool_id"] for item in results] == [tool_id]
assert conn.execute("SELECT COUNT(*) FROM tool_audits").fetchone()[0] == 2
def test_degraded_background_score_is_logged(audit_db):
conn, tool_id = audit_db
conn.execute("UPDATE tools SET downloads = 500 WHERE id = ?", [tool_id])
conn.commit()
audit_tool_version(conn, tool_id)
conn.execute("UPDATE tools SET downloads = 0 WHERE id = ?", [tool_id])
conn.commit()
result = audit_tool_version(conn, tool_id)
assert result["quality_change"] < 0
log = conn.execute(
"SELECT action FROM audit_log WHERE target_id = ?", [str(tool_id)]
).fetchone()
assert log["action"] == "automated_audit_changed"
+17 -2
View File
@@ -561,6 +561,7 @@ category: text-processing
assert data["status"] == "validated"
assert data["name"] == tool_name
assert data["owner"] == slug
assert data["preflight"]["errors"] == []
def test_publish_and_retrieve(self, session, base_url, auth_headers):
"""Test publishing a tool and retrieving it."""
@@ -588,8 +589,13 @@ tags:
assert data["name"] == tool_name
assert data["version"] == "1.0.0"
# Retrieve
# Pending public tools are hidden from anonymous callers, but remain
# visible to their owner while awaiting moderation.
resp = session.get(f"{base_url}/tools/{slug}/{tool_name}")
assert resp.status_code == 404
resp = session.get(
f"{base_url}/tools/{slug}/{tool_name}", headers=headers
)
assert resp.status_code == 200
tool = resp.json()["data"]
assert tool["name"] == tool_name
@@ -620,11 +626,20 @@ description: First version
})
assert resp.status_code == 201
# Duplicate publish
# An identical retry is idempotent.
resp = session.post(f"{base_url}/tools", headers=headers, json={
"config": config,
"readme": ""
})
assert resp.status_code == 200
assert resp.json()["data"]["already_published"] is True
# Different content at the same version is a real conflict.
changed_config = config.replace("First version", "Changed content")
resp = session.post(f"{base_url}/tools", headers=headers, json={
"config": changed_config,
"readme": ""
})
assert resp.status_code == 409
assert resp.json()["error"]["code"] == "VERSION_EXISTS"
+248 -2
View File
@@ -1,18 +1,20 @@
"""Tests for runner.py - Tool execution engine."""
import pytest
from types import SimpleNamespace
from unittest.mock import patch, MagicMock
from cmdforge.runner import (
substitute_variables,
execute_prompt_step,
execute_code_step,
execute_tool_step,
run_tool,
create_argument_parser,
collect_custom_args
)
from cmdforge.tool import Tool, ToolArgument, PromptStep, CodeStep
from cmdforge.providers import ProviderResult
from cmdforge.tool import Tool, ToolArgument, PromptStep, CodeStep, ToolStep, McpStep
from cmdforge.providers import Provider, ProviderResult
class TestSubstituteVariables:
@@ -227,6 +229,23 @@ class TestExecutePromptStep:
assert success is False
assert output == ""
@patch('cmdforge.runner.call_provider')
def test_step_can_deny_fallback_independently_of_caller(self, mock_call):
mock_call.return_value = ProviderResult(
text="private", success=True,
)
step = PromptStep(
prompt="Private", provider="local", output_var="out",
plain_text=True, fallback_policy="deny",
)
output, success = execute_prompt_step(step, {"input": ""})
assert success is True
assert output == "private"
policy = mock_call.call_args.kwargs["execution_policy"]
assert policy.allow_fallback is False
@patch('cmdforge.runner.mock_provider')
def test_mock_provider_used(self, mock_mock):
mock_mock.return_value = ProviderResult(text="mock response", success=True)
@@ -986,6 +1005,58 @@ class TestCreateArgumentParser:
assert args.count == "10"
def test_parser_enforces_typed_boolean_integer_and_enum_arguments(self):
tool = Tool(
name="typed",
arguments=[
ToolArgument(
flag="--project", variable="project",
type="boolean", default=False,
),
ToolArgument(
flag="--count", variable="count",
type="integer", required=True,
),
ToolArgument(
flag="--mode", variable="mode",
enum=["safe", "fast"], default="safe",
),
],
)
parser = create_argument_parser(tool)
args = parser.parse_args(["--project", "--count", "3", "--mode", "fast"])
assert args.project is True
assert args.count == 3
assert args.mode == "fast"
def test_parser_reuses_matching_legacy_boolean_universal_flag(self):
tool = Tool(
name="legacy",
arguments=[
ToolArgument(
flag="--verbose", variable="verbose", type="bool"
)
],
)
args = create_argument_parser(tool).parse_args(["--verbose"])
assert args.verbose is True
assert collect_custom_args(tool, args) == {"verbose": True}
def test_parser_rejects_incompatible_universal_flag_collision(self):
tool = Tool(
name="conflict",
arguments=[
ToolArgument(flag="--provider", variable="destination")
],
)
with pytest.raises(ValueError, match="universal run options"):
create_argument_parser(tool)
def test_parser_input_output_flags(self):
tool = Tool(name="test")
parser = create_argument_parser(tool)
@@ -1010,3 +1081,178 @@ class TestCreateArgumentParser:
"max_size": "50",
"format": "json",
}
class TestProviderExecutionPolicies:
def test_denied_tool_is_not_resolved(self, monkeypatch):
resolver = MagicMock()
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", tools=[]),
)
output, success = execute_tool_step(
ToolStep(tool="forbidden", output_var="out", provider="locked"),
{"input": ""},
)
assert (output, success) == ("", False)
resolver.assert_not_called()
def test_unknown_provider_is_rejected_before_resolution(self, monkeypatch):
resolver = MagicMock()
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
monkeypatch.setattr("cmdforge.providers.get_provider", lambda name: None)
_, success = execute_tool_step(
ToolStep(tool="anything", output_var="out", provider="missing"),
{"input": ""},
)
assert success is False
resolver.assert_not_called()
def test_provider_and_delegated_allowlists_both_apply(self, monkeypatch):
leaf = Tool(name="safe-lint", output="completed")
resolver = MagicMock(return_value=SimpleNamespace(tool=leaf))
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", tools=["safe-*"]),
)
output, success = execute_tool_step(
ToolStep(tool="safe-lint", output_var="out", provider="restricted"),
{"input": ""},
agent_tool_policies=[["*-lint"]],
)
assert (output, success) == ("completed", True)
resolver.assert_called_once_with("safe-lint")
def test_delegated_allowlist_can_further_restrict_provider(self, monkeypatch):
resolver = MagicMock()
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", tools=["*"]),
)
_, success = execute_tool_step(
ToolStep(tool="unsafe-delete", output_var="out", provider="broad"),
{"input": ""},
agent_tool_policies=[["safe-*"]],
)
assert success is False
resolver.assert_not_called()
def test_mcp_server_policy_denies_before_manager_creation(self, monkeypatch):
manager = MagicMock()
monkeypatch.setattr("cmdforge.runner.McpClientManager", manager)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", mcp_servers=[]),
)
tool = Tool(
name="mcp-tool",
steps=[McpStep(server="database", tool="query", output_var="result")],
output="{result}",
)
output, exit_code = run_tool(
tool, "", {}, provider_override="restricted"
)
assert (output, exit_code) == ("", 4)
manager.assert_not_called()
def test_mcp_server_policy_allows_matching_server(self, monkeypatch):
class FakeManager:
def call_tool(self, server, tool, arguments, result_mode="auto"):
return "allowed"
monkeypatch.setattr("cmdforge.runner.McpClientManager", FakeManager)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", mcp_servers=["data-*"]),
)
tool = Tool(
name="mcp-tool",
steps=[McpStep(server="data-local", tool="query", output_var="result")],
output="{result}",
)
assert run_tool(tool, "", {}, provider_override="restricted") == (
"allowed", 0
)
def test_agent_context_propagates_through_nested_tools(self, monkeypatch):
captured = {}
leaf = Tool(
name="leaf",
steps=[PromptStep("prompt", "original", "answer")],
output="{answer}",
)
middle = Tool(
name="middle",
steps=[ToolStep(tool="leaf", output_var="child")],
output="{child}",
)
root = Tool(
name="root",
steps=[
ToolStep(
tool="middle",
output_var="delegated",
provider="delegate",
profile="architect",
skills=["python"],
tools=["leaf"],
)
],
output="{delegated}",
)
def resolve(name):
return SimpleNamespace(tool={"middle": middle, "leaf": leaf}[name])
def execute_prompt(step, *args, **kwargs):
captured["step"] = step
captured["provider_override"] = args[1]
return "done", True
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolve)
monkeypatch.setattr("cmdforge.runner.execute_prompt_step", execute_prompt)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", tools=["middle", "leaf"]),
)
assert run_tool(root, "", {}) == ("done", 0)
assert captured["step"].profile == "architect"
assert captured["step"].skills == ["python"]
assert captured["provider_override"] == "delegate"
def test_descendant_cannot_broaden_ancestor_tool_policy(self, monkeypatch):
resolver = MagicMock()
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
monkeypatch.setattr(
"cmdforge.providers.get_provider",
lambda name: Provider(name, "cmd", tools=["*"]),
)
_, success = execute_tool_step(
ToolStep(
tool="dangerous",
output_var="out",
provider="delegate",
tools=["*"],
),
{"input": ""},
agent_tool_policies=[["safe-*"]],
)
assert success is False
resolver.assert_not_called()
+186
View File
@@ -0,0 +1,186 @@
"""Tests for conservative ToolStep JSON Schema compatibility."""
from unittest.mock import patch
from cmdforge.preflight import (
PreflightReport, _check_toolstep_compatibility, analyze_tool,
)
from cmdforge.schema_compat import compare_json_schemas
from cmdforge.tool import McpStep, PromptStep, Tool, ToolStep
class TestSchemaCompatibility:
def test_type_mismatch_is_incompatible(self):
result = compare_json_schemas(
{"type": "string"}, {"type": "integer"}
)
assert result.state == "incompatible"
def test_integer_is_accepted_by_number(self):
result = compare_json_schemas(
{"type": "integer"}, {"type": "number"}
)
assert result.state == "compatible"
def test_missing_required_property_is_incompatible(self):
result = compare_json_schemas(
{"type": "object", "properties": {}},
{
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
},
)
assert result.state == "incompatible"
assert "required" in result.detail
def test_matching_object_property_is_compatible(self):
schema = {
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
}
assert compare_json_schemas(schema, schema).state == "compatible"
def test_literal_enum_value_is_checked(self):
assert compare_json_schemas(
{"const": "fast"}, {"type": "string", "enum": ["fast", "slow"]}
).state == "compatible"
assert compare_json_schemas(
{"const": "invalid"}, {"type": "string", "enum": ["fast"]}
).state == "incompatible"
def test_unconstrained_producer_does_not_satisfy_consumer_enum(self):
result = compare_json_schemas(
{"type": "string"}, {"type": "string", "enum": ["fast"]}
)
assert result.state == "incompatible"
def test_closed_consumer_rejects_open_producer_object(self):
result = compare_json_schemas(
{"type": "object", "properties": {}},
{"type": "object", "properties": {}, "additionalProperties": False},
)
assert result.state == "incompatible"
def test_open_producer_cannot_guarantee_typed_optional_consumer_field(self):
result = compare_json_schemas(
{"type": "object", "properties": {}},
{
"type": "object",
"properties": {"limit": {"type": "integer"}},
},
)
assert result.state == "incompatible"
assert "limit" in result.detail
def test_unproven_pattern_is_unknown(self):
result = compare_json_schemas(
{"type": "string"}, {"type": "string", "pattern": "^[a-z]+$"}
)
assert result.state == "unknown"
class TestToolStepCompatibility:
@staticmethod
def child(input_schema):
return Tool(name="child", input_schema=input_schema)
def analyze(self, parent, child):
report = PreflightReport()
with patch("cmdforge.tool.load_tool", return_value=child):
_check_toolstep_compatibility(parent, report)
return report
def test_compatible_stdin_contract(self):
parent = Tool(
name="parent",
steps=[ToolStep(tool="child", output_var="result")],
)
child = self.child({
"type": "object",
"properties": {"input": {"type": "string"}},
"required": ["input"],
})
report = self.analyze(parent, child)
assert report.compatibility[0]["state"] == "compatible"
def test_upstream_serialization_mismatch_is_reported(self):
producer = PromptStep(
prompt="produce", provider="mock", output_var="value",
output_schema={"type": "string"},
)
parent = Tool(
name="parent",
steps=[
producer,
ToolStep(
tool="child", output_var="result", input_template="{value}"
),
],
)
child = self.child({
"type": "object",
"properties": {"input": {"type": "integer"}},
"required": ["input"],
})
report = self.analyze(parent, child)
finding = report.compatibility[0]
assert finding["state"] == "incompatible"
assert finding["upstream_schema"] == {"type": "string"}
assert any("incompatible" in warning for warning in report.warnings)
def test_missing_required_argument_is_incompatible(self):
parent = Tool(
name="parent",
steps=[ToolStep(tool="child", output_var="result")],
)
child = self.child({
"type": "object",
"properties": {
"input": {"type": "string"},
"limit": {"type": "integer"},
},
"required": ["input", "limit"],
})
report = self.analyze(parent, child)
assert report.compatibility[0]["state"] == "incompatible"
assert "limit" in report.compatibility[0]["detail"]
def test_explicit_argument_reflects_string_transport(self):
parent = Tool(
name="parent",
steps=[ToolStep(
tool="child", output_var="result", args={"limit": "{input}"}
)],
)
child = self.child({
"type": "object",
"properties": {
"input": {"type": "string"},
"limit": {"type": "integer"},
},
"required": ["input", "limit"],
})
assert self.analyze(parent, child).compatibility[0]["state"] == "incompatible"
def test_mcp_steps_are_ignored(self):
parent = Tool(
name="parent",
steps=[McpStep(server="remote", tool="same-name", output_var="result")],
)
report = PreflightReport()
with patch("cmdforge.tool.load_tool") as loader:
_check_toolstep_compatibility(parent, report)
loader.assert_not_called()
assert report.compatibility == []
def test_server_mode_does_not_resolve_local_tools(self):
parent = Tool(
name="published",
steps=[ToolStep(tool="child", output_var="result")],
)
with patch("cmdforge.tool.load_tool") as loader:
report = analyze_tool(parent, check_local_dependencies=False)
loader.assert_not_called()
assert report.compatibility == []
+11
View File
@@ -0,0 +1,11 @@
import stat
from cmdforge.signing import initialize_signing_key, load_signing_key
def test_signing_key_is_persistent_and_private(tmp_path):
path = tmp_path / "release-key.json"
created = initialize_signing_key(path)
assert load_signing_key(path) == created
assert stat.S_IMODE(path.stat().st_mode) == 0o600
assert initialize_signing_key(path) == created
+438
View File
@@ -0,0 +1,438 @@
"""Tests for provider-attached skills system."""
import textwrap
from types import SimpleNamespace
import pytest
from cmdforge.skills import (
Skill,
SkillSet,
_parse_frontmatter,
_validate_provider_name,
_validate_skill_name,
load_provider_skills,
inject_skills,
)
from cmdforge.providers import ProviderResult
from cmdforge.runner import execute_prompt_step
from cmdforge.tool import PromptStep
def write_skill(root, provider, name, content="Skill instructions.", description="A useful skill"):
directory = root / provider / "skills" / name
directory.mkdir(parents=True)
(directory / "SKILL.md").write_text(
f"---\nname: {name}\ndescription: {description}\n---\n{content}",
encoding="utf-8",
)
return directory
class TestValidation:
def test_valid_provider_name(self):
_validate_provider_name("claude")
_validate_provider_name("openrouter")
_validate_provider_name("my-provider_1")
def test_invalid_provider_name_slash(self):
with pytest.raises(ValueError):
_validate_provider_name("claude/evil")
def test_invalid_provider_name_dots(self):
with pytest.raises(ValueError):
_validate_provider_name("../../etc")
def test_invalid_provider_name_empty(self):
with pytest.raises(ValueError):
_validate_provider_name("")
def test_valid_skill_name(self):
_validate_skill_name("python-utils", "claude")
@pytest.mark.parametrize("name", ["Python", "python_utils", "-python", "python-", "python--utils"])
def test_invalid_standard_skill_names(self, name):
with pytest.raises(ValueError, match="Invalid skill name"):
_validate_skill_name(name, "claude")
def test_skill_name_max_length(self):
_validate_skill_name("a" * 64, "claude")
with pytest.raises(ValueError, match="1-64"):
_validate_skill_name("a" * 65, "claude")
def test_invalid_skill_name_slash(self):
with pytest.raises(ValueError):
_validate_skill_name("a/b", "claude")
def test_invalid_skill_name_dotdot(self):
with pytest.raises(ValueError):
_validate_skill_name("..", "claude")
class TestFrontmatter:
def test_basic(self):
text = textwrap.dedent("""\
---
name: python-utils
description: Python helpers
---
Use list comprehensions.
""")
meta, content = _parse_frontmatter(text)
assert meta["name"] == "python-utils"
assert meta["description"] == "Python helpers"
assert content == "Use list comprehensions."
def test_no_frontmatter(self):
with pytest.raises(ValueError, match="must begin"):
_parse_frontmatter("Just raw content")
def test_unterminated_frontmatter(self):
with pytest.raises(ValueError, match="not terminated"):
_parse_frontmatter("---\nname: test\ndescription: Test")
def test_invalid_yaml_frontmatter(self):
text = "---\n: invalid yaml\n---\ncontent"
with pytest.raises(ValueError, match="Invalid YAML"):
_parse_frontmatter(text)
def test_non_dict_frontmatter(self):
text = "---\n- list item\n---\ncontent"
with pytest.raises(ValueError, match="mapping"):
_parse_frontmatter(text)
class TestLoadProviderSkills:
@pytest.fixture
def temp_providers_dir(self, tmp_path, monkeypatch):
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
return tmp_path
def test_no_directory(self, temp_providers_dir):
skills = load_provider_skills("nonexistent")
assert isinstance(skills, SkillSet)
assert skills.skills == []
def test_empty_skills_dir(self, temp_providers_dir):
d = temp_providers_dir / "claude" / "skills"
d.mkdir(parents=True)
skills = load_provider_skills("claude")
assert skills.skills == []
def test_loads_skill(self, temp_providers_dir):
d = temp_providers_dir / "claude" / "skills" / "python"
d.mkdir(parents=True)
(d / "SKILL.md").write_text(textwrap.dedent("""\
---
name: python
description: Python coding patterns
---
Use type hints always.
"""))
skillset = load_provider_skills("claude")
assert len(skillset.skills) == 1
skill = skillset.skills[0]
assert skill.name == "python"
assert skill.description == "Python coding patterns"
assert "type hints" in skill.content
def test_skips_non_md_files(self, temp_providers_dir):
d = write_skill(temp_providers_dir, "claude", "python")
(d / "readme.txt").write_text("ignore me")
skillset = load_provider_skills("claude")
assert len(skillset.skills) == 1
def test_sorted_by_name(self, temp_providers_dir):
for name in ("z-skill", "a-skill", "m-skill"):
write_skill(temp_providers_dir, "claude", name)
skillset = load_provider_skills("claude")
names = [s.name for s in skillset.skills]
assert names == ["a-skill", "m-skill", "z-skill"]
def test_frontmatter_name_must_match_directory(self, temp_providers_dir):
d = temp_providers_dir / "claude" / "skills" / "python"
d.mkdir(parents=True)
(d / "SKILL.md").write_text(
"---\nname: wrong-name\ndescription: Wrong\n---\ncontent"
)
with pytest.raises(ValueError, match="does not match"):
load_provider_skills("claude")
def test_invalid_provider_name_traversal(self, temp_providers_dir):
with pytest.raises(ValueError):
load_provider_skills("../secrets")
def test_skips_non_directories(self, temp_providers_dir):
d = temp_providers_dir / "claude" / "skills"
d.mkdir(parents=True)
(d / "file.txt").write_text("not a skill dir")
skillset = load_provider_skills("claude")
assert skillset.skills == []
def test_no_skill_md(self, temp_providers_dir):
d = temp_providers_dir / "claude" / "skills" / "empty-dir"
d.mkdir(parents=True)
skillset = load_provider_skills("claude")
assert skillset.skills == []
@pytest.mark.parametrize(
("frontmatter", "message"),
[
("description: Useful", "required 'name'"),
("name: python", "required 'description'"),
("name: python\ndescription: ''", "non-empty"),
],
)
def test_required_metadata(self, temp_providers_dir, frontmatter, message):
directory = temp_providers_dir / "claude" / "skills" / "python"
directory.mkdir(parents=True)
(directory / "SKILL.md").write_text(
f"---\n{frontmatter}\n---\ncontent"
)
with pytest.raises(ValueError, match=message):
load_provider_skills("claude")
def test_description_max_length(self, temp_providers_dir):
write_skill(
temp_providers_dir,
"claude",
"python",
description="x" * 1025,
)
with pytest.raises(ValueError, match="1024"):
load_provider_skills("claude")
def test_rejects_symlinked_provider_directory(self, temp_providers_dir, tmp_path):
outside = tmp_path / "outside-provider"
outside.mkdir()
(temp_providers_dir / "claude").symlink_to(outside, target_is_directory=True)
with pytest.raises(ValueError, match="symlinked provider"):
load_provider_skills("claude")
def test_rejects_symlinked_skill_directory(self, temp_providers_dir, tmp_path):
skills_dir = temp_providers_dir / "claude" / "skills"
skills_dir.mkdir(parents=True)
outside = tmp_path / "outside-skill"
outside.mkdir()
(skills_dir / "python").symlink_to(outside, target_is_directory=True)
with pytest.raises(ValueError, match="symlinked skill directory"):
load_provider_skills("claude")
def test_rejects_symlinked_skill_file(self, temp_providers_dir, tmp_path):
directory = temp_providers_dir / "claude" / "skills" / "python"
directory.mkdir(parents=True)
outside = tmp_path / "outside-skill.md"
outside.write_text("---\nname: python\ndescription: Test\n---\ncontent")
(directory / "SKILL.md").symlink_to(outside)
with pytest.raises(ValueError, match="symlinked SKILL.md"):
load_provider_skills("claude")
class TestSkillSetSelection:
def test_get_all_when_none(self):
skills = SkillSet("test", [
Skill("a", "a-dir", "desc A", "content A"),
Skill("b", "b-dir", "desc B", "content B"),
])
result = skills.get(None)
assert len(result) == 2
def test_get_specific(self):
skills = SkillSet("test", [
Skill("a", "a-dir", "desc A", "content A"),
Skill("b", "b-dir", "desc B", "content B"),
])
result = skills.get(["a"])
assert len(result) == 1
assert result[0].name == "a"
def test_get_empty_list(self):
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
assert skills.get([]) == []
def test_warn_on_missing(self, capsys):
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
result = skills.get(["missing"], on_missing="warn")
assert result == []
captured = capsys.readouterr()
assert "not found" in captured.err
def test_error_on_missing(self):
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
with pytest.raises(KeyError, match="not found"):
skills.get(["missing"], on_missing="error")
def test_ignore_on_missing(self):
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
result = skills.get(["missing"], on_missing="ignore")
assert result == []
def test_invalid_missing_policy(self):
skills = SkillSet("test")
with pytest.raises(ValueError, match="missing-skill policy"):
skills.get([], on_missing="silent")
class TestInjectSkills:
@pytest.fixture
def temp_providers_dir(self, tmp_path, monkeypatch):
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
return tmp_path
def test_no_skills_no_injection(self, temp_providers_dir):
d = temp_providers_dir / "empty" / "skills"
d.mkdir(parents=True)
result = inject_skills("Hello", "empty")
assert result == "Hello"
def test_injects_skills_before_prompt(self, temp_providers_dir):
write_skill(temp_providers_dir, "claude", "python", "Use type hints.")
result = inject_skills("Do X", "claude")
assert result.startswith("## Skill: python")
assert "Do X" in result
assert result.index("## Skill") < result.index("Do X")
def test_specific_skill_names(self, temp_providers_dir):
for name in ("python", "git", "testing"):
write_skill(temp_providers_dir, "claude", name, f"{name} content")
result = inject_skills("Prompt", "claude", ["python"])
assert "python content" in result
assert "git content" not in result
assert "testing content" not in result
def test_invalid_provider_name(self, temp_providers_dir):
with pytest.raises(ValueError, match="Invalid provider"):
inject_skills("Prompt", "../../etc")
class TestPromptStepSkills:
def test_to_dict_excludes_none(self):
step = PromptStep(
prompt="Test",
provider="claude",
output_var="out",
)
d = step.to_dict()
assert "skills" not in d
def test_to_dict_includes_skills(self):
step = PromptStep(
prompt="Test",
provider="claude",
output_var="out",
skills=["python", "git"],
)
d = step.to_dict()
assert d["skills"] == ["python", "git"]
def test_roundtrip_with_skills(self):
step = PromptStep(
prompt="Test",
provider="claude",
output_var="out",
skills=["python"],
)
restored = PromptStep.from_dict(step.to_dict())
assert restored.skills == ["python"]
def test_roundtrip_without_skills(self):
step = PromptStep(
prompt="Test",
provider="claude",
output_var="out",
)
restored = PromptStep.from_dict(step.to_dict())
assert restored.skills is None
def test_roundtrip_with_explicit_empty_skills(self):
step = PromptStep("Test", "claude", "out", skills=[])
serialized = step.to_dict()
assert serialized["skills"] == []
assert PromptStep.from_dict(serialized).skills == []
@pytest.mark.parametrize("skills", ["python", [1], ["*", "python"], ["python", "python"]])
def test_rejects_invalid_skill_lists(self, skills):
with pytest.raises(ValueError):
PromptStep("Test", "claude", "out", skills=skills)
def test_rejects_invalid_skill_identifier(self):
with pytest.raises(ValueError, match="Invalid skill name"):
PromptStep("Test", "claude", "out", skills=["Python_Utils"])
class TestRunnerSkillIntegration:
@pytest.fixture
def configured_skill(self, tmp_path, monkeypatch):
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
write_skill(tmp_path, "override", "python", "SKILL INSTRUCTIONS")
return tmp_path
def test_plain_text_uses_effective_provider_and_context_order(
self, configured_skill, monkeypatch
):
captured = {}
def fake_call(provider, prompt, max_tokens=None, **kwargs):
captured.update(provider=provider, prompt=prompt)
return ProviderResult(text="done", success=True)
monkeypatch.setattr("cmdforge.runner.call_provider", fake_call)
monkeypatch.setattr(
"cmdforge.runner.load_profile",
lambda name: SimpleNamespace(system_prompt="PROFILE INSTRUCTIONS"),
)
step = PromptStep(
prompt="USER PROMPT",
provider="original",
output_var="out",
profile="reviewer",
plain_text=True,
skills=["python"],
)
output, success = execute_prompt_step(
step, {}, provider_override="override"
)
assert success is True
assert output == "done"
assert captured["provider"] == "override"
prompt = captured["prompt"]
assert prompt.index("PROFILE INSTRUCTIONS") < prompt.index("## Skill: python")
assert prompt.index("## Skill: python") < prompt.index("USER PROMPT")
def test_structured_path_preserves_context_order(
self, configured_skill, monkeypatch
):
captured = {}
def fake_call(provider, prompt, max_tokens=None, **kwargs):
captured.update(provider=provider, prompt=prompt)
return ProviderResult(text='{"output": "done"}', success=True)
monkeypatch.setattr("cmdforge.runner.call_provider", fake_call)
monkeypatch.setattr(
"cmdforge.runner.load_profile",
lambda name: SimpleNamespace(system_prompt="PROFILE INSTRUCTIONS"),
)
step = PromptStep(
prompt="USER PROMPT",
provider="override",
output_var="out",
profile="reviewer",
skills=["python"],
)
output, success = execute_prompt_step(step, {})
assert success is True
assert output == {"output": "done"}
prompt = captured["prompt"]
assert prompt.index("PROFILE INSTRUCTIONS") < prompt.index("## Skill: python")
assert prompt.index("## Skill: python") < prompt.index("USER PROMPT")
+195 -1
View File
@@ -8,7 +8,7 @@ import pytest
import yaml
from cmdforge.tool import (
Tool, ToolArgument, PromptStep, CodeStep,
Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
validate_tool_name, load_tool, save_tool, delete_tool,
list_tools, tool_exists, create_wrapper_script,
DEFAULT_CATEGORIES
@@ -80,6 +80,50 @@ class TestToolArgument:
assert restored.default == original.default
assert restored.description == original.description
def test_typed_metadata_roundtrip(self):
original = ToolArgument(
flag="--limit",
variable="limit",
default=10,
description="Maximum results",
type="integer",
enum=[10, 25, 50],
required=True,
)
assert ToolArgument.from_dict(original.to_dict()) == original
@pytest.mark.parametrize(
"values",
[
{"type": "object"},
{"enum": []},
{"enum": "fast"},
{"required": "yes"},
],
)
def test_rejects_invalid_typed_metadata(self, values):
with pytest.raises(ValueError):
ToolArgument(flag="--value", variable="value", **values)
@pytest.mark.parametrize(
("legacy_type", "canonical_type"),
[
("str", "string"),
("int", "integer"),
("float", "number"),
("bool", "boolean"),
],
)
def test_legacy_scalar_type_aliases_are_normalized(
self, legacy_type, canonical_type
):
argument = ToolArgument(
flag="--value", variable="value", type=legacy_type
)
assert argument.type == canonical_type
assert argument.to_dict().get("type", "string") == canonical_type
class TestPromptStep:
"""Tests for PromptStep dataclass."""
@@ -554,3 +598,153 @@ class TestDefaultCategories:
assert "Developer" in DEFAULT_CATEGORIES
assert "Data" in DEFAULT_CATEGORIES
assert "Other" in DEFAULT_CATEGORIES
class TestToolContracts:
def test_config_must_be_mapping(self):
with pytest.raises(ValueError, match="YAML mapping"):
Tool.from_dict(["not", "a", "mapping"])
def test_legacy_source_string_is_normalized(self):
tool = Tool.from_dict({"name": "legacy-source", "source": "old/tool"})
assert tool.source is not None
assert tool.source.type == "imported"
assert tool.source.original_tool == "old/tool"
def test_contracts_roundtrip(self):
tool = Tool(
name="contract-tool",
input_schema={"type": "string"},
output_schema={"type": "array", "items": {"type": "integer"}},
)
restored = Tool.from_dict(tool.to_dict())
assert restored.input_schema == {"type": "string"}
assert restored.output_schema["type"] == "array"
def test_empty_contracts_are_preserved(self):
tool = Tool(name="open-contract", input_schema={}, output_schema={})
serialized = tool.to_dict()
assert serialized["input_schema"] == {}
assert serialized["output_schema"] == {}
@pytest.mark.parametrize(
("field", "schema"),
[
("input_schema", "string"),
("output_schema", {"type": "not-a-json-schema-type"}),
("output_schema", {"required": "name"}),
],
)
def test_invalid_contract_is_rejected(self, field, schema):
with pytest.raises(ValueError, match=field):
Tool(name="bad-contract", **{field: schema})
class TestAgentContext:
def test_toolstep_profile_roundtrip(self):
step = ToolStep(tool="my-tool", output_var="out", profile="architect")
d = step.to_dict()
assert d["profile"] == "architect"
restored = ToolStep.from_dict(d)
assert restored.profile == "architect"
def test_toolstep_skills_roundtrip(self):
step = ToolStep(tool="my-tool", output_var="out", skills=["python"])
d = step.to_dict()
assert d["skills"] == ["python"]
restored = ToolStep.from_dict(d)
assert restored.skills == ["python"]
def test_toolstep_empty_skills_serialized(self):
step = ToolStep(tool="my-tool", output_var="out", skills=[])
d = step.to_dict()
assert d["skills"] == []
restored = ToolStep.from_dict(d)
assert restored.skills == []
def test_toolstep_no_skills_not_serialized(self):
step = ToolStep(tool="my-tool", output_var="out")
d = step.to_dict()
assert "skills" not in d
assert "profile" not in d
def test_toolstep_tools_roundtrip(self):
step = ToolStep(
tool="my-tool", output_var="out", tools=["safe-*", "official/lint"]
)
assert ToolStep.from_dict(step.to_dict()).tools == [
"safe-*", "official/lint"
]
def test_toolstep_empty_tools_roundtrip(self):
step = ToolStep(tool="my-tool", output_var="out", tools=[])
serialized = step.to_dict()
assert serialized["tools"] == []
assert ToolStep.from_dict(serialized).tools == []
@pytest.mark.parametrize(
"data",
[
{"skills": "python"},
{"skills": ["*", "python"]},
{"skills": ["Python"]},
{"tools": "safe-tool"},
{"tools": [""]},
{"profile": ""},
],
)
def test_toolstep_rejects_invalid_agent_context(self, data):
with pytest.raises(ValueError):
ToolStep(tool="my-tool", output_var="out", **data)
def test_promptstep_skills_roundtrip(self):
step = PromptStep(
prompt="Test", provider="claude", output_var="out",
skills=["python"],
)
restored = PromptStep.from_dict(step.to_dict())
assert restored.skills == ["python"]
def test_promptstep_deny_fallback_roundtrip(self):
step = PromptStep(
prompt="Private", provider="local", output_var="out",
fallback_policy="deny",
)
restored = PromptStep.from_dict(step.to_dict())
assert restored.fallback_policy == "deny"
def test_promptstep_rejects_invalid_fallback_policy(self):
with pytest.raises(ValueError, match="fallback_policy"):
PromptStep(
prompt="Test", provider="local", output_var="out",
fallback_policy="sometimes",
)
class TestDeprecationFields:
def test_round_trip_and_warning(self):
from cmdforge.tool import deprecation_warning
tool = Tool(
name="old", deprecated=True, deprecated_message="Moved.",
replacement="official/new",
)
restored = Tool.from_dict(tool.to_dict())
assert restored.deprecated is True
assert restored.replacement == "official/new"
assert "official/new" in deprecation_warning(restored)
@pytest.mark.parametrize(
"values",
[
{"deprecated": "false"},
{"deprecated_message": 123},
{"replacement": "../../escape"},
{"replacement": "owner/name/extra"},
],
)
def test_invalid_deprecation_metadata_is_rejected(self, values):
with pytest.raises(ValueError):
Tool(name="old", **values)
+67
View File
@@ -0,0 +1,67 @@
import json
import stat
from cmdforge.usage import (
build_composite_tool,
clear_usage,
get_suggestions,
is_enabled,
record_invocation,
set_enabled,
)
class Stream:
def __init__(self, inode=None):
self.inode = inode
def configure_paths(tmp_path, monkeypatch):
monkeypatch.setattr("cmdforge.usage.CONFIG_DIR", tmp_path)
monkeypatch.setattr("cmdforge.usage.USAGE_FILE", tmp_path / "usage.json")
monkeypatch.setattr("cmdforge.usage.USAGE_LOCK_FILE", tmp_path / ".usage.lock")
def test_tracking_is_disabled_by_default(tmp_path, monkeypatch):
configure_paths(tmp_path, monkeypatch)
assert not is_enabled()
monkeypatch.setattr("cmdforge.usage._pipe_inode", lambda stream: stream.inode)
record_invocation("one", Stream(), Stream())
assert not (tmp_path / "usage.json").exists()
def test_detects_only_pipe_link_and_stores_no_content(tmp_path, monkeypatch):
configure_paths(tmp_path, monkeypatch)
set_enabled(True)
monkeypatch.setattr("cmdforge.usage._pipe_inode", lambda stream: stream.inode)
for inode in range(44, 47):
record_invocation("tool-a", Stream(), Stream(inode))
record_invocation("tool-b", Stream(inode), Stream())
assert get_suggestions() == [{
"tools": ["tool-a", "tool-b"],
"count": 3,
"last_seen": get_suggestions()[0]["last_seen"],
}]
data = json.loads((tmp_path / "usage.json").read_text())
serialized = json.dumps(data)
for forbidden in ("input_text", "output", "arguments", "cwd", "environment"):
assert forbidden not in serialized
assert stat.S_IMODE((tmp_path / "usage.json").stat().st_mode) == 0o600
def test_disable_and_clear_preserve_consent_state(tmp_path, monkeypatch):
configure_paths(tmp_path, monkeypatch)
set_enabled(True)
clear_usage()
assert is_enabled()
set_enabled(False)
assert not is_enabled()
def test_build_composite_tool_preserves_pipeline_order():
tool = build_composite_tool("combined", ["tool-a", "tool-b"])
assert tool.dependencies == ["tool-a", "tool-b"]
assert tool.steps[0].input_template == "{input}"
assert tool.steps[1].input_template == "{pipeline_1}"
assert tool.output == "{pipeline_2}"
+125
View File
@@ -0,0 +1,125 @@
"""Regression tests for user-facing documentation embedded in the web app."""
import re
from cmdforge.web.docs_content import DOCS, get_doc, get_toc
def test_getting_started_documents_agent_first_workflow():
content = get_doc("getting-started")["content"]
expected_commands = (
"cmdforge list --json --filter",
"cmdforge registry search",
"cmdforge run-once",
"cmdforge create classify-docs --project",
"forge-tool --name classify-docs --project",
"cmdforge mcp configure codex --dry-run",
"cmdforge mcp configure claude-code --dry-run",
)
for command in expected_commands:
assert command in content
def test_first_tool_explains_project_ownership():
content = get_doc("first-tool")["content"]
assert "cmdforge create explain --project" in content
assert "./.cmdforge/" in content
def test_every_document_appears_once_in_the_book_navigation():
slugs = []
for chapter in get_toc():
slugs.append(chapter.slug)
slugs.extend(page.slug for page in chapter.children)
assert len(slugs) == len(set(slugs))
assert set(slugs) == set(DOCS)
def test_internal_documentation_links_resolve():
links = set()
for page in DOCS.values():
links.update(re.findall(r'href="/docs/([^"#?]+)', page["content"]))
assert links <= set(DOCS)
def test_modern_architecture_has_dedicated_chapters():
expected = {
"mcp-overview",
"mcp-client",
"mcp-server",
"agent-integration",
"contracts-quality",
"provider-policy",
"skills-delegation",
"optimization-usage",
"forge-tool",
"trust-publishing",
}
assert expected <= set(DOCS)
def test_mcp_documentation_covers_both_directions_and_transports():
content = " ".join(
get_doc(slug)["content"]
for slug in ("mcp-overview", "mcp-client", "mcp-server")
)
for term in (
"type: mcp",
"transport: stdio",
"transport: streamable-http",
"result_mode",
"cmdforge mcp connect",
"cmdforge mcp serve",
"expose",
"deny",
):
assert term in content
def test_provider_documentation_covers_strict_execution_contract():
content = get_doc("providers")["content"] + get_doc("provider-policy")["content"]
for term in (
"subprocess",
"API",
"PTY",
"--no-fallback",
"--require-local",
"--require-capability",
"--data-classification",
"--require-model-identity",
"--require-model-digest",
"--result-envelope json",
):
assert term in content
def test_command_atlas_covers_every_top_level_cli_family():
content = get_doc("cli-reference")["content"]
commands = (
"list", "create", "edit", "delete", "test", "run", "run-once", "ui",
"refresh", "docs", "inspect", "optimize", "check", "providers", "registry",
"collections", "deps", "install", "lock", "verify", "add", "remove", "init",
"config", "settings", "system-deps", "mcp", "usage",
)
for command in commands:
assert f"cmdforge {command}" in content
def test_installation_matches_runtime_and_current_provider_onboarding():
installation = get_doc("installation")["content"]
provider_setup = get_doc("provider-setup")["content"]
assert "Python 3.10" in installation
assert "Python 3.8" not in installation
assert "cmdforge[mcp,pty]" in installation
assert "cmdforge providers discover --add" in installation
assert "cmdforge providers discover --add" in provider_setup
assert "--type api" in provider_setup
assert "pip install claude-cli" not in provider_setup