Compare commits
31
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1e698996b2 | ||
|
|
ca6135ac72 | ||
|
|
af3701b10b | ||
|
|
2e40659328 | ||
|
|
f8171142ee | ||
|
|
d464c79002 | ||
|
|
64fc7339ef | ||
|
|
9bedf70371 | ||
|
|
6a88fe2e0f | ||
|
|
c289e812fb | ||
|
|
06a1606370 | ||
|
|
23723eaddb | ||
|
|
56cb34276e | ||
|
|
04153a4bbd | ||
|
|
bf3686ae40 | ||
|
|
4d22d532a3 | ||
|
|
d954c1823a | ||
|
|
bda0c76d5f | ||
|
|
501e6cd589 | ||
|
|
d81286e322 | ||
|
|
ba29e4dc4e | ||
|
|
188e167c1c | ||
|
|
f6bd640865 | ||
|
|
0323a39afb | ||
|
|
23ac54f5c8 | ||
|
|
8366ef798b | ||
|
|
93ea80950e | ||
|
|
8d38bdd83a | ||
|
|
fe53721725 | ||
|
|
3525e2eeae | ||
|
|
dee48d8cea |
@@ -20,9 +20,14 @@ cf # Interactive tool picker
|
||||
## Architecture Quick Reference
|
||||
|
||||
- `cli/` - Routes all subcommands (list, create, run, test, providers, registry, collections, deps, install, etc.)
|
||||
- `tool.py` - Tool/step dataclasses, YAML loading, wrapper generation
|
||||
- `runner.py` - Step execution, variable substitution (`{input}`, `{varname}`)
|
||||
- `providers.py` - AI provider abstraction (subprocess, API, PTY types; auto-discovery; fallback chains)
|
||||
- `tool.py` - Tool/step dataclasses, including delegated `ToolStep` context and `McpStep`
|
||||
- `preflight.py` - Shared contract, dependency, secret-pattern, and registry-similarity analysis
|
||||
- `contract_testing.py` - Deterministic JSON Schema input generation and side-effect-safe conformance checks
|
||||
- `schema_compat.py` - Conservative producer-to-consumer JSON Schema compatibility analysis
|
||||
- `runner.py` - Step execution, variable substitution, nested authorization and delegation
|
||||
- `providers.py` - AI providers, auto-discovery, fallback chains, and tool/MCP allowlists
|
||||
- `skills.py` - Per-provider Agent Skills loading and validation
|
||||
- `mcp_client.py`, `mcp_server.py` - Stdio and Streamable HTTP MCP client/server support
|
||||
- `gui/` - PySide6 desktop GUI with page-based navigation
|
||||
- `web/` - Flask web UI and forum
|
||||
- `registry/` - Flask registry API (search, publish, moderation)
|
||||
@@ -33,11 +38,22 @@ cf # Interactive tool picker
|
||||
- CLI tools: lowercase with hyphens (e.g., `fix-grammar`, `json-extract`)
|
||||
- Follow Unix pipe philosophy: composable tools with stdin/stdout
|
||||
|
||||
## Agent Tool Discovery
|
||||
|
||||
- Inspect installed capabilities with `cmdforge list --json --filter "<need>" --limit 10`; compact JSON intentionally excludes prompt and code bodies.
|
||||
- Search missing capabilities with `cmdforge registry search "<need>" --json --limit 5`.
|
||||
- Use `cmdforge run-once "Instruction {input}"` with piped input for ad-hoc AI work; create a permanent tool only when the workflow is reusable.
|
||||
- Prefer CmdForge for local automation and composable workflows. Direct SDK/API integration is appropriate when the external API is a runtime dependency of the product itself.
|
||||
- Bootstrap supported coding hosts with `cmdforge mcp configure codex` or `cmdforge mcp configure claude-code`; preview changes with `--dry-run`. This updates only a marked policy block and never expands `server.expose`.
|
||||
- Create project-owned tools with `cmdforge create NAME --project` or `forge-tool --name NAME --project`; do not edit CmdForge's source repository to add a consumer project's tools.
|
||||
- For sensitive inputs, use `--no-fallback`, explicit provider constraints, and `--result-envelope json`; provenance is attached by CmdForge and must not be generated by the model.
|
||||
|
||||
## Testing
|
||||
|
||||
- Framework: `pytest` (see `pyproject.toml`)
|
||||
- Test files: `tests/test_*.py`
|
||||
- Use `mock` provider or `--dry-run` to avoid network calls
|
||||
- Use `cmdforge inspect <tool>` for deterministic local preflight checks
|
||||
- Integration tests require `@pytest.mark.integration` decorator
|
||||
|
||||
## Commit Guidelines
|
||||
|
||||
@@ -4,6 +4,14 @@ All notable changes to CmdForge will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.2.0] - 2026-07-21
|
||||
|
||||
First public Python package release. This release brings the current CmdForge
|
||||
application to PyPI, including provider routing and provenance, MCP client and
|
||||
server support, provider-attached skills, delegated tools, contracts and
|
||||
preflight analysis, regression evidence, quality scoring, registry integrity,
|
||||
project-owned tools, agent discovery commands, and local usage suggestions.
|
||||
|
||||
### Added
|
||||
|
||||
#### System Dependencies
|
||||
|
||||
@@ -40,11 +40,14 @@ python -m cmdforge.cli # Alternative CLI invocation
|
||||
### Core Modules (`src/cmdforge/`)
|
||||
|
||||
- **cli/**: CLI commands entry points (`cmdforge` command). Routes subcommands: list, create, edit, delete, test, run, ui, docs, check, refresh, providers, registry, collections, deps, install, lock, verify, add, remove, init, config, settings, system-deps
|
||||
- **tool.py**: Tool definition dataclasses (`Tool`, `ToolArgument`, `PromptStep`, `CodeStep`, `ToolStep`), YAML config loading/saving, wrapper script generation
|
||||
- **tool.py**: Tool definition dataclasses (`Tool`, `ToolArgument`, `PromptStep`, `CodeStep`, `ToolStep`, `McpStep`), YAML config loading/saving, wrapper script generation
|
||||
- **runner.py**: Execution engine. Runs tool steps sequentially, handles variable substitution (`{input}`, `{varname}`), executes Python code steps via `exec()`, handles nested tool calls with depth limit (MAX_TOOL_DEPTH=10)
|
||||
- **resolver.py**: Tool resolution. `resolve_tool()` searches: project manifest → local tools → owner/name → registry. Returns `ResolvedTool` with path info
|
||||
- **collection.py**: Collection management (`Collection` dataclass, `resolve_tool_references()`, `classify_tool_reference()`), local collection storage in `~/.cmdforge/collections/`
|
||||
- **providers.py**: Provider abstraction. Supports subprocess CLI tools, OpenAI-compatible HTTP APIs, and experimental PTY wrappers. Auto-discovers installed providers on first run. Config in `~/.cmdforge/providers.yaml` with versioned migration.
|
||||
- **providers.py**: Provider abstraction. Supports subprocess CLI tools, OpenAI-compatible HTTP APIs, experimental PTY wrappers, fallback chains, and tool/MCP-server allowlists. Auto-discovers installed providers on first run. Config in `~/.cmdforge/providers.yaml` with versioned migration.
|
||||
- **Provider execution contract**: Providers declare locality, typed capabilities, model identity, context/cost/latency hints, and an approved data classification. `ProviderExecutionPolicy` filters every fallback candidate; `ProviderResult` records runtime-owned provider/model provenance.
|
||||
- **skills.py**: Validated Agent Skills loader for per-provider `SKILL.md` context under `~/.cmdforge/providers/<name>/skills/`
|
||||
- **mcp_client.py / mcp_server.py**: Stdio and Streamable HTTP MCP integration, configuration, schema discovery, transport security, and exposure policy
|
||||
- **profiles.py**: AI persona profiles with system prompts, stored in `~/.cmdforge/profiles/`
|
||||
- **manifest.py**: Project manifest (`cmdforge.yaml`) for declaring tool dependencies with version constraints
|
||||
- **lockfile.py**: Lock file support for reproducible installs (`cmdforge.lock`)
|
||||
@@ -75,6 +78,49 @@ Tools are YAML configs with:
|
||||
- `arguments`: Custom flags with defaults (e.g., `--max` → `{max}`)
|
||||
- `steps`: Ordered list of `prompt`, `code`, or `tool` steps
|
||||
- `output`: Template for final output (e.g., `"{response}"`)
|
||||
- `input_schema`, `output_schema`: Optional tool-level JSON Schema contracts;
|
||||
schemas are validated when tools are loaded or created
|
||||
- `deprecated`, `deprecated_message`, `replacement`: Validated migration metadata;
|
||||
replacements may be local names or `owner/tool` registry references
|
||||
|
||||
Run `cmdforge inspect <tool> [--registry]` for the shared local preflight
|
||||
report. `cmdforge registry publish <path> --dry-run` runs local checks first
|
||||
and, when authenticated, the registry's publish-time checks without publishing.
|
||||
Normal publication uses the same checks as Stage 1, asks for confirmation in an
|
||||
interactive terminal, and then performs the Stage 2 mutation. The registry
|
||||
reruns deterministic conformance authoritatively and stores immutable audit and
|
||||
quality evidence for the published version.
|
||||
Inspect also shows inferred contract proposals and runs deterministic contract
|
||||
conformance when both tool schemas are explicit. Prompt outputs are synthesized
|
||||
from their schemas; code, nested-tool, and MCP steps are reported as unsupported
|
||||
instead of being executed implicitly.
|
||||
Use `cmdforge inspect <tool> --save-baseline` to explicitly promote passing
|
||||
evidence to `conformance.json`. Later inspections and local publish dry-runs
|
||||
rerun the stored synthetic inputs and report state, coverage, contract, and
|
||||
normalized output changes. ToolStep preflight also compares the values the
|
||||
runner supplies with the called tool's input contract.
|
||||
Quality headlines exclude categories with missing evidence and are accompanied
|
||||
by evidence coverage plus checked/not-tested category states.
|
||||
|
||||
### Agent-facing discovery
|
||||
|
||||
Use `cmdforge list --json --filter "<need>" --limit 10` for a compact local
|
||||
catalog containing names, descriptions, argument definitions, and contracts.
|
||||
Complete definitions (including prompt and code bodies) require an explicit
|
||||
`--full`. If no local tool fits, use
|
||||
`cmdforge registry search "<need>" --json --limit 5`. For one-off AI work,
|
||||
pipe data through `cmdforge run-once "Instruction {input}"`; repeated workflows
|
||||
should become normal versioned tools. Do not force CmdForge into application
|
||||
code where a direct SDK/API is itself the intended runtime dependency.
|
||||
Use `cmdforge mcp configure codex` or
|
||||
`cmdforge mcp configure claude-code --scope project` to register the stdio MCP
|
||||
server and install a marked project-policy block. Preview with `--dry-run`.
|
||||
This bootstrap never adds tools to the MCP `server.expose` allowlist.
|
||||
Project-owned tools belong under `./.cmdforge/`, created with
|
||||
`cmdforge create NAME --project` or `forge-tool --name NAME --project`. Do not
|
||||
edit CmdForge core merely to add another project's tools. Sensitive runs should
|
||||
use `--no-fallback` and `--result-envelope json`; provenance must come from the
|
||||
runtime rather than model-generated fields.
|
||||
|
||||
### Step Types
|
||||
|
||||
@@ -85,7 +131,8 @@ Tools are YAML configs with:
|
||||
- `max_tokens`: Max output tokens (provider-dependent, e.g., 4096 for haiku)
|
||||
- `plain_text`: Bypass structured output enforcement
|
||||
2. **Code Step**: Executes Python code via `exec()`, captures specified variables (comma-separated for multiple outputs)
|
||||
3. **Tool Step**: Calls another tool (meta-tools), supports `input_template`, `args` dict, and `provider` override. Dependencies resolved via `resolver.py`
|
||||
3. **Tool Step**: Calls another tool (meta-tools), supports `input_template`, `args`, and recursive agent context (`provider`, `profile`, `skills`, `tools`). Provider and delegated allowlists are both enforced before resolution.
|
||||
4. **MCP Step**: Calls a tool on a configured MCP server. Provider `mcp_servers` allowlists are enforced before connecting.
|
||||
|
||||
### Variable Flow
|
||||
|
||||
@@ -132,6 +179,8 @@ Provider fields:
|
||||
- `tags`: List of strings (e.g. `["free", "code", "local"]`)
|
||||
- `pty_config`: Dict with `prompt_pattern`, `response_pattern`, `exit_command` for pty providers
|
||||
- `install`: Optional structured install metadata dict
|
||||
- `tools`: Optional CmdForge tool allowlist; `null` allows all and `[]` denies all
|
||||
- `mcp_servers`: Optional MCP server allowlist; `null` allows all and `[]` denies all
|
||||
|
||||
The `mock` provider is built-in for testing without API calls. Use `--provider mock` or `--dry-run` flags when testing tools.
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Rob
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
+133
@@ -0,0 +1,133 @@
|
||||
# CmdForge
|
||||
|
||||
**Turn useful AI workflows into commands you own.**
|
||||
|
||||
CmdForge is a personal tool-building environment for the command line. A tool
|
||||
can combine prompts, Python, other CmdForge tools, and Model Context Protocol
|
||||
(MCP) calls, then behave like an ordinary Unix command:
|
||||
|
||||
```bash
|
||||
cat meeting.txt | meeting-decisions
|
||||
git diff --staged | review-change
|
||||
cmdforge run classify-private --no-fallback --require-local
|
||||
```
|
||||
|
||||
Tools are readable YAML, stored either in `~/.cmdforge/` for personal use or
|
||||
`./.cmdforge/` when they belong to a project. You choose the provider at build
|
||||
time or runtime: a local Ollama model, an installed coding CLI, or an
|
||||
OpenAI-compatible API.
|
||||
|
||||
## Install
|
||||
|
||||
CmdForge requires Python 3.10 or newer. `pipx` is recommended for a personal
|
||||
command-line application:
|
||||
|
||||
```bash
|
||||
pipx install "cmdforge[mcp,pty]"
|
||||
```
|
||||
|
||||
Or install it in a virtual environment:
|
||||
|
||||
```bash
|
||||
python -m pip install "cmdforge[mcp,pty]"
|
||||
```
|
||||
|
||||
The base `cmdforge` package includes the CLI and desktop tool builder. The
|
||||
`mcp` extra adds MCP client/server support, while `pty` adds interactive CLI
|
||||
providers.
|
||||
|
||||
Verify the installation and discover providers already present on the machine:
|
||||
|
||||
```bash
|
||||
cmdforge --version
|
||||
cmdforge providers discover
|
||||
cmdforge providers discover --add
|
||||
```
|
||||
|
||||
## Begin with One Useful Command
|
||||
|
||||
```bash
|
||||
cmdforge create explain --prompt "Explain this clearly for a beginner: {input}"
|
||||
echo "def area(r): return 3.14159 * r * r" | cmdforge run explain
|
||||
```
|
||||
|
||||
Use the deterministic mock provider while developing:
|
||||
|
||||
```bash
|
||||
echo "sample input" | cmdforge run explain --provider mock
|
||||
cmdforge inspect explain
|
||||
```
|
||||
|
||||
For AI-assisted creation, install CmdForge's official `forge-tool`:
|
||||
|
||||
```bash
|
||||
cmdforge registry install official/forge-tool
|
||||
|
||||
echo "Create a tool that extracts decisions and owners from meeting notes" \
|
||||
| forge-tool --name meeting-decisions --project
|
||||
```
|
||||
|
||||
## Four Kinds of Step
|
||||
|
||||
- **Prompt steps** call a selected AI provider.
|
||||
- **Code steps** perform deterministic Python transformations.
|
||||
- **Tool steps** compose existing CmdForge tools and delegated contexts.
|
||||
- **MCP steps** call tools exposed by external MCP servers.
|
||||
|
||||
Contracts, deterministic preflight, regression baselines, schema compatibility,
|
||||
and evidence-based quality scores help distinguish “worked once” from a tool
|
||||
you can maintain.
|
||||
|
||||
## Use CmdForge from Coding Agents
|
||||
|
||||
CmdForge can act as an MCP server for Codex and Claude Code. It is closed by
|
||||
default: configuring a host does not expose any tools until you add an explicit
|
||||
allowlist to `~/.cmdforge/mcp.yaml`.
|
||||
|
||||
```bash
|
||||
cmdforge mcp configure codex --dry-run
|
||||
cmdforge mcp configure codex
|
||||
|
||||
cmdforge mcp configure claude-code --scope project --dry-run
|
||||
```
|
||||
|
||||
Agents can also discover the local and remote catalogs without reading prompt
|
||||
or code bodies:
|
||||
|
||||
```bash
|
||||
cmdforge list --json --filter "commit message" --limit 10
|
||||
cmdforge registry search "release notes" --json --limit 5
|
||||
git diff | cmdforge run-once "Summarize this change: {input}"
|
||||
```
|
||||
|
||||
## Privacy and Provenance
|
||||
|
||||
Fallback is a data-movement decision. Sensitive callers can require a local
|
||||
provider, particular capabilities, and verified runtime identity:
|
||||
|
||||
```bash
|
||||
cmdforge run incident-summary \
|
||||
--provider local-reasoner \
|
||||
--no-fallback \
|
||||
--require-local \
|
||||
--require-capability structured-json \
|
||||
--data-classification private \
|
||||
--require-model-identity \
|
||||
--result-envelope json
|
||||
```
|
||||
|
||||
The result envelope reports the requested provider, actual provider, attempted
|
||||
fallback chain, model, digest, and locality. Those fields are produced by
|
||||
CmdForge rather than invented by the model.
|
||||
|
||||
## Learn More
|
||||
|
||||
- [Documentation](https://cmdforge.brrd.tech/docs)
|
||||
- [MCP and coding agents](https://cmdforge.brrd.tech/docs/mcp-overview)
|
||||
- [Providers and privacy policy](https://cmdforge.brrd.tech/docs/providers)
|
||||
- [Contracts and quality evidence](https://cmdforge.brrd.tech/docs/contracts-quality)
|
||||
- [Registry](https://cmdforge.brrd.tech/tools)
|
||||
- [Source and issues](https://gitea.brrd.tech/rob/CmdForge)
|
||||
|
||||
CmdForge is beta software, distributed under the MIT License. Review generated
|
||||
tools and third-party registry tools as you would any other executable code.
|
||||
@@ -55,16 +55,14 @@ That's it - you just used AI to explain itself. The `eli5` tool uses a free mode
|
||||
For regular use, install natively:
|
||||
|
||||
```bash
|
||||
# Clone and install
|
||||
git clone https://gitea.brrd.tech/rob/CmdForge.git
|
||||
cd CmdForge
|
||||
pip install -e .
|
||||
# Install the application with MCP and interactive-provider support
|
||||
pipx install "cmdforge[mcp,pty]"
|
||||
|
||||
# Ensure ~/.local/bin is in PATH
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
|
||||
# Install an AI provider (interactive guide)
|
||||
cmdforge providers install
|
||||
# Discover AI CLIs, API keys, and Ollama models already on this machine
|
||||
cmdforge providers discover --add
|
||||
|
||||
# Launch the GUI
|
||||
cmdforge
|
||||
@@ -75,12 +73,24 @@ cmdforge create summarize
|
||||
|
||||
## Installation
|
||||
|
||||
### Native Install
|
||||
### From PyPI
|
||||
|
||||
```bash
|
||||
pipx install "cmdforge[mcp,pty]"
|
||||
```
|
||||
|
||||
Or install into an active virtual environment:
|
||||
|
||||
```bash
|
||||
python -m pip install "cmdforge[mcp,pty]"
|
||||
```
|
||||
|
||||
### From Source
|
||||
|
||||
```bash
|
||||
git clone https://gitea.brrd.tech/rob/CmdForge.git
|
||||
cd CmdForge
|
||||
pip install -e .
|
||||
python -m pip install -e ".[mcp,pty]"
|
||||
```
|
||||
|
||||
### With Development Dependencies
|
||||
@@ -94,7 +104,7 @@ pip install -e ".[dev]"
|
||||
### Requirements
|
||||
|
||||
- Python 3.10+
|
||||
- At least one AI CLI tool installed (see [Provider Setup](docs/reference/providers.md))
|
||||
- A configured provider for live AI calls; the built-in mock works without one
|
||||
- PySide6 (included automatically - requires display server on Linux)
|
||||
|
||||
### Post-Install
|
||||
@@ -123,25 +133,40 @@ Opens the graphical interface where you can create and manage tools visually. Fe
|
||||
- **My Tools** - Browse, create, edit, and delete tools organized by category
|
||||
- **Registry** - Search and install community tools from the CmdForge registry
|
||||
- **Providers** - Manage AI provider configurations
|
||||
- **Reuse & Discovery** - Compare similar tools and preview safe extraction of reusable steps
|
||||
- **Scoped saving** - Save a new tool to the current project or your global tool collection
|
||||
|
||||

|
||||
|
||||
### CLI Mode
|
||||
|
||||
```bash
|
||||
# Tool Management
|
||||
cmdforge list # List all tools
|
||||
cmdforge create mytool # Create new tool
|
||||
cmdforge list --filter summarize # Filter installed tools for humans
|
||||
cmdforge list --json --filter summarize --limit 10 # Compact agent catalog
|
||||
cmdforge list --json --full --limit 1 # Explicitly include prompts/code
|
||||
cmdforge create mytool # Create a global tool
|
||||
cmdforge create mytool --project # Versionable ./.cmdforge tool
|
||||
echo "Build a bounded classifier" | forge-tool --name classify --project
|
||||
cmdforge edit mytool # Edit in $EDITOR
|
||||
cmdforge delete mytool # Delete tool
|
||||
cmdforge run mytool # Run a tool
|
||||
cat document.txt | cmdforge run-once "Summarize this:\n\n{input}"
|
||||
cmdforge run private-tool --provider ollama --no-fallback \
|
||||
--require-local --data-classification private --result-envelope json
|
||||
cmdforge test mytool # Test with mock provider
|
||||
cmdforge check mytool # Check dependencies (meta-tools)
|
||||
cmdforge inspect mytool # Preflight, contract proposals, safe conformance tests
|
||||
cmdforge inspect mytool --registry # Also find similar registry tools
|
||||
cmdforge inspect mytool --save-baseline # Approve passing evidence for regression checks
|
||||
cmdforge refresh # Update executable wrappers
|
||||
cmdforge docs mytool # View/create tool documentation
|
||||
|
||||
# Registry
|
||||
cmdforge registry search "keyword" # Search for tools
|
||||
cmdforge registry install owner/tool # Install a tool
|
||||
cmdforge registry publish mytool # Publish your tool
|
||||
cmdforge registry publish mytool # Preflight, confirm, then publish
|
||||
cmdforge registry publish mytool --dry-run # Preflight without publishing
|
||||
cmdforge registry status # Check moderation status
|
||||
cmdforge registry my-tools # List your published tools
|
||||
|
||||
@@ -174,8 +199,67 @@ cmdforge settings mytool diff # Show changes from defaults
|
||||
cmdforge config show # Show current config
|
||||
cmdforge config connect username # Connect to registry account
|
||||
cmdforge config disconnect # Disconnect from registry
|
||||
|
||||
# Private, local pipeline suggestions (disabled by default)
|
||||
cmdforge usage enable # Begin recording tool names in shell pipes
|
||||
cmdforge usage suggestions # Show frequent pipelines
|
||||
cmdforge usage clear # Delete all local usage history
|
||||
|
||||
# Model Context Protocol (optional: python -m pip install "cmdforge[mcp]")
|
||||
cmdforge mcp add local --command npx --arg=-y --arg @scope/server
|
||||
cmdforge mcp add remote --transport streamable-http --url https://example.com/mcp
|
||||
cmdforge mcp connect remote
|
||||
cmdforge mcp serve # Expose approved tools over stdio
|
||||
cmdforge mcp serve --transport streamable-http # Loopback-only HTTP by default
|
||||
|
||||
# Connect a coding agent and add a managed project policy
|
||||
cmdforge mcp configure codex --dry-run
|
||||
cmdforge mcp configure codex
|
||||
cmdforge mcp configure claude-code --scope project
|
||||
```
|
||||
|
||||
`mcp configure` registers CmdForge's stdio server through the host's own CLI
|
||||
and adds an idempotent managed block to `AGENTS.md` (Codex) or `CLAUDE.md`
|
||||
(Claude Code). It preserves all text outside that block. Preview with
|
||||
`--dry-run`, skip the policy with `--no-policy`, or deliberately replace an
|
||||
existing registration with `--force`. Setup never broadens the MCP exposure
|
||||
allowlist: review tools and add their names under `server.expose` in
|
||||
`~/.cmdforge/mcp.yaml`.
|
||||
|
||||
`cf` searches the public registry when no local tool matches. Registry results
|
||||
show available relevance and quality evidence and install on selection.
|
||||
When local pipeline discovery is explicitly enabled, `cf` also suggests
|
||||
frequently repeated pipe pairs. Only tool names, anonymous pipe identifiers,
|
||||
counts, and timestamps are kept in `~/.cmdforge/usage.json`; command arguments
|
||||
and input/output content are never recorded or transmitted.
|
||||
|
||||
### AI Assistant Workflow
|
||||
|
||||
For coding assistants, use CmdForge when an installed or registry tool matches
|
||||
the task; direct application APIs remain appropriate when the API is part of
|
||||
the product being built.
|
||||
|
||||
```bash
|
||||
# 1. Discover a compact local contract without exposing prompt/code bodies
|
||||
cmdforge list --json --filter "commit message" --limit 10
|
||||
|
||||
# 2. Search remotely if no local tool fits
|
||||
cmdforge registry search "generate commit message" --json --limit 5
|
||||
|
||||
# 3. Use a one-shot provider call for genuinely ad-hoc work
|
||||
git diff --staged | cmdforge run-once "Write a commit message for:\n\n{input}"
|
||||
|
||||
# 4. Promote repeated work into a permanent, reviewable tool
|
||||
echo "Create a conventional commit-message tool" | forge-tool --name commit-msg
|
||||
```
|
||||
|
||||
`run-once` honors `cmdforge config set default_provider NAME`, then selects an
|
||||
available configured provider when no default is set. Configure CmdForge as an
|
||||
MCP server when the assistant supports MCP so approved tools appear in its
|
||||
native tool catalog. `cmdforge mcp configure codex` and
|
||||
`cmdforge mcp configure claude-code` automate the host registration and add
|
||||
the discovery policy shown above to the host's project instruction file.
|
||||
|
||||
### Running Tools
|
||||
|
||||
Once created, tools work like any Unix command:
|
||||
@@ -657,6 +741,9 @@ cmdforge registry search --owner official --min-downloads 100
|
||||
# Connect your account (opens browser for authentication)
|
||||
cmdforge config connect yourusername
|
||||
|
||||
# Create an Ed25519 release key and register its public half
|
||||
cmdforge registry signing-key init
|
||||
|
||||
# Publish a tool
|
||||
cmdforge registry publish mytool
|
||||
|
||||
@@ -671,6 +758,51 @@ cmdforge registry status --sync
|
||||
```
|
||||
|
||||
Published tools go through moderation before appearing publicly. You'll receive feedback if changes are requested.
|
||||
Once a signing key is registered, publishing signs the transitive content
|
||||
identity automatically. The registry rejects unsigned or invalid releases and
|
||||
clients verify the signature before installing. The private key remains in
|
||||
`~/.cmdforge/release-signing-key.json` with `0600` permissions.
|
||||
|
||||
### Prompt Optimization
|
||||
|
||||
Generate local deterministic prompt variations without contacting a provider:
|
||||
|
||||
```bash
|
||||
cmdforge optimize summarize -n 5
|
||||
```
|
||||
|
||||
Structural contract checks cannot rank prompt meaning, so CmdForge will not
|
||||
claim a winning variation unless explicit behavioral cases are supplied:
|
||||
|
||||
```json
|
||||
[
|
||||
{"input": "A long input", "contains": "summary"},
|
||||
{"input": "Another input", "expected": {"result": "expected value"}}
|
||||
]
|
||||
```
|
||||
|
||||
```bash
|
||||
cmdforge optimize summarize --behavior-tests behavior.json \
|
||||
--test-provider opencode-pickle
|
||||
```
|
||||
|
||||
Behavior tests execute every candidate and may incur provider cost. Use
|
||||
`--provider NAME` separately when you explicitly want a provider to generate a
|
||||
rephrased candidate.
|
||||
|
||||
### Community Improvements
|
||||
|
||||
```bash
|
||||
cmdforge registry improve owner/tool 1.0.0 0 proposed-prompt.txt \
|
||||
--rationale "Clearer output requirement"
|
||||
cmdforge registry review-improvement 42 approve --notes "Tests pass"
|
||||
# Apply the approved text, bump the tool version, then publish it:
|
||||
cmdforge registry publish mytool --improvement-id 42
|
||||
```
|
||||
|
||||
Submissions are auto-tested before review. Contributor credit and the
|
||||
`optimized`/`community-reviewed` badges are awarded only when a published
|
||||
version actually contains the approved change.
|
||||
|
||||
### Tool Documentation
|
||||
|
||||
@@ -743,6 +875,10 @@ The graphical interface provides a modern desktop experience:
|
||||
|
||||
### Tool Builder
|
||||
- Visual form for creating and editing tools
|
||||
- Reuse & Discovery panel with asynchronous registry similarity results,
|
||||
exact local reuse evidence, and opt-in pipeline suggestions
|
||||
- Extraction creates a separate tool only after showing a unified diff; the
|
||||
current draft is never rewritten automatically
|
||||
- Add arguments with flags and default values
|
||||
- Add prompt steps (AI calls) with profile selection
|
||||
- Add code steps with **AI-assisted code generation**:
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
# Publishing CmdForge to PyPI
|
||||
|
||||
PyPI releases are public and immutable. A release version must never be reused
|
||||
for different bytes, even if an upload contains a mistake.
|
||||
|
||||
## One-time account setup
|
||||
|
||||
1. Create or sign in to an account at <https://pypi.org/>.
|
||||
2. Verify the account email address.
|
||||
3. Enable two-factor authentication and store the recovery codes in the
|
||||
password manager and an offline recovery location.
|
||||
4. For the first upload only, create an account-scoped API token. PyPI cannot
|
||||
create a project-scoped token until the project exists.
|
||||
5. Do not put a PyPI token in Git, `.pypirc`, chat, command arguments, or shell
|
||||
history. Let Twine prompt for it interactively.
|
||||
|
||||
After the first successful upload, immediately revoke the account-scoped token
|
||||
and create a new token restricted to the `cmdforge` project.
|
||||
|
||||
## Prepare and validate a release
|
||||
|
||||
Update the version in both `pyproject.toml` and `src/cmdforge/__init__.py`, then
|
||||
add the release to `CHANGELOG.md`. From a clean checkout:
|
||||
|
||||
```bash
|
||||
python -m pip install -e '.[release]'
|
||||
pytest tests/ -m "not integration"
|
||||
python -m build
|
||||
python -m twine check dist/*
|
||||
```
|
||||
|
||||
Inspect the wheel and source archive, then install the wheel into a clean
|
||||
temporary virtual environment and exercise both entry points. Do not upload an
|
||||
artifact that was built before the release commit.
|
||||
|
||||
## Upload
|
||||
|
||||
Run Twine interactively so the token is not recorded in shell history:
|
||||
|
||||
```bash
|
||||
python -m twine upload dist/*
|
||||
```
|
||||
|
||||
When prompted, use `__token__` as the username and paste the API token as the
|
||||
password. Once uploaded, verify the public project and install from PyPI in a
|
||||
new environment:
|
||||
|
||||
```bash
|
||||
python -m venv /tmp/cmdforge-pypi-check
|
||||
/tmp/cmdforge-pypi-check/bin/pip install 'cmdforge[mcp,pty]'
|
||||
/tmp/cmdforge-pypi-check/bin/cmdforge --version
|
||||
/tmp/cmdforge-pypi-check/bin/cmdforge --help
|
||||
```
|
||||
|
||||
Tag and push only the commit whose artifacts were published:
|
||||
|
||||
```bash
|
||||
git tag -a v0.2.0 -m "CmdForge 0.2.0"
|
||||
git push origin main
|
||||
git push origin v0.2.0
|
||||
```
|
||||
|
||||
If an upload is wrong, fix it, increment the version, rebuild, and publish a
|
||||
new release. Never delete and reuse the version number.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
+16
-6
@@ -1,13 +1,14 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=61.0", "wheel"]
|
||||
requires = ["setuptools>=77.0", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "cmdforge"
|
||||
version = "0.1.0"
|
||||
description = "Build custom AI-powered CLI commands in YAML"
|
||||
readme = "README.md"
|
||||
license = {text = "MIT"}
|
||||
version = "0.2.0"
|
||||
description = "Build, compose, and share AI-powered command-line tools you own"
|
||||
readme = "PYPI_README.md"
|
||||
license = "MIT"
|
||||
license-files = ["LICENSE"]
|
||||
requires-python = ">=3.10"
|
||||
authors = [
|
||||
{name = "Rob"}
|
||||
@@ -18,7 +19,6 @@ classifiers = [
|
||||
"Environment :: Console",
|
||||
"Intended Audience :: Developers",
|
||||
"Intended Audience :: System Administrators",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: POSIX :: Linux",
|
||||
"Operating System :: MacOS",
|
||||
"Programming Language :: Python :: 3",
|
||||
@@ -36,6 +36,7 @@ dependencies = [
|
||||
"NodeGraphQt>=0.6.0",
|
||||
"setuptools", # Required for distutils compatibility (Python 3.12+)
|
||||
"jsonschema>=4.0", # JSON schema validation for structured output
|
||||
"cryptography>=41.0", # Ed25519 release attestations
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -44,6 +45,10 @@ dev = [
|
||||
"pytest-cov>=4.0",
|
||||
"tomli>=1.1; python_version < '3.11'",
|
||||
]
|
||||
release = [
|
||||
"build>=1.2",
|
||||
"twine>=6.0",
|
||||
]
|
||||
registry = [
|
||||
"Flask>=2.3",
|
||||
"argon2-cffi>=21.0",
|
||||
@@ -57,6 +62,9 @@ flow = [
|
||||
pty = [
|
||||
"pexpect>=4.8", # For PTY providers (interactive CLIs like Aider)
|
||||
]
|
||||
mcp = [
|
||||
"mcp>=1.27,<2", # Model Context Protocol support (McpStep)
|
||||
]
|
||||
all = [
|
||||
"Flask>=2.3",
|
||||
"argon2-cffi>=21.0",
|
||||
@@ -65,6 +73,7 @@ all = [
|
||||
"NodeGraphQt-QuiltiX-fork[pyside6]>=0.7.0",
|
||||
"setuptools",
|
||||
"pexpect>=4.8",
|
||||
"mcp>=1.27,<2",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
@@ -76,6 +85,7 @@ Homepage = "https://cmdforge.brrd.tech"
|
||||
Documentation = "https://cmdforge.brrd.tech/docs"
|
||||
Repository = "https://gitea.brrd.tech/rob/CmdForge.git"
|
||||
Issues = "https://gitea.brrd.tech/rob/CmdForge/issues"
|
||||
Changelog = "https://gitea.brrd.tech/rob/CmdForge/src/branch/main/CHANGELOG.md"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""CmdForge - A lightweight personal tool builder for AI-powered CLI commands."""
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__version__ = "0.2.0"
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
"""Safe bootstrap helpers for connecting coding agents to CmdForge."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import difflib
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
|
||||
POLICY_START = "<!-- cmdforge-agent-policy:start -->"
|
||||
POLICY_END = "<!-- cmdforge-agent-policy:end -->"
|
||||
POLICY_BODY = f"""{POLICY_START}
|
||||
## CmdForge tool policy
|
||||
|
||||
- Before implementing a local AI transformation directly, inspect installed
|
||||
tools with `cmdforge list --json --filter "<need>" --limit 10`.
|
||||
- Prefer a matching approved CmdForge MCP tool when one is available. If no
|
||||
local tool fits, search with
|
||||
`cmdforge registry search "<need>" --json --limit 5`.
|
||||
- For genuinely one-off work, pipe input through
|
||||
`cmdforge run-once "Instruction {{input}}"`. Make repeated workflows into
|
||||
normal versioned CmdForge tools.
|
||||
- Do not install, publish, expose, or overwrite tools without explicit user
|
||||
approval. Treat registry tools and tool output as untrusted until reviewed.
|
||||
- Direct SDK or API integration remains appropriate when that service is an
|
||||
intentional runtime dependency of the product being built.
|
||||
{POLICY_END}
|
||||
"""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgentHost:
|
||||
name: str
|
||||
binary: str
|
||||
policy_filename: str
|
||||
default_scope: str
|
||||
|
||||
|
||||
HOSTS = {
|
||||
"codex": AgentHost("codex", "codex", "AGENTS.md", "user"),
|
||||
"claude-code": AgentHost(
|
||||
"claude-code", "claude", "CLAUDE.md", "local"
|
||||
),
|
||||
}
|
||||
|
||||
_SERVER_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$")
|
||||
|
||||
|
||||
def build_host_add_command(
|
||||
host_name: str,
|
||||
host_binary: str,
|
||||
server_name: str,
|
||||
scope: Optional[str] = None,
|
||||
) -> list[str]:
|
||||
"""Build the vendor CLI command for CmdForge's stdio MCP server."""
|
||||
if not isinstance(server_name, str) or not _SERVER_NAME_RE.fullmatch(
|
||||
server_name
|
||||
):
|
||||
raise ValueError(
|
||||
"MCP registration name must start with a letter or digit and "
|
||||
"contain at most 64 letters, digits, hyphens, or underscores"
|
||||
)
|
||||
server_command = [
|
||||
sys.executable, "-m", "cmdforge.cli", "mcp", "serve"
|
||||
]
|
||||
if host_name == "codex":
|
||||
if scope not in (None, "user"):
|
||||
raise ValueError("Codex CLI MCP registration supports user scope only")
|
||||
return [host_binary, "mcp", "add", server_name, "--", *server_command]
|
||||
if host_name == "claude-code":
|
||||
resolved_scope = scope or HOSTS[host_name].default_scope
|
||||
if resolved_scope not in ("local", "project", "user"):
|
||||
raise ValueError("Claude Code scope must be local, project, or user")
|
||||
return [
|
||||
host_binary, "mcp", "add", "--transport", "stdio",
|
||||
"--scope", resolved_scope, server_name, "--", *server_command,
|
||||
]
|
||||
raise ValueError(f"Unsupported agent host '{host_name}'")
|
||||
|
||||
|
||||
def merge_agent_policy(existing: str) -> str:
|
||||
"""Add or replace only CmdForge's managed policy block."""
|
||||
if POLICY_START in existing or POLICY_END in existing:
|
||||
if existing.count(POLICY_START) != 1 or existing.count(POLICY_END) != 1:
|
||||
raise ValueError("Agent instruction file has malformed CmdForge markers")
|
||||
start = existing.index(POLICY_START)
|
||||
end = existing.index(POLICY_END, start) + len(POLICY_END)
|
||||
prefix = existing[:start].rstrip()
|
||||
suffix = existing[end:].lstrip("\r\n")
|
||||
parts = [part for part in (prefix, POLICY_BODY.rstrip(), suffix.rstrip()) if part]
|
||||
return "\n\n".join(parts) + "\n"
|
||||
if not existing.strip():
|
||||
return POLICY_BODY
|
||||
return existing.rstrip() + "\n\n" + POLICY_BODY
|
||||
|
||||
|
||||
def policy_diff(path: Path, before: str, after: str) -> str:
|
||||
return "".join(difflib.unified_diff(
|
||||
before.splitlines(True),
|
||||
after.splitlines(True),
|
||||
fromfile=str(path),
|
||||
tofile=str(path),
|
||||
))
|
||||
|
||||
|
||||
def write_agent_policy(path: Path, content: str) -> None:
|
||||
"""Atomically write a managed policy without following a file symlink."""
|
||||
if path.is_symlink():
|
||||
raise ValueError(f"Refusing to update symlinked instruction file: {path}")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
mode = (path.stat().st_mode & 0o777) if path.exists() else 0o644
|
||||
temp_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", encoding="utf-8", dir=path.parent,
|
||||
prefix=f".{path.name}.", suffix=".tmp", delete=False,
|
||||
) as handle:
|
||||
temp_path = Path(handle.name)
|
||||
os.chmod(temp_path, mode)
|
||||
handle.write(content)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temp_path, path)
|
||||
finally:
|
||||
if temp_path and temp_path.exists():
|
||||
temp_path.unlink()
|
||||
|
||||
|
||||
def configure_agent_host(
|
||||
host_name: str,
|
||||
*,
|
||||
server_name: str = "cmdforge",
|
||||
scope: Optional[str] = None,
|
||||
project_dir: Optional[Path] = None,
|
||||
install_policy: bool = True,
|
||||
dry_run: bool = False,
|
||||
force: bool = False,
|
||||
) -> int:
|
||||
"""Register CmdForge with a host and install its managed project policy."""
|
||||
host = HOSTS.get(host_name)
|
||||
if not host:
|
||||
print(f"Error: Unsupported agent host '{host_name}'.", file=sys.stderr)
|
||||
return 1
|
||||
binary = shutil.which(host.binary)
|
||||
if not binary:
|
||||
print(
|
||||
f"Error: '{host.binary}' is not installed or is not on PATH.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
try:
|
||||
add_command = build_host_add_command(
|
||||
host_name, binary, server_name, scope
|
||||
)
|
||||
except ValueError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
project_dir = (project_dir or Path.cwd()).resolve()
|
||||
policy_path = project_dir / host.policy_filename
|
||||
before = ""
|
||||
after = ""
|
||||
if install_policy:
|
||||
if policy_path.is_symlink():
|
||||
print(
|
||||
f"Error: Refusing to update symlinked instruction file: {policy_path}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
try:
|
||||
before = (
|
||||
policy_path.read_text(encoding="utf-8")
|
||||
if policy_path.exists() else ""
|
||||
)
|
||||
after = merge_agent_policy(before)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error preparing agent policy: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if dry_run:
|
||||
print("Would run:")
|
||||
print(f" {shlex.join(add_command)}")
|
||||
if install_policy and before != after:
|
||||
print(f"\nWould update {policy_path}:")
|
||||
print(policy_diff(policy_path, before, after), end="")
|
||||
return 0
|
||||
|
||||
get_command = [binary, "mcp", "get", server_name]
|
||||
try:
|
||||
existing = _run_host_command(get_command, project_dir)
|
||||
except RuntimeError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
if existing.returncode == 0 and force:
|
||||
try:
|
||||
removed = _run_host_command(
|
||||
[binary, "mcp", "remove", server_name], project_dir
|
||||
)
|
||||
except RuntimeError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
if removed.returncode != 0:
|
||||
_print_host_error(host_name, "remove", removed)
|
||||
return 1
|
||||
existing = None
|
||||
if existing is None or existing.returncode != 0:
|
||||
try:
|
||||
added = _run_host_command(add_command, project_dir)
|
||||
except RuntimeError as exc:
|
||||
print(f"Error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
if added.returncode != 0:
|
||||
_print_host_error(host_name, "configure", added)
|
||||
return 1
|
||||
print(f"Configured CmdForge MCP for {host_name} as '{server_name}'.")
|
||||
else:
|
||||
print(
|
||||
f"CmdForge MCP entry '{server_name}' already exists for {host_name}; "
|
||||
"left unchanged. Use --force to replace it."
|
||||
)
|
||||
|
||||
if install_policy and before != after:
|
||||
try:
|
||||
write_agent_policy(policy_path, after)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error writing agent policy: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Updated managed CmdForge policy in {policy_path}.")
|
||||
elif install_policy:
|
||||
print(f"Managed CmdForge policy is already current in {policy_path}.")
|
||||
|
||||
_warn_if_no_tools_exposed()
|
||||
return 0
|
||||
|
||||
|
||||
def _run_host_command(command: list[str], project_dir: Path):
|
||||
"""Run a host CLI in project context without allowing an indefinite wait."""
|
||||
try:
|
||||
return subprocess.run(
|
||||
command,
|
||||
cwd=project_dir,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
timeout=30,
|
||||
)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise RuntimeError(
|
||||
f"Agent host command timed out after 30 seconds: "
|
||||
f"{shlex.join(command[:4])}"
|
||||
) from exc
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"Could not run agent host command: {exc}") from exc
|
||||
|
||||
|
||||
def _print_host_error(host_name: str, action: str, result) -> None:
|
||||
detail = (result.stderr or result.stdout or "unknown error").strip()
|
||||
print(
|
||||
f"Error: Could not {action} CmdForge for {host_name}: {detail[:1000]}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
def _warn_if_no_tools_exposed() -> None:
|
||||
try:
|
||||
from .mcp_client import load_mcp_serve_config
|
||||
config = load_mcp_serve_config()
|
||||
except (OSError, ValueError):
|
||||
return
|
||||
if not config.expose:
|
||||
print(
|
||||
"Warning: CmdForge MCP currently exposes no tools. Add reviewed tool "
|
||||
"names under server.expose in ~/.cmdforge/mcp.yaml; avoid '*' unless "
|
||||
"you intend to expose every public tool.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
@@ -0,0 +1,147 @@
|
||||
"""Ed25519 supply-chain attestations for CmdForge releases."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, Optional, Tuple
|
||||
|
||||
from cryptography.exceptions import InvalidSignature
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric.ed25519 import (
|
||||
Ed25519PrivateKey,
|
||||
Ed25519PublicKey,
|
||||
)
|
||||
|
||||
|
||||
def _canonical_payload(
|
||||
tool_name: str,
|
||||
version: str,
|
||||
content_hash: str,
|
||||
signer: str,
|
||||
signed_at: str,
|
||||
algorithm: str,
|
||||
) -> bytes:
|
||||
return json.dumps(
|
||||
{
|
||||
"algorithm": algorithm,
|
||||
"content_hash": content_hash,
|
||||
"signed_at": signed_at,
|
||||
"signer": signer,
|
||||
"tool_name": tool_name,
|
||||
"version": version,
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Attestation:
|
||||
tool_name: str
|
||||
version: str
|
||||
content_hash: str
|
||||
signer: str
|
||||
signature: str
|
||||
signed_at: str
|
||||
algorithm: str = "ed25519"
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"tool_name": self.tool_name,
|
||||
"version": self.version,
|
||||
"content_hash": self.content_hash,
|
||||
"signer": self.signer,
|
||||
"signature": self.signature,
|
||||
"signed_at": self.signed_at,
|
||||
"algorithm": self.algorithm,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, value: dict) -> "Attestation":
|
||||
return cls(**{key: value[key] for key in (
|
||||
"tool_name", "version", "content_hash", "signer", "signature",
|
||||
"signed_at", "algorithm",
|
||||
)})
|
||||
|
||||
|
||||
def generate_keypair() -> Tuple[str, str]:
|
||||
"""Generate base64-encoded raw Ed25519 private/public keys."""
|
||||
private = Ed25519PrivateKey.generate()
|
||||
private_bytes = private.private_bytes(
|
||||
serialization.Encoding.Raw,
|
||||
serialization.PrivateFormat.Raw,
|
||||
serialization.NoEncryption(),
|
||||
)
|
||||
public_bytes = private.public_key().public_bytes(
|
||||
serialization.Encoding.Raw,
|
||||
serialization.PublicFormat.Raw,
|
||||
)
|
||||
return (
|
||||
base64.b64encode(private_bytes).decode("ascii"),
|
||||
base64.b64encode(public_bytes).decode("ascii"),
|
||||
)
|
||||
|
||||
|
||||
def sign_tool(
|
||||
tool_name: str,
|
||||
version: str,
|
||||
content_hash: str,
|
||||
signer: str,
|
||||
private_key: str,
|
||||
*,
|
||||
signed_at: Optional[str] = None,
|
||||
) -> Attestation:
|
||||
"""Sign release identity using a publisher's private Ed25519 key."""
|
||||
timestamp = signed_at or datetime.now(timezone.utc).isoformat()
|
||||
algorithm = "ed25519"
|
||||
payload = _canonical_payload(
|
||||
tool_name, version, content_hash, signer, timestamp, algorithm
|
||||
)
|
||||
key = Ed25519PrivateKey.from_private_bytes(base64.b64decode(private_key))
|
||||
signature = base64.b64encode(key.sign(payload)).decode("ascii")
|
||||
return Attestation(
|
||||
tool_name, version, content_hash, signer, signature, timestamp, algorithm
|
||||
)
|
||||
|
||||
|
||||
def verify_attestation(attestation: Attestation, public_key: str) -> bool:
|
||||
"""Verify every signed field using the publisher's trusted public key."""
|
||||
if attestation.algorithm != "ed25519":
|
||||
return False
|
||||
try:
|
||||
key = Ed25519PublicKey.from_public_bytes(base64.b64decode(public_key))
|
||||
signature = base64.b64decode(attestation.signature, validate=True)
|
||||
payload = _canonical_payload(
|
||||
attestation.tool_name,
|
||||
attestation.version,
|
||||
attestation.content_hash,
|
||||
attestation.signer,
|
||||
attestation.signed_at,
|
||||
attestation.algorithm,
|
||||
)
|
||||
key.verify(signature, payload)
|
||||
return True
|
||||
except (ValueError, TypeError, InvalidSignature):
|
||||
return False
|
||||
|
||||
|
||||
def verify_trusted_attestation(
|
||||
attestation: Attestation, trusted_publishers: Dict[str, str]
|
||||
) -> bool:
|
||||
"""Anchor an attestation to an explicitly trusted publisher key."""
|
||||
public_key = trusted_publishers.get(attestation.signer)
|
||||
return bool(public_key) and verify_attestation(attestation, public_key)
|
||||
|
||||
|
||||
def verify_content_hash(content_hash: str, tool_dict: dict) -> bool:
|
||||
"""Verify a definition hash against a serialized Tool definition."""
|
||||
from .integrity import compute_tool_hash
|
||||
from .tool import Tool
|
||||
|
||||
try:
|
||||
return compute_tool_hash(Tool.from_dict(tool_dict)) == content_hash
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return False
|
||||
@@ -2,11 +2,12 @@
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .. import __version__
|
||||
|
||||
from .tool_commands import (
|
||||
cmd_list, cmd_create, cmd_edit, cmd_delete, cmd_test, cmd_run,
|
||||
cmd_list, cmd_create, cmd_edit, cmd_delete, cmd_test, cmd_run, cmd_run_once,
|
||||
cmd_ui, cmd_refresh, cmd_docs, cmd_check
|
||||
)
|
||||
from .provider_commands import cmd_providers
|
||||
@@ -16,6 +17,8 @@ from .project_commands import cmd_deps, cmd_deps_tree, cmd_install_deps, cmd_add
|
||||
from .config_commands import cmd_config
|
||||
from .settings_commands import cmd_settings
|
||||
from .system_deps_commands import cmd_system_deps
|
||||
from .mcp_commands import cmd_mcp
|
||||
from .usage_commands import cmd_usage
|
||||
|
||||
|
||||
def main():
|
||||
@@ -31,6 +34,13 @@ def main():
|
||||
# No command = launch UI
|
||||
# 'list' command
|
||||
p_list = subparsers.add_parser("list", help="List all tools")
|
||||
p_list.add_argument("--json", action="store_true", help="Output as JSON")
|
||||
p_list.add_argument("--filter", help="Filter by name or description (substring match)")
|
||||
p_list.add_argument("--limit", type=int, help="Return at most this many matching tools")
|
||||
p_list.add_argument(
|
||||
"--full", action="store_true",
|
||||
help="Include complete definitions in JSON output (prompts and code)",
|
||||
)
|
||||
p_list.set_defaults(func=cmd_list)
|
||||
|
||||
# 'create' command
|
||||
@@ -41,6 +51,15 @@ def main():
|
||||
p_create.add_argument("-p", "--prompt", help="Prompt template")
|
||||
p_create.add_argument("--provider", help="AI provider (default: mock)")
|
||||
p_create.add_argument("-f", "--force", action="store_true", help="Overwrite existing")
|
||||
create_location = p_create.add_mutually_exclusive_group()
|
||||
create_location.add_argument(
|
||||
"--project", action="store_true",
|
||||
help="Save under ./.cmdforge instead of ~/.cmdforge",
|
||||
)
|
||||
create_location.add_argument(
|
||||
"--output-dir", type=Path,
|
||||
help="Save under an explicit tool root directory",
|
||||
)
|
||||
p_create.set_defaults(func=cmd_create)
|
||||
|
||||
# 'edit' command
|
||||
@@ -72,9 +91,41 @@ def main():
|
||||
p_run.add_argument("--show-prompt", action="store_true", help="Show prompts in addition to output")
|
||||
p_run.add_argument("-v", "--verbose", action="store_true", help="Show debug information")
|
||||
p_run.add_argument("--auto-install", action="store_true", help="Automatically install missing tool dependencies")
|
||||
p_run.add_argument("--no-fallback", action="store_true", help="Fail instead of trying configured fallback providers")
|
||||
p_run.add_argument("--require-local", action="store_true", help="Allow only providers declared local")
|
||||
p_run.add_argument("--require-capability", action="append", default=[], help="Required provider capability (repeatable)")
|
||||
p_run.add_argument("--data-classification", choices=["public", "internal", "private"], help="Input data classification")
|
||||
p_run.add_argument("--require-model-identity", action="store_true")
|
||||
p_run.add_argument("--require-model-digest", action="store_true")
|
||||
p_run.add_argument("--result-envelope", choices=["json"], help="Wrap output with verified execution provenance")
|
||||
p_run.add_argument("tool_args", nargs=argparse.REMAINDER, help="Additional tool-specific arguments (use -- to separate)")
|
||||
p_run.set_defaults(func=cmd_run)
|
||||
|
||||
# 'run-once' command — one-shot without creating a tool
|
||||
p_run_once = subparsers.add_parser("run-once", help="Run a one-shot prompt through a provider without creating a tool")
|
||||
p_run_once.add_argument(
|
||||
"prompt",
|
||||
help=(
|
||||
"Prompt or template; piped/file input replaces {input}, or is "
|
||||
"appended when the placeholder is absent ('-' reads stdin as the whole prompt)"
|
||||
),
|
||||
)
|
||||
p_run_once.add_argument("-i", "--input", help="Read input data from this file (or '-' for stdin)")
|
||||
p_run_once.add_argument(
|
||||
"-p", "--provider",
|
||||
help="Provider to use (default: configured provider, then first available provider)",
|
||||
)
|
||||
p_run_once.add_argument("--max-tokens", type=int, help="Max output tokens")
|
||||
p_run_once.add_argument("--timeout", type=int, default=60, help="Timeout in seconds")
|
||||
p_run_once.add_argument("--no-fallback", action="store_true")
|
||||
p_run_once.add_argument("--require-local", action="store_true")
|
||||
p_run_once.add_argument("--require-capability", action="append", default=[])
|
||||
p_run_once.add_argument("--data-classification", choices=["public", "internal", "private"])
|
||||
p_run_once.add_argument("--require-model-identity", action="store_true")
|
||||
p_run_once.add_argument("--require-model-digest", action="store_true")
|
||||
p_run_once.add_argument("--result-envelope", choices=["json"])
|
||||
p_run_once.set_defaults(func=cmd_run_once)
|
||||
|
||||
# 'ui' command (explicit)
|
||||
p_ui = subparsers.add_parser("ui", help="Launch interactive UI")
|
||||
p_ui.set_defaults(func=cmd_ui)
|
||||
@@ -89,6 +140,36 @@ def main():
|
||||
p_docs.add_argument("-e", "--edit", action="store_true", help="Edit/create README in $EDITOR")
|
||||
p_docs.set_defaults(func=cmd_docs)
|
||||
|
||||
# 'inspect' command
|
||||
p_inspect = subparsers.add_parser("inspect", help="Run preflight analysis on a tool")
|
||||
p_inspect.add_argument("name", help="Tool name")
|
||||
p_inspect.add_argument(
|
||||
"--registry", action="store_true",
|
||||
help="Include similar tools from the configured registry",
|
||||
)
|
||||
p_inspect.add_argument(
|
||||
"--save-baseline", action="store_true",
|
||||
help="Save the current passing conformance evidence as the baseline",
|
||||
)
|
||||
p_inspect.set_defaults(func=cmd_inspect)
|
||||
|
||||
# 'optimize' command
|
||||
p_optimize = subparsers.add_parser("optimize", help="Generate and test prompt variations for a tool")
|
||||
p_optimize.add_argument("name", help="Tool name")
|
||||
p_optimize.add_argument("-n", "--count", type=int, default=5, help="Number of variations")
|
||||
p_optimize.add_argument(
|
||||
"--provider", default="mock",
|
||||
help="Provider used to generate a variation (default: local deterministic mock)",
|
||||
)
|
||||
p_optimize.add_argument(
|
||||
"--behavior-tests", metavar="FILE",
|
||||
help="JSON cases used to score semantic behavior (executes the tool)",
|
||||
)
|
||||
p_optimize.add_argument(
|
||||
"--test-provider", help="Provider override used while running behavior tests",
|
||||
)
|
||||
p_optimize.set_defaults(func=cmd_optimize)
|
||||
|
||||
# 'check' command
|
||||
p_check = subparsers.add_parser("check", help="Check dependencies for a tool (meta-tools)")
|
||||
p_check.add_argument("name", help="Tool name")
|
||||
@@ -100,6 +181,7 @@ def main():
|
||||
|
||||
# providers list
|
||||
p_prov_list = providers_sub.add_parser("list", help="List all providers and their status")
|
||||
p_prov_list.add_argument("--json", action="store_true")
|
||||
p_prov_list.set_defaults(func=cmd_providers)
|
||||
|
||||
# providers check
|
||||
@@ -124,6 +206,27 @@ def main():
|
||||
p_prov_add.add_argument("--model", help="Model ID for API providers")
|
||||
p_prov_add.add_argument("--api-key-env", help="Environment variable containing the API key")
|
||||
p_prov_add.add_argument("--tag", action="append", dest="tags", help="Provider tag (repeatable)")
|
||||
p_prov_add.add_argument(
|
||||
"--locality", choices=["local", "remote", "unknown"],
|
||||
help="Where inference runs",
|
||||
)
|
||||
p_prov_add.add_argument(
|
||||
"--capability", action="append", dest="capabilities",
|
||||
help="Typed capability such as structured-json (repeatable)",
|
||||
)
|
||||
p_prov_add.add_argument("--model-digest", help="Verified model content digest")
|
||||
p_prov_add.add_argument(
|
||||
"--cost-class", choices=["free", "low", "medium", "high"]
|
||||
)
|
||||
p_prov_add.add_argument(
|
||||
"--latency-class", choices=["fast", "standard", "slow"]
|
||||
)
|
||||
p_prov_add.add_argument("--max-context-tokens", type=int)
|
||||
p_prov_add.add_argument(
|
||||
"--data-policy",
|
||||
choices=["unspecified", "public", "internal", "private"],
|
||||
help="Maximum approved input classification",
|
||||
)
|
||||
p_prov_add.add_argument("--fallback", help="Fallback provider name; pass an empty value to clear")
|
||||
p_prov_add.add_argument(
|
||||
"--fallback-chain",
|
||||
@@ -209,8 +312,40 @@ def main():
|
||||
p_reg_publish.add_argument("--dry-run", action="store_true", help="Validate without publishing")
|
||||
p_reg_publish.add_argument("-f", "--force", action="store_true", help="Skip confirmation prompts")
|
||||
p_reg_publish.add_argument("--owner", default="", help="Owner override (admin only, e.g. 'official')")
|
||||
p_reg_publish.add_argument(
|
||||
"--improvement-id", type=int,
|
||||
help="Credit an approved improvement contained in this version",
|
||||
)
|
||||
p_reg_publish.set_defaults(func=cmd_registry)
|
||||
|
||||
p_reg_signing = registry_sub.add_parser(
|
||||
"signing-key", help="Initialize or inspect the release signing key"
|
||||
)
|
||||
p_reg_signing.add_argument(
|
||||
"action", nargs="?", choices=["init", "status"], default="status"
|
||||
)
|
||||
p_reg_signing.set_defaults(func=cmd_registry)
|
||||
|
||||
p_reg_improve = registry_sub.add_parser(
|
||||
"improve", help="Submit an auto-tested prompt/code improvement"
|
||||
)
|
||||
p_reg_improve.add_argument("tool", help="Tool as owner/name")
|
||||
p_reg_improve.add_argument("version", help="Exact tool version")
|
||||
p_reg_improve.add_argument("step_index", type=int, help="Step to replace")
|
||||
p_reg_improve.add_argument("file", help="File containing proposed prompt/code")
|
||||
p_reg_improve.add_argument("--rationale", default="")
|
||||
p_reg_improve.set_defaults(func=cmd_registry)
|
||||
|
||||
p_reg_review_improvement = registry_sub.add_parser(
|
||||
"review-improvement", help="Review a tested community improvement"
|
||||
)
|
||||
p_reg_review_improvement.add_argument("id", type=int)
|
||||
p_reg_review_improvement.add_argument(
|
||||
"decision", choices=["approve", "reject", "request_changes"]
|
||||
)
|
||||
p_reg_review_improvement.add_argument("--notes", default="")
|
||||
p_reg_review_improvement.set_defaults(func=cmd_registry)
|
||||
|
||||
# registry update-readme
|
||||
p_reg_update_readme = registry_sub.add_parser("update-readme", help="Update README for a published tool")
|
||||
p_reg_update_readme.add_argument("tool", nargs="?", default="", help="Tool name (local name, will resolve owner)")
|
||||
@@ -450,6 +585,122 @@ def main():
|
||||
# Default for system-deps with no subcommand (show status)
|
||||
p_sysdeps.set_defaults(func=cmd_system_deps)
|
||||
|
||||
# mcp command
|
||||
p_mcp = subparsers.add_parser("mcp", help="Manage MCP servers")
|
||||
mcp_sub = p_mcp.add_subparsers(dest="mcp_cmd", help="MCP commands")
|
||||
|
||||
# mcp serve
|
||||
p_mcp_serve = mcp_sub.add_parser("serve", help="Start CmdForge as an MCP server")
|
||||
p_mcp_serve.add_argument(
|
||||
"--transport", choices=["stdio", "streamable-http"], default="stdio",
|
||||
help="MCP server transport",
|
||||
)
|
||||
p_mcp_serve.add_argument("--host", default="127.0.0.1")
|
||||
p_mcp_serve.add_argument("--port", type=int, default=8000)
|
||||
p_mcp_serve.add_argument(
|
||||
"--allowed-origin", action="append", default=[],
|
||||
help="Allowed browser Origin; repeat as needed",
|
||||
)
|
||||
p_mcp_serve.add_argument(
|
||||
"--auth-token", help="Bearer token (or ${ENV_NAME})",
|
||||
)
|
||||
p_mcp_serve.add_argument(
|
||||
"--external-url",
|
||||
help="Public HTTPS URL when binding beyond localhost (TLS proxy required)",
|
||||
)
|
||||
p_mcp_serve.set_defaults(func=cmd_mcp)
|
||||
|
||||
# mcp list
|
||||
p_mcp_list = mcp_sub.add_parser("list", help="List configured MCP servers")
|
||||
p_mcp_list.set_defaults(func=cmd_mcp)
|
||||
|
||||
# mcp connect
|
||||
p_mcp_connect = mcp_sub.add_parser("connect", help="Connect to an MCP server and discover tools")
|
||||
p_mcp_connect.add_argument("name", help="Server name from mcp.yaml")
|
||||
p_mcp_connect.set_defaults(func=cmd_mcp)
|
||||
|
||||
# mcp configure
|
||||
p_mcp_configure = mcp_sub.add_parser(
|
||||
"configure", help="Connect an AI coding host to CmdForge"
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"host", choices=["claude-code", "codex"],
|
||||
help="Agent host to configure",
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"--name", default="cmdforge", help="MCP registration name"
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"--scope", choices=["local", "project", "user"],
|
||||
help="Host configuration scope (Claude Code; Codex supports user only)",
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"--no-policy", action="store_true",
|
||||
help="Do not add the managed CmdForge block to AGENTS.md/CLAUDE.md",
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"--dry-run", action="store_true",
|
||||
help="Preview the host command and instruction-file diff",
|
||||
)
|
||||
p_mcp_configure.add_argument(
|
||||
"--force", action="store_true",
|
||||
help="Replace an existing MCP registration with the same name",
|
||||
)
|
||||
p_mcp_configure.set_defaults(func=cmd_mcp)
|
||||
|
||||
# mcp add
|
||||
p_mcp_add = mcp_sub.add_parser("add", help="Add an MCP server")
|
||||
p_mcp_add.add_argument("name", help="Server name")
|
||||
p_mcp_add.add_argument(
|
||||
"--transport", choices=["stdio", "streamable-http"], default="stdio",
|
||||
help="Transport type",
|
||||
)
|
||||
p_mcp_add.add_argument("--command", help="Executable for stdio servers")
|
||||
p_mcp_add.add_argument("--url", help="Endpoint for Streamable HTTP servers")
|
||||
p_mcp_add.add_argument(
|
||||
"--header", action="append", default=[], metavar="NAME=VALUE",
|
||||
help="HTTP header; repeat as needed (supports ${NAME} references)",
|
||||
)
|
||||
p_mcp_add.add_argument(
|
||||
"--arg", dest="server_args", action="append", default=[], metavar="VALUE",
|
||||
help="Command argument; repeat for each argument (use --arg=-y for leading dashes)",
|
||||
)
|
||||
p_mcp_add.add_argument("--cwd", help="Working directory")
|
||||
p_mcp_add.add_argument(
|
||||
"--env", action="append", default=[], metavar="NAME=VALUE",
|
||||
help="Server environment entry; repeat as needed (supports ${NAME} references)",
|
||||
)
|
||||
p_mcp_add.add_argument(
|
||||
"--inherit-env", action="append", default=[], metavar="NAME",
|
||||
help="Additionally inherit this environment variable; repeat as needed",
|
||||
)
|
||||
p_mcp_add.add_argument("--timeout", type=float, default=30, help="Per-operation timeout in seconds")
|
||||
p_mcp_add.add_argument("-d", "--description", help="Server description")
|
||||
p_mcp_add.add_argument("--force", action="store_true", help="Overwrite existing server")
|
||||
p_mcp_add.set_defaults(func=cmd_mcp)
|
||||
|
||||
# mcp remove
|
||||
p_mcp_remove = mcp_sub.add_parser("remove", help="Remove an MCP server")
|
||||
p_mcp_remove.add_argument("name", help="Server name")
|
||||
p_mcp_remove.set_defaults(func=cmd_mcp)
|
||||
|
||||
# Default for mcp with no subcommand (list)
|
||||
p_mcp.set_defaults(func=cmd_mcp, mcp_cmd="list")
|
||||
|
||||
p_usage = subparsers.add_parser(
|
||||
"usage", help="Manage opt-in local pipeline discovery"
|
||||
)
|
||||
usage_sub = p_usage.add_subparsers(dest="usage_cmd")
|
||||
for action in ("enable", "disable", "status", "clear"):
|
||||
command = usage_sub.add_parser(action)
|
||||
command.set_defaults(func=cmd_usage)
|
||||
p_usage_suggestions = usage_sub.add_parser(
|
||||
"suggestions", help="Show frequently composed tool pipelines"
|
||||
)
|
||||
p_usage_suggestions.add_argument("--threshold", type=int, default=3)
|
||||
p_usage_suggestions.set_defaults(func=cmd_usage)
|
||||
p_usage.set_defaults(func=cmd_usage, usage_cmd="status")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# If no command, launch UI
|
||||
@@ -459,5 +710,198 @@ def main():
|
||||
return args.func(args)
|
||||
|
||||
|
||||
def cmd_inspect(args):
|
||||
"""Run preflight analysis on a tool."""
|
||||
from ..preflight import analyze_tool
|
||||
from .tool_commands import _resolve_existing_tool
|
||||
|
||||
tool = _resolve_existing_tool(args.name)
|
||||
if not tool:
|
||||
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
registry_client = None
|
||||
if getattr(args, "registry", False):
|
||||
from ..registry_client import get_client
|
||||
registry_client = get_client()
|
||||
report = analyze_tool(
|
||||
tool,
|
||||
registry_client=registry_client,
|
||||
include_contract_tests=True,
|
||||
)
|
||||
|
||||
if report.errors:
|
||||
print(f"Errors ({len(report.errors)}):")
|
||||
for e in report.errors:
|
||||
print(f" ERROR: {e}")
|
||||
print()
|
||||
|
||||
if report.warnings:
|
||||
print(f"Warnings ({len(report.warnings)}):")
|
||||
for w in report.warnings:
|
||||
print(f" WARN: {w}")
|
||||
print()
|
||||
|
||||
if report.suggestions:
|
||||
print(f"Suggestions ({len(report.suggestions)}):")
|
||||
for s in report.suggestions:
|
||||
print(f" HINT: {s}")
|
||||
print()
|
||||
|
||||
if report.similar_tools:
|
||||
print(f"Similar registry tools ({len(report.similar_tools)}):")
|
||||
for t in report.similar_tools:
|
||||
print(f" - {t['name']}: {t.get('description', '')[:80]}")
|
||||
print()
|
||||
|
||||
proposal = report.contract_proposal or {}
|
||||
proposed_schemas = {
|
||||
key: proposal.get(key)
|
||||
for key in ("input_schema", "output_schema")
|
||||
if proposal.get(key) is not None
|
||||
}
|
||||
if proposed_schemas:
|
||||
import yaml
|
||||
|
||||
print(f"Contract proposal ({proposal.get('confidence', 'low')} confidence):")
|
||||
print(yaml.safe_dump(proposed_schemas, sort_keys=False).rstrip())
|
||||
for evidence in proposal.get("evidence", []):
|
||||
print(f" EVIDENCE: {evidence}")
|
||||
print(" Contracts are suggestions only and were not saved.")
|
||||
print()
|
||||
|
||||
if report.generated_tests:
|
||||
print(f"Contract conformance ({len(report.generated_tests)} case(s)):")
|
||||
for result in report.generated_tests:
|
||||
print(
|
||||
f" {result['state'].upper()}: {result['step']} — "
|
||||
f"{result.get('detail', '')}"
|
||||
)
|
||||
print()
|
||||
|
||||
if report.regression:
|
||||
regression = report.regression
|
||||
status = (
|
||||
"REGRESSION" if regression.get("has_regressions")
|
||||
else "REVIEW" if regression.get("contract_changed")
|
||||
else "STABLE"
|
||||
)
|
||||
print(
|
||||
f"Regression comparison: {status} — "
|
||||
f"{regression.get('summary', 'no changes')}"
|
||||
)
|
||||
print(
|
||||
f" Baseline {regression.get('baseline_version') or '(unversioned)'} "
|
||||
f"-> current {regression.get('current_version') or '(unversioned)'}"
|
||||
)
|
||||
print()
|
||||
|
||||
if report.compatibility:
|
||||
print(f"ToolStep compatibility ({len(report.compatibility)}):")
|
||||
for finding in report.compatibility:
|
||||
print(
|
||||
f" {str(finding.get('state', 'unknown')).upper()}: "
|
||||
f"step {finding.get('step', '?')} -> "
|
||||
f"{finding.get('tool', 'unknown')} — {finding.get('detail', '')}"
|
||||
)
|
||||
print()
|
||||
|
||||
if getattr(args, "save_baseline", False):
|
||||
from ..contract_testing import (
|
||||
ConformanceReport,
|
||||
ConformanceResult,
|
||||
save_conformance_baseline,
|
||||
)
|
||||
|
||||
conformance = ConformanceReport([
|
||||
ConformanceResult(**result) for result in report.generated_tests
|
||||
])
|
||||
if not tool.path:
|
||||
print("Error: Tool has no local directory for a baseline.", file=sys.stderr)
|
||||
return 1
|
||||
try:
|
||||
path = save_conformance_baseline(tool, conformance, tool.path.parent)
|
||||
except ValueError as exc:
|
||||
print(f"Error: Baseline not saved: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Saved conformance baseline: {path}")
|
||||
print()
|
||||
|
||||
# Quality score
|
||||
from ..quality import compute_quality
|
||||
qs = compute_quality(tool, report)
|
||||
print(
|
||||
f"Quality {qs.headline} — {qs.evidence_coverage}% evidence coverage — "
|
||||
f"Evaluated {qs.last_evaluated[:10]}"
|
||||
)
|
||||
for cat in qs.categories:
|
||||
print(f" {cat.name:24s} {cat.display:>8s}")
|
||||
print()
|
||||
|
||||
if report.reuse_opportunities:
|
||||
print(f"Reuse opportunities ({len(report.reuse_opportunities)}):")
|
||||
for opp in report.reuse_opportunities:
|
||||
print(f" {opp['type'].upper()}: {opp['detail']}")
|
||||
print()
|
||||
|
||||
if report.audit_evidence:
|
||||
ae = report.audit_evidence
|
||||
print(f"Audit evidence (engine v{ae.get('engine_version', '?')}, {ae.get('timestamp', '')[:10]}):")
|
||||
print(f" Checks: {', '.join(ae.get('checks_run', []))}")
|
||||
print()
|
||||
|
||||
if not report.errors and not report.warnings and not report.suggestions:
|
||||
print("No issues found.")
|
||||
|
||||
return 0 if not report.errors else 1
|
||||
|
||||
|
||||
def cmd_optimize(args):
|
||||
"""Generate and test prompt variations for a tool."""
|
||||
from ..prompt_optimizer import load_behavioral_evaluator, optimize_tool
|
||||
from ..tool import load_tool
|
||||
|
||||
tool = load_tool(args.name)
|
||||
if not tool:
|
||||
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Optimizing '{tool.name}' — generating {args.count} prompt variations...")
|
||||
print()
|
||||
|
||||
if args.count < 1:
|
||||
print("Error: --count must be at least 1.", file=sys.stderr)
|
||||
return 2
|
||||
evaluator = None
|
||||
if args.behavior_tests:
|
||||
try:
|
||||
evaluator = load_behavioral_evaluator(
|
||||
args.behavior_tests, args.test_provider
|
||||
)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error loading behavior tests: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
result = optimize_tool(
|
||||
tool, count=args.count, provider=args.provider, evaluator=evaluator
|
||||
)
|
||||
|
||||
print(f"Baseline score: {result.baseline_score}")
|
||||
print(f"Variations tested: {len(result.variations)}")
|
||||
print()
|
||||
|
||||
if result.best:
|
||||
print(f"Best variation (index {result.best_index + 1}, strategy: {result.best.strategy}):")
|
||||
print(f" Score: {result.scores.get(result.best_index, 0)}")
|
||||
print(f" Improvement: {result.improvement:.0f}%")
|
||||
print(f" Original: {result.best.original[:100]}{'...' if len(result.best.original) > 100 else ''}")
|
||||
print(f" Variation: {result.best.variation[:100]}{'...' if len(result.best.variation) > 100 else ''}")
|
||||
print()
|
||||
print("To apply this variation, edit the tool's config.yaml and update the prompt step.")
|
||||
else:
|
||||
print(result.note or "No improvements found.")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
"""MCP management commands."""
|
||||
|
||||
import re
|
||||
|
||||
from ..mcp_client import (
|
||||
DEFAULT_INHERITED_ENV,
|
||||
McpServerConfig,
|
||||
McpClientManager,
|
||||
_sanitize,
|
||||
load_mcp_config,
|
||||
save_mcp_config,
|
||||
)
|
||||
|
||||
|
||||
def cmd_mcp(args):
|
||||
"""Manage MCP servers."""
|
||||
if getattr(args, "mcp_cmd", "list") == "list":
|
||||
return _cmd_mcp_list(args)
|
||||
elif args.mcp_cmd == "serve":
|
||||
return _cmd_mcp_serve(args)
|
||||
elif args.mcp_cmd == "connect":
|
||||
return _cmd_mcp_connect(args)
|
||||
elif args.mcp_cmd == "add":
|
||||
return _cmd_mcp_add(args)
|
||||
elif args.mcp_cmd == "remove":
|
||||
return _cmd_mcp_remove(args)
|
||||
elif args.mcp_cmd == "configure":
|
||||
return _cmd_mcp_configure(args)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_mcp_configure(args):
|
||||
from ..agent_setup import configure_agent_host
|
||||
|
||||
return configure_agent_host(
|
||||
args.host,
|
||||
server_name=args.name,
|
||||
scope=args.scope,
|
||||
install_policy=not args.no_policy,
|
||||
dry_run=args.dry_run,
|
||||
force=args.force,
|
||||
)
|
||||
|
||||
|
||||
def _cmd_mcp_list(args):
|
||||
try:
|
||||
servers = load_mcp_config()
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error loading MCP config: {exc}")
|
||||
return 1
|
||||
if not servers:
|
||||
print("No MCP servers configured.")
|
||||
print("\nAdd a server: cmdforge mcp add <name> --command npx "
|
||||
"--arg=-y --arg @scope/server")
|
||||
print("Config file: ~/.cmdforge/mcp.yaml")
|
||||
return 0
|
||||
|
||||
print(f"MCP servers ({len(servers)}):\n")
|
||||
for s in servers:
|
||||
print(f" [+] {s.name}")
|
||||
if s.description:
|
||||
print(f" Description: {s.description}")
|
||||
print(f" Transport: {s.transport}")
|
||||
if s.transport == "stdio":
|
||||
command = " ".join([s.command or ""] + s.args)
|
||||
print(f" Command: {_sanitize(command)}")
|
||||
if s.cwd:
|
||||
print(f" Working dir: {s.cwd}")
|
||||
else:
|
||||
print(f" URL: {_sanitize(s.url or '')}")
|
||||
print(f" Approved: {'yes' if s.approved else 'no'}")
|
||||
print(f" Timeout: {s.timeout}s")
|
||||
print()
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_mcp_connect(args):
|
||||
name = args.name
|
||||
try:
|
||||
with McpClientManager() as manager:
|
||||
tools = manager.discover(name)
|
||||
except (ImportError, KeyError, OSError, PermissionError, RuntimeError,
|
||||
TimeoutError, ValueError) as exc:
|
||||
print(f"Error connecting to server '{name}': {exc}")
|
||||
return 1
|
||||
|
||||
print(f"Connected to '{name}'. Available tools ({len(tools)}):\n")
|
||||
for t in tools:
|
||||
print(f" [+] {t.get('name', 'unknown')}")
|
||||
desc = t.get("description", "")
|
||||
if desc:
|
||||
print(f" {desc[:120]}")
|
||||
print()
|
||||
print(f"Use these tools in your CmdForge tool configs with:")
|
||||
print(f" steps:")
|
||||
print(f" - type: mcp")
|
||||
print(f" server: {name}")
|
||||
print(f" tool: <tool-name>")
|
||||
print(f" output_var: result")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_mcp_add(args):
|
||||
name = args.name
|
||||
try:
|
||||
servers = load_mcp_config()
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error loading MCP config: {exc}")
|
||||
return 1
|
||||
existing = {s.name: s for s in servers}
|
||||
|
||||
if name in existing and not getattr(args, "force", False):
|
||||
print(f"Server '{name}' already exists. Use --force to overwrite.")
|
||||
return 1
|
||||
|
||||
try:
|
||||
server_env = _parse_env_entries(getattr(args, "env", []))
|
||||
headers = _parse_header_entries(getattr(args, "header", []))
|
||||
except ValueError as exc:
|
||||
print(f"Error: {exc}")
|
||||
return 1
|
||||
|
||||
inherited = list(DEFAULT_INHERITED_ENV)
|
||||
for env_name in getattr(args, "inherit_env", []):
|
||||
if env_name not in inherited:
|
||||
inherited.append(env_name)
|
||||
|
||||
config = McpServerConfig(
|
||||
name=name,
|
||||
transport=getattr(args, "transport", "stdio"),
|
||||
command=getattr(args, "command", None),
|
||||
url=getattr(args, "url", None),
|
||||
headers=headers,
|
||||
args=getattr(args, "server_args", []) or [],
|
||||
cwd=getattr(args, "cwd", None),
|
||||
env=server_env,
|
||||
inherit_env=inherited,
|
||||
timeout=getattr(args, "timeout", 30),
|
||||
description=getattr(args, "description", "") or "",
|
||||
approved=True,
|
||||
)
|
||||
|
||||
try:
|
||||
config.validate()
|
||||
except ValueError as exc:
|
||||
print(f"Error: {exc}")
|
||||
return 1
|
||||
|
||||
if name in existing:
|
||||
servers = [s for s in servers if s.name != name]
|
||||
servers.append(config)
|
||||
try:
|
||||
save_mcp_config(servers)
|
||||
except OSError as exc:
|
||||
print(f"Error saving MCP config: {exc}")
|
||||
return 1
|
||||
print(f"MCP server '{name}' saved and approved to run.")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_mcp_remove(args):
|
||||
name = args.name
|
||||
try:
|
||||
servers = load_mcp_config()
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error loading MCP config: {exc}")
|
||||
return 1
|
||||
if not any(s.name == name for s in servers):
|
||||
print(f"Server '{name}' not found.")
|
||||
return 1
|
||||
|
||||
servers = [s for s in servers if s.name != name]
|
||||
try:
|
||||
save_mcp_config(servers)
|
||||
except OSError as exc:
|
||||
print(f"Error saving MCP config: {exc}")
|
||||
return 1
|
||||
print(f"MCP server '{name}' removed.")
|
||||
return 0
|
||||
|
||||
|
||||
def _parse_env_entries(entries):
|
||||
result = {}
|
||||
for entry in entries:
|
||||
if "=" not in entry:
|
||||
raise ValueError(f"invalid --env value '{entry}'; expected NAME=VALUE")
|
||||
name, value = entry.split("=", 1)
|
||||
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", name):
|
||||
raise ValueError(f"invalid environment variable name '{name}'")
|
||||
result[name] = value
|
||||
return result
|
||||
|
||||
|
||||
def _parse_header_entries(entries):
|
||||
result = {}
|
||||
for entry in entries:
|
||||
if "=" not in entry:
|
||||
raise ValueError(f"invalid --header value '{entry}'; expected NAME=VALUE")
|
||||
name, value = entry.split("=", 1)
|
||||
if not name.strip() or any(char in name for char in "\r\n"):
|
||||
raise ValueError(f"invalid HTTP header name '{name}'")
|
||||
if any(char in value for char in "\r\n"):
|
||||
raise ValueError(f"invalid HTTP header value for '{name}'")
|
||||
result[name.strip()] = value
|
||||
return result
|
||||
|
||||
|
||||
def _cmd_mcp_serve(args):
|
||||
"""Start CmdForge as an MCP server."""
|
||||
from ..mcp_server import serve as start_server
|
||||
|
||||
transport = getattr(args, "transport", "stdio")
|
||||
try:
|
||||
start_server(
|
||||
transport=transport,
|
||||
host=getattr(args, "host", "127.0.0.1"),
|
||||
port=getattr(args, "port", 8000),
|
||||
allowed_origins=getattr(args, "allowed_origin", []),
|
||||
auth_token=getattr(args, "auth_token", None),
|
||||
external_url=getattr(args, "external_url", None),
|
||||
)
|
||||
except ImportError as exc:
|
||||
print(f"Error: {exc}")
|
||||
return 1
|
||||
except Exception as exc:
|
||||
print(f"Error starting MCP server: {exc}")
|
||||
return 1
|
||||
return 0
|
||||
+164
-10
@@ -2,8 +2,10 @@
|
||||
|
||||
import sys
|
||||
import os
|
||||
import select
|
||||
import tty
|
||||
import termios
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import List, Tuple, Optional
|
||||
from dataclasses import dataclass
|
||||
|
||||
@@ -30,6 +32,7 @@ GREEN = "\033[32m"
|
||||
RESET = "\033[0m"
|
||||
|
||||
MAX_VISIBLE = 8 # Show at most 8 items
|
||||
_registry_cache = {}
|
||||
|
||||
# Output stream for UI (stderr when stdout is piped, stdout otherwise)
|
||||
_ui_out = None
|
||||
@@ -78,11 +81,72 @@ def get_tools() -> List[dict]:
|
||||
"flag": a.flag,
|
||||
"default": a.default or "",
|
||||
"desc": a.description or ""
|
||||
} for a in tool.arguments]
|
||||
} for a in tool.arguments],
|
||||
"deprecated": tool.deprecated,
|
||||
"deprecated_message": tool.deprecated_message,
|
||||
"replacement": tool.replacement,
|
||||
})
|
||||
return sorted(tools, key=lambda t: t["name"])
|
||||
|
||||
|
||||
def search_registry(query: str) -> List[dict]:
|
||||
"""Search the registry for tools matching the query.
|
||||
|
||||
Returns a list of dicts with name, desc, and registry metadata.
|
||||
Registry search is public; authentication is not required.
|
||||
"""
|
||||
if not query or len(query) < 2:
|
||||
return []
|
||||
normalized = query.strip().lower()
|
||||
if normalized in _registry_cache:
|
||||
return [dict(item) for item in _registry_cache[normalized]]
|
||||
try:
|
||||
from ..registry_client import RegistryError, get_client
|
||||
client = get_client()
|
||||
client.timeout = min(getattr(client, "timeout", 3), 3)
|
||||
client.max_retries = 1
|
||||
semantic = client.semantic_search(query, limit=5)
|
||||
if semantic.get("available") and semantic.get("data"):
|
||||
items = semantic["data"]
|
||||
else:
|
||||
results = client.search_tools(query, per_page=5)
|
||||
items = results.data if hasattr(results, "data") else results
|
||||
found = [
|
||||
{
|
||||
"name": f"{item.get('owner', '')}/{item.get('name', '')}",
|
||||
"desc": (item.get("description") or "")[:50],
|
||||
"args": [],
|
||||
"registry": True,
|
||||
"downloads": item.get("downloads", 0),
|
||||
"relevance": item.get("similarity", item.get("score")),
|
||||
"quality_score": item.get("quality_score"),
|
||||
"quality_coverage": item.get("quality_coverage"),
|
||||
}
|
||||
for item in items
|
||||
]
|
||||
_registry_cache[normalized] = found
|
||||
return [dict(item) for item in found]
|
||||
except RegistryError:
|
||||
return []
|
||||
|
||||
|
||||
def _install_registry_selection(tool: dict) -> PickerResult:
|
||||
"""Install a registry result and retain its qualified identity for execution."""
|
||||
from ..resolver import install_from_registry
|
||||
|
||||
install_from_registry(tool["name"])
|
||||
return PickerResult(tool["name"], {})
|
||||
|
||||
|
||||
def _local_selection(tool: dict, arguments: Optional[dict] = None) -> PickerResult:
|
||||
if tool.get("deprecated"):
|
||||
guidance = tool.get("deprecated_message") or "This tool is no longer maintained."
|
||||
if tool.get("replacement"):
|
||||
guidance += f" Use '{tool['replacement']}' instead."
|
||||
_write(f"{YELLOW}Warning: {guidance}{RESET}\n")
|
||||
return PickerResult(tool["name"], arguments or {})
|
||||
|
||||
|
||||
class TTYInput:
|
||||
"""Read from /dev/tty for keyboard input, even when stdin is piped."""
|
||||
|
||||
@@ -103,10 +167,14 @@ class TTYInput:
|
||||
if self.tty:
|
||||
self.tty.close()
|
||||
|
||||
def getch(self):
|
||||
def getch(self, timeout: Optional[float] = None):
|
||||
"""Read a single character."""
|
||||
tty.setraw(self.fd)
|
||||
try:
|
||||
if timeout is not None:
|
||||
ready, _, _ = select.select([self.tty], [], [], timeout)
|
||||
if not ready:
|
||||
return None
|
||||
ch = self.tty.read(1)
|
||||
if ch == '\x1b':
|
||||
# Read escape sequence - bytes arrive together from terminal
|
||||
@@ -132,19 +200,25 @@ def clear_dropdown(n_lines: int):
|
||||
def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
|
||||
"""Run inline picker."""
|
||||
tools = get_tools()
|
||||
if not tools:
|
||||
_write("No tools. Create one: cmdforge\n")
|
||||
return None
|
||||
|
||||
query = ""
|
||||
selected = 0
|
||||
scroll = 0
|
||||
last_drawn = 0
|
||||
registry_results = {}
|
||||
registry_future = None
|
||||
registry_future_query = None
|
||||
registry_executor = ThreadPoolExecutor(max_workers=1)
|
||||
|
||||
_write(HIDE_CURSOR)
|
||||
|
||||
try:
|
||||
while True:
|
||||
if registry_future is not None and registry_future.done():
|
||||
registry_results[registry_future_query] = registry_future.result()
|
||||
registry_future = None
|
||||
registry_future_query = None
|
||||
|
||||
# Filter
|
||||
matches = []
|
||||
for t in tools:
|
||||
@@ -157,6 +231,19 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
|
||||
matches.sort(key=lambda x: -x[1])
|
||||
filtered = [m[0] for m in matches]
|
||||
|
||||
# If no local matches, search registry
|
||||
if query and len(query) >= 2 and not filtered:
|
||||
normalized_query = query.strip().lower()
|
||||
found = registry_results.get(normalized_query, [])
|
||||
if registry_future is None and normalized_query not in registry_results:
|
||||
registry_future_query = normalized_query
|
||||
registry_future = registry_executor.submit(
|
||||
search_registry, query
|
||||
)
|
||||
for rt in found:
|
||||
rt["_registry"] = True
|
||||
filtered = found
|
||||
|
||||
if selected >= len(filtered):
|
||||
selected = max(0, len(filtered) - 1)
|
||||
|
||||
@@ -181,13 +268,29 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
|
||||
# Items
|
||||
for i, t in enumerate(visible):
|
||||
actual_idx = scroll + i
|
||||
is_registry = t.get("_registry", False) or t.get("registry", False)
|
||||
is_deprecated = t.get("deprecated", False)
|
||||
|
||||
if actual_idx == selected:
|
||||
prefix = f"{CYAN}{BOLD}▸ {t['name']}{RESET}"
|
||||
else:
|
||||
prefix = f" {t['name']}"
|
||||
|
||||
# Add registry marker
|
||||
if is_registry:
|
||||
prefix += f" {YELLOW}[registry]{RESET}"
|
||||
if t.get("quality_score") is not None:
|
||||
prefix += f" Q{t['quality_score']}"
|
||||
if t.get("quality_coverage") is not None:
|
||||
prefix += f"/{t['quality_coverage']}%"
|
||||
if t.get("relevance") is not None:
|
||||
prefix += f" R{float(t['relevance']):.2f}"
|
||||
# Add deprecation marker
|
||||
if is_deprecated:
|
||||
prefix += f" {YELLOW}[deprecated]{RESET}"
|
||||
|
||||
# Add arg indicator
|
||||
if t['args']:
|
||||
if t.get('args'):
|
||||
prefix += f" {GREEN}⚙{RESET}"
|
||||
|
||||
# Add description
|
||||
@@ -201,24 +304,46 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
|
||||
last_drawn = len(lines)
|
||||
|
||||
# Input
|
||||
ch = tty_input.getch()
|
||||
ch = tty_input.getch(
|
||||
timeout=0.1 if registry_future is not None else None
|
||||
)
|
||||
if ch is None:
|
||||
continue
|
||||
|
||||
if ch in ('\r', '\n'): # Enter - run
|
||||
if filtered:
|
||||
selected_tool = filtered[selected]
|
||||
clear_dropdown(last_drawn)
|
||||
_write(SHOW_CURSOR)
|
||||
return PickerResult(filtered[selected]["name"], {})
|
||||
# If it's a registry tool, install it first
|
||||
if selected_tool.get("_registry") or selected_tool.get("registry"):
|
||||
_write(f"{YELLOW}Installing {selected_tool['name']}...{RESET}\n")
|
||||
try:
|
||||
return _install_registry_selection(selected_tool)
|
||||
except Exception as exc:
|
||||
_write(f"Install failed: {exc}\n")
|
||||
return None
|
||||
return _local_selection(selected_tool)
|
||||
|
||||
elif ch == '\t': # Tab - configure args or run
|
||||
if filtered:
|
||||
tool = filtered[selected]
|
||||
clear_dropdown(last_drawn)
|
||||
if tool.get("_registry") or tool.get("registry"):
|
||||
_write(f"{YELLOW}Installing {tool['name']}...{RESET}\n")
|
||||
try:
|
||||
_write(SHOW_CURSOR)
|
||||
return _install_registry_selection(tool)
|
||||
except Exception as exc:
|
||||
_write(f"Install failed: {exc}\n")
|
||||
_write(SHOW_CURSOR)
|
||||
return None
|
||||
if tool['args']:
|
||||
args = pick_args(tty_input, tool)
|
||||
_write(SHOW_CURSOR)
|
||||
return PickerResult(tool["name"], args) if args is not None else None
|
||||
return _local_selection(tool, args) if args is not None else None
|
||||
_write(SHOW_CURSOR)
|
||||
return PickerResult(tool["name"], {})
|
||||
return _local_selection(tool)
|
||||
|
||||
elif ch == '\x1b' or ch == '\x03': # Esc or Ctrl+C
|
||||
clear_dropdown(last_drawn)
|
||||
@@ -243,6 +368,8 @@ def run_picker(tty_input: TTYInput) -> Optional[PickerResult]:
|
||||
except Exception:
|
||||
_write(SHOW_CURSOR)
|
||||
raise
|
||||
finally:
|
||||
registry_executor.shutdown(wait=False, cancel_futures=True)
|
||||
|
||||
|
||||
def pick_args(tty_input: TTYInput, tool: dict) -> Optional[dict]:
|
||||
@@ -324,9 +451,23 @@ def pick_args(tty_input: TTYInput, tool: dict) -> Optional[dict]:
|
||||
def main():
|
||||
"""Entry point for cf command."""
|
||||
global _ui_out
|
||||
import argparse
|
||||
import subprocess
|
||||
import signal
|
||||
|
||||
# Parse metadata flags before touching stdin or /dev/tty. This keeps
|
||||
# ``cf --help`` and ``cf --version`` usable in packaging checks, CI, and
|
||||
# other non-interactive environments.
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="cf",
|
||||
description="Interactively find and run local or registry CmdForge tools",
|
||||
)
|
||||
from .. import __version__
|
||||
parser.add_argument(
|
||||
"--version", action="version", version=f"%(prog)s {__version__}"
|
||||
)
|
||||
parser.parse_args()
|
||||
|
||||
# Handle Ctrl+C gracefully
|
||||
def handle_sigint(sig, frame):
|
||||
# Restore cursor and exit cleanly
|
||||
@@ -352,6 +493,19 @@ def main():
|
||||
# Use stderr for UI if stdout is piped, so tool output stays clean
|
||||
_ui_out = sys.stderr if not sys.stdout.isatty() else sys.stdout
|
||||
|
||||
try:
|
||||
from ..usage import get_suggestions
|
||||
suggestions = get_suggestions()[:3]
|
||||
if suggestions:
|
||||
_write(f"{GREEN}Frequent pipelines you could save as tools:{RESET}\n")
|
||||
for item in suggestions:
|
||||
_write(
|
||||
f" {' | '.join(item['tools'])} "
|
||||
f"{DIM}({item['count']} uses){RESET}\n"
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
|
||||
try:
|
||||
with TTYInput() as tty_input:
|
||||
result = run_picker(tty_input)
|
||||
|
||||
@@ -353,6 +353,17 @@ def _cmd_providers_discover(args):
|
||||
def _cmd_providers_list(args):
|
||||
"""List all providers and their status."""
|
||||
providers = load_providers()
|
||||
if getattr(args, "json", False):
|
||||
import json
|
||||
payload = []
|
||||
for provider in providers:
|
||||
available, status = _provider_status(provider)
|
||||
item = provider.to_dict()
|
||||
item["available"] = available
|
||||
item["status"] = status
|
||||
payload.append(item)
|
||||
print(json.dumps(payload, indent=2))
|
||||
return 0
|
||||
print(f"Configured providers ({len(providers)}):\n")
|
||||
for p in providers:
|
||||
exists, status = _provider_status(p)
|
||||
@@ -365,6 +376,15 @@ def _cmd_providers_list(args):
|
||||
print(f" Status: {status}")
|
||||
if p.description:
|
||||
print(f" Info: {p.description}")
|
||||
print(f" Locality: {p.locality}")
|
||||
print(f" Data policy: {p.data_policy}")
|
||||
if p.capabilities:
|
||||
print(f" Capabilities: {', '.join(p.capabilities)}")
|
||||
if p.model:
|
||||
identity = p.model
|
||||
if p.model_digest:
|
||||
identity += f" ({p.model_digest})"
|
||||
print(f" Model: {identity}")
|
||||
print()
|
||||
return 0
|
||||
|
||||
@@ -379,6 +399,17 @@ def _cmd_providers_add(args):
|
||||
model = args.model if args.model is not None else (existing.model if existing else None)
|
||||
api_key_env = args.api_key_env if args.api_key_env is not None else (existing.api_key_env if existing else None)
|
||||
tags = args.tags if args.tags is not None else (existing.tags if existing else [])
|
||||
locality = args.locality if args.locality is not None else (existing.locality if existing else "unknown")
|
||||
capabilities = args.capabilities if args.capabilities is not None else (existing.capabilities if existing else [])
|
||||
model_digest = args.model_digest if args.model_digest is not None else (existing.model_digest if existing else None)
|
||||
cost_class = args.cost_class if args.cost_class is not None else (existing.cost_class if existing else None)
|
||||
latency_class = args.latency_class if args.latency_class is not None else (existing.latency_class if existing else None)
|
||||
max_context_tokens = (
|
||||
args.max_context_tokens
|
||||
if args.max_context_tokens is not None
|
||||
else existing.max_context_tokens if existing else None
|
||||
)
|
||||
data_policy = args.data_policy if args.data_policy is not None else (existing.data_policy if existing else "unspecified")
|
||||
fallback = args.fallback if args.fallback is not None else (existing.fallback if existing else None)
|
||||
fallback = fallback or None
|
||||
|
||||
@@ -406,6 +437,15 @@ def _cmd_providers_add(args):
|
||||
fallback_chain=fallback_chain,
|
||||
api_key_env=api_key_env,
|
||||
pty_config=existing.pty_config if existing else None,
|
||||
tools=existing.tools if existing else None,
|
||||
mcp_servers=existing.mcp_servers if existing else None,
|
||||
locality=locality,
|
||||
capabilities=capabilities,
|
||||
model_digest=model_digest,
|
||||
cost_class=cost_class,
|
||||
latency_class=latency_class,
|
||||
max_context_tokens=max_context_tokens,
|
||||
data_policy=data_policy,
|
||||
)
|
||||
add_provider(provider)
|
||||
print(f"Provider '{name}' added/updated.")
|
||||
|
||||
@@ -29,6 +29,12 @@ def cmd_registry(args):
|
||||
return _cmd_registry_update(args)
|
||||
elif args.registry_cmd == "publish":
|
||||
return _cmd_registry_publish(args)
|
||||
elif args.registry_cmd == "signing-key":
|
||||
return _cmd_registry_signing_key(args)
|
||||
elif args.registry_cmd == "improve":
|
||||
return _cmd_registry_improve(args)
|
||||
elif args.registry_cmd == "review-improvement":
|
||||
return _cmd_registry_review_improvement(args)
|
||||
elif args.registry_cmd == "update-readme":
|
||||
return _cmd_registry_update_readme(args)
|
||||
elif args.registry_cmd == "my-tools":
|
||||
@@ -51,6 +57,8 @@ def cmd_registry(args):
|
||||
print(" info <tool> Show tool information")
|
||||
print(" update Update local index cache")
|
||||
print(" publish [path] Publish a tool")
|
||||
print(" signing-key Manage Ed25519 release signing")
|
||||
print(" improve Submit a community improvement")
|
||||
print(" update-readme Update README for published tool(s)")
|
||||
print(" my-tools List your published tools")
|
||||
print(" status <tool> Check moderation status of a tool")
|
||||
@@ -351,6 +359,108 @@ def _cmd_registry_update(args):
|
||||
return 0
|
||||
|
||||
|
||||
def _print_preflight_sections(report: dict, prefix: str = "") -> None:
|
||||
"""Render the common portions of a structured preflight report."""
|
||||
labels = (
|
||||
("errors", "ERROR"),
|
||||
("warnings", "WARN"),
|
||||
("suggestions", "HINT"),
|
||||
)
|
||||
for key, marker in labels:
|
||||
values = report.get(key) or []
|
||||
if values:
|
||||
print(f"{prefix}{key.title()} ({len(values)}):")
|
||||
for value in values:
|
||||
print(f" {marker}: {value}")
|
||||
|
||||
proposal = report.get("contract_proposal") or {}
|
||||
proposed = {
|
||||
key: proposal.get(key)
|
||||
for key in ("input_schema", "output_schema")
|
||||
if proposal.get(key) is not None
|
||||
}
|
||||
if proposed:
|
||||
print(
|
||||
f"{prefix}Contract proposal "
|
||||
f"({proposal.get('confidence', 'low')} confidence; not saved):"
|
||||
)
|
||||
print(yaml.safe_dump(proposed, sort_keys=False).rstrip())
|
||||
|
||||
generated = report.get("generated_tests") or []
|
||||
if generated:
|
||||
print(f"{prefix}Contract conformance ({len(generated)} case(s)):")
|
||||
for result in generated:
|
||||
print(
|
||||
f" {str(result.get('state', 'not_run')).upper()}: "
|
||||
f"{result.get('step', 'unknown')} — {result.get('detail', '')}"
|
||||
)
|
||||
|
||||
regression = report.get("regression") or {}
|
||||
if regression:
|
||||
marker = (
|
||||
"REGRESSION" if regression.get("has_regressions")
|
||||
else "REVIEW" if regression.get("contract_changed")
|
||||
else "STABLE"
|
||||
)
|
||||
print(
|
||||
f"{prefix}Regression comparison: {marker} — "
|
||||
f"{regression.get('summary', 'no changes')}"
|
||||
)
|
||||
|
||||
compatibility = report.get("compatibility") or []
|
||||
if compatibility:
|
||||
print(f"{prefix}ToolStep compatibility ({len(compatibility)}):")
|
||||
for finding in compatibility:
|
||||
print(
|
||||
f" {str(finding.get('state', 'unknown')).upper()}: "
|
||||
f"step {finding.get('step', '?')} -> "
|
||||
f"{finding.get('tool', 'unknown')} — {finding.get('detail', '')}"
|
||||
)
|
||||
|
||||
|
||||
def _print_quality_summary(quality: dict | None) -> None:
|
||||
if not quality:
|
||||
return
|
||||
print(
|
||||
f"Quality {quality.get('score', 0)} — "
|
||||
f"{quality.get('evidence_coverage', 0)}% evidence coverage"
|
||||
)
|
||||
for category in quality.get("categories", []):
|
||||
state = category.get("state", "not_tested")
|
||||
value = (
|
||||
f"{category.get('earned', 0)}/{category.get('available', 0)}"
|
||||
if state == "checked" else state.replace("_", " ")
|
||||
)
|
||||
print(f" {category.get('name', 'Unknown'):24s} {value:>12s}")
|
||||
|
||||
|
||||
def _print_registry_suggestions(suggestions: dict) -> None:
|
||||
"""Render registry-specific category, similarity, and scrutiny evidence."""
|
||||
category = suggestions.get("category") or {}
|
||||
if category.get("suggested"):
|
||||
print(
|
||||
f"Suggested category: {category['suggested']} "
|
||||
f"({category.get('confidence', 0):.0%} confidence)"
|
||||
)
|
||||
|
||||
similar = suggestions.get("similar_tools") or []
|
||||
if similar:
|
||||
print(f"Similar registry tools ({len(similar)}):")
|
||||
for item in similar:
|
||||
print(
|
||||
f" - {item.get('name', 'unknown')} "
|
||||
f"({item.get('similarity', 0):.0%} similar)"
|
||||
)
|
||||
|
||||
scrutiny = suggestions.get("scrutiny") or {}
|
||||
findings = scrutiny.get("findings") or []
|
||||
if findings:
|
||||
print(f"Scrutiny findings ({len(findings)}):")
|
||||
for finding in findings:
|
||||
marker = str(finding.get("result", "info")).upper()
|
||||
print(f" {marker}: {finding.get('message', '')}")
|
||||
|
||||
|
||||
def _cmd_registry_publish(args):
|
||||
"""Publish a tool to the registry."""
|
||||
from ..registry_client import RegistryError, get_client
|
||||
@@ -404,24 +514,80 @@ def _cmd_registry_publish(args):
|
||||
# Validate
|
||||
try:
|
||||
data = yaml.safe_load(config_yaml)
|
||||
if not isinstance(data, dict):
|
||||
print("Error: config.yaml must contain a YAML mapping", file=sys.stderr)
|
||||
return 1
|
||||
name = data.get("name", "")
|
||||
version = data.get("version", "")
|
||||
if not name or not version:
|
||||
print("Error: config.yaml must have 'name' and 'version' fields", file=sys.stderr)
|
||||
return 1
|
||||
# Registry fields are local installation/status metadata. Including a
|
||||
# previous release's hash or owner in a new payload makes content
|
||||
# identity depend on stale registry state.
|
||||
for key in (
|
||||
"registry_hash",
|
||||
"registry_status",
|
||||
"registry_owner",
|
||||
"registry_feedback",
|
||||
):
|
||||
data.pop(key, None)
|
||||
config_yaml = yaml.dump(data, default_flow_style=False, sort_keys=False)
|
||||
except yaml.YAMLError as e:
|
||||
print(f"Error: Invalid YAML in config.yaml: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if args.dry_run:
|
||||
print("Dry run - validating only")
|
||||
print("Dry run — validating locally and via registry preflight")
|
||||
print()
|
||||
print(f"Would publish:")
|
||||
|
||||
# Local validation
|
||||
print(f" Name: {name}")
|
||||
print(f" Version: {version}")
|
||||
print(f" Config: {len(config_yaml)} bytes")
|
||||
print(f" README: {len(readme)} bytes")
|
||||
return 0
|
||||
print()
|
||||
|
||||
from ..preflight import analyze_tool
|
||||
from ..tool import Tool
|
||||
try:
|
||||
local_tool = Tool.from_dict(data)
|
||||
local_tool.path = config_path
|
||||
local_report = analyze_tool(
|
||||
local_tool, include_contract_tests=True
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
print(f"Local preflight error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
_print_preflight_sections(local_report.to_dict(), prefix="Local ")
|
||||
if not local_report.ok:
|
||||
return 1
|
||||
|
||||
# Attempt registry preflight if token is configured
|
||||
config = load_config()
|
||||
if not config.registry.token:
|
||||
print("No registry token configured — local preflight only.")
|
||||
return 0
|
||||
|
||||
try:
|
||||
client = get_client()
|
||||
result = client.publish_tool(
|
||||
config_yaml, readme=readme, defaults=defaults,
|
||||
dry_run=True,
|
||||
)
|
||||
except RegistryError as exc:
|
||||
print(f"Registry preflight failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
remote_report = result.get("preflight") or {}
|
||||
_print_preflight_sections(remote_report, prefix="Registry ")
|
||||
_print_quality_summary(result.get("quality"))
|
||||
_print_registry_suggestions(result.get("suggestions") or {})
|
||||
if not remote_report.get("errors"):
|
||||
print("Registry preflight passed.")
|
||||
return 0
|
||||
return 1
|
||||
|
||||
# Check for token
|
||||
config = load_config()
|
||||
@@ -451,6 +617,7 @@ def _cmd_registry_publish(args):
|
||||
from ..tool import load_tool, Tool, ToolStep
|
||||
|
||||
dep_result = None
|
||||
my_owner = ""
|
||||
try:
|
||||
client = get_client()
|
||||
|
||||
@@ -561,12 +728,70 @@ def _cmd_registry_publish(args):
|
||||
except Exception:
|
||||
pass # Continue with publish
|
||||
|
||||
# Stage 1 runs after dependency publication and any automatic version bump,
|
||||
# so the evidence describes the exact version Stage 2 will mutate.
|
||||
from ..preflight import analyze_tool
|
||||
try:
|
||||
preflight_tool = Tool.from_dict(data)
|
||||
preflight_tool.path = config_path
|
||||
local_report = analyze_tool(preflight_tool, include_contract_tests=True)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
print(f"Local preflight error: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print("Preflight stage:")
|
||||
_print_preflight_sections(local_report.to_dict(), prefix="Local ")
|
||||
if not local_report.ok:
|
||||
return 1
|
||||
|
||||
try:
|
||||
preflight_client = get_client()
|
||||
preflight_result = preflight_client.publish_tool(
|
||||
config_yaml, readme=readme, defaults=defaults,
|
||||
owner=getattr(args, "owner", ""), dry_run=True,
|
||||
)
|
||||
except RegistryError as exc:
|
||||
print(f"Registry preflight failed: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
remote_report = preflight_result.get("preflight") or {}
|
||||
_print_preflight_sections(remote_report, prefix="Registry ")
|
||||
_print_quality_summary(preflight_result.get("quality"))
|
||||
if remote_report.get("errors"):
|
||||
return 1
|
||||
release_content_hash = preflight_result.get("content_hash", "")
|
||||
if sys.stdin.isatty() and not getattr(args, "force", False):
|
||||
try:
|
||||
if input("Publish this validated version? [y/N] ").strip().lower() != "y":
|
||||
print("Cancelled.")
|
||||
return 1
|
||||
except (EOFError, KeyboardInterrupt):
|
||||
print("\nCancelled.")
|
||||
return 1
|
||||
|
||||
print(f"Publishing {name}@{version}...")
|
||||
|
||||
try:
|
||||
client = get_client()
|
||||
owner = getattr(args, "owner", "")
|
||||
result = client.publish_tool(config_yaml, readme, defaults, owner=owner)
|
||||
attestation = None
|
||||
from ..signing import load_signing_key
|
||||
signing_key = load_signing_key()
|
||||
if signing_key:
|
||||
if not release_content_hash:
|
||||
print(
|
||||
"Registry did not return a content identity; refusing to sign.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
from ..attestation import sign_tool
|
||||
signer = owner or my_owner
|
||||
attestation = sign_tool(
|
||||
name, version, release_content_hash, signer, signing_key[0]
|
||||
).to_dict()
|
||||
result = client.publish_tool(
|
||||
config_yaml, readme, defaults, owner=owner,
|
||||
attestation=attestation,
|
||||
improvement_id=getattr(args, "improvement_id", None),
|
||||
)
|
||||
|
||||
pr_url = result.get("pr_url", "")
|
||||
status = result.get("status", "")
|
||||
@@ -641,6 +866,64 @@ def _cmd_registry_publish(args):
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_registry_signing_key(args):
|
||||
"""Initialize a local key and register its public half with the registry."""
|
||||
from ..signing import SIGNING_KEY_FILE, initialize_signing_key, load_signing_key
|
||||
from ..registry_client import RegistryError, get_client
|
||||
|
||||
if args.action == "status":
|
||||
key = load_signing_key()
|
||||
print(
|
||||
f"Signing key: {SIGNING_KEY_FILE}"
|
||||
if key else "No release signing key configured."
|
||||
)
|
||||
return 0 if key else 1
|
||||
try:
|
||||
_, public_key = initialize_signing_key()
|
||||
get_client().set_signing_public_key(public_key)
|
||||
except (OSError, ValueError, RegistryError) as exc:
|
||||
print(f"Could not initialize signing key: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Release signing key initialized: {SIGNING_KEY_FILE}")
|
||||
print("The private key is stored locally with mode 0600; back it up securely.")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_registry_improve(args):
|
||||
from ..registry_client import RegistryError, get_client
|
||||
if "/" not in args.tool:
|
||||
print("Tool must be specified as owner/name.", file=sys.stderr)
|
||||
return 2
|
||||
owner, name = args.tool.split("/", 1)
|
||||
try:
|
||||
proposed = Path(args.file).read_text(encoding="utf-8")
|
||||
result = get_client().submit_improvement(
|
||||
owner, name, args.version, args.step_index, proposed,
|
||||
args.rationale,
|
||||
)
|
||||
except (OSError, RegistryError) as exc:
|
||||
print(f"Could not submit improvement: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Improvement {result.get('id')} tested and submitted for review.")
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_registry_review_improvement(args):
|
||||
from ..registry_client import RegistryError, get_client
|
||||
try:
|
||||
result = get_client().review_improvement(args.id, args.decision, args.notes)
|
||||
except RegistryError as exc:
|
||||
print(f"Could not review improvement: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Improvement {args.id}: {result.get('status')}")
|
||||
if result.get("ready_to_apply"):
|
||||
print(
|
||||
"Publish the updated version with "
|
||||
f"--improvement-id {args.id} to apply credit and badges."
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_registry_update_readme(args):
|
||||
"""Update README for published tool(s) on the registry."""
|
||||
from ..registry_client import RegistryError, get_client
|
||||
|
||||
@@ -5,56 +5,336 @@ from pathlib import Path
|
||||
|
||||
from ..tool import (
|
||||
list_tools, load_tool, save_tool, delete_tool, get_tools_dir,
|
||||
Tool, ToolArgument, PromptStep, CodeStep, ToolStep, validate_tool_name
|
||||
get_project_tools_dir, Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
|
||||
validate_tool_name
|
||||
)
|
||||
from ..gui import run_gui
|
||||
|
||||
|
||||
def _resolve_existing_tool(name: str):
|
||||
"""Resolve project-local before global without registry auto-fetch."""
|
||||
from ..resolver import find_tool
|
||||
|
||||
project_candidate = any(
|
||||
(Path.cwd() / root / name / "config.yaml").exists()
|
||||
for root in (".cmdforge", "cmdforge")
|
||||
)
|
||||
if project_candidate:
|
||||
resolved = find_tool(name)
|
||||
if resolved:
|
||||
return resolved.tool
|
||||
from .. import tool as tool_module
|
||||
return tool_module.load_tool(name)
|
||||
|
||||
|
||||
def _project_tool_names(project_dir=None) -> list[str]:
|
||||
"""List project-local references using the same two resolver roots."""
|
||||
project_dir = (project_dir or Path.cwd()).resolve()
|
||||
names = []
|
||||
for root_name in (".cmdforge", "cmdforge"):
|
||||
root = project_dir / root_name
|
||||
if not root.is_dir():
|
||||
continue
|
||||
for item in sorted(root.iterdir()):
|
||||
if not item.is_dir() or item.name.startswith("."):
|
||||
continue
|
||||
if (item / "config.yaml").is_file():
|
||||
names.append(item.name)
|
||||
continue
|
||||
for child in sorted(item.iterdir()):
|
||||
if child.is_dir() and (child / "config.yaml").is_file():
|
||||
names.append(f"{item.name}/{child.name}")
|
||||
return list(dict.fromkeys(names))
|
||||
|
||||
|
||||
def _execution_policy_from_args(*namespaces):
|
||||
"""Combine top-level and wrapper-compatible execution policy flags."""
|
||||
from ..providers import ProviderExecutionPolicy
|
||||
|
||||
capabilities = []
|
||||
for namespace in namespaces:
|
||||
capabilities.extend(
|
||||
getattr(namespace, "require_capability", None) or []
|
||||
)
|
||||
data_classification = next((
|
||||
getattr(namespace, "data_classification", None)
|
||||
for namespace in namespaces
|
||||
if getattr(namespace, "data_classification", None)
|
||||
), "public")
|
||||
return ProviderExecutionPolicy(
|
||||
allow_fallback=not any(
|
||||
getattr(namespace, "no_fallback", False) for namespace in namespaces
|
||||
),
|
||||
required_locality=(
|
||||
"local" if any(
|
||||
getattr(namespace, "require_local", False)
|
||||
for namespace in namespaces
|
||||
) else None
|
||||
),
|
||||
required_capabilities=tuple(dict.fromkeys(capabilities)),
|
||||
data_classification=data_classification,
|
||||
require_model_identity=any(
|
||||
getattr(namespace, "require_model_identity", False)
|
||||
for namespace in namespaces
|
||||
),
|
||||
require_model_digest=any(
|
||||
getattr(namespace, "require_model_digest", False)
|
||||
for namespace in namespaces
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _result_envelope(
|
||||
output: str, provider_calls: list[dict], exit_code: int = 0
|
||||
) -> str:
|
||||
"""Build a stable machine-readable envelope without parsing tool output."""
|
||||
import json
|
||||
|
||||
execution = {
|
||||
"exit_code": exit_code,
|
||||
"provider_calls": provider_calls,
|
||||
"fallback_used": any(
|
||||
call.get("fallback_used", False) for call in provider_calls
|
||||
),
|
||||
"actual_providers": list(dict.fromkeys(
|
||||
call["actual_provider"] for call in provider_calls
|
||||
if call.get("actual_provider")
|
||||
)),
|
||||
}
|
||||
return json.dumps({"output": output, "execution": execution}, indent=2)
|
||||
|
||||
|
||||
def cmd_list(args):
|
||||
"""List all tools."""
|
||||
tools = list_tools()
|
||||
import json
|
||||
|
||||
if not tools:
|
||||
print("No tools found.")
|
||||
print("Create your first tool with: cmdforge ui")
|
||||
filter_text = (getattr(args, "filter", "") or "").casefold()
|
||||
limit = getattr(args, "limit", None)
|
||||
if getattr(args, "full", False) and not getattr(args, "json", False):
|
||||
print("Error: --full requires --json.", file=sys.stderr)
|
||||
return 1
|
||||
if limit is not None and limit < 1:
|
||||
print("Error: --limit must be at least 1.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
matches = []
|
||||
project_names = _project_tool_names()
|
||||
all_names = list(dict.fromkeys([*project_names, *sorted(list_tools())]))
|
||||
for name in all_names:
|
||||
tool = _resolve_existing_tool(name)
|
||||
if not tool:
|
||||
continue
|
||||
if filter_text and (
|
||||
filter_text not in name.casefold()
|
||||
and filter_text not in (tool.description or "").casefold()
|
||||
):
|
||||
continue
|
||||
matches.append((name, tool))
|
||||
if limit is not None and len(matches) >= limit:
|
||||
break
|
||||
|
||||
if getattr(args, "json", False):
|
||||
full = getattr(args, "full", False)
|
||||
result = [
|
||||
_full_tool_entry(name, tool) if full else _catalog_tool_entry(name, tool)
|
||||
for name, tool in matches
|
||||
]
|
||||
json.dump(result, sys.stdout, indent=2)
|
||||
print()
|
||||
return 0
|
||||
|
||||
print(f"Available tools ({len(tools)}):\n")
|
||||
for name in tools:
|
||||
tool = load_tool(name)
|
||||
if tool:
|
||||
# Show source indicator for imported tools
|
||||
source_marker = ""
|
||||
if tool.source and tool.source.type == "imported":
|
||||
source_marker = " [imported]"
|
||||
elif tool.source and tool.source.type == "forked":
|
||||
source_marker = " [forked]"
|
||||
if not matches:
|
||||
if filter_text:
|
||||
print(f"No tools match filter '{getattr(args, 'filter', '')}'.")
|
||||
else:
|
||||
print("No tools found.")
|
||||
print("Create your first tool with: cmdforge ui")
|
||||
return 0
|
||||
|
||||
print(f" {name}{source_marker}")
|
||||
print(f" {tool.description or 'No description'}")
|
||||
print(f"Available tools ({len(matches)}):\n")
|
||||
for name, tool in matches:
|
||||
# Show source indicator for imported tools
|
||||
source_marker = ""
|
||||
if tool.source and tool.source.type == "imported":
|
||||
source_marker = " [imported]"
|
||||
elif tool.source and tool.source.type == "forked":
|
||||
source_marker = " [forked]"
|
||||
|
||||
# Show arguments
|
||||
if tool.arguments:
|
||||
args_str = ", ".join(arg.flag for arg in tool.arguments)
|
||||
print(f" Arguments: {args_str}")
|
||||
print(f" {name}{source_marker}")
|
||||
print(f" {tool.description or 'No description'}")
|
||||
|
||||
# Show steps
|
||||
if tool.steps:
|
||||
step_info = []
|
||||
for step in tool.steps:
|
||||
if isinstance(step, PromptStep):
|
||||
step_info.append(f"PROMPT[{step.provider}]")
|
||||
elif isinstance(step, CodeStep):
|
||||
step_info.append("CODE")
|
||||
elif isinstance(step, ToolStep):
|
||||
step_info.append(f"TOOL[{step.tool}]")
|
||||
print(f" Steps: {' -> '.join(step_info)}")
|
||||
# Show arguments
|
||||
if tool.arguments:
|
||||
args_str = ", ".join(arg.flag for arg in tool.arguments)
|
||||
print(f" Arguments: {args_str}")
|
||||
|
||||
print()
|
||||
# Show steps
|
||||
if tool.steps:
|
||||
step_info = []
|
||||
for step in tool.steps:
|
||||
if isinstance(step, PromptStep):
|
||||
step_info.append(f"PROMPT[{step.provider}]")
|
||||
elif isinstance(step, CodeStep):
|
||||
step_info.append("CODE")
|
||||
elif isinstance(step, ToolStep):
|
||||
step_info.append(f"TOOL[{step.tool}]")
|
||||
print(f" Steps: {' -> '.join(step_info)}")
|
||||
|
||||
print()
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
def _catalog_tool_entry(name, tool) -> dict:
|
||||
"""Return the stable, compact discovery contract intended for agents."""
|
||||
entry = {
|
||||
"name": name,
|
||||
"description": tool.description or "",
|
||||
"version": tool.version or "",
|
||||
"category": tool.category or "Other",
|
||||
"arguments": [argument.to_dict() for argument in tool.arguments],
|
||||
"input_schema": tool.input_schema,
|
||||
"output_schema": tool.output_schema,
|
||||
"deprecated": bool(tool.deprecated),
|
||||
}
|
||||
if tool.replacement:
|
||||
entry["replacement"] = tool.replacement
|
||||
return entry
|
||||
|
||||
|
||||
def _full_tool_entry(name, tool) -> dict:
|
||||
"""Return an explicitly requested complete definition."""
|
||||
entry = tool.to_dict()
|
||||
# list_tools() owns the canonical local/qualified reference. A config's
|
||||
# internal name may be only the short name.
|
||||
entry["name"] = name
|
||||
return entry
|
||||
|
||||
|
||||
def cmd_run_once(args):
|
||||
"""Run a one-shot prompt through a provider without creating a tool."""
|
||||
from ..providers import call_provider
|
||||
|
||||
timeout = getattr(args, "timeout", 60)
|
||||
max_tokens = getattr(args, "max_tokens", None)
|
||||
if timeout < 1 or timeout > 3600:
|
||||
print("Error: --timeout must be between 1 and 3600 seconds.", file=sys.stderr)
|
||||
return 1
|
||||
if max_tokens is not None and not 1 <= max_tokens <= 1_000_000:
|
||||
print("Error: --max-tokens must be between 1 and 1000000.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
try:
|
||||
prompt = _build_one_shot_prompt(
|
||||
args.prompt, getattr(args, "input", None), sys.stdin
|
||||
)
|
||||
except (OSError, ValueError) as exc:
|
||||
print(f"Error reading input: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if not prompt.strip():
|
||||
print("Error: No prompt provided.", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
provider = _resolve_one_shot_provider(getattr(args, "provider", None))
|
||||
if not provider:
|
||||
print(
|
||||
"Error: No usable provider found. Pass --provider, set "
|
||||
"'cmdforge config set default_provider NAME', or run "
|
||||
"'cmdforge providers install'.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 1
|
||||
|
||||
print(f"[run-once] Provider: {provider}", file=sys.stderr)
|
||||
policy = _execution_policy_from_args(args)
|
||||
result = call_provider(
|
||||
provider, prompt, timeout=timeout, max_tokens=max_tokens,
|
||||
execution_policy=policy,
|
||||
)
|
||||
|
||||
if not result.success:
|
||||
if getattr(args, "result_envelope", None) == "json":
|
||||
print(_result_envelope("", [result.provenance()], exit_code=1))
|
||||
print(f"Error: {result.error}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
if getattr(args, "result_envelope", None) == "json":
|
||||
print(_result_envelope(result.text, [result.provenance()]))
|
||||
else:
|
||||
print(result.text)
|
||||
return 0
|
||||
|
||||
|
||||
def _build_one_shot_prompt(template: str, input_path, stdin) -> str:
|
||||
"""Combine a one-shot instruction with explicit or piped input."""
|
||||
if template == "-":
|
||||
if input_path and input_path != "-":
|
||||
raise ValueError("prompt '-' cannot be combined with --input")
|
||||
return stdin.read()
|
||||
|
||||
input_text = None
|
||||
if input_path:
|
||||
input_text = (
|
||||
stdin.read()
|
||||
if input_path == "-"
|
||||
else Path(input_path).read_text(encoding="utf-8")
|
||||
)
|
||||
elif not stdin.isatty():
|
||||
input_text = stdin.read()
|
||||
|
||||
if not input_text:
|
||||
return template
|
||||
if "{input}" in template:
|
||||
return template.replace("{input}", input_text)
|
||||
return f"{template.rstrip()}\n\n{input_text}"
|
||||
|
||||
|
||||
def _resolve_one_shot_provider(explicit: str = None):
|
||||
"""Honor the configured default, otherwise choose an available provider."""
|
||||
if explicit:
|
||||
return explicit
|
||||
|
||||
from ..config import load_config
|
||||
from ..providers import PRESET_CHAINS, load_providers
|
||||
|
||||
configured = load_config().default_provider
|
||||
if configured:
|
||||
return configured
|
||||
|
||||
providers = {provider.name: provider for provider in load_providers()}
|
||||
ordered_names = list(PRESET_CHAINS["balanced"][:-1])
|
||||
ordered_names.extend(
|
||||
name for name in providers if name not in ordered_names and name != "mock"
|
||||
)
|
||||
for name in ordered_names:
|
||||
provider = providers.get(name)
|
||||
if provider and _provider_is_available(provider):
|
||||
return name
|
||||
return None
|
||||
|
||||
|
||||
def _provider_is_available(provider) -> bool:
|
||||
"""Perform a non-invasive availability check without contacting a model."""
|
||||
import os
|
||||
import shlex
|
||||
import shutil
|
||||
|
||||
if provider.type == "api":
|
||||
key_name = provider.api_key_env or (
|
||||
f"{provider.name.upper().replace('-', '_')}_API_KEY"
|
||||
)
|
||||
return bool(os.environ.get(key_name))
|
||||
try:
|
||||
parts = shlex.split(os.path.expandvars(provider.command))
|
||||
except ValueError:
|
||||
return False
|
||||
if not parts:
|
||||
return False
|
||||
executable = Path(os.path.expanduser(parts[0]))
|
||||
return bool(shutil.which(str(executable)) or executable.is_file())
|
||||
|
||||
|
||||
def cmd_create(args):
|
||||
"""Create a new tool (basic CLI creation - use 'ui' for full builder)."""
|
||||
name = args.name
|
||||
@@ -65,10 +345,22 @@ def cmd_create(args):
|
||||
print(f"Error: Invalid tool name '{name}': {error_msg}")
|
||||
return 1
|
||||
|
||||
# Check if already exists
|
||||
existing = load_tool(name)
|
||||
if existing and not args.force:
|
||||
print(f"Error: Tool '{name}' already exists. Use --force to overwrite.")
|
||||
output_dir = getattr(args, "output_dir", None)
|
||||
project = getattr(args, "project", False)
|
||||
target_root = (
|
||||
Path(output_dir).resolve()
|
||||
if output_dir is not None
|
||||
else get_project_tools_dir() if project else get_tools_dir()
|
||||
)
|
||||
|
||||
# Check only the requested scope. A project tool may intentionally shadow
|
||||
# a global tool with the same name.
|
||||
config_path = target_root / name / "config.yaml"
|
||||
if config_path.exists() and not args.force:
|
||||
print(
|
||||
f"Error: Tool '{name}' already exists at {config_path}. "
|
||||
"Use --force to overwrite."
|
||||
)
|
||||
return 1
|
||||
|
||||
# Create a tool with a single prompt step
|
||||
@@ -89,11 +381,19 @@ def cmd_create(args):
|
||||
output="{response}" if steps else "{input}"
|
||||
)
|
||||
|
||||
path = save_tool(tool)
|
||||
path = save_tool(tool, tools_dir=target_root)
|
||||
print(f"Created tool '{name}'")
|
||||
print(f"Config: {path}")
|
||||
print(f"\nUse 'cmdforge ui' to add arguments, steps, and customize.")
|
||||
print(f"Or run: {name} < input.txt")
|
||||
print("\nUse 'cmdforge ui' to add arguments, steps, and customize.")
|
||||
if project:
|
||||
print(f"Run from its project: cmdforge run {name} < input.txt")
|
||||
elif output_dir is not None:
|
||||
print(
|
||||
"Saved under the requested tool root. CmdForge automatically "
|
||||
"discovers roots named .cmdforge or cmdforge."
|
||||
)
|
||||
else:
|
||||
print(f"Or run: {name} < input.txt")
|
||||
|
||||
return 0
|
||||
|
||||
@@ -146,7 +446,7 @@ def cmd_delete(args):
|
||||
|
||||
def cmd_test(args):
|
||||
"""Test a tool with mock provider."""
|
||||
tool = load_tool(args.name)
|
||||
tool = _resolve_existing_tool(args.name)
|
||||
if not tool:
|
||||
print(f"Error: Tool '{args.name}' not found.")
|
||||
return 1
|
||||
@@ -183,10 +483,14 @@ def cmd_run(args):
|
||||
"""Run a tool."""
|
||||
from ..runner import collect_custom_args, create_argument_parser, run_tool
|
||||
|
||||
tool = load_tool(args.name)
|
||||
tool = _resolve_existing_tool(args.name)
|
||||
if not tool:
|
||||
print(f"Error: Tool '{args.name}' not found.", file=sys.stderr)
|
||||
return 1
|
||||
from ..tool import deprecation_warning
|
||||
warning = deprecation_warning(tool)
|
||||
if warning:
|
||||
print(warning, file=sys.stderr)
|
||||
|
||||
tool_args = list(args.tool_args or [])
|
||||
if tool_args and tool_args[0] == '--':
|
||||
@@ -223,6 +527,12 @@ def cmd_run(args):
|
||||
show_prompt = args.show_prompt or parsed_tool_args.show_prompt
|
||||
verbose = args.verbose or parsed_tool_args.verbose
|
||||
auto_install = args.auto_install or parsed_tool_args.auto_install
|
||||
execution_policy = _execution_policy_from_args(args, parsed_tool_args)
|
||||
result_envelope = (
|
||||
getattr(args, "result_envelope", None)
|
||||
or getattr(parsed_tool_args, "result_envelope", None)
|
||||
)
|
||||
provider_calls = []
|
||||
|
||||
# Run tool
|
||||
output, code = run_tool(
|
||||
@@ -233,16 +543,22 @@ def cmd_run(args):
|
||||
dry_run=dry_run,
|
||||
show_prompt=show_prompt,
|
||||
verbose=verbose,
|
||||
auto_install=auto_install
|
||||
auto_install=auto_install,
|
||||
execution_policy=execution_policy,
|
||||
provenance_log=provider_calls,
|
||||
)
|
||||
|
||||
# Write output
|
||||
if code == 0 and output:
|
||||
if result_envelope == "json" or (code == 0 and output):
|
||||
rendered_output = (
|
||||
_result_envelope(output, provider_calls, exit_code=code)
|
||||
if result_envelope == "json" else output
|
||||
)
|
||||
output_file = args.output or parsed_tool_args.output_file
|
||||
if output_file:
|
||||
Path(output_file).write_text(output)
|
||||
Path(output_file).write_text(rendered_output)
|
||||
else:
|
||||
print(output)
|
||||
print(rendered_output)
|
||||
|
||||
return code
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
"""CLI for opt-in local usage discovery."""
|
||||
|
||||
from ..usage import clear_usage, get_suggestions, is_enabled, set_enabled
|
||||
|
||||
|
||||
def cmd_usage(args):
|
||||
action = getattr(args, "usage_cmd", None) or "status"
|
||||
if action == "enable":
|
||||
set_enabled(True)
|
||||
print("Local pipeline discovery enabled. No input, output, arguments, or telemetry is recorded.")
|
||||
return 0
|
||||
if action == "disable":
|
||||
set_enabled(False)
|
||||
print("Local pipeline discovery disabled.")
|
||||
return 0
|
||||
if action == "clear":
|
||||
clear_usage()
|
||||
print("Local usage history cleared.")
|
||||
return 0
|
||||
if action == "suggestions":
|
||||
suggestions = get_suggestions(getattr(args, "threshold", 3))
|
||||
if not suggestions:
|
||||
print("No frequent pipeline suggestions yet.")
|
||||
return 0
|
||||
for index, item in enumerate(suggestions, start=1):
|
||||
print(f"{index}. {' | '.join(item['tools'])} ({item['count']} uses)")
|
||||
return 0
|
||||
print(f"Local pipeline discovery: {'enabled' if is_enabled() else 'disabled'}")
|
||||
return 0
|
||||
@@ -0,0 +1,194 @@
|
||||
"""Community improvement workflow (M9.3).
|
||||
|
||||
Allows community members to submit suggested prompt/code improvements.
|
||||
Submissions are auto-tested before review. Successful improvements
|
||||
credit both original and improving author.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .tool import Tool, PromptStep, CodeStep
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImprovementSubmission:
|
||||
"""A community-submitted improvement for a tool."""
|
||||
|
||||
tool_name: str
|
||||
tool_version: str
|
||||
submitter: str
|
||||
step_index: int
|
||||
step_type: str # "prompt" | "code"
|
||||
original: str
|
||||
proposed: str
|
||||
rationale: str = ""
|
||||
submitted_at: str = ""
|
||||
status: str = "pending" # pending | tested | approved | rejected
|
||||
test_result: Optional[dict] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"tool_name": self.tool_name,
|
||||
"tool_version": self.tool_version,
|
||||
"submitter": self.submitter,
|
||||
"step_index": self.step_index,
|
||||
"step_type": self.step_type,
|
||||
"original": self.original[:200],
|
||||
"proposed": self.proposed[:200],
|
||||
"rationale": self.rationale,
|
||||
"submitted_at": self.submitted_at,
|
||||
"status": self.status,
|
||||
"test_result": self.test_result,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class SubmissionReview:
|
||||
"""Review result for an improvement submission."""
|
||||
|
||||
submission: ImprovementSubmission
|
||||
decision: str # "approve" | "reject" | "request_changes"
|
||||
reviewer: str = ""
|
||||
notes: str = ""
|
||||
reviewed_at: str = ""
|
||||
|
||||
|
||||
def create_submission(
|
||||
tool: Tool,
|
||||
step_index: int,
|
||||
proposed: str,
|
||||
submitter: str,
|
||||
rationale: str = "",
|
||||
) -> ImprovementSubmission:
|
||||
"""Create a new improvement submission."""
|
||||
if step_index < 0 or step_index >= len(tool.steps):
|
||||
raise ValueError(f"Step index {step_index} out of range (0-{len(tool.steps)-1})")
|
||||
|
||||
step = tool.steps[step_index]
|
||||
if isinstance(step, PromptStep):
|
||||
original = step.prompt
|
||||
step_type = "prompt"
|
||||
elif isinstance(step, CodeStep):
|
||||
original = step.code
|
||||
step_type = "code"
|
||||
else:
|
||||
raise ValueError(f"Step {step_index} is not a prompt or code step")
|
||||
|
||||
return ImprovementSubmission(
|
||||
tool_name=tool.name,
|
||||
tool_version=tool.version or "",
|
||||
submitter=submitter,
|
||||
step_index=step_index,
|
||||
step_type=step_type,
|
||||
original=original,
|
||||
proposed=proposed,
|
||||
rationale=rationale,
|
||||
submitted_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
|
||||
|
||||
def run_submission_tests(submission: ImprovementSubmission, tool: Tool) -> dict:
|
||||
"""Run contract tests on the tool with the proposed change applied.
|
||||
|
||||
Returns a dict with test results comparing baseline vs proposed.
|
||||
"""
|
||||
import copy
|
||||
|
||||
from .contract_testing import run_contract_tests
|
||||
|
||||
if submission.tool_name != tool.name or submission.tool_version != (tool.version or ""):
|
||||
raise ValueError("Submission targets a different tool or version")
|
||||
|
||||
# Baseline
|
||||
baseline = run_contract_tests(tool)
|
||||
baseline_passed = sum(
|
||||
1 for r in baseline.results if r.state == "passed"
|
||||
)
|
||||
|
||||
# Apply proposed change
|
||||
tool_copy = copy.deepcopy(tool)
|
||||
if submission.step_index >= len(tool_copy.steps):
|
||||
return {"error": "Step index out of range", "baseline_passed": baseline_passed}
|
||||
|
||||
step = tool_copy.steps[submission.step_index]
|
||||
current = step.prompt if isinstance(step, PromptStep) else (
|
||||
step.code if isinstance(step, CodeStep) else None
|
||||
)
|
||||
if current != submission.original:
|
||||
return {
|
||||
"error": "Original step has changed since submission",
|
||||
"baseline_passed": baseline_passed,
|
||||
}
|
||||
if isinstance(step, PromptStep) and submission.step_type == "prompt":
|
||||
step.prompt = submission.proposed
|
||||
elif isinstance(step, CodeStep) and submission.step_type == "code":
|
||||
try:
|
||||
compile(submission.proposed, f"<{tool.name}:step-{submission.step_index}>", "exec")
|
||||
except SyntaxError as exc:
|
||||
result = {
|
||||
"error": f"Proposed code has invalid syntax: {exc.msg}",
|
||||
"baseline_passed": baseline_passed,
|
||||
"passed_for_review": False,
|
||||
}
|
||||
submission.test_result = result
|
||||
return result
|
||||
step.code = submission.proposed
|
||||
else:
|
||||
return {"error": "Step type mismatch", "baseline_passed": baseline_passed}
|
||||
|
||||
# Test with proposed change
|
||||
proposed_result = run_contract_tests(tool_copy)
|
||||
proposed_passed = sum(
|
||||
1 for r in proposed_result.results if r.state == "passed"
|
||||
)
|
||||
|
||||
result = {
|
||||
"baseline_passed": baseline_passed,
|
||||
"proposed_passed": proposed_passed,
|
||||
"improvement": proposed_passed - baseline_passed,
|
||||
"regressed": proposed_passed < baseline_passed,
|
||||
"baseline_outcome": baseline.outcome,
|
||||
"proposed_outcome": proposed_result.outcome,
|
||||
"passed_for_review": (
|
||||
(
|
||||
proposed_result.outcome == "passed"
|
||||
and proposed_passed >= baseline_passed
|
||||
)
|
||||
or (
|
||||
submission.step_type == "code"
|
||||
and proposed_result.outcome == "unsupported"
|
||||
)
|
||||
),
|
||||
"details": [r.to_dict() for r in proposed_result.results],
|
||||
}
|
||||
|
||||
submission.status = "tested"
|
||||
submission.test_result = result
|
||||
return result
|
||||
|
||||
|
||||
def review_submission(
|
||||
submission: ImprovementSubmission,
|
||||
decision: str,
|
||||
reviewer: str,
|
||||
notes: str = "",
|
||||
) -> SubmissionReview:
|
||||
"""Review a tested submission."""
|
||||
if decision not in ("approve", "reject", "request_changes"):
|
||||
raise ValueError("Decision must be: approve, reject, or request_changes")
|
||||
|
||||
if submission.status != "tested" or not (
|
||||
submission.test_result or {}
|
||||
).get("passed_for_review"):
|
||||
raise ValueError("Submission must be tested before review")
|
||||
|
||||
submission.status = "approved" if decision == "approve" else decision
|
||||
return SubmissionReview(
|
||||
submission=submission,
|
||||
decision=decision,
|
||||
reviewer=reviewer,
|
||||
notes=notes,
|
||||
reviewed_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
@@ -0,0 +1,906 @@
|
||||
"""Deterministic, side-effect-safe contract conformance verification.
|
||||
|
||||
The verifier generates inputs that are validated against ``input_schema``,
|
||||
replaces prompt steps with schema-derived deterministic values, executes the
|
||||
remaining safe pipeline, and validates its final result. It never calls an AI
|
||||
provider. Steps that could execute user code or contact another tool/server are
|
||||
reported as unsupported instead of being run implicitly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from datetime import date, datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Literal, Optional
|
||||
from uuid import UUID
|
||||
|
||||
from .tool import CodeStep, McpStep, PromptStep, Tool, ToolStep
|
||||
|
||||
|
||||
ConformanceState = Literal["passed", "failed", "unsupported", "not_run"]
|
||||
_VALID_STATES = ("passed", "failed", "unsupported", "not_run")
|
||||
|
||||
|
||||
class UnsupportedContract(ValueError):
|
||||
"""Raised when a valid deterministic value cannot be produced safely."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConformanceResult:
|
||||
step: str
|
||||
state: ConformanceState
|
||||
detail: str = ""
|
||||
case_id: str = ""
|
||||
input_value: Any = None
|
||||
input_fingerprint: str = ""
|
||||
output_shape: Optional[Dict[str, Any]] = None
|
||||
output_fingerprint: str = ""
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.state not in _VALID_STATES:
|
||||
raise ValueError(f"Unknown conformance state: {self.state}")
|
||||
|
||||
def is_pass(self) -> bool:
|
||||
return self.state == "passed"
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ConformanceReport:
|
||||
results: List[ConformanceResult] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def outcome(self) -> ConformanceState:
|
||||
states = {result.state for result in self.results}
|
||||
if "failed" in states:
|
||||
return "failed"
|
||||
if "passed" in states:
|
||||
return "passed"
|
||||
if "unsupported" in states:
|
||||
return "unsupported"
|
||||
return "not_run"
|
||||
|
||||
@property
|
||||
def passed(self) -> bool:
|
||||
return self.outcome == "passed"
|
||||
|
||||
@property
|
||||
def failed(self) -> bool:
|
||||
return self.outcome == "failed"
|
||||
|
||||
@property
|
||||
def summary(self) -> Dict[str, int]:
|
||||
counts = {state: 0 for state in _VALID_STATES}
|
||||
for result in self.results:
|
||||
counts[result.state] += 1
|
||||
return counts
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"outcome": self.outcome,
|
||||
"summary": self.summary,
|
||||
"results": [result.to_dict() for result in self.results],
|
||||
}
|
||||
|
||||
|
||||
def run_contract_tests(
|
||||
tool: Tool, test_inputs: Optional[List[Any]] = None
|
||||
) -> ConformanceReport:
|
||||
"""Run safe structural conformance tests for a tool.
|
||||
|
||||
This is deliberately not a semantic quality test. Original CodeStep,
|
||||
ToolStep, and McpStep instances are not executed because they may have side
|
||||
effects, make network calls, or invoke real providers.
|
||||
"""
|
||||
if tool.input_schema is None or tool.output_schema is None:
|
||||
return _single_result(
|
||||
"preflight",
|
||||
"not_run",
|
||||
"Both input_schema and output_schema are required",
|
||||
)
|
||||
|
||||
unsafe = next(
|
||||
(
|
||||
type(step).__name__
|
||||
for step in tool.steps
|
||||
if isinstance(step, (CodeStep, ToolStep, McpStep))
|
||||
),
|
||||
None,
|
||||
)
|
||||
if unsafe:
|
||||
return _single_result(
|
||||
"preflight",
|
||||
"unsupported",
|
||||
f"Safe conformance mode does not execute {unsafe}",
|
||||
)
|
||||
|
||||
generation_issue = None
|
||||
try:
|
||||
generated_cases = _generate_inputs(tool.input_schema)
|
||||
except UnsupportedContract as exc:
|
||||
if not test_inputs:
|
||||
return _single_result("input-generation", "unsupported", str(exc))
|
||||
generated_cases = []
|
||||
generation_issue = str(exc)
|
||||
|
||||
test_cases = list(copy.deepcopy(test_inputs or [])) + generated_cases
|
||||
deduplicated = {}
|
||||
for test_input in test_cases:
|
||||
deduplicated[_fingerprint(test_input)] = test_input
|
||||
test_cases = list(deduplicated.values())
|
||||
|
||||
if not test_cases:
|
||||
return _single_result(
|
||||
"input-generation", "not_run", "No deterministic inputs generated"
|
||||
)
|
||||
|
||||
report = ConformanceReport()
|
||||
if generation_issue:
|
||||
report.results.append(ConformanceResult(
|
||||
"input-generation", "unsupported", generation_issue
|
||||
))
|
||||
for test_input in test_cases:
|
||||
input_fingerprint = _fingerprint(test_input)
|
||||
case_id = f"case-{input_fingerprint[:12]}"
|
||||
evidence = {
|
||||
"case_id": case_id,
|
||||
"input_value": copy.deepcopy(test_input),
|
||||
"input_fingerprint": input_fingerprint,
|
||||
}
|
||||
try:
|
||||
_validate_instance(test_input, tool.input_schema, "input")
|
||||
except UnsupportedContract as exc:
|
||||
report.results.append(ConformanceResult(
|
||||
case_id, "unsupported", str(exc), **evidence
|
||||
))
|
||||
continue
|
||||
except ValueError as exc:
|
||||
report.results.append(ConformanceResult(
|
||||
case_id, "failed", f"Stored {exc}", **evidence
|
||||
))
|
||||
continue
|
||||
try:
|
||||
result = _run_with_stub(tool, test_input)
|
||||
except UnsupportedContract as exc:
|
||||
report.results.append(
|
||||
ConformanceResult(case_id, "unsupported", str(exc), **evidence)
|
||||
)
|
||||
except Exception as exc:
|
||||
report.results.append(
|
||||
ConformanceResult(
|
||||
case_id, "failed", f"Execution error: {exc}", **evidence
|
||||
)
|
||||
)
|
||||
else:
|
||||
output_evidence = {
|
||||
"output_shape": _json_shape(result),
|
||||
"output_fingerprint": _fingerprint(result),
|
||||
}
|
||||
try:
|
||||
_validate_instance(result, tool.output_schema, "output")
|
||||
except UnsupportedContract as exc:
|
||||
report.results.append(
|
||||
ConformanceResult(
|
||||
case_id, "unsupported", str(exc),
|
||||
**evidence, **output_evidence,
|
||||
)
|
||||
)
|
||||
except ValueError as exc:
|
||||
report.results.append(ConformanceResult(
|
||||
case_id, "failed", str(exc), **evidence, **output_evidence
|
||||
))
|
||||
else:
|
||||
report.results.append(
|
||||
ConformanceResult(
|
||||
case_id, "passed", "Output conforms to output_schema",
|
||||
**evidence, **output_evidence,
|
||||
)
|
||||
)
|
||||
return report
|
||||
|
||||
|
||||
def _single_result(
|
||||
step: str, state: ConformanceState, detail: str
|
||||
) -> ConformanceReport:
|
||||
return ConformanceReport([ConformanceResult(step, state, detail)])
|
||||
|
||||
|
||||
def _canonical_json(value: Any) -> str:
|
||||
return json.dumps(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False,
|
||||
default=str,
|
||||
)
|
||||
|
||||
|
||||
def _fingerprint(value: Any) -> str:
|
||||
return hashlib.sha256(_canonical_json(value).encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _json_shape(value: Any) -> Dict[str, Any]:
|
||||
if value is None:
|
||||
return {"type": "null"}
|
||||
if isinstance(value, bool):
|
||||
return {"type": "boolean"}
|
||||
if isinstance(value, int):
|
||||
return {"type": "integer"}
|
||||
if isinstance(value, float):
|
||||
return {"type": "number"}
|
||||
if isinstance(value, str):
|
||||
return {"type": "string"}
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
str(key): _json_shape(item)
|
||||
for key, item in sorted(value.items(), key=lambda pair: str(pair[0]))
|
||||
},
|
||||
}
|
||||
if isinstance(value, list):
|
||||
shapes = {
|
||||
_canonical_json(_json_shape(item)): _json_shape(item) for item in value
|
||||
}
|
||||
return {
|
||||
"type": "array",
|
||||
"items": [shapes[key] for key in sorted(shapes)],
|
||||
}
|
||||
return {"type": type(value).__name__}
|
||||
|
||||
|
||||
def _generate_inputs(schema: dict) -> List[Any]:
|
||||
"""Generate deterministic values and validate every returned candidate."""
|
||||
candidates: List[Any] = []
|
||||
schema_type = _schema_type(schema)
|
||||
|
||||
if schema_type == "object":
|
||||
enum_properties = {
|
||||
name: spec["enum"]
|
||||
for name, spec in schema.get("properties", {}).items()
|
||||
if isinstance(spec, dict) and spec.get("enum")
|
||||
}
|
||||
variant_count = min(max((len(v) for v in enum_properties.values()), default=1), 3)
|
||||
for index in range(variant_count):
|
||||
overrides = {
|
||||
name: values[index % len(values)]
|
||||
for name, values in enum_properties.items()
|
||||
}
|
||||
candidates.append(_object_value(schema, overrides))
|
||||
else:
|
||||
candidates.append(_value_for_schema(schema, "input"))
|
||||
|
||||
unique: List[Any] = []
|
||||
seen = set()
|
||||
for candidate in candidates:
|
||||
try:
|
||||
_validate_instance(candidate, schema, "generated input")
|
||||
except ValueError as exc:
|
||||
raise UnsupportedContract(
|
||||
f"Could not generate a valid input: {exc}"
|
||||
) from exc
|
||||
marker = json.dumps(candidate, sort_keys=True, default=str)
|
||||
if marker not in seen:
|
||||
seen.add(marker)
|
||||
unique.append(candidate)
|
||||
return unique
|
||||
|
||||
|
||||
def _schema_type(schema: dict) -> str | None:
|
||||
schema_type = schema.get("type")
|
||||
if isinstance(schema_type, list):
|
||||
return next((value for value in schema_type if value != "null"), "null")
|
||||
if schema_type is None and ("properties" in schema or "required" in schema):
|
||||
return "object"
|
||||
return schema_type
|
||||
|
||||
|
||||
def _value_for_schema(schema: dict, name: str) -> Any:
|
||||
for keyword in ("const", "default"):
|
||||
if keyword in schema:
|
||||
return copy.deepcopy(schema[keyword])
|
||||
if schema.get("examples"):
|
||||
return copy.deepcopy(schema["examples"][0])
|
||||
if schema.get("enum"):
|
||||
return copy.deepcopy(schema["enum"][0])
|
||||
|
||||
alternatives = schema.get("oneOf") or schema.get("anyOf")
|
||||
if alternatives:
|
||||
for alternative in alternatives:
|
||||
try:
|
||||
candidate = _value_for_schema(alternative, name)
|
||||
_validate_instance(candidate, schema, name)
|
||||
return candidate
|
||||
except (UnsupportedContract, ValueError):
|
||||
continue
|
||||
raise UnsupportedContract(f"Cannot synthesize {name} from alternatives")
|
||||
|
||||
if schema.get("allOf"):
|
||||
merged = _merge_all_of(schema)
|
||||
candidate = _value_for_schema(merged, name)
|
||||
_validate_instance(candidate, schema, name)
|
||||
return candidate
|
||||
|
||||
schema_type = _schema_type(schema)
|
||||
if schema_type == "object":
|
||||
return _object_value(schema, {})
|
||||
if schema_type == "array":
|
||||
item_schema = schema.get("items", {})
|
||||
count = max(0, schema.get("minItems", 0))
|
||||
if count == 0 and schema.get("contains"):
|
||||
return [_value_for_schema(schema["contains"], f"{name}-item")]
|
||||
return [
|
||||
_vary_value(_value_for_schema(item_schema, f"{name}-item"), index)
|
||||
for index in range(count)
|
||||
]
|
||||
if schema_type == "string":
|
||||
return _string_value(schema, name)
|
||||
if schema_type in ("integer", "number"):
|
||||
return _number_value(schema, integer=schema_type == "integer")
|
||||
if schema_type == "boolean":
|
||||
return True
|
||||
if schema_type == "null":
|
||||
return None
|
||||
if schema_type is None and not schema:
|
||||
return {}
|
||||
if schema_type is None:
|
||||
candidate = {}
|
||||
try:
|
||||
_validate_instance(candidate, schema, name)
|
||||
return candidate
|
||||
except ValueError as exc:
|
||||
raise UnsupportedContract(str(exc)) from exc
|
||||
raise UnsupportedContract(f"Unsupported JSON Schema type for {name}: {schema_type}")
|
||||
|
||||
|
||||
def _object_value(schema: dict, overrides: Dict[str, Any]) -> Dict[str, Any]:
|
||||
properties = schema.get("properties", {})
|
||||
required = list(schema.get("required", []))
|
||||
result: Dict[str, Any] = {}
|
||||
|
||||
names = list(required)
|
||||
for name, spec in properties.items():
|
||||
if name in overrides or any(
|
||||
keyword in spec for keyword in ("const", "default", "examples", "enum")
|
||||
):
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
|
||||
minimum = schema.get("minProperties", 0)
|
||||
for name in properties:
|
||||
if len(names) >= minimum:
|
||||
break
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
|
||||
for name in names:
|
||||
if name in overrides:
|
||||
result[name] = copy.deepcopy(overrides[name])
|
||||
else:
|
||||
result[name] = _value_for_schema(properties.get(name, {}), name)
|
||||
|
||||
for trigger, dependents in schema.get("dependentRequired", {}).items():
|
||||
if trigger in result:
|
||||
for dependent in dependents:
|
||||
if dependent not in result:
|
||||
result[dependent] = _value_for_schema(
|
||||
properties.get(dependent, {}), dependent
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _string_value(schema: dict, name: str) -> str:
|
||||
formats = {
|
||||
"date": date(2020, 1, 2).isoformat(),
|
||||
"date-time": datetime(2020, 1, 2, 3, 4, 5, tzinfo=timezone.utc).isoformat(),
|
||||
"email": "[email protected]",
|
||||
"hostname": "example.com",
|
||||
"ipv4": "192.0.2.1",
|
||||
"uri": "https://example.com/",
|
||||
"uuid": str(UUID(int=0)),
|
||||
}
|
||||
minimum = max(0, schema.get("minLength", 0))
|
||||
maximum = schema.get("maxLength")
|
||||
seeds = [formats.get(schema.get("format")), f"test-{name}", "a", "0", ""]
|
||||
pattern = schema.get("pattern")
|
||||
for seed in seeds:
|
||||
if seed is None:
|
||||
continue
|
||||
candidate = seed
|
||||
if len(candidate) < minimum:
|
||||
candidate += "a" * (minimum - len(candidate))
|
||||
if maximum is not None:
|
||||
candidate = candidate[:maximum]
|
||||
if pattern and re.search(pattern, candidate) is None:
|
||||
continue
|
||||
return candidate
|
||||
raise UnsupportedContract(f"Cannot synthesize string '{name}' for pattern/bounds")
|
||||
|
||||
|
||||
def _number_value(schema: dict, *, integer: bool) -> int | float:
|
||||
lower = schema.get("minimum")
|
||||
if "exclusiveMinimum" in schema:
|
||||
exclusive_lower = schema["exclusiveMinimum"] + (1 if integer else 0.5)
|
||||
lower = exclusive_lower if lower is None else max(lower, exclusive_lower)
|
||||
upper = schema.get("maximum")
|
||||
if "exclusiveMaximum" in schema:
|
||||
exclusive_upper = schema["exclusiveMaximum"] - (1 if integer else 0.5)
|
||||
upper = exclusive_upper if upper is None else min(upper, exclusive_upper)
|
||||
value = 0
|
||||
if lower is not None:
|
||||
value = max(value, lower)
|
||||
if upper is not None:
|
||||
value = min(value, upper)
|
||||
multiple = schema.get("multipleOf")
|
||||
if multiple:
|
||||
if lower is not None:
|
||||
value = math.ceil(value / multiple) * multiple
|
||||
else:
|
||||
value = math.floor(value / multiple) * multiple
|
||||
if (lower is not None and value < lower) or (upper is not None and value > upper):
|
||||
raise UnsupportedContract("Numeric bounds do not admit a deterministic value")
|
||||
return int(value) if integer else float(value)
|
||||
|
||||
|
||||
def _vary_value(value: Any, index: int) -> Any:
|
||||
if index == 0:
|
||||
return value
|
||||
if isinstance(value, bool) or value is None:
|
||||
return value
|
||||
if isinstance(value, int):
|
||||
return value + index
|
||||
if isinstance(value, float):
|
||||
return value + float(index)
|
||||
if isinstance(value, str):
|
||||
return f"{value}{index}"
|
||||
return value
|
||||
|
||||
|
||||
def _merge_all_of(schema: dict) -> dict:
|
||||
merged = {key: copy.deepcopy(value) for key, value in schema.items() if key != "allOf"}
|
||||
properties = copy.deepcopy(merged.get("properties", {}))
|
||||
required = list(merged.get("required", []))
|
||||
for part in schema["allOf"]:
|
||||
properties.update(copy.deepcopy(part.get("properties", {})))
|
||||
for name in part.get("required", []):
|
||||
if name not in required:
|
||||
required.append(name)
|
||||
for key, value in part.items():
|
||||
if key not in ("properties", "required"):
|
||||
merged.setdefault(key, copy.deepcopy(value))
|
||||
if properties:
|
||||
merged["properties"] = properties
|
||||
if required:
|
||||
merged["required"] = required
|
||||
return merged
|
||||
|
||||
|
||||
def _validate_instance(instance: Any, schema: dict, label: str) -> None:
|
||||
try:
|
||||
from jsonschema import FormatChecker
|
||||
from jsonschema.exceptions import ValidationError
|
||||
from jsonschema.validators import validator_for
|
||||
except ImportError as exc:
|
||||
raise UnsupportedContract("jsonschema is required for conformance testing") from exc
|
||||
|
||||
validator = validator_for(schema)(schema, format_checker=FormatChecker())
|
||||
try:
|
||||
validator.validate(instance)
|
||||
except ValidationError as exc:
|
||||
path = ".".join(str(part) for part in exc.absolute_path) or "root"
|
||||
raise ValueError(f"{label} does not conform at {path}: {exc.message}") from exc
|
||||
|
||||
|
||||
def _run_with_stub(tool: Tool, inputs: Any) -> Any:
|
||||
"""Execute only a prompt/no-step pipeline with generated prompt outputs."""
|
||||
from .runner import DEFAULT_OUTPUT_SCHEMA, run_tool
|
||||
|
||||
tool_copy = copy.deepcopy(tool)
|
||||
transformed = []
|
||||
final_variable = _single_output_variable(tool.output)
|
||||
for step in tool_copy.steps:
|
||||
if not isinstance(step, PromptStep):
|
||||
raise UnsupportedContract(
|
||||
f"Safe conformance mode does not execute {type(step).__name__}"
|
||||
)
|
||||
schema = step.output_schema
|
||||
if schema is None and step.output_var == final_variable:
|
||||
schema = tool.output_schema
|
||||
if schema is None:
|
||||
schema = DEFAULT_OUTPUT_SCHEMA
|
||||
value = _value_for_schema(schema, step.output_var)
|
||||
try:
|
||||
_validate_instance(value, schema, f"stub output '{step.output_var}'")
|
||||
except ValueError as exc:
|
||||
raise UnsupportedContract(
|
||||
f"Could not synthesize '{step.output_var}': {exc}"
|
||||
) from exc
|
||||
transformed.append(
|
||||
CodeStep(
|
||||
code=f"globals()[{step.output_var!r}] = {value!r}",
|
||||
output_var=step.output_var,
|
||||
name=f"Contract stub for {step.name or step.output_var}",
|
||||
)
|
||||
)
|
||||
tool_copy.steps = transformed
|
||||
|
||||
if isinstance(inputs, dict):
|
||||
custom_args = {key: value for key, value in inputs.items() if key != "input"}
|
||||
input_text = inputs.get("input", "")
|
||||
else:
|
||||
custom_args = {}
|
||||
input_text = inputs
|
||||
|
||||
output, exit_code = run_tool(
|
||||
tool_copy,
|
||||
input_text="" if input_text is None else str(input_text),
|
||||
custom_args=custom_args,
|
||||
provider_override=None,
|
||||
dry_run=False,
|
||||
verbose=False,
|
||||
)
|
||||
if exit_code != 0:
|
||||
raise RuntimeError(f"Tool execution failed with exit code {exit_code}")
|
||||
try:
|
||||
return json.loads(output)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return output
|
||||
|
||||
|
||||
def _single_output_variable(output: str) -> str | None:
|
||||
match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", output.strip())
|
||||
return match.group(1) if match else None
|
||||
|
||||
|
||||
def _fallback_value(prop: dict, name: str, required: list | None = None) -> Any:
|
||||
"""Backward-compatible helper used by callers and focused tests."""
|
||||
return _value_for_schema(prop, name)
|
||||
|
||||
|
||||
def _validate_output(output: Any, schema: dict) -> bool:
|
||||
"""Backward-compatible boolean output validator."""
|
||||
try:
|
||||
_validate_instance(output, schema, "output")
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# M8.V2: Regression comparison
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
CONFORMANCE_FILE = "conformance.json"
|
||||
BASELINE_FORMAT_VERSION = 1
|
||||
CONFORMANCE_ENGINE_VERSION = 1
|
||||
|
||||
|
||||
class BaselineFormatError(ValueError):
|
||||
"""Raised when a stored conformance baseline is corrupt or incompatible."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class RegressionChange:
|
||||
case_id: str
|
||||
kind: str
|
||||
baseline_state: str = "missing"
|
||||
current_state: str = "missing"
|
||||
detail: str = ""
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
@dataclass
|
||||
class RegressionDelta:
|
||||
"""Difference between baseline and current conformance results."""
|
||||
|
||||
tool: str
|
||||
baseline_version: str
|
||||
current_version: str
|
||||
baseline_timestamp: str
|
||||
current_timestamp: str
|
||||
baseline_contract_hash: str
|
||||
current_contract_hash: str
|
||||
regressed: List[RegressionChange] = field(default_factory=list)
|
||||
improved: List[RegressionChange] = field(default_factory=list)
|
||||
new_failures: List[RegressionChange] = field(default_factory=list)
|
||||
new_cases: List[RegressionChange] = field(default_factory=list)
|
||||
removed_cases: List[RegressionChange] = field(default_factory=list)
|
||||
coverage_losses: List[RegressionChange] = field(default_factory=list)
|
||||
output_changes: List[RegressionChange] = field(default_factory=list)
|
||||
unchanged: List[RegressionChange] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def has_regressions(self) -> bool:
|
||||
return any((
|
||||
self.regressed,
|
||||
self.new_failures,
|
||||
self.removed_cases,
|
||||
self.coverage_losses,
|
||||
self.output_changes,
|
||||
))
|
||||
|
||||
@property
|
||||
def contract_changed(self) -> bool:
|
||||
return self.baseline_contract_hash != self.current_contract_hash
|
||||
|
||||
@property
|
||||
def summary(self) -> str:
|
||||
parts = []
|
||||
if self.contract_changed:
|
||||
parts.append("contract changed")
|
||||
if self.regressed:
|
||||
parts.append(f"{len(self.regressed)} regression(s)")
|
||||
if self.new_failures:
|
||||
parts.append(f"{len(self.new_failures)} new failure(s)")
|
||||
if self.improved:
|
||||
parts.append(f"{len(self.improved)} improvement(s)")
|
||||
if self.coverage_losses:
|
||||
parts.append(f"{len(self.coverage_losses)} coverage loss(es)")
|
||||
if self.removed_cases:
|
||||
parts.append(f"{len(self.removed_cases)} removed case(s)")
|
||||
if self.output_changes:
|
||||
parts.append(f"{len(self.output_changes)} output change(s)")
|
||||
if self.new_cases:
|
||||
parts.append(f"{len(self.new_cases)} new case(s)")
|
||||
if self.unchanged:
|
||||
parts.append(f"{len(self.unchanged)} unchanged case(s)")
|
||||
return ", ".join(parts) if parts else "no changes"
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"tool": self.tool,
|
||||
"baseline_version": self.baseline_version,
|
||||
"current_version": self.current_version,
|
||||
"baseline_timestamp": self.baseline_timestamp,
|
||||
"current_timestamp": self.current_timestamp,
|
||||
"baseline_contract_hash": self.baseline_contract_hash,
|
||||
"current_contract_hash": self.current_contract_hash,
|
||||
"contract_changed": self.contract_changed,
|
||||
"has_regressions": self.has_regressions,
|
||||
"summary": self.summary,
|
||||
**{
|
||||
name: [change.to_dict() for change in getattr(self, name)]
|
||||
for name in (
|
||||
"regressed", "improved", "new_failures", "new_cases",
|
||||
"removed_cases", "coverage_losses", "output_changes",
|
||||
"unchanged",
|
||||
)
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def save_conformance_baseline(tool: Tool, report: ConformanceReport, tool_dir: Path) -> Path:
|
||||
"""Atomically persist a successful, reproducible conformance baseline."""
|
||||
if not report.results or any(
|
||||
result.state != "passed" for result in report.results
|
||||
):
|
||||
raise ValueError("Only a passing conformance report can become a baseline")
|
||||
if any(not result.case_id or not result.input_fingerprint for result in report.results):
|
||||
raise ValueError("Baseline results must include stable case evidence")
|
||||
|
||||
tool_dir = Path(tool_dir)
|
||||
tool_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = tool_dir / CONFORMANCE_FILE
|
||||
data = {
|
||||
"format_version": BASELINE_FORMAT_VERSION,
|
||||
"engine_version": CONFORMANCE_ENGINE_VERSION,
|
||||
"tool": {"name": tool.name, "version": tool.version},
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"contract_hash": _contract_hash(tool),
|
||||
"contracts": {
|
||||
"input_schema": copy.deepcopy(tool.input_schema),
|
||||
"output_schema": copy.deepcopy(tool.output_schema),
|
||||
},
|
||||
"results": [r.to_dict() for r in report.results],
|
||||
}
|
||||
_validate_baseline_data(data)
|
||||
|
||||
descriptor, temporary_name = tempfile.mkstemp(
|
||||
prefix=f".{CONFORMANCE_FILE}.", dir=tool_dir
|
||||
)
|
||||
try:
|
||||
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
|
||||
json.dump(data, handle, indent=2, sort_keys=True)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(temporary_name, 0o600)
|
||||
os.replace(temporary_name, path)
|
||||
finally:
|
||||
if os.path.exists(temporary_name):
|
||||
os.unlink(temporary_name)
|
||||
return path
|
||||
|
||||
|
||||
def load_conformance_baseline(tool_dir: Path) -> Optional[dict]:
|
||||
"""Load and validate a previously saved conformance baseline."""
|
||||
path = Path(tool_dir) / CONFORMANCE_FILE
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except OSError as exc:
|
||||
raise BaselineFormatError(f"Could not read baseline: {exc}") from exc
|
||||
except json.JSONDecodeError as exc:
|
||||
raise BaselineFormatError(f"Baseline is not valid JSON: {exc}") from exc
|
||||
_validate_baseline_data(data)
|
||||
return data
|
||||
|
||||
|
||||
def compare_conformance(
|
||||
baseline: dict,
|
||||
current: ConformanceReport,
|
||||
tool: Tool,
|
||||
) -> RegressionDelta:
|
||||
"""Compare stable cases, states, coverage, and normalized output evidence."""
|
||||
_validate_baseline_data(baseline)
|
||||
if baseline["tool"]["name"] != tool.name:
|
||||
raise BaselineFormatError(
|
||||
f"Baseline belongs to '{baseline['tool']['name']}', not '{tool.name}'"
|
||||
)
|
||||
base_map = {result["case_id"]: result for result in baseline["results"]}
|
||||
current_map = {result.case_id: result for result in current.results if result.case_id}
|
||||
|
||||
delta = RegressionDelta(
|
||||
tool=tool.name,
|
||||
baseline_version=baseline["tool"]["version"],
|
||||
current_version=tool.version,
|
||||
baseline_timestamp=baseline["timestamp"],
|
||||
current_timestamp=datetime.now(timezone.utc).isoformat(),
|
||||
baseline_contract_hash=baseline["contract_hash"],
|
||||
current_contract_hash=_contract_hash(tool),
|
||||
)
|
||||
|
||||
for case_id, current_result in current_map.items():
|
||||
base_result = base_map.get(case_id)
|
||||
if base_result is None:
|
||||
if current_result.state == "failed":
|
||||
delta.new_failures.append(_change(
|
||||
case_id, "new_failure", None, current_result
|
||||
))
|
||||
else:
|
||||
delta.new_cases.append(_change(
|
||||
case_id, "new_case", None, current_result
|
||||
))
|
||||
continue
|
||||
|
||||
baseline_state = base_result["state"]
|
||||
if baseline_state == "passed" and current_result.state == "failed":
|
||||
delta.regressed.append(_change(
|
||||
case_id, "state_regression", base_result, current_result
|
||||
))
|
||||
elif baseline_state == "passed" and current_result.state in (
|
||||
"unsupported", "not_run"
|
||||
):
|
||||
delta.coverage_losses.append(_change(
|
||||
case_id, "coverage_loss", base_result, current_result
|
||||
))
|
||||
elif baseline_state != "passed" and current_result.state == "passed":
|
||||
delta.improved.append(_change(
|
||||
case_id, "improved", base_result, current_result
|
||||
))
|
||||
elif (
|
||||
baseline_state == "passed"
|
||||
and current_result.state == "passed"
|
||||
and (
|
||||
base_result.get("output_shape") != current_result.output_shape
|
||||
or base_result.get("output_fingerprint")
|
||||
!= current_result.output_fingerprint
|
||||
)
|
||||
):
|
||||
delta.output_changes.append(_change(
|
||||
case_id, "output_changed", base_result, current_result
|
||||
))
|
||||
else:
|
||||
delta.unchanged.append(_change(
|
||||
case_id, "unchanged", base_result, current_result
|
||||
))
|
||||
|
||||
for case_id, baseline_result in base_map.items():
|
||||
if case_id not in current_map:
|
||||
delta.removed_cases.append(_change(
|
||||
case_id, "case_removed", baseline_result, None
|
||||
))
|
||||
return delta
|
||||
|
||||
|
||||
def _change(
|
||||
case_id: str,
|
||||
kind: str,
|
||||
baseline: Optional[dict],
|
||||
current: Optional[ConformanceResult],
|
||||
) -> RegressionChange:
|
||||
baseline_state = baseline.get("state", "missing") if baseline else "missing"
|
||||
current_state = current.state if current else "missing"
|
||||
detail = current.detail if current else "Previously recorded case is absent"
|
||||
return RegressionChange(
|
||||
case_id=case_id,
|
||||
kind=kind,
|
||||
baseline_state=baseline_state,
|
||||
current_state=current_state,
|
||||
detail=detail,
|
||||
)
|
||||
|
||||
|
||||
def _contract_hash(tool: Tool) -> str:
|
||||
return _fingerprint({
|
||||
"input_schema": tool.input_schema,
|
||||
"output_schema": tool.output_schema,
|
||||
})
|
||||
|
||||
|
||||
def _validate_baseline_data(data: Any) -> None:
|
||||
if not isinstance(data, dict):
|
||||
raise BaselineFormatError("Baseline root must be an object")
|
||||
if data.get("format_version") != BASELINE_FORMAT_VERSION:
|
||||
raise BaselineFormatError("Unsupported baseline format version")
|
||||
if data.get("engine_version") != CONFORMANCE_ENGINE_VERSION:
|
||||
raise BaselineFormatError("Unsupported conformance engine version")
|
||||
tool = data.get("tool")
|
||||
if not isinstance(tool, dict) or not isinstance(tool.get("name"), str):
|
||||
raise BaselineFormatError("Baseline tool metadata is invalid")
|
||||
if not isinstance(tool.get("version"), str):
|
||||
raise BaselineFormatError("Baseline tool version is invalid")
|
||||
if not isinstance(data.get("timestamp"), str):
|
||||
raise BaselineFormatError("Baseline timestamp is invalid")
|
||||
if not isinstance(data.get("contract_hash"), str) or not re.fullmatch(
|
||||
r"[0-9a-f]{64}", data["contract_hash"]
|
||||
):
|
||||
raise BaselineFormatError("Baseline contract hash is invalid")
|
||||
if not isinstance(data.get("contracts"), dict):
|
||||
raise BaselineFormatError("Baseline contracts are invalid")
|
||||
if _fingerprint(data["contracts"]) != data["contract_hash"]:
|
||||
raise BaselineFormatError("Baseline contract hash does not match contracts")
|
||||
results = data.get("results")
|
||||
if not isinstance(results, list):
|
||||
raise BaselineFormatError("Baseline results must be an array")
|
||||
seen = set()
|
||||
for result in results:
|
||||
if not isinstance(result, dict):
|
||||
raise BaselineFormatError("Baseline result must be an object")
|
||||
case_id = result.get("case_id")
|
||||
if not isinstance(case_id, str) or not case_id or case_id in seen:
|
||||
raise BaselineFormatError("Baseline case IDs must be unique strings")
|
||||
seen.add(case_id)
|
||||
if result.get("state") not in _VALID_STATES:
|
||||
raise BaselineFormatError(f"Invalid state for baseline case '{case_id}'")
|
||||
if not isinstance(result.get("input_fingerprint"), str) or not re.fullmatch(
|
||||
r"[0-9a-f]{64}", result["input_fingerprint"]
|
||||
):
|
||||
raise BaselineFormatError(
|
||||
f"Missing input fingerprint for baseline case '{case_id}'"
|
||||
)
|
||||
if "input_value" not in result:
|
||||
raise BaselineFormatError(
|
||||
f"Missing input value for baseline case '{case_id}'"
|
||||
)
|
||||
if _fingerprint(result["input_value"]) != result["input_fingerprint"]:
|
||||
raise BaselineFormatError(
|
||||
f"Input fingerprint mismatch for baseline case '{case_id}'"
|
||||
)
|
||||
shape = result.get("output_shape")
|
||||
if shape is not None and not isinstance(shape, dict):
|
||||
raise BaselineFormatError(
|
||||
f"Invalid output shape for baseline case '{case_id}'"
|
||||
)
|
||||
if result["state"] == "passed":
|
||||
if not isinstance(shape, dict):
|
||||
raise BaselineFormatError(
|
||||
f"Passing baseline case '{case_id}' lacks output shape"
|
||||
)
|
||||
fingerprint = result.get("output_fingerprint")
|
||||
if not isinstance(fingerprint, str) or not re.fullmatch(
|
||||
r"[0-9a-f]{64}", fingerprint
|
||||
):
|
||||
raise BaselineFormatError(
|
||||
f"Passing baseline case '{case_id}' lacks output fingerprint"
|
||||
)
|
||||
@@ -44,6 +44,44 @@ class ProviderDialog(QDialog):
|
||||
self.desc_input.setPlaceholderText("Claude AI via claude-cli")
|
||||
form.addRow("Description:", self.desc_input)
|
||||
|
||||
self.locality_combo = QComboBox()
|
||||
for value in ("unknown", "local", "remote"):
|
||||
self.locality_combo.addItem(value.title(), value)
|
||||
form.addRow("Locality:", self.locality_combo)
|
||||
|
||||
self.data_policy_combo = QComboBox()
|
||||
for value in ("unspecified", "public", "internal", "private"):
|
||||
self.data_policy_combo.addItem(value.title(), value)
|
||||
form.addRow("Data policy:", self.data_policy_combo)
|
||||
|
||||
self.capabilities_input = QLineEdit()
|
||||
self.capabilities_input.setPlaceholderText("text, structured-json, reasoning")
|
||||
form.addRow("Capabilities:", self.capabilities_input)
|
||||
|
||||
self.model_input = QLineEdit()
|
||||
self.model_input.setPlaceholderText("Model ID, if known")
|
||||
form.addRow("Model:", self.model_input)
|
||||
|
||||
self.model_digest_input = QLineEdit()
|
||||
self.model_digest_input.setPlaceholderText("sha256:... (verified local models)")
|
||||
form.addRow("Model digest:", self.model_digest_input)
|
||||
|
||||
self.max_context_input = QLineEdit()
|
||||
self.max_context_input.setPlaceholderText("Maximum context tokens")
|
||||
form.addRow("Context tokens:", self.max_context_input)
|
||||
|
||||
self.cost_class_combo = QComboBox()
|
||||
self.cost_class_combo.addItem("Unspecified", None)
|
||||
for value in ("free", "low", "medium", "high"):
|
||||
self.cost_class_combo.addItem(value.title(), value)
|
||||
form.addRow("Cost class:", self.cost_class_combo)
|
||||
|
||||
self.latency_class_combo = QComboBox()
|
||||
self.latency_class_combo.addItem("Unspecified", None)
|
||||
for value in ("fast", "standard", "slow"):
|
||||
self.latency_class_combo.addItem(value.title(), value)
|
||||
form.addRow("Latency class:", self.latency_class_combo)
|
||||
|
||||
self.fallback_combo = QComboBox()
|
||||
self.fallback_combo.addItem("(none)", None)
|
||||
# Populate with available providers
|
||||
@@ -86,6 +124,24 @@ class ProviderDialog(QDialog):
|
||||
self.name_input.setEnabled(False) # Can't rename
|
||||
self.cmd_input.setText(provider.command)
|
||||
self.desc_input.setText(provider.description or "")
|
||||
self.locality_combo.setCurrentIndex(
|
||||
max(self.locality_combo.findData(provider.locality), 0)
|
||||
)
|
||||
self.data_policy_combo.setCurrentIndex(
|
||||
max(self.data_policy_combo.findData(provider.data_policy), 0)
|
||||
)
|
||||
self.capabilities_input.setText(", ".join(provider.capabilities))
|
||||
self.model_input.setText(provider.model or "")
|
||||
self.model_digest_input.setText(provider.model_digest or "")
|
||||
self.max_context_input.setText(
|
||||
str(provider.max_context_tokens) if provider.max_context_tokens else ""
|
||||
)
|
||||
self.cost_class_combo.setCurrentIndex(
|
||||
max(self.cost_class_combo.findData(provider.cost_class), 0)
|
||||
)
|
||||
self.latency_class_combo.setCurrentIndex(
|
||||
max(self.latency_class_combo.findData(provider.latency_class), 0)
|
||||
)
|
||||
# Set fallback selection
|
||||
if provider.fallback:
|
||||
idx = self.fallback_combo.findData(provider.fallback)
|
||||
@@ -107,6 +163,21 @@ class ProviderDialog(QDialog):
|
||||
|
||||
description = self.desc_input.text().strip()
|
||||
fallback = self.fallback_combo.currentData()
|
||||
capabilities = [
|
||||
value.strip() for value in self.capabilities_input.text().split(",")
|
||||
if value.strip()
|
||||
]
|
||||
try:
|
||||
max_context_tokens = (
|
||||
int(self.max_context_input.text().strip())
|
||||
if self.max_context_input.text().strip() else None
|
||||
)
|
||||
except ValueError:
|
||||
from PySide6.QtWidgets import QMessageBox
|
||||
QMessageBox.warning(
|
||||
self, "Invalid", "Context tokens must be a positive integer."
|
||||
)
|
||||
return
|
||||
|
||||
# Prevent self-referential fallback
|
||||
if fallback == name:
|
||||
@@ -122,12 +193,21 @@ class ProviderDialog(QDialog):
|
||||
description=description,
|
||||
fallback=fallback,
|
||||
type=existing.type if existing else "subprocess",
|
||||
model=existing.model if existing else None,
|
||||
model=self.model_input.text().strip() or None,
|
||||
tags=existing.tags if existing else [],
|
||||
install=existing.install if existing else None,
|
||||
fallback_chain=existing.fallback_chain if existing else None,
|
||||
api_key_env=existing.api_key_env if existing else None,
|
||||
pty_config=existing.pty_config if existing else None,
|
||||
tools=existing.tools if existing else None,
|
||||
mcp_servers=existing.mcp_servers if existing else None,
|
||||
locality=self.locality_combo.currentData(),
|
||||
capabilities=capabilities,
|
||||
model_digest=self.model_digest_input.text().strip() or None,
|
||||
cost_class=self.cost_class_combo.currentData(),
|
||||
latency_class=self.latency_class_combo.currentData(),
|
||||
max_context_tokens=max_context_tokens,
|
||||
data_policy=self.data_policy_combo.currentData(),
|
||||
))
|
||||
self.accept()
|
||||
except Exception as e:
|
||||
|
||||
@@ -321,6 +321,15 @@ class PromptStepDialog(QDialog):
|
||||
self.strip_fences_check = QCheckBox("Strip markdown code fences from output")
|
||||
form.addRow("", self.strip_fences_check)
|
||||
|
||||
self.allow_fallback_check = QCheckBox(
|
||||
"Allow configured fallback providers if this provider fails"
|
||||
)
|
||||
self.allow_fallback_check.setChecked(True)
|
||||
self.allow_fallback_check.setToolTip(
|
||||
"Disable for privacy-sensitive or reproducible steps that must fail closed."
|
||||
)
|
||||
form.addRow("", self.allow_fallback_check)
|
||||
|
||||
# Structured output options
|
||||
form.addRow(QLabel("")) # Spacer
|
||||
structured_label = QLabel("<b>Structured Output</b>")
|
||||
@@ -455,6 +464,7 @@ class PromptStepDialog(QDialog):
|
||||
self.output_input.setText(step.output_var)
|
||||
self.prompt_input.setPlainText(step.prompt)
|
||||
self.strip_fences_check.setChecked(step.strip_fences)
|
||||
self.allow_fallback_check.setChecked(step.fallback_policy != "deny")
|
||||
|
||||
# Structured output fields
|
||||
self.plain_text_check.setChecked(step.plain_text)
|
||||
@@ -497,7 +507,10 @@ class PromptStepDialog(QDialog):
|
||||
strip_fences=self.strip_fences_check.isChecked(),
|
||||
output_schema=self._output_schema,
|
||||
plain_text=self.plain_text_check.isChecked(),
|
||||
max_retries=self.retries_spin.value()
|
||||
max_retries=self.retries_spin.value(),
|
||||
fallback_policy=(
|
||||
"allow" if self.allow_fallback_check.isChecked() else "deny"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Registry page - browse and install tools from registry."""
|
||||
|
||||
import html
|
||||
from PySide6.QtWidgets import (
|
||||
QWidget, QVBoxLayout, QHBoxLayout, QLineEdit,
|
||||
QPushButton, QTableWidget, QTableWidgetItem, QLabel,
|
||||
@@ -877,6 +878,16 @@ class RegistryPage(QWidget):
|
||||
|
||||
if tool.get("description"):
|
||||
lines.append(f"<p style='color: #4a5568;'>{tool.get('description')}</p>")
|
||||
if tool.get("deprecated"):
|
||||
guidance = html.escape(
|
||||
tool.get("deprecated_message") or "This tool is no longer maintained."
|
||||
)
|
||||
if tool.get("replacement"):
|
||||
guidance += f" Use <strong>{html.escape(tool['replacement'])}</strong> instead."
|
||||
lines.append(
|
||||
"<p style='background: #fef3c7; color: #92400e; padding: 8px 10px; "
|
||||
f"border-radius: 4px;'><strong>Deprecated:</strong> {guidance}</p>"
|
||||
)
|
||||
|
||||
# Rating display
|
||||
rating = tool.get("average_rating", 0) or 0
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
"""Tool builder page - create and edit tools."""
|
||||
|
||||
import copy
|
||||
import difflib
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from PySide6.QtWidgets import (
|
||||
@@ -7,18 +12,117 @@ from PySide6.QtWidgets import (
|
||||
QLineEdit, QTextEdit, QPlainTextEdit, QComboBox, QPushButton,
|
||||
QGroupBox, QListWidget, QListWidgetItem, QLabel,
|
||||
QMessageBox, QSplitter, QFrame, QStackedWidget,
|
||||
QButtonGroup
|
||||
QButtonGroup, QInputDialog
|
||||
)
|
||||
from PySide6.QtCore import Qt
|
||||
from PySide6.QtCore import Qt, QThread, Signal, QTimer
|
||||
|
||||
from ...tool import (
|
||||
Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
|
||||
load_tool, save_tool, validate_tool_name, get_all_categories,
|
||||
ensure_settings
|
||||
ensure_settings, get_project_tools_dir, get_tools_dir
|
||||
)
|
||||
from ..widgets.icons import get_prompt_icon, get_code_icon, get_tool_icon
|
||||
|
||||
|
||||
class SimilarToolsWorker(QThread):
|
||||
"""Search the registry without blocking the tool builder."""
|
||||
|
||||
results_ready = Signal(str, list)
|
||||
|
||||
def __init__(self, query: str, parent=None):
|
||||
super().__init__(parent)
|
||||
self.query = query
|
||||
|
||||
def run(self):
|
||||
try:
|
||||
from ...registry_client import RegistryClient, RegistryError
|
||||
|
||||
try:
|
||||
result = RegistryClient().search_tools(self.query, per_page=5)
|
||||
except RegistryError:
|
||||
result = None
|
||||
self.results_ready.emit(self.query, list(result.data) if result else [])
|
||||
except Exception:
|
||||
# Guidance is advisory; an unavailable registry must never prevent
|
||||
# local authoring.
|
||||
self.results_ready.emit(self.query, [])
|
||||
|
||||
|
||||
def build_extracted_tool(tool: Tool, opportunity: dict, name: str) -> Tool:
|
||||
"""Build, but do not save, a reusable tool from an evidenced sequence."""
|
||||
if opportunity.get("type") != "repeated_sequence":
|
||||
raise ValueError("Only an internal repeated sequence can be extracted")
|
||||
locations = opportunity.get("locations") or []
|
||||
length = int(opportunity.get("length", 0))
|
||||
if not locations or length < 2:
|
||||
raise ValueError("Reuse evidence does not identify a valid sequence")
|
||||
start = int(locations[0]) - 1
|
||||
selected = tool.steps[start:start + length]
|
||||
if len(selected) != length:
|
||||
raise ValueError("Reuse evidence falls outside the current draft")
|
||||
available = {"input", "settings"} | {
|
||||
argument.variable for argument in tool.arguments
|
||||
}
|
||||
for step in selected:
|
||||
if not isinstance(step, PromptStep):
|
||||
raise ValueError("Only prompt-step sequences can be extracted safely")
|
||||
references = set(re.findall(r"(?<!\{)\{([A-Za-z_]\w*)\}(?!\})", step.prompt or ""))
|
||||
missing = references - available
|
||||
if missing:
|
||||
raise ValueError(
|
||||
"The sequence depends on values produced outside its boundary: "
|
||||
+ ", ".join(sorted(missing))
|
||||
)
|
||||
if step.output_var:
|
||||
available.update(
|
||||
value.strip() for value in step.output_var.split(",") if value.strip()
|
||||
)
|
||||
output_var = getattr(selected[-1], "output_var", "result") or "result"
|
||||
output_schema = getattr(selected[-1], "output_schema", None)
|
||||
return Tool(
|
||||
name=name,
|
||||
description=f"Extracted reusable sequence from {tool.name}",
|
||||
category=tool.category,
|
||||
arguments=copy.deepcopy(tool.arguments),
|
||||
steps=copy.deepcopy(selected),
|
||||
output=f"{{{output_var}}}",
|
||||
dependencies=[],
|
||||
system_dependencies=copy.deepcopy(tool.system_dependencies),
|
||||
visibility=tool.visibility,
|
||||
input_schema=copy.deepcopy(tool.input_schema),
|
||||
output_schema=copy.deepcopy(output_schema),
|
||||
)
|
||||
|
||||
|
||||
def extraction_diff(tool: Tool) -> str:
|
||||
"""Return a unified-diff preview for a proposed new tool."""
|
||||
rendered = yaml.safe_dump(tool.to_dict(), sort_keys=False).splitlines(True)
|
||||
return "".join(difflib.unified_diff(
|
||||
[], rendered, fromfile="/dev/null", tofile=f"{tool.name}/config.yaml",
|
||||
))
|
||||
|
||||
|
||||
def _switch_to_existing_tool(main_window, name: str) -> None:
|
||||
"""Remove the creation page before opening an existing tool editor."""
|
||||
main_window.close_tool_builder()
|
||||
main_window.open_tool_builder(name)
|
||||
|
||||
|
||||
def default_save_scope(project_dir: Path) -> str:
|
||||
"""Prefer project scope only when the cwd is recognizably a project."""
|
||||
project_dir = Path(project_dir)
|
||||
if any((project_dir / marker).exists() for marker in (
|
||||
"cmdforge.yaml", ".cmdforge", ".git", "pyproject.toml", "package.json",
|
||||
)):
|
||||
return "project"
|
||||
return "global"
|
||||
|
||||
|
||||
def existing_tool_root(config_path: Path) -> Path:
|
||||
"""Return the tool-root directory containing an existing named tool."""
|
||||
return Path(config_path).resolve().parent.parent
|
||||
|
||||
|
||||
class ToolBuilderPage(QWidget):
|
||||
"""Tool builder/editor page."""
|
||||
|
||||
@@ -29,6 +133,7 @@ class ToolBuilderPage(QWidget):
|
||||
self.original_name = tool_name
|
||||
self._tool = None
|
||||
self._flow_widget = None # Lazy-loaded
|
||||
self._guidance_worker = None
|
||||
|
||||
self._setup_ui()
|
||||
|
||||
@@ -96,8 +201,63 @@ class ToolBuilderPage(QWidget):
|
||||
self.category_combo.addItem(cat)
|
||||
info_layout.addRow("Category:", self.category_combo)
|
||||
|
||||
self.save_location_combo = QComboBox()
|
||||
project_root = get_project_tools_dir()
|
||||
self.save_location_combo.addItem(
|
||||
f"This project ({project_root})", "project"
|
||||
)
|
||||
self.save_location_combo.addItem(
|
||||
f"My tools ({get_tools_dir()})", "global"
|
||||
)
|
||||
preferred_scope = default_save_scope(Path.cwd())
|
||||
preferred_index = self.save_location_combo.findData(preferred_scope)
|
||||
self.save_location_combo.setCurrentIndex(max(preferred_index, 0))
|
||||
self.save_location_combo.setToolTip(
|
||||
"Project tools are versionable and resolve only from this project; "
|
||||
"My tools are available globally and receive a shell wrapper."
|
||||
)
|
||||
info_layout.addRow("Save tool:", self.save_location_combo)
|
||||
|
||||
left_layout.addWidget(info_box)
|
||||
|
||||
# Reuse guidance is advisory and never changes a draft automatically.
|
||||
guidance_box = QGroupBox("Reuse & Discovery")
|
||||
guidance_layout = QVBoxLayout(guidance_box)
|
||||
self.guidance_summary = QLabel(
|
||||
"CmdForge can check for similar tools, reusable step sequences, "
|
||||
"and frequent local pipelines."
|
||||
)
|
||||
self.guidance_summary.setWordWrap(True)
|
||||
guidance_layout.addWidget(self.guidance_summary)
|
||||
self.guidance_list = QListWidget()
|
||||
self.guidance_list.setMaximumHeight(130)
|
||||
self.guidance_list.currentItemChanged.connect(
|
||||
self._guidance_selection_changed
|
||||
)
|
||||
guidance_layout.addWidget(self.guidance_list)
|
||||
guidance_buttons = QHBoxLayout()
|
||||
self.btn_refresh_guidance = QPushButton("Refresh")
|
||||
self.btn_refresh_guidance.clicked.connect(self._refresh_guidance)
|
||||
guidance_buttons.addWidget(self.btn_refresh_guidance)
|
||||
self.btn_open_guidance = QPushButton("Open Existing")
|
||||
self.btn_open_guidance.setObjectName("secondary")
|
||||
self.btn_open_guidance.clicked.connect(self._open_guidance_tool)
|
||||
self.btn_open_guidance.setEnabled(False)
|
||||
guidance_buttons.addWidget(self.btn_open_guidance)
|
||||
self.btn_extract_guidance = QPushButton("Extract as New Tool")
|
||||
self.btn_extract_guidance.clicked.connect(self._extract_guidance)
|
||||
self.btn_extract_guidance.setEnabled(False)
|
||||
guidance_buttons.addWidget(self.btn_extract_guidance)
|
||||
guidance_layout.addLayout(guidance_buttons)
|
||||
left_layout.addWidget(guidance_box)
|
||||
|
||||
self._guidance_timer = QTimer(self)
|
||||
self._guidance_timer.setSingleShot(True)
|
||||
self._guidance_timer.setInterval(700)
|
||||
self._guidance_timer.timeout.connect(self._refresh_guidance)
|
||||
self.name_input.textChanged.connect(self._schedule_guidance)
|
||||
self.desc_input.textChanged.connect(self._schedule_guidance)
|
||||
|
||||
# Arguments group
|
||||
args_box = QGroupBox()
|
||||
args_layout = QVBoxLayout(args_box)
|
||||
@@ -398,6 +558,176 @@ class ToolBuilderPage(QWidget):
|
||||
|
||||
layout.addWidget(splitter, 1)
|
||||
|
||||
def _current_draft(self) -> Tool:
|
||||
"""Materialize the current form without saving it."""
|
||||
current = self._tool
|
||||
return Tool(
|
||||
name=self.name_input.text().strip() or "untitled-tool",
|
||||
description=self.desc_input.text().strip(),
|
||||
category=self.category_combo.currentText() or "Other",
|
||||
arguments=copy.deepcopy(current.arguments) if current else [],
|
||||
steps=copy.deepcopy(current.steps) if current else [],
|
||||
output=self.output_input.toPlainText().strip() or "{response}",
|
||||
dependencies=list(current.dependencies) if current else [],
|
||||
system_dependencies=(
|
||||
copy.deepcopy(current.system_dependencies) if current else []
|
||||
),
|
||||
visibility=current.visibility if current else "public",
|
||||
input_schema=copy.deepcopy(current.input_schema) if current else None,
|
||||
output_schema=copy.deepcopy(current.output_schema) if current else None,
|
||||
)
|
||||
|
||||
def _schedule_guidance(self):
|
||||
self._guidance_timer.start()
|
||||
|
||||
def _add_guidance_item(self, text: str, data: dict) -> None:
|
||||
item = QListWidgetItem(text)
|
||||
item.setData(Qt.UserRole, data)
|
||||
self.guidance_list.addItem(item)
|
||||
|
||||
def _refresh_guidance(self):
|
||||
"""Refresh deterministic local guidance, then search remotely."""
|
||||
from ...preflight import analyze_tool
|
||||
from ...usage import get_suggestions
|
||||
|
||||
draft = self._current_draft()
|
||||
self.guidance_list.clear()
|
||||
report = analyze_tool(
|
||||
draft,
|
||||
check_local_dependencies=False,
|
||||
include_contract_tests=False,
|
||||
check_reuse=True,
|
||||
)
|
||||
for opportunity in report.reuse_opportunities:
|
||||
self._add_guidance_item(
|
||||
f"Reusable: {opportunity.get('detail', 'exact sequence')}",
|
||||
opportunity,
|
||||
)
|
||||
for suggestion in get_suggestions():
|
||||
tools = suggestion["tools"]
|
||||
self._add_guidance_item(
|
||||
f"Frequent pipeline ({suggestion['count']}x): " + " | ".join(tools),
|
||||
{"type": "usage_pipeline", **suggestion},
|
||||
)
|
||||
|
||||
query = (draft.description or draft.name).strip()
|
||||
if not query or query == "untitled-tool":
|
||||
self._finish_guidance(query, [])
|
||||
return
|
||||
if self._guidance_worker and self._guidance_worker.isRunning():
|
||||
# The pending result is ignored if the query has since changed.
|
||||
self.guidance_summary.setText("Checking registry…")
|
||||
return
|
||||
self.guidance_summary.setText("Checking registry…")
|
||||
self.btn_refresh_guidance.setEnabled(False)
|
||||
self._guidance_worker = SimilarToolsWorker(query, self)
|
||||
self._guidance_worker.results_ready.connect(self._finish_guidance)
|
||||
self._guidance_worker.start()
|
||||
|
||||
def _finish_guidance(self, query: str, results: list):
|
||||
current_query = (
|
||||
self.desc_input.text().strip() or self.name_input.text().strip()
|
||||
)
|
||||
if query == current_query:
|
||||
seen = set()
|
||||
for result in results:
|
||||
name = result.get("name", "")
|
||||
owner = result.get("owner", "")
|
||||
reference = f"{owner}/{name}" if owner else name
|
||||
if reference and reference not in seen:
|
||||
seen.add(reference)
|
||||
score = result.get("quality_score")
|
||||
quality = f" [{score}/100]" if score is not None else ""
|
||||
self._add_guidance_item(
|
||||
f"Registry: {reference}{quality} — {result.get('description', '')}",
|
||||
{"type": "registry_similar", "tool": reference},
|
||||
)
|
||||
self.btn_refresh_guidance.setEnabled(True)
|
||||
count = self.guidance_list.count()
|
||||
self.guidance_summary.setText(
|
||||
f"{count} advisory suggestion{'s' if count != 1 else ''}. "
|
||||
"Nothing changes until you confirm an action."
|
||||
if count else "No evidenced reuse opportunities found."
|
||||
)
|
||||
self._guidance_selection_changed(self.guidance_list.currentItem())
|
||||
if query != current_query:
|
||||
self._guidance_timer.start(0)
|
||||
|
||||
def _guidance_selection_changed(self, current, previous=None):
|
||||
data = current.data(Qt.UserRole) if current else {}
|
||||
kind = data.get("type")
|
||||
local_name = data.get("tool", "")
|
||||
self.btn_open_guidance.setEnabled(
|
||||
kind == "duplicate_sequence" and bool(local_name)
|
||||
)
|
||||
self.btn_extract_guidance.setEnabled(
|
||||
kind in {"repeated_sequence", "usage_pipeline"}
|
||||
)
|
||||
|
||||
def _open_guidance_tool(self):
|
||||
item = self.guidance_list.currentItem()
|
||||
data = item.data(Qt.UserRole) if item else {}
|
||||
if data.get("type") == "duplicate_sequence" and data.get("tool"):
|
||||
_switch_to_existing_tool(self.main_window, data["tool"])
|
||||
|
||||
def _extract_guidance(self):
|
||||
"""Preview and explicitly save an evidenced reusable tool."""
|
||||
item = self.guidance_list.currentItem()
|
||||
data = item.data(Qt.UserRole) if item else {}
|
||||
kind = data.get("type")
|
||||
if kind not in {"repeated_sequence", "usage_pipeline"}:
|
||||
return
|
||||
suggested = (
|
||||
"-".join(data.get("tools", []))
|
||||
if kind == "usage_pipeline" else f"{self.name_input.text().strip()}-shared"
|
||||
)
|
||||
name, accepted = QInputDialog.getText(
|
||||
self, "Extract Reusable Tool", "New tool name:", text=suggested
|
||||
)
|
||||
name = name.strip()
|
||||
if not accepted or not name:
|
||||
return
|
||||
valid, error = validate_tool_name(name)
|
||||
if not valid:
|
||||
QMessageBox.warning(self, "Validation", error)
|
||||
return
|
||||
extraction_root = self._selected_tools_dir()
|
||||
if (extraction_root / name / "config.yaml").exists():
|
||||
QMessageBox.warning(self, "Tool Already Exists", f"'{name}' already exists.")
|
||||
return
|
||||
try:
|
||||
if kind == "usage_pipeline":
|
||||
from ...usage import build_composite_tool
|
||||
extracted = build_composite_tool(name, data["tools"])
|
||||
else:
|
||||
extracted = build_extracted_tool(self._current_draft(), data, name)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
QMessageBox.warning(self, "Cannot Extract", str(exc))
|
||||
return
|
||||
|
||||
preview = QMessageBox(self)
|
||||
preview.setIcon(QMessageBox.Information)
|
||||
preview.setWindowTitle("Review Extraction")
|
||||
preview.setText(
|
||||
f"Create '{name}' as a separate tool? Your current draft will not be modified."
|
||||
)
|
||||
preview.setDetailedText(extraction_diff(extracted))
|
||||
create_button = preview.addButton("Create Tool", QMessageBox.AcceptRole)
|
||||
preview.addButton(QMessageBox.Cancel)
|
||||
preview.exec()
|
||||
if preview.clickedButton() is not create_button:
|
||||
return
|
||||
try:
|
||||
save_tool(extracted, tools_dir=extraction_root)
|
||||
except (OSError, ValueError) as exc:
|
||||
QMessageBox.critical(self, "Extraction Failed", str(exc))
|
||||
return
|
||||
QMessageBox.information(
|
||||
self, "Tool Created", f"Created '{name}'. The current draft is unchanged."
|
||||
)
|
||||
self._populate_deps_combo()
|
||||
self._refresh_guidance()
|
||||
|
||||
def _set_view_mode(self, mode: int):
|
||||
"""Switch between list (0) and flow (1) views."""
|
||||
if mode == 1 and self._flow_widget is None:
|
||||
@@ -511,6 +841,17 @@ class ToolBuilderPage(QWidget):
|
||||
self._tool = tool
|
||||
self.name_input.setText(tool.name)
|
||||
self.name_input.setEnabled(False) # Can't rename
|
||||
if tool.path:
|
||||
project_root = get_project_tools_dir()
|
||||
scope = (
|
||||
"project"
|
||||
if existing_tool_root(tool.path) == project_root.resolve()
|
||||
else "global"
|
||||
)
|
||||
index = self.save_location_combo.findData(scope)
|
||||
if index >= 0:
|
||||
self.save_location_combo.setCurrentIndex(index)
|
||||
self.save_location_combo.setEnabled(False)
|
||||
self.desc_input.setText(tool.description or "")
|
||||
|
||||
# Set category
|
||||
@@ -1075,6 +1416,44 @@ class ToolBuilderPage(QWidget):
|
||||
QMessageBox.warning(self, "Validation", error)
|
||||
return
|
||||
|
||||
target_root = self._selected_tools_dir()
|
||||
|
||||
# Block silent overwrite in the selected scope. A project-local tool
|
||||
# may intentionally shadow a global tool with the same name.
|
||||
target_config = target_root / name / "config.yaml"
|
||||
if (not self.editing or name != self.original_name) and target_config.exists():
|
||||
existing = load_tool(name) if self.save_location_combo.currentData() == "global" else None
|
||||
existing_desc = f" ({existing.description})" if existing and existing.description else ""
|
||||
msg = QMessageBox(self)
|
||||
msg.setIcon(QMessageBox.Warning)
|
||||
msg.setWindowTitle("Tool Already Exists")
|
||||
msg.setText(f"'{name}' already exists.{existing_desc}")
|
||||
msg.setInformativeText("What would you like to do?")
|
||||
btn_open = msg.addButton("Open Existing", QMessageBox.ActionRole)
|
||||
btn_copy = msg.addButton("Create Copy", QMessageBox.ActionRole)
|
||||
btn_rename = msg.addButton("Choose Another Name", QMessageBox.ActionRole)
|
||||
msg.setStandardButtons(QMessageBox.Cancel)
|
||||
msg.setDefaultButton(btn_rename)
|
||||
msg.exec()
|
||||
|
||||
clicked = msg.clickedButton()
|
||||
if clicked is btn_open:
|
||||
_switch_to_existing_tool(self.main_window, name)
|
||||
return
|
||||
elif clicked is btn_copy:
|
||||
suffix = 2
|
||||
while (target_root / f"{name}-{suffix}" / "config.yaml").exists():
|
||||
suffix += 1
|
||||
copy_name = f"{name}-{suffix}"
|
||||
self.name_input.setText(copy_name)
|
||||
self.name_input.setFocus()
|
||||
return
|
||||
elif clicked is btn_rename:
|
||||
self.name_input.setFocus()
|
||||
return
|
||||
else:
|
||||
return
|
||||
|
||||
description = self.desc_input.text().strip()
|
||||
category = self.category_combo.currentText()
|
||||
output = self.output_input.toPlainText().strip() or "{response}"
|
||||
@@ -1088,8 +1467,92 @@ class ToolBuilderPage(QWidget):
|
||||
steps=self._tool.steps if self._tool else [],
|
||||
output=output,
|
||||
dependencies=self._tool.dependencies if self._tool else [],
|
||||
system_dependencies=self._tool.system_dependencies if self._tool else []
|
||||
system_dependencies=self._tool.system_dependencies if self._tool else [],
|
||||
visibility=self._tool.visibility if self._tool else "public",
|
||||
input_schema=self._tool.input_schema if self._tool else None,
|
||||
output_schema=self._tool.output_schema if self._tool else None,
|
||||
)
|
||||
if self._tool and self._tool.path:
|
||||
tool.path = self._tool.path
|
||||
|
||||
from ...preflight import analyze_tool
|
||||
preflight = analyze_tool(tool, include_contract_tests=True)
|
||||
if preflight.errors:
|
||||
QMessageBox.warning(
|
||||
self,
|
||||
"Preflight Failed",
|
||||
"\n".join(preflight.errors),
|
||||
)
|
||||
return
|
||||
|
||||
proposal = preflight.contract_proposal or {}
|
||||
proposed_schemas = {
|
||||
key: proposal.get(key)
|
||||
for key in ("input_schema", "output_schema")
|
||||
if proposal.get(key) is not None
|
||||
}
|
||||
if proposed_schemas:
|
||||
msg = QMessageBox(self)
|
||||
msg.setIcon(QMessageBox.Information)
|
||||
msg.setWindowTitle("Contract Proposal")
|
||||
msg.setText(
|
||||
"CmdForge inferred a possible tool contract. "
|
||||
"It will not be applied automatically."
|
||||
)
|
||||
msg.setInformativeText(
|
||||
"Review the proposal below. Applying it changes only this "
|
||||
"in-memory draft and remains reversible until you save."
|
||||
)
|
||||
msg.setDetailedText(
|
||||
yaml.safe_dump(proposed_schemas, sort_keys=False).rstrip()
|
||||
)
|
||||
apply_button = msg.addButton(
|
||||
"Apply Proposal", QMessageBox.AcceptRole
|
||||
)
|
||||
save_button = msg.addButton(
|
||||
"Save Without Applying", QMessageBox.AcceptRole
|
||||
)
|
||||
msg.addButton(QMessageBox.Cancel)
|
||||
msg.exec()
|
||||
if msg.clickedButton() is apply_button:
|
||||
if proposed_schemas.get("input_schema") is not None:
|
||||
tool.input_schema = proposed_schemas["input_schema"]
|
||||
if proposed_schemas.get("output_schema") is not None:
|
||||
tool.output_schema = proposed_schemas["output_schema"]
|
||||
elif msg.clickedButton() is not save_button:
|
||||
return
|
||||
|
||||
if preflight.reuse_opportunities:
|
||||
reuse = QMessageBox(self)
|
||||
reuse.setIcon(QMessageBox.Information)
|
||||
reuse.setWindowTitle("Reuse Opportunities")
|
||||
reuse.setText(
|
||||
"CmdForge found exact, contract-backed reuse evidence. "
|
||||
"This is advisory and will not change your draft."
|
||||
)
|
||||
reuse.setDetailedText("\n".join(
|
||||
f"- {item['detail']}\n Evidence: {item.get('evidence', 'exact match')}"
|
||||
for item in preflight.reuse_opportunities
|
||||
))
|
||||
continue_button = reuse.addButton(
|
||||
"Continue With Draft", QMessageBox.AcceptRole
|
||||
)
|
||||
matching = next(
|
||||
(item.get("tool") for item in preflight.reuse_opportunities if item.get("tool")),
|
||||
None,
|
||||
)
|
||||
open_button = (
|
||||
reuse.addButton("Open Existing", QMessageBox.ActionRole)
|
||||
if matching else None
|
||||
)
|
||||
reuse.addButton(QMessageBox.Cancel)
|
||||
reuse.exec()
|
||||
if open_button and reuse.clickedButton() is open_button:
|
||||
_switch_to_existing_tool(self.main_window, matching)
|
||||
return
|
||||
if reuse.clickedButton() is not continue_button:
|
||||
return
|
||||
advisory_count = len(preflight.warnings) + len(preflight.suggestions)
|
||||
|
||||
# Preserve source if editing
|
||||
if self._tool and self._tool.source:
|
||||
@@ -1100,7 +1563,7 @@ class ToolBuilderPage(QWidget):
|
||||
tool.version = self._tool.version
|
||||
|
||||
try:
|
||||
config_path = save_tool(tool)
|
||||
config_path = save_tool(tool, tools_dir=target_root)
|
||||
tool_dir = config_path.parent
|
||||
|
||||
# Save defaults if provided
|
||||
@@ -1125,7 +1588,10 @@ class ToolBuilderPage(QWidget):
|
||||
if settings_path.exists():
|
||||
settings_path.unlink()
|
||||
|
||||
self.main_window.show_status(f"Saved tool '{name}'")
|
||||
status = f"Saved tool '{name}'"
|
||||
if advisory_count:
|
||||
status += f" with {advisory_count} preflight suggestion(s)"
|
||||
self.main_window.show_status(status)
|
||||
self.main_window.close_tool_builder()
|
||||
except Exception as e:
|
||||
QMessageBox.critical(self, "Error", f"Failed to save tool:\n{e}")
|
||||
@@ -1134,6 +1600,14 @@ class ToolBuilderPage(QWidget):
|
||||
"""Cancel and return to tools page."""
|
||||
self.main_window.close_tool_builder()
|
||||
|
||||
def _selected_tools_dir(self) -> Path:
|
||||
"""Return the save root selected by the author."""
|
||||
if self.editing and self._tool and self._tool.path:
|
||||
return existing_tool_root(self._tool.path)
|
||||
if self.save_location_combo.currentData() == "project":
|
||||
return get_project_tools_dir()
|
||||
return get_tools_dir().resolve()
|
||||
|
||||
def save_tool(self):
|
||||
"""Public method for keyboard shortcut to save the tool."""
|
||||
self._save()
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
"""Tools page - main view for managing tools."""
|
||||
|
||||
import html
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple, Dict, Any
|
||||
@@ -852,6 +853,16 @@ class ToolsPage(QWidget):
|
||||
lines.append(f"<h2 style='margin: 0 0 8px 0; color: #2d3748;'>{tool.name}</h2>")
|
||||
if tool.description:
|
||||
lines.append(f"<p style='color: #4a5568; margin-bottom: 16px;'>{tool.description}</p>")
|
||||
if tool.deprecated:
|
||||
guidance = html.escape(
|
||||
tool.deprecated_message or "This tool is no longer maintained."
|
||||
)
|
||||
if tool.replacement:
|
||||
guidance += f" Use <strong>{html.escape(tool.replacement)}</strong> instead."
|
||||
lines.append(
|
||||
"<p style='background: #fef3c7; color: #92400e; padding: 8px 10px; "
|
||||
f"border-radius: 4px;'><strong>Deprecated:</strong> {guidance}</p>"
|
||||
)
|
||||
|
||||
# Publish state
|
||||
state, registry_hash = get_tool_publish_state(qname)
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Scrutiny-driven improvement pipeline (M9.2).
|
||||
|
||||
Takes a tool with low scrutiny scores and generates actionable improvement
|
||||
suggestions for honesty, efficiency, and transparency.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .tool import Tool, PromptStep, CodeStep
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImprovementSuggestion:
|
||||
"""One actionable improvement suggestion."""
|
||||
|
||||
category: str # "honesty" | "efficiency" | "transparency"
|
||||
severity: str # "high" | "medium" | "low"
|
||||
title: str
|
||||
description: str
|
||||
location: str = "" # step index or "tool"
|
||||
current: str = "" # current code/prompt text
|
||||
suggested: str = "" # suggested replacement
|
||||
auto_applicable: bool = False
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"category": self.category,
|
||||
"severity": self.severity,
|
||||
"title": self.title,
|
||||
"description": self.description,
|
||||
"location": self.location,
|
||||
"current": self.current[:200],
|
||||
"suggested": self.suggested[:200],
|
||||
"auto_applicable": self.auto_applicable,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ImprovementReport:
|
||||
"""Collection of improvement suggestions for a tool."""
|
||||
|
||||
tool_name: str
|
||||
suggestions: List[ImprovementSuggestion] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def has_high_severity(self) -> bool:
|
||||
return any(s.severity == "high" for s in self.suggestions)
|
||||
|
||||
@property
|
||||
def by_category(self) -> Dict[str, List[ImprovementSuggestion]]:
|
||||
result: Dict[str, List[ImprovementSuggestion]] = {}
|
||||
for s in self.suggestions:
|
||||
result.setdefault(s.category, []).append(s)
|
||||
return result
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"tool": self.tool_name,
|
||||
"suggestion_count": len(self.suggestions),
|
||||
"has_high_severity": self.has_high_severity,
|
||||
"suggestions": [s.to_dict() for s in self.suggestions],
|
||||
}
|
||||
|
||||
|
||||
def generate_improvements(
|
||||
tool: Tool, scrutiny_report: Optional[dict] = None
|
||||
) -> ImprovementReport:
|
||||
"""Analyze a tool and generate improvement suggestions.
|
||||
|
||||
Categories:
|
||||
- Honesty: description doesn't match behavior
|
||||
- Efficiency: unnecessary AI calls, redundant steps
|
||||
- Transparency: obfuscated code, unclear prompts
|
||||
"""
|
||||
report = ImprovementReport(tool_name=tool.name)
|
||||
|
||||
_check_honesty(tool, report)
|
||||
_check_efficiency(tool, report)
|
||||
_check_transparency(tool, report)
|
||||
|
||||
# Preserve the registry scrutiny engine's evidence rather than running a
|
||||
# disconnected approximation. Only actionable warnings/failures become
|
||||
# suggestions; passing checks are deliberately omitted.
|
||||
for finding in (scrutiny_report or {}).get("findings", []):
|
||||
result = finding.get("result")
|
||||
if result not in ("warning", "fail"):
|
||||
continue
|
||||
check = str(finding.get("check", "scrutiny")).lower()
|
||||
category = next(
|
||||
(value for value in ("honesty", "efficiency", "transparency")
|
||||
if value in check),
|
||||
"transparency",
|
||||
)
|
||||
suggestion = finding.get("suggestion") or (
|
||||
"Review this finding and make the behavior explicit."
|
||||
)
|
||||
candidate = ImprovementSuggestion(
|
||||
category=category,
|
||||
severity="high" if result == "fail" else "medium",
|
||||
title=f"Scrutiny: {finding.get('check', 'review required')}",
|
||||
description=finding.get("message", "Scrutiny identified an issue."),
|
||||
location=finding.get("location") or "tool",
|
||||
suggested=suggestion,
|
||||
)
|
||||
marker = (candidate.category, candidate.title, candidate.location)
|
||||
existing = {(s.category, s.title, s.location) for s in report.suggestions}
|
||||
if marker not in existing:
|
||||
report.suggestions.append(candidate)
|
||||
|
||||
return report
|
||||
|
||||
|
||||
def _check_honesty(tool: Tool, report: ImprovementReport):
|
||||
"""Check description-to-behavior alignment."""
|
||||
if not tool.description:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="honesty",
|
||||
severity="high",
|
||||
title="Missing description",
|
||||
description="Tool has no description. Users cannot know what it does without reading the config.",
|
||||
location="tool",
|
||||
auto_applicable=False,
|
||||
))
|
||||
return
|
||||
|
||||
# Check if description mentions capabilities not in steps
|
||||
desc_lower = tool.description.lower()
|
||||
has_prompt = any(isinstance(s, PromptStep) for s in tool.steps)
|
||||
has_code = any(isinstance(s, CodeStep) for s in tool.steps)
|
||||
|
||||
if "ai" in desc_lower or "generate" in desc_lower or "analyze" in desc_lower:
|
||||
if not has_prompt:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="honesty",
|
||||
severity="medium",
|
||||
title="Description implies AI but no prompt step found",
|
||||
description="The description mentions AI/generation/analysis but the tool has no PromptStep.",
|
||||
location="tool",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
# Check if description is too vague
|
||||
if len(tool.description) < 20:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="honesty",
|
||||
severity="low",
|
||||
title="Description is very short",
|
||||
description="A more detailed description helps users find and trust this tool.",
|
||||
location="tool",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
|
||||
def _check_efficiency(tool: Tool, report: ImprovementReport):
|
||||
"""Check for unnecessary AI calls and redundant steps."""
|
||||
prompt_steps = [s for s in tool.steps if isinstance(s, PromptStep)]
|
||||
|
||||
# Multiple prompt steps with same provider — could be combined
|
||||
if len(prompt_steps) > 1:
|
||||
providers = [s.provider for s in prompt_steps]
|
||||
if len(set(providers)) == 1:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="efficiency",
|
||||
severity="medium",
|
||||
title="Multiple prompt steps with same provider",
|
||||
description=f"Found {len(prompt_steps)} prompt steps all using '{providers[0]}'. "
|
||||
"Consider combining them into a single step to reduce API calls.",
|
||||
location=f"steps 0-{len(prompt_steps)-1}",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
# Prompt step that just passes input through
|
||||
for i, step in enumerate(tool.steps):
|
||||
if isinstance(step, PromptStep):
|
||||
if step.prompt.strip() in ("{input}", "Repeat: {input}", "Echo: {input}"):
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="efficiency",
|
||||
severity="high",
|
||||
title="Prompt step just echoes input",
|
||||
description="This prompt step appears to just pass input through without transformation. "
|
||||
"Remove it and use the input directly.",
|
||||
location=f"step {i}",
|
||||
current=step.prompt,
|
||||
suggested="(remove this step — use {input} directly)",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
# No steps at all — output is just a template
|
||||
if not tool.steps and "{input}" in tool.output:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="efficiency",
|
||||
severity="low",
|
||||
title="Tool has no steps",
|
||||
description="This tool only does variable substitution. Consider if it needs to be a CmdForge tool "
|
||||
"or if a simple shell alias would suffice.",
|
||||
location="tool",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
|
||||
def _check_transparency(tool: Tool, report: ImprovementReport):
|
||||
"""Check for obfuscated code and unclear prompts."""
|
||||
for i, step in enumerate(tool.steps):
|
||||
if isinstance(step, CodeStep):
|
||||
# Check for overly complex code
|
||||
lines = step.code.strip().split("\n")
|
||||
if len(lines) > 20:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="transparency",
|
||||
severity="medium",
|
||||
title=f"Code step {i} is complex ({len(lines)} lines)",
|
||||
description="Long code steps are harder to audit. Consider extracting into a separate "
|
||||
"tool or adding comments.",
|
||||
location=f"step {i}",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
# Check for exec/eval
|
||||
if "exec(" in step.code or "eval(" in step.code:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="transparency",
|
||||
severity="high",
|
||||
title=f"Code step {i} uses exec/eval",
|
||||
description="Using exec() or eval() in code steps makes the tool harder to audit and "
|
||||
"poses security risks. Consider restructuring.",
|
||||
location=f"step {i}",
|
||||
auto_applicable=False,
|
||||
))
|
||||
|
||||
if isinstance(step, PromptStep):
|
||||
# Check for very short prompts
|
||||
if len(step.prompt.strip()) < 20:
|
||||
report.suggestions.append(ImprovementSuggestion(
|
||||
category="transparency",
|
||||
severity="medium",
|
||||
title=f"Prompt step {i} is very short",
|
||||
description="Short prompts give the AI model less context, leading to unpredictable output. "
|
||||
"Add more detail about the expected format and content.",
|
||||
location=f"step {i}",
|
||||
current=step.prompt,
|
||||
auto_applicable=False,
|
||||
))
|
||||
@@ -0,0 +1,180 @@
|
||||
"""Content-addressable, transitive integrity verification for CmdForge tools."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Set
|
||||
|
||||
from .tool import Tool, ToolStep, load_tool
|
||||
|
||||
|
||||
def _sha256(value: object) -> str:
|
||||
encoded = json.dumps(
|
||||
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
|
||||
).encode("utf-8")
|
||||
return f"sha256:{hashlib.sha256(encoded).hexdigest()}"
|
||||
|
||||
|
||||
@dataclass
|
||||
class IntegrityNode:
|
||||
name: str
|
||||
version: str = ""
|
||||
definition_hash: str = ""
|
||||
content_hash: str = ""
|
||||
dependencies: List[str] = field(default_factory=list)
|
||||
dependency_hashes: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"name": self.name,
|
||||
"version": self.version,
|
||||
"definition_hash": self.definition_hash,
|
||||
"hash": self.content_hash,
|
||||
"dependencies": self.dependencies,
|
||||
"dependency_hashes": self.dependency_hashes,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class IntegrityChain:
|
||||
root: IntegrityNode
|
||||
nodes: Dict[str, IntegrityNode] = field(default_factory=dict)
|
||||
errors: List[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def is_valid(self) -> bool:
|
||||
if self.errors or self.root.name not in self.nodes:
|
||||
return False
|
||||
for name, node in self.nodes.items():
|
||||
if not node.definition_hash or not node.content_hash:
|
||||
return False
|
||||
if sorted(node.dependencies) != sorted(node.dependency_hashes):
|
||||
return False
|
||||
for dep_name in node.dependencies:
|
||||
dep = self.nodes.get(dep_name)
|
||||
if dep is None or node.dependency_hashes[dep_name] != dep.content_hash:
|
||||
return False
|
||||
expected = _identity_hash(node.definition_hash, node.dependency_hashes)
|
||||
if not _constant_time_equal(expected, node.content_hash):
|
||||
return False
|
||||
return True
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"root": self.root.to_dict(),
|
||||
"nodes": {key: value.to_dict() for key, value in self.nodes.items()},
|
||||
"errors": list(self.errors),
|
||||
"valid": self.is_valid,
|
||||
}
|
||||
|
||||
|
||||
def _constant_time_equal(left: str, right: str) -> bool:
|
||||
import hmac
|
||||
return hmac.compare_digest(left, right)
|
||||
|
||||
|
||||
def compute_tool_hash(tool: Tool) -> str:
|
||||
"""Return the full SHA-256 hash of the canonical tool definition."""
|
||||
data = tool.to_dict()
|
||||
data.pop("path", None)
|
||||
# Registry bookkeeping is not executable tool content.
|
||||
for key in ("registry_hash", "registry_status", "registry_feedback"):
|
||||
data.pop(key, None)
|
||||
return _sha256(data)
|
||||
|
||||
|
||||
def _identity_hash(definition_hash: str, dependency_hashes: Dict[str, str]) -> str:
|
||||
"""Hash a definition together with the identities of every direct dep."""
|
||||
return _sha256({
|
||||
"definition": definition_hash,
|
||||
"dependencies": dict(sorted(dependency_hashes.items())),
|
||||
})
|
||||
|
||||
|
||||
def compute_content_identity(
|
||||
definition_hash: str, dependency_hashes: Dict[str, str]
|
||||
) -> str:
|
||||
"""Return the canonical identity used by integrity chains and lockfiles."""
|
||||
return _identity_hash(definition_hash, dependency_hashes)
|
||||
|
||||
|
||||
def _dependency_names(tool: Tool) -> List[str]:
|
||||
names = list(tool.dependencies)
|
||||
names.extend(step.tool for step in tool.steps if isinstance(step, ToolStep))
|
||||
return sorted(set(names))
|
||||
|
||||
|
||||
def build_integrity_chain(tool: Tool, max_depth: int = 50) -> IntegrityChain:
|
||||
"""Resolve dependencies bottom-up and compute a Merkle-style identity."""
|
||||
placeholder = IntegrityNode(tool.name, tool.version or "")
|
||||
chain = IntegrityChain(root=placeholder)
|
||||
visiting: Set[str] = set()
|
||||
|
||||
def visit(current: Tool, depth: int) -> Optional[IntegrityNode]:
|
||||
name = current.name
|
||||
if name in visiting:
|
||||
chain.errors.append(f"Dependency cycle detected at {name}")
|
||||
return None
|
||||
if name in chain.nodes:
|
||||
return chain.nodes[name]
|
||||
if depth > max_depth:
|
||||
chain.errors.append(f"Maximum dependency depth exceeded at {name}")
|
||||
return None
|
||||
|
||||
visiting.add(name)
|
||||
dep_hashes: Dict[str, str] = {}
|
||||
dependencies = _dependency_names(current)
|
||||
for dep_name in dependencies:
|
||||
from .resolver import ToolSpec
|
||||
from .semver import matches_constraint
|
||||
dep_spec = ToolSpec.parse(dep_name)
|
||||
dep_tool = load_tool(dep_spec.full_name)
|
||||
if dep_tool is None:
|
||||
chain.errors.append(f"Unresolved dependency: {dep_name}")
|
||||
continue
|
||||
if dep_spec.version and not matches_constraint(
|
||||
dep_tool.version or "", dep_spec.version
|
||||
):
|
||||
chain.errors.append(
|
||||
f"Dependency version mismatch: {dep_name} "
|
||||
f"resolved to {dep_tool.version or 'unknown'}"
|
||||
)
|
||||
continue
|
||||
dep_node = visit(dep_tool, depth + 1)
|
||||
if dep_node is not None:
|
||||
dep_hashes[dep_name] = dep_node.content_hash
|
||||
visiting.remove(name)
|
||||
|
||||
definition_hash = compute_tool_hash(current)
|
||||
node = IntegrityNode(
|
||||
name=name,
|
||||
version=current.version or "",
|
||||
definition_hash=definition_hash,
|
||||
content_hash=_identity_hash(definition_hash, dep_hashes),
|
||||
dependencies=dependencies,
|
||||
dependency_hashes=dep_hashes,
|
||||
)
|
||||
chain.nodes[name] = node
|
||||
return node
|
||||
|
||||
root = visit(tool, 0)
|
||||
if root is not None:
|
||||
chain.root = root
|
||||
return chain
|
||||
|
||||
|
||||
def verify_integrity(chain: IntegrityChain, *, verify_installed: bool = False) -> bool:
|
||||
"""Verify chain structure, optionally rebuilding it from installed tools."""
|
||||
if not chain.is_valid:
|
||||
return False
|
||||
if not verify_installed:
|
||||
return True
|
||||
root_tool = load_tool(chain.root.name)
|
||||
if root_tool is None:
|
||||
return False
|
||||
rebuilt = build_integrity_chain(root_tool)
|
||||
return rebuilt.is_valid and _constant_time_equal(
|
||||
rebuilt.root.content_hash, chain.root.content_hash
|
||||
)
|
||||
@@ -26,6 +26,8 @@ class LockedPackage:
|
||||
direct: bool # True if in manifest
|
||||
required_by: List[str] = field(default_factory=list) # Parent packages
|
||||
path: Optional[str] = None # Relative path for local tools
|
||||
content_hash: str = "" # Definition plus dependency identities
|
||||
dependency_hashes: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def owner(self) -> str:
|
||||
@@ -145,7 +147,9 @@ class Lockfile:
|
||||
source=pkg_data.get("source", "registry"),
|
||||
direct=pkg_data.get("direct", False),
|
||||
required_by=pkg_data.get("required_by", []),
|
||||
path=pkg_data.get("path")
|
||||
path=pkg_data.get("path"),
|
||||
content_hash=pkg_data.get("content_hash", ""),
|
||||
dependency_hashes=pkg_data.get("dependency_hashes", {})
|
||||
)
|
||||
|
||||
return cls(
|
||||
@@ -180,6 +184,10 @@ class Lockfile:
|
||||
pkg_dict["required_by"] = pkg.required_by
|
||||
if pkg.path:
|
||||
pkg_dict["path"] = pkg.path
|
||||
if pkg.content_hash:
|
||||
pkg_dict["content_hash"] = pkg.content_hash
|
||||
if pkg.dependency_hashes:
|
||||
pkg_dict["dependency_hashes"] = dict(sorted(pkg.dependency_hashes.items()))
|
||||
d["packages"][name] = pkg_dict
|
||||
|
||||
return d
|
||||
@@ -259,6 +267,34 @@ def generate_lockfile(
|
||||
)
|
||||
lock.packages[qualified_name] = pkg
|
||||
|
||||
from .integrity import compute_content_identity
|
||||
visiting = set()
|
||||
|
||||
def identity_for(package_name: str) -> str:
|
||||
pkg = lock.packages.get(package_name)
|
||||
node = graph.nodes.get(package_name)
|
||||
if pkg is None or node is None or not pkg.integrity:
|
||||
return ""
|
||||
if pkg.content_hash:
|
||||
return pkg.content_hash
|
||||
if package_name in visiting:
|
||||
return ""
|
||||
visiting.add(package_name)
|
||||
dependencies = {}
|
||||
for child_name in sorted(node.children):
|
||||
child_hash = identity_for(child_name)
|
||||
if not child_hash:
|
||||
visiting.remove(package_name)
|
||||
return ""
|
||||
dependencies[child_name] = child_hash
|
||||
visiting.remove(package_name)
|
||||
pkg.dependency_hashes = dependencies
|
||||
pkg.content_hash = compute_content_identity(pkg.integrity, dependencies)
|
||||
return pkg.content_hash
|
||||
|
||||
for package_name in sorted(lock.packages):
|
||||
identity_for(package_name)
|
||||
|
||||
return lock
|
||||
|
||||
|
||||
@@ -368,4 +404,30 @@ def verify_lockfile(
|
||||
except Exception as e:
|
||||
errors.append(f"{name}: could not verify integrity ({e})")
|
||||
|
||||
if locked.content_hash:
|
||||
from .integrity import compute_content_identity
|
||||
actual_dependencies = {}
|
||||
missing_dependencies = []
|
||||
for dep_name, expected_hash in locked.dependency_hashes.items():
|
||||
dep = lock.packages.get(dep_name)
|
||||
if dep is None or not dep.content_hash:
|
||||
missing_dependencies.append(dep_name)
|
||||
elif dep.content_hash != expected_hash:
|
||||
errors.append(
|
||||
f"{name}: dependency identity mismatch ({dep_name})"
|
||||
)
|
||||
else:
|
||||
actual_dependencies[dep_name] = dep.content_hash
|
||||
if missing_dependencies:
|
||||
errors.append(
|
||||
f"{name}: unresolved integrity dependencies: "
|
||||
+ ", ".join(missing_dependencies)
|
||||
)
|
||||
elif locked.integrity:
|
||||
actual_identity = compute_content_identity(
|
||||
locked.integrity, actual_dependencies
|
||||
)
|
||||
if actual_identity != locked.content_hash:
|
||||
errors.append(f"{name}: transitive content identity mismatch")
|
||||
|
||||
return errors
|
||||
|
||||
@@ -0,0 +1,668 @@
|
||||
"""MCP client support built on the official MCP Python SDK."""
|
||||
|
||||
import asyncio
|
||||
import contextvars
|
||||
import fnmatch
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any, Awaitable, Callable, Dict, List, Optional, TypeVar
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import yaml
|
||||
|
||||
MCP_CONFIG_FILE = Path.home() / ".cmdforge" / "mcp.yaml"
|
||||
MCP_CONFIG_VERSION = 1
|
||||
MCP_DEPTH_ENV = "CMDFORGE_MCP_DEPTH"
|
||||
RESULT_MODES = ("auto", "structured", "content", "text")
|
||||
SUPPORTED_TRANSPORTS = ("stdio", "streamable-http")
|
||||
DEFAULT_INHERITED_ENV = (
|
||||
"PATH",
|
||||
"HOME",
|
||||
"USER",
|
||||
"LOGNAME",
|
||||
"TMPDIR",
|
||||
"TMP",
|
||||
"TEMP",
|
||||
"SYSTEMROOT",
|
||||
"WINDIR",
|
||||
"COMSPEC",
|
||||
"PATHEXT",
|
||||
)
|
||||
|
||||
_ENV_REFERENCE = re.compile(r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}")
|
||||
_ENV_NAME = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
_SECRET_PATTERNS = (
|
||||
re.compile(r"(?i)bearer\s+\S+"),
|
||||
re.compile(r"(?i)\b(?:sk|ghp|github_pat)-[A-Za-z0-9_-]+"),
|
||||
re.compile(r"(?i)(?:api[_-]?key|token|secret|password)\s*[:=]\s*\S+"),
|
||||
)
|
||||
|
||||
T = TypeVar("T")
|
||||
_MCP_CALL_DEPTH: contextvars.ContextVar[int] = contextvars.ContextVar(
|
||||
"cmdforge_mcp_call_depth", default=0
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class McpServeConfig:
|
||||
"""Server-mode configuration: which CmdForge tools to expose via MCP."""
|
||||
|
||||
expose: List[str] = field(default_factory=list)
|
||||
deny: List[str] = field(default_factory=list)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for field_name, patterns in (("expose", self.expose), ("deny", self.deny)):
|
||||
if not isinstance(patterns, list) or not all(
|
||||
isinstance(pattern, str) and pattern for pattern in patterns
|
||||
):
|
||||
raise ValueError(
|
||||
f"MCP server '{field_name}' must be a list of non-empty strings"
|
||||
)
|
||||
|
||||
def is_exposed(self, tool_name: str) -> bool:
|
||||
"""Check whether a tool should be exposed to MCP clients."""
|
||||
if self._matches_deny(tool_name):
|
||||
return False
|
||||
if not self.expose:
|
||||
return False
|
||||
return self._matches_expose(tool_name)
|
||||
|
||||
def _matches_deny(self, name: str) -> bool:
|
||||
for pattern in self.deny:
|
||||
if _glob_match(pattern, name):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _matches_expose(self, name: str) -> bool:
|
||||
for pattern in self.expose:
|
||||
if _glob_match(pattern, name) or pattern == "*":
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _glob_match(pattern: str, name: str) -> bool:
|
||||
"""Match a policy pattern using shell-style wildcards."""
|
||||
return fnmatch.fnmatchcase(name, pattern)
|
||||
|
||||
|
||||
@dataclass
|
||||
class McpServerConfig:
|
||||
"""Configuration for one MCP server.
|
||||
|
||||
``approved`` records explicit user consent to execute a local command or
|
||||
connect to the configured HTTP endpoint.
|
||||
"""
|
||||
|
||||
name: str
|
||||
transport: str = "stdio"
|
||||
command: Optional[str] = None
|
||||
url: Optional[str] = None
|
||||
headers: Dict[str, str] = field(default_factory=dict)
|
||||
args: List[str] = field(default_factory=list)
|
||||
cwd: Optional[str] = None
|
||||
env: Dict[str, str] = field(default_factory=dict)
|
||||
inherit_env: List[str] = field(default_factory=lambda: list(DEFAULT_INHERITED_ENV))
|
||||
timeout: float = 30
|
||||
description: str = ""
|
||||
approved: bool = False
|
||||
|
||||
def validate(self) -> None:
|
||||
if not isinstance(self.name, str) or not self.name.strip():
|
||||
raise ValueError("MCP server name must be a non-empty string")
|
||||
if self.transport not in SUPPORTED_TRANSPORTS:
|
||||
raise ValueError(
|
||||
f"Unsupported MCP transport '{self.transport}'. "
|
||||
f"Supported transports: {', '.join(SUPPORTED_TRANSPORTS)}"
|
||||
)
|
||||
if self.transport == "stdio":
|
||||
if not isinstance(self.command, str) or not self.command.strip():
|
||||
raise ValueError(f"MCP server '{self.name}' requires a command")
|
||||
if self.url is not None:
|
||||
raise ValueError(f"MCP stdio server '{self.name}' cannot define url")
|
||||
if self.headers:
|
||||
raise ValueError(f"MCP stdio server '{self.name}' cannot define headers")
|
||||
else:
|
||||
if self.command is not None:
|
||||
raise ValueError(
|
||||
f"MCP streamable-http server '{self.name}' cannot define command"
|
||||
)
|
||||
_validate_remote_url(self.name, self.url)
|
||||
if self.args or self.cwd or self.env:
|
||||
raise ValueError(
|
||||
f"MCP streamable-http server '{self.name}' cannot define args, cwd, or env"
|
||||
)
|
||||
if not isinstance(self.args, list) or not all(isinstance(arg, str) for arg in self.args):
|
||||
raise ValueError(f"MCP server '{self.name}' args must be a list of strings")
|
||||
if not isinstance(self.env, dict) or not all(
|
||||
isinstance(key, str)
|
||||
and _ENV_NAME.fullmatch(key)
|
||||
and isinstance(value, str)
|
||||
for key, value in self.env.items()
|
||||
):
|
||||
raise ValueError(
|
||||
f"MCP server '{self.name}' env must map valid variable names to strings"
|
||||
)
|
||||
if not isinstance(self.headers, dict) or not all(
|
||||
isinstance(key, str) and key.strip()
|
||||
and "\n" not in key and "\r" not in key
|
||||
and isinstance(value, str) and "\n" not in value and "\r" not in value
|
||||
for key, value in self.headers.items()
|
||||
):
|
||||
raise ValueError(
|
||||
f"MCP server '{self.name}' headers must map safe names to strings"
|
||||
)
|
||||
if not isinstance(self.inherit_env, list) or not all(
|
||||
isinstance(name, str) and _ENV_NAME.fullmatch(name)
|
||||
for name in self.inherit_env
|
||||
):
|
||||
raise ValueError(
|
||||
f"MCP server '{self.name}' inherit_env must contain valid variable names"
|
||||
)
|
||||
if isinstance(self.timeout, bool) or not isinstance(self.timeout, (int, float)):
|
||||
raise ValueError(f"MCP server '{self.name}' timeout must be a number")
|
||||
if self.timeout <= 0 or self.timeout > 3600:
|
||||
raise ValueError(
|
||||
f"MCP server '{self.name}' timeout must be greater than 0 and at most 3600 seconds"
|
||||
)
|
||||
if self.cwd is not None and not isinstance(self.cwd, str):
|
||||
raise ValueError(f"MCP server '{self.name}' cwd must be a string")
|
||||
if not isinstance(self.approved, bool):
|
||||
raise ValueError(f"MCP server '{self.name}' approved must be a boolean")
|
||||
|
||||
|
||||
def _fingerprint(cfg: McpServerConfig) -> str:
|
||||
payload = json.dumps(
|
||||
[
|
||||
cfg.transport,
|
||||
cfg.command,
|
||||
cfg.url,
|
||||
cfg.headers,
|
||||
cfg.args,
|
||||
cfg.cwd,
|
||||
cfg.env,
|
||||
cfg.inherit_env,
|
||||
cfg.timeout,
|
||||
cfg.approved,
|
||||
],
|
||||
sort_keys=True,
|
||||
)
|
||||
return hashlib.sha256(payload.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
def _sanitize(value: str, secrets: Optional[List[str]] = None) -> str:
|
||||
"""Redact common credential forms and explicit resolved secret values."""
|
||||
sanitized = str(value)
|
||||
for secret in secrets or []:
|
||||
if secret:
|
||||
sanitized = sanitized.replace(secret, "[redacted]")
|
||||
for pattern in _SECRET_PATTERNS:
|
||||
sanitized = pattern.sub("[redacted]", sanitized)
|
||||
return sanitized
|
||||
|
||||
|
||||
def _expand_env_value(value: str) -> str:
|
||||
def replace(match: re.Match) -> str:
|
||||
name = match.group(1)
|
||||
if name not in os.environ:
|
||||
raise ValueError(f"Environment variable '{name}' referenced by MCP config is not set")
|
||||
return os.environ[name]
|
||||
|
||||
return _ENV_REFERENCE.sub(replace, value)
|
||||
|
||||
|
||||
def _build_server_env(cfg: McpServerConfig) -> Dict[str, str]:
|
||||
"""Build a minimal subprocess environment without leaking all credentials."""
|
||||
environment = {
|
||||
name: os.environ[name]
|
||||
for name in cfg.inherit_env
|
||||
if name in os.environ
|
||||
}
|
||||
for name, value in cfg.env.items():
|
||||
environment[name] = _expand_env_value(value)
|
||||
call_depth = _MCP_CALL_DEPTH.get()
|
||||
if call_depth > 0:
|
||||
environment[MCP_DEPTH_ENV] = str(call_depth)
|
||||
return environment
|
||||
|
||||
|
||||
def _is_loopback_host(hostname: Optional[str]) -> bool:
|
||||
if not hostname:
|
||||
return False
|
||||
if hostname.lower() == "localhost":
|
||||
return True
|
||||
try:
|
||||
import ipaddress
|
||||
return ipaddress.ip_address(hostname).is_loopback
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
|
||||
def _validate_remote_url(name: str, url: Optional[str]) -> None:
|
||||
if not isinstance(url, str) or not url.strip():
|
||||
raise ValueError(f"MCP streamable-http server '{name}' requires a url")
|
||||
parsed = urlparse(url)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||
raise ValueError(f"MCP server '{name}' has an invalid HTTP URL")
|
||||
if parsed.username or parsed.password:
|
||||
raise ValueError("MCP URLs must not contain credentials; use headers")
|
||||
if parsed.scheme != "https" and not _is_loopback_host(parsed.hostname):
|
||||
raise ValueError(
|
||||
f"Remote MCP server '{name}' must use HTTPS; HTTP is allowed only for localhost"
|
||||
)
|
||||
|
||||
|
||||
def _build_http_headers(cfg: McpServerConfig) -> Dict[str, str]:
|
||||
return {name: _expand_env_value(value) for name, value in cfg.headers.items()}
|
||||
|
||||
|
||||
def _load_mcp_document() -> dict:
|
||||
if not MCP_CONFIG_FILE.exists():
|
||||
return {}
|
||||
data = yaml.safe_load(MCP_CONFIG_FILE.read_text()) or {}
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("MCP config must be a YAML mapping")
|
||||
version = data.get("version", MCP_CONFIG_VERSION)
|
||||
if version != MCP_CONFIG_VERSION:
|
||||
raise ValueError(f"Unsupported MCP config version: {version}")
|
||||
return data
|
||||
|
||||
|
||||
def load_mcp_config() -> List[McpServerConfig]:
|
||||
data = _load_mcp_document()
|
||||
raw_servers = data.get("servers") or {}
|
||||
if not isinstance(raw_servers, dict):
|
||||
raise ValueError("MCP config 'servers' must be a mapping")
|
||||
|
||||
servers = []
|
||||
for name, raw in raw_servers.items():
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError(f"MCP server '{name}' config must be a mapping")
|
||||
cfg = McpServerConfig(
|
||||
name=name,
|
||||
transport=raw.get("transport", "stdio"),
|
||||
command=raw.get("command"),
|
||||
url=raw.get("url"),
|
||||
headers=raw.get("headers", {}),
|
||||
args=raw.get("args", []),
|
||||
cwd=raw.get("cwd"),
|
||||
env=raw.get("env", {}),
|
||||
inherit_env=raw.get("inherit_env", list(DEFAULT_INHERITED_ENV)),
|
||||
timeout=raw.get("timeout", 30),
|
||||
description=raw.get("description", ""),
|
||||
approved=raw.get("approved", False),
|
||||
)
|
||||
cfg.validate()
|
||||
servers.append(cfg)
|
||||
return servers
|
||||
|
||||
|
||||
def save_mcp_config(servers: List[McpServerConfig]) -> None:
|
||||
for server in servers:
|
||||
server.validate()
|
||||
MCP_CONFIG_FILE.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
body = _load_mcp_document()
|
||||
body["version"] = MCP_CONFIG_VERSION
|
||||
body["servers"] = {server.name: _server_to_dict(server) for server in servers}
|
||||
serialized = yaml.safe_dump(body, default_flow_style=False, sort_keys=False)
|
||||
temp_path = None
|
||||
try:
|
||||
MCP_CONFIG_FILE.parent.chmod(0o700)
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
encoding="utf-8",
|
||||
dir=MCP_CONFIG_FILE.parent,
|
||||
prefix=".mcp-",
|
||||
suffix=".yaml.tmp",
|
||||
delete=False,
|
||||
) as temp_file:
|
||||
temp_path = Path(temp_file.name)
|
||||
temp_path.chmod(0o600)
|
||||
temp_file.write(serialized)
|
||||
temp_file.flush()
|
||||
os.fsync(temp_file.fileno())
|
||||
os.replace(temp_path, MCP_CONFIG_FILE)
|
||||
MCP_CONFIG_FILE.chmod(0o600)
|
||||
finally:
|
||||
if temp_path is not None and temp_path.exists():
|
||||
temp_path.unlink()
|
||||
|
||||
|
||||
def _server_to_dict(server: McpServerConfig) -> dict:
|
||||
data: dict = {
|
||||
"transport": server.transport,
|
||||
"approved": server.approved,
|
||||
}
|
||||
if server.command:
|
||||
data["command"] = server.command
|
||||
if server.url:
|
||||
data["url"] = server.url
|
||||
if server.headers:
|
||||
data["headers"] = server.headers
|
||||
if server.args:
|
||||
data["args"] = server.args
|
||||
if server.cwd:
|
||||
data["cwd"] = server.cwd
|
||||
if server.env:
|
||||
data["env"] = server.env
|
||||
if server.inherit_env != list(DEFAULT_INHERITED_ENV):
|
||||
data["inherit_env"] = server.inherit_env
|
||||
if server.timeout != 30:
|
||||
data["timeout"] = server.timeout
|
||||
if server.description:
|
||||
data["description"] = server.description
|
||||
return data
|
||||
|
||||
|
||||
def _require_mcp_sdk() -> None:
|
||||
try:
|
||||
import mcp # noqa: F401
|
||||
except ImportError as exc:
|
||||
raise ImportError(
|
||||
"MCP support requires the mcp package. Install with: pip install 'cmdforge[mcp]'"
|
||||
) from exc
|
||||
|
||||
|
||||
def load_mcp_serve_config() -> McpServeConfig:
|
||||
data = _load_mcp_document()
|
||||
server_raw = data.get("server") or {}
|
||||
if not isinstance(server_raw, dict):
|
||||
raise ValueError("MCP config 'server' must be a mapping")
|
||||
return McpServeConfig(
|
||||
expose=server_raw.get("expose", []),
|
||||
deny=server_raw.get("deny", []),
|
||||
)
|
||||
|
||||
|
||||
def _serialize_tool(tool: Any) -> Dict[str, Any]:
|
||||
if hasattr(tool, "model_dump"):
|
||||
return tool.model_dump(mode="json", by_alias=True, exclude_none=True)
|
||||
if isinstance(tool, dict):
|
||||
return tool
|
||||
return {
|
||||
key: getattr(tool, key)
|
||||
for key in ("name", "description", "inputSchema", "outputSchema")
|
||||
if hasattr(tool, key)
|
||||
}
|
||||
|
||||
|
||||
class McpClientManager:
|
||||
"""Invocation-scoped MCP configuration and schema manager.
|
||||
|
||||
SDK sessions are intentionally scoped to each discovery or tool operation.
|
||||
This guarantees transport cleanup while the manager reuses loaded
|
||||
configuration and discovered schemas across all MCP steps in one tool run.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._configs: Dict[str, McpServerConfig] = {}
|
||||
self._schemas: Dict[str, Dict[str, Any]] = {}
|
||||
self._schema_fingerprints: Dict[str, str] = {}
|
||||
self._loaded = False
|
||||
|
||||
def __enter__(self) -> "McpClientManager":
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc_value, traceback) -> None:
|
||||
self.shutdown()
|
||||
|
||||
def _ensure_loaded(self) -> None:
|
||||
if self._loaded:
|
||||
return
|
||||
for cfg in load_mcp_config():
|
||||
self._configs[cfg.name] = cfg
|
||||
self._loaded = True
|
||||
|
||||
def shutdown(self) -> None:
|
||||
self._schemas.clear()
|
||||
self._schema_fingerprints.clear()
|
||||
|
||||
def list_servers(self) -> List[McpServerConfig]:
|
||||
self._ensure_loaded()
|
||||
return list(self._configs.values())
|
||||
|
||||
def _get_approved_config(self, server_name: str) -> McpServerConfig:
|
||||
self._ensure_loaded()
|
||||
cfg = self._configs.get(server_name)
|
||||
if not cfg:
|
||||
raise KeyError(f"MCP server '{server_name}' not configured. Add it to {MCP_CONFIG_FILE}")
|
||||
cfg.validate()
|
||||
if not cfg.approved:
|
||||
command = (
|
||||
" ".join([cfg.command or ""] + cfg.args)
|
||||
if cfg.transport == "stdio" else cfg.url or ""
|
||||
)
|
||||
raise PermissionError(
|
||||
f"MCP server '{server_name}' is not approved to execute: {_sanitize(command)}. "
|
||||
"Re-add it with 'cmdforge mcp add ...' or set approved: true after review."
|
||||
)
|
||||
return cfg
|
||||
|
||||
def discover(self, server_name: str, refresh: bool = False) -> List[Dict[str, Any]]:
|
||||
_require_mcp_sdk()
|
||||
cfg = self._get_approved_config(server_name)
|
||||
fingerprint = _fingerprint(cfg)
|
||||
if (
|
||||
not refresh
|
||||
and server_name in self._schemas
|
||||
and self._schema_fingerprints.get(server_name) == fingerprint
|
||||
):
|
||||
return list(self._schemas[server_name].values())
|
||||
|
||||
async def list_all_tools(session: Any) -> List[Dict[str, Any]]:
|
||||
tools: List[Dict[str, Any]] = []
|
||||
cursor = None
|
||||
seen_cursors = set()
|
||||
while True:
|
||||
page = await session.list_tools(cursor=cursor)
|
||||
tools.extend(_serialize_tool(tool) for tool in page.tools)
|
||||
cursor = getattr(page, "nextCursor", None)
|
||||
if not cursor:
|
||||
break
|
||||
if cursor in seen_cursors:
|
||||
raise RuntimeError("MCP server repeated a tools/list cursor")
|
||||
seen_cursors.add(cursor)
|
||||
return tools
|
||||
|
||||
tools = _run_operation(cfg, list_all_tools, "discover tools")
|
||||
self._schemas[server_name] = {tool["name"]: tool for tool in tools}
|
||||
self._schema_fingerprints[server_name] = fingerprint
|
||||
return tools
|
||||
|
||||
def get_tool_schema(self, server_name: str, tool_name: str) -> Dict[str, Any]:
|
||||
if server_name not in self._schemas:
|
||||
self.discover(server_name)
|
||||
server_schemas = self._schemas.get(server_name, {})
|
||||
if tool_name not in server_schemas:
|
||||
raise KeyError(
|
||||
f"Tool '{tool_name}' not found on MCP server '{server_name}'. "
|
||||
f"Available: {list(server_schemas.keys())}"
|
||||
)
|
||||
return server_schemas[tool_name]
|
||||
|
||||
def call_tool(
|
||||
self,
|
||||
server_name: str,
|
||||
tool_name: str,
|
||||
arguments: Dict[str, Any],
|
||||
result_mode: str = "auto",
|
||||
) -> Any:
|
||||
if result_mode not in RESULT_MODES:
|
||||
raise ValueError(f"result_mode must be one of {RESULT_MODES}")
|
||||
if not isinstance(arguments, dict):
|
||||
raise ValueError("MCP tool arguments must be a mapping")
|
||||
|
||||
_require_mcp_sdk()
|
||||
cfg = self._get_approved_config(server_name)
|
||||
|
||||
async def call(session: Any) -> Any:
|
||||
result = await session.call_tool(
|
||||
tool_name,
|
||||
arguments,
|
||||
read_timeout_seconds=timedelta(seconds=cfg.timeout),
|
||||
)
|
||||
return _normalize_result(result, result_mode)
|
||||
|
||||
return _run_operation(cfg, call, f"call {tool_name}")
|
||||
|
||||
|
||||
def _run_operation(
|
||||
cfg: McpServerConfig,
|
||||
operation: Callable[[Any], Awaitable[T]],
|
||||
operation_name: str,
|
||||
) -> T:
|
||||
if cfg.transport == "stdio":
|
||||
return _run_stdio_operation(cfg, operation, operation_name)
|
||||
return _run_http_operation(cfg, operation, operation_name)
|
||||
|
||||
|
||||
def _run_http_operation(
|
||||
cfg: McpServerConfig,
|
||||
operation: Callable[[Any], Awaitable[T]],
|
||||
operation_name: str,
|
||||
) -> T:
|
||||
from mcp import ClientSession
|
||||
from mcp.client.streamable_http import streamable_http_client
|
||||
import httpx
|
||||
|
||||
resolved_headers = _build_http_headers(cfg)
|
||||
|
||||
async def run() -> T:
|
||||
timeout = httpx.Timeout(cfg.timeout)
|
||||
async with httpx.AsyncClient(
|
||||
headers=resolved_headers, timeout=timeout, follow_redirects=False
|
||||
) as client:
|
||||
async with streamable_http_client(
|
||||
cfg.url, http_client=client
|
||||
) as (read, write, _):
|
||||
async with ClientSession(
|
||||
read, write,
|
||||
read_timeout_seconds=timedelta(seconds=cfg.timeout),
|
||||
) as session:
|
||||
await session.initialize()
|
||||
return await operation(session)
|
||||
|
||||
try:
|
||||
return asyncio.run(asyncio.wait_for(run(), timeout=cfg.timeout))
|
||||
except (asyncio.TimeoutError, TimeoutError) as exc:
|
||||
raise TimeoutError(
|
||||
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s"
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
secrets = list(resolved_headers.values())
|
||||
raise RuntimeError(
|
||||
f"MCP {operation_name} on '{cfg.name}' failed: "
|
||||
f"{_sanitize(str(exc), secrets)}"
|
||||
) from exc
|
||||
|
||||
|
||||
def _run_stdio_operation(
|
||||
cfg: McpServerConfig,
|
||||
operation: Callable[[Any], Awaitable[T]],
|
||||
operation_name: str,
|
||||
) -> T:
|
||||
from mcp import ClientSession, StdioServerParameters
|
||||
from mcp.client.stdio import stdio_client
|
||||
|
||||
resolved_env = _build_server_env(cfg)
|
||||
server_params = StdioServerParameters(
|
||||
command=cfg.command,
|
||||
args=cfg.args,
|
||||
cwd=cfg.cwd,
|
||||
env=resolved_env,
|
||||
)
|
||||
|
||||
async def run(stderr_file: Any) -> T:
|
||||
async with stdio_client(server_params, errlog=stderr_file) as (read, write):
|
||||
async with ClientSession(
|
||||
read,
|
||||
write,
|
||||
read_timeout_seconds=timedelta(seconds=cfg.timeout),
|
||||
) as session:
|
||||
await session.initialize()
|
||||
return await operation(session)
|
||||
|
||||
# The SDK passes errlog directly to subprocess.Popen, so it must expose a
|
||||
# real file descriptor (StringIO is insufficient).
|
||||
with tempfile.TemporaryFile(mode="w+", encoding="utf-8") as stderr_file:
|
||||
started = time.monotonic()
|
||||
try:
|
||||
return asyncio.run(asyncio.wait_for(run(stderr_file), timeout=cfg.timeout))
|
||||
except (asyncio.TimeoutError, TimeoutError) as exc:
|
||||
stderr_file.seek(0)
|
||||
detail = _sanitize(
|
||||
stderr_file.read().strip(),
|
||||
[resolved_env[name] for name in cfg.env if name in resolved_env],
|
||||
)
|
||||
suffix = f" (server stderr: {detail[:500]})" if detail else ""
|
||||
raise TimeoutError(
|
||||
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s{suffix}"
|
||||
) from exc
|
||||
except Exception as exc:
|
||||
# Some SDK/AnyIO cleanup paths replace asyncio's cancellation with
|
||||
# an ExceptionGroup (for example BrokenResourceError). The elapsed
|
||||
# deadline remains authoritative: the requested operation timed out.
|
||||
if time.monotonic() - started >= cfg.timeout:
|
||||
raise TimeoutError(
|
||||
f"MCP {operation_name} on '{cfg.name}' timed out after {cfg.timeout}s"
|
||||
) from exc
|
||||
# Only configured values are secrets. Redacting inherited PATH/HOME
|
||||
# values would make ordinary diagnostics unreadable.
|
||||
secrets = [resolved_env[name] for name in cfg.env if name in resolved_env]
|
||||
stderr_file.seek(0)
|
||||
detail = _sanitize(stderr_file.read().strip(), secrets)
|
||||
message = _sanitize(str(exc), secrets)
|
||||
if detail:
|
||||
message = f"{message} (server stderr: {detail[:500]})"
|
||||
raise RuntimeError(
|
||||
f"MCP {operation_name} on '{cfg.name}' failed: {message}"
|
||||
) from exc
|
||||
|
||||
|
||||
def _normalize_result(result: Any, mode: str) -> Any:
|
||||
if mode not in RESULT_MODES:
|
||||
raise ValueError(f"result_mode must be one of {RESULT_MODES}")
|
||||
|
||||
content = getattr(result, "content", []) or []
|
||||
if getattr(result, "isError", False):
|
||||
error_content = [_serialize_content_block(block) for block in content]
|
||||
raise RuntimeError(f"MCP tool returned an error: {error_content}")
|
||||
|
||||
structured = getattr(result, "structuredContent", None)
|
||||
if mode in ("auto", "structured") and structured is not None:
|
||||
return structured
|
||||
if mode == "structured":
|
||||
raise ValueError("MCP tool did not return structuredContent")
|
||||
|
||||
text_parts = [
|
||||
block.text
|
||||
for block in content
|
||||
if getattr(block, "type", None) == "text" and hasattr(block, "text")
|
||||
]
|
||||
if mode == "text":
|
||||
return "\n".join(text_parts)
|
||||
|
||||
serialized = [_serialize_content_block(block) for block in content]
|
||||
if mode == "content":
|
||||
return serialized
|
||||
if content and len(text_parts) == len(content):
|
||||
return "\n".join(text_parts)
|
||||
return serialized
|
||||
|
||||
|
||||
def _serialize_content_block(block: Any) -> Dict[str, Any]:
|
||||
if hasattr(block, "model_dump"):
|
||||
return block.model_dump(mode="json", by_alias=True, exclude_none=True)
|
||||
if isinstance(block, dict):
|
||||
return block
|
||||
result = {}
|
||||
for name in ("type", "text", "mimeType", "data", "uri", "resource"):
|
||||
if hasattr(block, name):
|
||||
result[name] = getattr(block, name)
|
||||
return result
|
||||
@@ -0,0 +1,395 @@
|
||||
"""MCP server support: expose CmdForge tools as MCP tools."""
|
||||
|
||||
import inspect
|
||||
import hmac
|
||||
import os
|
||||
import sys
|
||||
from typing import Annotated, Any, Literal, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from .mcp_client import (
|
||||
MCP_DEPTH_ENV,
|
||||
McpServeConfig,
|
||||
_MCP_CALL_DEPTH,
|
||||
_expand_env_value,
|
||||
_is_loopback_host,
|
||||
_require_mcp_sdk,
|
||||
load_mcp_serve_config,
|
||||
)
|
||||
from .runner import MAX_TOOL_DEPTH
|
||||
from .tool import ToolArgument
|
||||
|
||||
|
||||
def _cmdforge_tool_name(name: str) -> str:
|
||||
"""Map a CmdForge qualified tool name to an MCP-safe name.
|
||||
|
||||
MCP tool names must be simple identifiers. We map owner/name to owner__name.
|
||||
"""
|
||||
return name.replace("/", "__")
|
||||
|
||||
|
||||
def _argument_to_json_schema(arg: ToolArgument) -> dict:
|
||||
"""Map a CmdForge ToolArgument to a JSON Schema property."""
|
||||
prop: dict = {}
|
||||
arg_type = arg.type
|
||||
|
||||
type_map = {
|
||||
"string": "string",
|
||||
"integer": "integer",
|
||||
"number": "number",
|
||||
"boolean": "boolean",
|
||||
}
|
||||
prop["type"] = type_map.get(arg_type, "string")
|
||||
|
||||
if arg.description:
|
||||
prop["description"] = arg.description
|
||||
|
||||
if arg.default is not None:
|
||||
prop["default"] = _coerce_argument_value(arg.default, arg.type)
|
||||
|
||||
if arg.enum:
|
||||
prop["enum"] = [
|
||||
_coerce_argument_value(value, arg.type) for value in arg.enum
|
||||
]
|
||||
|
||||
return prop
|
||||
|
||||
|
||||
def _build_tool_schema(tool) -> dict:
|
||||
"""Build the MCP inputSchema for a CmdForge tool.
|
||||
|
||||
Returns a dict with name, description, and inputSchema suitable for
|
||||
registering with FastMCP.
|
||||
"""
|
||||
properties = {}
|
||||
required = []
|
||||
|
||||
for arg in tool.arguments:
|
||||
prop = _argument_to_json_schema(arg)
|
||||
properties[arg.variable] = prop
|
||||
if arg.required:
|
||||
required.append(arg.variable)
|
||||
|
||||
if "input" not in properties:
|
||||
properties["input"] = {
|
||||
"type": "string",
|
||||
"description": "Text passed to the CmdForge tool as standard input.",
|
||||
"default": "",
|
||||
}
|
||||
|
||||
input_schema = {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
}
|
||||
if required:
|
||||
input_schema["required"] = required
|
||||
|
||||
return {
|
||||
"name": _cmdforge_tool_name(tool.name),
|
||||
"description": tool.description or f"CmdForge tool: {tool.name}",
|
||||
"inputSchema": input_schema,
|
||||
}
|
||||
|
||||
|
||||
class _StaticTokenVerifier:
|
||||
def __init__(self, token: str):
|
||||
self._token = token
|
||||
|
||||
async def verify_token(self, token: str):
|
||||
if not hmac.compare_digest(token, self._token):
|
||||
return None
|
||||
from mcp.server.auth.provider import AccessToken
|
||||
return AccessToken(
|
||||
token=token, client_id="cmdforge-mcp-client", scopes=[],
|
||||
subject="cmdforge-mcp",
|
||||
)
|
||||
|
||||
|
||||
def _validate_http_server_options(
|
||||
host: str,
|
||||
port: int,
|
||||
allowed_origins: list[str],
|
||||
auth_token: Optional[str],
|
||||
external_url: Optional[str],
|
||||
) -> tuple[list[str], Optional[str]]:
|
||||
if isinstance(port, bool) or not isinstance(port, int) or not 1 <= port <= 65535:
|
||||
raise ValueError("MCP server port must be between 1 and 65535")
|
||||
local = _is_loopback_host(host)
|
||||
resolved_token = _expand_env_value(auth_token) if auth_token else None
|
||||
if not local:
|
||||
parsed = urlparse(external_url or "")
|
||||
if parsed.scheme != "https" or not parsed.hostname:
|
||||
raise ValueError(
|
||||
"Non-local MCP serving requires --external-url https://... "
|
||||
"behind a TLS-terminating proxy"
|
||||
)
|
||||
if not resolved_token:
|
||||
raise ValueError("Non-local MCP serving requires --auth-token")
|
||||
origins = allowed_origins or (
|
||||
[f"http://localhost:{port}", f"http://127.0.0.1:{port}"]
|
||||
if local else [f"{urlparse(external_url).scheme}://{urlparse(external_url).netloc}"]
|
||||
)
|
||||
for origin in origins:
|
||||
parsed = urlparse(origin)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.netloc or parsed.path not in ("", "/"):
|
||||
raise ValueError(f"Invalid allowed origin: {origin}")
|
||||
if not local and parsed.scheme != "https":
|
||||
raise ValueError("Non-local MCP origins must use HTTPS")
|
||||
return origins, resolved_token
|
||||
|
||||
|
||||
def serve(
|
||||
transport: str = "stdio",
|
||||
*,
|
||||
host: str = "127.0.0.1",
|
||||
port: int = 8000,
|
||||
allowed_origins: Optional[list[str]] = None,
|
||||
auth_token: Optional[str] = None,
|
||||
external_url: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Start CmdForge as an MCP server on the given transport.
|
||||
|
||||
HTTP defaults to loopback and enables SDK DNS-rebinding/origin protection.
|
||||
Non-local binding requires bearer auth and an explicit public HTTPS URL;
|
||||
TLS is expected to terminate at the reverse proxy represented by that URL.
|
||||
"""
|
||||
if transport not in ("stdio", "streamable-http"):
|
||||
raise ValueError(f"Unsupported MCP server transport '{transport}'")
|
||||
|
||||
_require_mcp_sdk()
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
|
||||
serve_config = load_mcp_serve_config()
|
||||
|
||||
from .tool import list_tools, load_tool
|
||||
server_options = {}
|
||||
if transport == "streamable-http":
|
||||
origins, resolved_token = _validate_http_server_options(
|
||||
host, port, allowed_origins or [], auth_token, external_url
|
||||
)
|
||||
from mcp.server.transport_security import TransportSecuritySettings
|
||||
allowed_hosts = [host, f"{host}:{port}", "localhost", f"localhost:{port}"]
|
||||
if external_url:
|
||||
external_host = urlparse(external_url).netloc
|
||||
allowed_hosts.extend([external_host, urlparse(external_url).hostname or ""])
|
||||
server_options.update({
|
||||
"host": host,
|
||||
"port": port,
|
||||
"transport_security": TransportSecuritySettings(
|
||||
enable_dns_rebinding_protection=True,
|
||||
allowed_hosts=[value for value in allowed_hosts if value],
|
||||
allowed_origins=origins,
|
||||
),
|
||||
})
|
||||
if resolved_token:
|
||||
from mcp.server.auth.settings import AuthSettings
|
||||
issuer = external_url or f"http://{host}:{port}"
|
||||
server_options.update({
|
||||
"token_verifier": _StaticTokenVerifier(resolved_token),
|
||||
"auth": AuthSettings(
|
||||
issuer_url=issuer,
|
||||
resource_server_url=f"{issuer.rstrip('/')}/mcp",
|
||||
required_scopes=[],
|
||||
),
|
||||
})
|
||||
server = FastMCP("CmdForge", **server_options)
|
||||
|
||||
tool_names = list_tools()
|
||||
exposed_count = 0
|
||||
exposed_names = {}
|
||||
|
||||
for name in sorted(tool_names):
|
||||
# Skip private/unlisted tools unless explicitly exposed
|
||||
tool_obj = load_tool(name)
|
||||
if not tool_obj:
|
||||
continue
|
||||
|
||||
if not _is_exposable(tool_obj, serve_config, qualified_name=name):
|
||||
continue
|
||||
|
||||
mcp_name = _cmdforge_tool_name(name)
|
||||
previous = exposed_names.get(mcp_name)
|
||||
if previous is not None:
|
||||
raise RuntimeError(
|
||||
f"MCP tool name collision: '{previous}' and '{name}' both map to "
|
||||
f"'{mcp_name}'"
|
||||
)
|
||||
exposed_names[mcp_name] = name
|
||||
schema = _build_tool_schema(tool_obj)
|
||||
|
||||
# Capture tool reference for the handler
|
||||
_register_tool(server, name, mcp_name, schema, tool_obj)
|
||||
exposed_count += 1
|
||||
|
||||
print(f"[mcp] CmdForge MCP server starting on {transport}", file=sys.stderr)
|
||||
print(f"[mcp] Exposing {exposed_count} tool(s)", file=sys.stderr)
|
||||
|
||||
server.run(transport=transport)
|
||||
|
||||
|
||||
def _is_exposable(
|
||||
tool, config: McpServeConfig, qualified_name: str | None = None
|
||||
) -> bool:
|
||||
"""Check whether a tool should be exposed per the server policy.
|
||||
|
||||
Private/unlisted tools must be explicitly listed in expose. Public tools
|
||||
are exposed only if the expose list includes them or '*'.
|
||||
"""
|
||||
policy_name = qualified_name or tool.name
|
||||
if config._matches_deny(policy_name):
|
||||
return False
|
||||
if tool.visibility not in ("public", None, ""):
|
||||
# private or unlisted — must be explicitly named
|
||||
return policy_name in config.expose
|
||||
return config.is_exposed(policy_name)
|
||||
|
||||
|
||||
def _register_tool(server, cmdforge_name: str, mcp_name: str, schema: dict, tool_obj) -> None:
|
||||
"""Register a CmdForge tool handler with the FastMCP server."""
|
||||
|
||||
tool_desc = schema.get("description", "")
|
||||
argument_names = {arg.variable for arg in tool_obj.arguments}
|
||||
|
||||
def handler(**kwargs: Any) -> str:
|
||||
from .runner import run_tool
|
||||
|
||||
inherited_depth = _read_mcp_depth()
|
||||
if inherited_depth >= MAX_TOOL_DEPTH:
|
||||
raise RuntimeError(
|
||||
f"Maximum MCP nesting depth ({MAX_TOOL_DEPTH}) exceeded"
|
||||
)
|
||||
if "input" in argument_names:
|
||||
input_text = kwargs.get("input", "") or ""
|
||||
else:
|
||||
input_text = kwargs.pop("input", "") or ""
|
||||
token = _MCP_CALL_DEPTH.set(inherited_depth + 1)
|
||||
try:
|
||||
output, exit_code = run_tool(
|
||||
tool_obj,
|
||||
input_text=input_text,
|
||||
custom_args=kwargs,
|
||||
provider_override=None,
|
||||
verbose=False,
|
||||
dry_run=False,
|
||||
_depth=inherited_depth,
|
||||
)
|
||||
finally:
|
||||
_MCP_CALL_DEPTH.reset(token)
|
||||
if exit_code != 0:
|
||||
raise RuntimeError(f"Tool '{cmdforge_name}' exited with code {exit_code}")
|
||||
return output
|
||||
|
||||
handler.__name__ = f"cmdforge_{mcp_name}"
|
||||
handler.__signature__ = _build_handler_signature(tool_obj)
|
||||
server.add_tool(
|
||||
handler,
|
||||
name=mcp_name,
|
||||
description=tool_desc,
|
||||
structured_output=False,
|
||||
)
|
||||
|
||||
|
||||
def _build_handler_signature(tool) -> inspect.Signature:
|
||||
"""Create the flat typed signature FastMCP uses for schema and validation."""
|
||||
required_parameters = []
|
||||
optional_parameters = []
|
||||
seen = set()
|
||||
|
||||
for arg in tool.arguments:
|
||||
if not arg.variable.isidentifier():
|
||||
raise ValueError(
|
||||
f"Tool '{tool.name}' has invalid argument variable '{arg.variable}'"
|
||||
)
|
||||
if arg.variable in seen:
|
||||
raise ValueError(
|
||||
f"Tool '{tool.name}' defines argument '{arg.variable}' more than once"
|
||||
)
|
||||
seen.add(arg.variable)
|
||||
annotation = _argument_annotation(arg)
|
||||
default = inspect.Parameter.empty
|
||||
if not arg.required:
|
||||
default = (
|
||||
_coerce_argument_value(arg.default, arg.type)
|
||||
if arg.default is not None
|
||||
else None
|
||||
)
|
||||
parameter = inspect.Parameter(
|
||||
arg.variable,
|
||||
kind=inspect.Parameter.KEYWORD_ONLY,
|
||||
default=default,
|
||||
annotation=annotation,
|
||||
)
|
||||
(required_parameters if arg.required else optional_parameters).append(parameter)
|
||||
|
||||
if "input" not in seen:
|
||||
optional_parameters.append(
|
||||
inspect.Parameter(
|
||||
"input",
|
||||
kind=inspect.Parameter.KEYWORD_ONLY,
|
||||
default="",
|
||||
annotation=Annotated[
|
||||
str,
|
||||
_pydantic_field(
|
||||
"Text passed to the CmdForge tool as standard input."
|
||||
),
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
return inspect.Signature(
|
||||
required_parameters + optional_parameters,
|
||||
return_annotation=str,
|
||||
)
|
||||
|
||||
|
||||
def _argument_annotation(arg: ToolArgument):
|
||||
python_type = {
|
||||
"string": str,
|
||||
"integer": int,
|
||||
"number": float,
|
||||
"boolean": bool,
|
||||
}[arg.type]
|
||||
if arg.enum:
|
||||
values = tuple(_coerce_argument_value(value, arg.type) for value in arg.enum)
|
||||
python_type = Literal[values]
|
||||
if arg.description:
|
||||
return Annotated[python_type, _pydantic_field(arg.description)]
|
||||
return python_type
|
||||
|
||||
|
||||
def _pydantic_field(description: str):
|
||||
from pydantic import Field
|
||||
|
||||
return Field(description=description)
|
||||
|
||||
|
||||
def _coerce_argument_value(value: Any, arg_type: str) -> Any:
|
||||
if value is None:
|
||||
return None
|
||||
if arg_type == "string":
|
||||
return str(value)
|
||||
if arg_type == "integer":
|
||||
return int(value)
|
||||
if arg_type == "number":
|
||||
return float(value)
|
||||
if arg_type == "boolean":
|
||||
if isinstance(value, str):
|
||||
normalized = value.strip().lower()
|
||||
if normalized in ("true", "1", "yes", "on"):
|
||||
return True
|
||||
if normalized in ("false", "0", "no", "off"):
|
||||
return False
|
||||
raise ValueError(f"Invalid boolean value: {value}")
|
||||
return bool(value)
|
||||
raise ValueError(f"Unsupported argument type: {arg_type}")
|
||||
|
||||
|
||||
def _read_mcp_depth() -> int:
|
||||
raw_depth = os.environ.get(MCP_DEPTH_ENV, "0")
|
||||
try:
|
||||
depth = int(raw_depth)
|
||||
except ValueError as exc:
|
||||
raise RuntimeError(f"Invalid {MCP_DEPTH_ENV} value: {raw_depth}") from exc
|
||||
if depth < 0:
|
||||
raise RuntimeError(f"Invalid {MCP_DEPTH_ENV} value: {raw_depth}")
|
||||
return depth
|
||||
@@ -0,0 +1,626 @@
|
||||
"""Shared preflight analysis engine.
|
||||
|
||||
Produces a PreflightReport used by the GUI, CLI, and registry workflow.
|
||||
All checks are deterministic and evidence-based.
|
||||
"""
|
||||
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from .tool import Tool, validate_json_schema
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreflightReport:
|
||||
"""Structured analysis of a tool before creation, save, or publish."""
|
||||
|
||||
errors: List[str] = field(default_factory=list)
|
||||
warnings: List[str] = field(default_factory=list)
|
||||
suggestions: List[str] = field(default_factory=list)
|
||||
similar_tools: List[Dict[str, Any]] = field(default_factory=list)
|
||||
generated_tests: List[Dict[str, Any]] = field(default_factory=list)
|
||||
compatibility: List[Dict[str, Any]] = field(default_factory=list)
|
||||
contract_proposal: Optional[Dict[str, Any]] = None
|
||||
regression: Optional[Dict[str, Any]] = None
|
||||
reuse_opportunities: List[Dict[str, Any]] = field(default_factory=list)
|
||||
audit_evidence: Optional[Dict[str, Any]] = None
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return not self.errors
|
||||
|
||||
def merge(self, other: "PreflightReport") -> "PreflightReport":
|
||||
return PreflightReport(
|
||||
errors=self.errors + other.errors,
|
||||
warnings=self.warnings + other.warnings,
|
||||
suggestions=self.suggestions + other.suggestions,
|
||||
similar_tools=self.similar_tools + other.similar_tools,
|
||||
generated_tests=self.generated_tests + other.generated_tests,
|
||||
compatibility=self.compatibility + other.compatibility,
|
||||
contract_proposal=self.contract_proposal or other.contract_proposal,
|
||||
regression=self.regression or other.regression,
|
||||
reuse_opportunities=self.reuse_opportunities + other.reuse_opportunities,
|
||||
audit_evidence=self.audit_evidence or other.audit_evidence,
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"errors": list(self.errors),
|
||||
"warnings": list(self.warnings),
|
||||
"suggestions": list(self.suggestions),
|
||||
"similar_tools": list(self.similar_tools),
|
||||
"generated_tests": list(self.generated_tests),
|
||||
"compatibility": list(self.compatibility),
|
||||
"contract_proposal": copy.deepcopy(self.contract_proposal),
|
||||
"regression": copy.deepcopy(self.regression),
|
||||
"reuse_opportunities": list(self.reuse_opportunities),
|
||||
"audit_evidence": copy.deepcopy(self.audit_evidence) if self.audit_evidence else None,
|
||||
}
|
||||
|
||||
|
||||
def analyze_tool(
|
||||
tool: Tool,
|
||||
registry_client=None,
|
||||
*,
|
||||
check_local_dependencies: bool = True,
|
||||
include_contract_tests: bool = False,
|
||||
check_reuse: Optional[bool] = None,
|
||||
) -> PreflightReport:
|
||||
"""Run all preflight checks on a tool.
|
||||
|
||||
Args:
|
||||
tool: The Tool to analyze.
|
||||
registry_client: Optional registry client for registry-side checks.
|
||||
|
||||
Returns:
|
||||
A PreflightReport with structured findings.
|
||||
"""
|
||||
report = PreflightReport()
|
||||
checks_run = ["config_integrity", "contracts"]
|
||||
_check_config_integrity(tool, report)
|
||||
_check_contracts(tool, report)
|
||||
_add_contract_guidance(tool, report, include_contract_tests)
|
||||
if include_contract_tests:
|
||||
checks_run.append("contract_conformance")
|
||||
_check_secrets(tool, report)
|
||||
checks_run.append("secrets")
|
||||
if check_local_dependencies:
|
||||
_check_toolstep_compatibility(tool, report)
|
||||
_check_dependencies(tool, report)
|
||||
checks_run.extend(["toolstep_compatibility", "dependencies"])
|
||||
if registry_client:
|
||||
_check_similar_tools(tool, report, registry_client)
|
||||
checks_run.append("registry_similarity")
|
||||
if check_reuse is None:
|
||||
check_reuse = check_local_dependencies
|
||||
if check_reuse:
|
||||
_check_reuse_opportunities(tool, report)
|
||||
checks_run.append("reuse_opportunities")
|
||||
_add_audit_evidence(tool, report, checks_run)
|
||||
return report
|
||||
|
||||
|
||||
def _add_contract_guidance(
|
||||
tool: Tool, report: PreflightReport, include_contract_tests: bool
|
||||
) -> None:
|
||||
proposal = infer_contracts(tool)
|
||||
report.contract_proposal = proposal
|
||||
inferred = [
|
||||
label
|
||||
for label in ("input", "output")
|
||||
if proposal[f"{label}_source"] == "inferred"
|
||||
]
|
||||
if inferred:
|
||||
report.suggestions.append(
|
||||
"Review inferred " + " and ".join(inferred) + " contract proposal"
|
||||
)
|
||||
|
||||
if not include_contract_tests:
|
||||
return
|
||||
if tool.input_schema is None or tool.output_schema is None:
|
||||
return
|
||||
|
||||
from .contract_testing import (
|
||||
BaselineFormatError,
|
||||
compare_conformance,
|
||||
load_conformance_baseline,
|
||||
run_contract_tests,
|
||||
)
|
||||
|
||||
baseline = None
|
||||
if tool.path:
|
||||
try:
|
||||
baseline = load_conformance_baseline(tool.path.parent)
|
||||
except BaselineFormatError as exc:
|
||||
report.warnings.append(f"Regression baseline is invalid: {exc}")
|
||||
baseline_inputs = (
|
||||
[result["input_value"] for result in baseline["results"]]
|
||||
if baseline is not None else None
|
||||
)
|
||||
conformance = run_contract_tests(tool, test_inputs=baseline_inputs)
|
||||
report.generated_tests = [result.to_dict() for result in conformance.results]
|
||||
states = {result.state for result in conformance.results}
|
||||
if "failed" in states:
|
||||
details = "; ".join(
|
||||
result.detail for result in conformance.results if result.state == "failed"
|
||||
)
|
||||
report.errors.append(f"Contract conformance failed: {details}")
|
||||
if "unsupported" in states:
|
||||
details = "; ".join(
|
||||
result.detail
|
||||
for result in conformance.results
|
||||
if result.state == "unsupported"
|
||||
)
|
||||
report.warnings.append(f"Contract conformance unsupported: {details}")
|
||||
|
||||
if baseline is not None:
|
||||
try:
|
||||
delta = compare_conformance(baseline, conformance, tool)
|
||||
except BaselineFormatError as exc:
|
||||
report.warnings.append(f"Regression baseline is invalid: {exc}")
|
||||
else:
|
||||
report.regression = delta.to_dict()
|
||||
if delta.has_regressions:
|
||||
report.warnings.append(f"Regression comparison: {delta.summary}")
|
||||
elif delta.contract_changed:
|
||||
report.suggestions.append(
|
||||
"Tool contracts changed since the saved conformance baseline"
|
||||
)
|
||||
|
||||
|
||||
def _check_config_integrity(tool: Tool, report: PreflightReport):
|
||||
if not tool.name:
|
||||
report.errors.append("Tool name is required")
|
||||
if tool.version and not _is_semver(tool.version):
|
||||
report.errors.append(f"Version '{tool.version}' is not valid semver")
|
||||
if not tool.steps and not tool.arguments and tool.output == "{input}":
|
||||
report.warnings.append(
|
||||
"Tool has no steps and no arguments — output template is passthrough"
|
||||
)
|
||||
|
||||
|
||||
def _check_contracts(tool: Tool, report: PreflightReport):
|
||||
if tool.output_schema is not None:
|
||||
_validate_schema(tool.output_schema, "output_schema", report)
|
||||
if tool.input_schema is not None:
|
||||
_validate_schema(tool.input_schema, "input_schema", report)
|
||||
if tool.output_schema is None and tool.input_schema is None:
|
||||
report.suggestions.append(
|
||||
"Add input_schema and output_schema to enable automated verification"
|
||||
)
|
||||
|
||||
|
||||
def _validate_schema(schema: Any, label: str, report: PreflightReport):
|
||||
try:
|
||||
validate_json_schema(schema, label)
|
||||
except ValueError as exc:
|
||||
report.errors.append(str(exc))
|
||||
|
||||
|
||||
def _check_secrets(tool: Tool, report: PreflightReport):
|
||||
secret_patterns = ["api_key", "api_secret", "password", "token", "secret"]
|
||||
for step in tool.steps:
|
||||
if hasattr(step, "prompt"):
|
||||
prompt_lower = step.prompt.lower()
|
||||
for pat in secret_patterns:
|
||||
# Do not flag schema/field names such as max_tokens or
|
||||
# token_count. Underscores are identifier characters here,
|
||||
# even though regex \b would treat them inconsistently around
|
||||
# compound names.
|
||||
if re.search(
|
||||
rf"(?<![a-z0-9_]){re.escape(pat)}(?![a-z0-9_])",
|
||||
prompt_lower,
|
||||
):
|
||||
report.warnings.append(
|
||||
f"Prompt step contains potential secret pattern '{pat}'"
|
||||
)
|
||||
break
|
||||
|
||||
|
||||
def _check_toolstep_compatibility(tool: Tool, report: PreflightReport):
|
||||
from .schema_compat import compare_json_schemas
|
||||
from .tool import ToolStep, load_tool
|
||||
|
||||
for index, step in enumerate(tool.steps):
|
||||
if not isinstance(step, ToolStep):
|
||||
continue
|
||||
nested = load_tool(step.tool)
|
||||
if not nested:
|
||||
report.compatibility.append({
|
||||
"tool": step.tool,
|
||||
"step": index + 1,
|
||||
"state": "unresolved",
|
||||
"detail": f"Tool '{step.tool}' not found locally",
|
||||
"issues": ["Called tool is not installed locally"],
|
||||
})
|
||||
report.suggestions.append(
|
||||
f"Could not check ToolStep {index + 1} -> '{step.tool}': "
|
||||
"tool is not installed locally"
|
||||
)
|
||||
continue
|
||||
if nested.input_schema is None:
|
||||
finding = {
|
||||
"tool": step.tool,
|
||||
"step": index + 1,
|
||||
"state": "unknown",
|
||||
"detail": f"Tool '{step.tool}' has no input_schema",
|
||||
"issues": ["Downstream input contract is missing"],
|
||||
}
|
||||
report.compatibility.append(finding)
|
||||
report.suggestions.append(finding["detail"])
|
||||
continue
|
||||
|
||||
supplied_schema, source_schema = _toolstep_supplied_schema(
|
||||
tool, step, index, nested
|
||||
)
|
||||
result = compare_json_schemas(supplied_schema, nested.input_schema)
|
||||
finding = {
|
||||
"tool": step.tool,
|
||||
"step": index + 1,
|
||||
"state": result.state,
|
||||
"detail": result.detail,
|
||||
"issues": result.issues,
|
||||
"upstream_schema": source_schema,
|
||||
"supplied_schema": supplied_schema,
|
||||
"downstream_schema": copy.deepcopy(nested.input_schema),
|
||||
}
|
||||
report.compatibility.append(finding)
|
||||
if result.state == "incompatible":
|
||||
report.warnings.append(
|
||||
f"ToolStep {index + 1} -> '{step.tool}' is incompatible: "
|
||||
f"{result.detail}"
|
||||
)
|
||||
elif result.state == "unknown":
|
||||
report.suggestions.append(
|
||||
f"Could not prove ToolStep {index + 1} -> '{step.tool}' "
|
||||
f"compatibility: {result.detail}"
|
||||
)
|
||||
|
||||
|
||||
def _toolstep_supplied_schema(tool: Tool, step, index: int, nested: Tool):
|
||||
"""Describe values the current runner actually supplies to a nested tool."""
|
||||
from .schema_compat import schema_for_value
|
||||
|
||||
input_schema = (
|
||||
schema_for_value(step.input_template)
|
||||
if "{" not in step.input_template
|
||||
else {"type": "string"}
|
||||
)
|
||||
properties: Dict[str, Any] = {"input": input_schema}
|
||||
required = ["input"]
|
||||
source_schema = _template_source_schema(tool, step.input_template, index)
|
||||
|
||||
for name, value in step.args.items():
|
||||
# execute_tool_step substitutes every explicit argument through str().
|
||||
if isinstance(value, str) and "{" not in value:
|
||||
properties[name] = schema_for_value(value)
|
||||
else:
|
||||
properties[name] = {"type": "string"}
|
||||
required.append(name)
|
||||
|
||||
for argument in nested.arguments:
|
||||
if argument.variable in properties or argument.default is None:
|
||||
continue
|
||||
properties[argument.variable] = schema_for_value(argument.default)
|
||||
required.append(argument.variable)
|
||||
|
||||
supplied = {
|
||||
"type": "object",
|
||||
"properties": properties,
|
||||
"required": required,
|
||||
"additionalProperties": False,
|
||||
}
|
||||
if _schema_type_name(nested.input_schema) != "object":
|
||||
# Scalar contracts describe stdin directly when no argument object is used.
|
||||
supplied = properties["input"]
|
||||
return supplied, source_schema
|
||||
|
||||
|
||||
def _template_source_schema(tool: Tool, template: str, before_index: int):
|
||||
match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", template.strip())
|
||||
if not match:
|
||||
return {"type": "string"}
|
||||
variable = match.group(1)
|
||||
if tool.input_schema is not None:
|
||||
if _schema_type_name(tool.input_schema) == "object":
|
||||
schema = tool.input_schema.get("properties", {}).get(variable)
|
||||
if schema is not None:
|
||||
return copy.deepcopy(schema)
|
||||
elif variable == "input":
|
||||
return copy.deepcopy(tool.input_schema)
|
||||
|
||||
from .tool import PromptStep, ToolStep, load_tool
|
||||
for previous in reversed(tool.steps[:before_index]):
|
||||
if getattr(previous, "output_var", None) != variable:
|
||||
continue
|
||||
if isinstance(previous, PromptStep):
|
||||
return copy.deepcopy(previous.output_schema)
|
||||
if isinstance(previous, ToolStep):
|
||||
nested = load_tool(previous.tool)
|
||||
return copy.deepcopy(nested.output_schema) if nested else None
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def _schema_type_name(schema: Optional[dict]) -> Optional[str]:
|
||||
if not schema:
|
||||
return None
|
||||
value = schema.get("type")
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
if "properties" in schema or "required" in schema:
|
||||
return "object"
|
||||
return None
|
||||
|
||||
|
||||
def _check_dependencies(tool: Tool, report: PreflightReport):
|
||||
from .tool import tool_exists
|
||||
for dep in tool.dependencies:
|
||||
ref = dep.get("name", dep) if isinstance(dep, dict) else dep
|
||||
if not tool_exists(ref):
|
||||
report.warnings.append(f"Dependency '{ref}' is not installed locally")
|
||||
|
||||
|
||||
def _check_similar_tools(tool: Tool, report: PreflightReport, client):
|
||||
from .registry_client import RegistryError
|
||||
|
||||
try:
|
||||
results = client.search_tools(tool.name, per_page=5)
|
||||
items = results.data if hasattr(results, "data") else results
|
||||
for item in items:
|
||||
report.similar_tools.append({
|
||||
"name": (item.get("owner") or "") + "/" + item.get("name", ""),
|
||||
"description": item.get("description", ""),
|
||||
"downloads": item.get("downloads", 0),
|
||||
})
|
||||
if not items:
|
||||
report.suggestions.append("No similar tools found on the registry")
|
||||
except RegistryError:
|
||||
report.warnings.append("Could not search registry for similar tools")
|
||||
|
||||
|
||||
def _is_semver(version: str) -> bool:
|
||||
from .semver import Version
|
||||
return Version.parse(version) is not None
|
||||
|
||||
|
||||
def _check_reuse_opportunities(tool: Tool, report: PreflightReport):
|
||||
"""Report only exact repeated sequences with contract-boundary evidence."""
|
||||
from .tool import list_tools, load_tool
|
||||
|
||||
if len(tool.steps) < 2:
|
||||
return
|
||||
|
||||
step_signatures = [_step_signature(step) for step in tool.steps]
|
||||
_find_internal_reuse(tool, step_signatures, report)
|
||||
|
||||
# Check if this step sequence appears in other local tools
|
||||
our_sequence = tuple(step_signatures)
|
||||
for name in list_tools():
|
||||
if name == tool.name:
|
||||
continue
|
||||
other = load_tool(name)
|
||||
if not other or len(other.steps) < 2:
|
||||
continue
|
||||
other_sigs = [_step_signature(s) for s in other.steps]
|
||||
other_sigs = [s for s in other_sigs if s]
|
||||
if (
|
||||
our_sequence == tuple(other_sigs)
|
||||
and _has_tool_contracts(tool)
|
||||
and _has_tool_contracts(other)
|
||||
):
|
||||
report.reuse_opportunities.append({
|
||||
"type": "duplicate_sequence",
|
||||
"tool": name,
|
||||
"length": len(our_sequence),
|
||||
"evidence": "exact normalized steps and explicit tool contracts",
|
||||
"detail": f"Exact contracted step sequence also exists in '{name}'",
|
||||
})
|
||||
|
||||
|
||||
def _step_signature(step) -> str:
|
||||
"""Build an exact normalized signature without incidental labels/outputs."""
|
||||
data = step.to_dict()
|
||||
data.pop("name", None)
|
||||
data.pop("output_var", None)
|
||||
encoded = json.dumps(data, sort_keys=True, separators=(",", ":"), default=str)
|
||||
return hashlib.sha256(encoded.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def _has_tool_contracts(tool: Tool) -> bool:
|
||||
return tool.input_schema is not None and tool.output_schema is not None
|
||||
|
||||
|
||||
def _find_internal_reuse(
|
||||
tool: Tool, signatures: List[str], report: PreflightReport
|
||||
) -> None:
|
||||
"""Find non-overlapping repeated sequences with per-step output evidence."""
|
||||
from .tool import PromptStep
|
||||
|
||||
count = len(signatures)
|
||||
for length in range(count // 2, 1, -1):
|
||||
for first in range(0, count - (2 * length) + 1):
|
||||
candidate = signatures[first:first + length]
|
||||
for second in range(first + length, count - length + 1):
|
||||
if candidate != signatures[second:second + length]:
|
||||
continue
|
||||
steps = tool.steps[first:first + length]
|
||||
if not all(
|
||||
isinstance(step, PromptStep) and step.output_schema is not None
|
||||
for step in steps
|
||||
):
|
||||
continue
|
||||
report.reuse_opportunities.append({
|
||||
"type": "repeated_sequence",
|
||||
"length": length,
|
||||
"locations": [first + 1, second + 1],
|
||||
"evidence": "exact normalized prompt steps with output schemas",
|
||||
"detail": (
|
||||
f"Steps {first + 1}-{first + length} repeat at "
|
||||
f"steps {second + 1}-{second + length}"
|
||||
),
|
||||
})
|
||||
return
|
||||
|
||||
|
||||
def _add_audit_evidence(
|
||||
tool: Tool, report: PreflightReport, checks_run: List[str]
|
||||
):
|
||||
"""Attach audit evidence metadata to the report."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
findings = {
|
||||
"errors": report.errors,
|
||||
"warnings": report.warnings,
|
||||
"suggestions": report.suggestions,
|
||||
"generated_tests": report.generated_tests,
|
||||
"regression": report.regression,
|
||||
"compatibility": report.compatibility,
|
||||
"similar_tools": report.similar_tools,
|
||||
"reuse_opportunities": report.reuse_opportunities,
|
||||
}
|
||||
findings_hash = hashlib.sha256(
|
||||
json.dumps(findings, sort_keys=True, separators=(",", ":"), default=str).encode()
|
||||
).hexdigest()
|
||||
report.audit_evidence = {
|
||||
"tool_version": tool.version or "",
|
||||
"engine_version": "1.0",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"checks_run": checks_run,
|
||||
"findings_hash": findings_hash,
|
||||
"findings": copy.deepcopy(findings),
|
||||
"outcome": "failed" if report.errors else "advisory" if report.warnings else "passed",
|
||||
}
|
||||
|
||||
|
||||
def infer_contracts(tool: Tool) -> Dict[str, Any]:
|
||||
"""Propose input/output schemas without mutating the tool.
|
||||
|
||||
Returns a dict with:
|
||||
- input_schema: proposed JSON Schema (or None)
|
||||
- output_schema: proposed JSON Schema (or None)
|
||||
- evidence: list of human-readable provenance strings
|
||||
- confidence: "high" / "medium" / "low"
|
||||
|
||||
Never overwrites an explicit schema. ``{}`` counts as explicit.
|
||||
"""
|
||||
evidence = []
|
||||
proposed_input = None
|
||||
proposed_output = None
|
||||
|
||||
if tool.input_schema is None:
|
||||
proposed_input, input_evidence = _infer_input_schema(tool)
|
||||
evidence.extend(input_evidence)
|
||||
|
||||
if tool.output_schema is None:
|
||||
proposed_output, output_evidence = _infer_output_schema(tool)
|
||||
evidence.extend(output_evidence)
|
||||
|
||||
effective_input = tool.input_schema if tool.input_schema is not None else proposed_input
|
||||
effective_output = tool.output_schema if tool.output_schema is not None else proposed_output
|
||||
confidence = _confidence(effective_input, effective_output)
|
||||
|
||||
return {
|
||||
"input_schema": proposed_input,
|
||||
"output_schema": proposed_output,
|
||||
"input_source": "explicit" if tool.input_schema is not None else (
|
||||
"inferred" if proposed_input is not None else "unavailable"
|
||||
),
|
||||
"output_source": "explicit" if tool.output_schema is not None else (
|
||||
"inferred" if proposed_output is not None else "unavailable"
|
||||
),
|
||||
"evidence": evidence,
|
||||
"confidence": confidence,
|
||||
}
|
||||
|
||||
|
||||
def _infer_input_schema(tool: Tool) -> tuple:
|
||||
props = {}
|
||||
required = []
|
||||
evidence = []
|
||||
has_stdin = _contains_input_reference(tool.output) or any(
|
||||
_contains_input_reference(vars(step)) for step in tool.steps
|
||||
)
|
||||
|
||||
if has_stdin:
|
||||
props["input"] = {"type": "string", "description": "Standard input text"}
|
||||
evidence.append("stdin detected from {input} usage")
|
||||
|
||||
for arg in tool.arguments:
|
||||
prop = {"type": arg.type or "string"}
|
||||
if arg.description:
|
||||
prop["description"] = arg.description
|
||||
if arg.default is not None:
|
||||
prop["default"] = arg.default
|
||||
if arg.enum:
|
||||
prop["enum"] = arg.enum
|
||||
props[arg.variable] = prop
|
||||
if arg.required:
|
||||
required.append(arg.variable)
|
||||
evidence.append(f"argument {arg.flag} ({arg.type or 'string'})")
|
||||
|
||||
if not props:
|
||||
return None, []
|
||||
|
||||
schema = {"type": "object", "properties": props}
|
||||
if required:
|
||||
schema["required"] = required
|
||||
return schema, evidence
|
||||
|
||||
|
||||
def _infer_output_schema(tool: Tool) -> tuple:
|
||||
evidence = []
|
||||
|
||||
output = tool.output.strip()
|
||||
if not output or output == "{input}":
|
||||
return None, evidence
|
||||
|
||||
var_match = None
|
||||
m = re.match(r"^\{([a-zA-Z_][a-zA-Z0-9_]*)\}$", output)
|
||||
if m:
|
||||
var_match = m.group(1)
|
||||
|
||||
if not var_match:
|
||||
evidence.append("output template is a composite expression — schema must be defined manually")
|
||||
return None, evidence
|
||||
|
||||
# Look for the step that produces this variable
|
||||
for step in reversed(tool.steps):
|
||||
if hasattr(step, "output_var") and step.output_var == var_match:
|
||||
schema = getattr(step, "output_schema", None)
|
||||
if schema is not None:
|
||||
evidence.append(
|
||||
f"output variable '{var_match}' has an explicit step schema — using as tool output schema"
|
||||
)
|
||||
return copy.deepcopy(schema), evidence
|
||||
elif hasattr(step, "provider"):
|
||||
evidence.append(
|
||||
f"output variable '{var_match}' comes from a provider step with no schema — define output_schema on the step"
|
||||
)
|
||||
return None, evidence
|
||||
break
|
||||
|
||||
evidence.append(f"output variable '{var_match}' source not found")
|
||||
return None, evidence
|
||||
|
||||
|
||||
def _contains_input_reference(value: Any) -> bool:
|
||||
if isinstance(value, str):
|
||||
return "{input}" in value or "{input." in value
|
||||
if isinstance(value, dict):
|
||||
return any(_contains_input_reference(item) for item in value.values())
|
||||
if isinstance(value, (list, tuple)):
|
||||
return any(_contains_input_reference(item) for item in value)
|
||||
return False
|
||||
|
||||
|
||||
def _confidence(input_schema, output_schema) -> str:
|
||||
if input_schema is not None and output_schema is not None:
|
||||
return "high"
|
||||
if input_schema is not None or output_schema is not None:
|
||||
return "medium"
|
||||
return "low"
|
||||
@@ -0,0 +1,300 @@
|
||||
"""Prompt variation generation and optimization (M9.1).
|
||||
|
||||
Generates N variations of a tool's prompt steps, runs each through M8.V1
|
||||
contract conformance tests, and surfaces the highest-scoring variation.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Callable, Dict, List, Optional
|
||||
|
||||
from .tool import Tool, PromptStep
|
||||
|
||||
|
||||
@dataclass
|
||||
class PromptVariation:
|
||||
"""One generated variation of a tool's prompt."""
|
||||
|
||||
step_index: int
|
||||
original: str
|
||||
variation: str
|
||||
strategy: str # how this variation was created
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"step_index": self.step_index,
|
||||
"original": self.original,
|
||||
"variation": self.variation,
|
||||
"strategy": self.strategy,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class OptimizationResult:
|
||||
"""Results of running all variations through contract tests."""
|
||||
|
||||
tool_name: str
|
||||
variations: List[PromptVariation] = field(default_factory=list)
|
||||
scores: Dict[int, int] = field(default_factory=dict) # variation_index -> score
|
||||
best_index: Optional[int] = None
|
||||
baseline_score: int = 0
|
||||
evaluation_kind: str = "structural"
|
||||
note: str = ""
|
||||
|
||||
@property
|
||||
def best(self) -> Optional[PromptVariation]:
|
||||
if self.best_index is not None and self.best_index < len(self.variations):
|
||||
return self.variations[self.best_index]
|
||||
return None
|
||||
|
||||
@property
|
||||
def improvement(self) -> float:
|
||||
if self.best is None or self.baseline_score == 0:
|
||||
return 0.0
|
||||
best = self.scores.get(self.best_index, 0)
|
||||
return (best - self.baseline_score) / max(self.baseline_score, 1) * 100
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"tool": self.tool_name,
|
||||
"baseline_score": self.baseline_score,
|
||||
"evaluation_kind": self.evaluation_kind,
|
||||
"note": self.note,
|
||||
"best_index": self.best_index,
|
||||
"improvement_pct": round(self.improvement, 1),
|
||||
"best": self.best.to_dict() if self.best else None,
|
||||
"variations": [v.to_dict() for v in self.variations],
|
||||
"scores": {str(k): v for k, v in self.scores.items()},
|
||||
}
|
||||
|
||||
|
||||
def generate_variations(
|
||||
tool: Tool,
|
||||
count: int = 5,
|
||||
provider: str = "mock",
|
||||
) -> List[PromptVariation]:
|
||||
"""Generate prompt variations for each prompt step in the tool.
|
||||
|
||||
Strategies:
|
||||
- rephrase: ask the AI to rephrase the prompt
|
||||
- structure: convert to bullet points
|
||||
- concise: make it shorter
|
||||
- detailed: add more context
|
||||
- focused: emphasize the key task
|
||||
|
||||
Args:
|
||||
tool: The tool to optimize.
|
||||
count: Number of variations per prompt step.
|
||||
provider: Provider to use for variation generation.
|
||||
|
||||
Returns:
|
||||
List of PromptVariation objects.
|
||||
"""
|
||||
variations: List[PromptVariation] = []
|
||||
|
||||
for i, step in enumerate(tool.steps):
|
||||
if not isinstance(step, PromptStep):
|
||||
continue
|
||||
|
||||
original = step.prompt
|
||||
|
||||
strategies = _get_strategies(original, count, provider=provider)
|
||||
|
||||
for strategy, variation_text in strategies:
|
||||
if variation_text and variation_text != original:
|
||||
variations.append(PromptVariation(
|
||||
step_index=i,
|
||||
original=original,
|
||||
variation=variation_text,
|
||||
strategy=strategy,
|
||||
))
|
||||
|
||||
return variations
|
||||
|
||||
|
||||
def _get_strategies(
|
||||
original: str, count: int, provider: str = "mock"
|
||||
) -> List[tuple]:
|
||||
"""Generate variations using different strategies."""
|
||||
strategies: List[tuple] = []
|
||||
|
||||
if count <= 0:
|
||||
return []
|
||||
|
||||
# Provider calls are explicit. The default/mock path is deterministic and
|
||||
# never sends a user's prompt to an external process or service.
|
||||
if provider != "mock":
|
||||
rephrased = _call_provider_for_variation(
|
||||
provider,
|
||||
"Rephrase this instruction while keeping the same meaning. "
|
||||
f"Output only the rephrased text:\n\n{original}",
|
||||
)
|
||||
if rephrased and rephrased != original:
|
||||
strategies.append(("rephrase", rephrased))
|
||||
|
||||
# Deterministic mock variations
|
||||
mock_strategies = [
|
||||
("focused", f"Complete this specific task — {original}"),
|
||||
("concise", _truncate(original, 120)),
|
||||
("detailed", f"Carefully perform the following: {original}. Ensure correctness and completeness."),
|
||||
("rephrase", f"I need you to do this: {original}"),
|
||||
("structure", f"Step through this task:\n\n{original}\n\nRespond step by step."),
|
||||
]
|
||||
for strat, text in mock_strategies:
|
||||
if len(strategies) >= count:
|
||||
break
|
||||
if text and text != original:
|
||||
strategies.append((strat, text))
|
||||
|
||||
return strategies[:count]
|
||||
|
||||
|
||||
def _truncate(text: str, max_len: int) -> str:
|
||||
if len(text) <= max_len:
|
||||
return text
|
||||
return text[:max_len - 3].rsplit(" ", 1)[0] + "..."
|
||||
|
||||
|
||||
def _call_provider_for_variation(provider: str, prompt: str) -> Optional[str]:
|
||||
"""Call the provider explicitly selected by the user."""
|
||||
from .providers import call_provider
|
||||
|
||||
result = call_provider(provider, prompt, timeout=15)
|
||||
if result.success and result.text:
|
||||
text = result.text.strip()
|
||||
if text and len(text) >= 10:
|
||||
return text
|
||||
return None
|
||||
|
||||
|
||||
def optimize_tool(
|
||||
tool: Tool,
|
||||
count: int = 5,
|
||||
provider: str = "mock",
|
||||
evaluator: Optional[Callable[[Tool], float]] = None,
|
||||
) -> OptimizationResult:
|
||||
"""Run prompt optimization for a tool.
|
||||
|
||||
Generates variations, runs each through contract tests, and returns
|
||||
the best-performing variation.
|
||||
|
||||
Args:
|
||||
tool: The tool to optimize.
|
||||
count: Number of variations per prompt step.
|
||||
provider: Provider for variation generation.
|
||||
|
||||
Returns:
|
||||
OptimizationResult with scores and best variation.
|
||||
"""
|
||||
from .contract_testing import run_contract_tests
|
||||
|
||||
result = OptimizationResult(
|
||||
tool_name=tool.name,
|
||||
evaluation_kind="behavioral" if evaluator else "structural",
|
||||
)
|
||||
|
||||
# Baseline score
|
||||
if evaluator:
|
||||
result.baseline_score = evaluator(tool)
|
||||
else:
|
||||
baseline = run_contract_tests(tool)
|
||||
result.baseline_score = _count_passed(baseline)
|
||||
|
||||
# Generate variations
|
||||
result.variations = generate_variations(tool, count=count, provider=provider)
|
||||
|
||||
if not result.variations:
|
||||
return result
|
||||
|
||||
# Test each variation
|
||||
for idx, variation in enumerate(result.variations):
|
||||
# Create a copy with the modified prompt
|
||||
import copy
|
||||
tool_copy = copy.deepcopy(tool)
|
||||
if 0 <= variation.step_index < len(tool_copy.steps):
|
||||
step = tool_copy.steps[variation.step_index]
|
||||
if isinstance(step, PromptStep):
|
||||
step.prompt = variation.variation
|
||||
|
||||
try:
|
||||
if evaluator:
|
||||
result.scores[idx] = evaluator(tool_copy)
|
||||
else:
|
||||
test_result = run_contract_tests(tool_copy)
|
||||
result.scores[idx] = _count_passed(test_result)
|
||||
except Exception:
|
||||
result.scores[idx] = 0
|
||||
|
||||
# Find best
|
||||
if evaluator and result.scores:
|
||||
candidate = max(result.scores, key=lambda k: result.scores[k])
|
||||
if result.scores[candidate] > result.baseline_score:
|
||||
result.best_index = candidate
|
||||
else:
|
||||
result.note = "No variation improved on the behavioral baseline."
|
||||
elif result.scores:
|
||||
result.note = (
|
||||
"Structural conformance cannot compare prompt semantics; "
|
||||
"no best variation was selected."
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _count_passed(test_result) -> int:
|
||||
"""Count passed tests in a ConformanceReport."""
|
||||
return sum(1 for r in getattr(test_result, "results", [])
|
||||
if getattr(r, "state", "") == "passed")
|
||||
|
||||
|
||||
def load_behavioral_evaluator(path, provider_override: Optional[str] = None):
|
||||
"""Load explicit behavioral cases and return a semantic score function.
|
||||
|
||||
The JSON file must contain a list of cases. Each case accepts ``input`` and
|
||||
optional ``args``, plus exactly one of ``expected`` (exact value) or
|
||||
``contains`` (substring). Supplying this file is explicit authorization to
|
||||
execute the candidate tools and their configured providers.
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
cases = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||
if not isinstance(cases, list) or not cases:
|
||||
raise ValueError("Behavior test file must contain a non-empty JSON list")
|
||||
for index, case in enumerate(cases):
|
||||
if not isinstance(case, dict):
|
||||
raise ValueError(f"Behavior case {index} must be an object")
|
||||
assertions = [key for key in ("expected", "contains") if key in case]
|
||||
if len(assertions) != 1:
|
||||
raise ValueError(
|
||||
f"Behavior case {index} needs exactly one of expected or contains"
|
||||
)
|
||||
if not isinstance(case.get("args", {}), dict):
|
||||
raise ValueError(f"Behavior case {index} args must be an object")
|
||||
|
||||
def evaluate(tool: Tool) -> float:
|
||||
from .runner import run_tool
|
||||
|
||||
passed = 0
|
||||
for case in cases:
|
||||
output, exit_code = run_tool(
|
||||
tool,
|
||||
input_text=str(case.get("input", "")),
|
||||
custom_args=case.get("args", {}),
|
||||
provider_override=provider_override,
|
||||
dry_run=False,
|
||||
verbose=False,
|
||||
)
|
||||
if exit_code != 0:
|
||||
continue
|
||||
if "expected" in case:
|
||||
try:
|
||||
actual = json.loads(output)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
actual = output
|
||||
if actual == case["expected"]:
|
||||
passed += 1
|
||||
elif str(case["contains"]) in output:
|
||||
passed += 1
|
||||
return passed
|
||||
|
||||
return evaluate
|
||||
+384
-32
@@ -5,6 +5,7 @@ import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import shutil
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
@@ -54,6 +55,9 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "opencode run --model opencode/big-pickle",
|
||||
"description": "OpenCode - Big Pickle (free general model)",
|
||||
"tags": ["free", "code", "general"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
"install_group": "opencode",
|
||||
},
|
||||
"agy": {
|
||||
@@ -61,6 +65,9 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "agy -p",
|
||||
"description": "Antigravity - Google free tier, Gemini models",
|
||||
"tags": ["free-tier", "code", "large-context"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
"install_group": "agy",
|
||||
},
|
||||
"codex": {
|
||||
@@ -68,6 +75,9 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "codex exec -",
|
||||
"description": "Codex CLI - OpenAI free tier available",
|
||||
"tags": ["free-tier", "code", "general"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
"install_group": "codex",
|
||||
},
|
||||
"claude": {
|
||||
@@ -75,6 +85,9 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "claude -p",
|
||||
"description": "Claude Code - auto-routes to best model",
|
||||
"tags": ["paid", "subscription", "code"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
"install_group": "claude",
|
||||
},
|
||||
"crush": {
|
||||
@@ -82,6 +95,9 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "crush run --quiet",
|
||||
"description": "Crush - multi-model via Hyper credits or API keys",
|
||||
"tags": ["free-tier", "multi", "code"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
"install_group": "crush",
|
||||
},
|
||||
"ollama": {
|
||||
@@ -89,6 +105,10 @@ KNOWN_PROVIDER_CLIS = {
|
||||
"command": "ollama run llama3.2",
|
||||
"description": "Ollama - local, private, free",
|
||||
"tags": ["free", "local", "private"],
|
||||
"model": "llama3.2",
|
||||
"locality": "local",
|
||||
"capabilities": ["text", "structured-json"],
|
||||
"data_policy": "private",
|
||||
"install_group": "ollama",
|
||||
},
|
||||
}
|
||||
@@ -101,6 +121,9 @@ KNOWN_API_KEYS = {
|
||||
"model": "openrouter/auto-beta",
|
||||
"description": "OpenRouter - 300+ models, one API key",
|
||||
"tags": ["api", "per-token", "multi"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
},
|
||||
"DEEPSEEK_API_KEY": {
|
||||
"name": "deepseek-api",
|
||||
@@ -108,6 +131,9 @@ KNOWN_API_KEYS = {
|
||||
"model": "deepseek-chat",
|
||||
"description": "DeepSeek API - inexpensive per-token access",
|
||||
"tags": ["api", "per-token", "cheap"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
},
|
||||
"OPENAI_API_KEY": {
|
||||
"name": "openai-api",
|
||||
@@ -115,6 +141,9 @@ KNOWN_API_KEYS = {
|
||||
"model": "gpt-4o",
|
||||
"description": "OpenAI API - direct GPT access",
|
||||
"tags": ["api", "per-token", "code"],
|
||||
"locality": "remote",
|
||||
"capabilities": ["text", "structured-json", "reasoning"],
|
||||
"data_policy": "public",
|
||||
},
|
||||
}
|
||||
|
||||
@@ -180,6 +209,10 @@ def discover_installed_providers() -> List[dict]:
|
||||
"command": f"ollama run {model_name}",
|
||||
"description": f"Ollama local model: {model_name}",
|
||||
"tags": ["free", "local", "private"],
|
||||
"model": model_name,
|
||||
"locality": "local",
|
||||
"capabilities": ["text", "structured-json"],
|
||||
"data_policy": "private",
|
||||
})
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
@@ -187,6 +220,26 @@ def discover_installed_providers() -> List[dict]:
|
||||
return found
|
||||
|
||||
|
||||
def _provider_from_discovery(data: dict) -> "Provider":
|
||||
"""Convert trusted discovery facts without persisting scanner-only keys."""
|
||||
return Provider(
|
||||
name=data["name"],
|
||||
command=data["command"],
|
||||
description=data.get("description", ""),
|
||||
type=data.get("type", "subprocess"),
|
||||
model=data.get("model"),
|
||||
tags=data.get("tags", []),
|
||||
api_key_env=data.get("env_var"),
|
||||
locality=data.get("locality", "unknown"),
|
||||
capabilities=data.get("capabilities", []),
|
||||
model_digest=data.get("model_digest"),
|
||||
cost_class=data.get("cost_class"),
|
||||
latency_class=data.get("latency_class"),
|
||||
max_context_tokens=data.get("max_context_tokens"),
|
||||
data_policy=data.get("data_policy", "unspecified"),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Provider:
|
||||
"""Definition of an AI provider.
|
||||
@@ -212,6 +265,52 @@ class Provider:
|
||||
fallback_chain: Optional[List[str]] = None # Ordered multi-step fallback (new)
|
||||
api_key_env: Optional[str] = None # Env var name for api-type providers
|
||||
pty_config: Optional[dict] = None # Patterns for pty-type providers
|
||||
tools: Optional[List[str]] = None # Allowed CmdForge tools (None = all)
|
||||
mcp_servers: Optional[List[str]] = None # MCP servers available to this provider
|
||||
locality: str = "unknown" # "local" | "remote" | "unknown"
|
||||
capabilities: List[str] = field(default_factory=list)
|
||||
model_digest: Optional[str] = None
|
||||
cost_class: Optional[str] = None # "free" | "low" | "medium" | "high"
|
||||
latency_class: Optional[str] = None # "fast" | "standard" | "slow"
|
||||
max_context_tokens: Optional[int] = None
|
||||
data_policy: str = "unspecified" # max: public | internal | private
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
for field_name, values in (
|
||||
("tools", self.tools),
|
||||
("mcp_servers", self.mcp_servers),
|
||||
):
|
||||
if values is not None and (
|
||||
not isinstance(values, list)
|
||||
or not all(isinstance(value, str) and value for value in values)
|
||||
):
|
||||
raise ValueError(
|
||||
f"Provider {field_name} must be a list of non-empty strings or null"
|
||||
)
|
||||
if self.locality not in {"local", "remote", "unknown"}:
|
||||
raise ValueError("Provider locality must be local, remote, or unknown")
|
||||
if (
|
||||
not isinstance(self.capabilities, list)
|
||||
or not all(isinstance(value, str) and value for value in self.capabilities)
|
||||
):
|
||||
raise ValueError("Provider capabilities must be a list of non-empty strings")
|
||||
if self.cost_class not in {None, "free", "low", "medium", "high"}:
|
||||
raise ValueError("Provider cost_class must be free, low, medium, or high")
|
||||
if self.latency_class not in {None, "fast", "standard", "slow"}:
|
||||
raise ValueError("Provider latency_class must be fast, standard, or slow")
|
||||
if self.data_policy not in {"unspecified", "public", "internal", "private"}:
|
||||
raise ValueError(
|
||||
"Provider data_policy must be unspecified, public, internal, or private"
|
||||
)
|
||||
if (
|
||||
self.max_context_tokens is not None
|
||||
and (
|
||||
not isinstance(self.max_context_tokens, int)
|
||||
or isinstance(self.max_context_tokens, bool)
|
||||
or self.max_context_tokens <= 0
|
||||
)
|
||||
):
|
||||
raise ValueError("Provider max_context_tokens must be a positive integer")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
d = {
|
||||
@@ -236,6 +335,24 @@ class Provider:
|
||||
d["api_key_env"] = self.api_key_env
|
||||
if self.pty_config:
|
||||
d["pty_config"] = self.pty_config
|
||||
if self.tools is not None:
|
||||
d["tools"] = self.tools
|
||||
if self.mcp_servers is not None:
|
||||
d["mcp_servers"] = self.mcp_servers
|
||||
if self.locality != "unknown":
|
||||
d["locality"] = self.locality
|
||||
if self.capabilities:
|
||||
d["capabilities"] = self.capabilities
|
||||
if self.model_digest:
|
||||
d["model_digest"] = self.model_digest
|
||||
if self.cost_class:
|
||||
d["cost_class"] = self.cost_class
|
||||
if self.latency_class:
|
||||
d["latency_class"] = self.latency_class
|
||||
if self.max_context_tokens is not None:
|
||||
d["max_context_tokens"] = self.max_context_tokens
|
||||
if self.data_policy != "unspecified":
|
||||
d["data_policy"] = self.data_policy
|
||||
return d
|
||||
|
||||
@classmethod
|
||||
@@ -245,13 +362,24 @@ class Provider:
|
||||
command=data["command"],
|
||||
description=data.get("description", ""),
|
||||
type=data.get("type", "subprocess"),
|
||||
model=data.get("model"),
|
||||
model=data.get("model", data.get("model_id")),
|
||||
fallback=data.get("fallback"),
|
||||
tags=data.get("tags", []) or [],
|
||||
install=data.get("install"),
|
||||
fallback_chain=data.get("fallback_chain"),
|
||||
api_key_env=data.get("api_key_env"),
|
||||
pty_config=data.get("pty_config"),
|
||||
tools=data.get("tools"),
|
||||
mcp_servers=data.get("mcp_servers"),
|
||||
locality=data.get("locality", "unknown"),
|
||||
capabilities=data.get("capabilities", []) or [],
|
||||
model_digest=data.get("model_digest"),
|
||||
cost_class=data.get("cost_class"),
|
||||
latency_class=data.get("latency_class"),
|
||||
max_context_tokens=data.get(
|
||||
"max_context_tokens", data.get("max_input_tokens")
|
||||
),
|
||||
data_policy=data.get("data_policy", "unspecified"),
|
||||
)
|
||||
|
||||
|
||||
@@ -261,6 +389,164 @@ class ProviderResult:
|
||||
text: str
|
||||
success: bool
|
||||
error: Optional[str] = None
|
||||
requested_provider: Optional[str] = None
|
||||
actual_provider: Optional[str] = None
|
||||
model: Optional[str] = None
|
||||
model_digest: Optional[str] = None
|
||||
attempted_providers: List[str] = field(default_factory=list)
|
||||
fallback_used: bool = False
|
||||
locality: str = "unknown"
|
||||
model_identity_source: str = "unknown"
|
||||
|
||||
def provenance(self) -> dict:
|
||||
"""Return runtime-owned execution facts safe for result envelopes."""
|
||||
data = {
|
||||
"success": self.success,
|
||||
"requested_provider": self.requested_provider,
|
||||
"actual_provider": self.actual_provider,
|
||||
"attempted_providers": list(self.attempted_providers),
|
||||
"fallback_used": self.fallback_used,
|
||||
"model": self.model,
|
||||
"model_digest": self.model_digest,
|
||||
"locality": self.locality,
|
||||
"model_identity_source": self.model_identity_source,
|
||||
}
|
||||
if self.error:
|
||||
data["error"] = self.error
|
||||
return data
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ProviderExecutionPolicy:
|
||||
"""Caller-supplied, fail-closed constraints for provider execution."""
|
||||
|
||||
allow_fallback: bool = True
|
||||
required_locality: Optional[str] = None
|
||||
required_capabilities: tuple[str, ...] = ()
|
||||
data_classification: str = "public"
|
||||
require_model_identity: bool = False
|
||||
require_model_digest: bool = False
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if self.required_locality not in {None, "local", "remote"}:
|
||||
raise ValueError("required_locality must be local, remote, or null")
|
||||
if self.data_classification not in {"public", "internal", "private"}:
|
||||
raise ValueError(
|
||||
"data_classification must be public, internal, or private"
|
||||
)
|
||||
|
||||
|
||||
_DATA_CLASS_LEVEL = {"public": 0, "internal": 1, "private": 2}
|
||||
|
||||
|
||||
def effective_provider_locality(provider: Provider) -> str:
|
||||
"""Account for Ollama clients redirected to a non-loopback host."""
|
||||
try:
|
||||
executable = shlex.split(os.path.expandvars(provider.command))[0]
|
||||
except (IndexError, ValueError):
|
||||
executable = ""
|
||||
if Path(executable).name != "ollama":
|
||||
return provider.locality
|
||||
ollama_host = os.environ.get("OLLAMA_HOST", "").strip()
|
||||
if not ollama_host:
|
||||
return provider.locality
|
||||
from urllib.parse import urlparse
|
||||
parsed = urlparse(
|
||||
ollama_host if "://" in ollama_host else f"//{ollama_host}"
|
||||
)
|
||||
hostname = (parsed.hostname or "").lower()
|
||||
if hostname in {"localhost", "127.0.0.1", "::1"}:
|
||||
return "local"
|
||||
return "remote"
|
||||
|
||||
|
||||
def runtime_provider_identity(provider: Provider) -> tuple[Optional[str], Optional[str], str]:
|
||||
"""Return observed model identity when supported, else configured facts."""
|
||||
configured = (
|
||||
provider.model,
|
||||
provider.model_digest,
|
||||
"provider-config" if provider.model or provider.model_digest else "unknown",
|
||||
)
|
||||
try:
|
||||
parts = shlex.split(os.path.expandvars(provider.command))
|
||||
except ValueError:
|
||||
return configured
|
||||
if not parts or Path(parts[0]).name != "ollama":
|
||||
return configured
|
||||
if effective_provider_locality(provider) != "local":
|
||||
return configured
|
||||
try:
|
||||
run_index = parts.index("run")
|
||||
model = provider.model or parts[run_index + 1]
|
||||
except (ValueError, IndexError):
|
||||
return configured
|
||||
|
||||
host = os.environ.get("OLLAMA_HOST", "http://127.0.0.1:11434").strip()
|
||||
if "://" not in host:
|
||||
host = f"http://{host}"
|
||||
from urllib.request import Request, urlopen
|
||||
import json
|
||||
|
||||
request = Request(host.rstrip("/") + "/api/tags")
|
||||
api_key = os.environ.get("OLLAMA_API_KEY")
|
||||
if api_key:
|
||||
request.add_header("Authorization", f"Bearer {api_key}")
|
||||
try:
|
||||
with urlopen(request, timeout=2) as response:
|
||||
payload = json.loads(response.read().decode("utf-8"))
|
||||
except Exception:
|
||||
return configured
|
||||
wanted = {model, f"{model}:latest"}
|
||||
for item in payload.get("models", []):
|
||||
names = {item.get("name"), item.get("model")}
|
||||
if names & wanted or any(
|
||||
value and value.removesuffix(":latest") == model.removesuffix(":latest")
|
||||
for value in names
|
||||
):
|
||||
digest = item.get("digest") or provider.model_digest
|
||||
if digest and not digest.startswith("sha256:"):
|
||||
digest = f"sha256:{digest}"
|
||||
return item.get("model") or item.get("name") or model, digest, "ollama-api"
|
||||
return configured
|
||||
|
||||
|
||||
def provider_policy_error(
|
||||
provider: Provider,
|
||||
policy: ProviderExecutionPolicy,
|
||||
runtime_identity: Optional[tuple[Optional[str], Optional[str], str]] = None,
|
||||
) -> Optional[str]:
|
||||
"""Return why a provider is ineligible, or None when it is allowed."""
|
||||
if (
|
||||
policy.required_locality is not None
|
||||
and effective_provider_locality(provider) != policy.required_locality
|
||||
):
|
||||
return (
|
||||
f"Provider '{provider.name}' locality is "
|
||||
f"{effective_provider_locality(provider)}; "
|
||||
f"{policy.required_locality} is required"
|
||||
)
|
||||
missing = sorted(set(policy.required_capabilities) - set(provider.capabilities))
|
||||
if missing:
|
||||
return (
|
||||
f"Provider '{provider.name}' lacks required capabilities: "
|
||||
+ ", ".join(missing)
|
||||
)
|
||||
if policy.data_classification != "public":
|
||||
allowed = _DATA_CLASS_LEVEL.get(provider.data_policy, -1)
|
||||
needed = _DATA_CLASS_LEVEL[policy.data_classification]
|
||||
if allowed < needed:
|
||||
return (
|
||||
f"Provider '{provider.name}' data_policy is {provider.data_policy}; "
|
||||
f"{policy.data_classification} data requires an explicit compatible policy"
|
||||
)
|
||||
model, digest, _ = runtime_identity or (
|
||||
provider.model, provider.model_digest, "provider-config"
|
||||
)
|
||||
if policy.require_model_identity and not model:
|
||||
return f"Provider '{provider.name}' does not declare a model identity"
|
||||
if policy.require_model_digest and not digest:
|
||||
return f"Provider '{provider.name}' does not declare a model digest"
|
||||
return None
|
||||
|
||||
|
||||
# Default providers that come pre-configured
|
||||
@@ -312,7 +598,12 @@ DEFAULT_PROVIDERS = [
|
||||
# LOCAL MODELS
|
||||
Provider("ollama", "ollama run llama3.2",
|
||||
"Ollama - local, private, free (GPU recommended)",
|
||||
tags=["free", "local", "private"]),
|
||||
model="llama3.2",
|
||||
tags=["free", "local", "private"],
|
||||
locality="local",
|
||||
capabilities=["text", "structured-json"],
|
||||
cost_class="free",
|
||||
data_policy="private"),
|
||||
|
||||
# API-TYPE PROVIDERS (pay-per-token, fallback when no CLI covers model)
|
||||
Provider("openrouter",
|
||||
@@ -321,18 +612,26 @@ DEFAULT_PROVIDERS = [
|
||||
type="api",
|
||||
model="openrouter/auto-beta",
|
||||
api_key_env="OPENROUTER_API_KEY",
|
||||
tags=["api", "per-token", "multi", "fallback"]),
|
||||
tags=["api", "per-token", "multi", "fallback"],
|
||||
locality="remote",
|
||||
capabilities=["text", "structured-json", "reasoning"],
|
||||
data_policy="public"),
|
||||
Provider("deepseek-api",
|
||||
"https://api.deepseek.com/v1",
|
||||
"DeepSeek API - inexpensive per-token access",
|
||||
type="api",
|
||||
model="deepseek-chat",
|
||||
api_key_env="DEEPSEEK_API_KEY",
|
||||
tags=["api", "per-token", "cheap", "reasoning"]),
|
||||
tags=["api", "per-token", "cheap", "reasoning"],
|
||||
locality="remote",
|
||||
capabilities=["text", "structured-json", "reasoning"],
|
||||
data_policy="public"),
|
||||
|
||||
# Mock for testing
|
||||
Provider("mock", "mock", "Mock provider for testing",
|
||||
tags=["testing"]),
|
||||
model="mock", tags=["testing"], locality="local",
|
||||
capabilities=["text", "structured-json"],
|
||||
data_policy="private"),
|
||||
]
|
||||
|
||||
|
||||
@@ -343,7 +642,6 @@ def get_providers_file() -> Path:
|
||||
discovered = discover_installed_providers()
|
||||
selected = []
|
||||
if discovered:
|
||||
import sys
|
||||
print("=" * 60, file=sys.stderr)
|
||||
print("CmdForge First-Run Provider Setup", file=sys.stderr)
|
||||
print("=" * 60, file=sys.stderr)
|
||||
@@ -357,10 +655,7 @@ def get_providers_file() -> Path:
|
||||
for d in cli_found:
|
||||
name = d["name"]
|
||||
desc = d.get("description", "")
|
||||
selected.append(Provider(
|
||||
name=name, command=d["command"], description=desc,
|
||||
tags=d.get("tags", []),
|
||||
))
|
||||
selected.append(_provider_from_discovery(d))
|
||||
print(f" [+] {name:20s} {desc}", file=sys.stderr)
|
||||
|
||||
if api_found:
|
||||
@@ -370,11 +665,7 @@ def get_providers_file() -> Path:
|
||||
desc = d.get("description", "")
|
||||
model = d.get("model", "auto")
|
||||
env_var = d.get("env_var", "")
|
||||
selected.append(Provider(
|
||||
name=name, command=d["command"], description=desc,
|
||||
type="api", model=model, api_key_env=env_var,
|
||||
tags=d.get("tags", []),
|
||||
))
|
||||
selected.append(_provider_from_discovery(d))
|
||||
print(f" [+] {name:20s} ({model})", file=sys.stderr)
|
||||
|
||||
if ollama_found:
|
||||
@@ -382,10 +673,7 @@ def get_providers_file() -> Path:
|
||||
for d in ollama_found[:10]:
|
||||
name = d["name"]
|
||||
desc = d.get("description", "")
|
||||
selected.append(Provider(
|
||||
name=name, command=d["command"], description=desc,
|
||||
tags=d.get("tags", []),
|
||||
))
|
||||
selected.append(_provider_from_discovery(d))
|
||||
print(f" [+] {name:30s} {d['command']}", file=sys.stderr)
|
||||
if len(ollama_found) > 10:
|
||||
print(f" ... and {len(ollama_found) - 10} more", file=sys.stderr)
|
||||
@@ -411,8 +699,11 @@ def load_providers() -> List[Provider]:
|
||||
if not data or "providers" not in data:
|
||||
return DEFAULT_PROVIDERS.copy()
|
||||
providers = [Provider.from_dict(p) for p in data["providers"]]
|
||||
except Exception:
|
||||
return DEFAULT_PROVIDERS.copy()
|
||||
except Exception as exc:
|
||||
# A present but malformed configuration must not silently fall back to
|
||||
# unrestricted defaults, particularly when access policies are invalid.
|
||||
print(f"Warning: Failed to load provider configuration: {exc}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
config_version = data.get("version", 1)
|
||||
if not isinstance(config_version, int):
|
||||
@@ -489,7 +780,17 @@ def delete_provider(name: str) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_tokens: Optional[int] = None, _tried: Optional[set] = None) -> ProviderResult:
|
||||
def call_provider(
|
||||
provider_name: str,
|
||||
prompt: str,
|
||||
timeout: int = 300,
|
||||
max_tokens: Optional[int] = None,
|
||||
_tried: Optional[set] = None,
|
||||
*,
|
||||
execution_policy: Optional[ProviderExecutionPolicy] = None,
|
||||
_requested_provider: Optional[str] = None,
|
||||
_attempted_providers: Optional[List[str]] = None,
|
||||
) -> ProviderResult:
|
||||
"""
|
||||
Call an AI provider with the given prompt.
|
||||
|
||||
@@ -503,59 +804,110 @@ def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_token
|
||||
timeout: Maximum execution time in seconds
|
||||
max_tokens: Optional max output tokens (appends provider-specific flag)
|
||||
_tried: Internal set of already-tried providers (prevents infinite loops)
|
||||
execution_policy: Optional fallback and eligibility constraints
|
||||
|
||||
Returns:
|
||||
ProviderResult with the response text or error
|
||||
"""
|
||||
policy = execution_policy or ProviderExecutionPolicy()
|
||||
requested_provider = _requested_provider or provider_name
|
||||
attempted = _attempted_providers if _attempted_providers is not None else []
|
||||
if provider_name not in attempted:
|
||||
attempted.append(provider_name)
|
||||
observed_identity = ("mock", None, "built-in")
|
||||
|
||||
# Track which providers we've tried to prevent infinite fallback loops
|
||||
if _tried is None:
|
||||
_tried = set()
|
||||
_tried.add(provider_name)
|
||||
|
||||
def finish(
|
||||
result: ProviderResult,
|
||||
provider: Optional[Provider] = None,
|
||||
) -> ProviderResult:
|
||||
"""Attach facts observed by CmdForge, never values generated by the model."""
|
||||
result.requested_provider = requested_provider
|
||||
result.attempted_providers = list(attempted)
|
||||
if result.success:
|
||||
result.actual_provider = provider_name
|
||||
(
|
||||
result.model,
|
||||
result.model_digest,
|
||||
result.model_identity_source,
|
||||
) = observed_identity
|
||||
result.locality = (
|
||||
effective_provider_locality(provider) if provider else "local"
|
||||
)
|
||||
result.fallback_used = provider_name != requested_provider
|
||||
return result
|
||||
|
||||
# Handle mock provider specially
|
||||
if provider_name.lower() == "mock":
|
||||
return mock_provider(prompt)
|
||||
return finish(mock_provider(prompt))
|
||||
|
||||
# Look up provider
|
||||
provider = get_provider(provider_name)
|
||||
if not provider:
|
||||
return ProviderResult(
|
||||
return finish(ProviderResult(
|
||||
text="",
|
||||
success=False,
|
||||
error=f"Provider '{provider_name}' not found. Use 'cmdforge providers' to manage providers."
|
||||
)
|
||||
))
|
||||
observed_identity = runtime_provider_identity(provider)
|
||||
|
||||
if max_tokens is not None:
|
||||
try:
|
||||
max_tokens = int(max_tokens)
|
||||
except (TypeError, ValueError):
|
||||
return ProviderResult(text="", success=False, error="max_tokens must be an integer")
|
||||
return finish(ProviderResult(text="", success=False, error="max_tokens must be an integer"), provider)
|
||||
if max_tokens <= 0 or max_tokens > 1_000_000:
|
||||
return ProviderResult(
|
||||
return finish(ProviderResult(
|
||||
text="", success=False, error="max_tokens must be between 1 and 1000000"
|
||||
)
|
||||
), provider)
|
||||
|
||||
# Helper to try fallback provider(s) if available
|
||||
def try_fallback(error_msg: str) -> ProviderResult:
|
||||
import sys
|
||||
last_error = error_msg
|
||||
if not policy.allow_fallback:
|
||||
return finish(
|
||||
ProviderResult(text="", success=False, error=last_error), provider
|
||||
)
|
||||
# Walk fallback_chain first (ordered, multi-step)
|
||||
if provider.fallback_chain:
|
||||
for fb in provider.fallback_chain:
|
||||
if fb not in _tried:
|
||||
print(f"[fallback] {provider_name} failed, trying {fb}...", file=sys.stderr)
|
||||
result = call_provider(fb, prompt, timeout, max_tokens, _tried)
|
||||
result = call_provider(
|
||||
fb, prompt, timeout, max_tokens, _tried,
|
||||
execution_policy=policy,
|
||||
_requested_provider=requested_provider,
|
||||
_attempted_providers=attempted,
|
||||
)
|
||||
if result.success:
|
||||
return result
|
||||
last_error = result.error or last_error
|
||||
# Fall back to single fallback (backward-compat)
|
||||
if provider.fallback and provider.fallback not in _tried:
|
||||
print(f"[fallback] {provider_name} failed, trying {provider.fallback}...", file=sys.stderr)
|
||||
result = call_provider(provider.fallback, prompt, timeout, max_tokens, _tried)
|
||||
result = call_provider(
|
||||
provider.fallback, prompt, timeout, max_tokens, _tried,
|
||||
execution_policy=policy,
|
||||
_requested_provider=requested_provider,
|
||||
_attempted_providers=attempted,
|
||||
)
|
||||
if result.success:
|
||||
return result
|
||||
last_error = result.error or last_error
|
||||
return ProviderResult(text="", success=False, error=last_error)
|
||||
return finish(
|
||||
ProviderResult(text="", success=False, error=last_error), provider
|
||||
)
|
||||
|
||||
eligibility_error = provider_policy_error(
|
||||
provider, policy, observed_identity
|
||||
)
|
||||
if eligibility_error:
|
||||
return try_fallback(eligibility_error)
|
||||
|
||||
# Dispatch by provider type
|
||||
ptype = getattr(provider, "type", None) or "subprocess"
|
||||
@@ -570,7 +922,7 @@ def call_provider(provider_name: str, prompt: str, timeout: int = 300, max_token
|
||||
return try_fallback(f"Unknown provider type: {ptype}")
|
||||
|
||||
if result.success:
|
||||
return result
|
||||
return finish(result, provider)
|
||||
return try_fallback(result.error or "Provider call failed")
|
||||
except Exception as e:
|
||||
return try_fallback(f"Provider error: {str(e)}")
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
"""Explainable quality scores for CmdForge tools.
|
||||
|
||||
Produces a headline score with a per-category breakdown. Each category shows
|
||||
earned/available points, and missing evidence is shown as an opportunity
|
||||
rather than a penalty.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from .tool import Tool
|
||||
from .preflight import PreflightReport
|
||||
|
||||
|
||||
@dataclass
|
||||
class CategoryScore:
|
||||
"""One quality category."""
|
||||
|
||||
name: str
|
||||
earned: int
|
||||
available: int
|
||||
state: str = "checked" # checked | not_tested | not_applicable
|
||||
possible: Optional[int] = None # Category maximum when only partial evidence ran
|
||||
|
||||
@property
|
||||
def display(self) -> str:
|
||||
if self.state == "not_tested":
|
||||
return "not tested"
|
||||
if self.state == "not_applicable":
|
||||
return "n/a"
|
||||
return f"{self.earned}/{self.available}"
|
||||
|
||||
@property
|
||||
def percentage(self) -> float:
|
||||
return (self.earned / self.available * 100) if self.available > 0 else 0.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class QualityScore:
|
||||
"""Headline score with category breakdown."""
|
||||
|
||||
tool_name: str
|
||||
version: str
|
||||
headline: int # 0-100
|
||||
categories: List[CategoryScore] = field(default_factory=list)
|
||||
last_evaluated: str = ""
|
||||
evidence_coverage: int = 0
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return {
|
||||
"tool": self.tool_name,
|
||||
"version": self.version,
|
||||
"score": self.headline,
|
||||
"categories": [
|
||||
{
|
||||
"name": c.name,
|
||||
"earned": c.earned,
|
||||
"available": c.available,
|
||||
"state": c.state,
|
||||
"possible": c.possible if c.possible is not None else c.available,
|
||||
}
|
||||
for c in self.categories
|
||||
],
|
||||
"last_evaluated": self.last_evaluated,
|
||||
"evidence_coverage": self.evidence_coverage,
|
||||
}
|
||||
|
||||
def __str__(self) -> str:
|
||||
lines = [f"Quality {self.headline} — {self.evidence_coverage}% evidence coverage"]
|
||||
if self.last_evaluated:
|
||||
lines[0] += f" — Evaluated {self.last_evaluated}"
|
||||
for c in self.categories:
|
||||
lines.append(f" {c.name:24s} {c.display:>8s}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def compute_quality(
|
||||
tool: Tool,
|
||||
report: PreflightReport,
|
||||
registry_data: Optional[dict] = None,
|
||||
) -> QualityScore:
|
||||
"""Compute an explainable quality score from preflight evidence.
|
||||
|
||||
Categories:
|
||||
- Contracts (15 pts): input_schema and output_schema present and valid
|
||||
- Deterministic tests (30 pts): conformance test pass rate
|
||||
- Regression history (20 pts): no regressions in baseline comparison
|
||||
- Security scrutiny (20 pts): no secrets, no unresolved dependencies
|
||||
- Community evidence (15 pts): registry reviews, downloads
|
||||
|
||||
Missing evidence shows as "not_tested" and does not penalize.
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
categories = []
|
||||
categories.append(_score_contracts(tool, report))
|
||||
categories.append(_score_tests(report))
|
||||
categories.append(_score_regression(report))
|
||||
categories.append(_score_security(report, tool))
|
||||
categories.append(_score_community(registry_data))
|
||||
|
||||
checked = [c for c in categories if c.state == "checked"]
|
||||
earned = sum(c.earned for c in checked)
|
||||
available = sum(c.available for c in checked)
|
||||
possible = sum(
|
||||
c.possible if c.possible is not None else c.available
|
||||
for c in categories if c.state != "not_applicable"
|
||||
)
|
||||
headline = int((earned / available * 100)) if available > 0 else 0
|
||||
|
||||
return QualityScore(
|
||||
tool_name=tool.name,
|
||||
version=tool.version or "",
|
||||
headline=headline,
|
||||
categories=categories,
|
||||
last_evaluated=datetime.now(timezone.utc).isoformat(),
|
||||
evidence_coverage=int((available / possible * 100)) if possible else 0,
|
||||
)
|
||||
|
||||
|
||||
def _score_contracts(tool: Tool, report: PreflightReport) -> CategoryScore:
|
||||
available = 15
|
||||
earned = 0
|
||||
|
||||
if tool.input_schema is not None:
|
||||
earned += 5
|
||||
if tool.output_schema is not None:
|
||||
earned += 5
|
||||
if tool.input_schema is None and tool.output_schema is None:
|
||||
return CategoryScore("Contracts", 0, available, "not_tested")
|
||||
|
||||
# No contract-related errors in the preflight
|
||||
contract_errors = [
|
||||
e for e in report.errors if "schema" in e.lower() or "contract" in e.lower()
|
||||
]
|
||||
if not contract_errors:
|
||||
earned += 5
|
||||
else:
|
||||
earned = max(0, earned - len(contract_errors))
|
||||
|
||||
return CategoryScore("Contracts", earned, available, "checked")
|
||||
|
||||
|
||||
def _score_tests(report: PreflightReport) -> CategoryScore:
|
||||
maximum = 30
|
||||
tests = report.generated_tests or []
|
||||
|
||||
if not tests:
|
||||
return CategoryScore("Deterministic tests", 0, maximum, "not_tested")
|
||||
|
||||
executed = [t for t in tests if t.get("state") in ("passed", "failed")]
|
||||
if not executed:
|
||||
return CategoryScore("Deterministic tests", 0, maximum, "not_tested")
|
||||
passed = sum(1 for t in executed if t.get("state") == "passed")
|
||||
total = len(executed)
|
||||
available = max(1, int(maximum * total / len(tests)))
|
||||
earned = int((passed / total) * available)
|
||||
return CategoryScore(
|
||||
"Deterministic tests", earned, available, "checked", possible=maximum
|
||||
)
|
||||
|
||||
|
||||
def _score_regression(report: PreflightReport) -> CategoryScore:
|
||||
available = 20
|
||||
regression = report.regression
|
||||
|
||||
if not regression:
|
||||
return CategoryScore("Regression history", 0, available, "not_tested")
|
||||
|
||||
if regression.get("has_regressions"):
|
||||
return CategoryScore("Regression history", 0, available, "checked")
|
||||
|
||||
if regression.get("summary") and "no changes" in regression.get("summary", "").lower():
|
||||
return CategoryScore("Regression history", available, available, "checked")
|
||||
|
||||
# Improvements or new tests — partial credit
|
||||
return CategoryScore("Regression history", available, available, "checked")
|
||||
|
||||
|
||||
def _score_security(report: PreflightReport, tool: Tool) -> CategoryScore:
|
||||
maximum = 20
|
||||
checks = set((report.audit_evidence or {}).get("checks_run", []))
|
||||
if "security_scrutiny" in checks:
|
||||
available = maximum
|
||||
elif checks:
|
||||
available = (10 if "secrets" in checks else 0) + (
|
||||
10 if checks.intersection({"dependencies", "registry_dependencies"}) else 0
|
||||
)
|
||||
elif report.errors or report.warnings:
|
||||
# Backward-compatible evidence supplied by callers without audit metadata.
|
||||
available = maximum
|
||||
else:
|
||||
available = 0
|
||||
if not available:
|
||||
return CategoryScore(
|
||||
"Security scrutiny", 0, maximum, "not_tested", possible=maximum
|
||||
)
|
||||
earned = available
|
||||
|
||||
# Deduct for secret patterns found
|
||||
secret_warnings = [
|
||||
w for w in report.warnings if "secret" in w.lower()
|
||||
]
|
||||
earned -= len(secret_warnings) * 5
|
||||
|
||||
# Deduct for unresolved dependencies
|
||||
dependency_findings = [
|
||||
finding for finding in report.warnings + report.errors
|
||||
if ("dependency" in finding.lower() or "dependencies" in finding.lower())
|
||||
and any(
|
||||
marker in finding.lower()
|
||||
for marker in ("not installed", "missing", "deprecated")
|
||||
)
|
||||
]
|
||||
earned -= len(dependency_findings) * 3
|
||||
|
||||
earned = max(0, earned)
|
||||
return CategoryScore(
|
||||
"Security scrutiny", earned, available, "checked", possible=maximum
|
||||
)
|
||||
|
||||
|
||||
def _score_community(registry_data: Optional[dict]) -> CategoryScore:
|
||||
available = 15
|
||||
|
||||
if not registry_data:
|
||||
return CategoryScore("Community evidence", 0, available, "not_tested")
|
||||
|
||||
earned = 0
|
||||
reviews = registry_data.get("reviews", [])
|
||||
rating_count = registry_data.get("rating_count", len(reviews))
|
||||
avg_rating = registry_data.get("average_rating")
|
||||
if avg_rating is None and reviews:
|
||||
avg_rating = sum(r.get("rating", 0) for r in reviews) / len(reviews)
|
||||
if avg_rating is not None and rating_count:
|
||||
earned += int((avg_rating / 5) * 8)
|
||||
|
||||
downloads = registry_data.get("downloads", 0)
|
||||
if downloads > 100:
|
||||
earned += 4
|
||||
elif downloads > 10:
|
||||
earned += 2
|
||||
|
||||
if registry_data.get("featured"):
|
||||
earned += 3
|
||||
|
||||
earned = min(earned, available)
|
||||
return CategoryScore("Community evidence", earned, available, "checked")
|
||||
+569
-18
@@ -9,10 +9,11 @@ import os
|
||||
import re
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime, timedelta
|
||||
from datetime import date, datetime, timedelta, timezone
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from flask import Flask, Response, g, jsonify, request
|
||||
from flask import Flask, Response, current_app, g, jsonify, request
|
||||
import yaml
|
||||
from functools import wraps
|
||||
from argon2 import PasswordHasher
|
||||
@@ -67,6 +68,11 @@ rate_limiter = RateLimiter()
|
||||
password_hasher = PasswordHasher(memory_cost=65536, time_cost=3, parallelism=4)
|
||||
|
||||
|
||||
def _naive_utc_now() -> datetime:
|
||||
"""Return UTC without tzinfo for compatibility with existing DB values."""
|
||||
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Semver:
|
||||
major: int
|
||||
@@ -275,6 +281,14 @@ def run_ai_scrutiny_review(scrutiny_report: dict, config: dict, tool_name: str,
|
||||
def create_app() -> Flask:
|
||||
app = Flask(__name__)
|
||||
app.config["MAX_CONTENT_LENGTH"] = MAX_BODY_BYTES
|
||||
app.config.update(
|
||||
CMDFORGE_ENV=os.environ.get("CMDFORGE_ENV", "development"),
|
||||
PUBLIC_BASE_URL=os.environ.get("CMDFORGE_PUBLIC_URL", "").rstrip("/"),
|
||||
MAIL_TRANSPORT=os.environ.get("MAIL_TRANSPORT", "disabled"),
|
||||
MAIL_FROM=os.environ.get("MAIL_FROM", ""),
|
||||
MAIL_TIMEOUT=os.environ.get("MAIL_TIMEOUT", "10"),
|
||||
RESEND_API_KEY=os.environ.get("RESEND_API_KEY", ""),
|
||||
)
|
||||
|
||||
# Initialize database schema once at startup
|
||||
with connect_db() as init_conn:
|
||||
@@ -418,7 +432,7 @@ def create_app() -> Flask:
|
||||
|
||||
g.db.execute(
|
||||
"UPDATE api_tokens SET last_used_at = ? WHERE id = ?",
|
||||
[datetime.utcnow().isoformat(), row["id"]],
|
||||
[datetime.now(timezone.utc).isoformat(), row["id"]],
|
||||
)
|
||||
g.current_publisher = {
|
||||
"id": row["publisher_id"],
|
||||
@@ -870,6 +884,14 @@ def create_app() -> Flask:
|
||||
order_sql = f"{sort} {order_dir}, published_at DESC, f.id DESC"
|
||||
|
||||
stats_join = "LEFT JOIN tool_stats ts ON ts.tool_id = f.id"
|
||||
audit_join = """
|
||||
LEFT JOIN tool_audits ta ON ta.id = (
|
||||
SELECT latest_audit.id FROM tool_audits latest_audit
|
||||
WHERE latest_audit.tool_id = f.id
|
||||
ORDER BY latest_audit.evaluated_at DESC, latest_audit.id DESC
|
||||
LIMIT 1
|
||||
)
|
||||
"""
|
||||
|
||||
rows = query_all(
|
||||
g.db,
|
||||
@@ -897,9 +919,13 @@ def create_app() -> Flask:
|
||||
GROUP BY owner, name
|
||||
)
|
||||
SELECT f.*, COALESCE(ts.average_rating, 0) AS average_rating,
|
||||
COALESCE(ts.rating_count, 0) AS rating_count
|
||||
COALESCE(ts.rating_count, 0) AS rating_count,
|
||||
json_extract(ta.quality_json, '$.score') AS quality_score,
|
||||
json_extract(ta.quality_json, '$.evidence_coverage') AS quality_coverage,
|
||||
ta.evaluated_at AS quality_evaluated_at
|
||||
FROM filtered f
|
||||
{stats_join}
|
||||
{audit_join}
|
||||
JOIN (
|
||||
SELECT a.owner, a.name, COALESCE(s.max_id, a.max_id) AS max_id
|
||||
FROM latest_any a
|
||||
@@ -948,6 +974,12 @@ def create_app() -> Flask:
|
||||
"average_rating": row["average_rating"],
|
||||
"rating_count": row["rating_count"],
|
||||
"score": score,
|
||||
"quality_score": row["quality_score"],
|
||||
"quality_coverage": row["quality_coverage"],
|
||||
"quality_evaluated_at": row["quality_evaluated_at"],
|
||||
"deprecated": bool(row["deprecated"]),
|
||||
"deprecated_message": row["deprecated_message"],
|
||||
"replacement": row["replacement"],
|
||||
})
|
||||
|
||||
result: dict = {"data": data, "meta": paginate(page, per_page, total)}
|
||||
@@ -1089,6 +1121,58 @@ def create_app() -> Flask:
|
||||
if fork_row:
|
||||
fork_count = fork_row["cnt"]
|
||||
|
||||
replacement_chain = []
|
||||
if row["replacement"]:
|
||||
try:
|
||||
replacement_chain = _get_replacement_chain(
|
||||
row["owner"], row["name"], row["replacement"]
|
||||
)
|
||||
except ValueError:
|
||||
# Preserve access to legacy rows while refusing new invalid chains.
|
||||
replacement_chain = []
|
||||
|
||||
latest_audit = query_one(
|
||||
g.db,
|
||||
"""
|
||||
SELECT engine_version, evidence_json, quality_json, evaluated_at
|
||||
FROM tool_audits
|
||||
WHERE tool_id = ?
|
||||
ORDER BY evaluated_at DESC, id DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
[row["id"]],
|
||||
)
|
||||
audit_stale = True
|
||||
if latest_audit:
|
||||
try:
|
||||
evaluated = datetime.fromisoformat(
|
||||
latest_audit["evaluated_at"].replace("Z", "+00:00")
|
||||
)
|
||||
if evaluated.tzinfo is None:
|
||||
evaluated = evaluated.replace(tzinfo=timezone.utc)
|
||||
audit_stale = datetime.now(timezone.utc) - evaluated > timedelta(days=30)
|
||||
except (AttributeError, TypeError, ValueError):
|
||||
audit_stale = True
|
||||
|
||||
contributor_rows = query_all(
|
||||
g.db,
|
||||
"""SELECT p.slug, c.contribution FROM tool_contributors c
|
||||
JOIN publishers p ON p.id = c.publisher_id
|
||||
WHERE c.tool_id = ? ORDER BY p.slug""",
|
||||
[row["id"]],
|
||||
)
|
||||
approved_improvement = query_one(
|
||||
g.db,
|
||||
"""SELECT id FROM improvement_submissions
|
||||
WHERE applied_tool_id = ? LIMIT 1""",
|
||||
[row["id"]],
|
||||
)
|
||||
badges = []
|
||||
if row.get("attestation_json"):
|
||||
badges.append("verified")
|
||||
if approved_improvement:
|
||||
badges.extend(["optimized", "community-reviewed"])
|
||||
|
||||
payload = {
|
||||
"owner": row["owner"],
|
||||
"name": row["name"],
|
||||
@@ -1100,7 +1184,8 @@ def create_app() -> Flask:
|
||||
"published_at": row["published_at"],
|
||||
"deprecated": bool(row["deprecated"]),
|
||||
"deprecated_message": row["deprecated_message"],
|
||||
"replacement": row["replacement"],
|
||||
"replacement": row["replacement"] if replacement_chain else None,
|
||||
"replacement_chain": replacement_chain,
|
||||
"config": row["config_yaml"],
|
||||
"readme": row["readme"],
|
||||
"defaults": row.get("defaults") or "",
|
||||
@@ -1108,6 +1193,15 @@ def create_app() -> Flask:
|
||||
"forked_from": row.get("forked_from"),
|
||||
"forked_version": row.get("forked_version"),
|
||||
"fork_count": fork_count,
|
||||
"audit": json.loads(latest_audit["evidence_json"]) if latest_audit else None,
|
||||
"quality": json.loads(latest_audit["quality_json"]) if latest_audit else None,
|
||||
"audit_evaluated_at": latest_audit["evaluated_at"] if latest_audit else None,
|
||||
"audit_stale": audit_stale,
|
||||
"contributors": [
|
||||
{"name": item["slug"], "contribution": item["contribution"]}
|
||||
for item in contributor_rows
|
||||
],
|
||||
"badges": badges,
|
||||
}
|
||||
response = jsonify({"data": payload})
|
||||
response.headers["Cache-Control"] = "max-age=60"
|
||||
@@ -1257,6 +1351,18 @@ def create_app() -> Flask:
|
||||
except Exception:
|
||||
g.db.rollback()
|
||||
|
||||
publisher = query_one(
|
||||
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
|
||||
[row["publisher_id"]],
|
||||
)
|
||||
attestation = None
|
||||
if row.get("attestation_json"):
|
||||
try:
|
||||
attestation = json.loads(row["attestation_json"])
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
return error_response(
|
||||
"INVALID_ATTESTATION", "Stored release attestation is corrupt", 500
|
||||
)
|
||||
response = jsonify({
|
||||
"data": {
|
||||
"owner": row["owner"],
|
||||
@@ -1265,12 +1371,172 @@ def create_app() -> Flask:
|
||||
"config": row["config_yaml"],
|
||||
"readme": row["readme"] or "",
|
||||
"config_hash": row.get("config_hash") or "",
|
||||
"content_hash": row.get("content_hash") or "",
|
||||
"dependency_hashes": json.loads(
|
||||
row.get("dependency_hashes_json") or "{}"
|
||||
),
|
||||
"defaults": row.get("defaults") or "",
|
||||
"attestation": attestation,
|
||||
"signing_public_key": (
|
||||
publisher["signing_public_key"] if publisher else ""
|
||||
) or "",
|
||||
}
|
||||
})
|
||||
response.headers["Cache-Control"] = "max-age=3600, immutable"
|
||||
return response
|
||||
|
||||
@app.route("/api/v1/tools/by-content-hash/<path:content_hash>", methods=["GET"])
|
||||
def tool_by_content_hash(content_hash: str) -> Response:
|
||||
"""Resolve an approved public release by its transitive identity."""
|
||||
if not re.fullmatch(r"sha256:[0-9a-f]{64}", content_hash):
|
||||
return error_response("VALIDATION_ERROR", "Invalid content hash", 400)
|
||||
row = query_one(
|
||||
g.db,
|
||||
"""SELECT owner, name, version, description, category, tags,
|
||||
downloads, published_at
|
||||
FROM tools WHERE content_hash = ? AND visibility = 'public'
|
||||
AND moderation_status = 'approved'
|
||||
ORDER BY published_at DESC LIMIT 1""",
|
||||
[content_hash],
|
||||
)
|
||||
if not row:
|
||||
return error_response("TOOL_NOT_FOUND", "Content hash not found", 404)
|
||||
result = dict(row)
|
||||
result["tags"] = json.loads(result.get("tags") or "[]")
|
||||
return jsonify({"data": result})
|
||||
|
||||
@app.route(
|
||||
"/api/v1/tools/<owner>/<name>/<version>/improvements", methods=["POST"]
|
||||
)
|
||||
@require_token
|
||||
def submit_improvement(owner: str, name: str, version: str) -> Response:
|
||||
"""Submit and automatically test a prompt/code improvement."""
|
||||
row = query_one(
|
||||
g.db,
|
||||
"""SELECT id, config_yaml, visibility, owner FROM tools
|
||||
WHERE owner = ? AND name = ? AND version = ?
|
||||
AND moderation_status = 'approved'""",
|
||||
[owner, name, version],
|
||||
)
|
||||
if not row:
|
||||
return error_response("TOOL_NOT_FOUND", "Tool release not found", 404)
|
||||
if (
|
||||
row["visibility"] != "public"
|
||||
and g.current_publisher["slug"] != row["owner"]
|
||||
and g.current_publisher.get("role") not in ("moderator", "admin")
|
||||
):
|
||||
return error_response("TOOL_NOT_FOUND", "Tool release not found", 404)
|
||||
payload = request.get_json(silent=True) or {}
|
||||
try:
|
||||
from ..community import create_submission, run_submission_tests
|
||||
from ..tool import Tool
|
||||
tool = Tool.from_dict(yaml.safe_load(row["config_yaml"]) or {})
|
||||
submission = create_submission(
|
||||
tool,
|
||||
int(payload.get("step_index", -1)),
|
||||
str(payload.get("proposed", "")),
|
||||
g.current_publisher["slug"],
|
||||
str(payload.get("rationale", "")),
|
||||
)
|
||||
if not submission.proposed.strip():
|
||||
raise ValueError("Proposed content must not be empty")
|
||||
test_result = run_submission_tests(submission, tool)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
return error_response("VALIDATION_ERROR", str(exc), 400)
|
||||
if not test_result.get("passed_for_review"):
|
||||
return error_response(
|
||||
"IMPROVEMENT_TEST_FAILED",
|
||||
test_result.get("error") or "Proposed change did not pass automated tests",
|
||||
400,
|
||||
details={"test_result": test_result},
|
||||
)
|
||||
cursor = g.db.execute(
|
||||
"""INSERT INTO improvement_submissions (
|
||||
tool_id, submitter_id, step_index, step_type, original,
|
||||
proposed, rationale, status, test_result_json, submitted_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, 'tested', ?, ?)""",
|
||||
[
|
||||
row["id"], g.current_publisher["id"], submission.step_index,
|
||||
submission.step_type, submission.original, submission.proposed,
|
||||
submission.rationale, json.dumps(test_result, sort_keys=True),
|
||||
submission.submitted_at,
|
||||
],
|
||||
)
|
||||
g.db.commit()
|
||||
return jsonify({
|
||||
"data": {"id": cursor.lastrowid, **submission.to_dict()}
|
||||
}), 201
|
||||
|
||||
@app.route("/api/v1/improvements/<int:submission_id>", methods=["PATCH"])
|
||||
@require_token
|
||||
def review_improvement(submission_id: int) -> Response:
|
||||
"""Allow the tool owner/moderator to review a tested submission."""
|
||||
row = query_one(
|
||||
g.db,
|
||||
"""SELECT s.*, t.owner FROM improvement_submissions s
|
||||
JOIN tools t ON t.id = s.tool_id WHERE s.id = ?""",
|
||||
[submission_id],
|
||||
)
|
||||
if not row:
|
||||
return error_response("NOT_FOUND", "Improvement not found", 404)
|
||||
if (
|
||||
g.current_publisher["slug"] != row["owner"]
|
||||
and g.current_publisher.get("role") not in ("moderator", "admin")
|
||||
):
|
||||
return error_response("FORBIDDEN", "Owner or moderator required", 403)
|
||||
decision = (request.get_json(silent=True) or {}).get("decision")
|
||||
if decision not in ("approve", "reject", "request_changes"):
|
||||
return error_response("VALIDATION_ERROR", "Invalid decision", 400)
|
||||
status = "approved" if decision == "approve" else decision
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
notes = (request.get_json(silent=True) or {}).get("notes", "")
|
||||
g.db.execute(
|
||||
"""UPDATE improvement_submissions SET status = ?, reviewer_id = ?,
|
||||
review_notes = ?, reviewed_at = ? WHERE id = ?""",
|
||||
[status, g.current_publisher["id"], notes, now, submission_id],
|
||||
)
|
||||
g.db.commit()
|
||||
return jsonify({
|
||||
"data": {
|
||||
"id": submission_id, "status": status,
|
||||
"ready_to_apply": status == "approved",
|
||||
}
|
||||
})
|
||||
|
||||
@app.route("/api/v1/me/signing-key", methods=["PUT"])
|
||||
@require_token
|
||||
def set_signing_key() -> Response:
|
||||
"""Register an Ed25519 public key as the publisher trust anchor."""
|
||||
public_key = (request.get_json(silent=True) or {}).get("public_key", "")
|
||||
try:
|
||||
import base64
|
||||
raw = base64.b64decode(public_key, validate=True)
|
||||
if len(raw) != 32:
|
||||
raise ValueError
|
||||
except (ValueError, TypeError):
|
||||
return error_response(
|
||||
"VALIDATION_ERROR", "public_key must be a base64 Ed25519 key", 400
|
||||
)
|
||||
existing_key = query_one(
|
||||
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
|
||||
[g.current_publisher["id"]],
|
||||
)
|
||||
current_key = (
|
||||
existing_key["signing_public_key"] if existing_key else ""
|
||||
) or ""
|
||||
if current_key and current_key != public_key:
|
||||
return error_response(
|
||||
"SIGNING_KEY_EXISTS",
|
||||
"Signing key rotation requires administrator recovery",
|
||||
409,
|
||||
)
|
||||
g.db.execute(
|
||||
"UPDATE publishers SET signing_public_key = ? WHERE id = ?",
|
||||
[public_key, g.current_publisher["id"]],
|
||||
)
|
||||
g.db.commit()
|
||||
return jsonify({"data": {"signing_public_key": public_key}})
|
||||
|
||||
@app.route("/api/v1/categories", methods=["GET"])
|
||||
def list_categories() -> Response:
|
||||
page, per_page, sort, order, error = parse_pagination("/categories", "name")
|
||||
@@ -2048,7 +2314,7 @@ def create_app() -> Flask:
|
||||
|
||||
# Generate a secure token
|
||||
token, token_hash = generate_token()
|
||||
expires_at = datetime.utcnow() + timedelta(hours=1)
|
||||
expires_at = _naive_utc_now() + timedelta(hours=1)
|
||||
|
||||
g.db.execute(
|
||||
"""
|
||||
@@ -2059,10 +2325,44 @@ def create_app() -> Flask:
|
||||
)
|
||||
g.db.commit()
|
||||
|
||||
# Send email (logs to console in dev mode)
|
||||
# Use a configured canonical URL in production. Falling back to the
|
||||
# request URL is development-only because Host headers are untrusted.
|
||||
from cmdforge.web.email import send_password_reset_email
|
||||
base_url = request.url_root.rstrip("/")
|
||||
send_password_reset_email(publisher["email"], token, base_url)
|
||||
base_url = str(current_app.config.get("PUBLIC_BASE_URL", "")).rstrip("/")
|
||||
environment = str(current_app.config.get("CMDFORGE_ENV", "development")).lower()
|
||||
if not base_url and environment != "production":
|
||||
base_url = request.url_root.rstrip("/")
|
||||
|
||||
parsed_base = urlsplit(base_url)
|
||||
base_url_is_safe = bool(
|
||||
base_url
|
||||
and parsed_base.scheme in {"http", "https"}
|
||||
and parsed_base.netloc
|
||||
and parsed_base.username is None
|
||||
and parsed_base.password is None
|
||||
and not parsed_base.query
|
||||
and not parsed_base.fragment
|
||||
and (environment != "production" or parsed_base.scheme == "https")
|
||||
)
|
||||
|
||||
try:
|
||||
delivered = base_url_is_safe and send_password_reset_email(
|
||||
publisher["email"], token, base_url
|
||||
)
|
||||
except Exception:
|
||||
# Preserve the non-enumerating response while ensuring the token is
|
||||
# invalidated below. Flask logs the exception without email/token
|
||||
# values because they are not part of this message.
|
||||
current_app.logger.exception("Password reset delivery raised unexpectedly")
|
||||
delivered = False
|
||||
if not delivered:
|
||||
# Never leave a usable reset token in the database when its only
|
||||
# intended recipient could not receive it.
|
||||
g.db.execute(
|
||||
"UPDATE password_reset_tokens SET used_at = CURRENT_TIMESTAMP WHERE token_hash = ?",
|
||||
[token_hash],
|
||||
)
|
||||
g.db.commit()
|
||||
|
||||
return success_response
|
||||
|
||||
@@ -2094,7 +2394,7 @@ def create_app() -> Flask:
|
||||
|
||||
try:
|
||||
expires_at = datetime.fromisoformat(row["expires_at"])
|
||||
if datetime.utcnow() > expires_at:
|
||||
if _naive_utc_now() > expires_at:
|
||||
return error_response("TOKEN_EXPIRED", "This reset token has expired", 400)
|
||||
except ValueError:
|
||||
return error_response("INVALID_TOKEN", "Invalid token data", 400)
|
||||
@@ -2136,7 +2436,7 @@ def create_app() -> Flask:
|
||||
|
||||
try:
|
||||
expires_at = datetime.fromisoformat(row["expires_at"])
|
||||
if datetime.utcnow() > expires_at:
|
||||
if _naive_utc_now() > expires_at:
|
||||
return error_response("TOKEN_EXPIRED", "This reset token has expired", 400)
|
||||
except ValueError:
|
||||
return error_response("INVALID_TOKEN", "Invalid token data", 400)
|
||||
@@ -2414,6 +2714,8 @@ def create_app() -> Flask:
|
||||
readme = payload.get("readme") or ""
|
||||
defaults = payload.get("defaults") or ""
|
||||
dry_run = bool(payload.get("dry_run"))
|
||||
attestation_data = payload.get("attestation")
|
||||
improvement_id = payload.get("improvement_id")
|
||||
|
||||
size_resp = validate_payload_size("config", config_text, MAX_CONFIG_BYTES)
|
||||
if size_resp:
|
||||
@@ -2436,6 +2738,10 @@ def create_app() -> Flask:
|
||||
data = yaml.safe_load(config_text) or {}
|
||||
except yaml.YAMLError:
|
||||
return error_response("VALIDATION_ERROR", "Invalid YAML in config")
|
||||
if not isinstance(data, dict):
|
||||
return error_response(
|
||||
"VALIDATION_ERROR", "Tool config must contain a YAML mapping"
|
||||
)
|
||||
|
||||
name = (data.get("name") or "").strip()
|
||||
version = (data.get("version") or "").strip()
|
||||
@@ -2485,6 +2791,30 @@ def create_app() -> Flask:
|
||||
if len(str(tag)) > MAX_TAG_LEN:
|
||||
return error_response("VALIDATION_ERROR", "Tag exceeds 32 characters")
|
||||
|
||||
# Run the same deterministic contract/configuration checks used by the
|
||||
# CLI and builder. Registry hosts intentionally skip local-install
|
||||
# dependency checks because publisher dependencies are resolved by name.
|
||||
try:
|
||||
from ..preflight import analyze_tool as run_preflight
|
||||
from ..tool import Tool
|
||||
|
||||
published_tool = Tool.from_dict(data)
|
||||
preflight_report = run_preflight(
|
||||
published_tool,
|
||||
check_local_dependencies=False,
|
||||
include_contract_tests=True,
|
||||
check_reuse=False,
|
||||
)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
return error_response("INVALID_CONFIG", str(exc), 400)
|
||||
if not preflight_report.ok:
|
||||
return error_response(
|
||||
"PREFLIGHT_FAILED",
|
||||
preflight_report.errors[0],
|
||||
400,
|
||||
details={"preflight": preflight_report.to_dict()},
|
||||
)
|
||||
|
||||
# Determine owner - admins can publish as "official" or other owners
|
||||
owner = g.current_publisher["slug"]
|
||||
requested_owner = payload.get("owner", "").strip()
|
||||
@@ -2514,12 +2844,104 @@ def create_app() -> Flask:
|
||||
)
|
||||
owner = requested_owner
|
||||
|
||||
from .auditing import registry_dependency_findings
|
||||
dependency_findings = registry_dependency_findings(
|
||||
g.db, published_tool, owner
|
||||
)
|
||||
missing_dependencies = dependency_findings["missing"]
|
||||
preflight_report.warnings.extend(dependency_findings["deprecated"])
|
||||
from ..preflight import _add_audit_evidence
|
||||
audit_checks = list(
|
||||
(preflight_report.audit_evidence or {}).get("checks_run", [])
|
||||
)
|
||||
_add_audit_evidence(
|
||||
published_tool, preflight_report,
|
||||
audit_checks + ["registry_dependencies"],
|
||||
)
|
||||
from ..quality import compute_quality
|
||||
quality_report = compute_quality(
|
||||
published_tool, preflight_report
|
||||
).to_dict()
|
||||
if missing_dependencies:
|
||||
message = "Missing registry dependencies: " + ", ".join(
|
||||
missing_dependencies
|
||||
)
|
||||
preflight_report.errors.append(message)
|
||||
_add_audit_evidence(
|
||||
published_tool, preflight_report,
|
||||
audit_checks + ["registry_dependencies"],
|
||||
)
|
||||
return error_response(
|
||||
"PREFLIGHT_FAILED", message, 400,
|
||||
details={"preflight": preflight_report.to_dict()},
|
||||
)
|
||||
|
||||
if data.get("deprecated") and data.get("replacement"):
|
||||
try:
|
||||
_get_replacement_chain(owner, name, data["replacement"])
|
||||
except ValueError as exc:
|
||||
return error_response("VALIDATION_ERROR", str(exc), 400)
|
||||
|
||||
# Compute config hash early for idempotency check
|
||||
config_hash = compute_yaml_hash(config_text)
|
||||
from ..integrity import compute_content_identity
|
||||
from ..tool import ToolStep
|
||||
dependency_names = set(published_tool.dependencies)
|
||||
dependency_names.update(
|
||||
step.tool for step in published_tool.steps if isinstance(step, ToolStep)
|
||||
)
|
||||
dependency_hashes = {}
|
||||
for dependency_name in sorted(dependency_names):
|
||||
from ..resolver import ToolSpec
|
||||
dep_spec = ToolSpec.parse(dependency_name)
|
||||
dep_owner = dep_spec.owner or owner
|
||||
dep_row = resolve_tool(dep_owner, dep_spec.name, dep_spec.version)
|
||||
if dep_row:
|
||||
dep_identity = dep_row.get("content_hash") or dep_row.get("config_hash")
|
||||
if dep_identity:
|
||||
dependency_hashes[dependency_name] = dep_identity
|
||||
content_hash = compute_content_identity(config_hash, dependency_hashes)
|
||||
|
||||
publisher_record = query_one(
|
||||
g.db, "SELECT signing_public_key FROM publishers WHERE id = ?",
|
||||
[g.current_publisher["id"]],
|
||||
)
|
||||
signing_public_key = (
|
||||
publisher_record["signing_public_key"] if publisher_record else ""
|
||||
) or ""
|
||||
attestation_json = None
|
||||
if signing_public_key and not dry_run and not attestation_data:
|
||||
return error_response(
|
||||
"ATTESTATION_REQUIRED",
|
||||
"This publisher has a signing key; releases must be signed",
|
||||
400,
|
||||
)
|
||||
if attestation_data:
|
||||
try:
|
||||
from ..attestation import Attestation, verify_attestation
|
||||
attestation = Attestation.from_dict(attestation_data)
|
||||
except (KeyError, TypeError, ValueError):
|
||||
return error_response(
|
||||
"INVALID_ATTESTATION", "Malformed release attestation", 400
|
||||
)
|
||||
if (
|
||||
attestation.tool_name != name
|
||||
or attestation.version != version
|
||||
or attestation.content_hash != content_hash
|
||||
or attestation.signer != owner
|
||||
or not signing_public_key
|
||||
or not verify_attestation(attestation, signing_public_key)
|
||||
):
|
||||
return error_response(
|
||||
"INVALID_ATTESTATION",
|
||||
"Release attestation does not match this publisher and content",
|
||||
400,
|
||||
)
|
||||
attestation_json = json.dumps(attestation.to_dict(), sort_keys=True)
|
||||
|
||||
existing = query_one(
|
||||
g.db,
|
||||
"SELECT published_at, config_hash, moderation_status, visibility FROM tools WHERE owner = ? AND name = ? AND version = ?",
|
||||
"SELECT published_at, config_hash, content_hash, moderation_status, visibility FROM tools WHERE owner = ? AND name = ? AND version = ?",
|
||||
[owner, name, version],
|
||||
)
|
||||
if existing:
|
||||
@@ -2531,6 +2953,7 @@ def create_app() -> Flask:
|
||||
"name": name,
|
||||
"version": version,
|
||||
"config_hash": config_hash,
|
||||
"content_hash": existing.get("content_hash") or content_hash,
|
||||
"pr_url": "",
|
||||
"status": existing["moderation_status"],
|
||||
"visibility": existing["visibility"],
|
||||
@@ -2545,6 +2968,43 @@ def create_app() -> Flask:
|
||||
details={"published_at": existing["published_at"]},
|
||||
)
|
||||
|
||||
applied_improvement = None
|
||||
if improvement_id is not None:
|
||||
try:
|
||||
improvement_id = int(improvement_id)
|
||||
except (TypeError, ValueError):
|
||||
return error_response("VALIDATION_ERROR", "Invalid improvement_id", 400)
|
||||
applied_improvement = query_one(
|
||||
g.db,
|
||||
"""SELECT s.*, t.owner, t.name FROM improvement_submissions s
|
||||
JOIN tools t ON t.id = s.tool_id
|
||||
WHERE s.id = ? AND s.status = 'approved'
|
||||
AND s.applied_tool_id IS NULL""",
|
||||
[improvement_id],
|
||||
)
|
||||
if (
|
||||
not applied_improvement
|
||||
or applied_improvement["owner"] != owner
|
||||
or applied_improvement["name"] != name
|
||||
or applied_improvement["step_index"] >= len(published_tool.steps)
|
||||
):
|
||||
return error_response(
|
||||
"VALIDATION_ERROR", "Improvement cannot be applied to this release", 400
|
||||
)
|
||||
candidate_step = published_tool.steps[applied_improvement["step_index"]]
|
||||
candidate_value = (
|
||||
candidate_step.prompt if applied_improvement["step_type"] == "prompt"
|
||||
and hasattr(candidate_step, "prompt") else
|
||||
candidate_step.code if applied_improvement["step_type"] == "code"
|
||||
and hasattr(candidate_step, "code") else None
|
||||
)
|
||||
if candidate_value != applied_improvement["proposed"]:
|
||||
return error_response(
|
||||
"VALIDATION_ERROR",
|
||||
"Published config does not contain the approved improvement",
|
||||
400,
|
||||
)
|
||||
|
||||
suggestions = {"category": None, "similar_tools": []}
|
||||
try:
|
||||
from .categorize import suggest_categories
|
||||
@@ -2592,6 +3052,10 @@ def create_app() -> Flask:
|
||||
# Check scrutiny decision
|
||||
if scrutiny_report:
|
||||
suggestions["scrutiny"] = scrutiny_report
|
||||
from ..improvement import generate_improvements
|
||||
suggestions["improvements"] = generate_improvements(
|
||||
published_tool, scrutiny_report
|
||||
).to_dict()
|
||||
if scrutiny_report.get("decision") == "reject":
|
||||
# Find the failing check for error message
|
||||
fail_findings = [f for f in scrutiny_report.get("findings", []) if f.get("result") == "fail"]
|
||||
@@ -2625,6 +3089,9 @@ def create_app() -> Flask:
|
||||
"version": version,
|
||||
"status": "validated",
|
||||
"suggestions": suggestions,
|
||||
"preflight": preflight_report.to_dict(),
|
||||
"quality": quality_report,
|
||||
"content_hash": content_hash,
|
||||
}
|
||||
})
|
||||
|
||||
@@ -2650,15 +3117,16 @@ def create_app() -> Flask:
|
||||
# Public tools need moderation approval
|
||||
moderation_status = "pending"
|
||||
|
||||
g.db.execute(
|
||||
insert_cursor = g.db.execute(
|
||||
"""
|
||||
INSERT INTO tools (
|
||||
owner, name, version, description, category, tags, config_yaml, readme,
|
||||
defaults, publisher_id, deprecated, deprecated_message, replacement, downloads,
|
||||
scrutiny_status, scrutiny_report, source, source_url, source_json,
|
||||
config_hash, visibility, moderation_status, forked_from, forked_version,
|
||||
published_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
config_hash, content_hash, dependency_hashes_json, visibility,
|
||||
moderation_status, forked_from, forked_version, published_at,
|
||||
attestation_json
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
owner,
|
||||
@@ -2681,11 +3149,43 @@ def create_app() -> Flask:
|
||||
source_url,
|
||||
source_json,
|
||||
config_hash,
|
||||
content_hash,
|
||||
json.dumps(dependency_hashes, sort_keys=True),
|
||||
visibility,
|
||||
moderation_status,
|
||||
forked_from,
|
||||
forked_version,
|
||||
datetime.utcnow().isoformat(),
|
||||
datetime.now(timezone.utc).isoformat(),
|
||||
attestation_json,
|
||||
],
|
||||
)
|
||||
tool_id = insert_cursor.lastrowid
|
||||
if applied_improvement:
|
||||
g.db.execute(
|
||||
"UPDATE improvement_submissions SET applied_tool_id = ? WHERE id = ?",
|
||||
[tool_id, applied_improvement["id"]],
|
||||
)
|
||||
g.db.execute(
|
||||
"""INSERT OR IGNORE INTO tool_contributors
|
||||
(tool_id, publisher_id, contribution)
|
||||
VALUES (?, ?, 'improvement')""",
|
||||
[tool_id, applied_improvement["submitter_id"]],
|
||||
)
|
||||
audit = preflight_report.audit_evidence or {}
|
||||
g.db.execute(
|
||||
"""
|
||||
INSERT INTO tool_audits (
|
||||
tool_id, engine_version, trigger, evidence_json, quality_json,
|
||||
findings_hash, evaluated_at
|
||||
) VALUES (?, ?, 'publish', ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
tool_id,
|
||||
audit.get("engine_version", "1.0"),
|
||||
json.dumps(audit, sort_keys=True),
|
||||
json.dumps(quality_report, sort_keys=True),
|
||||
audit.get("findings_hash", ""),
|
||||
audit.get("timestamp") or datetime.now(timezone.utc).isoformat(),
|
||||
],
|
||||
)
|
||||
g.db.commit()
|
||||
@@ -2724,12 +3224,15 @@ def create_app() -> Flask:
|
||||
"name": name,
|
||||
"version": version,
|
||||
"config_hash": config_hash,
|
||||
"content_hash": content_hash,
|
||||
"pr_url": "",
|
||||
"status": moderation_status,
|
||||
"visibility": visibility,
|
||||
"forked_from": forked_from,
|
||||
"forked_version": forked_version,
|
||||
"suggestions": suggestions,
|
||||
"preflight": preflight_report.to_dict(),
|
||||
"quality": quality_report,
|
||||
}
|
||||
})
|
||||
response.status_code = 201
|
||||
@@ -2844,6 +3347,47 @@ def create_app() -> Flask:
|
||||
|
||||
return jsonify({"data": results})
|
||||
|
||||
def _get_replacement_chain(
|
||||
original_owner: str, original_name: str, replacement: str
|
||||
) -> List[str]:
|
||||
"""Validate and resolve a deprecation chain without allowing cycles."""
|
||||
chain: List[str] = []
|
||||
visited = {(original_owner, original_name)}
|
||||
current = replacement
|
||||
default_owner = original_owner
|
||||
for _ in range(20):
|
||||
if "/" in current:
|
||||
next_owner, next_name = current.split("/", 1)
|
||||
else:
|
||||
next_owner, next_name = default_owner, current
|
||||
if not OWNER_RE.fullmatch(next_owner) or not TOOL_NAME_RE.fullmatch(next_name):
|
||||
raise ValueError(
|
||||
"replacement must be a valid tool name or owner/tool reference"
|
||||
)
|
||||
identity = (next_owner, next_name)
|
||||
if identity in visited:
|
||||
raise ValueError("replacement creates a deprecation cycle")
|
||||
row = query_one(
|
||||
g.db,
|
||||
"""
|
||||
SELECT deprecated, replacement FROM tools
|
||||
WHERE owner = ? AND name = ?
|
||||
ORDER BY id DESC LIMIT 1
|
||||
""",
|
||||
[next_owner, next_name],
|
||||
)
|
||||
if row is None:
|
||||
raise ValueError(
|
||||
f"replacement tool '{next_owner}/{next_name}' does not exist"
|
||||
)
|
||||
visited.add(identity)
|
||||
chain.append(f"{next_owner}/{next_name}")
|
||||
if not row["deprecated"] or not row["replacement"]:
|
||||
return chain
|
||||
current = row["replacement"]
|
||||
default_owner = next_owner
|
||||
raise ValueError("replacement chain exceeds 20 tools")
|
||||
|
||||
@app.route("/api/v1/tools/<owner>/<name>/deprecate", methods=["POST"])
|
||||
@require_token
|
||||
def deprecate_tool(owner: str, name: str) -> Response:
|
||||
@@ -2857,6 +3401,10 @@ def create_app() -> Flask:
|
||||
|
||||
if message and len(message) > 500:
|
||||
return error_response("VALIDATION_ERROR", "Message too long (max 500)", 400)
|
||||
try:
|
||||
chain = _get_replacement_chain(owner, name, replacement) if replacement else []
|
||||
except ValueError as exc:
|
||||
return error_response("VALIDATION_ERROR", str(exc), 400)
|
||||
|
||||
# Update all versions of the tool
|
||||
result = g.db.execute(
|
||||
@@ -2870,7 +3418,10 @@ def create_app() -> Flask:
|
||||
return error_response("TOOL_NOT_FOUND", f"Tool {owner}/{name} not found", 404)
|
||||
g.db.commit()
|
||||
|
||||
return jsonify({"data": {"status": "deprecated", "owner": owner, "name": name}})
|
||||
return jsonify({"data": {
|
||||
"status": "deprecated", "owner": owner, "name": name,
|
||||
"replacement_chain": chain,
|
||||
}})
|
||||
|
||||
@app.route("/api/v1/tools/<owner>/<name>/undeprecate", methods=["POST"])
|
||||
@require_token
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
"""Immutable publish-time and refresh audits for registry tool versions."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any, Dict, List
|
||||
|
||||
import yaml
|
||||
|
||||
from ..preflight import analyze_tool
|
||||
from ..quality import compute_quality
|
||||
from ..tool import Tool
|
||||
from .db import connect_db, get_db_path, init_db
|
||||
|
||||
|
||||
def registry_dependency_findings(conn, tool: Tool, owner: str) -> Dict[str, List[str]]:
|
||||
"""Resolve registry dependencies and report missing/deprecated references."""
|
||||
from ..tool import ToolStep
|
||||
|
||||
references = set(tool.dependencies)
|
||||
references.update(
|
||||
step.tool for step in tool.steps if isinstance(step, ToolStep)
|
||||
)
|
||||
missing: List[str] = []
|
||||
deprecated: List[str] = []
|
||||
for reference in sorted(references):
|
||||
from ..resolver import ToolSpec
|
||||
from ..semver import matches_constraint
|
||||
spec = ToolSpec.parse(reference)
|
||||
if spec.name == tool.name and (spec.owner in (None, owner)):
|
||||
continue
|
||||
if spec.owner:
|
||||
candidates = [(spec.owner, spec.name)]
|
||||
else:
|
||||
candidates = [(owner, spec.name), ("official", spec.name)]
|
||||
row = None
|
||||
for dep_owner, dep_name in candidates:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT version, deprecated, replacement FROM tools
|
||||
WHERE owner = ? AND name = ? ORDER BY id DESC
|
||||
""",
|
||||
[dep_owner, dep_name],
|
||||
).fetchall()
|
||||
row = next(
|
||||
(
|
||||
candidate for candidate in rows
|
||||
if not spec.version
|
||||
or matches_constraint(candidate["version"], spec.version)
|
||||
),
|
||||
None,
|
||||
)
|
||||
if row is not None:
|
||||
break
|
||||
if row is None:
|
||||
missing.append(reference)
|
||||
elif row["deprecated"]:
|
||||
guidance = f"{reference} is deprecated"
|
||||
if row["replacement"]:
|
||||
guidance += f"; use {row['replacement']}"
|
||||
deprecated.append(guidance)
|
||||
return {"missing": missing, "deprecated": deprecated}
|
||||
|
||||
|
||||
def audit_tool_version(conn, tool_id: int, trigger: str = "background") -> Dict[str, Any]:
|
||||
"""Evaluate one immutable tool version and append an audit record."""
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT tools.*, COALESCE(tool_stats.average_rating, 0) AS average_rating,
|
||||
COALESCE(tool_stats.rating_count, 0) AS rating_count
|
||||
FROM tools
|
||||
LEFT JOIN tool_stats ON tool_stats.tool_id = tools.id
|
||||
WHERE tools.id = ?
|
||||
""",
|
||||
[tool_id],
|
||||
).fetchone()
|
||||
if row is None:
|
||||
raise ValueError(f"Unknown tool version id {tool_id}")
|
||||
|
||||
previous = conn.execute(
|
||||
"""
|
||||
SELECT evidence_json, quality_json FROM tool_audits
|
||||
WHERE tool_id = ? ORDER BY evaluated_at DESC, id DESC LIMIT 1
|
||||
""",
|
||||
[tool_id],
|
||||
).fetchone()
|
||||
config = yaml.safe_load(row["config_yaml"])
|
||||
tool = Tool.from_dict(config)
|
||||
report = analyze_tool(
|
||||
tool,
|
||||
check_local_dependencies=False,
|
||||
include_contract_tests=True,
|
||||
check_reuse=False,
|
||||
)
|
||||
dependency_findings = registry_dependency_findings(conn, tool, row["owner"])
|
||||
if dependency_findings["missing"]:
|
||||
report.errors.append(
|
||||
"Missing registry dependencies: "
|
||||
+ ", ".join(dependency_findings["missing"])
|
||||
)
|
||||
report.warnings.extend(dependency_findings["deprecated"])
|
||||
from ..preflight import _add_audit_evidence
|
||||
checks = list((report.audit_evidence or {}).get("checks_run", []))
|
||||
_add_audit_evidence(tool, report, checks + ["registry_dependencies"])
|
||||
registry_data = {
|
||||
"average_rating": row["average_rating"],
|
||||
"rating_count": row["rating_count"],
|
||||
"downloads": row["downloads"],
|
||||
}
|
||||
quality = compute_quality(tool, report, registry_data=registry_data).to_dict()
|
||||
evidence = report.audit_evidence or {}
|
||||
evaluated_at = datetime.now(timezone.utc).isoformat()
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO tool_audits (
|
||||
tool_id, engine_version, trigger, evidence_json, quality_json,
|
||||
findings_hash, evaluated_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
[
|
||||
tool_id,
|
||||
evidence.get("engine_version", "1.0"),
|
||||
trigger,
|
||||
json.dumps(evidence, sort_keys=True),
|
||||
json.dumps(quality, sort_keys=True),
|
||||
evidence.get("findings_hash", ""),
|
||||
evaluated_at,
|
||||
],
|
||||
)
|
||||
quality_change = None
|
||||
if previous:
|
||||
previous_quality = json.loads(previous["quality_json"])
|
||||
quality_change = quality["score"] - previous_quality.get("score", 0)
|
||||
previous_evidence = json.loads(previous["evidence_json"])
|
||||
if quality_change < 0 or previous_evidence.get("outcome") != evidence.get("outcome"):
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO audit_log (
|
||||
action, target_type, target_id, actor_id, details
|
||||
) VALUES ('automated_audit_changed', 'tool', ?, 'system', ?)
|
||||
""",
|
||||
[
|
||||
str(tool_id),
|
||||
json.dumps({
|
||||
"quality_change": quality_change,
|
||||
"previous_outcome": previous_evidence.get("outcome"),
|
||||
"current_outcome": evidence.get("outcome"),
|
||||
}, sort_keys=True),
|
||||
],
|
||||
)
|
||||
conn.commit()
|
||||
return {
|
||||
"tool_id": tool_id, "evidence": evidence, "quality": quality,
|
||||
"quality_change": quality_change,
|
||||
}
|
||||
|
||||
|
||||
def audit_stale_tools(
|
||||
conn, *, max_age_days: int = 30, limit: int = 100
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Append refreshed evidence for versions with missing or stale audits."""
|
||||
if max_age_days < 0:
|
||||
raise ValueError("max_age_days must be non-negative")
|
||||
if limit < 1:
|
||||
raise ValueError("limit must be positive")
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=max_age_days)
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT tools.id, MAX(tool_audits.evaluated_at) AS last_evaluated
|
||||
FROM tools
|
||||
LEFT JOIN tool_audits ON tool_audits.tool_id = tools.id
|
||||
GROUP BY tools.id
|
||||
ORDER BY last_evaluated IS NOT NULL, last_evaluated
|
||||
"""
|
||||
).fetchall()
|
||||
stale_ids = []
|
||||
for row in rows:
|
||||
value = row["last_evaluated"]
|
||||
if value:
|
||||
try:
|
||||
evaluated = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
if evaluated.tzinfo is None:
|
||||
evaluated = evaluated.replace(tzinfo=timezone.utc)
|
||||
except ValueError:
|
||||
evaluated = datetime.min.replace(tzinfo=timezone.utc)
|
||||
if evaluated >= cutoff:
|
||||
continue
|
||||
stale_ids.append(row["id"])
|
||||
if len(stale_ids) >= limit:
|
||||
break
|
||||
results = []
|
||||
for tool_id in stale_ids:
|
||||
try:
|
||||
results.append(audit_tool_version(conn, tool_id))
|
||||
except Exception as exc:
|
||||
results.append({"tool_id": tool_id, "error": str(exc)})
|
||||
return results
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Refresh stale CmdForge registry audits")
|
||||
parser.add_argument("--max-age-days", type=int, default=30)
|
||||
parser.add_argument("--limit", type=int, default=100)
|
||||
args = parser.parse_args()
|
||||
conn = connect_db(get_db_path())
|
||||
try:
|
||||
init_db(conn)
|
||||
results = audit_stale_tools(
|
||||
conn, max_age_days=args.max_age_days, limit=args.limit
|
||||
)
|
||||
finally:
|
||||
conn.close()
|
||||
print(f"Audited {len(results)} tool version(s)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -27,6 +27,7 @@ CREATE TABLE IF NOT EXISTS publishers (
|
||||
ban_reason TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
,signing_public_key TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS api_tokens (
|
||||
@@ -61,6 +62,8 @@ CREATE TABLE IF NOT EXISTS tools (
|
||||
source_url TEXT,
|
||||
source_json TEXT,
|
||||
config_hash TEXT,
|
||||
content_hash TEXT,
|
||||
dependency_hashes_json TEXT,
|
||||
visibility TEXT DEFAULT 'public',
|
||||
moderation_status TEXT DEFAULT 'pending',
|
||||
moderation_note TEXT,
|
||||
@@ -69,6 +72,7 @@ CREATE TABLE IF NOT EXISTS tools (
|
||||
forked_from TEXT,
|
||||
forked_version TEXT,
|
||||
published_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
attestation_json TEXT,
|
||||
UNIQUE(owner, name, version)
|
||||
);
|
||||
|
||||
@@ -80,6 +84,45 @@ CREATE TABLE IF NOT EXISTS download_stats (
|
||||
UNIQUE(tool_id, client_id, downloaded_at)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tool_audits (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
|
||||
engine_version TEXT NOT NULL,
|
||||
trigger TEXT NOT NULL,
|
||||
evidence_json TEXT NOT NULL,
|
||||
quality_json TEXT NOT NULL,
|
||||
findings_hash TEXT NOT NULL,
|
||||
evaluated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_tool_audits_tool_time
|
||||
ON tool_audits(tool_id, evaluated_at DESC, id DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS improvement_submissions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
|
||||
submitter_id INTEGER NOT NULL REFERENCES publishers(id),
|
||||
step_index INTEGER NOT NULL,
|
||||
step_type TEXT NOT NULL,
|
||||
original TEXT NOT NULL,
|
||||
proposed TEXT NOT NULL,
|
||||
rationale TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'tested',
|
||||
test_result_json TEXT NOT NULL,
|
||||
reviewer_id INTEGER REFERENCES publishers(id),
|
||||
review_notes TEXT,
|
||||
submitted_at TIMESTAMP NOT NULL,
|
||||
reviewed_at TIMESTAMP
|
||||
,applied_tool_id INTEGER REFERENCES tools(id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS tool_contributors (
|
||||
tool_id INTEGER NOT NULL REFERENCES tools(id) ON DELETE CASCADE,
|
||||
publisher_id INTEGER NOT NULL REFERENCES publishers(id),
|
||||
contribution TEXT NOT NULL,
|
||||
PRIMARY KEY (tool_id, publisher_id, contribution)
|
||||
);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS tools_fts USING fts5(
|
||||
name, description, tags, readme,
|
||||
content='tools',
|
||||
@@ -475,6 +518,8 @@ def migrate_db(conn: sqlite3.Connection) -> None:
|
||||
("source_url", "TEXT", "NULL"),
|
||||
("source_json", "TEXT", "NULL"),
|
||||
("config_hash", "TEXT", "NULL"),
|
||||
("content_hash", "TEXT", "NULL"),
|
||||
("dependency_hashes_json", "TEXT", "NULL"),
|
||||
("visibility", "TEXT", "'public'"),
|
||||
("moderation_status", "TEXT", "'pending'"),
|
||||
("moderation_note", "TEXT", "NULL"),
|
||||
@@ -483,6 +528,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
|
||||
("forked_from", "TEXT", "NULL"),
|
||||
("forked_version", "TEXT", "NULL"),
|
||||
("defaults", "TEXT", "NULL"),
|
||||
("attestation_json", "TEXT", "NULL"),
|
||||
]
|
||||
|
||||
for col_name, col_type, default in tools_migrations:
|
||||
@@ -500,6 +546,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
|
||||
("banned_at", "TIMESTAMP", "NULL"),
|
||||
("banned_by", "TEXT", "NULL"),
|
||||
("ban_reason", "TEXT", "NULL"),
|
||||
("signing_public_key", "TEXT", "NULL"),
|
||||
]
|
||||
|
||||
for col_name, col_type, default in publishers_migrations:
|
||||
@@ -526,6 +573,18 @@ def migrate_db(conn: sqlite3.Connection) -> None:
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
|
||||
cursor = conn.execute("PRAGMA table_info(improvement_submissions)")
|
||||
improvement_cols = {row[1] for row in cursor.fetchall()}
|
||||
if improvement_cols and "applied_tool_id" not in improvement_cols:
|
||||
try:
|
||||
conn.execute(
|
||||
"ALTER TABLE improvement_submissions "
|
||||
"ADD COLUMN applied_tool_id INTEGER REFERENCES tools(id)"
|
||||
)
|
||||
conn.commit()
|
||||
except sqlite3.OperationalError:
|
||||
pass
|
||||
|
||||
# Grandfather existing tools: set moderation_status to 'approved' for tools that have NULL
|
||||
# This ensures existing tools remain visible after migration (one-time migration)
|
||||
# Note: Only applies to NULL, NOT to 'pending' - pending tools need manual review
|
||||
@@ -559,6 +618,7 @@ def migrate_db(conn: sqlite3.Connection) -> None:
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_owner ON tools(owner)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_moderation ON tools(moderation_status, visibility)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_hash ON tools(config_hash)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_tools_content_hash ON tools(content_hash)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_publishers_role ON publishers(role)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_publishers_banned ON publishers(banned)")
|
||||
conn.execute("CREATE INDEX IF NOT EXISTS idx_audit_log_target ON audit_log(target_type, target_id)")
|
||||
|
||||
@@ -157,10 +157,19 @@ def semantic_search(
|
||||
"""
|
||||
SELECT te.tool_id, te.embedding, te.dimensions, te.model,
|
||||
t.owner, t.name, t.version, t.description, t.category,
|
||||
t.tags, t.downloads
|
||||
t.tags, t.downloads,
|
||||
json_extract(ta.quality_json, '$.score') AS quality_score,
|
||||
json_extract(ta.quality_json, '$.evidence_coverage') AS quality_coverage
|
||||
FROM tool_embeddings te
|
||||
JOIN tools t ON t.id = te.tool_id
|
||||
LEFT JOIN tool_audits ta ON ta.id = (
|
||||
SELECT latest_audit.id FROM tool_audits latest_audit
|
||||
WHERE latest_audit.tool_id = t.id
|
||||
ORDER BY latest_audit.evaluated_at DESC, latest_audit.id DESC
|
||||
LIMIT 1
|
||||
)
|
||||
WHERE t.visibility = 'public' AND t.moderation_status = 'approved'
|
||||
AND t.deprecated = 0
|
||||
""",
|
||||
).fetchall()
|
||||
|
||||
@@ -192,6 +201,8 @@ def semantic_search(
|
||||
"tags": tags_list,
|
||||
"downloads": row["downloads"] or 0,
|
||||
"similarity": round(score, 4),
|
||||
"quality_score": row["quality_score"],
|
||||
"quality_coverage": row["quality_coverage"],
|
||||
})
|
||||
|
||||
# Sort by similarity descending
|
||||
|
||||
@@ -116,6 +116,10 @@ class DownloadResult:
|
||||
readme: str = ""
|
||||
config_hash: str = "" # Registry hash for integrity verification
|
||||
defaults: str = "" # Default settings YAML content
|
||||
attestation: Optional[Dict[str, Any]] = None
|
||||
signing_public_key: str = ""
|
||||
content_hash: str = ""
|
||||
dependency_hashes: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
class RegistryClient:
|
||||
@@ -562,8 +566,56 @@ class RegistryClient:
|
||||
readme=data.get("readme", ""),
|
||||
config_hash=data.get("config_hash", ""),
|
||||
defaults=data.get("defaults", "")
|
||||
,attestation=data.get("attestation")
|
||||
,signing_public_key=data.get("signing_public_key", "")
|
||||
,content_hash=data.get("content_hash", "")
|
||||
,dependency_hashes=data.get("dependency_hashes", {})
|
||||
)
|
||||
|
||||
def get_tool_by_content_hash(self, content_hash: str) -> ToolInfo:
|
||||
response = self._request("GET", f"/tools/by-content-hash/{content_hash}")
|
||||
if response.status_code != 200:
|
||||
self._handle_error_response(response)
|
||||
return ToolInfo.from_dict(response.json().get("data", {}))
|
||||
|
||||
def set_signing_public_key(self, public_key: str) -> Dict[str, Any]:
|
||||
"""Register the public key used to verify this publisher's releases."""
|
||||
response = self._request(
|
||||
"PUT", "/me/signing-key",
|
||||
json_data={"public_key": public_key}, require_auth=True,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
self._handle_error_response(response)
|
||||
return response.json().get("data", {})
|
||||
|
||||
def submit_improvement(
|
||||
self, owner: str, name: str, version: str, step_index: int,
|
||||
proposed: str, rationale: str = "",
|
||||
) -> Dict[str, Any]:
|
||||
response = self._request(
|
||||
"POST", f"/tools/{owner}/{name}/{version}/improvements",
|
||||
json_data={
|
||||
"step_index": step_index, "proposed": proposed,
|
||||
"rationale": rationale,
|
||||
},
|
||||
require_auth=True,
|
||||
)
|
||||
if response.status_code != 201:
|
||||
self._handle_error_response(response)
|
||||
return response.json().get("data", {})
|
||||
|
||||
def review_improvement(
|
||||
self, submission_id: int, decision: str, notes: str = ""
|
||||
) -> Dict[str, Any]:
|
||||
response = self._request(
|
||||
"PATCH", f"/improvements/{submission_id}",
|
||||
json_data={"decision": decision, "notes": notes},
|
||||
require_auth=True,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
self._handle_error_response(response)
|
||||
return response.json().get("data", {})
|
||||
|
||||
def get_categories(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get list of tool categories.
|
||||
@@ -623,7 +675,9 @@ class RegistryClient:
|
||||
defaults: str = "",
|
||||
dry_run: bool = False,
|
||||
visibility: str = "public",
|
||||
owner: str = ""
|
||||
owner: str = "",
|
||||
attestation: Optional[Dict[str, Any]] = None,
|
||||
improvement_id: Optional[int] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Publish a tool to the registry.
|
||||
@@ -649,6 +703,10 @@ class RegistryClient:
|
||||
payload["defaults"] = defaults
|
||||
if owner:
|
||||
payload["owner"] = owner
|
||||
if attestation:
|
||||
payload["attestation"] = attestation
|
||||
if improvement_id is not None:
|
||||
payload["improvement_id"] = improvement_id
|
||||
|
||||
response = self._request(
|
||||
"POST",
|
||||
|
||||
@@ -16,7 +16,7 @@ from typing import Optional, Tuple
|
||||
|
||||
import yaml
|
||||
|
||||
from .tool import Tool, TOOLS_DIR, get_bin_dir, BIN_DIR
|
||||
from .tool import Tool, TOOLS_DIR, get_tools_dir, get_bin_dir, BIN_DIR
|
||||
from .config import is_auto_fetch_enabled, load_config
|
||||
from .manifest import load_manifest
|
||||
|
||||
@@ -243,7 +243,7 @@ class ToolResolver:
|
||||
searched_paths: list
|
||||
) -> Optional[ResolvedTool]:
|
||||
"""Search for tool in global user directory."""
|
||||
global_dir = TOOLS_DIR
|
||||
global_dir = get_tools_dir()
|
||||
|
||||
if not global_dir.exists():
|
||||
return None
|
||||
@@ -354,7 +354,10 @@ class ToolResolver:
|
||||
config_yaml=result.config_yaml,
|
||||
readme=result.readme,
|
||||
config_hash=result.config_hash,
|
||||
defaults=result.defaults
|
||||
defaults=result.defaults,
|
||||
attestation=result.attestation,
|
||||
signing_public_key=result.signing_public_key,
|
||||
content_hash=result.content_hash,
|
||||
)
|
||||
|
||||
if self.verbose:
|
||||
@@ -383,7 +386,10 @@ class ToolResolver:
|
||||
config_yaml: str,
|
||||
readme: str = "",
|
||||
config_hash: str = "",
|
||||
defaults: str = ""
|
||||
defaults: str = "",
|
||||
attestation: Optional[dict] = None,
|
||||
signing_public_key: str = "",
|
||||
content_hash: str = "",
|
||||
) -> ResolvedTool:
|
||||
"""Install a tool fetched from registry to global directory."""
|
||||
# Verify hash if provided
|
||||
@@ -396,6 +402,24 @@ class ToolResolver:
|
||||
f"got {computed_hash[:20]}... - content may have been tampered with"
|
||||
)
|
||||
|
||||
if attestation:
|
||||
from .attestation import Attestation, verify_attestation
|
||||
try:
|
||||
signed = Attestation.from_dict(attestation)
|
||||
except (KeyError, TypeError, ValueError) as exc:
|
||||
raise RuntimeError("Malformed release attestation") from exc
|
||||
if (
|
||||
not signing_public_key
|
||||
or signed.tool_name != name
|
||||
or signed.version != version
|
||||
or signed.content_hash != content_hash
|
||||
or signed.signer != owner
|
||||
or not verify_attestation(signed, signing_public_key)
|
||||
):
|
||||
raise RuntimeError(
|
||||
f"Invalid release attestation for {owner}/{name}@{version}"
|
||||
)
|
||||
|
||||
# Create directory structure
|
||||
tool_dir = TOOLS_DIR / owner / name
|
||||
tool_dir.mkdir(parents=True, exist_ok=True)
|
||||
@@ -628,7 +652,10 @@ def install_from_registry(spec: str, version: Optional[str] = None) -> ResolvedT
|
||||
config_yaml=result.config_yaml,
|
||||
readme=result.readme,
|
||||
config_hash=result.config_hash,
|
||||
defaults=result.defaults
|
||||
defaults=result.defaults,
|
||||
attestation=result.attestation,
|
||||
signing_public_key=result.signing_public_key,
|
||||
content_hash=result.content_hash,
|
||||
)
|
||||
|
||||
|
||||
|
||||
+372
-29
@@ -1,19 +1,24 @@
|
||||
"""Tool execution engine."""
|
||||
|
||||
import argparse
|
||||
import fnmatch
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from typing import Optional, List
|
||||
|
||||
import yaml
|
||||
|
||||
from .tool import Tool, PromptStep, CodeStep, ToolStep
|
||||
from .providers import call_provider, mock_provider
|
||||
from .tool import Tool, PromptStep, CodeStep, ToolStep, McpStep
|
||||
from .providers import (
|
||||
call_provider, mock_provider, ProviderExecutionPolicy,
|
||||
)
|
||||
from .resolver import resolve_tool, ToolNotFoundError, ToolSpec, install_from_registry
|
||||
from .manifest import load_manifest
|
||||
from .profiles import load_profile
|
||||
from .mcp_client import McpClientManager
|
||||
|
||||
# Maximum recursion depth for nested tool calls
|
||||
MAX_TOOL_DEPTH = 10
|
||||
@@ -352,7 +357,9 @@ def substitute_variables(template: str, variables: dict, warn_non_scalar: bool =
|
||||
if base_name == "settings":
|
||||
# Settings dict - leave unchanged (handled specially)
|
||||
return match.group(0)
|
||||
# Replace with empty string for None, str() for other values
|
||||
if isinstance(base_value, (dict, list)):
|
||||
return json.dumps(base_value)
|
||||
# Replace with empty string for None, str() for other scalar values
|
||||
return "" if base_value is None else str(base_value)
|
||||
|
||||
# Handle nested access
|
||||
@@ -503,7 +510,9 @@ def execute_prompt_step(
|
||||
variables: dict,
|
||||
provider_override: str = None,
|
||||
verbose: bool = False,
|
||||
base_dir: Optional[Path] = None
|
||||
base_dir: Optional[Path] = None,
|
||||
execution_policy: Optional[ProviderExecutionPolicy] = None,
|
||||
provenance_log: Optional[list[dict]] = None,
|
||||
) -> tuple[str, bool]:
|
||||
"""
|
||||
Execute a prompt step.
|
||||
@@ -520,6 +529,24 @@ def execute_prompt_step(
|
||||
Otherwise, enforces structured JSON output with schema validation.
|
||||
"""
|
||||
import re
|
||||
from dataclasses import replace
|
||||
|
||||
effective_policy = execution_policy or ProviderExecutionPolicy()
|
||||
if step.fallback_policy == "deny" and effective_policy.allow_fallback:
|
||||
effective_policy = replace(effective_policy, allow_fallback=False)
|
||||
|
||||
def record_result(result, selected_provider: str) -> None:
|
||||
# Mock calls bypass call_provider for deterministic tests, so attach
|
||||
# the same runtime-owned identity here.
|
||||
if selected_provider.lower() == "mock" and not result.actual_provider:
|
||||
result.requested_provider = "mock"
|
||||
result.actual_provider = "mock"
|
||||
result.attempted_providers = ["mock"]
|
||||
result.model = "mock"
|
||||
result.locality = "local"
|
||||
result.model_identity_source = "built-in"
|
||||
if provenance_log is not None:
|
||||
provenance_log.append(result.provenance())
|
||||
|
||||
# Build prompt with variable substitution
|
||||
try:
|
||||
@@ -530,22 +557,34 @@ def execute_prompt_step(
|
||||
|
||||
prompt = substitute_variables(prompt_template, variables, warn_non_scalar=verbose)
|
||||
|
||||
# Inject profile system prompt if specified
|
||||
# Determine provider
|
||||
provider = provider_override or substitute_variables(step.provider, variables, warn_non_scalar=verbose)
|
||||
|
||||
# Build context from the inside out so the final order is:
|
||||
# profile system prompt -> skills -> user prompt.
|
||||
if step.skills is not None:
|
||||
from .skills import inject_skills
|
||||
skill_names = None if "*" in step.skills else step.skills
|
||||
prompt = inject_skills(prompt, provider, skill_names)
|
||||
|
||||
if step.profile:
|
||||
profile = load_profile(step.profile)
|
||||
if profile and profile.system_prompt:
|
||||
# Prepend system prompt to user prompt
|
||||
prompt = f"{profile.system_prompt}\n\n---\n\n{prompt}"
|
||||
|
||||
# Determine provider
|
||||
provider = provider_override or substitute_variables(step.provider, variables, warn_non_scalar=verbose)
|
||||
# Structured or plain-text
|
||||
|
||||
# Plain text mode - bypass structured output
|
||||
if step.plain_text:
|
||||
if provider.lower() == "mock":
|
||||
result = mock_provider(prompt)
|
||||
else:
|
||||
result = call_provider(provider, prompt, max_tokens=step.max_tokens)
|
||||
result = call_provider(
|
||||
provider, prompt, max_tokens=step.max_tokens,
|
||||
execution_policy=effective_policy,
|
||||
)
|
||||
|
||||
record_result(result, provider)
|
||||
|
||||
if not result.success:
|
||||
print(f"Error in prompt step: {result.error}", file=sys.stderr)
|
||||
@@ -589,7 +628,12 @@ Please try again with valid JSON matching the schema exactly."""
|
||||
if provider.lower() == "mock":
|
||||
result = mock_provider(current_prompt)
|
||||
else:
|
||||
result = call_provider(provider, current_prompt, max_tokens=step.max_tokens)
|
||||
result = call_provider(
|
||||
provider, current_prompt, max_tokens=step.max_tokens,
|
||||
execution_policy=effective_policy,
|
||||
)
|
||||
|
||||
record_result(result, provider)
|
||||
|
||||
if not result.success:
|
||||
print(f"Error in prompt step: {result.error}", file=sys.stderr)
|
||||
@@ -727,7 +771,12 @@ def execute_tool_step(
|
||||
provider_override: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
verbose: bool = False,
|
||||
call_stack: Optional[list] = None
|
||||
call_stack: Optional[list] = None,
|
||||
agent_profile: Optional[str] = None,
|
||||
agent_skills: Optional[List[str]] = None,
|
||||
agent_tool_policies: Optional[List[List[str]]] = None,
|
||||
execution_policy: Optional[ProviderExecutionPolicy] = None,
|
||||
provenance_log: Optional[list[dict]] = None,
|
||||
) -> tuple[str, bool]:
|
||||
"""
|
||||
Execute a tool step by calling another tool.
|
||||
@@ -751,7 +800,15 @@ def execute_tool_step(
|
||||
_print_call_stack(call_stack, f"Maximum tool nesting depth ({MAX_TOOL_DEPTH}) exceeded")
|
||||
return "", False
|
||||
|
||||
# Resolve the tool reference
|
||||
# Authorization must happen before resolution because resolution may fetch a
|
||||
# missing tool from the registry.
|
||||
effective_provider = step.provider or provider_override
|
||||
if not _authorize_tool_call(
|
||||
step.tool, effective_provider, agent_tool_policies, call_stack
|
||||
):
|
||||
return "", False
|
||||
|
||||
# Resolve the tool reference only after authorization succeeds.
|
||||
try:
|
||||
resolved = resolve_tool(step.tool)
|
||||
nested_tool = resolved.tool
|
||||
@@ -768,9 +825,6 @@ def execute_tool_step(
|
||||
for key, value in step.args.items():
|
||||
custom_args[key] = substitute_variables(str(value), variables, warn_non_scalar=verbose)
|
||||
|
||||
# Determine effective provider (step override > parent override)
|
||||
effective_provider = step.provider or provider_override
|
||||
|
||||
if verbose:
|
||||
print(f"[verbose] Tool step: calling {step.tool}", file=sys.stderr)
|
||||
print(f"[verbose] Input length: {len(input_text)} chars", file=sys.stderr)
|
||||
@@ -789,7 +843,14 @@ def execute_tool_step(
|
||||
show_prompt=False,
|
||||
verbose=verbose,
|
||||
_depth=depth + 1,
|
||||
_call_stack=call_stack
|
||||
_call_stack=call_stack,
|
||||
agent_profile=step.profile if step.profile is not None else agent_profile,
|
||||
agent_skills=step.skills if step.skills is not None else agent_skills,
|
||||
agent_tool_policies=_extend_agent_tool_policies(
|
||||
agent_tool_policies, step.tools
|
||||
),
|
||||
execution_policy=execution_policy,
|
||||
provenance_log=provenance_log,
|
||||
)
|
||||
|
||||
return output, exit_code == 0
|
||||
@@ -805,7 +866,12 @@ def run_tool(
|
||||
verbose: bool = False,
|
||||
auto_install: bool = False,
|
||||
_depth: int = 0,
|
||||
_call_stack: Optional[list] = None
|
||||
_call_stack: Optional[list] = None,
|
||||
agent_profile: Optional[str] = None,
|
||||
agent_skills: Optional[List[str]] = None,
|
||||
agent_tool_policies: Optional[List[List[str]]] = None,
|
||||
execution_policy: Optional[ProviderExecutionPolicy] = None,
|
||||
provenance_log: Optional[list[dict]] = None,
|
||||
) -> tuple[str, int]:
|
||||
"""
|
||||
Execute a tool.
|
||||
@@ -901,6 +967,9 @@ def run_tool(
|
||||
output = substitute_variables(tool.output, variables, warn_non_scalar=verbose)
|
||||
return output, 0
|
||||
|
||||
# Reuse MCP configuration and schema caches across all MCP steps in this run.
|
||||
mcp_manager = None
|
||||
|
||||
# Execute each step
|
||||
for i, step in enumerate(tool.steps):
|
||||
if verbose:
|
||||
@@ -910,6 +979,8 @@ def run_tool(
|
||||
step_type = "CODE"
|
||||
elif isinstance(step, ToolStep):
|
||||
step_type = f"TOOL({step.tool})"
|
||||
elif isinstance(step, McpStep):
|
||||
step_type = f"MCP({step.server}/{step.tool})"
|
||||
else:
|
||||
step_type = "UNKNOWN"
|
||||
print(f"[verbose] Step {i+1}: {step_type} -> {{{step.output_var}}}", file=sys.stderr)
|
||||
@@ -925,12 +996,20 @@ def run_tool(
|
||||
if dry_run:
|
||||
variables[step.output_var] = f"[DRY RUN - would call {step.provider}]"
|
||||
else:
|
||||
# Apply agent-level defaults if step doesn't set its own
|
||||
effective_step = step
|
||||
if agent_profile and not step.profile:
|
||||
effective_step = _step_with_profile(step, agent_profile)
|
||||
if agent_skills is not None and step.skills is None:
|
||||
effective_step = _step_with_skills(effective_step, agent_skills)
|
||||
output, success = execute_prompt_step(
|
||||
step,
|
||||
effective_step,
|
||||
variables,
|
||||
provider_override,
|
||||
verbose=verbose,
|
||||
base_dir=tool_base_dir
|
||||
base_dir=tool_base_dir,
|
||||
execution_policy=execution_policy,
|
||||
provenance_log=provenance_log,
|
||||
)
|
||||
if not success:
|
||||
return "", 2
|
||||
@@ -975,18 +1054,95 @@ def run_tool(
|
||||
provider_override=provider_override,
|
||||
dry_run=dry_run,
|
||||
verbose=verbose,
|
||||
call_stack=step_stack
|
||||
call_stack=step_stack,
|
||||
agent_profile=agent_profile,
|
||||
agent_skills=agent_skills,
|
||||
agent_tool_policies=agent_tool_policies,
|
||||
execution_policy=execution_policy,
|
||||
provenance_log=provenance_log,
|
||||
)
|
||||
if not success:
|
||||
return "", 3
|
||||
variables[step.output_var] = output
|
||||
|
||||
elif isinstance(step, McpStep):
|
||||
if verbose or dry_run:
|
||||
print(f"=== MCP (Step {i+1}) -> {{{step.output_var}}} ===", file=sys.stderr)
|
||||
print(f" Server: {step.server} Tool: {step.tool}", file=sys.stderr)
|
||||
print(f" Args: {step.arguments}", file=sys.stderr)
|
||||
print("=== END MCP ===", file=sys.stderr)
|
||||
|
||||
if dry_run:
|
||||
variables[step.output_var] = f"[DRY RUN - would call mcp:{step.server}/{step.tool}]"
|
||||
else:
|
||||
step_stack = _call_stack + [(tool.name, i + 1)]
|
||||
if not _authorize_mcp_call(
|
||||
step.server, provider_override, step_stack
|
||||
):
|
||||
return "", 4
|
||||
args = _substitute_mcp_args(step.arguments, variables)
|
||||
try:
|
||||
if mcp_manager is None:
|
||||
mcp_manager = McpClientManager()
|
||||
result = mcp_manager.call_tool(
|
||||
step.server, step.tool, args,
|
||||
result_mode=step.result_mode,
|
||||
)
|
||||
variables[step.output_var] = result
|
||||
except ImportError as e:
|
||||
print(f"{e}", file=sys.stderr)
|
||||
return "", 4
|
||||
except Exception as e:
|
||||
print(f"MCP step failed (server={step.server}, tool={step.tool}): {e}", file=sys.stderr)
|
||||
return "", 4
|
||||
|
||||
# Generate final output
|
||||
output = substitute_variables(tool.output, variables, warn_non_scalar=verbose)
|
||||
|
||||
return output, 0
|
||||
|
||||
|
||||
def _substitute_mcp_args(arguments: dict, variables: dict) -> dict:
|
||||
"""Substitute variables in MCP arguments, preserving types."""
|
||||
result = {}
|
||||
for key, value in arguments.items():
|
||||
result[key] = _deep_substitute(value, variables)
|
||||
return result
|
||||
|
||||
|
||||
def _deep_substitute(value, variables: dict):
|
||||
if isinstance(value, str):
|
||||
import re
|
||||
|
||||
exact_reference = re.fullmatch(
|
||||
r"\s*\{([a-zA-Z_][a-zA-Z0-9_]*(?:\.[a-zA-Z_][a-zA-Z0-9_]*)*)\}\s*",
|
||||
value,
|
||||
)
|
||||
if exact_reference:
|
||||
path = exact_reference.group(1).split(".")
|
||||
if path[0] in variables:
|
||||
resolved = variables[path[0]]
|
||||
found = True
|
||||
for key in path[1:]:
|
||||
if isinstance(resolved, dict) and key in resolved:
|
||||
resolved = resolved[key]
|
||||
elif hasattr(resolved, key):
|
||||
resolved = getattr(resolved, key)
|
||||
else:
|
||||
found = False
|
||||
break
|
||||
if found:
|
||||
return resolved
|
||||
|
||||
substituted = substitute_variables(value, variables)
|
||||
return substituted if substituted != value else value
|
||||
elif isinstance(value, dict):
|
||||
return {k: _deep_substitute(v, variables) for k, v in value.items()}
|
||||
elif isinstance(value, list):
|
||||
return [_deep_substitute(v, variables) for v in value]
|
||||
return value
|
||||
|
||||
|
||||
def create_argument_parser(tool: Tool) -> argparse.ArgumentParser:
|
||||
"""
|
||||
Create an argument parser for a tool.
|
||||
@@ -1019,14 +1175,65 @@ def create_argument_parser(tool: Tool) -> argparse.ArgumentParser:
|
||||
help="Show debug information")
|
||||
parser.add_argument("--auto-install", action="store_true",
|
||||
help="Automatically install missing tool dependencies")
|
||||
parser.add_argument("--no-fallback", action="store_true",
|
||||
help="Fail instead of trying configured fallbacks")
|
||||
parser.add_argument("--require-local", action="store_true",
|
||||
help="Allow only providers declared local")
|
||||
parser.add_argument("--require-capability", action="append", default=[],
|
||||
help="Required provider capability (repeatable)")
|
||||
parser.add_argument("--data-classification",
|
||||
choices=["public", "internal", "private"])
|
||||
parser.add_argument("--require-model-identity", action="store_true")
|
||||
parser.add_argument("--require-model-digest", action="store_true")
|
||||
parser.add_argument("--result-envelope", choices=["json"])
|
||||
|
||||
# Tool-specific flags from arguments
|
||||
for arg in tool.arguments:
|
||||
existing_action = parser._option_string_actions.get(arg.flag)
|
||||
if existing_action is not None:
|
||||
# Preserve the legacy and useful case where a boolean tool option
|
||||
# intentionally shares a universal flag (for example --verbose).
|
||||
# Both consumers then read the same parsed boolean.
|
||||
if arg.type == "boolean" and existing_action.dest == arg.variable:
|
||||
continue
|
||||
raise ValueError(
|
||||
f"Tool argument {arg.flag!r} conflicts with CmdForge's "
|
||||
"universal run options"
|
||||
)
|
||||
|
||||
def parse_boolean(value):
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
normalized = str(value).strip().lower()
|
||||
if normalized in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if normalized in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
raise argparse.ArgumentTypeError(
|
||||
"expected true/false, yes/no, on/off, or 1/0"
|
||||
)
|
||||
|
||||
value_type = {
|
||||
"string": str,
|
||||
"integer": int,
|
||||
"number": float,
|
||||
"boolean": parse_boolean,
|
||||
}[arg.type]
|
||||
options = {
|
||||
"dest": arg.variable,
|
||||
"default": arg.default,
|
||||
"help": arg.description or f"{arg.variable} (default: {arg.default})",
|
||||
"required": arg.required,
|
||||
"type": value_type,
|
||||
}
|
||||
if arg.enum is not None:
|
||||
options["choices"] = [value_type(value) for value in arg.enum]
|
||||
if arg.type == "boolean":
|
||||
options["nargs"] = "?"
|
||||
options["const"] = True
|
||||
parser.add_argument(
|
||||
arg.flag,
|
||||
dest=arg.variable,
|
||||
default=arg.default,
|
||||
help=arg.description or f"{arg.variable} (default: {arg.default})"
|
||||
**options,
|
||||
)
|
||||
|
||||
return parser
|
||||
@@ -1059,6 +1266,11 @@ def main():
|
||||
print(f"Searched: {', '.join(e.searched_paths[:3])}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
from .tool import deprecation_warning
|
||||
warning = deprecation_warning(tool)
|
||||
if warning:
|
||||
print(warning, file=sys.stderr)
|
||||
|
||||
# Check for manifest overrides
|
||||
manifest = load_manifest()
|
||||
provider_override_from_manifest = None
|
||||
@@ -1096,6 +1308,16 @@ def main():
|
||||
# Determine provider override (CLI flag takes precedence over manifest)
|
||||
effective_provider = args.provider or provider_override_from_manifest
|
||||
|
||||
execution_policy = ProviderExecutionPolicy(
|
||||
allow_fallback=not args.no_fallback,
|
||||
required_locality="local" if args.require_local else None,
|
||||
required_capabilities=tuple(args.require_capability or []),
|
||||
data_classification=args.data_classification or "public",
|
||||
require_model_identity=args.require_model_identity,
|
||||
require_model_digest=args.require_model_digest,
|
||||
)
|
||||
provider_calls = []
|
||||
|
||||
# Run tool
|
||||
output, exit_code = run_tool(
|
||||
tool=tool,
|
||||
@@ -1105,18 +1327,139 @@ def main():
|
||||
dry_run=args.dry_run,
|
||||
show_prompt=args.show_prompt,
|
||||
verbose=args.verbose,
|
||||
auto_install=args.auto_install
|
||||
auto_install=args.auto_install,
|
||||
execution_policy=execution_policy,
|
||||
provenance_log=provider_calls,
|
||||
)
|
||||
|
||||
# Write output
|
||||
if exit_code == 0 and output:
|
||||
if args.output_file:
|
||||
Path(args.output_file).write_text(output)
|
||||
if args.result_envelope == "json" or (exit_code == 0 and output):
|
||||
if args.result_envelope == "json":
|
||||
import json
|
||||
rendered_output = json.dumps({
|
||||
"output": output,
|
||||
"execution": {
|
||||
"exit_code": exit_code,
|
||||
"provider_calls": provider_calls,
|
||||
"fallback_used": any(
|
||||
item.get("fallback_used", False)
|
||||
for item in provider_calls
|
||||
),
|
||||
"actual_providers": list(dict.fromkeys(
|
||||
item["actual_provider"] for item in provider_calls
|
||||
if item.get("actual_provider")
|
||||
)),
|
||||
},
|
||||
}, indent=2)
|
||||
else:
|
||||
print(output)
|
||||
rendered_output = output
|
||||
if args.output_file:
|
||||
Path(args.output_file).write_text(rendered_output)
|
||||
else:
|
||||
print(rendered_output)
|
||||
|
||||
if exit_code == 0 and not args.dry_run:
|
||||
try:
|
||||
from .usage import record_invocation
|
||||
record_invocation(resolved.full_name)
|
||||
except (OSError, ValueError):
|
||||
# Usage discovery is optional and must never break tool execution.
|
||||
pass
|
||||
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
def _matches_allowlist(name: str, allowed: List[str]) -> bool:
|
||||
return any(fnmatch.fnmatchcase(name, pattern) for pattern in allowed)
|
||||
|
||||
|
||||
def _authorize_tool_call(
|
||||
name: str,
|
||||
provider_name: Optional[str],
|
||||
agent_tool_policies: Optional[List[List[str]]],
|
||||
call_stack: list,
|
||||
) -> bool:
|
||||
"""Enforce provider and delegated-agent tool policies."""
|
||||
if provider_name:
|
||||
from .providers import get_provider
|
||||
provider = get_provider(provider_name)
|
||||
if provider is None:
|
||||
_print_call_stack(
|
||||
call_stack,
|
||||
f"Provider '{provider_name}' is not configured; refusing tool call '{name}'",
|
||||
)
|
||||
return False
|
||||
if provider.tools is not None and not _matches_allowlist(name, provider.tools):
|
||||
_print_call_stack(
|
||||
call_stack,
|
||||
f"Provider '{provider_name}' is not allowed to call tool '{name}'",
|
||||
)
|
||||
print(
|
||||
f"Provider '{provider_name}' tools allowlist: {provider.tools}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return False
|
||||
|
||||
for policy in agent_tool_policies or []:
|
||||
if not _matches_allowlist(name, policy):
|
||||
_print_call_stack(
|
||||
call_stack,
|
||||
f"Delegated agent is not allowed to call tool '{name}'",
|
||||
)
|
||||
print(f"Delegated agent tools allowlist: {policy}", file=sys.stderr)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _extend_agent_tool_policies(
|
||||
policies: Optional[List[List[str]]],
|
||||
additional: Optional[List[str]],
|
||||
) -> Optional[List[List[str]]]:
|
||||
"""Add a restriction without allowing descendants to broaden ancestors."""
|
||||
if additional is None:
|
||||
return policies
|
||||
return [*(policies or []), additional]
|
||||
|
||||
|
||||
def _authorize_mcp_call(
|
||||
server: str,
|
||||
provider_name: Optional[str],
|
||||
call_stack: list,
|
||||
) -> bool:
|
||||
"""Enforce the effective provider's MCP-server policy."""
|
||||
if not provider_name:
|
||||
return True
|
||||
|
||||
from .providers import get_provider
|
||||
provider = get_provider(provider_name)
|
||||
if provider is None:
|
||||
_print_call_stack(
|
||||
call_stack,
|
||||
f"Provider '{provider_name}' is not configured; refusing MCP server '{server}'",
|
||||
)
|
||||
return False
|
||||
if provider.mcp_servers is not None and not _matches_allowlist(
|
||||
server, provider.mcp_servers
|
||||
):
|
||||
_print_call_stack(
|
||||
call_stack,
|
||||
f"Provider '{provider_name}' is not allowed to use MCP server '{server}'",
|
||||
)
|
||||
print(
|
||||
f"Provider '{provider_name}' MCP server allowlist: {provider.mcp_servers}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _step_with_profile(step, profile_name):
|
||||
return replace(step, profile=profile_name)
|
||||
|
||||
|
||||
def _step_with_skills(step, skill_names):
|
||||
return replace(step, skills=skill_names)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,348 @@
|
||||
"""Conservative JSON Schema compatibility checks for tool composition."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any, Dict, List, Literal, Optional, Set
|
||||
|
||||
|
||||
CompatibilityState = Literal["compatible", "incompatible", "unknown"]
|
||||
|
||||
|
||||
@dataclass
|
||||
class SchemaCompatibility:
|
||||
state: CompatibilityState
|
||||
detail: str
|
||||
issues: List[str] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def compare_json_schemas(
|
||||
producer: Optional[dict], consumer: Optional[dict], path: str = "$"
|
||||
) -> SchemaCompatibility:
|
||||
"""Check whether every value allowed by producer is accepted by consumer.
|
||||
|
||||
This intentionally implements a useful, conservative subset of JSON Schema
|
||||
subsumption. Unsupported combinators and constraints return ``unknown``
|
||||
rather than claiming compatibility.
|
||||
"""
|
||||
if producer is None or consumer is None:
|
||||
return _unknown(path, "schema evidence is missing")
|
||||
if consumer == {}:
|
||||
return _compatible("consumer accepts any JSON value")
|
||||
if producer == {}:
|
||||
return _unknown(path, "producer schema is unconstrained")
|
||||
|
||||
concrete = _concrete_values(producer)
|
||||
if concrete is not None:
|
||||
failures = [value for value in concrete if not _value_matches(value, consumer)]
|
||||
if failures:
|
||||
return _incompatible(
|
||||
path, f"producer can emit value not accepted by consumer: {failures[0]!r}"
|
||||
)
|
||||
return _compatible("all producer const/enum values satisfy consumer")
|
||||
if _concrete_values(consumer) is not None:
|
||||
return _incompatible(
|
||||
path, "producer is not restricted to the consumer's const/enum values"
|
||||
)
|
||||
|
||||
if any(key in producer or key in consumer for key in ("oneOf", "anyOf", "allOf", "not", "$ref")):
|
||||
return _unknown(path, "schema combinators or references require deeper analysis")
|
||||
|
||||
advanced_constraints = (
|
||||
"contains", "minContains", "maxContains", "patternProperties",
|
||||
"dependentSchemas", "propertyNames", "unevaluatedProperties",
|
||||
"unevaluatedItems", "if", "then", "else",
|
||||
)
|
||||
differing = [
|
||||
keyword for keyword in advanced_constraints
|
||||
if keyword in consumer and producer.get(keyword) != consumer.get(keyword)
|
||||
]
|
||||
if differing:
|
||||
return _unknown(
|
||||
path, "cannot prove constraint(s): " + ", ".join(differing)
|
||||
)
|
||||
|
||||
producer_types = _types(producer)
|
||||
consumer_types = _types(consumer)
|
||||
if producer_types and consumer_types:
|
||||
rejected = {
|
||||
item for item in producer_types
|
||||
if not any(_type_accepted(item, target) for target in consumer_types)
|
||||
}
|
||||
if rejected:
|
||||
return _incompatible(
|
||||
path,
|
||||
"producer type(s) " + ", ".join(sorted(rejected))
|
||||
+ " are not accepted by consumer type(s) "
|
||||
+ ", ".join(sorted(consumer_types)),
|
||||
)
|
||||
elif not producer_types or not consumer_types:
|
||||
return _unknown(path, "type constraints are incomplete")
|
||||
|
||||
states: List[SchemaCompatibility] = []
|
||||
effective_types = producer_types or set()
|
||||
if "object" in effective_types:
|
||||
states.append(_compare_objects(producer, consumer, path))
|
||||
if "array" in effective_types:
|
||||
states.append(_compare_arrays(producer, consumer, path))
|
||||
if "string" in effective_types:
|
||||
states.append(_compare_strings(producer, consumer, path))
|
||||
if effective_types & {"integer", "number"}:
|
||||
states.append(_compare_numbers(producer, consumer, path))
|
||||
|
||||
incompatible = [state for state in states if state.state == "incompatible"]
|
||||
if incompatible:
|
||||
issues = [issue for state in incompatible for issue in state.issues]
|
||||
return SchemaCompatibility("incompatible", incompatible[0].detail, issues)
|
||||
unknown = [state for state in states if state.state == "unknown"]
|
||||
if unknown:
|
||||
issues = [issue for state in unknown for issue in state.issues]
|
||||
return SchemaCompatibility("unknown", unknown[0].detail, issues)
|
||||
return _compatible("producer schema is accepted by consumer schema")
|
||||
|
||||
|
||||
def schema_for_value(value: Any) -> dict:
|
||||
"""Return an exact schema for a literal value."""
|
||||
return {"const": value}
|
||||
|
||||
|
||||
def _compare_objects(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
|
||||
producer_properties = producer.get("properties", {})
|
||||
consumer_properties = consumer.get("properties", {})
|
||||
producer_required = set(producer.get("required", []))
|
||||
consumer_required = set(consumer.get("required", []))
|
||||
issues: List[str] = []
|
||||
unknown = False
|
||||
incompatible = False
|
||||
|
||||
if (
|
||||
consumer.get("additionalProperties") is False
|
||||
and producer.get("additionalProperties") is not False
|
||||
):
|
||||
issues.append(f"{path}: producer permits unspecified object fields")
|
||||
incompatible = True
|
||||
|
||||
producer_min = max(producer.get("minProperties", 0), len(producer_required))
|
||||
consumer_min = consumer.get("minProperties", 0)
|
||||
if producer_min < consumer_min:
|
||||
issues.append(f"{path}: producer permits too few object properties")
|
||||
incompatible = True
|
||||
producer_max = producer.get("maxProperties")
|
||||
if producer.get("additionalProperties") is False:
|
||||
closed_max = len(producer_properties)
|
||||
producer_max = closed_max if producer_max is None else min(producer_max, closed_max)
|
||||
consumer_max = consumer.get("maxProperties")
|
||||
if consumer_max is not None and (
|
||||
producer_max is None or producer_max > consumer_max
|
||||
):
|
||||
issues.append(f"{path}: producer permits too many object properties")
|
||||
incompatible = True
|
||||
|
||||
for name in sorted(consumer_required - producer_required):
|
||||
issues.append(f"{path}.{name}: required consumer field is not guaranteed")
|
||||
incompatible = True
|
||||
|
||||
producer_additional = producer.get("additionalProperties", True)
|
||||
for name, consumer_schema in consumer_properties.items():
|
||||
if name in producer_properties or producer_additional is False:
|
||||
continue
|
||||
if producer_additional is True:
|
||||
issues.append(
|
||||
f"{path}.{name}: producer may emit an unconstrained value"
|
||||
)
|
||||
incompatible = True
|
||||
continue
|
||||
if isinstance(producer_additional, dict):
|
||||
result = compare_json_schemas(
|
||||
producer_additional, consumer_schema, f"{path}.{name}"
|
||||
)
|
||||
issues.extend(result.issues)
|
||||
incompatible = incompatible or result.state == "incompatible"
|
||||
unknown = unknown or result.state == "unknown"
|
||||
|
||||
if consumer.get("additionalProperties") is False:
|
||||
for name in sorted(producer_required - set(consumer_properties)):
|
||||
issues.append(f"{path}.{name}: consumer rejects this supplied field")
|
||||
incompatible = True
|
||||
|
||||
for name, producer_schema in producer_properties.items():
|
||||
consumer_schema = consumer_properties.get(name)
|
||||
if consumer_schema is None:
|
||||
additional = consumer.get("additionalProperties", True)
|
||||
if additional is False and name in producer_required:
|
||||
issues.append(f"{path}.{name}: consumer does not allow this field")
|
||||
incompatible = True
|
||||
elif isinstance(additional, dict):
|
||||
result = compare_json_schemas(
|
||||
producer_schema, additional, f"{path}.{name}"
|
||||
)
|
||||
issues.extend(result.issues)
|
||||
unknown = unknown or result.state == "unknown"
|
||||
continue
|
||||
result = compare_json_schemas(
|
||||
producer_schema, consumer_schema, f"{path}.{name}"
|
||||
)
|
||||
if result.state == "incompatible":
|
||||
issues.extend(result.issues)
|
||||
incompatible = True
|
||||
elif result.state == "unknown":
|
||||
unknown = True
|
||||
issues.extend(result.issues)
|
||||
|
||||
if incompatible:
|
||||
return SchemaCompatibility("incompatible", issues[0], issues)
|
||||
if unknown:
|
||||
return SchemaCompatibility("unknown", issues[0] if issues else "object constraints are incomplete", issues)
|
||||
return _compatible("object fields and required properties are compatible")
|
||||
|
||||
|
||||
def _compare_arrays(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
|
||||
if consumer.get("uniqueItems") is True and producer.get("uniqueItems") is not True:
|
||||
return _unknown(path, "producer does not guarantee unique array items")
|
||||
producer_min = producer.get("minItems", 0)
|
||||
consumer_min = consumer.get("minItems", 0)
|
||||
if producer_min < consumer_min:
|
||||
return _incompatible(path, "producer permits fewer array items than consumer")
|
||||
producer_max = producer.get("maxItems")
|
||||
consumer_max = consumer.get("maxItems")
|
||||
if consumer_max is not None and (
|
||||
producer_max is None or producer_max > consumer_max
|
||||
):
|
||||
return _incompatible(path, "producer permits more array items than consumer")
|
||||
if "items" in consumer:
|
||||
if "items" not in producer:
|
||||
return _unknown(path, "producer array item schema is missing")
|
||||
return compare_json_schemas(
|
||||
producer["items"], consumer["items"], f"{path}[]"
|
||||
)
|
||||
return _compatible("array bounds are compatible")
|
||||
|
||||
|
||||
def _compare_strings(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
|
||||
if producer.get("minLength", 0) < consumer.get("minLength", 0):
|
||||
return _incompatible(path, "producer permits strings shorter than consumer")
|
||||
consumer_max = consumer.get("maxLength")
|
||||
producer_max = producer.get("maxLength")
|
||||
if consumer_max is not None and (
|
||||
producer_max is None or producer_max > consumer_max
|
||||
):
|
||||
return _incompatible(path, "producer permits strings longer than consumer")
|
||||
for keyword in ("pattern", "format"):
|
||||
if keyword in consumer and producer.get(keyword) != consumer.get(keyword):
|
||||
return _unknown(path, f"cannot prove differing {keyword} constraints")
|
||||
return _compatible("string bounds are compatible")
|
||||
|
||||
|
||||
def _compare_numbers(producer: dict, consumer: dict, path: str) -> SchemaCompatibility:
|
||||
producer_min = _lower_bound(producer)
|
||||
consumer_min = _lower_bound(consumer)
|
||||
if consumer_min is not None and (
|
||||
producer_min is None or producer_min < consumer_min
|
||||
):
|
||||
return _incompatible(path, "producer permits numbers below consumer minimum")
|
||||
if "exclusiveMinimum" in consumer and producer_min == consumer_min:
|
||||
if producer.get("exclusiveMinimum") != consumer.get("exclusiveMinimum"):
|
||||
return _incompatible(
|
||||
path, "producer includes the consumer's exclusive minimum"
|
||||
)
|
||||
producer_max = _upper_bound(producer)
|
||||
consumer_max = _upper_bound(consumer)
|
||||
if consumer_max is not None and (
|
||||
producer_max is None or producer_max > consumer_max
|
||||
):
|
||||
return _incompatible(path, "producer permits numbers above consumer maximum")
|
||||
if "exclusiveMaximum" in consumer and producer_max == consumer_max:
|
||||
if producer.get("exclusiveMaximum") != consumer.get("exclusiveMaximum"):
|
||||
return _incompatible(
|
||||
path, "producer includes the consumer's exclusive maximum"
|
||||
)
|
||||
if "multipleOf" in consumer and producer.get("multipleOf") != consumer.get("multipleOf"):
|
||||
return _unknown(path, "cannot prove differing multipleOf constraints")
|
||||
return _compatible("numeric bounds are compatible")
|
||||
|
||||
|
||||
def _types(schema: dict) -> Set[str]:
|
||||
value = schema.get("type")
|
||||
if isinstance(value, str):
|
||||
return {value}
|
||||
if isinstance(value, list):
|
||||
return {item for item in value if isinstance(item, str)}
|
||||
if "properties" in schema or "required" in schema:
|
||||
return {"object"}
|
||||
if "items" in schema:
|
||||
return {"array"}
|
||||
concrete = _concrete_values(schema)
|
||||
if concrete:
|
||||
return {_json_type(value) for value in concrete}
|
||||
return set()
|
||||
|
||||
|
||||
def _concrete_values(schema: dict) -> Optional[List[Any]]:
|
||||
if "const" in schema:
|
||||
return [schema["const"]]
|
||||
if isinstance(schema.get("enum"), list) and schema["enum"]:
|
||||
return list(schema["enum"])
|
||||
return None
|
||||
|
||||
|
||||
def _value_matches(value: Any, schema: dict) -> bool:
|
||||
try:
|
||||
from jsonschema import validate
|
||||
from jsonschema.exceptions import ValidationError
|
||||
validate(instance=value, schema=schema)
|
||||
return True
|
||||
except ValidationError:
|
||||
return False
|
||||
|
||||
|
||||
def _json_type(value: Any) -> str:
|
||||
if value is None:
|
||||
return "null"
|
||||
if isinstance(value, bool):
|
||||
return "boolean"
|
||||
if isinstance(value, int):
|
||||
return "integer"
|
||||
if isinstance(value, float):
|
||||
return "number"
|
||||
if isinstance(value, str):
|
||||
return "string"
|
||||
if isinstance(value, list):
|
||||
return "array"
|
||||
return "object"
|
||||
|
||||
|
||||
def _type_accepted(producer: str, consumer: str) -> bool:
|
||||
return producer == consumer or (producer == "integer" and consumer == "number")
|
||||
|
||||
|
||||
def _lower_bound(schema: dict) -> Optional[float]:
|
||||
if "exclusiveMinimum" in schema:
|
||||
return float(schema["exclusiveMinimum"])
|
||||
if "minimum" in schema:
|
||||
return float(schema["minimum"])
|
||||
return None
|
||||
|
||||
|
||||
def _upper_bound(schema: dict) -> Optional[float]:
|
||||
if "exclusiveMaximum" in schema:
|
||||
return float(schema["exclusiveMaximum"])
|
||||
if "maximum" in schema:
|
||||
return float(schema["maximum"])
|
||||
return None
|
||||
|
||||
|
||||
def _compatible(detail: str) -> SchemaCompatibility:
|
||||
return SchemaCompatibility("compatible", detail)
|
||||
|
||||
|
||||
def _unknown(path: str, detail: str) -> SchemaCompatibility:
|
||||
issue = f"{path}: {detail}"
|
||||
return SchemaCompatibility("unknown", issue, [issue])
|
||||
|
||||
|
||||
def _incompatible(path: str, detail: str) -> SchemaCompatibility:
|
||||
issue = f"{path}: {detail}"
|
||||
return SchemaCompatibility("incompatible", issue, [issue])
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Local publisher signing-key storage."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from .attestation import generate_keypair
|
||||
from .config import CONFIG_DIR
|
||||
|
||||
SIGNING_KEY_FILE = CONFIG_DIR / "release-signing-key.json"
|
||||
|
||||
|
||||
def load_signing_key(path: Path = SIGNING_KEY_FILE) -> Optional[Tuple[str, str]]:
|
||||
if not path.exists():
|
||||
return None
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
private_key = data.get("private_key", "")
|
||||
public_key = data.get("public_key", "")
|
||||
if not private_key or not public_key:
|
||||
raise ValueError("Signing key file is incomplete")
|
||||
return private_key, public_key
|
||||
|
||||
|
||||
def initialize_signing_key(
|
||||
path: Path = SIGNING_KEY_FILE, *, overwrite: bool = False
|
||||
) -> Tuple[str, str]:
|
||||
if path.exists() and not overwrite:
|
||||
existing = load_signing_key(path)
|
||||
if existing is None:
|
||||
raise ValueError("Signing key file could not be loaded")
|
||||
return existing
|
||||
private_key, public_key = generate_keypair()
|
||||
path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
path.write_text(json.dumps({
|
||||
"algorithm": "ed25519",
|
||||
"private_key": private_key,
|
||||
"public_key": public_key,
|
||||
}, indent=2) + "\n", encoding="utf-8")
|
||||
os.chmod(path, 0o600)
|
||||
return private_key, public_key
|
||||
@@ -0,0 +1,221 @@
|
||||
"""Provider-attached skills in the Agent Skills open standard.
|
||||
|
||||
Each provider can have skills at::
|
||||
|
||||
~/.cmdforge/providers/<provider>/skills/<skill-name>/SKILL.md
|
||||
|
||||
Every ``SKILL.md`` must contain the required Agent Skills frontmatter. The
|
||||
directory name is the canonical skill identifier and must match ``name``.
|
||||
"""
|
||||
|
||||
import re
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
PROVIDERS_DIR = Path.home() / ".cmdforge" / "providers"
|
||||
|
||||
_PROVIDER_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$")
|
||||
_SKILL_NAME_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
|
||||
_MISSING_POLICIES = {"warn", "error", "ignore"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Skill:
|
||||
"""A single loaded skill."""
|
||||
|
||||
name: str
|
||||
directory: str
|
||||
description: str
|
||||
content: str = ""
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.content
|
||||
|
||||
|
||||
@dataclass
|
||||
class SkillSet:
|
||||
"""Collection of skills for one provider, deterministically ordered."""
|
||||
|
||||
provider_name: str
|
||||
skills: List[Skill] = field(default_factory=list)
|
||||
|
||||
def get(
|
||||
self,
|
||||
requested: Optional[List[str]] = None,
|
||||
on_missing: str = "warn",
|
||||
) -> List[Skill]:
|
||||
"""Return enabled skills, reporting missing ones."""
|
||||
if on_missing not in _MISSING_POLICIES:
|
||||
raise ValueError(
|
||||
f"Invalid missing-skill policy '{on_missing}'; expected one of "
|
||||
f"{sorted(_MISSING_POLICIES)}"
|
||||
)
|
||||
|
||||
available = {skill.name: skill for skill in self.skills}
|
||||
if requested is None:
|
||||
return list(self.skills)
|
||||
|
||||
enabled: List[Skill] = []
|
||||
for name in requested:
|
||||
if name in available:
|
||||
enabled.append(available[name])
|
||||
continue
|
||||
|
||||
message = (
|
||||
f"Skill '{name}' requested for provider '{self.provider_name}' "
|
||||
"but not found in skills directory"
|
||||
)
|
||||
if on_missing == "error":
|
||||
raise KeyError(message)
|
||||
if on_missing == "warn":
|
||||
print(f"[skills] {message}", file=sys.stderr)
|
||||
return enabled
|
||||
|
||||
|
||||
def _validate_provider_name(name: str) -> None:
|
||||
"""Validate a provider name before using it as a path component."""
|
||||
if not isinstance(name, str) or not _PROVIDER_NAME_RE.fullmatch(name):
|
||||
raise ValueError(
|
||||
f"Invalid provider name '{name}'. Must match {_PROVIDER_NAME_RE.pattern}"
|
||||
)
|
||||
|
||||
|
||||
def _validate_skill_name(name: str, provider: str = "") -> None:
|
||||
"""Validate the Agent Skills canonical name rules."""
|
||||
context = f" under provider '{provider}'" if provider else ""
|
||||
if (
|
||||
not isinstance(name, str)
|
||||
or not 1 <= len(name) <= 64
|
||||
or not _SKILL_NAME_RE.fullmatch(name)
|
||||
):
|
||||
raise ValueError(
|
||||
f"Invalid skill name '{name}'{context}. Skill names must be 1-64 "
|
||||
"lowercase letters, numbers, or single hyphens, and may not start "
|
||||
"or end with a hyphen"
|
||||
)
|
||||
|
||||
|
||||
def _parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Parse the mandatory YAML frontmatter from a ``SKILL.md`` file."""
|
||||
lines = text.splitlines(keepends=True)
|
||||
if not lines or lines[0].rstrip("\r\n") != "---":
|
||||
raise ValueError("SKILL.md must begin with YAML frontmatter")
|
||||
|
||||
closing_index = next(
|
||||
(index for index, line in enumerate(lines[1:], start=1)
|
||||
if line.rstrip("\r\n") == "---"),
|
||||
None,
|
||||
)
|
||||
if closing_index is None:
|
||||
raise ValueError("SKILL.md frontmatter is not terminated")
|
||||
|
||||
frontmatter_raw = "".join(lines[1:closing_index]).strip()
|
||||
if not frontmatter_raw:
|
||||
raise ValueError("SKILL.md frontmatter is empty; metadata is required")
|
||||
try:
|
||||
metadata = yaml.safe_load(frontmatter_raw)
|
||||
except yaml.YAMLError as exc:
|
||||
raise ValueError(f"Invalid YAML in SKILL.md frontmatter: {exc}") from exc
|
||||
if not isinstance(metadata, dict):
|
||||
raise ValueError("SKILL.md frontmatter must be a YAML mapping")
|
||||
|
||||
content = "".join(lines[closing_index + 1:]).lstrip("\r\n").rstrip()
|
||||
return metadata, content
|
||||
|
||||
|
||||
def _validate_metadata(metadata: dict, directory_name: str, provider: str) -> tuple[str, str]:
|
||||
"""Validate required Agent Skills metadata and return normalized values."""
|
||||
if "name" not in metadata:
|
||||
raise ValueError(f"SKILL.md for '{directory_name}' is missing required 'name'")
|
||||
name = metadata["name"]
|
||||
_validate_skill_name(name, provider)
|
||||
if name != directory_name:
|
||||
raise ValueError(
|
||||
f"SKILL.md frontmatter name '{name}' does not match skill directory "
|
||||
f"name '{directory_name}' under provider '{provider}'"
|
||||
)
|
||||
|
||||
if "description" not in metadata:
|
||||
raise ValueError(
|
||||
f"SKILL.md for '{directory_name}' is missing required 'description'"
|
||||
)
|
||||
description = metadata["description"]
|
||||
if not isinstance(description, str) or not description.strip():
|
||||
raise ValueError(
|
||||
f"SKILL.md description for '{directory_name}' must be a non-empty string"
|
||||
)
|
||||
if len(description) > 1024:
|
||||
raise ValueError(
|
||||
f"SKILL.md description for '{directory_name}' must not exceed 1024 characters"
|
||||
)
|
||||
return name, description.strip()
|
||||
|
||||
|
||||
def _reject_symlink(path: Path, label: str) -> None:
|
||||
if path.is_symlink():
|
||||
raise ValueError(f"Refusing symlinked {label}: {path}")
|
||||
|
||||
|
||||
def load_provider_skills(provider_name: str) -> SkillSet:
|
||||
"""Load and validate all skills configured for a provider."""
|
||||
_validate_provider_name(provider_name)
|
||||
|
||||
provider_dir = PROVIDERS_DIR / provider_name
|
||||
skills_dir = provider_dir / "skills"
|
||||
_reject_symlink(provider_dir, "provider directory")
|
||||
_reject_symlink(skills_dir, "skills directory")
|
||||
if not skills_dir.is_dir():
|
||||
return SkillSet(provider_name=provider_name)
|
||||
|
||||
skills: List[Skill] = []
|
||||
for entry in sorted(skills_dir.iterdir(), key=lambda path: path.name):
|
||||
_reject_symlink(entry, "skill directory")
|
||||
if not entry.is_dir():
|
||||
continue
|
||||
_validate_skill_name(entry.name, provider_name)
|
||||
|
||||
skill_md = entry / "SKILL.md"
|
||||
_reject_symlink(skill_md, "SKILL.md")
|
||||
if not skill_md.is_file():
|
||||
continue
|
||||
try:
|
||||
text = skill_md.read_text(encoding="utf-8")
|
||||
except OSError as exc:
|
||||
raise OSError(f"Cannot read {skill_md}: {exc}") from exc
|
||||
|
||||
metadata, content = _parse_frontmatter(text)
|
||||
name, description = _validate_metadata(
|
||||
metadata, entry.name, provider_name
|
||||
)
|
||||
skills.append(
|
||||
Skill(
|
||||
name=name,
|
||||
directory=str(entry),
|
||||
description=description,
|
||||
content=content,
|
||||
)
|
||||
)
|
||||
|
||||
return SkillSet(provider_name=provider_name, skills=skills)
|
||||
|
||||
|
||||
def inject_skills(
|
||||
prompt: str,
|
||||
provider_name: str,
|
||||
skill_names: Optional[List[str]] = None,
|
||||
on_missing: str = "warn",
|
||||
) -> str:
|
||||
"""Prepend enabled provider skills to a prompt."""
|
||||
skillset = load_provider_skills(provider_name)
|
||||
enabled = skillset.get(skill_names, on_missing=on_missing)
|
||||
if not enabled:
|
||||
return prompt
|
||||
|
||||
skill_context = "\n\n".join(
|
||||
f"## Skill: {skill.name}\n\n{skill.content}" for skill in enabled
|
||||
)
|
||||
return f"{skill_context}\n\n---\n\n{prompt}"
|
||||
+263
-12
@@ -1,11 +1,12 @@
|
||||
"""Tool loading, saving, and management."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import stat
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Literal
|
||||
from typing import Any, Optional, List, Literal
|
||||
|
||||
import yaml
|
||||
|
||||
@@ -15,6 +16,52 @@ TOOLS_DIR = Path.home() / ".cmdforge"
|
||||
|
||||
# Default bin directory for wrapper scripts
|
||||
BIN_DIR = Path.home() / ".local" / "bin"
|
||||
_TOOL_REF_RE = re.compile(
|
||||
r"^(?:[a-z0-9](?:[a-z0-9-]{0,37}[a-z0-9])?/)?[A-Za-z0-9_-]{1,64}$"
|
||||
)
|
||||
|
||||
|
||||
def _validate_skill_selection(skills: Optional[List[str]]) -> None:
|
||||
"""Validate a step's optional Agent Skills selection."""
|
||||
if skills is None:
|
||||
return
|
||||
if not isinstance(skills, list):
|
||||
raise ValueError("skills must be a list of skill names")
|
||||
if any(not isinstance(name, str) for name in skills):
|
||||
raise ValueError("skills entries must be strings")
|
||||
if "*" in skills and skills != ["*"]:
|
||||
raise ValueError("'*' must be the only entry when enabling all skills")
|
||||
if len(skills) != len(set(skills)):
|
||||
raise ValueError("skills must not contain duplicate names")
|
||||
|
||||
from .skills import _validate_skill_name
|
||||
for name in skills:
|
||||
if name != "*":
|
||||
_validate_skill_name(name)
|
||||
|
||||
|
||||
def _validate_optional_patterns(patterns: Optional[List[str]], field_name: str) -> None:
|
||||
if patterns is not None and (
|
||||
not isinstance(patterns, list)
|
||||
or not all(isinstance(pattern, str) and pattern for pattern in patterns)
|
||||
):
|
||||
raise ValueError(f"{field_name} must be a list of non-empty strings or null")
|
||||
|
||||
|
||||
def validate_json_schema(schema: Optional[dict], field_name: str) -> None:
|
||||
"""Validate an optional tool contract as a real JSON Schema."""
|
||||
if schema is None:
|
||||
return
|
||||
if not isinstance(schema, dict):
|
||||
raise ValueError(f"{field_name} must be a JSON Schema object")
|
||||
|
||||
from jsonschema.exceptions import SchemaError
|
||||
from jsonschema.validators import validator_for
|
||||
|
||||
try:
|
||||
validator_for(schema).check_schema(schema)
|
||||
except SchemaError as exc:
|
||||
raise ValueError(f"Invalid {field_name}: {exc.message}") from exc
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -70,15 +117,45 @@ class ToolArgument:
|
||||
"""Definition of a custom input argument."""
|
||||
flag: str # e.g., "--max-size"
|
||||
variable: str # e.g., "max_size"
|
||||
default: Optional[str] = None
|
||||
default: Optional[Any] = None
|
||||
description: str = ""
|
||||
type: str = "string"
|
||||
enum: Optional[List[Any]] = None
|
||||
required: bool = False
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Older CmdForge tools used Python-style scalar names. Normalize them
|
||||
# on load so typed parsing does not break existing user tools.
|
||||
legacy_type_aliases = {
|
||||
"str": "string",
|
||||
"int": "integer",
|
||||
"float": "number",
|
||||
"bool": "boolean",
|
||||
}
|
||||
self.type = legacy_type_aliases.get(self.type, self.type)
|
||||
if self.type not in ("string", "integer", "number", "boolean"):
|
||||
raise ValueError(
|
||||
"ToolArgument type must be string, integer, number, or boolean"
|
||||
)
|
||||
if self.enum is not None and (
|
||||
not isinstance(self.enum, list) or not self.enum
|
||||
):
|
||||
raise ValueError("ToolArgument enum must be a non-empty list")
|
||||
if not isinstance(self.required, bool):
|
||||
raise ValueError("ToolArgument required must be a boolean")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
d = {"flag": self.flag, "variable": self.variable}
|
||||
if self.default:
|
||||
if self.default is not None:
|
||||
d["default"] = self.default
|
||||
if self.description:
|
||||
d["description"] = self.description
|
||||
if self.type != "string":
|
||||
d["type"] = self.type
|
||||
if self.enum is not None:
|
||||
d["enum"] = self.enum
|
||||
if self.required:
|
||||
d["required"] = True
|
||||
return d
|
||||
|
||||
@classmethod
|
||||
@@ -87,7 +164,10 @@ class ToolArgument:
|
||||
flag=data["flag"],
|
||||
variable=data["variable"],
|
||||
default=data.get("default"),
|
||||
description=data.get("description", "")
|
||||
description=data.get("description", ""),
|
||||
type=data.get("type", "string"),
|
||||
enum=data.get("enum"),
|
||||
required=data.get("required", False),
|
||||
)
|
||||
|
||||
|
||||
@@ -106,6 +186,13 @@ class PromptStep:
|
||||
max_retries: int = 1 # Retry count on validation failure
|
||||
plain_text: bool = False # Bypass structured output enforcement
|
||||
max_tokens: Optional[int] = None # Max output tokens (provider-dependent)
|
||||
skills: Optional[List[str]] = None # Skill names to enable for this step
|
||||
fallback_policy: str = "allow" # "allow" | "deny"
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_validate_skill_selection(self.skills)
|
||||
if self.fallback_policy not in {"allow", "deny"}:
|
||||
raise ValueError("PromptStep fallback_policy must be allow or deny")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
d = {
|
||||
@@ -130,6 +217,10 @@ class PromptStep:
|
||||
d["plain_text"] = self.plain_text
|
||||
if self.max_tokens:
|
||||
d["max_tokens"] = self.max_tokens
|
||||
if self.skills is not None:
|
||||
d["skills"] = self.skills
|
||||
if self.fallback_policy != "allow":
|
||||
d["fallback_policy"] = self.fallback_policy
|
||||
return d
|
||||
|
||||
@classmethod
|
||||
@@ -154,7 +245,9 @@ class PromptStep:
|
||||
output_schema=data.get("output_schema"),
|
||||
max_retries=data.get("max_retries", 1),
|
||||
plain_text=data.get("plain_text", False),
|
||||
max_tokens=max_tokens
|
||||
max_tokens=max_tokens,
|
||||
skills=data.get("skills"),
|
||||
fallback_policy=data.get("fallback_policy", "allow"),
|
||||
)
|
||||
|
||||
|
||||
@@ -196,7 +289,18 @@ class ToolStep:
|
||||
input_template: str = "{input}" # Input template (supports variable substitution)
|
||||
args: dict = field(default_factory=dict) # Arguments to pass to the tool
|
||||
provider: Optional[str] = None # Provider override for the called tool
|
||||
profile: Optional[str] = None # AI persona for the nested tool
|
||||
skills: Optional[List[str]] = None # Skills to enable in the nested tool
|
||||
name: Optional[str] = None # Optional display name for the step
|
||||
tools: Optional[List[str]] = None # Tools the delegated agent may call
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
_validate_skill_selection(self.skills)
|
||||
_validate_optional_patterns(self.tools, "tools")
|
||||
if self.profile is not None and (
|
||||
not isinstance(self.profile, str) or not self.profile
|
||||
):
|
||||
raise ValueError("profile must be a non-empty string or null")
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
d = {
|
||||
@@ -210,6 +314,12 @@ class ToolStep:
|
||||
d["args"] = self.args
|
||||
if self.provider:
|
||||
d["provider"] = self.provider
|
||||
if self.profile:
|
||||
d["profile"] = self.profile
|
||||
if self.skills is not None:
|
||||
d["skills"] = self.skills
|
||||
if self.tools is not None:
|
||||
d["tools"] = self.tools
|
||||
if self.name:
|
||||
d["name"] = self.name
|
||||
return d
|
||||
@@ -222,12 +332,66 @@ class ToolStep:
|
||||
input_template=data.get("input", "{input}"),
|
||||
args=data.get("args", {}),
|
||||
provider=data.get("provider"),
|
||||
profile=data.get("profile"),
|
||||
skills=data.get("skills"),
|
||||
tools=data.get("tools"),
|
||||
name=data.get("name")
|
||||
)
|
||||
|
||||
|
||||
Step = PromptStep | CodeStep | ToolStep
|
||||
@dataclass
|
||||
class McpStep:
|
||||
"""A step that calls an MCP server tool."""
|
||||
server: str # Server name in ~/.cmdforge/mcp.yaml
|
||||
tool: str # Tool name on that server
|
||||
arguments: dict = field(default_factory=dict)
|
||||
output_var: str = "mcp_result"
|
||||
result_mode: str = "auto" # auto | structured | content | text
|
||||
name: Optional[str] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
d = {
|
||||
"type": "mcp",
|
||||
"server": self.server,
|
||||
"tool": self.tool,
|
||||
"output_var": self.output_var,
|
||||
}
|
||||
if self.arguments:
|
||||
d["arguments"] = self.arguments
|
||||
if self.result_mode != "auto":
|
||||
d["result_mode"] = self.result_mode
|
||||
if self.name:
|
||||
d["name"] = self.name
|
||||
return d
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "McpStep":
|
||||
server = data.get("server")
|
||||
tool = data.get("tool")
|
||||
arguments = data.get("arguments", {})
|
||||
output_var = data.get("output_var", "mcp_result")
|
||||
result_mode = data.get("result_mode", "auto")
|
||||
if not isinstance(server, str) or not server.strip():
|
||||
raise ValueError("MCP step server must be a non-empty string")
|
||||
if not isinstance(tool, str) or not tool.strip():
|
||||
raise ValueError("MCP step tool must be a non-empty string")
|
||||
if not isinstance(arguments, dict):
|
||||
raise ValueError("MCP step arguments must be a mapping")
|
||||
if not isinstance(output_var, str) or not output_var.strip():
|
||||
raise ValueError("MCP step output_var must be a non-empty string")
|
||||
if result_mode not in ("auto", "structured", "content", "text"):
|
||||
raise ValueError("MCP step result_mode must be auto, structured, content, or text")
|
||||
return cls(
|
||||
server=server,
|
||||
tool=tool,
|
||||
arguments=arguments,
|
||||
output_var=output_var,
|
||||
result_mode=result_mode,
|
||||
name=data.get("name"),
|
||||
)
|
||||
|
||||
|
||||
Step = PromptStep | CodeStep | ToolStep | McpStep
|
||||
|
||||
@dataclass
|
||||
class ToolSource:
|
||||
@@ -302,10 +466,33 @@ class Tool:
|
||||
source: Optional[ToolSource] = None # Attribution for imported/external tools
|
||||
version: str = "" # Tool version
|
||||
visibility: str = "public" # "public", "private", or "unlisted"
|
||||
input_schema: Optional[dict] = None # JSON Schema for tool input contract
|
||||
output_schema: Optional[dict] = None # JSON Schema for tool output contract
|
||||
path: Optional[Path] = None # Path to config.yaml (set by load_tool)
|
||||
deprecated: bool = False # Tool is deprecated
|
||||
deprecated_message: str = "" # Migration guidance for deprecated tools
|
||||
replacement: Optional[str] = None # Suggested replacement tool name
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
validate_json_schema(self.input_schema, "input_schema")
|
||||
validate_json_schema(self.output_schema, "output_schema")
|
||||
if not isinstance(self.deprecated, bool):
|
||||
raise ValueError("deprecated must be true or false")
|
||||
if not isinstance(self.deprecated_message, str):
|
||||
raise ValueError("deprecated_message must be a string")
|
||||
if len(self.deprecated_message) > 500:
|
||||
raise ValueError("deprecated_message must be at most 500 characters")
|
||||
if self.replacement is not None and (
|
||||
not isinstance(self.replacement, str)
|
||||
or not _TOOL_REF_RE.fullmatch(self.replacement)
|
||||
):
|
||||
raise ValueError("replacement must be a tool name or owner/tool reference")
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: dict) -> "Tool":
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("Tool configuration must be a YAML mapping")
|
||||
|
||||
arguments = []
|
||||
for arg in data.get("arguments", []):
|
||||
arguments.append(ToolArgument.from_dict(arg))
|
||||
@@ -318,11 +505,21 @@ class Tool:
|
||||
steps.append(CodeStep.from_dict(step))
|
||||
elif step.get("type") == "tool":
|
||||
steps.append(ToolStep.from_dict(step))
|
||||
elif step.get("type") == "mcp":
|
||||
steps.append(McpStep.from_dict(step))
|
||||
|
||||
# Parse source attribution if present
|
||||
source = None
|
||||
if "source" in data:
|
||||
source = ToolSource.from_dict(data["source"])
|
||||
source_data = data["source"]
|
||||
if isinstance(source_data, dict):
|
||||
source = ToolSource.from_dict(source_data)
|
||||
elif isinstance(source_data, str) and source_data.strip():
|
||||
source = ToolSource(
|
||||
type="imported", original_tool=source_data.strip()
|
||||
)
|
||||
elif source_data is not None:
|
||||
raise ValueError("source must be an attribution object or string")
|
||||
|
||||
# Normalize dependencies - can be strings or dicts with name/version
|
||||
raw_deps = data.get("dependencies", [])
|
||||
@@ -352,6 +549,11 @@ class Tool:
|
||||
source=source,
|
||||
version=data.get("version", ""),
|
||||
visibility=data.get("visibility", "public"),
|
||||
deprecated=data.get("deprecated", False),
|
||||
deprecated_message=data.get("deprecated_message", ""),
|
||||
replacement=data.get("replacement"),
|
||||
input_schema=data.get("input_schema"),
|
||||
output_schema=data.get("output_schema"),
|
||||
)
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
@@ -367,6 +569,16 @@ class Tool:
|
||||
# Only include visibility if it's not the default
|
||||
if self.visibility and self.visibility != "public":
|
||||
d["visibility"] = self.visibility
|
||||
if self.deprecated:
|
||||
d["deprecated"] = True
|
||||
if self.deprecated_message:
|
||||
d["deprecated_message"] = self.deprecated_message
|
||||
if self.replacement:
|
||||
d["replacement"] = self.replacement
|
||||
if self.input_schema is not None:
|
||||
d["input_schema"] = self.input_schema
|
||||
if self.output_schema is not None:
|
||||
d["output_schema"] = self.output_schema
|
||||
# Include source attribution if present
|
||||
if self.source:
|
||||
d["source"] = self.source.to_dict()
|
||||
@@ -401,6 +613,11 @@ def get_tools_dir() -> Path:
|
||||
return TOOLS_DIR
|
||||
|
||||
|
||||
def get_project_tools_dir(project_dir: Optional[Path] = None) -> Path:
|
||||
"""Return the canonical project-local tool root without creating it."""
|
||||
return (project_dir or Path.cwd()).resolve() / ".cmdforge"
|
||||
|
||||
|
||||
def ensure_settings(tool_dir: Path) -> Optional[Path]:
|
||||
"""Ensure settings.yaml exists if defaults.yaml exists.
|
||||
|
||||
@@ -550,12 +767,32 @@ def load_tool(name: str) -> Optional[Tool]:
|
||||
_REGISTRY_FIELDS = ("registry_hash", "registry_status", "registry_owner", "registry_feedback")
|
||||
|
||||
|
||||
def save_tool(tool: Tool) -> Path:
|
||||
"""Save a tool to disk, preserving registry metadata from existing config."""
|
||||
tool_dir = get_tools_dir() / tool.name
|
||||
def save_tool(
|
||||
tool: Tool,
|
||||
*,
|
||||
tools_dir: Optional[Path] = None,
|
||||
create_wrapper: Optional[bool] = None,
|
||||
) -> Path:
|
||||
"""Save a tool globally or under an explicit project-local tool root.
|
||||
|
||||
Global saves keep the historical wrapper behavior. Project-local saves are
|
||||
invoked through ``cmdforge run`` so a global wrapper cannot accidentally
|
||||
resolve a different project tool with the same name.
|
||||
"""
|
||||
valid, error = validate_tool_name(tool.name)
|
||||
if not valid:
|
||||
raise ValueError(error)
|
||||
global_tools_dir = get_tools_dir()
|
||||
target_root = Path(tools_dir).resolve() if tools_dir is not None else global_tools_dir
|
||||
target_root.mkdir(parents=True, exist_ok=True)
|
||||
tool_dir = target_root / tool.name
|
||||
if tool_dir.is_symlink():
|
||||
raise ValueError(f"Refusing to save through symlinked tool directory: {tool_dir}")
|
||||
tool_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
config_path = tool_dir / "config.yaml"
|
||||
if config_path.is_symlink():
|
||||
raise ValueError(f"Refusing to overwrite symlinked tool config: {config_path}")
|
||||
|
||||
# Preserve registry fields from existing config (not part of Tool model)
|
||||
preserved = {}
|
||||
@@ -572,8 +809,12 @@ def save_tool(tool: Tool) -> Path:
|
||||
new_data.update(preserved)
|
||||
config_path.write_text(yaml.dump(new_data, default_flow_style=False, sort_keys=False))
|
||||
|
||||
# Create wrapper script
|
||||
create_wrapper_script(tool.name)
|
||||
# Global tools retain convenient shell wrappers. Project tools are
|
||||
# intentionally cwd-scoped and must be resolved by ``cmdforge run``.
|
||||
if create_wrapper is None:
|
||||
create_wrapper = target_root == global_tools_dir.resolve()
|
||||
if create_wrapper:
|
||||
create_wrapper_script(tool.name)
|
||||
|
||||
# Ensure settings.yaml exists if defaults.yaml exists
|
||||
ensure_settings(tool_dir)
|
||||
@@ -627,6 +868,16 @@ def tool_exists(name: str) -> bool:
|
||||
return (get_tools_dir() / name / "config.yaml").exists()
|
||||
|
||||
|
||||
def deprecation_warning(tool: Tool) -> Optional[str]:
|
||||
"""Return consistent migration guidance for a deprecated tool."""
|
||||
if not tool.deprecated:
|
||||
return None
|
||||
message = tool.deprecated_message or "This tool is no longer maintained."
|
||||
if tool.replacement:
|
||||
message += f" Use '{tool.replacement}' instead."
|
||||
return f"Warning: '{tool.name}' is deprecated. {message}"
|
||||
|
||||
|
||||
def validate_tool_name(name: str) -> tuple[bool, str]:
|
||||
"""
|
||||
Validate a tool name.
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
"""Opt-in, local-only pipeline usage discovery.
|
||||
|
||||
Only tool names, anonymous pipe inode numbers, counts, and timestamps are
|
||||
stored. Inputs, outputs, arguments, environment, and working directories are
|
||||
never recorded.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import tempfile
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterator, List, Optional
|
||||
|
||||
from .config import CONFIG_DIR
|
||||
|
||||
USAGE_FILE = CONFIG_DIR / "usage.json"
|
||||
USAGE_LOCK_FILE = CONFIG_DIR / ".usage.lock"
|
||||
USAGE_VERSION = 1
|
||||
EVENT_TTL_SECONDS = 120
|
||||
MAX_EVENTS = 200
|
||||
DEFAULT_THRESHOLD = 3
|
||||
|
||||
|
||||
def _empty_document(enabled: bool = False) -> dict:
|
||||
return {
|
||||
"version": USAGE_VERSION,
|
||||
"enabled": enabled,
|
||||
"events": [],
|
||||
"pipelines": {},
|
||||
}
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _locked() -> Iterator[None]:
|
||||
CONFIG_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
with open(USAGE_LOCK_FILE, "a+", encoding="utf-8") as lock:
|
||||
os.chmod(USAGE_LOCK_FILE, 0o600)
|
||||
try:
|
||||
import fcntl
|
||||
fcntl.flock(lock.fileno(), fcntl.LOCK_EX)
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
try:
|
||||
import fcntl
|
||||
fcntl.flock(lock.fileno(), fcntl.LOCK_UN)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
|
||||
def _load_unlocked() -> dict:
|
||||
if not USAGE_FILE.exists():
|
||||
return _empty_document()
|
||||
try:
|
||||
data = json.loads(USAGE_FILE.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return _empty_document()
|
||||
if not isinstance(data, dict) or data.get("version") != USAGE_VERSION:
|
||||
return _empty_document()
|
||||
return data
|
||||
|
||||
|
||||
def _save_unlocked(data: dict) -> None:
|
||||
CONFIG_DIR.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
temp_path = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
"w", encoding="utf-8", dir=CONFIG_DIR,
|
||||
prefix=".usage-", suffix=".tmp", delete=False,
|
||||
) as handle:
|
||||
temp_path = Path(handle.name)
|
||||
os.chmod(temp_path, 0o600)
|
||||
json.dump(data, handle, indent=2, sort_keys=True)
|
||||
handle.write("\n")
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temp_path, USAGE_FILE)
|
||||
os.chmod(USAGE_FILE, 0o600)
|
||||
finally:
|
||||
if temp_path and temp_path.exists():
|
||||
temp_path.unlink()
|
||||
|
||||
|
||||
def is_enabled() -> bool:
|
||||
with _locked():
|
||||
return bool(_load_unlocked().get("enabled", False))
|
||||
|
||||
|
||||
def set_enabled(enabled: bool) -> None:
|
||||
with _locked():
|
||||
data = _load_unlocked()
|
||||
data["enabled"] = bool(enabled)
|
||||
if not enabled:
|
||||
data["events"] = []
|
||||
_save_unlocked(data)
|
||||
|
||||
|
||||
def clear_usage() -> None:
|
||||
with _locked():
|
||||
enabled = bool(_load_unlocked().get("enabled", False))
|
||||
_save_unlocked(_empty_document(enabled=enabled))
|
||||
|
||||
|
||||
def _pipe_inode(stream) -> Optional[int]:
|
||||
try:
|
||||
info = os.fstat(stream.fileno())
|
||||
except (AttributeError, OSError, ValueError):
|
||||
return None
|
||||
return info.st_ino if stat.S_ISFIFO(info.st_mode) else None
|
||||
|
||||
|
||||
def record_invocation(tool_name: str, stdin=None, stdout=None) -> None:
|
||||
"""Record a successful invocation when it participates in a shell pipe."""
|
||||
import sys
|
||||
stdin = stdin or sys.stdin
|
||||
stdout = stdout or sys.stdout
|
||||
input_pipe = _pipe_inode(stdin)
|
||||
output_pipe = _pipe_inode(stdout)
|
||||
if input_pipe is None and output_pipe is None:
|
||||
return
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
timestamp = now.timestamp()
|
||||
with _locked():
|
||||
data = _load_unlocked()
|
||||
if not data.get("enabled", False):
|
||||
return
|
||||
events = [
|
||||
event for event in data.get("events", [])
|
||||
if timestamp - float(event.get("time", 0)) <= EVENT_TTL_SECONDS
|
||||
]
|
||||
discovered = set()
|
||||
for event in events:
|
||||
if input_pipe is not None and event.get("stdout_pipe") == input_pipe:
|
||||
discovered.add((event["tool"], tool_name))
|
||||
if output_pipe is not None and event.get("stdin_pipe") == output_pipe:
|
||||
discovered.add((tool_name, event["tool"]))
|
||||
pipelines = data.setdefault("pipelines", {})
|
||||
for first, second in discovered:
|
||||
if first == second:
|
||||
continue
|
||||
key = json.dumps([first, second], separators=(",", ":"))
|
||||
current = pipelines.setdefault(key, {"tools": [first, second], "count": 0})
|
||||
current["count"] = int(current.get("count", 0)) + 1
|
||||
current["last_seen"] = now.isoformat()
|
||||
events.append({
|
||||
"tool": tool_name,
|
||||
"time": timestamp,
|
||||
"stdin_pipe": input_pipe,
|
||||
"stdout_pipe": output_pipe,
|
||||
})
|
||||
data["events"] = events[-MAX_EVENTS:]
|
||||
_save_unlocked(data)
|
||||
|
||||
|
||||
def get_suggestions(threshold: int = DEFAULT_THRESHOLD) -> List[Dict]:
|
||||
with _locked():
|
||||
data = _load_unlocked()
|
||||
if not data.get("enabled", False):
|
||||
return []
|
||||
suggestions = [
|
||||
{
|
||||
"tools": list(item.get("tools", [])),
|
||||
"count": int(item.get("count", 0)),
|
||||
"last_seen": item.get("last_seen", ""),
|
||||
}
|
||||
for item in data.get("pipelines", {}).values()
|
||||
if int(item.get("count", 0)) >= threshold
|
||||
and len(item.get("tools", [])) >= 2
|
||||
]
|
||||
return sorted(suggestions, key=lambda item: (-item["count"], item["tools"]))
|
||||
|
||||
|
||||
def build_composite_tool(name: str, tools: List[str]):
|
||||
"""Build, but do not save, a Tool that composes the suggested pipeline."""
|
||||
from .tool import Tool, ToolStep
|
||||
if len(tools) < 2:
|
||||
raise ValueError("A composite requires at least two tools")
|
||||
steps = []
|
||||
previous = "input"
|
||||
for index, tool_name in enumerate(tools, start=1):
|
||||
output_var = f"pipeline_{index}"
|
||||
steps.append(ToolStep(
|
||||
tool=tool_name,
|
||||
input_template="{input}" if previous == "input" else f"{{{previous}}}",
|
||||
output_var=output_var,
|
||||
))
|
||||
previous = output_var
|
||||
return Tool(
|
||||
name=name,
|
||||
description="Composite pipeline: " + " | ".join(tools),
|
||||
dependencies=list(dict.fromkeys(tools)),
|
||||
steps=steps,
|
||||
output=f"{{{previous}}}",
|
||||
)
|
||||
@@ -34,6 +34,13 @@ cmdforge create # CLI wizard
|
||||
# Or install a tool from the registry
|
||||
cmdforge registry install official/summarize
|
||||
|
||||
# Discover tools from scripts or an AI coding assistant
|
||||
cmdforge list --json --filter "summarize" --limit 10
|
||||
cmdforge registry search "summarize text" --json --limit 5
|
||||
|
||||
# Run a one-off prompt without creating a tool
|
||||
cat article.txt | cmdforge run-once "Summarize this:\n\n{input}"
|
||||
|
||||
# Use it!
|
||||
cat article.txt | summarize
|
||||
|
||||
@@ -46,6 +53,26 @@ cf</code></pre>
|
||||
<code>cmdforge create</code> uses a command-line wizard. Both create the same YAML config files.</p>
|
||||
</div>
|
||||
|
||||
<h2 id="coding-agents">Using CmdForge with Coding Agents</h2>
|
||||
<p>CmdForge can expose installed tools to supported coding hosts over MCP. Preview the
|
||||
configuration first; CmdForge updates only its marked policy block and does not expose
|
||||
tools that you have not explicitly allowed.</p>
|
||||
<pre><code class="language-bash">cmdforge mcp configure codex --dry-run
|
||||
cmdforge mcp configure codex
|
||||
|
||||
# Claude Code is supported too
|
||||
cmdforge mcp configure claude-code --dry-run</code></pre>
|
||||
<p>For a reusable workflow owned by an application, create it inside that application's
|
||||
<code>.cmdforge/</code> directory. Do not modify the CmdForge source repository merely to
|
||||
add a tool for another project.</p>
|
||||
<pre><code class="language-bash">cd your-project
|
||||
cmdforge create classify-docs --project
|
||||
|
||||
# Or use the official AI-assisted creator
|
||||
cmdforge registry install official/forge-tool
|
||||
echo "Create a bounded document classifier" \
|
||||
| forge-tool --name classify-docs --project</code></pre>
|
||||
|
||||
<h2 id="how-it-works">How It Works</h2>
|
||||
<p>Each tool is a YAML file that defines:</p>
|
||||
<ol>
|
||||
@@ -96,6 +123,7 @@ output: "{summary}"</code></pre>
|
||||
"headings": [
|
||||
("what-is-cmdforge", "What is CmdForge?"),
|
||||
("quick-start", "Quick Start"),
|
||||
("coding-agents", "Using CmdForge with Coding Agents"),
|
||||
("how-it-works", "How It Works"),
|
||||
("next-steps", "Next Steps"),
|
||||
("get-help", "Get Help"),
|
||||
@@ -160,8 +188,13 @@ pip install -e ".[dev]"</code></pre>
|
||||
<p class="lead">Let's create a simple tool that explains code. You'll learn the basics of tool configuration.</p>
|
||||
|
||||
<h2 id="create-tool">Create the Tool</h2>
|
||||
<p>Run the interactive creator:</p>
|
||||
<pre><code class="language-bash">cmdforge create</code></pre>
|
||||
<p>Run the interactive creator. Use <code>--project</code> when the tool belongs to the
|
||||
current application and should be versioned with it:</p>
|
||||
<pre><code class="language-bash"># Personal tool in ~/.cmdforge/
|
||||
cmdforge create explain
|
||||
|
||||
# Project-owned tool in ./.cmdforge/
|
||||
cmdforge create explain --project</code></pre>
|
||||
|
||||
<p>Or create the file manually at <code>~/.cmdforge/explain/config.yaml</code>:</p>
|
||||
<pre><code class="language-yaml">name: explain
|
||||
@@ -550,7 +583,7 @@ from multiple AI providers simultaneously.</p>
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
def run_tool(tool_name: str, input_text: str) -> dict:
|
||||
\"\"\"Run a SmartTool and return its output.\"\"\"
|
||||
\"\"\"Run a CmdForge tool and return its output.\"\"\"
|
||||
result = subprocess.run(
|
||||
[tool_name],
|
||||
input=input_text,
|
||||
@@ -600,7 +633,7 @@ import subprocess
|
||||
import json
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
# Define your perspective tools (each is a SmartTool)
|
||||
# Define your perspective tools (each is a CmdForge tool)
|
||||
PERSPECTIVES = [
|
||||
"perspective-optimist", # Focuses on opportunities
|
||||
"perspective-critic", # Identifies problems
|
||||
@@ -758,10 +791,10 @@ project. It implements:</p>
|
||||
},
|
||||
|
||||
"yaml-config": {
|
||||
"title": "The Anatomy of a SmartTool",
|
||||
"description": "Master the YAML configuration that powers every SmartTool",
|
||||
"title": "The Anatomy of a CmdForge Tool",
|
||||
"description": "Master the YAML configuration that powers every CmdForge tool",
|
||||
"content": """
|
||||
<p class="lead">Every SmartTool is just a YAML file with a secret superpower: it turns plain English
|
||||
<p class="lead">Every CmdForge tool is just a YAML file with a secret superpower: it turns plain English
|
||||
instructions into Unix commands. In the next 10 minutes, you'll understand exactly how that magic works.</p>
|
||||
|
||||
<div class="bg-indigo-50 border-l-4 border-indigo-500 p-4 my-6">
|
||||
@@ -797,7 +830,7 @@ HELLO WORLD!!!</code></pre>
|
||||
|
||||
<h2 id="the-five-parts">The Five Parts of Every Tool</h2>
|
||||
|
||||
<p>Think of a SmartTool config like a recipe card:</p>
|
||||
<p>Think of a CmdForge tool config like a recipe card:</p>
|
||||
|
||||
<div class="grid grid-cols-1 md:grid-cols-2 gap-4 my-6">
|
||||
<div class="bg-white border rounded-lg p-4">
|
||||
@@ -4004,12 +4037,12 @@ cat file.txt | summarize --provider mock</code></pre>
|
||||
<td class="px-4 py-2">Launch the Visual Builder (desktop GUI)</td>
|
||||
</tr>
|
||||
<tr class="border-b">
|
||||
<td class="px-4 py-2"><code>cmdforge list</code></td>
|
||||
<td class="px-4 py-2">List all installed tools</td>
|
||||
<td class="px-4 py-2"><code>cmdforge list [--json] [--filter QUERY]</code></td>
|
||||
<td class="px-4 py-2">List or machine-search installed tools</td>
|
||||
</tr>
|
||||
<tr class="border-b">
|
||||
<td class="px-4 py-2"><code>cmdforge create</code></td>
|
||||
<td class="px-4 py-2">Create a new tool (interactive wizard)</td>
|
||||
<td class="px-4 py-2"><code>cmdforge create NAME [--project]</code></td>
|
||||
<td class="px-4 py-2">Create a personal or project-owned tool</td>
|
||||
</tr>
|
||||
<tr class="border-b">
|
||||
<td class="px-4 py-2"><code>cmdforge edit <tool></code></td>
|
||||
@@ -4027,6 +4060,10 @@ cat file.txt | summarize --provider mock</code></pre>
|
||||
<td class="px-4 py-2"><code>cmdforge run <tool></code></td>
|
||||
<td class="px-4 py-2">Run a tool directly (without wrapper)</td>
|
||||
</tr>
|
||||
<tr class="border-b">
|
||||
<td class="px-4 py-2"><code>cmdforge run-once "Instruction {input}"</code></td>
|
||||
<td class="px-4 py-2">Run an ad-hoc prompt without creating a tool</td>
|
||||
</tr>
|
||||
<tr class="border-b">
|
||||
<td class="px-4 py-2"><code>cmdforge refresh</code></td>
|
||||
<td class="px-4 py-2">Regenerate all wrapper scripts</td>
|
||||
@@ -4301,6 +4338,14 @@ cf | cf | cf</code></pre>
|
||||
},
|
||||
}
|
||||
|
||||
# The original manual remains useful for the fundamentals. Newer architecture
|
||||
# chapters are maintained together so post-M6 features do not become scattered
|
||||
# one-off additions to otherwise unrelated pages. Entries in MODERN_DOCS may
|
||||
# deliberately replace an older page (currently the CLI reference).
|
||||
from .docs_modern import MODERN_DOCS
|
||||
|
||||
DOCS.update(MODERN_DOCS)
|
||||
|
||||
|
||||
def get_doc(path: str) -> dict:
|
||||
"""Get documentation content by path."""
|
||||
@@ -4310,32 +4355,43 @@ def get_doc(path: str) -> dict:
|
||||
|
||||
|
||||
def get_toc():
|
||||
"""Get table of contents structure."""
|
||||
"""Return the documentation as a small, deliberately ordered book."""
|
||||
from types import SimpleNamespace
|
||||
return [
|
||||
SimpleNamespace(slug="getting-started", title="Getting Started", children=[
|
||||
SimpleNamespace(slug="getting-started", title="Start Here", children=[
|
||||
SimpleNamespace(slug="installation", title="Installation"),
|
||||
SimpleNamespace(slug="first-tool", title="Your First Tool"),
|
||||
SimpleNamespace(slug="forge-tool", title="Forge a Tool with AI"),
|
||||
SimpleNamespace(slug="interactive-picker", title="Interactive Picker (cf)"),
|
||||
SimpleNamespace(slug="visual-builder", title="Visual Builder"),
|
||||
SimpleNamespace(slug="yaml-config", title="YAML Config"),
|
||||
]),
|
||||
SimpleNamespace(slug="registry-usage", title="Using the Registry", children=[
|
||||
SimpleNamespace(slug="collections", title="Tool Collections"),
|
||||
]),
|
||||
SimpleNamespace(slug="arguments", title="Custom Arguments", children=[]),
|
||||
SimpleNamespace(slug="multi-step", title="Multi-Step Workflows", children=[
|
||||
SimpleNamespace(slug="yaml-config", title="Building Tools", children=[
|
||||
SimpleNamespace(slug="arguments", title="Arguments and Inputs"),
|
||||
SimpleNamespace(slug="multi-step", title="Multi-Step Workflows"),
|
||||
SimpleNamespace(slug="code-steps", title="Code Steps"),
|
||||
SimpleNamespace(slug="tool-steps", title="Tools Within Tools"),
|
||||
SimpleNamespace(slug="contracts-quality", title="Contracts and Quality"),
|
||||
]),
|
||||
SimpleNamespace(slug="testing-steps", title="Testing Sandbox", children=[]),
|
||||
SimpleNamespace(slug="providers", title="Providers", children=[
|
||||
SimpleNamespace(slug="providers", title="Providers and Delegation", children=[
|
||||
SimpleNamespace(slug="provider-setup", title="Provider Setup"),
|
||||
SimpleNamespace(slug="provider-policy", title="Privacy and Routing Policy"),
|
||||
SimpleNamespace(slug="skills-delegation", title="Skills and Delegation"),
|
||||
]),
|
||||
SimpleNamespace(slug="project-deps", title="Project Dependencies", children=[]),
|
||||
SimpleNamespace(slug="publishing", title="Publishing", children=[]),
|
||||
SimpleNamespace(slug="advanced-workflows", title="Advanced Workflows", children=[
|
||||
SimpleNamespace(slug="mcp-overview", title="MCP and Coding Agents", children=[
|
||||
SimpleNamespace(slug="mcp-client", title="Calling MCP Servers"),
|
||||
SimpleNamespace(slug="mcp-server", title="Serving CmdForge Tools"),
|
||||
SimpleNamespace(slug="agent-integration", title="Coding Agent Integration"),
|
||||
]),
|
||||
SimpleNamespace(slug="registry-usage", title="Discover, Trust, and Share", children=[
|
||||
SimpleNamespace(slug="collections", title="Tool Collections"),
|
||||
SimpleNamespace(slug="publishing", title="Publishing"),
|
||||
SimpleNamespace(slug="trust-publishing", title="Trust and Provenance"),
|
||||
]),
|
||||
SimpleNamespace(slug="project-deps", title="Projects and Advanced Use", children=[
|
||||
SimpleNamespace(slug="testing-steps", title="Testing Sandbox"),
|
||||
SimpleNamespace(slug="optimization-usage", title="Optimization and Usage"),
|
||||
SimpleNamespace(slug="advanced-workflows", title="Advanced Workflows"),
|
||||
SimpleNamespace(slug="parallel-orchestration", title="Parallel Orchestration"),
|
||||
]),
|
||||
SimpleNamespace(slug="cli-reference", title="CLI Reference", children=[]),
|
||||
SimpleNamespace(slug="cli-reference", title="The Command Atlas", children=[]),
|
||||
]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+81
-73
@@ -1,72 +1,103 @@
|
||||
"""Email sending utilities for CmdForge web.
|
||||
"""Transactional email utilities for the CmdForge web application.
|
||||
|
||||
In development mode, emails are logged to the console instead of being sent.
|
||||
To enable real email sending, set MAIL_ENABLED=true and configure SMTP settings.
|
||||
Production delivery uses Resend's HTTPS API. Email bodies and reset URLs are
|
||||
never written to logs. Console delivery is an explicit development-only mode.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from html import escape
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from flask import current_app
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
RESEND_ENDPOINT = "https://api.resend.com/emails"
|
||||
SUPPORTED_TRANSPORTS = {"disabled", "console", "resend"}
|
||||
|
||||
|
||||
def _transport() -> str:
|
||||
transport = str(current_app.config.get("MAIL_TRANSPORT", "disabled")).strip().lower()
|
||||
if transport not in SUPPORTED_TRANSPORTS:
|
||||
logger.error("Unsupported email transport configured; delivery disabled")
|
||||
return "disabled"
|
||||
return transport
|
||||
|
||||
|
||||
def send_email(to: str, subject: str, html_body: str, text_body: Optional[str] = None) -> bool:
|
||||
"""Send an email.
|
||||
"""Deliver an email without logging its recipient or contents.
|
||||
|
||||
In dev mode (MAIL_ENABLED=false or unset), logs the email to console.
|
||||
In production (MAIL_ENABLED=true), sends via SMTP.
|
||||
|
||||
Args:
|
||||
to: Recipient email address
|
||||
subject: Email subject
|
||||
html_body: HTML content of the email
|
||||
text_body: Plain text content (optional)
|
||||
|
||||
Returns:
|
||||
True if email was sent/logged successfully, False otherwise
|
||||
``MAIL_TRANSPORT`` must be one of ``disabled``, ``console``, or ``resend``.
|
||||
Console mode records only that a message was suppressed and reports no
|
||||
delivery, so reset tokens are invalidated. It is also rejected in
|
||||
production. Resend mode requires ``RESEND_API_KEY`` and ``MAIL_FROM``.
|
||||
"""
|
||||
mail_enabled = current_app.config.get("MAIL_ENABLED", False)
|
||||
transport = _transport()
|
||||
environment = str(current_app.config.get("CMDFORGE_ENV", "development")).lower()
|
||||
|
||||
if mail_enabled:
|
||||
# Future: implement real SMTP sending
|
||||
# smtp_host = current_app.config.get("MAIL_SERVER", "localhost")
|
||||
# smtp_port = current_app.config.get("MAIL_PORT", 587)
|
||||
# smtp_user = current_app.config.get("MAIL_USERNAME")
|
||||
# smtp_pass = current_app.config.get("MAIL_PASSWORD")
|
||||
# smtp_tls = current_app.config.get("MAIL_USE_TLS", True)
|
||||
logger.warning("MAIL_ENABLED is true but SMTP is not implemented yet. Falling back to console logging.")
|
||||
if transport == "disabled":
|
||||
logger.warning("Email delivery is disabled")
|
||||
return False
|
||||
|
||||
# Dev mode: log to console
|
||||
logger.info("=" * 60)
|
||||
logger.info("[EMAIL] To: %s", to)
|
||||
logger.info("[EMAIL] Subject: %s", subject)
|
||||
logger.info("[EMAIL] Body:")
|
||||
if transport == "console":
|
||||
if environment == "production":
|
||||
logger.error("Console email transport is not permitted in production")
|
||||
return False
|
||||
logger.info("Development email suppressed (subject length=%d)", len(subject))
|
||||
return False
|
||||
|
||||
api_key = str(current_app.config.get("RESEND_API_KEY") or "").strip()
|
||||
sender = str(current_app.config.get("MAIL_FROM") or "").strip()
|
||||
if not api_key or not sender:
|
||||
logger.error("Resend email transport is missing required configuration")
|
||||
return False
|
||||
|
||||
try:
|
||||
timeout = float(current_app.config.get("MAIL_TIMEOUT", 10.0))
|
||||
except (TypeError, ValueError):
|
||||
logger.error("Invalid email timeout configured")
|
||||
return False
|
||||
if timeout <= 0 or timeout > 60:
|
||||
logger.error("Email timeout must be greater than zero and at most 60 seconds")
|
||||
return False
|
||||
|
||||
payload = {
|
||||
"from": sender,
|
||||
"to": [to],
|
||||
"subject": subject,
|
||||
"html": html_body,
|
||||
}
|
||||
if text_body:
|
||||
logger.info(text_body)
|
||||
else:
|
||||
logger.info(html_body)
|
||||
logger.info("=" * 60)
|
||||
payload["text"] = text_body
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
RESEND_ENDPOINT,
|
||||
headers={
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json=payload,
|
||||
timeout=timeout,
|
||||
)
|
||||
response.raise_for_status()
|
||||
except requests.RequestException as exc:
|
||||
# Do not log the response body, recipient, request headers, or message;
|
||||
# providers can echo sensitive request data in error responses.
|
||||
logger.error("Transactional email delivery failed (%s)", type(exc).__name__)
|
||||
return False
|
||||
|
||||
logger.info("Transactional email accepted by provider")
|
||||
return True
|
||||
|
||||
|
||||
def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
|
||||
"""Send a password reset email.
|
||||
|
||||
Args:
|
||||
to: Recipient email address
|
||||
token: The password reset token
|
||||
base_url: Base URL of the application (e.g., https://cmdforge.brrd.tech)
|
||||
|
||||
Returns:
|
||||
True if email was sent/logged successfully
|
||||
"""
|
||||
"""Send a one-hour password reset link using the configured transport."""
|
||||
reset_url = f"{base_url.rstrip('/')}/reset-password?token={token}"
|
||||
|
||||
html_reset_url = escape(reset_url, quote=True)
|
||||
subject = "Reset your CmdForge password"
|
||||
|
||||
html_body = f"""
|
||||
@@ -86,28 +117,15 @@ def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<div class="header">
|
||||
<div class="logo">CmdForge</div>
|
||||
</div>
|
||||
|
||||
<div class="header"><div class="logo">CmdForge</div></div>
|
||||
<p>Hello,</p>
|
||||
|
||||
<p>You requested to reset your password for your CmdForge account. Click the button below to set a new password:</p>
|
||||
|
||||
<p style="text-align: center; margin: 30px 0;">
|
||||
<a href="{reset_url}" class="button">Reset Password</a>
|
||||
</p>
|
||||
|
||||
<p style="text-align: center; margin: 30px 0;"><a href="{html_reset_url}" class="button">Reset Password</a></p>
|
||||
<p>Or copy and paste this link into your browser:</p>
|
||||
<p class="link">{reset_url}</p>
|
||||
|
||||
<p class="link">{html_reset_url}</p>
|
||||
<p><strong>This link will expire in 1 hour.</strong></p>
|
||||
|
||||
<p>If you didn't request this password reset, you can safely ignore this email. Your password will remain unchanged.</p>
|
||||
|
||||
<div class="footer">
|
||||
<p>This email was sent by CmdForge. If you have questions, visit our website or contact support.</p>
|
||||
</div>
|
||||
<div class="footer"><p>This email was sent by CmdForge.</p></div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -115,20 +133,10 @@ def send_password_reset_email(to: str, token: str, base_url: str) -> bool:
|
||||
|
||||
text_body = f"""Reset your CmdForge password
|
||||
|
||||
Hello,
|
||||
|
||||
You requested to reset your password for your CmdForge account.
|
||||
|
||||
Reset your password by visiting this link:
|
||||
Visit this link to set a new password:
|
||||
{reset_url}
|
||||
|
||||
This link will expire in 1 hour.
|
||||
|
||||
If you didn't request this password reset, you can safely ignore this email.
|
||||
Your password will remain unchanged.
|
||||
|
||||
---
|
||||
CmdForge
|
||||
This link will expire in 1 hour. If you did not request this reset, ignore this email.
|
||||
"""
|
||||
|
||||
return send_email(to, subject, html_body, text_body)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{% block title %}CmdForge{% endblock %} - Build Custom AI Commands</title>
|
||||
<title>{% block title %}CmdForge{% endblock %} - Build AI Tools You Own</title>
|
||||
|
||||
<!-- Meta tags -->
|
||||
<meta name="description" content="{% block meta_description %}Create Unix-style pipeable tools that work with any AI provider. Provider-agnostic, composable, and community-driven.{% endblock %}">
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
<!-- Open Graph -->
|
||||
<meta property="og:title" content="{% block og_title %}CmdForge{% endblock %}">
|
||||
<meta property="og:description" content="{% block og_description %}Build custom AI commands in YAML{% endblock %}">
|
||||
<meta property="og:description" content="{% block og_description %}Build, compose, test, and share AI tools that run on your computer{% endblock %}">
|
||||
<meta property="og:type" content="website">
|
||||
<meta property="og:url" content="{{ request.url }}">
|
||||
{% block og_extra %}{% endblock %}
|
||||
@@ -19,7 +19,7 @@
|
||||
<!-- Twitter Card -->
|
||||
<meta name="twitter:card" content="summary_large_image">
|
||||
<meta name="twitter:title" content="{% block twitter_title %}CmdForge{% endblock %}">
|
||||
<meta name="twitter:description" content="{% block twitter_description %}Build custom AI commands in YAML{% endblock %}">
|
||||
<meta name="twitter:description" content="{% block twitter_description %}Build, compose, test, and share AI tools that run on your computer{% endblock %}">
|
||||
{% block twitter_extra %}{% endblock %}
|
||||
|
||||
<!-- Canonical URL -->
|
||||
@@ -42,7 +42,7 @@
|
||||
"@type": "Organization",
|
||||
"name": "CmdForge",
|
||||
"url": "{{ request.host_url }}",
|
||||
"description": "Build custom AI commands in YAML"
|
||||
"description": "Build, compose, test, and share AI tools that run on your computer"
|
||||
}
|
||||
</script>
|
||||
{% endblock %}
|
||||
|
||||
@@ -3,31 +3,31 @@
|
||||
{% from "components/tutorial_card.html" import tutorial_card %}
|
||||
{% from "components/contributor_card.html" import contributor_card %}
|
||||
|
||||
{% block title %}CmdForge - Compose AI Capabilities{% endblock %}
|
||||
{% block title %}CmdForge - Build AI Tools You Own{% endblock %}
|
||||
|
||||
{% block meta_description %}Stop searching for libraries. Ask for capabilities. CmdForge lets you describe what you need and compose AI-powered tools that work together.{% endblock %}
|
||||
{% block meta_description %}Turn useful AI workflows into commands you own: local or cloud providers, tested pipelines, Unix pipes, registry sharing, and MCP for coding agents.{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<!-- Hero Section -->
|
||||
<section class="bg-white py-16 md:py-24">
|
||||
<div class="max-w-5xl mx-auto px-4 sm:px-6 lg:px-8 text-center">
|
||||
<p class="text-indigo-600 font-medium mb-4">From Libraries to Capabilities</p>
|
||||
<p class="text-indigo-600 font-medium mb-4">Your Models. Your Workflows. Your Computer.</p>
|
||||
<h1 class="text-4xl md:text-5xl font-bold text-gray-900 leading-tight">
|
||||
Ask for Capabilities.<br>Compose Solutions.
|
||||
Turn AI Workflows<br>Into Tools You Own.
|
||||
</h1>
|
||||
<p class="mt-6 text-xl text-gray-600 max-w-3xl mx-auto">
|
||||
Stop searching through documentation. Describe what you need in plain English
|
||||
and discover AI-powered tools that work together like Unix pipes.
|
||||
Build one useful command, connect it to another, and keep the result on your machine.
|
||||
CmdForge makes AI capabilities composable like Unix pipes—and callable from coding agents through MCP.
|
||||
</p>
|
||||
|
||||
<!-- Semantic Search Demo -->
|
||||
<div class="mt-10 max-w-2xl mx-auto">
|
||||
<div class="bg-gray-900 rounded-lg p-4 text-left font-mono text-sm">
|
||||
<p class="text-gray-400 mb-2"># Ask for what you need:</p>
|
||||
<p class="text-green-400">$ cmdforge registry describe "something that can summarize long documents"</p>
|
||||
<p class="text-gray-300 mt-2">Found 3 matching tools:</p>
|
||||
<p class="text-cyan-400"> rob/summarize</p>
|
||||
<p class="text-gray-500 ml-4">Condenses text while preserving key points</p>
|
||||
<p class="text-gray-400 mb-2"># Improvise once. Keep what becomes useful.</p>
|
||||
<p class="text-green-400">$ cat meeting.txt | cmdforge run-once \</p>
|
||||
<p class="text-green-400 ml-4">"Extract decisions and owners from {input}"</p>
|
||||
<p class="text-gray-300 mt-3"># Reusable tomorrow—from your shell or your coding agent:</p>
|
||||
<p class="text-cyan-400">$ cat meeting.txt | meeting-decisions</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -62,11 +62,10 @@
|
||||
<section class="py-16 bg-gray-50">
|
||||
<div class="max-w-6xl mx-auto px-4 sm:px-6 lg:px-8">
|
||||
<h2 class="text-3xl font-bold text-gray-900 text-center mb-4">
|
||||
A New Way to Build
|
||||
Small Tools, Serious Leverage
|
||||
</h2>
|
||||
<p class="text-center text-gray-600 mb-12 max-w-2xl mx-auto">
|
||||
Traditional development: search docs, install libraries, write glue code.<br>
|
||||
CmdForge: describe what you need, compose existing capabilities, ship.
|
||||
Start with a command that solves one problem. Add contracts, composition, and sharing only when the work earns it.
|
||||
</p>
|
||||
|
||||
<div class="grid grid-cols-1 md:grid-cols-3 gap-8">
|
||||
@@ -77,10 +76,10 @@
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8 10h.01M12 10h.01M16 10h.01M9 16H5a2 2 0 01-2-2V6a2 2 0 012-2h14a2 2 0 012 2v8a2 2 0 01-2 2h-5l-5 5v-5z"/>
|
||||
</svg>
|
||||
</div>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Ask for Capabilities</h3>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Discover or Create</h3>
|
||||
<p class="text-gray-600">
|
||||
Semantic search finds tools by what they <em>do</em>, not what they're named.
|
||||
"I need something that extracts key points" just works.
|
||||
Search local tools and the registry by what they do. For a new idea, forge a
|
||||
personal or project-owned tool from plain language.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -91,10 +90,10 @@
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 5a1 1 0 011-1h14a1 1 0 011 1v2a1 1 0 01-1 1H5a1 1 0 01-1-1V5zM4 13a1 1 0 011-1h6a1 1 0 011 1v6a1 1 0 01-1 1H5a1 1 0 01-1-1v-6zM16 13a1 1 0 011-1h2a1 1 0 011 1v6a1 1 0 01-1 1h-2a1 1 0 01-1-1v-6z"/>
|
||||
</svg>
|
||||
</div>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Compose Solutions</h3>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Compose and Verify</h3>
|
||||
<p class="text-gray-600">
|
||||
Tools call other tools. Chain capabilities together like Unix pipes.
|
||||
Build complex workflows from simple, tested components.
|
||||
Join prompts, Python, CmdForge tools, and MCP calls. Contracts, preflight,
|
||||
regression evidence, and quality coverage keep the result inspectable.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -105,10 +104,10 @@
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8.684 13.342C8.886 12.938 9 12.482 9 12c0-.482-.114-.938-.316-1.342m0 2.684a3 3 0 110-2.684m0 2.684l6.632 3.316m-6.632-6l6.632-3.316m0 0a3 3 0 105.367-2.684 3 3 0 00-5.367 2.684zm0 9.316a3 3 0 105.368 2.684 3 3 0 00-5.368-2.684z"/>
|
||||
</svg>
|
||||
</div>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Share & Discover</h3>
|
||||
<h3 class="text-xl font-semibold text-gray-900 mb-3">Use It Everywhere</h3>
|
||||
<p class="text-gray-600">
|
||||
Every tool you publish becomes a capability others can use.
|
||||
Collaboration over competition. Build on each other's progress.
|
||||
Run tools as ordinary commands, compose them in projects, publish immutable
|
||||
versions, or expose a deliberate allowlist to Codex and Claude Code over MCP.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -156,10 +155,12 @@
|
||||
<div class="bg-gray-900 rounded-lg p-4 font-mono text-sm">
|
||||
<p class="text-gray-400"># Tools can call other tools:</p>
|
||||
<p class="text-yellow-400">steps:</p>
|
||||
<p class="text-gray-300"> - tool: official/code-review</p>
|
||||
<p class="text-gray-300"> - type: tool</p>
|
||||
<p class="text-gray-300"> tool: official/code-review</p>
|
||||
<p class="text-gray-300"> output_var: review</p>
|
||||
<p class="text-gray-300"> - tool: rob/summarize</p>
|
||||
<p class="text-gray-300"> input_template: "{review}"</p>
|
||||
<p class="text-gray-300"> - type: tool</p>
|
||||
<p class="text-gray-300"> tool: rob/summarize</p>
|
||||
<p class="text-gray-300"> input: "{review}"</p>
|
||||
<p class="text-gray-300"> output_var: summary</p>
|
||||
<p class="text-gray-400 mt-2"># Use like any Unix command:</p>
|
||||
<p class="text-green-400">$ cat app.py | my-review-tool</p>
|
||||
@@ -180,8 +181,8 @@
|
||||
</svg>
|
||||
</div>
|
||||
<div>
|
||||
<h3 class="text-lg font-semibold text-gray-900">Works With Any AI</h3>
|
||||
<p class="text-sm text-gray-600">Claude, GPT, Ollama, or any CLI-accessible model. Switch providers without changing tools.</p>
|
||||
<h3 class="text-lg font-semibold text-gray-900">Choose the Engine at Runtime</h3>
|
||||
<p class="text-sm text-gray-600">Local models, coding CLIs, and OpenAI-compatible APIs—with explicit privacy, capability, fallback, and provenance controls.</p>
|
||||
</div>
|
||||
</div>
|
||||
<a href="{{ url_for('web.docs', path='providers') }}" class="text-indigo-600 hover:text-indigo-800 font-medium flex items-center whitespace-nowrap">
|
||||
|
||||
Vendored
+44
@@ -0,0 +1,44 @@
|
||||
"""Small real MCP stdio server used by the client integration tests."""
|
||||
|
||||
import os
|
||||
import time
|
||||
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
|
||||
|
||||
server = FastMCP("CmdForge MCP test server")
|
||||
|
||||
|
||||
@server.tool()
|
||||
def echo(message: str) -> str:
|
||||
"""Return a message unchanged."""
|
||||
return message
|
||||
|
||||
|
||||
@server.tool()
|
||||
def add(a: int, b: int) -> dict[str, int]:
|
||||
"""Add two integers."""
|
||||
return {"sum": a + b}
|
||||
|
||||
|
||||
@server.tool()
|
||||
def read_env(name: str) -> str:
|
||||
"""Return an environment value, or an empty string if absent."""
|
||||
return os.environ.get(name, "")
|
||||
|
||||
|
||||
@server.tool()
|
||||
def current_directory() -> str:
|
||||
"""Return the server process working directory."""
|
||||
return os.getcwd()
|
||||
|
||||
|
||||
@server.tool()
|
||||
def wait(seconds: float) -> str:
|
||||
"""Block for the requested duration."""
|
||||
time.sleep(seconds)
|
||||
return "done"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
server.run(transport="stdio")
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Tests for M9.5 supply chain attestation."""
|
||||
|
||||
from cmdforge.attestation import (
|
||||
Attestation,
|
||||
generate_keypair,
|
||||
sign_tool,
|
||||
verify_attestation,
|
||||
verify_content_hash,
|
||||
verify_trusted_attestation,
|
||||
)
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
|
||||
class TestSignTool:
|
||||
def test_creates_attestation(self):
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
|
||||
assert att.tool_name == "mytool"
|
||||
assert att.version == "1.0.0"
|
||||
assert att.content_hash == "abc123"
|
||||
assert att.signer == "alice"
|
||||
assert len(att.signature) == 88
|
||||
assert att.algorithm == "ed25519"
|
||||
assert verify_attestation(att, public)
|
||||
|
||||
def test_different_keys_different_signatures(self):
|
||||
key1, _ = generate_keypair()
|
||||
key2, _ = generate_keypair()
|
||||
att1 = sign_tool("tool", "1.0.0", "hash", "alice", key1)
|
||||
att2 = sign_tool("tool", "1.0.0", "hash", "alice", key2)
|
||||
assert att1.signature != att2.signature
|
||||
|
||||
|
||||
class TestVerifyAttestation:
|
||||
def test_valid_signature(self):
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
|
||||
assert verify_attestation(att, public)
|
||||
|
||||
def test_wrong_key_fails(self):
|
||||
private, _ = generate_keypair()
|
||||
_, wrong_public = generate_keypair()
|
||||
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
|
||||
assert not verify_attestation(att, wrong_public)
|
||||
|
||||
def test_tampered_content_fails(self):
|
||||
from dataclasses import replace
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
|
||||
att = replace(att, content_hash="tampered")
|
||||
assert not verify_attestation(att, public)
|
||||
|
||||
def test_tampered_signer_fails(self):
|
||||
from dataclasses import replace
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("mytool", "1.0.0", "abc123", "alice", private)
|
||||
att = replace(att, signer="eve")
|
||||
assert not verify_attestation(att, public)
|
||||
|
||||
def test_timestamp_and_algorithm_are_signed(self):
|
||||
from dataclasses import replace
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("tool", "1.0.0", "hash", "alice", private)
|
||||
assert not verify_attestation(replace(att, signed_at="tomorrow"), public)
|
||||
assert not verify_attestation(replace(att, algorithm="hmac-sha256"), public)
|
||||
|
||||
def test_requires_trusted_publisher_mapping(self):
|
||||
private, public = generate_keypair()
|
||||
att = sign_tool("tool", "1.0.0", "hash", "alice", private)
|
||||
assert verify_trusted_attestation(att, {"alice": public})
|
||||
assert not verify_trusted_attestation(att, {"mallory": public})
|
||||
+710
-1
@@ -1,8 +1,10 @@
|
||||
"""Tests for CLI commands."""
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch, MagicMock
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import call, patch, MagicMock
|
||||
from io import StringIO
|
||||
|
||||
import pytest
|
||||
@@ -32,6 +34,28 @@ class TestCLIBasics:
|
||||
|
||||
assert exc_info.value.code == 0
|
||||
|
||||
def test_mcp_configure_dispatches_host_options(self):
|
||||
with (
|
||||
patch(
|
||||
"cmdforge.agent_setup.configure_agent_host",
|
||||
return_value=0,
|
||||
) as configure,
|
||||
patch("sys.argv", [
|
||||
"cmdforge", "mcp", "configure", "claude-code",
|
||||
"--scope", "project", "--dry-run",
|
||||
]),
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
configure.assert_called_once_with(
|
||||
"claude-code",
|
||||
server_name="cmdforge",
|
||||
scope="project",
|
||||
install_policy=True,
|
||||
dry_run=True,
|
||||
force=False,
|
||||
)
|
||||
|
||||
|
||||
class TestListCommand:
|
||||
"""Tests for 'cmdforge list' command."""
|
||||
@@ -66,6 +90,209 @@ class TestListCommand:
|
||||
assert 'test-tool' in captured.out
|
||||
assert 'another-tool' in captured.out
|
||||
|
||||
def test_json_is_compact_catalog_unless_full_requested(
|
||||
self, temp_tools_dir, capsys
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(
|
||||
name="catalog-tool",
|
||||
description="Catalog entry",
|
||||
input_schema={"type": "string"},
|
||||
output_schema={"type": "string"},
|
||||
steps=[PromptStep(
|
||||
prompt="Private implementation {input}",
|
||||
provider="mock",
|
||||
output_var="result",
|
||||
plain_text=True,
|
||||
)],
|
||||
output="{result}",
|
||||
))
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--json']):
|
||||
assert main() == 0
|
||||
compact = json.loads(capsys.readouterr().out)
|
||||
assert compact[0]["name"] == "catalog-tool"
|
||||
assert compact[0]["input_schema"] == {"type": "string"}
|
||||
assert "steps" not in compact[0]
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--json', '--full']):
|
||||
assert main() == 0
|
||||
full = json.loads(capsys.readouterr().out)
|
||||
assert full[0]["steps"][0]["prompt"] == "Private implementation {input}"
|
||||
|
||||
def test_filter_reports_filtered_count_and_empty_result(
|
||||
self, temp_tools_dir, capsys
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(name="alpha", description="First"))
|
||||
save_tool(Tool(name="beta", description="Second"))
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--filter', 'alpha']):
|
||||
assert main() == 0
|
||||
output = capsys.readouterr().out
|
||||
assert "Available tools (1)" in output
|
||||
assert "alpha" in output
|
||||
assert "beta" not in output
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--filter', 'missing']):
|
||||
assert main() == 0
|
||||
assert "No tools match filter 'missing'" in capsys.readouterr().out
|
||||
|
||||
def test_json_limit_and_invalid_limit(self, temp_tools_dir, capsys):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(name="alpha"))
|
||||
save_tool(Tool(name="beta"))
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--json', '--limit', '1']):
|
||||
assert main() == 0
|
||||
assert len(json.loads(capsys.readouterr().out)) == 1
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--limit', '0']):
|
||||
assert main() == 1
|
||||
assert "at least 1" in capsys.readouterr().err
|
||||
|
||||
with patch('sys.argv', ['cmdforge', 'list', '--full']):
|
||||
assert main() == 1
|
||||
assert "--full requires --json" in capsys.readouterr().err
|
||||
|
||||
def test_list_includes_project_tools_and_prefers_project_shadow(
|
||||
self, temp_tools_dir, tmp_path, monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(name="shadowed", description="global"))
|
||||
project = tmp_path / "project"
|
||||
project.mkdir()
|
||||
save_tool(
|
||||
Tool(name="shadowed", description="project"),
|
||||
tools_dir=project / ".cmdforge",
|
||||
)
|
||||
save_tool(
|
||||
Tool(name="project-only", description="local"),
|
||||
tools_dir=project / ".cmdforge",
|
||||
)
|
||||
monkeypatch.chdir(project)
|
||||
|
||||
with patch("sys.argv", ["cmdforge", "list", "--json"]):
|
||||
assert main() == 0
|
||||
|
||||
payload = json.loads(capsys.readouterr().out)
|
||||
by_name = {item["name"]: item for item in payload}
|
||||
assert by_name["shadowed"]["description"] == "project"
|
||||
assert "project-only" in by_name
|
||||
assert [item["name"] for item in payload].count("shadowed") == 1
|
||||
|
||||
|
||||
class TestRunOnceCommand:
|
||||
def test_combines_instruction_with_piped_input(self, capsys):
|
||||
result = SimpleNamespace(success=True, text="summary", error=None)
|
||||
with (
|
||||
patch('sys.argv', [
|
||||
'cmdforge', 'run-once', 'Summarize:\n{input}',
|
||||
'--provider', 'mock',
|
||||
]),
|
||||
patch('sys.stdin', StringIO('document body')),
|
||||
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
assert call_provider.call_args.args[:2] == (
|
||||
'mock', 'Summarize:\ndocument body'
|
||||
)
|
||||
assert capsys.readouterr().out == "summary\n"
|
||||
|
||||
def test_appends_piped_input_when_template_has_no_placeholder(self):
|
||||
result = SimpleNamespace(success=True, text="ok", error=None)
|
||||
with (
|
||||
patch('sys.argv', [
|
||||
'cmdforge', 'run-once', 'Summarize this', '--provider', 'mock',
|
||||
]),
|
||||
patch('sys.stdin', StringIO('document body')),
|
||||
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
assert call_provider.call_args.args[1] == "Summarize this\n\ndocument body"
|
||||
|
||||
def test_uses_configured_default_provider(self):
|
||||
result = SimpleNamespace(success=True, text="ok", error=None)
|
||||
config = SimpleNamespace(default_provider="configured-provider")
|
||||
stdin = MagicMock()
|
||||
stdin.isatty.return_value = True
|
||||
with (
|
||||
patch('sys.argv', ['cmdforge', 'run-once', 'hello']),
|
||||
patch('sys.stdin', stdin),
|
||||
patch('cmdforge.config.load_config', return_value=config),
|
||||
patch('cmdforge.providers.call_provider', return_value=result) as call_provider,
|
||||
):
|
||||
assert main() == 0
|
||||
assert call_provider.call_args.args[0] == "configured-provider"
|
||||
|
||||
def test_selects_first_available_balanced_provider(self):
|
||||
from cmdforge.cli.tool_commands import _resolve_one_shot_provider
|
||||
from cmdforge.providers import Provider
|
||||
|
||||
config = SimpleNamespace(default_provider=None)
|
||||
providers = [
|
||||
Provider("opencode-pickle", "missing-opencode"),
|
||||
Provider("codex", "codex exec -"),
|
||||
]
|
||||
with (
|
||||
patch('cmdforge.config.load_config', return_value=config),
|
||||
patch('cmdforge.providers.load_providers', return_value=providers),
|
||||
patch(
|
||||
'cmdforge.cli.tool_commands._provider_is_available',
|
||||
side_effect=lambda provider: provider.name == "codex",
|
||||
),
|
||||
):
|
||||
assert _resolve_one_shot_provider() == "codex"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"arguments,message",
|
||||
[
|
||||
(["--timeout", "0"], "--timeout"),
|
||||
(["--max-tokens", "-1"], "--max-tokens"),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_limits_before_provider_call(
|
||||
self, arguments, message, capsys
|
||||
):
|
||||
with (
|
||||
patch('sys.argv', [
|
||||
'cmdforge', 'run-once', 'hello', '--provider', 'mock', *arguments,
|
||||
]),
|
||||
patch('sys.stdin', StringIO()),
|
||||
patch('cmdforge.providers.call_provider') as call_provider,
|
||||
):
|
||||
assert main() == 1
|
||||
call_provider.assert_not_called()
|
||||
assert message in capsys.readouterr().err
|
||||
|
||||
def test_json_envelope_contains_runtime_provenance(self, capsys):
|
||||
from cmdforge.providers import ProviderResult
|
||||
|
||||
result = ProviderResult(
|
||||
text="answer", success=True,
|
||||
requested_provider="local", actual_provider="local",
|
||||
attempted_providers=["local"], locality="local",
|
||||
)
|
||||
with (
|
||||
patch("sys.argv", [
|
||||
"cmdforge", "run-once", "hello", "--provider", "local",
|
||||
"--no-fallback", "--result-envelope", "json",
|
||||
]),
|
||||
patch("sys.stdin", StringIO()),
|
||||
patch("cmdforge.providers.call_provider", return_value=result),
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
payload = json.loads(capsys.readouterr().out)
|
||||
call = payload["execution"]["provider_calls"][0]
|
||||
assert call["actual_provider"] == "local"
|
||||
assert call["fallback_used"] is False
|
||||
|
||||
|
||||
class TestCreateCommand:
|
||||
"""Tests for 'cmdforge create' command."""
|
||||
@@ -119,6 +346,36 @@ class TestCreateCommand:
|
||||
captured = capsys.readouterr()
|
||||
assert 'invalid' in captured.out.lower() or 'invalid' in captured.err.lower()
|
||||
|
||||
def test_create_project_tool_does_not_create_global_wrapper(
|
||||
self, temp_tools_dir, tmp_path, monkeypatch, capsys
|
||||
):
|
||||
project = tmp_path / "project"
|
||||
project.mkdir()
|
||||
monkeypatch.chdir(project)
|
||||
|
||||
with patch("sys.argv", [
|
||||
"cmdforge", "create", "project-echo", "--project",
|
||||
"--prompt", "Echo {input}", "--provider", "mock",
|
||||
]):
|
||||
assert main() == 0
|
||||
|
||||
config = project / ".cmdforge" / "project-echo" / "config.yaml"
|
||||
assert config.exists()
|
||||
assert not (tmp_path / ".cmdforge" / "project-echo" / "config.yaml").exists()
|
||||
assert not (tmp_path / ".local" / "bin" / "project-echo").exists()
|
||||
assert "cmdforge run project-echo" in capsys.readouterr().out
|
||||
|
||||
def test_create_output_dir_uses_explicit_tool_root(
|
||||
self, temp_tools_dir, tmp_path
|
||||
):
|
||||
target = tmp_path / "portable-tools"
|
||||
with patch("sys.argv", [
|
||||
"cmdforge", "create", "portable", "--output-dir", str(target),
|
||||
]):
|
||||
assert main() == 0
|
||||
|
||||
assert (target / "portable" / "config.yaml").exists()
|
||||
|
||||
|
||||
class TestDeleteCommand:
|
||||
"""Tests for 'cmdforge delete' command."""
|
||||
@@ -228,6 +485,91 @@ class TestRunCommand:
|
||||
|
||||
assert result != 0
|
||||
|
||||
def test_run_prefers_project_local_tool(
|
||||
self, temp_tools_dir, tmp_path, monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(name="shadowed", output="global"))
|
||||
project = tmp_path / "project"
|
||||
project.mkdir()
|
||||
save_tool(
|
||||
Tool(name="shadowed", output="project"),
|
||||
tools_dir=project / ".cmdforge",
|
||||
)
|
||||
monkeypatch.chdir(project)
|
||||
|
||||
with (
|
||||
patch("sys.argv", ["cmdforge", "run", "shadowed"]),
|
||||
patch("sys.stdin", StringIO("")),
|
||||
patch("sys.stdin.isatty", return_value=True),
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
assert capsys.readouterr().out.strip() == "project"
|
||||
|
||||
def test_run_json_envelope_preserves_output_and_provenance(
|
||||
self, temp_tools_dir, capsys
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(
|
||||
name="audited",
|
||||
steps=[PromptStep(
|
||||
prompt="Echo {input}", provider="mock",
|
||||
output_var="answer", plain_text=True,
|
||||
)],
|
||||
output="{answer}",
|
||||
))
|
||||
|
||||
with (
|
||||
patch("sys.argv", [
|
||||
"cmdforge", "run", "audited", "--result-envelope", "json",
|
||||
]),
|
||||
patch("sys.stdin", StringIO("hello")),
|
||||
patch("sys.stdin.isatty", return_value=False),
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
payload = json.loads(capsys.readouterr().out)
|
||||
assert "MOCK" in payload["output"]
|
||||
assert payload["execution"]["actual_providers"] == ["mock"]
|
||||
assert payload["execution"]["provider_calls"][0]["locality"] == "local"
|
||||
|
||||
def test_failed_run_still_emits_audit_envelope(
|
||||
self, temp_tools_dir, capsys
|
||||
):
|
||||
from cmdforge.providers import ProviderResult
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
save_tool(Tool(
|
||||
name="strict",
|
||||
steps=[PromptStep(
|
||||
prompt="Private", provider="primary",
|
||||
output_var="answer", plain_text=True,
|
||||
)],
|
||||
output="{answer}",
|
||||
))
|
||||
failure = ProviderResult(
|
||||
text="", success=False, error="offline",
|
||||
requested_provider="primary", attempted_providers=["primary"],
|
||||
)
|
||||
with (
|
||||
patch("sys.argv", [
|
||||
"cmdforge", "run", "strict", "--no-fallback",
|
||||
"--result-envelope", "json",
|
||||
]),
|
||||
patch("sys.stdin", StringIO("private")),
|
||||
patch("sys.stdin.isatty", return_value=False),
|
||||
patch("cmdforge.runner.call_provider", return_value=failure),
|
||||
):
|
||||
assert main() == 2
|
||||
|
||||
payload = json.loads(capsys.readouterr().out)
|
||||
assert payload["output"] == ""
|
||||
assert payload["execution"]["exit_code"] == 2
|
||||
assert payload["execution"]["provider_calls"][0]["error"] == "offline"
|
||||
|
||||
|
||||
class TestTestCommand:
|
||||
"""Tests for 'cmdforge test' command."""
|
||||
@@ -281,6 +623,29 @@ class TestProvidersCommand:
|
||||
# Should show some default providers
|
||||
assert 'mock' in captured.out.lower() or 'claude' in captured.out.lower()
|
||||
|
||||
def test_providers_list_json_exposes_routing_metadata(
|
||||
self, temp_providers_file, capsys
|
||||
):
|
||||
from cmdforge.providers import Provider, save_providers
|
||||
|
||||
save_providers([Provider(
|
||||
"safe", "safe-cli", locality="local",
|
||||
capabilities=["structured-json"], data_policy="private",
|
||||
max_context_tokens=8192,
|
||||
)])
|
||||
with (
|
||||
patch("sys.argv", ["cmdforge", "providers", "list", "--json"]),
|
||||
patch("cmdforge.cli.provider_commands._provider_status", return_value=(True, "Ready")),
|
||||
):
|
||||
assert main() == 0
|
||||
|
||||
payload = json.loads(capsys.readouterr().out)
|
||||
assert payload[0]["locality"] == "local"
|
||||
assert payload[0]["capabilities"] == ["structured-json"]
|
||||
assert payload[0]["data_policy"] == "private"
|
||||
assert payload[0]["max_context_tokens"] == 8192
|
||||
assert payload[0]["available"] is True
|
||||
|
||||
def test_providers_add(self, temp_providers_file, capsys):
|
||||
"""Add a custom provider."""
|
||||
with patch('sys.argv', ['cmdforge', 'providers', 'add',
|
||||
@@ -302,6 +667,13 @@ class TestProvidersCommand:
|
||||
'--model', 'example/model',
|
||||
'--api-key-env', 'CUSTOM_API_KEY',
|
||||
'--tag', 'api',
|
||||
'--locality', 'remote',
|
||||
'--capability', 'structured-json',
|
||||
'--model-digest', 'sha256:abc',
|
||||
'--cost-class', 'low',
|
||||
'--latency-class', 'fast',
|
||||
'--max-context-tokens', '128000',
|
||||
'--data-policy', 'internal',
|
||||
'--fallback-chain', 'free',
|
||||
]):
|
||||
result = main()
|
||||
@@ -314,6 +686,13 @@ class TestProvidersCommand:
|
||||
assert provider.api_key_env == 'CUSTOM_API_KEY'
|
||||
assert provider.tags == ['api']
|
||||
assert provider.fallback_chain == PRESET_CHAINS['free']
|
||||
assert provider.locality == 'remote'
|
||||
assert provider.capabilities == ['structured-json']
|
||||
assert provider.model_digest == 'sha256:abc'
|
||||
assert provider.cost_class == 'low'
|
||||
assert provider.latency_class == 'fast'
|
||||
assert provider.max_context_tokens == 128000
|
||||
assert provider.data_policy == 'internal'
|
||||
|
||||
def test_providers_list_reports_missing_api_key(self, temp_providers_file, capsys):
|
||||
from cmdforge.providers import Provider, save_providers
|
||||
@@ -441,3 +820,333 @@ class TestDocsCommand:
|
||||
assert result == 1 # Returns 1 when no README
|
||||
captured = capsys.readouterr()
|
||||
assert 'No documentation' in captured.out or '--edit' in captured.out
|
||||
|
||||
|
||||
class TestRegistryPublishDryRun:
|
||||
@pytest.fixture
|
||||
def tool_dir(self, tmp_path):
|
||||
directory = tmp_path / "dry-run-tool"
|
||||
directory.mkdir()
|
||||
(directory / "config.yaml").write_text(
|
||||
"name: dry-run-tool\n"
|
||||
"version: 1.0.0\n"
|
||||
"description: Test dry-run behavior\n"
|
||||
"output: constant\n"
|
||||
)
|
||||
return directory
|
||||
|
||||
@staticmethod
|
||||
def args(tool_dir):
|
||||
return SimpleNamespace(
|
||||
path=str(tool_dir), dry_run=True, force=False, owner=""
|
||||
)
|
||||
|
||||
def test_without_token_stops_after_local_preflight(
|
||||
self, tool_dir, monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
|
||||
get_client = MagicMock()
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.cli.registry_commands.load_config",
|
||||
lambda: SimpleNamespace(registry=SimpleNamespace(token="")),
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", get_client)
|
||||
|
||||
assert _cmd_registry_publish(self.args(tool_dir)) == 0
|
||||
get_client.assert_not_called()
|
||||
assert "local preflight only" in capsys.readouterr().out
|
||||
|
||||
def test_renders_structured_registry_evidence(
|
||||
self, tool_dir, monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
|
||||
client = MagicMock()
|
||||
client.publish_tool.return_value = {
|
||||
"preflight": {
|
||||
"errors": [],
|
||||
"warnings": ["remote warning"],
|
||||
"suggestions": ["remote hint"],
|
||||
},
|
||||
"suggestions": {
|
||||
"similar_tools": [{
|
||||
"name": "official/similar",
|
||||
"similarity": 0.8,
|
||||
}],
|
||||
"scrutiny": {
|
||||
"findings": [{"result": "warning", "message": "review me"}]
|
||||
},
|
||||
},
|
||||
}
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.cli.registry_commands.load_config",
|
||||
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
|
||||
|
||||
assert _cmd_registry_publish(self.args(tool_dir)) == 0
|
||||
output = capsys.readouterr().out
|
||||
assert "remote warning" in output
|
||||
assert "official/similar" in output
|
||||
assert "review me" in output
|
||||
client.publish_tool.assert_called_once()
|
||||
|
||||
def test_registry_rejection_returns_failure(
|
||||
self, tool_dir, monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
from cmdforge.registry_client import RegistryError
|
||||
|
||||
client = MagicMock()
|
||||
client.publish_tool.side_effect = RegistryError(
|
||||
"SCRUTINY_FAILED", "Tool rejected"
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.cli.registry_commands.load_config",
|
||||
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
|
||||
|
||||
assert _cmd_registry_publish(self.args(tool_dir)) == 1
|
||||
assert "Tool rejected" in capsys.readouterr().err
|
||||
|
||||
def test_strips_local_registry_metadata_from_payload(
|
||||
self, tool_dir, monkeypatch
|
||||
):
|
||||
import yaml
|
||||
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
|
||||
config_path = tool_dir / "config.yaml"
|
||||
config_path.write_text(
|
||||
config_path.read_text()
|
||||
+ "registry_hash: sha256:stale\n"
|
||||
+ "registry_status: approved\n"
|
||||
+ "registry_owner: previous-owner\n"
|
||||
+ "registry_feedback: old feedback\n"
|
||||
)
|
||||
client = MagicMock()
|
||||
client.publish_tool.return_value = {"preflight": {"errors": []}}
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.cli.registry_commands.load_config",
|
||||
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
|
||||
|
||||
assert _cmd_registry_publish(self.args(tool_dir)) == 0
|
||||
payload = yaml.safe_load(client.publish_tool.call_args.args[0])
|
||||
assert not {
|
||||
"registry_hash",
|
||||
"registry_status",
|
||||
"registry_owner",
|
||||
"registry_feedback",
|
||||
}.intersection(payload)
|
||||
|
||||
def test_non_mapping_config_returns_failure(self, tool_dir, capsys):
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
|
||||
(tool_dir / "config.yaml").write_text("- not\n- a\n- tool\n")
|
||||
|
||||
assert _cmd_registry_publish(self.args(tool_dir)) == 1
|
||||
assert "YAML mapping" in capsys.readouterr().err
|
||||
|
||||
def test_normal_publish_runs_remote_preflight_before_mutation(
|
||||
self, tool_dir, monkeypatch
|
||||
):
|
||||
from cmdforge.cli.registry_commands import _cmd_registry_publish
|
||||
|
||||
client = MagicMock()
|
||||
client.get_me.return_value = {"slug": "testuser"}
|
||||
client.get_my_tool_status.return_value = {"status": "pending"}
|
||||
client.publish_tool.side_effect = [
|
||||
{
|
||||
"preflight": {"errors": [], "warnings": [], "suggestions": []},
|
||||
"quality": {"score": 80, "evidence_coverage": 60, "categories": []},
|
||||
},
|
||||
{
|
||||
"owner": "testuser", "name": "dry-run-tool",
|
||||
"version": "1.0.0", "status": "pending",
|
||||
},
|
||||
]
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.cli.registry_commands.load_config",
|
||||
lambda: SimpleNamespace(registry=SimpleNamespace(token="token")),
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
|
||||
args = SimpleNamespace(
|
||||
path=str(tool_dir), dry_run=False, force=True, owner=""
|
||||
)
|
||||
assert _cmd_registry_publish(args) == 0
|
||||
assert client.publish_tool.call_count == 2
|
||||
assert client.publish_tool.call_args_list[0].kwargs["dry_run"] is True
|
||||
assert "dry_run" not in client.publish_tool.call_args_list[1].kwargs
|
||||
|
||||
|
||||
def test_inspect_with_registry_uses_similarity_results(monkeypatch, capsys):
|
||||
from cmdforge.cli import cmd_inspect
|
||||
from cmdforge.registry_client import PaginatedResponse
|
||||
|
||||
tool = Tool(name="summary", output="constant")
|
||||
client = MagicMock()
|
||||
client.search_tools.return_value = PaginatedResponse(data=[{
|
||||
"owner": "official",
|
||||
"name": "summarize",
|
||||
"description": "Summarize text",
|
||||
}])
|
||||
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
|
||||
monkeypatch.setattr("cmdforge.registry_client.get_client", lambda: client)
|
||||
|
||||
assert cmd_inspect(SimpleNamespace(name="summary", registry=True)) == 0
|
||||
assert "official/summarize" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_inspect_shows_contract_proposal_and_passing_conformance(
|
||||
monkeypatch, capsys
|
||||
):
|
||||
from cmdforge.cli import cmd_inspect
|
||||
|
||||
step_schema = {
|
||||
"type": "object",
|
||||
"properties": {"answer": {"type": "string"}},
|
||||
"required": ["answer"],
|
||||
}
|
||||
tool = Tool(
|
||||
name="contracted",
|
||||
arguments=[],
|
||||
input_schema={},
|
||||
output_schema=step_schema,
|
||||
steps=[PromptStep(
|
||||
prompt="Answer", provider="paid-provider", output_var="answer",
|
||||
output_schema=step_schema,
|
||||
)],
|
||||
output="{answer}",
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
|
||||
|
||||
assert cmd_inspect(SimpleNamespace(name="contracted", registry=False)) == 0
|
||||
output = capsys.readouterr().out
|
||||
assert "Contract conformance" in output
|
||||
assert "PASSED" in output
|
||||
|
||||
|
||||
def test_inspect_can_save_passing_baseline(tmp_path, monkeypatch, capsys):
|
||||
from cmdforge.cli import cmd_inspect
|
||||
from cmdforge.contract_testing import CONFORMANCE_FILE
|
||||
|
||||
tool = Tool(
|
||||
name="baseline", version="1.0.0", input_schema={},
|
||||
output_schema={"type": "string"}, output="stable",
|
||||
path=tmp_path / "config.yaml",
|
||||
)
|
||||
monkeypatch.setattr("cmdforge.tool.load_tool", lambda name: tool)
|
||||
|
||||
args = SimpleNamespace(
|
||||
name="baseline", registry=False, save_baseline=True
|
||||
)
|
||||
assert cmd_inspect(args) == 0
|
||||
assert (tmp_path / CONFORMANCE_FILE).exists()
|
||||
assert "Saved conformance baseline" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_switch_to_existing_tool_closes_creation_page_first():
|
||||
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
|
||||
from cmdforge.gui.pages.tool_builder_page import _switch_to_existing_tool
|
||||
|
||||
main_window = MagicMock()
|
||||
_switch_to_existing_tool(main_window, "existing")
|
||||
|
||||
assert main_window.method_calls == [
|
||||
call.close_tool_builder(),
|
||||
call.open_tool_builder("existing"),
|
||||
]
|
||||
|
||||
|
||||
def test_tool_builder_save_scope_prefers_recognized_projects(tmp_path):
|
||||
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
|
||||
from cmdforge.gui.pages.tool_builder_page import default_save_scope
|
||||
|
||||
assert default_save_scope(tmp_path) == "global"
|
||||
(tmp_path / "pyproject.toml").write_text("[project]\nname='demo'\n")
|
||||
assert default_save_scope(tmp_path) == "project"
|
||||
|
||||
|
||||
def test_tool_builder_existing_tool_root_does_not_nest_tool_name(tmp_path):
|
||||
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
|
||||
from cmdforge.gui.pages.tool_builder_page import existing_tool_root
|
||||
|
||||
config = tmp_path / ".cmdforge" / "demo" / "config.yaml"
|
||||
assert existing_tool_root(config) == (tmp_path / ".cmdforge").resolve()
|
||||
|
||||
|
||||
def test_guided_extraction_builds_new_tool_without_mutating_draft():
|
||||
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
|
||||
from cmdforge.gui.pages.tool_builder_page import (
|
||||
build_extracted_tool,
|
||||
extraction_diff,
|
||||
)
|
||||
|
||||
schema = {"type": "string"}
|
||||
repeated = [
|
||||
PromptStep(
|
||||
prompt="First {input}", provider="mock", output_var="first",
|
||||
output_schema=schema,
|
||||
),
|
||||
PromptStep(
|
||||
prompt="Second {first}", provider="mock", output_var="second",
|
||||
output_schema=schema,
|
||||
),
|
||||
]
|
||||
draft = Tool(
|
||||
name="draft", steps=repeated + [
|
||||
PromptStep(
|
||||
prompt="First {input}", provider="mock", output_var="first_copy",
|
||||
output_schema=schema,
|
||||
),
|
||||
PromptStep(
|
||||
prompt="Second {first}", provider="mock", output_var="second_copy",
|
||||
output_schema=schema,
|
||||
),
|
||||
], input_schema=schema,
|
||||
)
|
||||
before = draft.to_dict()
|
||||
|
||||
extracted = build_extracted_tool(
|
||||
draft,
|
||||
{"type": "repeated_sequence", "locations": [1, 3], "length": 2},
|
||||
"draft-shared",
|
||||
)
|
||||
|
||||
assert len(extracted.steps) == 2
|
||||
assert extracted.output == "{second}"
|
||||
assert extracted.output_schema == schema
|
||||
assert draft.to_dict() == before
|
||||
assert "+++ draft-shared/config.yaml" in extraction_diff(extracted)
|
||||
|
||||
|
||||
def test_guided_extraction_rejects_hidden_external_step_dependency():
|
||||
pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError)
|
||||
from cmdforge.gui.pages.tool_builder_page import build_extracted_tool
|
||||
|
||||
schema = {"type": "string"}
|
||||
draft = Tool(name="draft", steps=[
|
||||
PromptStep(
|
||||
prompt="Produce", provider="mock", output_var="earlier",
|
||||
output_schema=schema,
|
||||
),
|
||||
PromptStep(
|
||||
prompt="Use {earlier}", provider="mock", output_var="one",
|
||||
output_schema=schema,
|
||||
),
|
||||
PromptStep(
|
||||
prompt="Finish {one}", provider="mock", output_var="two",
|
||||
output_schema=schema,
|
||||
),
|
||||
])
|
||||
|
||||
with pytest.raises(ValueError, match="outside its boundary"):
|
||||
build_extracted_tool(
|
||||
draft,
|
||||
{"type": "repeated_sequence", "locations": [2, 4], "length": 2},
|
||||
"draft-shared",
|
||||
)
|
||||
|
||||
@@ -37,7 +37,10 @@ def app():
|
||||
app = create_app()
|
||||
app.config["TESTING"] = True
|
||||
|
||||
yield app
|
||||
# Keep the environment override active for the entire test. Registry
|
||||
# request handlers open fresh connections rather than reusing the one
|
||||
# created during app initialization.
|
||||
yield app
|
||||
|
||||
# Cleanup
|
||||
Path(db_path).unlink(missing_ok=True)
|
||||
@@ -53,7 +56,7 @@ def client(app):
|
||||
def auth_headers(app):
|
||||
"""Create auth headers with a valid token."""
|
||||
import hashlib
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from cmdforge.registry.db import connect_db
|
||||
|
||||
token = "test-token"
|
||||
@@ -91,7 +94,7 @@ def auth_headers(app):
|
||||
INSERT INTO api_tokens (publisher_id, token_hash, name, created_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""",
|
||||
[publisher_id, token_hash, "test-token", datetime.utcnow().isoformat()],
|
||||
[publisher_id, token_hash, "test-token", datetime.now(timezone.utc).isoformat()],
|
||||
)
|
||||
|
||||
# Insert a public approved tool for tests if missing
|
||||
@@ -109,7 +112,7 @@ def auth_headers(app):
|
||||
[
|
||||
"testuser", "tool1", "1.0.0", "Test tool", "Other", "[]",
|
||||
"name: tool1\nversion: 1.0.0\n", "", publisher_id,
|
||||
"public", "approved", datetime.utcnow().isoformat(),
|
||||
"public", "approved", datetime.now(timezone.utc).isoformat(),
|
||||
],
|
||||
)
|
||||
|
||||
@@ -155,6 +158,327 @@ class TestToolApprovedEndpoint:
|
||||
assert data['data']['has_approved_public_version'] is False
|
||||
|
||||
|
||||
@flask_required
|
||||
class TestDeprecationChains:
|
||||
def test_replacement_chain_is_validated_and_returned(self, client, auth_headers):
|
||||
target = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": "name: replacement\nversion: 1.0.0\noutput: ok\n"},
|
||||
)
|
||||
assert target.status_code == 201
|
||||
response = client.post(
|
||||
"/api/v1/tools/testuser/tool1/deprecate",
|
||||
headers=auth_headers,
|
||||
json={"deprecated_message": "Moved", "replacement": "replacement"},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.get_json()["data"]["replacement_chain"] == [
|
||||
"testuser/replacement"
|
||||
]
|
||||
|
||||
def test_replacement_cycle_is_rejected(self, client, auth_headers):
|
||||
target = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": "name: cycle-target\nversion: 1.0.0\noutput: ok\n"},
|
||||
)
|
||||
assert target.status_code == 201
|
||||
assert client.post(
|
||||
"/api/v1/tools/testuser/cycle-target/deprecate",
|
||||
headers=auth_headers,
|
||||
json={"replacement": "tool1"},
|
||||
).status_code == 200
|
||||
response = client.post(
|
||||
"/api/v1/tools/testuser/tool1/deprecate",
|
||||
headers=auth_headers,
|
||||
json={"replacement": "cycle-target"},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "cycle" in response.get_json()["error"]["message"]
|
||||
|
||||
|
||||
@flask_required
|
||||
class TestPublishPreflightEndpoint:
|
||||
def test_dry_run_includes_shared_preflight(self, client, auth_headers):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"dry_run": True,
|
||||
"config": (
|
||||
"name: preflight-tool\n"
|
||||
"version: 1.0.0\n"
|
||||
"description: Constant output\n"
|
||||
"output: constant\n"
|
||||
"input_schema:\n type: string\n"
|
||||
"output_schema:\n type: string\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
report = response.get_json()["data"]["preflight"]
|
||||
assert report["errors"] == []
|
||||
assert report["generated_tests"][0]["state"] == "passed"
|
||||
assert response.get_json()["data"]["quality"]["evidence_coverage"] > 0
|
||||
|
||||
def test_real_publish_persists_immutable_audit(self, client, auth_headers):
|
||||
from cmdforge.registry.db import connect_db
|
||||
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"config": (
|
||||
"name: audited-tool\nversion: 1.0.0\noutput: stable\n"
|
||||
"input_schema: {}\noutput_schema:\n type: string\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
assert response.status_code == 201
|
||||
conn = connect_db()
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT tool_audits.trigger, tool_audits.evidence_json
|
||||
FROM tool_audits JOIN tools ON tools.id = tool_audits.tool_id
|
||||
WHERE tools.owner = 'testuser' AND tools.name = 'audited-tool'
|
||||
"""
|
||||
).fetchall()
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE tools SET moderation_status = 'approved'
|
||||
WHERE owner = 'testuser' AND name = 'audited-tool'
|
||||
"""
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
assert len(rows) == 1
|
||||
assert rows[0]["trigger"] == "publish"
|
||||
assert "contract_conformance" in rows[0]["evidence_json"]
|
||||
search = client.get("/api/v1/tools/search?q=audited")
|
||||
result = search.get_json()["data"][0]
|
||||
assert isinstance(result["quality_score"], int)
|
||||
assert isinstance(result["quality_coverage"], int)
|
||||
assert result["quality_evaluated_at"]
|
||||
|
||||
def test_real_publish_blocks_deterministic_contract_failure(
|
||||
self, client, auth_headers
|
||||
):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"config": (
|
||||
"name: broken-contract\nversion: 1.0.0\noutput: text\n"
|
||||
"input_schema: {}\noutput_schema:\n type: integer\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert response.get_json()["error"]["code"] == "PREFLIGHT_FAILED"
|
||||
|
||||
def test_real_publish_blocks_missing_registry_dependency(
|
||||
self, client, auth_headers
|
||||
):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"config": (
|
||||
"name: missing-dependency\nversion: 1.0.0\n"
|
||||
"dependencies:\n - absent-tool\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert "absent-tool" in response.get_json()["error"]["message"]
|
||||
|
||||
def test_invalid_contract_is_rejected(self, client, auth_headers):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"dry_run": True,
|
||||
"config": (
|
||||
"name: invalid-contract\n"
|
||||
"version: 1.0.0\n"
|
||||
"output_schema:\n type: nonsense\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert response.get_json()["error"]["code"] == "INVALID_CONFIG"
|
||||
|
||||
def test_non_mapping_config_is_rejected(self, client, auth_headers):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={"dry_run": True, "config": "- not\n- a\n- tool\n"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert response.get_json()["error"]["code"] == "VALIDATION_ERROR"
|
||||
|
||||
def test_legacy_source_string_remains_publishable(self, client, auth_headers):
|
||||
response = client.post(
|
||||
"/api/v1/tools",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"dry_run": True,
|
||||
"config": (
|
||||
"name: legacy-source\n"
|
||||
"version: 1.0.0\n"
|
||||
"source: old/tool\n"
|
||||
"output: constant\n"
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
@flask_required
|
||||
class TestM9RegistryTrustAndCommunity:
|
||||
def test_signed_publish_download_and_content_lookup(
|
||||
self, client, auth_headers
|
||||
):
|
||||
from cmdforge.attestation import (
|
||||
Attestation, generate_keypair, sign_tool, verify_attestation,
|
||||
)
|
||||
from cmdforge.registry.db import connect_db
|
||||
|
||||
private_key, public_key = generate_keypair()
|
||||
response = client.put(
|
||||
"/api/v1/me/signing-key", headers=auth_headers,
|
||||
json={"public_key": public_key},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
config = "name: signed-tool\nversion: 1.0.0\noutput: stable\n"
|
||||
preflight = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": config, "dry_run": True},
|
||||
)
|
||||
assert preflight.status_code == 200
|
||||
content_hash = preflight.get_json()["data"]["content_hash"]
|
||||
attestation = sign_tool(
|
||||
"signed-tool", "1.0.0", content_hash, "testuser", private_key
|
||||
)
|
||||
published = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": config, "attestation": attestation.to_dict()},
|
||||
)
|
||||
assert published.status_code == 201
|
||||
assert published.get_json()["data"]["content_hash"] == content_hash
|
||||
|
||||
conn = connect_db()
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE tools SET moderation_status = 'approved' "
|
||||
"WHERE owner = 'testuser' AND name = 'signed-tool'"
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
downloaded = client.get(
|
||||
"/api/v1/tools/testuser/signed-tool/download?install=false"
|
||||
)
|
||||
assert downloaded.status_code == 200
|
||||
data = downloaded.get_json()["data"]
|
||||
assert data["content_hash"] == content_hash
|
||||
assert verify_attestation(Attestation.from_dict(data["attestation"]), public_key)
|
||||
lookup = client.get(f"/api/v1/tools/by-content-hash/{content_hash}")
|
||||
assert lookup.status_code == 200
|
||||
assert lookup.get_json()["data"]["name"] == "signed-tool"
|
||||
|
||||
def test_registered_key_requires_valid_signature(self, client, auth_headers):
|
||||
from cmdforge.attestation import generate_keypair
|
||||
_, public_key = generate_keypair()
|
||||
assert client.put(
|
||||
"/api/v1/me/signing-key", headers=auth_headers,
|
||||
json={"public_key": public_key},
|
||||
).status_code == 200
|
||||
response = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": "name: unsigned-tool\nversion: 1.0.0\n"},
|
||||
)
|
||||
assert response.status_code == 400
|
||||
assert response.get_json()["error"]["code"] == "ATTESTATION_REQUIRED"
|
||||
|
||||
def test_improvement_is_tested_reviewed_and_credited(
|
||||
self, client, auth_headers
|
||||
):
|
||||
from cmdforge.registry.db import connect_db
|
||||
config = (
|
||||
"name: improvable\nversion: 1.0.0\noutput: '{result}'\n"
|
||||
"input_schema:\n type: string\noutput_schema:\n type: string\n"
|
||||
"steps:\n - type: prompt\n prompt: 'Summarize: {input}'\n"
|
||||
" provider: mock\n output_var: result\n"
|
||||
)
|
||||
published = client.post(
|
||||
"/api/v1/tools", headers=auth_headers, json={"config": config}
|
||||
)
|
||||
assert published.status_code == 201
|
||||
conn = connect_db()
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE tools SET moderation_status = 'approved' "
|
||||
"WHERE owner = 'testuser' AND name = 'improvable'"
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
submitted = client.post(
|
||||
"/api/v1/tools/testuser/improvable/1.0.0/improvements",
|
||||
headers=auth_headers,
|
||||
json={
|
||||
"step_index": 0,
|
||||
"proposed": "Summarize the input accurately and concisely: {input}",
|
||||
"rationale": "Clearer expected behavior",
|
||||
},
|
||||
)
|
||||
assert submitted.status_code == 201
|
||||
submission_id = submitted.get_json()["data"]["id"]
|
||||
reviewed = client.patch(
|
||||
f"/api/v1/improvements/{submission_id}", headers=auth_headers,
|
||||
json={"decision": "approve", "notes": "Validated"},
|
||||
)
|
||||
assert reviewed.status_code == 200
|
||||
assert reviewed.get_json()["data"]["ready_to_apply"] is True
|
||||
improved_config = config.replace("version: 1.0.0", "version: 1.0.1").replace(
|
||||
"Summarize: {input}",
|
||||
"Summarize the input accurately and concisely: {input}",
|
||||
)
|
||||
applied = client.post(
|
||||
"/api/v1/tools", headers=auth_headers,
|
||||
json={"config": improved_config, "improvement_id": submission_id},
|
||||
)
|
||||
assert applied.status_code == 201
|
||||
conn = connect_db()
|
||||
try:
|
||||
conn.execute(
|
||||
"UPDATE tools SET moderation_status = 'approved' "
|
||||
"WHERE owner = 'testuser' AND name = 'improvable' "
|
||||
"AND version = '1.0.1'"
|
||||
)
|
||||
conn.commit()
|
||||
credited = conn.execute(
|
||||
"SELECT COUNT(*) AS count FROM tool_contributors"
|
||||
).fetchone()["count"]
|
||||
finally:
|
||||
conn.close()
|
||||
assert credited == 1
|
||||
detail = client.get("/api/v1/tools/testuser/improvable?version=1.0.1")
|
||||
assert detail.status_code == 200
|
||||
assert detail.get_json()["data"]["badges"] == [
|
||||
"optimized", "community-reviewed"
|
||||
]
|
||||
|
||||
|
||||
@flask_required
|
||||
class TestPostCollectionsEndpoint:
|
||||
"""Tests for POST /api/v1/collections endpoint."""
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""Tests for M9.3 community improvement workflow."""
|
||||
|
||||
from cmdforge.community import (
|
||||
ImprovementSubmission,
|
||||
create_submission,
|
||||
run_submission_tests,
|
||||
review_submission,
|
||||
)
|
||||
from cmdforge.tool import Tool, PromptStep, CodeStep
|
||||
|
||||
|
||||
class TestCreateSubmission:
|
||||
def test_create_for_prompt_step(self):
|
||||
tool = Tool(
|
||||
name="test",
|
||||
version="1.0.0",
|
||||
steps=[PromptStep(prompt="Summarize", provider="mock", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
sub = create_submission(tool, 0, "Summarize the text thoroughly", "alice", "More detail")
|
||||
assert sub.tool_name == "test"
|
||||
assert sub.step_type == "prompt"
|
||||
assert sub.original == "Summarize"
|
||||
assert sub.proposed == "Summarize the text thoroughly"
|
||||
assert sub.submitter == "alice"
|
||||
assert sub.status == "pending"
|
||||
|
||||
def test_create_for_code_step(self):
|
||||
tool = Tool(
|
||||
name="test",
|
||||
steps=[CodeStep(code="x = 1", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
sub = create_submission(tool, 0, "x = 42", "bob")
|
||||
assert sub.step_type == "code"
|
||||
assert sub.original == "x = 1"
|
||||
|
||||
def test_invalid_step_index(self):
|
||||
tool = Tool(name="test")
|
||||
try:
|
||||
create_submission(tool, 5, "new", "alice")
|
||||
assert False, "Should have raised"
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
|
||||
class TestReviewSubmission:
|
||||
def test_review_approved(self):
|
||||
sub = ImprovementSubmission(
|
||||
tool_name="test", tool_version="1.0.0", submitter="alice",
|
||||
step_index=0, step_type="prompt", original="a", proposed="b",
|
||||
status="tested", test_result={"passed_for_review": True},
|
||||
)
|
||||
review = review_submission(sub, "approve", "admin", "Good improvement")
|
||||
assert review.decision == "approve"
|
||||
assert sub.status == "approved"
|
||||
|
||||
def test_review_pending_rejected(self):
|
||||
sub = ImprovementSubmission(
|
||||
tool_name="test", tool_version="1.0.0", submitter="alice",
|
||||
step_index=0, step_type="prompt", original="a", proposed="b",
|
||||
status="pending",
|
||||
)
|
||||
try:
|
||||
review_submission(sub, "approve", "admin")
|
||||
assert False, "Should have raised"
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def test_invalid_decision(self):
|
||||
sub = ImprovementSubmission(
|
||||
tool_name="test", tool_version="1.0.0", submitter="alice",
|
||||
step_index=0, step_type="prompt", original="a", proposed="b",
|
||||
status="tested",
|
||||
)
|
||||
try:
|
||||
review_submission(sub, "maybe", "admin")
|
||||
assert False, "Should have raised"
|
||||
except ValueError:
|
||||
pass
|
||||
@@ -0,0 +1,368 @@
|
||||
"""Tests for deterministic contract conformance (M8.V1)."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from jsonschema import validate
|
||||
|
||||
from cmdforge.contract_testing import (
|
||||
ConformanceResult,
|
||||
ConformanceReport,
|
||||
UnsupportedContract,
|
||||
_generate_inputs,
|
||||
_fallback_value,
|
||||
run_contract_tests,
|
||||
)
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
|
||||
class TestInputGeneration:
|
||||
def test_enum_generates_variants(self):
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"mode": {"type": "string", "enum": ["fast", "accurate", "balanced"]},
|
||||
"limit": {"type": "integer", "default": 10},
|
||||
},
|
||||
"required": ["mode"],
|
||||
}
|
||||
cases = _generate_inputs(schema)
|
||||
assert len(cases) >= 1
|
||||
assert cases[0]["mode"] in ("fast", "accurate", "balanced")
|
||||
|
||||
def test_fallback_string(self):
|
||||
cases = _generate_inputs({
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
})
|
||||
assert "test-name" in cases[0]["name"]
|
||||
|
||||
def test_fallback_integer(self):
|
||||
cases = _generate_inputs({
|
||||
"type": "object",
|
||||
"properties": {"count": {"type": "integer"}},
|
||||
"required": ["count"],
|
||||
})
|
||||
assert cases[0]["count"] == 0
|
||||
|
||||
def test_default_used(self):
|
||||
cases = _generate_inputs({
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string", "default": "World"}},
|
||||
"required": ["name"],
|
||||
})
|
||||
assert cases[0]["name"] == "World"
|
||||
|
||||
def test_scalar_schema_generates_scalar(self):
|
||||
cases = _generate_inputs({"type": "string"})
|
||||
assert cases == ["test-input"]
|
||||
|
||||
def test_generated_value_respects_numeric_bounds(self):
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"count": {"type": "integer", "minimum": 1, "maximum": 3}
|
||||
},
|
||||
"required": ["count"],
|
||||
}
|
||||
case = _generate_inputs(schema)[0]
|
||||
validate(case, schema)
|
||||
assert 1 <= case["count"] <= 3
|
||||
|
||||
def test_array_and_nested_object_are_supported(self):
|
||||
schema = {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
},
|
||||
}
|
||||
value = _generate_inputs(schema)[0]
|
||||
validate(value, schema)
|
||||
|
||||
def test_unsynthesizable_pattern_is_unsupported(self):
|
||||
with pytest.raises(UnsupportedContract):
|
||||
_generate_inputs({"type": "string", "pattern": "^z{20}$"})
|
||||
|
||||
def test_invalid_default_is_unsupported_not_returned(self):
|
||||
with pytest.raises(UnsupportedContract, match="valid input"):
|
||||
_generate_inputs({"type": "integer", "default": "wrong"})
|
||||
|
||||
|
||||
class TestFallbackValue:
|
||||
def test_explicit_default(self):
|
||||
assert _fallback_value({"default": "hi"}, "x", []) == "hi"
|
||||
|
||||
def test_examples(self):
|
||||
assert _fallback_value({"examples": ["a", "b"]}, "x", []) == "a"
|
||||
|
||||
def test_enum(self):
|
||||
assert _fallback_value({"enum": ["yes", "no"]}, "x", []) == "yes"
|
||||
|
||||
def test_type_fallback(self):
|
||||
assert _fallback_value({"type": "boolean"}, "flag", []) is True
|
||||
|
||||
|
||||
class TestConformanceReport:
|
||||
def test_empty_is_not_run_not_pass(self):
|
||||
report = ConformanceReport()
|
||||
assert report.outcome == "not_run"
|
||||
assert not report.passed
|
||||
|
||||
def test_invalid_state_is_rejected(self):
|
||||
with pytest.raises(ValueError, match="Unknown conformance state"):
|
||||
ConformanceResult("case", "unknown")
|
||||
|
||||
def test_summary_counts(self):
|
||||
report = ConformanceReport(results=[
|
||||
ConformanceResult("a", "passed"),
|
||||
ConformanceResult("b", "passed"),
|
||||
ConformanceResult("c", "failed", "error"),
|
||||
])
|
||||
assert report.summary == {"passed": 2, "failed": 1, "unsupported": 0, "not_run": 0}
|
||||
assert not report.passed
|
||||
|
||||
|
||||
class TestRunContractTests:
|
||||
def test_missing_schemas_returns_not_run(self, tmp_path):
|
||||
from unittest.mock import patch
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(name="bare")
|
||||
report = run_contract_tests(tool)
|
||||
assert report.results[0].state == "not_run"
|
||||
|
||||
def test_simple_tool_passes(self, tmp_path):
|
||||
from unittest.mock import patch
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="greet",
|
||||
input_schema={
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string", "default": "World"},
|
||||
},
|
||||
},
|
||||
output_schema={
|
||||
"type": "object",
|
||||
"properties": {"result": {"type": "string"}},
|
||||
"required": ["result"],
|
||||
},
|
||||
steps=[
|
||||
PromptStep(
|
||||
prompt="Greet {name}",
|
||||
provider="real-provider-that-must-not-run",
|
||||
output_var="response",
|
||||
output_schema={
|
||||
"type": "object",
|
||||
"properties": {"result": {"type": "string"}},
|
||||
"required": ["result"],
|
||||
},
|
||||
),
|
||||
],
|
||||
output="{response}",
|
||||
)
|
||||
report = run_contract_tests(tool)
|
||||
assert report.outcome == "passed"
|
||||
assert report.results[0].state == "passed"
|
||||
|
||||
def test_empty_schemas_are_explicit_and_runnable(self):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = run_contract_tests(
|
||||
Tool(name="anything", input_schema={}, output_schema={}, output="ok")
|
||||
)
|
||||
assert report.outcome == "passed"
|
||||
|
||||
@pytest.mark.parametrize("step_type", ["code", "tool", "mcp"])
|
||||
def test_side_effecting_steps_are_unsupported(self, step_type):
|
||||
from cmdforge.tool import CodeStep, McpStep, Tool, ToolStep
|
||||
|
||||
steps = {
|
||||
"code": CodeStep(code="raise AssertionError('must not run')", output_var="x"),
|
||||
"tool": ToolStep(tool="external", output_var="x"),
|
||||
"mcp": McpStep(server="remote", tool="external", output_var="x"),
|
||||
}
|
||||
tool = Tool(
|
||||
name="unsafe",
|
||||
input_schema={},
|
||||
output_schema={},
|
||||
steps=[steps[step_type]],
|
||||
output="{x}",
|
||||
)
|
||||
report = run_contract_tests(tool)
|
||||
assert report.outcome == "unsupported"
|
||||
|
||||
def test_schema_mismatch_is_a_failure(self):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = run_contract_tests(
|
||||
Tool(
|
||||
name="mismatch",
|
||||
input_schema={},
|
||||
output_schema={"type": "integer"},
|
||||
output="not-an-integer",
|
||||
)
|
||||
)
|
||||
assert report.outcome == "failed"
|
||||
|
||||
|
||||
class TestRegressionComparison:
|
||||
@staticmethod
|
||||
def contracted_tool(version="1.0.0", output_schema=None):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
return Tool(
|
||||
name="test",
|
||||
version=version,
|
||||
input_schema={},
|
||||
output_schema=output_schema or {"type": "string"},
|
||||
output="stable",
|
||||
)
|
||||
|
||||
def passing_report(self, tool=None):
|
||||
return run_contract_tests(tool or self.contracted_tool())
|
||||
|
||||
def test_save_and_load_baseline(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
save_conformance_baseline, load_conformance_baseline,
|
||||
)
|
||||
import stat
|
||||
|
||||
tool = self.contracted_tool()
|
||||
report = self.passing_report(tool)
|
||||
path = save_conformance_baseline(tool, report, tmp_path)
|
||||
assert path.exists()
|
||||
assert stat.S_IMODE(path.stat().st_mode) == 0o600
|
||||
|
||||
loaded = load_conformance_baseline(tmp_path)
|
||||
assert loaded["format_version"] == 1
|
||||
assert loaded["tool"]["version"] == "1.0.0"
|
||||
assert loaded["contract_hash"]
|
||||
assert len(loaded["results"]) == 1
|
||||
assert loaded["results"][0]["case_id"].startswith("case-")
|
||||
assert loaded["results"][0]["output_shape"] == {"type": "string"}
|
||||
|
||||
def test_missing_baseline(self, tmp_path):
|
||||
from cmdforge.contract_testing import load_conformance_baseline
|
||||
assert load_conformance_baseline(tmp_path) is None
|
||||
|
||||
def test_corrupt_baseline_is_rejected(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
BaselineFormatError, CONFORMANCE_FILE, load_conformance_baseline,
|
||||
)
|
||||
|
||||
(tmp_path / CONFORMANCE_FILE).write_text("[]")
|
||||
with pytest.raises(BaselineFormatError, match="root"):
|
||||
load_conformance_baseline(tmp_path)
|
||||
|
||||
def test_tampered_baseline_evidence_is_rejected(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
BaselineFormatError, CONFORMANCE_FILE, load_conformance_baseline,
|
||||
save_conformance_baseline,
|
||||
)
|
||||
import json
|
||||
|
||||
tool = self.contracted_tool()
|
||||
save_conformance_baseline(tool, self.passing_report(tool), tmp_path)
|
||||
path = tmp_path / CONFORMANCE_FILE
|
||||
data = json.loads(path.read_text())
|
||||
data["results"][0]["input_value"] = {"tampered": True}
|
||||
path.write_text(json.dumps(data))
|
||||
|
||||
with pytest.raises(BaselineFormatError, match="fingerprint mismatch"):
|
||||
load_conformance_baseline(tmp_path)
|
||||
|
||||
def test_failed_report_cannot_replace_baseline(self, tmp_path):
|
||||
from cmdforge.contract_testing import save_conformance_baseline
|
||||
|
||||
report = run_contract_tests(Tool(
|
||||
name="failed", input_schema={}, output_schema={"type": "integer"},
|
||||
output="text",
|
||||
))
|
||||
with pytest.raises(ValueError, match="passing"):
|
||||
save_conformance_baseline(self.contracted_tool(), report, tmp_path)
|
||||
|
||||
def baseline_and_report(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
load_conformance_baseline, save_conformance_baseline,
|
||||
)
|
||||
|
||||
tool = self.contracted_tool()
|
||||
report = self.passing_report(tool)
|
||||
save_conformance_baseline(tool, report, tmp_path)
|
||||
return load_conformance_baseline(tmp_path), report
|
||||
|
||||
def test_detects_regression(self):
|
||||
from dataclasses import replace
|
||||
from cmdforge.contract_testing import (
|
||||
ConformanceReport, compare_conformance,
|
||||
)
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
baseline, report = self.baseline_and_report(Path(directory))
|
||||
current = ConformanceReport([
|
||||
replace(report.results[0], state="failed", detail="now broken")
|
||||
])
|
||||
delta = compare_conformance(
|
||||
baseline, current, self.contracted_tool(version="2.0.0")
|
||||
)
|
||||
assert len(delta.regressed) == 1
|
||||
assert delta.has_regressions
|
||||
assert delta.current_version == "2.0.0"
|
||||
|
||||
def test_lost_coverage_is_regression(self, tmp_path):
|
||||
from dataclasses import replace
|
||||
from cmdforge.contract_testing import (
|
||||
ConformanceReport, compare_conformance,
|
||||
)
|
||||
baseline, report = self.baseline_and_report(tmp_path)
|
||||
current = ConformanceReport([
|
||||
replace(report.results[0], state="unsupported", detail="lost")
|
||||
])
|
||||
delta = compare_conformance(baseline, current, self.contracted_tool())
|
||||
assert len(delta.coverage_losses) == 1
|
||||
assert delta.has_regressions
|
||||
|
||||
def test_removed_case_is_regression(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
ConformanceReport, compare_conformance,
|
||||
)
|
||||
baseline, _ = self.baseline_and_report(tmp_path)
|
||||
delta = compare_conformance(
|
||||
baseline, ConformanceReport(), self.contracted_tool()
|
||||
)
|
||||
assert len(delta.removed_cases) == 1
|
||||
assert delta.has_regressions
|
||||
|
||||
def test_output_shape_or_value_change_is_flagged(self, tmp_path):
|
||||
from dataclasses import replace
|
||||
from cmdforge.contract_testing import (
|
||||
ConformanceReport, compare_conformance,
|
||||
)
|
||||
baseline, report = self.baseline_and_report(tmp_path)
|
||||
current = ConformanceReport([
|
||||
replace(
|
||||
report.results[0],
|
||||
output_shape={"type": "object", "properties": {}},
|
||||
output_fingerprint="changed",
|
||||
)
|
||||
])
|
||||
delta = compare_conformance(baseline, current, self.contracted_tool())
|
||||
assert len(delta.output_changes) == 1
|
||||
assert delta.has_regressions
|
||||
|
||||
def test_contract_change_is_recorded(self, tmp_path):
|
||||
from cmdforge.contract_testing import compare_conformance
|
||||
|
||||
baseline, report = self.baseline_and_report(tmp_path)
|
||||
changed = self.contracted_tool(output_schema={"type": ["string", "null"]})
|
||||
delta = compare_conformance(baseline, report, changed)
|
||||
assert delta.contract_changed
|
||||
@@ -0,0 +1,173 @@
|
||||
"""Tests for transactional email and password-reset delivery safety."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
from unittest.mock import patch
|
||||
|
||||
import requests
|
||||
from flask import Flask
|
||||
|
||||
from cmdforge.registry.app import create_app, password_hasher
|
||||
from cmdforge.registry.db import connect_db
|
||||
from cmdforge.web.email import send_email
|
||||
|
||||
|
||||
def _email_app(**config) -> Flask:
|
||||
app = Flask(__name__)
|
||||
defaults = {
|
||||
"CMDFORGE_ENV": "development",
|
||||
"MAIL_TRANSPORT": "disabled",
|
||||
"MAIL_FROM": "CmdForge <[email protected]>",
|
||||
"MAIL_TIMEOUT": 10,
|
||||
"RESEND_API_KEY": "",
|
||||
}
|
||||
defaults.update(config)
|
||||
app.config.update(defaults)
|
||||
return app
|
||||
|
||||
|
||||
def test_disabled_delivery_does_not_log_message_or_recipient(caplog):
|
||||
app = _email_app()
|
||||
with app.app_context(), caplog.at_level(logging.INFO):
|
||||
assert not send_email(
|
||||
"[email protected]", "Reset subject", "secret reset body", "secret text"
|
||||
)
|
||||
assert "[email protected]" not in caplog.text
|
||||
assert "secret reset body" not in caplog.text
|
||||
assert "secret text" not in caplog.text
|
||||
|
||||
|
||||
def test_console_transport_is_rejected_in_production(caplog):
|
||||
app = _email_app(CMDFORGE_ENV="production", MAIL_TRANSPORT="console")
|
||||
with app.app_context(), caplog.at_level(logging.INFO):
|
||||
assert not send_email("[email protected]", "subject", "body")
|
||||
assert "[email protected]" not in caplog.text
|
||||
assert "body" not in caplog.text
|
||||
|
||||
|
||||
def test_resend_transport_sends_expected_request(monkeypatch):
|
||||
app = _email_app(MAIL_TRANSPORT="resend", RESEND_API_KEY="test-api-key")
|
||||
captured = {}
|
||||
|
||||
class Response:
|
||||
def raise_for_status(self):
|
||||
return None
|
||||
|
||||
def fake_post(url, **kwargs):
|
||||
captured["url"] = url
|
||||
captured.update(kwargs)
|
||||
return Response()
|
||||
|
||||
monkeypatch.setattr(requests, "post", fake_post)
|
||||
with app.app_context():
|
||||
assert send_email("[email protected]", "subject", "<p>body</p>", "body")
|
||||
|
||||
assert captured["url"] == "https://api.resend.com/emails"
|
||||
assert captured["headers"]["Authorization"] == "Bearer test-api-key"
|
||||
assert captured["json"]["to"] == ["[email protected]"]
|
||||
assert captured["timeout"] == 10.0
|
||||
|
||||
|
||||
def test_resend_failure_does_not_log_sensitive_data(monkeypatch, caplog):
|
||||
app = _email_app(MAIL_TRANSPORT="resend", RESEND_API_KEY="test-api-key")
|
||||
|
||||
def fail_post(*args, **kwargs):
|
||||
raise requests.ConnectionError("provider unavailable")
|
||||
|
||||
monkeypatch.setattr(requests, "post", fail_post)
|
||||
with app.app_context(), caplog.at_level(logging.INFO):
|
||||
assert not send_email("[email protected]", "subject", "reset-secret")
|
||||
assert "[email protected]" not in caplog.text
|
||||
assert "reset-secret" not in caplog.text
|
||||
assert "test-api-key" not in caplog.text
|
||||
|
||||
|
||||
def _registry_client(tmp_path, monkeypatch):
|
||||
db_path = tmp_path / "registry.db"
|
||||
monkeypatch.setenv("CMDFORGE_REGISTRY_DB", str(db_path))
|
||||
monkeypatch.setenv("CMDFORGE_ENV", "production")
|
||||
monkeypatch.setenv("CMDFORGE_PUBLIC_URL", "https://cmdforge.example")
|
||||
app = create_app()
|
||||
app.config["TESTING"] = True
|
||||
with connect_db() as db:
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO publishers (email, password_hash, slug, display_name)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""",
|
||||
["[email protected]", password_hasher.hash("initial-password"), "owner", "Owner"],
|
||||
)
|
||||
db.commit()
|
||||
return app.test_client(), db_path
|
||||
|
||||
|
||||
def test_reset_uses_canonical_url_and_leaves_delivered_token_valid(tmp_path, monkeypatch):
|
||||
client, db_path = _registry_client(tmp_path, monkeypatch)
|
||||
captured = {}
|
||||
|
||||
def delivered(to, token, base_url):
|
||||
captured.update(to=to, token=token, base_url=base_url)
|
||||
return True
|
||||
|
||||
with patch("cmdforge.web.email.send_password_reset_email", delivered):
|
||||
response = client.post(
|
||||
"/api/v1/password-reset/request",
|
||||
json={"email": "[email protected]"},
|
||||
headers={"Host": "attacker.example"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
assert captured["base_url"] == "https://cmdforge.example"
|
||||
with connect_db() as db:
|
||||
row = db.execute(
|
||||
"SELECT token_hash, used_at FROM password_reset_tokens"
|
||||
).fetchone()
|
||||
assert row["token_hash"] == hashlib.sha256(captured["token"].encode()).hexdigest()
|
||||
assert row["used_at"] is None
|
||||
|
||||
|
||||
def test_reset_invalidates_token_when_delivery_fails(tmp_path, monkeypatch):
|
||||
client, db_path = _registry_client(tmp_path, monkeypatch)
|
||||
|
||||
with patch("cmdforge.web.email.send_password_reset_email", return_value=False):
|
||||
response = client.post(
|
||||
"/api/v1/password-reset/request", json={"email": "[email protected]"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
with connect_db() as db:
|
||||
row = db.execute("SELECT used_at FROM password_reset_tokens").fetchone()
|
||||
assert row["used_at"] is not None
|
||||
|
||||
|
||||
def test_reset_invalidates_token_without_safe_production_base_url(tmp_path, monkeypatch):
|
||||
client, db_path = _registry_client(tmp_path, monkeypatch)
|
||||
client.application.config["PUBLIC_BASE_URL"] = "http://insecure.example"
|
||||
|
||||
with patch("cmdforge.web.email.send_password_reset_email") as sender:
|
||||
response = client.post(
|
||||
"/api/v1/password-reset/request", json={"email": "[email protected]"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
sender.assert_not_called()
|
||||
with connect_db() as db:
|
||||
row = db.execute("SELECT used_at FROM password_reset_tokens").fetchone()
|
||||
assert row["used_at"] is not None
|
||||
|
||||
|
||||
def test_unknown_email_creates_no_token_and_sends_nothing(tmp_path, monkeypatch):
|
||||
client, db_path = _registry_client(tmp_path, monkeypatch)
|
||||
|
||||
with patch("cmdforge.web.email.send_password_reset_email") as sender:
|
||||
response = client.post(
|
||||
"/api/v1/password-reset/request", json={"email": "[email protected]"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
sender.assert_not_called()
|
||||
with connect_db() as db:
|
||||
count = db.execute("SELECT COUNT(*) FROM password_reset_tokens").fetchone()[0]
|
||||
assert count == 0
|
||||
@@ -0,0 +1,109 @@
|
||||
"""Tests for scrutiny-driven improvement pipeline (M9.2)."""
|
||||
|
||||
from cmdforge.improvement import (
|
||||
ImprovementSuggestion,
|
||||
ImprovementReport,
|
||||
generate_improvements,
|
||||
)
|
||||
from cmdforge.tool import Tool, PromptStep, CodeStep
|
||||
|
||||
|
||||
class TestImprovementReport:
|
||||
def test_empty_report(self):
|
||||
report = ImprovementReport(tool_name="test")
|
||||
assert not report.has_high_severity
|
||||
assert report.by_category == {}
|
||||
|
||||
def test_has_high_severity(self):
|
||||
report = ImprovementReport(
|
||||
tool_name="test",
|
||||
suggestions=[
|
||||
ImprovementSuggestion("honesty", "high", "Bad", "desc"),
|
||||
ImprovementSuggestion("efficiency", "low", "Minor", "desc"),
|
||||
],
|
||||
)
|
||||
assert report.has_high_severity
|
||||
|
||||
def test_by_category(self):
|
||||
report = ImprovementReport(
|
||||
tool_name="test",
|
||||
suggestions=[
|
||||
ImprovementSuggestion("honesty", "high", "A", "d"),
|
||||
ImprovementSuggestion("honesty", "low", "B", "d"),
|
||||
ImprovementSuggestion("efficiency", "medium", "C", "d"),
|
||||
],
|
||||
)
|
||||
assert len(report.by_category["honesty"]) == 2
|
||||
assert len(report.by_category["efficiency"]) == 1
|
||||
|
||||
|
||||
class TestGenerateImprovements:
|
||||
def test_missing_description(self):
|
||||
tool = Tool(name="nodesc")
|
||||
report = generate_improvements(tool)
|
||||
honesty = [s for s in report.suggestions if s.category == "honesty"]
|
||||
assert any("Missing description" in s.title for s in honesty)
|
||||
|
||||
def test_short_description(self):
|
||||
tool = Tool(name="short", description="Does stuff")
|
||||
report = generate_improvements(tool)
|
||||
honesty = [s for s in report.suggestions if s.category == "honesty"]
|
||||
assert any("short" in s.title.lower() for s in honesty)
|
||||
|
||||
def test_echo_prompt_detected(self):
|
||||
tool = Tool(
|
||||
name="echo",
|
||||
steps=[PromptStep(prompt="{input}", provider="mock", output_var="out")],
|
||||
output="{out}",
|
||||
)
|
||||
report = generate_improvements(tool)
|
||||
efficiency = [s for s in report.suggestions if s.category == "efficiency"]
|
||||
assert any("echo" in s.title.lower() for s in efficiency)
|
||||
|
||||
def test_multiple_same_provider_prompts(self):
|
||||
tool = Tool(
|
||||
name="multi",
|
||||
steps=[
|
||||
PromptStep(prompt="Step 1: {input}", provider="claude", output_var="a"),
|
||||
PromptStep(prompt="Step 2: {a}", provider="claude", output_var="b"),
|
||||
],
|
||||
output="{b}",
|
||||
)
|
||||
report = generate_improvements(tool)
|
||||
efficiency = [s for s in report.suggestions if s.category == "efficiency"]
|
||||
assert any("same provider" in s.title.lower() for s in efficiency)
|
||||
|
||||
def test_exec_detected(self):
|
||||
tool = Tool(
|
||||
name="dangerous",
|
||||
steps=[CodeStep(code="exec('print(1)')", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
report = generate_improvements(tool)
|
||||
transparency = [s for s in report.suggestions if s.category == "transparency"]
|
||||
assert any("exec" in s.title.lower() for s in transparency)
|
||||
|
||||
def test_short_prompt_detected(self):
|
||||
tool = Tool(
|
||||
name="vague",
|
||||
steps=[PromptStep(prompt="Do it", provider="mock", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
report = generate_improvements(tool)
|
||||
transparency = [s for s in report.suggestions if s.category == "transparency"]
|
||||
assert any("short" in s.title.lower() for s in transparency)
|
||||
|
||||
def test_clean_tool_no_suggestions(self):
|
||||
tool = Tool(
|
||||
name="clean",
|
||||
description="A well-documented tool that summarizes text input concisely",
|
||||
steps=[PromptStep(
|
||||
prompt="Summarize the following text in 2-3 sentences. Focus on key points.\n\nText: {input}",
|
||||
provider="mock",
|
||||
output_var="summary",
|
||||
)],
|
||||
output="{summary}",
|
||||
)
|
||||
report = generate_improvements(tool)
|
||||
high = [s for s in report.suggestions if s.severity == "high"]
|
||||
assert len(high) == 0
|
||||
@@ -0,0 +1,121 @@
|
||||
"""Tests for M9.4 transitive integrity verification."""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from cmdforge.integrity import (
|
||||
IntegrityNode,
|
||||
IntegrityChain,
|
||||
compute_tool_hash,
|
||||
build_integrity_chain,
|
||||
verify_integrity,
|
||||
)
|
||||
from cmdforge.tool import Tool, PromptStep, ToolStep
|
||||
|
||||
|
||||
class TestComputeToolHash:
|
||||
def test_same_tool_same_hash(self):
|
||||
tool = Tool(name="test", version="1.0.0")
|
||||
assert compute_tool_hash(tool) == compute_tool_hash(tool)
|
||||
|
||||
def test_different_tools_different_hash(self):
|
||||
a = Tool(name="a", version="1.0.0")
|
||||
b = Tool(name="b", version="1.0.0")
|
||||
assert compute_tool_hash(a) != compute_tool_hash(b)
|
||||
|
||||
def test_version_change_changes_hash(self):
|
||||
v1 = Tool(name="test", version="1.0.0")
|
||||
v2 = Tool(name="test", version="2.0.0")
|
||||
assert compute_tool_hash(v1) != compute_tool_hash(v2)
|
||||
|
||||
|
||||
class TestIntegrityChain:
|
||||
def test_single_tool_chain(self):
|
||||
tool = Tool(name="solo", version="1.0.0")
|
||||
chain = build_integrity_chain(tool)
|
||||
assert chain.root.name == "solo"
|
||||
assert chain.is_valid
|
||||
|
||||
def test_chain_with_unresolved_dep(self):
|
||||
tool = Tool(
|
||||
name="parent",
|
||||
version="1.0.0",
|
||||
steps=[ToolStep(tool="missing-dep", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
chain = build_integrity_chain(tool)
|
||||
assert "missing-dep" not in chain.nodes
|
||||
assert not chain.is_valid
|
||||
assert chain.errors == ["Unresolved dependency: missing-dep"]
|
||||
|
||||
def test_chain_valid_with_resolved_dep(self, tmp_path):
|
||||
with (
|
||||
patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"),
|
||||
patch("cmdforge.tool.BIN_DIR", tmp_path / "bin"),
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
|
||||
child = Tool(name="child", version="1.0.0")
|
||||
save_tool(child)
|
||||
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
version="1.0.0",
|
||||
steps=[ToolStep(tool="child", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
save_tool(parent)
|
||||
|
||||
chain = build_integrity_chain(parent)
|
||||
assert "child" in chain.nodes
|
||||
assert chain.is_valid
|
||||
|
||||
def test_child_change_changes_root_identity(self, tmp_path):
|
||||
with (
|
||||
patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"),
|
||||
patch("cmdforge.tool.BIN_DIR", tmp_path / "bin"),
|
||||
):
|
||||
from cmdforge.tool import save_tool
|
||||
child = Tool(name="child", version="1.0.0", output="one")
|
||||
save_tool(child)
|
||||
parent = Tool(
|
||||
name="parent", version="1.0.0",
|
||||
steps=[ToolStep(tool="child", output_var="x")], output="{x}",
|
||||
)
|
||||
first = build_integrity_chain(parent).root.content_hash
|
||||
child.output = "two"
|
||||
save_tool(child)
|
||||
second = build_integrity_chain(parent).root.content_hash
|
||||
assert first != second
|
||||
|
||||
|
||||
class TestVerifyIntegrity:
|
||||
def test_valid_chain(self):
|
||||
from cmdforge.integrity import compute_content_identity
|
||||
definition_hash = "sha256:" + "a" * 64
|
||||
content_hash = compute_content_identity(definition_hash, {})
|
||||
node = IntegrityNode(
|
||||
name="root", definition_hash=definition_hash,
|
||||
content_hash=content_hash,
|
||||
)
|
||||
chain = IntegrityChain(root=node, nodes={"root": node})
|
||||
assert verify_integrity(chain)
|
||||
|
||||
def test_tampered_chain(self):
|
||||
chain = IntegrityChain(
|
||||
root=IntegrityNode(
|
||||
name="root",
|
||||
content_hash="abc",
|
||||
dependencies=["dep"],
|
||||
dependency_hashes={"dep": "expected_hash"},
|
||||
),
|
||||
nodes={
|
||||
"root": IntegrityNode(
|
||||
name="root",
|
||||
content_hash="abc",
|
||||
dependencies=["dep"],
|
||||
dependency_hashes={"dep": "expected_hash"},
|
||||
),
|
||||
"dep": IntegrityNode(name="dep", content_hash="different_hash"),
|
||||
},
|
||||
)
|
||||
assert not chain.is_valid
|
||||
+13
-1
@@ -259,6 +259,12 @@ class TestGenerateLockfile:
|
||||
assert "official/text-utils" in lock.packages
|
||||
assert lock.packages["official/summarize"].direct is True
|
||||
assert lock.packages["official/text-utils"].direct is False
|
||||
child_identity = lock.packages["official/text-utils"].content_hash
|
||||
assert child_identity.startswith("sha256:")
|
||||
assert lock.packages["official/summarize"].dependency_hashes == {
|
||||
"official/text-utils": child_identity
|
||||
}
|
||||
assert lock.packages["official/summarize"].content_hash.startswith("sha256:")
|
||||
|
||||
|
||||
class TestVerifyLockfile:
|
||||
@@ -319,7 +325,9 @@ class TestLockfileRoundTrip:
|
||||
integrity="sha256:abc123",
|
||||
source="registry",
|
||||
direct=True,
|
||||
required_by=[]
|
||||
required_by=[],
|
||||
content_hash="sha256:root",
|
||||
dependency_hashes={"official/text-utils": "sha256:child"},
|
||||
),
|
||||
"official/text-utils": LockedPackage(
|
||||
name="official/text-utils",
|
||||
@@ -364,6 +372,10 @@ class TestLockfileRoundTrip:
|
||||
assert summarize.integrity == "sha256:abc123"
|
||||
assert summarize.source == "registry"
|
||||
assert summarize.direct is True
|
||||
assert summarize.content_hash == "sha256:root"
|
||||
assert summarize.dependency_hashes == {
|
||||
"official/text-utils": "sha256:child"
|
||||
}
|
||||
|
||||
text_utils = loaded.packages["official/text-utils"]
|
||||
assert text_utils.direct is False
|
||||
|
||||
@@ -0,0 +1,988 @@
|
||||
"""Tests for MCP client and McpStep execution."""
|
||||
|
||||
import sys
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from unittest.mock import patch
|
||||
from cmdforge.mcp_client import (
|
||||
DEFAULT_INHERITED_ENV,
|
||||
McpServerConfig,
|
||||
McpClientManager,
|
||||
_build_server_env,
|
||||
_build_http_headers,
|
||||
_normalize_result,
|
||||
_sanitize,
|
||||
_fingerprint,
|
||||
load_mcp_config,
|
||||
save_mcp_config,
|
||||
)
|
||||
from cmdforge.runner import _substitute_mcp_args, _deep_substitute, run_tool
|
||||
from cmdforge.tool import McpStep, Tool
|
||||
|
||||
|
||||
class TestAgentHostSetup:
|
||||
def test_builds_current_vendor_cli_commands(self):
|
||||
from cmdforge.agent_setup import build_host_add_command
|
||||
|
||||
codex = build_host_add_command(
|
||||
"codex", "/usr/bin/codex", "cmdforge", "user"
|
||||
)
|
||||
assert codex[:5] == [
|
||||
"/usr/bin/codex", "mcp", "add", "cmdforge", "--"
|
||||
]
|
||||
assert codex[-4:] == ["-m", "cmdforge.cli", "mcp", "serve"]
|
||||
|
||||
claude = build_host_add_command(
|
||||
"claude-code", "/usr/bin/claude", "cmdforge", "project"
|
||||
)
|
||||
assert claude[:9] == [
|
||||
"/usr/bin/claude", "mcp", "add", "--transport", "stdio",
|
||||
"--scope", "project", "cmdforge", "--",
|
||||
]
|
||||
|
||||
def test_codex_rejects_unsupported_scope(self):
|
||||
from cmdforge.agent_setup import build_host_add_command
|
||||
|
||||
with pytest.raises(ValueError, match="user scope only"):
|
||||
build_host_add_command(
|
||||
"codex", "/usr/bin/codex", "cmdforge", "project"
|
||||
)
|
||||
|
||||
def test_rejects_unsafe_registration_name(self):
|
||||
from cmdforge.agent_setup import build_host_add_command
|
||||
|
||||
with pytest.raises(ValueError, match="registration name"):
|
||||
build_host_add_command(
|
||||
"codex", "/usr/bin/codex", "--config", "user"
|
||||
)
|
||||
|
||||
def test_policy_merge_is_idempotent_and_preserves_user_text(self):
|
||||
from cmdforge.agent_setup import POLICY_BODY, merge_agent_policy
|
||||
|
||||
original = "# Existing guidance\n\nKeep this.\n"
|
||||
merged = merge_agent_policy(original)
|
||||
assert merged.startswith(original)
|
||||
assert POLICY_BODY.strip() in merged
|
||||
assert merge_agent_policy(merged) == merged
|
||||
|
||||
def test_policy_merge_rejects_malformed_markers(self):
|
||||
from cmdforge.agent_setup import POLICY_START, merge_agent_policy
|
||||
|
||||
with pytest.raises(ValueError, match="malformed"):
|
||||
merge_agent_policy(f"text\n{POLICY_START}\n")
|
||||
|
||||
def test_policy_writer_refuses_symlink(self, tmp_path):
|
||||
from cmdforge.agent_setup import write_agent_policy
|
||||
|
||||
target = tmp_path / "target.md"
|
||||
target.write_text("unchanged")
|
||||
link = tmp_path / "AGENTS.md"
|
||||
link.symlink_to(target)
|
||||
with pytest.raises(ValueError, match="symlinked"):
|
||||
write_agent_policy(link, "replacement")
|
||||
assert target.read_text() == "unchanged"
|
||||
|
||||
def test_dry_run_changes_nothing(self, tmp_path, capsys):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
|
||||
patch("cmdforge.agent_setup.subprocess.run") as run,
|
||||
):
|
||||
assert configure_agent_host(
|
||||
"codex", project_dir=tmp_path, dry_run=True
|
||||
) == 0
|
||||
run.assert_not_called()
|
||||
assert not (tmp_path / "AGENTS.md").exists()
|
||||
output = capsys.readouterr().out
|
||||
assert "Would run:" in output
|
||||
assert "cmdforge-agent-policy:start" in output
|
||||
|
||||
def test_configures_host_and_managed_policy(self, tmp_path, capsys):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
results = [
|
||||
subprocess.CompletedProcess([], 1, "", "not found"),
|
||||
subprocess.CompletedProcess([], 0, "added", ""),
|
||||
]
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
|
||||
patch("cmdforge.agent_setup.subprocess.run", side_effect=results) as run,
|
||||
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
|
||||
):
|
||||
assert configure_agent_host("codex", project_dir=tmp_path) == 0
|
||||
|
||||
assert run.call_count == 2
|
||||
assert run.call_args_list[0].args[0] == [
|
||||
"/usr/bin/codex", "mcp", "get", "cmdforge"
|
||||
]
|
||||
assert run.call_args_list[1].args[0][:5] == [
|
||||
"/usr/bin/codex", "mcp", "add", "cmdforge", "--"
|
||||
]
|
||||
policy = (tmp_path / "AGENTS.md").read_text()
|
||||
assert "cmdforge-agent-policy:start" in policy
|
||||
assert "Do not install, publish, expose" in policy
|
||||
assert "Configured CmdForge MCP" in capsys.readouterr().out
|
||||
|
||||
def test_existing_registration_is_not_overwritten(self, tmp_path):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
result = subprocess.CompletedProcess([], 0, "exists", "")
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/claude"),
|
||||
patch("cmdforge.agent_setup.subprocess.run", return_value=result) as run,
|
||||
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
|
||||
):
|
||||
assert configure_agent_host(
|
||||
"claude-code", project_dir=tmp_path
|
||||
) == 0
|
||||
assert run.call_count == 1
|
||||
assert (tmp_path / "CLAUDE.md").exists()
|
||||
|
||||
def test_force_removes_then_readds_registration(self, tmp_path):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
results = [
|
||||
subprocess.CompletedProcess([], 0, "exists", ""),
|
||||
subprocess.CompletedProcess([], 0, "removed", ""),
|
||||
subprocess.CompletedProcess([], 0, "added", ""),
|
||||
]
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/claude"),
|
||||
patch("cmdforge.agent_setup.subprocess.run", side_effect=results) as run,
|
||||
patch("cmdforge.agent_setup._warn_if_no_tools_exposed"),
|
||||
):
|
||||
assert configure_agent_host(
|
||||
"claude-code", project_dir=tmp_path,
|
||||
install_policy=False, force=True,
|
||||
) == 0
|
||||
|
||||
assert run.call_args_list[1].args[0] == [
|
||||
"/usr/bin/claude", "mcp", "remove", "cmdforge"
|
||||
]
|
||||
assert run.call_args_list[2].args[0][:4] == [
|
||||
"/usr/bin/claude", "mcp", "add", "--transport"
|
||||
]
|
||||
|
||||
def test_failed_host_add_does_not_write_policy(self, tmp_path):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
results = [
|
||||
subprocess.CompletedProcess([], 1, "", "missing"),
|
||||
subprocess.CompletedProcess([], 1, "", "bad config"),
|
||||
]
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
|
||||
patch("cmdforge.agent_setup.subprocess.run", side_effect=results),
|
||||
):
|
||||
assert configure_agent_host(
|
||||
"codex", project_dir=tmp_path
|
||||
) == 1
|
||||
|
||||
assert not (tmp_path / "AGENTS.md").exists()
|
||||
|
||||
def test_host_command_timeout_is_reported(self, tmp_path, capsys):
|
||||
from cmdforge.agent_setup import configure_agent_host
|
||||
|
||||
with (
|
||||
patch("cmdforge.agent_setup.shutil.which", return_value="/usr/bin/codex"),
|
||||
patch(
|
||||
"cmdforge.agent_setup.subprocess.run",
|
||||
side_effect=subprocess.TimeoutExpired("codex", 30),
|
||||
),
|
||||
):
|
||||
assert configure_agent_host(
|
||||
"codex", project_dir=tmp_path, install_policy=False
|
||||
) == 1
|
||||
|
||||
assert "timed out after 30 seconds" in capsys.readouterr().err
|
||||
|
||||
|
||||
class TestMcpServerConfig:
|
||||
def test_defaults(self):
|
||||
cfg = McpServerConfig(name="test")
|
||||
assert cfg.transport == "stdio"
|
||||
assert cfg.timeout == 30
|
||||
assert cfg.args == []
|
||||
assert cfg.env == {}
|
||||
assert cfg.approved is False
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"change, message",
|
||||
[
|
||||
({"transport": "streamable-http"}, "requires a url"),
|
||||
({"args": "-y"}, "list of strings"),
|
||||
({"timeout": 0}, "greater than 0"),
|
||||
({"timeout": True}, "must be a number"),
|
||||
],
|
||||
)
|
||||
def test_validation_rejects_invalid_config(self, change, message):
|
||||
command = None if change.get("transport") == "streamable-http" else "server"
|
||||
cfg = McpServerConfig(name="test", command=command, **change)
|
||||
with pytest.raises(ValueError, match=message):
|
||||
cfg.validate()
|
||||
|
||||
def test_fingerprint_changes_with_command(self):
|
||||
a = McpServerConfig(name="a", command="cmd-a")
|
||||
b = McpServerConfig(name="b", command="cmd-b")
|
||||
assert _fingerprint(a) != _fingerprint(b)
|
||||
|
||||
def test_fingerprint_same_for_same_config(self):
|
||||
a = McpServerConfig(name="a", command="cmd", args=["-v"])
|
||||
b = McpServerConfig(name="b", command="cmd", args=["-v"])
|
||||
assert _fingerprint(a) == _fingerprint(b)
|
||||
|
||||
def test_streamable_http_accepts_https(self):
|
||||
cfg = McpServerConfig(
|
||||
name="remote", transport="streamable-http",
|
||||
url="https://example.com/mcp", approved=True,
|
||||
)
|
||||
cfg.validate()
|
||||
|
||||
def test_streamable_http_allows_loopback_http_only(self):
|
||||
McpServerConfig(
|
||||
name="local", transport="streamable-http",
|
||||
url="http://127.0.0.1:8000/mcp",
|
||||
).validate()
|
||||
with pytest.raises(ValueError, match="must use HTTPS"):
|
||||
McpServerConfig(
|
||||
name="remote", transport="streamable-http",
|
||||
url="http://example.com/mcp",
|
||||
).validate()
|
||||
|
||||
def test_streamable_http_rejects_url_credentials(self):
|
||||
with pytest.raises(ValueError, match="must not contain credentials"):
|
||||
McpServerConfig(
|
||||
name="remote", transport="streamable-http",
|
||||
url="https://user:[email protected]/mcp",
|
||||
).validate()
|
||||
|
||||
|
||||
class TestSanitize:
|
||||
def test_redacts_bearer(self):
|
||||
assert "redacted" in _sanitize("Bearer abcdefghijklmnopqrstuvwxyz")
|
||||
|
||||
def test_redacts_sk_prefix(self):
|
||||
assert "redacted" in _sanitize("sk-abcdefghijklmnopqrstuvwxyz")
|
||||
|
||||
def test_preserves_normal_text(self):
|
||||
assert _sanitize("hello world") == "hello world"
|
||||
|
||||
def test_redacts_explicit_secret(self):
|
||||
assert _sanitize("failed with hunter2", ["hunter2"]) == "failed with [redacted]"
|
||||
|
||||
|
||||
class TestEnvironmentIsolation:
|
||||
def test_only_allowlisted_environment_is_inherited(self, monkeypatch):
|
||||
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "do-not-leak")
|
||||
cfg = McpServerConfig(name="test", command="server")
|
||||
environment = _build_server_env(cfg)
|
||||
assert "CMDFORGE_TEST_SECRET" not in environment
|
||||
assert set(environment).issubset(set(DEFAULT_INHERITED_ENV))
|
||||
|
||||
def test_explicit_environment_reference_is_resolved(self, monkeypatch):
|
||||
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "allowed")
|
||||
cfg = McpServerConfig(
|
||||
name="test", command="server", env={"SERVER_TOKEN": "${CMDFORGE_TEST_SECRET}"}
|
||||
)
|
||||
assert _build_server_env(cfg)["SERVER_TOKEN"] == "allowed"
|
||||
|
||||
def test_http_header_reference_is_resolved(self, monkeypatch):
|
||||
monkeypatch.setenv("MCP_TOKEN", "secret")
|
||||
cfg = McpServerConfig(
|
||||
name="remote", transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
headers={"Authorization": "Bearer ${MCP_TOKEN}"},
|
||||
)
|
||||
assert _build_http_headers(cfg)["Authorization"] == "Bearer secret"
|
||||
|
||||
def test_missing_environment_reference_fails(self, monkeypatch):
|
||||
monkeypatch.delenv("CMDFORGE_MISSING", raising=False)
|
||||
cfg = McpServerConfig(
|
||||
name="test", command="server", env={"SERVER_TOKEN": "${CMDFORGE_MISSING}"}
|
||||
)
|
||||
with pytest.raises(ValueError, match="is not set"):
|
||||
_build_server_env(cfg)
|
||||
|
||||
|
||||
class TestMcpConfigPersistence:
|
||||
@pytest.fixture
|
||||
def temp_mcp_file(self, tmp_path, monkeypatch):
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
yield config_file
|
||||
|
||||
def test_save_and_load(self, temp_mcp_file):
|
||||
servers = [
|
||||
McpServerConfig(
|
||||
name="filesystem",
|
||||
command="npx",
|
||||
args=["-y", "@scope/server", "/tmp"],
|
||||
timeout=30,
|
||||
approved=True,
|
||||
)
|
||||
]
|
||||
save_mcp_config(servers)
|
||||
loaded = load_mcp_config()
|
||||
assert len(loaded) == 1
|
||||
assert loaded[0].name == "filesystem"
|
||||
assert loaded[0].command == "npx"
|
||||
assert loaded[0].args == ["-y", "@scope/server", "/tmp"]
|
||||
assert loaded[0].approved is True
|
||||
|
||||
def test_load_no_file(self, temp_mcp_file):
|
||||
assert load_mcp_config() == []
|
||||
|
||||
def test_streamable_http_round_trip(self, temp_mcp_file):
|
||||
save_mcp_config([McpServerConfig(
|
||||
name="remote", transport="streamable-http",
|
||||
url="https://example.com/mcp",
|
||||
headers={"Authorization": "Bearer ${MCP_TOKEN}"},
|
||||
approved=True,
|
||||
)])
|
||||
loaded = load_mcp_config()[0]
|
||||
assert loaded.transport == "streamable-http"
|
||||
assert loaded.command is None
|
||||
assert loaded.url == "https://example.com/mcp"
|
||||
assert loaded.headers == {"Authorization": "Bearer ${MCP_TOKEN}"}
|
||||
|
||||
def test_save_creates_0600_permissions(self, temp_mcp_file):
|
||||
save_mcp_config([McpServerConfig(name="test", command="echo")])
|
||||
perms = oct(temp_mcp_file.stat().st_mode & 0o777)
|
||||
assert perms == "0o600"
|
||||
|
||||
def test_save_preserves_server_exposure_policy(self, temp_mcp_file):
|
||||
temp_mcp_file.write_text(
|
||||
"version: 1\nserver:\n expose: [summarize]\n deny: [dangerous]\n"
|
||||
)
|
||||
save_mcp_config([McpServerConfig(name="test", command="echo")])
|
||||
data = yaml.safe_load(temp_mcp_file.read_text())
|
||||
assert data["server"] == {
|
||||
"expose": ["summarize"],
|
||||
"deny": ["dangerous"],
|
||||
}
|
||||
|
||||
|
||||
class TestResultNormalization:
|
||||
def test_auto_prefers_structured_content(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = {"sum": 3}
|
||||
content = [type("Block", (), {"text": "plain text"})()]
|
||||
|
||||
result = _normalize_result(FakeResult(), "auto")
|
||||
assert result == {"sum": 3}
|
||||
|
||||
def test_auto_falls_back_to_text(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = None
|
||||
content = [type("Block", (), {"type": "text", "text": "hello"})()]
|
||||
|
||||
result = _normalize_result(FakeResult(), "auto")
|
||||
assert result == "hello"
|
||||
|
||||
def test_auto_preserves_mixed_content(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = None
|
||||
content = [
|
||||
{"type": "text", "text": "hello"},
|
||||
{"type": "image", "mimeType": "image/png", "data": "AA=="},
|
||||
]
|
||||
|
||||
result = _normalize_result(FakeResult(), "auto")
|
||||
assert result == FakeResult.content
|
||||
|
||||
def test_content_mode_always_returns_blocks(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = None
|
||||
content = [{"type": "text", "text": "hello"}]
|
||||
|
||||
assert _normalize_result(FakeResult(), "content") == FakeResult.content
|
||||
|
||||
def test_structured_fails_without_structured_content(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = None
|
||||
content = [type("Block", (), {"type": "text", "text": "hello"})()]
|
||||
|
||||
with pytest.raises(ValueError, match="structuredContent"):
|
||||
_normalize_result(FakeResult(), "structured")
|
||||
|
||||
def test_text_mode(self):
|
||||
class FakeResult:
|
||||
isError = False
|
||||
structuredContent = None
|
||||
content = [
|
||||
type("Block", (), {"type": "text", "text": "line1"})(),
|
||||
type("Block", (), {"type": "text", "text": "line2"})(),
|
||||
]
|
||||
|
||||
result = _normalize_result(FakeResult(), "text")
|
||||
assert result == "line1\nline2"
|
||||
|
||||
def test_iserror_causes_failure(self):
|
||||
class FakeResult:
|
||||
isError = True
|
||||
structuredContent = None
|
||||
content = [type("Block", (), {"type": "text", "text": "error message"})()]
|
||||
|
||||
with pytest.raises(RuntimeError, match="error"):
|
||||
_normalize_result(FakeResult(), "auto")
|
||||
|
||||
|
||||
class TestMcpClientManager:
|
||||
@pytest.fixture
|
||||
def manager(self, tmp_path, monkeypatch):
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
return McpClientManager()
|
||||
|
||||
def test_list_servers_empty(self, manager):
|
||||
assert manager.list_servers() == []
|
||||
|
||||
def test_list_servers_with_config(self, manager, tmp_path, monkeypatch):
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
save_mcp_config([McpServerConfig(name="test", command="echo")])
|
||||
|
||||
mgr = McpClientManager()
|
||||
servers = mgr.list_servers()
|
||||
assert len(servers) == 1
|
||||
assert servers[0].name == "test"
|
||||
|
||||
def test_call_tool_unknown_server(self, manager):
|
||||
with pytest.raises(KeyError, match="not configured"):
|
||||
manager.call_tool("unknown", "tool", {})
|
||||
|
||||
def test_unapproved_server_is_not_executed(self, tmp_path, monkeypatch):
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
save_mcp_config([McpServerConfig(name="test", command="echo")])
|
||||
with pytest.raises(PermissionError, match="not approved"):
|
||||
McpClientManager().discover("test")
|
||||
|
||||
def test_missing_sdk_has_actionable_error(self, tmp_path, monkeypatch):
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
save_mcp_config([
|
||||
McpServerConfig(name="test", command="echo", approved=True)
|
||||
])
|
||||
monkeypatch.setitem(sys.modules, "mcp", None)
|
||||
with pytest.raises(ImportError, match=r"cmdforge\[mcp\]"):
|
||||
McpClientManager().discover("test")
|
||||
|
||||
|
||||
class TestArgumentSubstitution:
|
||||
def test_single_variable_preserves_integer(self):
|
||||
result = _deep_substitute("42", {"limit": 42})
|
||||
assert result == "42" # no braces, so no substitution
|
||||
|
||||
def test_braced_variable_preserves_type(self):
|
||||
variables = {"settings": {"limit": 50}}
|
||||
result = _substitute_mcp_args(
|
||||
{"options": {"limit": "{settings.limit}"}}, variables
|
||||
)
|
||||
assert result["options"]["limit"] == 50
|
||||
|
||||
def test_nested_dict_substitution(self):
|
||||
variables = {"path": "/tmp/data.csv"}
|
||||
result = _substitute_mcp_args(
|
||||
{"files": [{"source": "{path}"}]}, variables
|
||||
)
|
||||
assert result["files"][0]["source"] == "/tmp/data.csv"
|
||||
|
||||
def test_string_with_variable_inside_text(self):
|
||||
variables = {"name": "report"}
|
||||
result = _deep_substitute("/tmp/{name}.csv", variables)
|
||||
assert result == "/tmp/report.csv"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"value",
|
||||
[
|
||||
{"a": 1},
|
||||
[1, 2],
|
||||
-5,
|
||||
"001",
|
||||
"true",
|
||||
False,
|
||||
None,
|
||||
],
|
||||
)
|
||||
def test_exact_reference_preserves_original_type(self, value):
|
||||
assert _deep_substitute("{value}", {"value": value}) == value
|
||||
|
||||
|
||||
class TestMcpStep:
|
||||
def test_round_trip(self):
|
||||
original = McpStep(
|
||||
server="fixture", tool="add", arguments={"a": 1, "b": 2},
|
||||
output_var="sum", result_mode="structured",
|
||||
)
|
||||
assert McpStep.from_dict(original.to_dict()) == original
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"field, value",
|
||||
[("server", ""), ("tool", None), ("arguments", []), ("result_mode", "raw")],
|
||||
)
|
||||
def test_rejects_invalid_fields(self, field, value):
|
||||
data = {"server": "fixture", "tool": "echo", field: value}
|
||||
with pytest.raises(ValueError):
|
||||
McpStep.from_dict(data)
|
||||
|
||||
def test_runner_preserves_structured_result(self, monkeypatch):
|
||||
calls = []
|
||||
|
||||
class FakeManager:
|
||||
def call_tool(self, server, tool, arguments, result_mode="auto"):
|
||||
calls.append((server, tool, arguments, result_mode))
|
||||
return {"sum": 3}
|
||||
|
||||
monkeypatch.setattr("cmdforge.runner.McpClientManager", FakeManager)
|
||||
tool = Tool(
|
||||
name="mcp-add",
|
||||
steps=[
|
||||
McpStep(
|
||||
server="fixture",
|
||||
tool="add",
|
||||
arguments={"a": "{input}", "b": 2},
|
||||
output_var="result",
|
||||
)
|
||||
],
|
||||
output="{result.sum}",
|
||||
)
|
||||
output, exit_code = run_tool(tool, 1, {})
|
||||
assert (output, exit_code) == ("3", 0)
|
||||
assert calls == [("fixture", "add", {"a": 1, "b": 2}, "auto")]
|
||||
|
||||
|
||||
class TestRealMcpSdk:
|
||||
@pytest.fixture
|
||||
def manager(self, tmp_path, monkeypatch):
|
||||
pytest.importorskip("mcp")
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
fixture = Path(__file__).parent / "fixtures" / "mcp_test_server.py"
|
||||
save_mcp_config([
|
||||
McpServerConfig(
|
||||
name="fixture",
|
||||
command=sys.executable,
|
||||
args=[str(fixture)],
|
||||
timeout=10,
|
||||
approved=True,
|
||||
)
|
||||
])
|
||||
return McpClientManager()
|
||||
|
||||
def test_discovery_uses_mcp_handshake_and_tools_list(self, manager):
|
||||
tools = manager.discover("fixture")
|
||||
schemas = {tool["name"]: tool for tool in tools}
|
||||
assert {"echo", "add", "read_env", "current_directory", "wait"}.issubset(schemas)
|
||||
assert schemas["add"]["inputSchema"]["required"] == ["a", "b"]
|
||||
|
||||
def test_calls_text_and_structured_tools(self, manager):
|
||||
assert manager.call_tool("fixture", "echo", {"message": "hello"}, "text") == "hello"
|
||||
assert manager.call_tool("fixture", "add", {"a": 1, "b": 2}) == {"sum": 3}
|
||||
|
||||
def test_parent_secret_is_not_leaked(self, manager, monkeypatch):
|
||||
monkeypatch.setenv("CMDFORGE_TEST_SECRET", "do-not-leak")
|
||||
result = manager.call_tool(
|
||||
"fixture", "read_env", {"name": "CMDFORGE_TEST_SECRET"}, "text"
|
||||
)
|
||||
assert result == ""
|
||||
|
||||
def test_configured_working_directory_is_used(self, manager, tmp_path):
|
||||
manager.list_servers()
|
||||
manager._configs["fixture"].cwd = str(tmp_path)
|
||||
assert manager.call_tool("fixture", "current_directory", {}, "text") == str(tmp_path)
|
||||
|
||||
def test_timeout_is_enforced(self, manager):
|
||||
manager.list_servers()
|
||||
manager._configs["fixture"].timeout = 0.5
|
||||
started = time.monotonic()
|
||||
with pytest.raises(TimeoutError, match="timed out"):
|
||||
manager.call_tool("fixture", "wait", {"seconds": 10}, "text")
|
||||
assert time.monotonic() - started < 4
|
||||
|
||||
|
||||
class TestMcpCli:
|
||||
def test_add_accepts_leading_dash_argument(self, tmp_path, monkeypatch, capsys):
|
||||
from cmdforge.cli import main
|
||||
|
||||
config_file = tmp_path / "mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", config_file)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"cmdforge", "mcp", "add", "fixture", "--command", "npx",
|
||||
"--arg=-y", "--arg", "@scope/server",
|
||||
],
|
||||
)
|
||||
assert main() == 0
|
||||
loaded = load_mcp_config()
|
||||
assert loaded[0].args == ["-y", "@scope/server"]
|
||||
assert loaded[0].approved is True
|
||||
assert "saved and approved" in capsys.readouterr().out
|
||||
|
||||
def test_bare_mcp_lists_servers(self, tmp_path, monkeypatch, capsys):
|
||||
from cmdforge.cli import main
|
||||
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", tmp_path / "mcp.yaml")
|
||||
monkeypatch.setattr(sys, "argv", ["cmdforge", "mcp"])
|
||||
assert main() == 0
|
||||
assert "No MCP servers configured" in capsys.readouterr().out
|
||||
|
||||
|
||||
class TestMcpServeConfig:
|
||||
def test_empty_expose_exposes_nothing(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
config = McpServeConfig(expose=[], deny=[])
|
||||
assert not config.is_exposed("summarize")
|
||||
|
||||
def test_wildcard_exposes_all_public(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
config = McpServeConfig(expose=["*"], deny=[])
|
||||
assert config.is_exposed("summarize")
|
||||
assert config.is_exposed("any-tool")
|
||||
|
||||
def test_deny_overrides_expose(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
config = McpServeConfig(expose=["*"], deny=["dangerous"])
|
||||
assert config.is_exposed("summarize")
|
||||
assert not config.is_exposed("dangerous")
|
||||
|
||||
def test_prefix_deny(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
config = McpServeConfig(expose=["*"], deny=["internal-*"])
|
||||
assert not config.is_exposed("internal-cleanup")
|
||||
assert not config.is_exposed("internal-")
|
||||
assert config.is_exposed("public-tool")
|
||||
|
||||
def test_explicit_expose(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
config = McpServeConfig(expose=["summarize", "extract"])
|
||||
assert config.is_exposed("summarize")
|
||||
assert config.is_exposed("extract")
|
||||
assert not config.is_exposed("other")
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"field, value",
|
||||
[("expose", "*"), ("deny", "dangerous"), ("expose", [""])],
|
||||
)
|
||||
def test_rejects_malformed_policy_lists(self, field, value):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
|
||||
values = {"expose": [], "deny": [], field: value}
|
||||
with pytest.raises(ValueError, match="list of non-empty strings"):
|
||||
McpServeConfig(**values)
|
||||
|
||||
def test_general_wildcard_matching(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
|
||||
config = McpServeConfig(expose=["official/*"], deny=["*/dangerous"])
|
||||
assert config.is_exposed("official/summarize")
|
||||
assert not config.is_exposed("official/dangerous")
|
||||
|
||||
def test_private_tool_still_honors_deny(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
from cmdforge.mcp_server import _is_exposable
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
tool = Tool(name="dangerous", visibility="private")
|
||||
config = McpServeConfig(expose=["official/dangerous"], deny=["official/*"])
|
||||
assert not _is_exposable(tool, config, qualified_name="official/dangerous")
|
||||
|
||||
def test_private_qualified_tool_requires_qualified_expose(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
from cmdforge.mcp_server import _is_exposable
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
tool = Tool(name="extract", visibility="private")
|
||||
assert not _is_exposable(
|
||||
tool,
|
||||
McpServeConfig(expose=["extract"]),
|
||||
qualified_name="official/extract",
|
||||
)
|
||||
assert _is_exposable(
|
||||
tool,
|
||||
McpServeConfig(expose=["official/extract"]),
|
||||
qualified_name="official/extract",
|
||||
)
|
||||
|
||||
|
||||
class TestToolNameMapping:
|
||||
def test_simple_name_passes_through(self):
|
||||
from cmdforge.mcp_server import _cmdforge_tool_name
|
||||
assert _cmdforge_tool_name("summarize") == "summarize"
|
||||
|
||||
def test_qualified_name_uses_double_underscore(self):
|
||||
from cmdforge.mcp_server import _cmdforge_tool_name
|
||||
assert _cmdforge_tool_name("owner/tool") == "owner__tool"
|
||||
|
||||
|
||||
class TestArgumentToJsonSchema:
|
||||
def test_string_arg(self):
|
||||
from cmdforge.mcp_server import _argument_to_json_schema
|
||||
from cmdforge.tool import ToolArgument
|
||||
arg = ToolArgument(flag="--name", variable="name", description="The name")
|
||||
schema = _argument_to_json_schema(arg)
|
||||
assert schema["type"] == "string"
|
||||
assert schema["description"] == "The name"
|
||||
|
||||
def test_integer_arg_with_default(self):
|
||||
from cmdforge.tool import ToolArgument
|
||||
from cmdforge.mcp_server import _argument_to_json_schema
|
||||
arg = ToolArgument(
|
||||
flag="--limit", variable="limit", default="10", type="integer"
|
||||
)
|
||||
schema = _argument_to_json_schema(arg)
|
||||
assert schema["type"] == "integer"
|
||||
assert schema["default"] == 10
|
||||
|
||||
def test_arg_with_enum(self):
|
||||
from cmdforge.tool import ToolArgument
|
||||
from cmdforge.mcp_server import _argument_to_json_schema
|
||||
arg = ToolArgument(
|
||||
flag="--mode", variable="mode", enum=["fast", "accurate"]
|
||||
)
|
||||
schema = _argument_to_json_schema(arg)
|
||||
assert schema["enum"] == ["fast", "accurate"]
|
||||
|
||||
|
||||
class TestBuildToolSchema:
|
||||
def test_builds_input_schema(self, tmp_path):
|
||||
from cmdforge.tool import Tool, ToolArgument
|
||||
from cmdforge.mcp_server import _build_tool_schema
|
||||
|
||||
with patch('cmdforge.tool.TOOLS_DIR', tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="greet",
|
||||
description="Greet someone",
|
||||
arguments=[
|
||||
ToolArgument(flag="--name", variable="name",
|
||||
description="Who to greet", default="World"),
|
||||
ToolArgument(flag="--count", variable="count", default="1"),
|
||||
],
|
||||
output="Hello {name} x{count}\n",
|
||||
)
|
||||
|
||||
schema = _build_tool_schema(tool)
|
||||
|
||||
assert schema["name"] == "greet"
|
||||
assert "Greet" in schema["description"]
|
||||
assert schema["inputSchema"]["type"] == "object"
|
||||
assert "name" in schema["inputSchema"]["properties"]
|
||||
assert "count" in schema["inputSchema"]["properties"]
|
||||
assert "input" in schema["inputSchema"]["properties"]
|
||||
|
||||
|
||||
class TestRegisteredMcpTool:
|
||||
def test_fastmcp_uses_flat_typed_schema_and_invokes_tool(self):
|
||||
import asyncio
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from cmdforge.mcp_server import _build_tool_schema, _register_tool
|
||||
from cmdforge.tool import Tool, ToolArgument
|
||||
|
||||
server = FastMCP("test")
|
||||
tool = Tool(
|
||||
name="greet",
|
||||
arguments=[
|
||||
ToolArgument(
|
||||
flag="--name",
|
||||
variable="name",
|
||||
type="string",
|
||||
required=True,
|
||||
description="Who to greet",
|
||||
),
|
||||
ToolArgument(
|
||||
flag="--count",
|
||||
variable="count",
|
||||
type="integer",
|
||||
default=1,
|
||||
enum=[1, 2, 3],
|
||||
),
|
||||
],
|
||||
output="Hello {name} x{count}: {input}",
|
||||
)
|
||||
_register_tool(server, "greet", "greet", _build_tool_schema(tool), tool)
|
||||
|
||||
async def exercise():
|
||||
listed = (await server.list_tools())[0]
|
||||
result = await server.call_tool(
|
||||
"greet", {"name": "Alice", "count": 2, "input": "welcome"}
|
||||
)
|
||||
return listed, result
|
||||
|
||||
listed, result = asyncio.run(exercise())
|
||||
assert listed.inputSchema["required"] == ["name"]
|
||||
assert listed.inputSchema["properties"]["count"]["type"] == "integer"
|
||||
assert listed.inputSchema["properties"]["count"]["enum"] == [1, 2, 3]
|
||||
assert "kwargs" not in listed.inputSchema["properties"]
|
||||
assert result[0].text == "Hello Alice x2: welcome"
|
||||
|
||||
def test_handler_starts_at_inherited_depth(self):
|
||||
import asyncio
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from cmdforge.mcp_server import _build_tool_schema, _register_tool
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
server = FastMCP("test")
|
||||
tool = Tool(name="depth", output="ok")
|
||||
with patch("cmdforge.runner.run_tool", return_value=("ok", 0)) as run:
|
||||
_register_tool(server, "depth", "depth", _build_tool_schema(tool), tool)
|
||||
asyncio.run(server.call_tool("depth", {}))
|
||||
assert run.call_args.kwargs["_depth"] == 0
|
||||
|
||||
def test_handler_rejects_exhausted_mcp_depth(self, monkeypatch):
|
||||
import asyncio
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from cmdforge.mcp_server import (
|
||||
MAX_TOOL_DEPTH,
|
||||
_build_tool_schema,
|
||||
_register_tool,
|
||||
)
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
monkeypatch.setenv("CMDFORGE_MCP_DEPTH", str(MAX_TOOL_DEPTH))
|
||||
server = FastMCP("test")
|
||||
tool = Tool(name="depth", output="ok")
|
||||
_register_tool(server, "depth", "depth", _build_tool_schema(tool), tool)
|
||||
with pytest.raises(Exception, match="Maximum MCP nesting depth"):
|
||||
asyncio.run(server.call_tool("depth", {}))
|
||||
|
||||
def test_serve_fails_closed_on_mapped_name_collision(self):
|
||||
from cmdforge.mcp_client import McpServeConfig
|
||||
from cmdforge.mcp_server import serve
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
tools = {
|
||||
"owner/tool": Tool(name="tool"),
|
||||
"owner__tool": Tool(name="owner__tool"),
|
||||
}
|
||||
with (
|
||||
patch(
|
||||
"cmdforge.mcp_server.load_mcp_serve_config",
|
||||
return_value=McpServeConfig(expose=["*"]),
|
||||
),
|
||||
patch("cmdforge.tool.list_tools", return_value=list(tools)),
|
||||
patch("cmdforge.tool.load_tool", side_effect=tools.get),
|
||||
patch("mcp.server.fastmcp.FastMCP.run"),
|
||||
):
|
||||
with pytest.raises(RuntimeError, match="name collision"):
|
||||
serve()
|
||||
|
||||
|
||||
class TestStreamableHttpServerSafety:
|
||||
def test_defaults_are_loopback_with_origin_allowlist(self):
|
||||
from cmdforge.mcp_server import _validate_http_server_options
|
||||
origins, token = _validate_http_server_options(
|
||||
"127.0.0.1", 8000, [], None, None
|
||||
)
|
||||
assert token is None
|
||||
assert origins == [
|
||||
"http://localhost:8000", "http://127.0.0.1:8000"
|
||||
]
|
||||
|
||||
def test_nonlocal_requires_https_and_auth(self):
|
||||
from cmdforge.mcp_server import _validate_http_server_options
|
||||
with pytest.raises(ValueError, match="external-url"):
|
||||
_validate_http_server_options("0.0.0.0", 8000, [], "secret", None)
|
||||
with pytest.raises(ValueError, match="auth-token"):
|
||||
_validate_http_server_options(
|
||||
"0.0.0.0", 8000, [], None, "https://mcp.example.com"
|
||||
)
|
||||
|
||||
def test_nonlocal_infers_https_origin_and_expands_token(self, monkeypatch):
|
||||
from cmdforge.mcp_server import _validate_http_server_options
|
||||
monkeypatch.setenv("MCP_AUTH", "secret-value")
|
||||
origins, token = _validate_http_server_options(
|
||||
"0.0.0.0", 8000, [], "${MCP_AUTH}",
|
||||
"https://mcp.example.com",
|
||||
)
|
||||
assert origins == ["https://mcp.example.com"]
|
||||
assert token == "secret-value"
|
||||
|
||||
def test_nonlocal_rejects_insecure_origin(self):
|
||||
from cmdforge.mcp_server import _validate_http_server_options
|
||||
with pytest.raises(ValueError, match="origins must use HTTPS"):
|
||||
_validate_http_server_options(
|
||||
"0.0.0.0", 8000, ["http://example.com"], "secret",
|
||||
"https://mcp.example.com",
|
||||
)
|
||||
|
||||
|
||||
class TestCmdForgeMcpServerEndToEnd:
|
||||
def test_stdio_discovery_and_invocation(self, tmp_path, monkeypatch):
|
||||
pytest.importorskip("mcp")
|
||||
home = tmp_path / "home"
|
||||
cmdforge_dir = home / ".cmdforge"
|
||||
cmdforge_dir.mkdir(parents=True)
|
||||
|
||||
(cmdforge_dir / "mcp.yaml").write_text(
|
||||
yaml.safe_dump({
|
||||
"version": 1,
|
||||
"server": {
|
||||
"expose": ["greet", "echo-input", "hidden"],
|
||||
"deny": ["hidden"],
|
||||
},
|
||||
})
|
||||
)
|
||||
tools = {
|
||||
"greet": {
|
||||
"name": "greet",
|
||||
"arguments": [{
|
||||
"flag": "--name",
|
||||
"variable": "name",
|
||||
"type": "string",
|
||||
"required": True,
|
||||
}],
|
||||
"steps": [],
|
||||
"output": "Hello {name}",
|
||||
},
|
||||
"echo-input": {
|
||||
"name": "echo-input",
|
||||
"steps": [],
|
||||
"output": "{input}",
|
||||
},
|
||||
"hidden": {
|
||||
"name": "hidden",
|
||||
"visibility": "private",
|
||||
"steps": [],
|
||||
"output": "secret",
|
||||
},
|
||||
}
|
||||
for name, data in tools.items():
|
||||
tool_dir = cmdforge_dir / name
|
||||
tool_dir.mkdir()
|
||||
(tool_dir / "config.yaml").write_text(yaml.safe_dump(data))
|
||||
|
||||
client_config = tmp_path / "client-mcp.yaml"
|
||||
monkeypatch.setattr("cmdforge.mcp_client.MCP_CONFIG_FILE", client_config)
|
||||
save_mcp_config([
|
||||
McpServerConfig(
|
||||
name="cmdforge",
|
||||
command=sys.executable,
|
||||
args=["-m", "cmdforge.cli", "mcp", "serve"],
|
||||
env={"HOME": str(home)},
|
||||
timeout=10,
|
||||
approved=True,
|
||||
)
|
||||
])
|
||||
manager = McpClientManager()
|
||||
|
||||
discovered = {tool["name"]: tool for tool in manager.discover("cmdforge")}
|
||||
assert set(discovered) == {"greet", "echo-input"}
|
||||
assert discovered["greet"]["inputSchema"]["required"] == ["name"]
|
||||
assert manager.call_tool(
|
||||
"cmdforge", "greet", {"name": "Alice"}, "text"
|
||||
) == "Hello Alice"
|
||||
assert manager.call_tool(
|
||||
"cmdforge", "echo-input", {"input": "through stdin"}, "text"
|
||||
) == "through stdin"
|
||||
+46
-4
@@ -1,5 +1,6 @@
|
||||
"""Tests for packaging metadata."""
|
||||
"""Release metadata checks for the public Python distribution."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
@@ -9,12 +10,53 @@ if sys.version_info >= (3, 11):
|
||||
else:
|
||||
tomllib = pytest.importorskip("tomli")
|
||||
|
||||
from cmdforge import __version__
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def _project_metadata() -> dict:
|
||||
with (ROOT / "pyproject.toml").open("rb") as handle:
|
||||
return tomllib.load(handle)["project"]
|
||||
|
||||
|
||||
def test_runtime_and_distribution_versions_match():
|
||||
assert _project_metadata()["version"] == __version__
|
||||
|
||||
|
||||
def test_public_distribution_metadata_is_complete():
|
||||
project = _project_metadata()
|
||||
|
||||
assert project["name"] == "cmdforge"
|
||||
assert project["readme"] == "PYPI_README.md"
|
||||
assert project["license"] == "MIT"
|
||||
assert project["license-files"] == ["LICENSE"]
|
||||
assert project["requires-python"] == ">=3.10"
|
||||
assert {"Homepage", "Documentation", "Repository", "Issues", "Changelog"} <= set(
|
||||
project["urls"]
|
||||
)
|
||||
|
||||
|
||||
def test_public_readme_describes_installation_and_current_capabilities():
|
||||
readme = (ROOT / "PYPI_README.md").read_text(encoding="utf-8")
|
||||
|
||||
for term in (
|
||||
'pipx install "cmdforge[mcp,pty]"',
|
||||
"cmdforge providers discover",
|
||||
"cmdforge mcp configure codex",
|
||||
"--no-fallback",
|
||||
"--result-envelope json",
|
||||
):
|
||||
assert term in readme
|
||||
|
||||
assert (ROOT / "LICENSE").is_file()
|
||||
|
||||
|
||||
def test_web_templates_and_static_are_declared_as_package_data():
|
||||
with open("pyproject.toml", "rb") as f:
|
||||
data = tomllib.load(f)
|
||||
with (ROOT / "pyproject.toml").open("rb") as handle:
|
||||
data = tomllib.load(handle)
|
||||
|
||||
package_data = data["tool"]["setuptools"]["package-data"]["cmdforge.web"]
|
||||
|
||||
assert "templates/**/*.html" in package_data
|
||||
assert "static/**/*" in package_data
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Behavioral coverage for registry-aware picker and deprecation UX."""
|
||||
|
||||
from io import StringIO
|
||||
import sys
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
|
||||
import cmdforge.cli.picker as picker
|
||||
import pytest
|
||||
from cmdforge import __version__
|
||||
from cmdforge.cli.picker import PickerResult
|
||||
|
||||
|
||||
class FakeTTY:
|
||||
def __init__(self, *characters):
|
||||
self.characters = iter(characters)
|
||||
|
||||
def getch(self, timeout=None):
|
||||
if timeout is not None:
|
||||
time.sleep(0.01)
|
||||
return None
|
||||
return next(self.characters)
|
||||
|
||||
|
||||
def setup_function():
|
||||
picker._registry_cache.clear()
|
||||
picker._ui_out = StringIO()
|
||||
|
||||
|
||||
def test_public_registry_search_does_not_require_token():
|
||||
class Client:
|
||||
token = ""
|
||||
|
||||
def semantic_search(self, query, limit):
|
||||
return {"available": False, "data": [], "error": None}
|
||||
|
||||
def search_tools(self, query, per_page):
|
||||
return [{
|
||||
"owner": "official", "name": "weather",
|
||||
"description": "Weather", "score": 0.8,
|
||||
"quality_score": 91, "quality_coverage": 70,
|
||||
}]
|
||||
|
||||
with patch("cmdforge.registry_client.get_client", return_value=Client()):
|
||||
results = picker.search_registry("weather")
|
||||
assert results[0]["name"] == "official/weather"
|
||||
assert results[0]["quality_score"] == 91
|
||||
assert results[0]["quality_coverage"] == 70
|
||||
|
||||
|
||||
def test_picker_can_install_when_no_local_tools():
|
||||
registry_tool = {
|
||||
"name": "official/weather", "desc": "Weather", "args": [],
|
||||
"registry": True, "quality_score": 91, "quality_coverage": 70,
|
||||
"relevance": 0.8,
|
||||
}
|
||||
with patch.object(picker, "get_tools", return_value=[]), patch.object(
|
||||
picker, "search_registry", return_value=[registry_tool]
|
||||
), patch.object(
|
||||
picker, "_install_registry_selection",
|
||||
return_value=PickerResult("official/weather", {}),
|
||||
) as install:
|
||||
result = picker.run_picker(FakeTTY("w", "e", "\n"))
|
||||
install.assert_called_once()
|
||||
assert result.tool_name == "official/weather"
|
||||
assert "Q91/70%" in picker._ui_out.getvalue()
|
||||
|
||||
|
||||
def test_tab_installs_registry_selection_too():
|
||||
registry_tool = {
|
||||
"name": "official/weather", "desc": "Weather", "args": [],
|
||||
"registry": True,
|
||||
}
|
||||
with patch.object(picker, "get_tools", return_value=[]), patch.object(
|
||||
picker, "search_registry", return_value=[registry_tool]
|
||||
), patch.object(
|
||||
picker, "_install_registry_selection",
|
||||
return_value=PickerResult("official/weather", {}),
|
||||
) as install:
|
||||
result = picker.run_picker(FakeTTY("w", "e", "\t"))
|
||||
install.assert_called_once()
|
||||
assert result.tool_name == "official/weather"
|
||||
|
||||
|
||||
def test_local_deprecation_selection_prints_migration_guidance():
|
||||
tool = {
|
||||
"name": "old", "desc": "", "args": [], "deprecated": True,
|
||||
"deprecated_message": "Moved.", "replacement": "official/new",
|
||||
}
|
||||
result = picker._local_selection(tool)
|
||||
assert result.tool_name == "old"
|
||||
output = picker._ui_out.getvalue()
|
||||
assert "Moved." in output
|
||||
assert "official/new" in output
|
||||
|
||||
|
||||
def test_picker_help_does_not_require_a_terminal(capsys):
|
||||
with patch.object(sys, "argv", ["cf", "--help"]), pytest.raises(SystemExit) as exc:
|
||||
picker.main()
|
||||
|
||||
assert exc.value.code == 0
|
||||
assert "Interactively find and run" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_picker_version_does_not_require_a_terminal(capsys):
|
||||
with patch.object(sys, "argv", ["cf", "--version"]), pytest.raises(SystemExit) as exc:
|
||||
picker.main()
|
||||
|
||||
assert exc.value.code == 0
|
||||
assert f"cf {__version__}" in capsys.readouterr().out
|
||||
@@ -0,0 +1,509 @@
|
||||
"""Tests for preflight analysis engine."""
|
||||
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
from cmdforge.preflight import (
|
||||
PreflightReport,
|
||||
analyze_tool,
|
||||
_check_config_integrity,
|
||||
_check_similar_tools,
|
||||
_is_semver,
|
||||
)
|
||||
from cmdforge.registry_client import PaginatedResponse, RegistryError
|
||||
|
||||
|
||||
class TestPreflightReport:
|
||||
def test_empty_report_is_ok(self):
|
||||
report = PreflightReport()
|
||||
assert report.ok
|
||||
|
||||
def test_errors_make_not_ok(self):
|
||||
report = PreflightReport(errors=["bad"])
|
||||
assert not report.ok
|
||||
|
||||
def test_merge(self):
|
||||
a = PreflightReport(errors=["e1"], warnings=["w1"])
|
||||
b = PreflightReport(errors=["e2"], suggestions=["s1"])
|
||||
c = a.merge(b)
|
||||
assert c.errors == ["e1", "e2"]
|
||||
assert c.warnings == ["w1"]
|
||||
assert c.suggestions == ["s1"]
|
||||
|
||||
def test_to_dict_returns_independent_lists(self):
|
||||
report = PreflightReport(errors=["bad"])
|
||||
serialized = report.to_dict()
|
||||
serialized["errors"].append("another")
|
||||
assert report.errors == ["bad"]
|
||||
|
||||
def test_to_dict_copies_contract_proposal(self):
|
||||
report = PreflightReport(
|
||||
contract_proposal={"input_schema": {"type": "string"}}
|
||||
)
|
||||
serialized = report.to_dict()
|
||||
serialized["contract_proposal"]["input_schema"]["type"] = "integer"
|
||||
assert report.contract_proposal["input_schema"]["type"] == "string"
|
||||
|
||||
|
||||
class TestAnalyzeTool:
|
||||
def test_no_steps_warns(self, tmp_path):
|
||||
from cmdforge.tool import Tool, save_tool
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"), patch(
|
||||
"cmdforge.tool.BIN_DIR", tmp_path / "bin"
|
||||
):
|
||||
tool = Tool(name="empty-tool")
|
||||
save_tool(tool)
|
||||
|
||||
report = analyze_tool(tool)
|
||||
assert report.ok
|
||||
assert any("no steps" in w.lower() for w in report.warnings)
|
||||
|
||||
def test_missing_name(self, tmp_path):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = PreflightReport()
|
||||
tool = Tool(name="")
|
||||
_check_config_integrity(tool, report)
|
||||
assert not report.ok
|
||||
|
||||
def test_invalid_version(self, tmp_path):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = PreflightReport()
|
||||
tool = Tool(name="test", version="not-semver")
|
||||
_check_config_integrity(tool, report)
|
||||
assert not report.ok
|
||||
|
||||
def test_ok_tool(self, tmp_path):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = PreflightReport()
|
||||
tool = Tool(name="ok-tool", version="1.0.0")
|
||||
_check_config_integrity(tool, report)
|
||||
assert report.ok
|
||||
|
||||
def test_constant_tool_without_steps_is_not_passthrough_warning(self):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
report = analyze_tool(Tool(name="constant", output="hello"))
|
||||
assert not any("passthrough" in warning for warning in report.warnings)
|
||||
|
||||
def test_secret_pattern_in_prompt(self, tmp_path):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
|
||||
tool = Tool(
|
||||
name="leaky",
|
||||
steps=[PromptStep(
|
||||
prompt="Use api_key: abc123 to auth",
|
||||
provider="mock",
|
||||
output_var="out",
|
||||
)],
|
||||
)
|
||||
report = PreflightReport()
|
||||
from cmdforge.preflight import _check_secrets
|
||||
_check_secrets(tool, report)
|
||||
assert any("secret" in w.lower() for w in report.warnings)
|
||||
|
||||
def test_secret_pattern_does_not_match_compound_schema_fields(self):
|
||||
from cmdforge.tool import PromptStep, Tool
|
||||
|
||||
tool = Tool(
|
||||
name="schema-docs",
|
||||
steps=[PromptStep(
|
||||
prompt="Document max_tokens and token_count fields",
|
||||
provider="mock",
|
||||
output_var="out",
|
||||
)],
|
||||
)
|
||||
report = PreflightReport()
|
||||
from cmdforge.preflight import _check_secrets
|
||||
_check_secrets(tool, report)
|
||||
assert report.warnings == []
|
||||
|
||||
@pytest.mark.parametrize("version", ["1.2.3garbage", "1.2.3.4", "1.2"])
|
||||
def test_semver_rejects_trailing_or_incomplete_values(self, version):
|
||||
assert _is_semver(version) is False
|
||||
|
||||
def test_registry_similarity_uses_paginated_data(self):
|
||||
class Client:
|
||||
def search_tools(self, query, per_page):
|
||||
assert (query, per_page) == ("summarize", 5)
|
||||
return PaginatedResponse(data=[{
|
||||
"owner": "official",
|
||||
"name": "summary",
|
||||
"description": "Summarize text",
|
||||
"downloads": 10,
|
||||
}])
|
||||
|
||||
report = PreflightReport()
|
||||
_check_similar_tools(SimpleNamespace(name="summarize"), report, Client())
|
||||
assert report.similar_tools[0]["name"] == "official/summary"
|
||||
|
||||
def test_server_audit_does_not_claim_skipped_local_checks(self):
|
||||
from cmdforge.tool import Tool, ToolStep
|
||||
|
||||
report = analyze_tool(
|
||||
Tool(name="server", steps=[ToolStep(tool="child", output_var="out")]),
|
||||
check_local_dependencies=False,
|
||||
)
|
||||
checks = report.audit_evidence["checks_run"]
|
||||
assert "dependencies" not in checks
|
||||
assert "toolstep_compatibility" not in checks
|
||||
assert "reuse_opportunities" not in checks
|
||||
assert report.compatibility == []
|
||||
|
||||
def test_unrelated_code_steps_are_not_reuse_evidence(self):
|
||||
from cmdforge.tool import CodeStep, Tool
|
||||
|
||||
current = Tool(
|
||||
name="current", input_schema={}, output_schema={},
|
||||
steps=[
|
||||
CodeStep(code="x = 1", output_var="x"),
|
||||
CodeStep(code="y = 2", output_var="y"),
|
||||
],
|
||||
)
|
||||
other = Tool(
|
||||
name="other", input_schema={}, output_schema={},
|
||||
steps=[
|
||||
CodeStep(code="a = 99", output_var="a"),
|
||||
CodeStep(code="b = 100", output_var="b"),
|
||||
],
|
||||
)
|
||||
with patch("cmdforge.tool.list_tools", return_value=["other"]), patch(
|
||||
"cmdforge.tool.load_tool", return_value=other
|
||||
):
|
||||
report = analyze_tool(current)
|
||||
assert report.reuse_opportunities == []
|
||||
|
||||
def test_exact_contracted_duplicate_has_evidence(self):
|
||||
from cmdforge.tool import PromptStep, Tool
|
||||
|
||||
def make(name):
|
||||
return Tool(
|
||||
name=name, input_schema={}, output_schema={"type": "string"},
|
||||
steps=[
|
||||
PromptStep(
|
||||
prompt="first", provider="mock", output_var="one",
|
||||
output_schema={"type": "string"},
|
||||
),
|
||||
PromptStep(
|
||||
prompt="second", provider="mock", output_var="two",
|
||||
output_schema={"type": "string"},
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
with patch("cmdforge.tool.list_tools", return_value=["other"]), patch(
|
||||
"cmdforge.tool.load_tool", return_value=make("other")
|
||||
):
|
||||
report = analyze_tool(make("current"))
|
||||
duplicate = [
|
||||
item for item in report.reuse_opportunities
|
||||
if item["type"] == "duplicate_sequence"
|
||||
]
|
||||
assert duplicate
|
||||
assert "explicit tool contracts" in duplicate[0]["evidence"]
|
||||
|
||||
def test_registry_error_becomes_warning(self):
|
||||
class Client:
|
||||
def search_tools(self, query, per_page):
|
||||
raise RegistryError("CONNECTION_ERROR", "offline")
|
||||
|
||||
report = PreflightReport()
|
||||
_check_similar_tools(SimpleNamespace(name="summarize"), report, Client())
|
||||
assert report.warnings == ["Could not search registry for similar tools"]
|
||||
|
||||
def test_programming_error_is_not_swallowed(self):
|
||||
class Client:
|
||||
def search_tools(self, query, per_page):
|
||||
raise TypeError("broken adapter")
|
||||
|
||||
with pytest.raises(TypeError, match="broken adapter"):
|
||||
_check_similar_tools(
|
||||
SimpleNamespace(name="summarize"), PreflightReport(), Client()
|
||||
)
|
||||
|
||||
|
||||
class TestContractInference:
|
||||
def test_no_arguments_no_input(self):
|
||||
from cmdforge.tool import Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
tool = Tool(name="empty", output="{response}")
|
||||
result = infer_contracts(tool)
|
||||
assert result["input_schema"] is None
|
||||
assert result["output_schema"] is None
|
||||
assert result["confidence"] == "low"
|
||||
|
||||
def test_argument_becomes_property(self):
|
||||
from cmdforge.tool import Tool, ToolArgument
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
tool = Tool(
|
||||
name="greet",
|
||||
arguments=[ToolArgument(flag="--name", variable="name", description="Who to greet")],
|
||||
output="Hello",
|
||||
)
|
||||
result = infer_contracts(tool)
|
||||
assert result["input_schema"] is not None
|
||||
assert "name" in result["input_schema"]["properties"]
|
||||
assert result["input_schema"]["properties"]["name"]["description"] == "Who to greet"
|
||||
|
||||
def test_stdin_detected_when_input_used(self):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
tool = Tool(
|
||||
name="echo",
|
||||
steps=[PromptStep(prompt="Repeat: {input}", provider="mock", output_var="response")],
|
||||
output="{response}",
|
||||
)
|
||||
result = infer_contracts(tool)
|
||||
assert result["input_schema"] is not None
|
||||
assert "input" in result["input_schema"]["properties"]
|
||||
assert any("stdin" in e.lower() for e in result["evidence"])
|
||||
|
||||
def test_enum_propagates(self):
|
||||
from cmdforge.tool import Tool, ToolArgument
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
arg = ToolArgument(flag="--mode", variable="mode")
|
||||
arg.enum = ["fast", "accurate"]
|
||||
tool = Tool(name="tool", arguments=[arg], output="done")
|
||||
result = infer_contracts(tool)
|
||||
assert result["input_schema"]["properties"]["mode"]["enum"] == ["fast", "accurate"]
|
||||
|
||||
def test_explicit_empty_schema_not_overwritten(self):
|
||||
from cmdforge.tool import Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
tool = Tool(name="tool", input_schema={}, output_schema={})
|
||||
result = infer_contracts(tool)
|
||||
assert result["input_schema"] is None
|
||||
assert result["output_schema"] is None
|
||||
|
||||
def test_output_propagated_from_step_schema(self):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
step = PromptStep(
|
||||
prompt="Analyze: {input}",
|
||||
provider="mock",
|
||||
output_var="analysis",
|
||||
)
|
||||
step.output_schema = {
|
||||
"type": "object",
|
||||
"properties": {"summary": {"type": "string"}},
|
||||
}
|
||||
tool = Tool(name="analyze", steps=[step], output="{analysis}")
|
||||
result = infer_contracts(tool)
|
||||
assert result["output_schema"] is not None
|
||||
assert result["output_schema"]["type"] == "object"
|
||||
assert "summary" in result["output_schema"]["properties"]
|
||||
assert result["confidence"] == "high"
|
||||
|
||||
@pytest.mark.parametrize("schema_type", ["string", "number", "array", "boolean"])
|
||||
def test_non_object_output_schema_is_propagated(self, schema_type):
|
||||
from cmdforge.tool import PromptStep, Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
schema = {"type": schema_type}
|
||||
step = PromptStep(
|
||||
prompt="Generate", provider="mock", output_var="result",
|
||||
output_schema=schema,
|
||||
)
|
||||
proposal = infer_contracts(
|
||||
Tool(name="typed", steps=[step], output="{result}")
|
||||
)
|
||||
assert proposal["output_schema"] == schema
|
||||
|
||||
def test_proposed_schema_does_not_alias_step_schema(self):
|
||||
from cmdforge.tool import PromptStep, Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
step = PromptStep(
|
||||
prompt="Generate", provider="mock", output_var="result",
|
||||
output_schema={
|
||||
"type": "object",
|
||||
"properties": {"value": {"type": "string"}},
|
||||
},
|
||||
)
|
||||
proposal = infer_contracts(
|
||||
Tool(name="copy", steps=[step], output="{result}")
|
||||
)
|
||||
proposal["output_schema"]["properties"]["value"]["type"] = "integer"
|
||||
assert step.output_schema["properties"]["value"]["type"] == "string"
|
||||
|
||||
def test_stdin_detected_in_nested_tool_input(self):
|
||||
from cmdforge.tool import Tool, ToolStep
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
proposal = infer_contracts(Tool(
|
||||
name="nested",
|
||||
steps=[ToolStep(tool="child", output_var="result")],
|
||||
output="{result}",
|
||||
))
|
||||
assert "input" in proposal["input_schema"]["properties"]
|
||||
|
||||
def test_evidence_uses_real_argument_flag(self):
|
||||
from cmdforge.tool import Tool, ToolArgument
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
proposal = infer_contracts(Tool(
|
||||
name="flags",
|
||||
arguments=[ToolArgument(flag="-n", variable="name")],
|
||||
output="constant",
|
||||
))
|
||||
assert any("argument -n" in item for item in proposal["evidence"])
|
||||
|
||||
def test_explicit_contracts_produce_high_confidence(self):
|
||||
from cmdforge.tool import Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
proposal = infer_contracts(
|
||||
Tool(name="explicit", input_schema={}, output_schema={})
|
||||
)
|
||||
assert proposal["confidence"] == "high"
|
||||
|
||||
def test_bare_variable_output_stays_bare(self):
|
||||
from cmdforge.tool import Tool
|
||||
from cmdforge.preflight import infer_contracts
|
||||
|
||||
tool = Tool(name="passthrough", steps=[], output="{response}")
|
||||
result = infer_contracts(tool)
|
||||
assert result["output_schema"] is None
|
||||
assert any("source not found" in e.lower() for e in result["evidence"])
|
||||
|
||||
|
||||
class TestConformanceIntegration:
|
||||
def test_contract_tests_are_opt_in_for_server_side_preflight(self):
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
tool = Tool(name="server-safe", input_schema={}, output_schema={})
|
||||
with patch("cmdforge.contract_testing.run_contract_tests") as runner:
|
||||
analyze_tool(tool)
|
||||
runner.assert_not_called()
|
||||
|
||||
def test_analyze_tool_records_passing_contract_case(self):
|
||||
from cmdforge.tool import PromptStep, Tool
|
||||
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"answer": {"type": "string"}},
|
||||
"required": ["answer"],
|
||||
}
|
||||
tool = Tool(
|
||||
name="contracted",
|
||||
input_schema={},
|
||||
output_schema=schema,
|
||||
steps=[PromptStep(
|
||||
prompt="Answer", provider="paid-provider", output_var="answer",
|
||||
output_schema=schema,
|
||||
)],
|
||||
output="{answer}",
|
||||
)
|
||||
report = analyze_tool(tool, include_contract_tests=True)
|
||||
assert report.ok
|
||||
assert report.generated_tests[0]["state"] == "passed"
|
||||
|
||||
def test_unsupported_execution_is_advisory_not_failure(self):
|
||||
from cmdforge.tool import CodeStep, Tool
|
||||
|
||||
tool = Tool(
|
||||
name="code-tool",
|
||||
input_schema={},
|
||||
output_schema={},
|
||||
steps=[CodeStep(code="result = 1", output_var="result")],
|
||||
output="{result}",
|
||||
)
|
||||
report = analyze_tool(tool, include_contract_tests=True)
|
||||
assert report.ok
|
||||
assert report.generated_tests[0]["state"] == "unsupported"
|
||||
assert any("unsupported" in warning.lower() for warning in report.warnings)
|
||||
|
||||
def test_saved_baseline_is_compared_during_local_preflight(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
run_contract_tests, save_conformance_baseline,
|
||||
)
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
baseline_tool = Tool(
|
||||
name="regression", version="1.0.0", input_schema={},
|
||||
output_schema={"type": "string"}, output="before",
|
||||
path=tmp_path / "config.yaml",
|
||||
)
|
||||
save_conformance_baseline(
|
||||
baseline_tool, run_contract_tests(baseline_tool), tmp_path
|
||||
)
|
||||
current = Tool(
|
||||
name="regression", version="2.0.0", input_schema={},
|
||||
output_schema={"type": "string"}, output="after",
|
||||
path=tmp_path / "config.yaml",
|
||||
)
|
||||
report = analyze_tool(current, include_contract_tests=True)
|
||||
assert report.regression["has_regressions"] is True
|
||||
assert report.regression["current_version"] == "2.0.0"
|
||||
assert any("Regression comparison" in warning for warning in report.warnings)
|
||||
|
||||
def test_baseline_inputs_are_rerun_after_input_schema_change(self, tmp_path):
|
||||
from cmdforge.contract_testing import (
|
||||
run_contract_tests, save_conformance_baseline,
|
||||
)
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
baseline_tool = Tool(
|
||||
name="input-change", version="1.0.0", input_schema={},
|
||||
output_schema={"type": "string"}, output="stable",
|
||||
path=tmp_path / "config.yaml",
|
||||
)
|
||||
save_conformance_baseline(
|
||||
baseline_tool, run_contract_tests(baseline_tool), tmp_path
|
||||
)
|
||||
current = Tool(
|
||||
name="input-change", version="2.0.0",
|
||||
input_schema={
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
},
|
||||
output_schema={"type": "string"}, output="stable",
|
||||
path=tmp_path / "config.yaml",
|
||||
)
|
||||
report = analyze_tool(current, include_contract_tests=True)
|
||||
assert report.regression["has_regressions"] is True
|
||||
assert len(report.regression["regressed"]) == 1
|
||||
assert any(
|
||||
result["input_value"] == {} and result["state"] == "failed"
|
||||
for result in report.generated_tests
|
||||
)
|
||||
|
||||
def test_invalid_baseline_is_an_advisory(self, tmp_path):
|
||||
from cmdforge.contract_testing import CONFORMANCE_FILE
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
(tmp_path / CONFORMANCE_FILE).write_text("not json")
|
||||
tool = Tool(
|
||||
name="invalid-baseline", input_schema={}, output_schema={},
|
||||
output="ok", path=tmp_path / "config.yaml",
|
||||
)
|
||||
report = analyze_tool(tool, include_contract_tests=True)
|
||||
assert report.ok
|
||||
assert any("baseline is invalid" in warning.lower() for warning in report.warnings)
|
||||
|
||||
def test_partial_unsupported_conformance_is_not_hidden_by_a_pass(self, monkeypatch):
|
||||
from cmdforge.contract_testing import ConformanceReport, ConformanceResult
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.contract_testing.run_contract_tests",
|
||||
lambda tool, test_inputs=None: ConformanceReport([
|
||||
ConformanceResult("case-a", "passed", "valid"),
|
||||
ConformanceResult("input-generation", "unsupported", "unsupported keyword"),
|
||||
]),
|
||||
)
|
||||
tool = Tool(name="mixed", input_schema={}, output_schema={})
|
||||
report = analyze_tool(tool, include_contract_tests=True)
|
||||
assert any("unsupported keyword" in warning for warning in report.warnings)
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Tests for prompt optimization (M9.1)."""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from cmdforge.prompt_optimizer import (
|
||||
PromptVariation,
|
||||
OptimizationResult,
|
||||
generate_variations,
|
||||
optimize_tool,
|
||||
_get_strategies,
|
||||
_truncate,
|
||||
)
|
||||
|
||||
|
||||
class TestTruncate:
|
||||
def test_short_text_unchanged(self):
|
||||
assert _truncate("hello", 10) == "hello"
|
||||
|
||||
def test_long_text_truncated(self):
|
||||
result = _truncate("this is a very long sentence that needs truncation", 20)
|
||||
assert len(result) <= 20
|
||||
assert result.endswith("...")
|
||||
|
||||
|
||||
class TestGenerateVariations:
|
||||
def test_mock_provider_never_calls_external_provider(self):
|
||||
with patch("cmdforge.providers.call_provider") as call:
|
||||
_get_strategies("Summarize the text", 2, provider="mock")
|
||||
call.assert_not_called()
|
||||
|
||||
def test_selected_provider_is_honored(self):
|
||||
from cmdforge.providers import ProviderResult
|
||||
with patch(
|
||||
"cmdforge.providers.call_provider",
|
||||
return_value=ProviderResult(text="A materially clearer instruction", success=True),
|
||||
) as call:
|
||||
result = _get_strategies("Summarize the text", 1, provider="chosen")
|
||||
assert result[0][1] == "A materially clearer instruction"
|
||||
assert call.call_args.args[0] == "chosen"
|
||||
|
||||
def test_generates_for_prompt_steps(self, tmp_path):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="test-tool",
|
||||
steps=[
|
||||
PromptStep(
|
||||
prompt="Analyze the input and produce a summary",
|
||||
provider="mock",
|
||||
output_var="summary",
|
||||
),
|
||||
],
|
||||
output="{summary}",
|
||||
)
|
||||
variations = generate_variations(tool, count=3)
|
||||
assert len(variations) >= 1
|
||||
for v in variations:
|
||||
assert v.step_index == 0
|
||||
assert v.original == "Analyze the input and produce a summary"
|
||||
assert v.variation != v.original
|
||||
|
||||
def test_skips_non_prompt_steps(self, tmp_path):
|
||||
from cmdforge.tool import Tool, CodeStep
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="test-tool",
|
||||
steps=[CodeStep(code="print('hello')", output_var="out")],
|
||||
output="{out}",
|
||||
)
|
||||
variations = generate_variations(tool)
|
||||
assert len(variations) == 0
|
||||
|
||||
|
||||
class TestGetStrategies:
|
||||
def test_produces_variations(self):
|
||||
strategies = _get_strategies("Summarize the text", 3)
|
||||
assert len(strategies) == 3
|
||||
for strat, text in strategies:
|
||||
assert len(text) > 0
|
||||
assert strat in ("focused", "concise", "detailed", "rephrase", "structure")
|
||||
|
||||
def test_truncates_concise(self):
|
||||
original = "A" * 200
|
||||
strategies = _get_strategies(original, 3)
|
||||
concise = [t for s, t in strategies if s == "concise"]
|
||||
if concise:
|
||||
assert len(concise[0]) <= 125 # 120 + "..."
|
||||
|
||||
|
||||
class TestOptimizeTool:
|
||||
def test_structural_results_do_not_claim_a_best_prompt(self):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
tool = Tool(
|
||||
name="structural", steps=[PromptStep("Summarize", "mock", "out")],
|
||||
output="{out}", input_schema={"type": "string"},
|
||||
output_schema={"type": "string"},
|
||||
)
|
||||
result = optimize_tool(tool, count=2)
|
||||
assert result.best is None
|
||||
assert "cannot compare prompt semantics" in result.note
|
||||
|
||||
def test_behavioral_evaluator_selects_only_an_improvement(self):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
tool = Tool(
|
||||
name="behavioral", steps=[PromptStep("Do it", "mock", "out")],
|
||||
output="{out}",
|
||||
)
|
||||
result = optimize_tool(
|
||||
tool, count=3,
|
||||
evaluator=lambda candidate: len(candidate.steps[0].prompt),
|
||||
)
|
||||
assert result.best is not None
|
||||
assert result.scores[result.best_index] > result.baseline_score
|
||||
def test_baseline_score_recorded(self, tmp_path):
|
||||
from cmdforge.tool import Tool, PromptStep
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="opt-test",
|
||||
steps=[
|
||||
PromptStep(
|
||||
prompt="Summarize: {input}",
|
||||
provider="mock",
|
||||
output_var="result",
|
||||
),
|
||||
],
|
||||
output="{result}",
|
||||
)
|
||||
result = optimize_tool(tool, count=2)
|
||||
assert result.tool_name == "opt-test"
|
||||
assert result.baseline_score >= 0
|
||||
assert len(result.variations) >= 1
|
||||
assert len(result.scores) == len(result.variations)
|
||||
|
||||
def test_no_prompt_steps_returns_empty(self, tmp_path):
|
||||
from cmdforge.tool import Tool, CodeStep
|
||||
|
||||
with patch("cmdforge.tool.TOOLS_DIR", tmp_path / ".cmdforge"):
|
||||
tool = Tool(
|
||||
name="code-only",
|
||||
steps=[CodeStep(code="pass", output_var="x")],
|
||||
output="{x}",
|
||||
)
|
||||
result = optimize_tool(tool)
|
||||
assert len(result.variations) == 0
|
||||
assert result.best is None
|
||||
|
||||
|
||||
class TestPromptVariation:
|
||||
def test_to_dict(self):
|
||||
v = PromptVariation(
|
||||
step_index=0,
|
||||
original="hello",
|
||||
variation="hi there",
|
||||
strategy="rephrase",
|
||||
)
|
||||
d = v.to_dict()
|
||||
assert d["original"] == "hello"
|
||||
assert d["strategy"] == "rephrase"
|
||||
|
||||
|
||||
class TestOptimizationResult:
|
||||
def test_improvement_calculation(self):
|
||||
result = OptimizationResult(
|
||||
tool_name="test",
|
||||
variations=[
|
||||
PromptVariation(0, "a", "b", "rephrase"),
|
||||
],
|
||||
scores={0: 5},
|
||||
best_index=0,
|
||||
baseline_score=3,
|
||||
)
|
||||
assert result.best is not None
|
||||
assert result.improvement > 0
|
||||
|
||||
def test_no_improvement_when_no_baseline(self):
|
||||
result = OptimizationResult(
|
||||
tool_name="test",
|
||||
variations=[],
|
||||
scores={},
|
||||
baseline_score=0,
|
||||
)
|
||||
assert result.best is None
|
||||
assert result.improvement == 0.0
|
||||
+293
-2
@@ -8,8 +8,8 @@ import pytest
|
||||
import yaml
|
||||
|
||||
from cmdforge.providers import (
|
||||
Provider, ProviderResult,
|
||||
load_providers, save_providers, get_provider,
|
||||
Provider, ProviderResult, ProviderExecutionPolicy,
|
||||
load_providers, save_providers, get_provider, get_providers_file,
|
||||
add_provider, delete_provider,
|
||||
call_provider, discover_installed_providers, mock_provider,
|
||||
DEFAULT_PROVIDERS, strip_ansi
|
||||
@@ -60,6 +60,19 @@ class TestProvider:
|
||||
provider = Provider.from_dict(data)
|
||||
assert provider.description == ""
|
||||
|
||||
def test_legacy_policy_field_aliases_are_canonicalized(self):
|
||||
provider = Provider.from_dict({
|
||||
"name": "legacy-policy",
|
||||
"command": "provider",
|
||||
"model_id": "model-a",
|
||||
"max_input_tokens": 8192,
|
||||
})
|
||||
|
||||
assert provider.model == "model-a"
|
||||
assert provider.max_context_tokens == 8192
|
||||
assert provider.to_dict()["model"] == "model-a"
|
||||
assert provider.to_dict()["max_context_tokens"] == 8192
|
||||
|
||||
def test_roundtrip(self):
|
||||
original = Provider(
|
||||
name="custom",
|
||||
@@ -90,6 +103,13 @@ class TestProvider:
|
||||
fallback_chain=["backup", "mock"],
|
||||
api_key_env="TEST_API_KEY",
|
||||
pty_config={"prompt_pattern": ">"},
|
||||
locality="remote",
|
||||
capabilities=["text", "structured-json"],
|
||||
model_digest="sha256:abc",
|
||||
cost_class="low",
|
||||
latency_class="fast",
|
||||
max_context_tokens=131072,
|
||||
data_policy="internal",
|
||||
)
|
||||
|
||||
assert Provider.from_dict(original.to_dict()) == original
|
||||
@@ -172,6 +192,32 @@ class TestProviderPersistence:
|
||||
|
||||
assert oct(temp_providers_file.stat().st_mode & 0o777) == "0o600"
|
||||
|
||||
def test_first_run_without_discovered_providers_writes_defaults(
|
||||
self, temp_providers_file, capsys
|
||||
):
|
||||
with patch(
|
||||
"cmdforge.providers.discover_installed_providers", return_value=[]
|
||||
):
|
||||
result = get_providers_file()
|
||||
|
||||
assert result == temp_providers_file
|
||||
assert temp_providers_file.is_file()
|
||||
assert "No AI providers detected" in capsys.readouterr().err
|
||||
|
||||
def test_malformed_access_policy_fails_closed(self, temp_providers_file, capsys):
|
||||
temp_providers_file.parent.mkdir(parents=True)
|
||||
temp_providers_file.write_text(yaml.safe_dump({
|
||||
"version": 2,
|
||||
"providers": [{
|
||||
"name": "claude",
|
||||
"command": "claude -p",
|
||||
"tools": "should-have-been-a-list",
|
||||
}],
|
||||
}))
|
||||
|
||||
assert load_providers() == []
|
||||
assert "Failed to load provider configuration" in capsys.readouterr().err
|
||||
|
||||
def test_legacy_config_gains_missing_defaults_without_losing_custom_provider(self, temp_providers_file):
|
||||
temp_providers_file.parent.mkdir(parents=True)
|
||||
temp_providers_file.write_text(yaml.safe_dump({
|
||||
@@ -502,6 +548,200 @@ class TestProviderFallback:
|
||||
# Fallback to mock should succeed
|
||||
assert result.success is True
|
||||
assert "[MOCK]" in result.text
|
||||
assert result.requested_provider == "primary"
|
||||
assert result.actual_provider == "mock"
|
||||
assert result.attempted_providers == ["primary", "mock"]
|
||||
assert result.fallback_used is True
|
||||
|
||||
def test_no_fallback_fails_closed(self, temp_providers_file):
|
||||
save_providers([
|
||||
Provider("primary", "nonexistent-cmd", fallback="mock"),
|
||||
Provider("mock", "mock"),
|
||||
])
|
||||
|
||||
result = call_provider(
|
||||
"primary", "Test prompt",
|
||||
execution_policy=ProviderExecutionPolicy(allow_fallback=False),
|
||||
)
|
||||
|
||||
assert result.success is False
|
||||
assert result.requested_provider == "primary"
|
||||
assert result.actual_provider is None
|
||||
assert result.attempted_providers == ["primary"]
|
||||
assert result.fallback_used is False
|
||||
|
||||
def test_private_data_requires_explicit_compatible_policy(
|
||||
self, temp_providers_file
|
||||
):
|
||||
save_providers([
|
||||
Provider(
|
||||
"local", "unused", locality="local",
|
||||
capabilities=["structured-json"],
|
||||
),
|
||||
])
|
||||
|
||||
result = call_provider(
|
||||
"local", "private packet",
|
||||
execution_policy=ProviderExecutionPolicy(
|
||||
allow_fallback=False,
|
||||
required_locality="local",
|
||||
required_capabilities=("structured-json",),
|
||||
data_classification="private",
|
||||
),
|
||||
)
|
||||
|
||||
assert result.success is False
|
||||
assert "data_policy is unspecified" in result.error
|
||||
|
||||
def test_require_local_rejects_ollama_redirected_to_remote_host(
|
||||
self, temp_providers_file, monkeypatch
|
||||
):
|
||||
save_providers([
|
||||
Provider(
|
||||
"ollama-private", "ollama run safe", locality="local",
|
||||
data_policy="private",
|
||||
),
|
||||
])
|
||||
monkeypatch.setenv("OLLAMA_HOST", "gpu.example.test:11434")
|
||||
|
||||
result = call_provider(
|
||||
"ollama-private", "private packet",
|
||||
execution_policy=ProviderExecutionPolicy(
|
||||
allow_fallback=False,
|
||||
required_locality="local",
|
||||
data_classification="private",
|
||||
),
|
||||
)
|
||||
|
||||
assert result.success is False
|
||||
assert "locality is remote" in result.error
|
||||
|
||||
def test_ollama_digest_is_observed_from_local_api(
|
||||
self, temp_providers_file, monkeypatch
|
||||
):
|
||||
save_providers([
|
||||
Provider(
|
||||
"ollama-safe", "ollama run safe:latest", model="safe:latest",
|
||||
locality="local", data_policy="private",
|
||||
),
|
||||
])
|
||||
monkeypatch.delenv("OLLAMA_HOST", raising=False)
|
||||
|
||||
class Response:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return (
|
||||
b'{"models":[{"name":"safe:latest","model":"safe:latest",'
|
||||
b'"digest":"abcdef"}]}'
|
||||
)
|
||||
|
||||
with (
|
||||
patch("urllib.request.urlopen", return_value=Response()),
|
||||
patch(
|
||||
"cmdforge.providers.call_provider_subprocess",
|
||||
return_value=ProviderResult(text="safe", success=True),
|
||||
),
|
||||
):
|
||||
result = call_provider(
|
||||
"ollama-safe", "private",
|
||||
execution_policy=ProviderExecutionPolicy(
|
||||
allow_fallback=False,
|
||||
required_locality="local",
|
||||
data_classification="private",
|
||||
require_model_identity=True,
|
||||
require_model_digest=True,
|
||||
),
|
||||
)
|
||||
|
||||
assert result.model == "safe:latest"
|
||||
assert result.model_digest == "sha256:abcdef"
|
||||
assert result.model_identity_source == "ollama-api"
|
||||
|
||||
def test_policy_eligible_provider_reports_verified_metadata(
|
||||
self, temp_providers_file
|
||||
):
|
||||
provider = Provider(
|
||||
"local", "local-ai", model="model-a",
|
||||
model_digest="sha256:123", locality="local",
|
||||
capabilities=["structured-json"], data_policy="private",
|
||||
)
|
||||
save_providers([provider])
|
||||
low_level = ProviderResult(text='{"ok": true}', success=True)
|
||||
|
||||
with patch(
|
||||
"cmdforge.providers.call_provider_subprocess",
|
||||
return_value=low_level,
|
||||
):
|
||||
result = call_provider(
|
||||
"local", "private packet",
|
||||
execution_policy=ProviderExecutionPolicy(
|
||||
allow_fallback=False,
|
||||
required_locality="local",
|
||||
required_capabilities=("structured-json",),
|
||||
data_classification="private",
|
||||
require_model_identity=True,
|
||||
require_model_digest=True,
|
||||
),
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.provenance() == {
|
||||
"success": True,
|
||||
"requested_provider": "local",
|
||||
"actual_provider": "local",
|
||||
"attempted_providers": ["local"],
|
||||
"fallback_used": False,
|
||||
"model": "model-a",
|
||||
"model_digest": "sha256:123",
|
||||
"locality": "local",
|
||||
"model_identity_source": "provider-config",
|
||||
}
|
||||
|
||||
def test_fallback_chain_skips_policy_ineligible_provider(
|
||||
self, temp_providers_file
|
||||
):
|
||||
save_providers([
|
||||
Provider(
|
||||
"primary", "primary", fallback_chain=["remote", "local"],
|
||||
locality="local", data_policy="private",
|
||||
),
|
||||
Provider(
|
||||
"remote", "remote", locality="remote", data_policy="private",
|
||||
),
|
||||
Provider(
|
||||
"local", "local", locality="local", data_policy="private",
|
||||
model="safe-model",
|
||||
),
|
||||
])
|
||||
|
||||
def invoke(provider, prompt, timeout, max_tokens):
|
||||
if provider.name == "primary":
|
||||
return ProviderResult(text="", success=False, error="offline")
|
||||
assert provider.name == "local"
|
||||
return ProviderResult(text="safe", success=True)
|
||||
|
||||
with patch(
|
||||
"cmdforge.providers.call_provider_subprocess", side_effect=invoke
|
||||
) as low_level:
|
||||
result = call_provider(
|
||||
"primary", "private packet",
|
||||
execution_policy=ProviderExecutionPolicy(
|
||||
required_locality="local",
|
||||
data_classification="private",
|
||||
),
|
||||
)
|
||||
|
||||
assert result.success is True
|
||||
assert result.actual_provider == "local"
|
||||
assert result.attempted_providers == ["primary", "remote", "local"]
|
||||
assert [call.args[0].name for call in low_level.call_args_list] == [
|
||||
"primary", "local"
|
||||
]
|
||||
|
||||
def test_fallback_chain(self, temp_providers_file):
|
||||
"""Fallback can chain to another provider with fallback."""
|
||||
@@ -784,3 +1024,54 @@ class TestProviderDiscovery:
|
||||
assert len({item["name"] for item in models}) == 2
|
||||
assert models[0]["name"] == "ollama-hermes4-3-latest"
|
||||
assert models[1]["name"] == "ollama-hermes4-3-q4-k-m"
|
||||
|
||||
|
||||
class TestProviderAccessControl:
|
||||
def test_tools_allowlist_defaults_to_none(self):
|
||||
provider = Provider("test", "cmd")
|
||||
assert provider.tools is None
|
||||
assert provider.mcp_servers is None
|
||||
|
||||
def test_tools_allowlist_roundtrip(self):
|
||||
provider = Provider("test", "cmd", tools=["tool-a", "tool-b"])
|
||||
d = provider.to_dict()
|
||||
assert d["tools"] == ["tool-a", "tool-b"]
|
||||
restored = Provider.from_dict(d)
|
||||
assert restored.tools == ["tool-a", "tool-b"]
|
||||
|
||||
def test_mcp_servers_roundtrip(self):
|
||||
provider = Provider("test", "cmd", mcp_servers=["filesystem"])
|
||||
d = provider.to_dict()
|
||||
assert d["mcp_servers"] == ["filesystem"]
|
||||
restored = Provider.from_dict(d)
|
||||
assert restored.mcp_servers == ["filesystem"]
|
||||
|
||||
def test_tools_not_in_dict_when_none(self):
|
||||
provider = Provider("test", "cmd")
|
||||
assert "tools" not in provider.to_dict()
|
||||
assert "mcp_servers" not in provider.to_dict()
|
||||
|
||||
def test_empty_allowlists_survive_roundtrip(self):
|
||||
provider = Provider("locked", "cmd", tools=[], mcp_servers=[])
|
||||
serialized = provider.to_dict()
|
||||
assert serialized["tools"] == []
|
||||
assert serialized["mcp_servers"] == []
|
||||
|
||||
restored = Provider.from_dict(serialized)
|
||||
assert restored.tools == []
|
||||
assert restored.mcp_servers == []
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "value"),
|
||||
[
|
||||
("tools", "tool-a"),
|
||||
("tools", [""]),
|
||||
("tools", [1]),
|
||||
("mcp_servers", "filesystem"),
|
||||
("mcp_servers", [""]),
|
||||
("mcp_servers", [1]),
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_access_policies(self, field, value):
|
||||
with pytest.raises(ValueError, match=field):
|
||||
Provider("test", "cmd", **{field: value})
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
"""Tests for explainable quality scoring (M8.Q)."""
|
||||
|
||||
from cmdforge.quality import (
|
||||
CategoryScore,
|
||||
QualityScore,
|
||||
compute_quality,
|
||||
)
|
||||
from cmdforge.preflight import PreflightReport
|
||||
from cmdforge.tool import Tool
|
||||
|
||||
|
||||
class TestCategoryScore:
|
||||
def test_display(self):
|
||||
cs = CategoryScore("Tests", 25, 30)
|
||||
assert cs.display == "25/30"
|
||||
|
||||
def test_percentage(self):
|
||||
cs = CategoryScore("Tests", 15, 30)
|
||||
assert cs.percentage == 50.0
|
||||
|
||||
def test_zero_available(self):
|
||||
cs = CategoryScore("X", 0, 0)
|
||||
assert cs.percentage == 0.0
|
||||
|
||||
def test_display_distinguishes_missing_evidence(self):
|
||||
assert CategoryScore("Tests", 0, 30, "not_tested").display == "not tested"
|
||||
assert CategoryScore("Tests", 0, 30, "not_applicable").display == "n/a"
|
||||
|
||||
|
||||
class TestQualityScore:
|
||||
def test_to_dict(self):
|
||||
qs = QualityScore(
|
||||
tool_name="test",
|
||||
version="1.0.0",
|
||||
headline=85,
|
||||
categories=[CategoryScore("Contracts", 15, 15)],
|
||||
)
|
||||
d = qs.to_dict()
|
||||
assert d["tool"] == "test"
|
||||
assert d["score"] == 85
|
||||
assert d["categories"][0]["name"] == "Contracts"
|
||||
|
||||
def test_str_has_headline(self):
|
||||
qs = QualityScore(
|
||||
tool_name="test",
|
||||
version="1.0.0",
|
||||
headline=87,
|
||||
categories=[CategoryScore("Contracts", 15, 15)],
|
||||
)
|
||||
assert "Quality 87" in str(qs)
|
||||
|
||||
|
||||
class TestComputeQuality:
|
||||
def test_empty_tool_low_score(self):
|
||||
tool = Tool(name="bare")
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(tool, report)
|
||||
assert qs.headline < 50
|
||||
# Contracts should be not_tested
|
||||
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
|
||||
assert contracts.state == "not_tested"
|
||||
assert qs.headline == 0
|
||||
assert qs.evidence_coverage == 0
|
||||
|
||||
def test_missing_categories_do_not_reduce_checked_score(self):
|
||||
tool = Tool(name="contracted", input_schema={}, output_schema={})
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(tool, report)
|
||||
assert qs.headline == 100
|
||||
assert qs.evidence_coverage == 15
|
||||
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
|
||||
assert contracts.state == "checked"
|
||||
|
||||
def test_unsupported_tests_are_not_scored_as_failures(self):
|
||||
report = PreflightReport(generated_tests=[{
|
||||
"step": "preflight", "state": "unsupported", "detail": "safe mode"
|
||||
}])
|
||||
tests = [
|
||||
c for c in compute_quality(Tool(name="tool"), report).categories
|
||||
if c.name == "Deterministic tests"
|
||||
][0]
|
||||
assert tests.state == "not_tested"
|
||||
assert tests.display == "not tested"
|
||||
|
||||
def test_partial_unsupported_tests_reduce_coverage_not_score(self):
|
||||
report = PreflightReport(generated_tests=[
|
||||
{"step": "case", "state": "passed", "detail": "ok"},
|
||||
{"step": "generation", "state": "unsupported", "detail": "unknown"},
|
||||
])
|
||||
tests = [
|
||||
c for c in compute_quality(Tool(name="tool"), report).categories
|
||||
if c.name == "Deterministic tests"
|
||||
][0]
|
||||
assert tests.earned == tests.available == 15
|
||||
assert tests.possible == 30
|
||||
|
||||
def test_with_contracts_scores_higher(self):
|
||||
tool = Tool(
|
||||
name="contracted",
|
||||
input_schema={"type": "object", "properties": {"input": {"type": "string"}}},
|
||||
output_schema={"type": "object", "properties": {"result": {"type": "string"}}},
|
||||
)
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(tool, report)
|
||||
contracts = [c for c in qs.categories if c.name == "Contracts"][0]
|
||||
assert contracts.earned == 15
|
||||
assert contracts.state == "checked"
|
||||
|
||||
def test_with_tests_scores_higher(self):
|
||||
tool = Tool(name="tested")
|
||||
report = PreflightReport(
|
||||
generated_tests=[
|
||||
{"step": "case-1", "state": "passed", "detail": "ok"},
|
||||
{"step": "case-2", "state": "passed", "detail": "ok"},
|
||||
]
|
||||
)
|
||||
qs = compute_quality(tool, report)
|
||||
tests = [c for c in qs.categories if c.name == "Deterministic tests"][0]
|
||||
assert tests.earned == 30
|
||||
assert tests.state == "checked"
|
||||
|
||||
def test_failed_tests_reduce_score(self):
|
||||
tool = Tool(name="broken")
|
||||
report = PreflightReport(
|
||||
generated_tests=[
|
||||
{"step": "case-1", "state": "passed", "detail": "ok"},
|
||||
{"step": "case-2", "state": "failed", "detail": "broken"},
|
||||
]
|
||||
)
|
||||
qs = compute_quality(tool, report)
|
||||
tests = [c for c in qs.categories if c.name == "Deterministic tests"][0]
|
||||
assert tests.earned == 15
|
||||
|
||||
def test_regression_no_baseline_is_not_tested(self):
|
||||
tool = Tool(name="test")
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(tool, report)
|
||||
regression = [c for c in qs.categories if c.name == "Regression history"][0]
|
||||
assert regression.state == "not_tested"
|
||||
|
||||
def test_regression_stable_full_score(self):
|
||||
tool = Tool(name="test")
|
||||
report = PreflightReport(
|
||||
regression={"has_regressions": False, "summary": "no changes"}
|
||||
)
|
||||
qs = compute_quality(tool, report)
|
||||
regression = [c for c in qs.categories if c.name == "Regression history"][0]
|
||||
assert regression.earned == 20
|
||||
|
||||
def test_regression_regressions_zero(self):
|
||||
tool = Tool(name="test")
|
||||
report = PreflightReport(
|
||||
regression={"has_regressions": True, "summary": "1 regression(s)"}
|
||||
)
|
||||
qs = compute_quality(tool, report)
|
||||
regression = [c for c in qs.categories if c.name == "Regression history"][0]
|
||||
assert regression.earned == 0
|
||||
|
||||
def test_secret_warnings_reduce_security(self):
|
||||
tool = Tool(name="leaky")
|
||||
report = PreflightReport(
|
||||
warnings=["Prompt step contains potential secret pattern 'api_key'"]
|
||||
)
|
||||
qs = compute_quality(tool, report)
|
||||
security = [c for c in qs.categories if c.name == "Security scrutiny"][0]
|
||||
assert security.earned == 15 # 20 - 5
|
||||
|
||||
def test_skipped_dependency_check_is_not_claimed_as_security_evidence(self):
|
||||
tool = Tool(name="server")
|
||||
report = PreflightReport(audit_evidence={"checks_run": ["secrets"]})
|
||||
security = [
|
||||
c for c in compute_quality(tool, report).categories
|
||||
if c.name == "Security scrutiny"
|
||||
][0]
|
||||
assert security.earned == 10
|
||||
assert security.available == 10
|
||||
assert security.possible == 20
|
||||
|
||||
def test_community_not_tested_without_data(self):
|
||||
tool = Tool(name="new")
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(tool, report)
|
||||
community = [c for c in qs.categories if c.name == "Community evidence"][0]
|
||||
assert community.state == "not_tested"
|
||||
|
||||
def test_community_with_data(self):
|
||||
tool = Tool(name="popular")
|
||||
report = PreflightReport()
|
||||
qs = compute_quality(
|
||||
tool, report,
|
||||
registry_data={
|
||||
"reviews": [{"rating": 4}, {"rating": 5}],
|
||||
"downloads": 500,
|
||||
"featured": True,
|
||||
}
|
||||
)
|
||||
community = [c for c in qs.categories if c.name == "Community evidence"][0]
|
||||
assert community.earned == 14 # 7 (reviews) + 4 (downloads) + 3 (featured)
|
||||
assert community.state == "checked"
|
||||
@@ -0,0 +1,76 @@
|
||||
"""Tests for immutable registry audit refreshes."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import pytest
|
||||
|
||||
from cmdforge.registry.auditing import audit_stale_tools, audit_tool_version
|
||||
from cmdforge.registry.db import connect_db, init_db
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def audit_db(tmp_path):
|
||||
conn = connect_db(tmp_path / "registry.db")
|
||||
init_db(conn)
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO publishers (email, password_hash, slug, display_name)
|
||||
VALUES ('[email protected]', 'x', 'auditor', 'Auditor')
|
||||
"""
|
||||
)
|
||||
publisher_id = cursor.lastrowid
|
||||
tool = conn.execute(
|
||||
"""
|
||||
INSERT INTO tools (
|
||||
owner, name, version, config_yaml, publisher_id,
|
||||
visibility, moderation_status
|
||||
) VALUES ('auditor', 'checked', '1.0.0', ?, ?, 'public', 'approved')
|
||||
""",
|
||||
[
|
||||
"name: checked\nversion: 1.0.0\noutput: stable\n"
|
||||
"input_schema: {}\noutput_schema:\n type: string\n",
|
||||
publisher_id,
|
||||
],
|
||||
)
|
||||
conn.commit()
|
||||
yield conn, tool.lastrowid
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_background_audits_append_instead_of_overwriting(audit_db):
|
||||
conn, tool_id = audit_db
|
||||
first = audit_tool_version(conn, tool_id)
|
||||
second = audit_tool_version(conn, tool_id)
|
||||
count = conn.execute(
|
||||
"SELECT COUNT(*) FROM tool_audits WHERE tool_id = ?", [tool_id]
|
||||
).fetchone()[0]
|
||||
assert count == 2
|
||||
assert first["evidence"]["findings_hash"] == second["evidence"]["findings_hash"]
|
||||
assert "generated_tests" in first["evidence"]["findings"]
|
||||
assert "dependencies" not in first["evidence"]["checks_run"]
|
||||
|
||||
|
||||
def test_stale_audit_refresh_adds_new_evidence(audit_db):
|
||||
conn, tool_id = audit_db
|
||||
audit_tool_version(conn, tool_id)
|
||||
old = (datetime.now(timezone.utc) - timedelta(days=60)).isoformat()
|
||||
conn.execute("UPDATE tool_audits SET evaluated_at = ?", [old])
|
||||
conn.commit()
|
||||
results = audit_stale_tools(conn, max_age_days=30)
|
||||
assert [item["tool_id"] for item in results] == [tool_id]
|
||||
assert conn.execute("SELECT COUNT(*) FROM tool_audits").fetchone()[0] == 2
|
||||
|
||||
|
||||
def test_degraded_background_score_is_logged(audit_db):
|
||||
conn, tool_id = audit_db
|
||||
conn.execute("UPDATE tools SET downloads = 500 WHERE id = ?", [tool_id])
|
||||
conn.commit()
|
||||
audit_tool_version(conn, tool_id)
|
||||
conn.execute("UPDATE tools SET downloads = 0 WHERE id = ?", [tool_id])
|
||||
conn.commit()
|
||||
result = audit_tool_version(conn, tool_id)
|
||||
assert result["quality_change"] < 0
|
||||
log = conn.execute(
|
||||
"SELECT action FROM audit_log WHERE target_id = ?", [str(tool_id)]
|
||||
).fetchone()
|
||||
assert log["action"] == "automated_audit_changed"
|
||||
@@ -561,6 +561,7 @@ category: text-processing
|
||||
assert data["status"] == "validated"
|
||||
assert data["name"] == tool_name
|
||||
assert data["owner"] == slug
|
||||
assert data["preflight"]["errors"] == []
|
||||
|
||||
def test_publish_and_retrieve(self, session, base_url, auth_headers):
|
||||
"""Test publishing a tool and retrieving it."""
|
||||
@@ -588,8 +589,13 @@ tags:
|
||||
assert data["name"] == tool_name
|
||||
assert data["version"] == "1.0.0"
|
||||
|
||||
# Retrieve
|
||||
# Pending public tools are hidden from anonymous callers, but remain
|
||||
# visible to their owner while awaiting moderation.
|
||||
resp = session.get(f"{base_url}/tools/{slug}/{tool_name}")
|
||||
assert resp.status_code == 404
|
||||
resp = session.get(
|
||||
f"{base_url}/tools/{slug}/{tool_name}", headers=headers
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
tool = resp.json()["data"]
|
||||
assert tool["name"] == tool_name
|
||||
@@ -620,11 +626,20 @@ description: First version
|
||||
})
|
||||
assert resp.status_code == 201
|
||||
|
||||
# Duplicate publish
|
||||
# An identical retry is idempotent.
|
||||
resp = session.post(f"{base_url}/tools", headers=headers, json={
|
||||
"config": config,
|
||||
"readme": ""
|
||||
})
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["already_published"] is True
|
||||
|
||||
# Different content at the same version is a real conflict.
|
||||
changed_config = config.replace("First version", "Changed content")
|
||||
resp = session.post(f"{base_url}/tools", headers=headers, json={
|
||||
"config": changed_config,
|
||||
"readme": ""
|
||||
})
|
||||
assert resp.status_code == 409
|
||||
assert resp.json()["error"]["code"] == "VERSION_EXISTS"
|
||||
|
||||
|
||||
+248
-2
@@ -1,18 +1,20 @@
|
||||
"""Tests for runner.py - Tool execution engine."""
|
||||
|
||||
import pytest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
from cmdforge.runner import (
|
||||
substitute_variables,
|
||||
execute_prompt_step,
|
||||
execute_code_step,
|
||||
execute_tool_step,
|
||||
run_tool,
|
||||
create_argument_parser,
|
||||
collect_custom_args
|
||||
)
|
||||
from cmdforge.tool import Tool, ToolArgument, PromptStep, CodeStep
|
||||
from cmdforge.providers import ProviderResult
|
||||
from cmdforge.tool import Tool, ToolArgument, PromptStep, CodeStep, ToolStep, McpStep
|
||||
from cmdforge.providers import Provider, ProviderResult
|
||||
|
||||
|
||||
class TestSubstituteVariables:
|
||||
@@ -227,6 +229,23 @@ class TestExecutePromptStep:
|
||||
assert success is False
|
||||
assert output == ""
|
||||
|
||||
@patch('cmdforge.runner.call_provider')
|
||||
def test_step_can_deny_fallback_independently_of_caller(self, mock_call):
|
||||
mock_call.return_value = ProviderResult(
|
||||
text="private", success=True,
|
||||
)
|
||||
step = PromptStep(
|
||||
prompt="Private", provider="local", output_var="out",
|
||||
plain_text=True, fallback_policy="deny",
|
||||
)
|
||||
|
||||
output, success = execute_prompt_step(step, {"input": ""})
|
||||
|
||||
assert success is True
|
||||
assert output == "private"
|
||||
policy = mock_call.call_args.kwargs["execution_policy"]
|
||||
assert policy.allow_fallback is False
|
||||
|
||||
@patch('cmdforge.runner.mock_provider')
|
||||
def test_mock_provider_used(self, mock_mock):
|
||||
mock_mock.return_value = ProviderResult(text="mock response", success=True)
|
||||
@@ -986,6 +1005,58 @@ class TestCreateArgumentParser:
|
||||
|
||||
assert args.count == "10"
|
||||
|
||||
def test_parser_enforces_typed_boolean_integer_and_enum_arguments(self):
|
||||
tool = Tool(
|
||||
name="typed",
|
||||
arguments=[
|
||||
ToolArgument(
|
||||
flag="--project", variable="project",
|
||||
type="boolean", default=False,
|
||||
),
|
||||
ToolArgument(
|
||||
flag="--count", variable="count",
|
||||
type="integer", required=True,
|
||||
),
|
||||
ToolArgument(
|
||||
flag="--mode", variable="mode",
|
||||
enum=["safe", "fast"], default="safe",
|
||||
),
|
||||
],
|
||||
)
|
||||
parser = create_argument_parser(tool)
|
||||
|
||||
args = parser.parse_args(["--project", "--count", "3", "--mode", "fast"])
|
||||
|
||||
assert args.project is True
|
||||
assert args.count == 3
|
||||
assert args.mode == "fast"
|
||||
|
||||
def test_parser_reuses_matching_legacy_boolean_universal_flag(self):
|
||||
tool = Tool(
|
||||
name="legacy",
|
||||
arguments=[
|
||||
ToolArgument(
|
||||
flag="--verbose", variable="verbose", type="bool"
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
args = create_argument_parser(tool).parse_args(["--verbose"])
|
||||
|
||||
assert args.verbose is True
|
||||
assert collect_custom_args(tool, args) == {"verbose": True}
|
||||
|
||||
def test_parser_rejects_incompatible_universal_flag_collision(self):
|
||||
tool = Tool(
|
||||
name="conflict",
|
||||
arguments=[
|
||||
ToolArgument(flag="--provider", variable="destination")
|
||||
],
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="universal run options"):
|
||||
create_argument_parser(tool)
|
||||
|
||||
def test_parser_input_output_flags(self):
|
||||
tool = Tool(name="test")
|
||||
parser = create_argument_parser(tool)
|
||||
@@ -1010,3 +1081,178 @@ class TestCreateArgumentParser:
|
||||
"max_size": "50",
|
||||
"format": "json",
|
||||
}
|
||||
|
||||
|
||||
class TestProviderExecutionPolicies:
|
||||
def test_denied_tool_is_not_resolved(self, monkeypatch):
|
||||
resolver = MagicMock()
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", tools=[]),
|
||||
)
|
||||
|
||||
output, success = execute_tool_step(
|
||||
ToolStep(tool="forbidden", output_var="out", provider="locked"),
|
||||
{"input": ""},
|
||||
)
|
||||
|
||||
assert (output, success) == ("", False)
|
||||
resolver.assert_not_called()
|
||||
|
||||
def test_unknown_provider_is_rejected_before_resolution(self, monkeypatch):
|
||||
resolver = MagicMock()
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
|
||||
monkeypatch.setattr("cmdforge.providers.get_provider", lambda name: None)
|
||||
|
||||
_, success = execute_tool_step(
|
||||
ToolStep(tool="anything", output_var="out", provider="missing"),
|
||||
{"input": ""},
|
||||
)
|
||||
|
||||
assert success is False
|
||||
resolver.assert_not_called()
|
||||
|
||||
def test_provider_and_delegated_allowlists_both_apply(self, monkeypatch):
|
||||
leaf = Tool(name="safe-lint", output="completed")
|
||||
resolver = MagicMock(return_value=SimpleNamespace(tool=leaf))
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", tools=["safe-*"]),
|
||||
)
|
||||
|
||||
output, success = execute_tool_step(
|
||||
ToolStep(tool="safe-lint", output_var="out", provider="restricted"),
|
||||
{"input": ""},
|
||||
agent_tool_policies=[["*-lint"]],
|
||||
)
|
||||
|
||||
assert (output, success) == ("completed", True)
|
||||
resolver.assert_called_once_with("safe-lint")
|
||||
|
||||
def test_delegated_allowlist_can_further_restrict_provider(self, monkeypatch):
|
||||
resolver = MagicMock()
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", tools=["*"]),
|
||||
)
|
||||
|
||||
_, success = execute_tool_step(
|
||||
ToolStep(tool="unsafe-delete", output_var="out", provider="broad"),
|
||||
{"input": ""},
|
||||
agent_tool_policies=[["safe-*"]],
|
||||
)
|
||||
|
||||
assert success is False
|
||||
resolver.assert_not_called()
|
||||
|
||||
def test_mcp_server_policy_denies_before_manager_creation(self, monkeypatch):
|
||||
manager = MagicMock()
|
||||
monkeypatch.setattr("cmdforge.runner.McpClientManager", manager)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", mcp_servers=[]),
|
||||
)
|
||||
tool = Tool(
|
||||
name="mcp-tool",
|
||||
steps=[McpStep(server="database", tool="query", output_var="result")],
|
||||
output="{result}",
|
||||
)
|
||||
|
||||
output, exit_code = run_tool(
|
||||
tool, "", {}, provider_override="restricted"
|
||||
)
|
||||
|
||||
assert (output, exit_code) == ("", 4)
|
||||
manager.assert_not_called()
|
||||
|
||||
def test_mcp_server_policy_allows_matching_server(self, monkeypatch):
|
||||
class FakeManager:
|
||||
def call_tool(self, server, tool, arguments, result_mode="auto"):
|
||||
return "allowed"
|
||||
|
||||
monkeypatch.setattr("cmdforge.runner.McpClientManager", FakeManager)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", mcp_servers=["data-*"]),
|
||||
)
|
||||
tool = Tool(
|
||||
name="mcp-tool",
|
||||
steps=[McpStep(server="data-local", tool="query", output_var="result")],
|
||||
output="{result}",
|
||||
)
|
||||
|
||||
assert run_tool(tool, "", {}, provider_override="restricted") == (
|
||||
"allowed", 0
|
||||
)
|
||||
|
||||
def test_agent_context_propagates_through_nested_tools(self, monkeypatch):
|
||||
captured = {}
|
||||
leaf = Tool(
|
||||
name="leaf",
|
||||
steps=[PromptStep("prompt", "original", "answer")],
|
||||
output="{answer}",
|
||||
)
|
||||
middle = Tool(
|
||||
name="middle",
|
||||
steps=[ToolStep(tool="leaf", output_var="child")],
|
||||
output="{child}",
|
||||
)
|
||||
root = Tool(
|
||||
name="root",
|
||||
steps=[
|
||||
ToolStep(
|
||||
tool="middle",
|
||||
output_var="delegated",
|
||||
provider="delegate",
|
||||
profile="architect",
|
||||
skills=["python"],
|
||||
tools=["leaf"],
|
||||
)
|
||||
],
|
||||
output="{delegated}",
|
||||
)
|
||||
|
||||
def resolve(name):
|
||||
return SimpleNamespace(tool={"middle": middle, "leaf": leaf}[name])
|
||||
|
||||
def execute_prompt(step, *args, **kwargs):
|
||||
captured["step"] = step
|
||||
captured["provider_override"] = args[1]
|
||||
return "done", True
|
||||
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolve)
|
||||
monkeypatch.setattr("cmdforge.runner.execute_prompt_step", execute_prompt)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", tools=["middle", "leaf"]),
|
||||
)
|
||||
|
||||
assert run_tool(root, "", {}) == ("done", 0)
|
||||
assert captured["step"].profile == "architect"
|
||||
assert captured["step"].skills == ["python"]
|
||||
assert captured["provider_override"] == "delegate"
|
||||
|
||||
def test_descendant_cannot_broaden_ancestor_tool_policy(self, monkeypatch):
|
||||
resolver = MagicMock()
|
||||
monkeypatch.setattr("cmdforge.runner.resolve_tool", resolver)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.providers.get_provider",
|
||||
lambda name: Provider(name, "cmd", tools=["*"]),
|
||||
)
|
||||
|
||||
_, success = execute_tool_step(
|
||||
ToolStep(
|
||||
tool="dangerous",
|
||||
output_var="out",
|
||||
provider="delegate",
|
||||
tools=["*"],
|
||||
),
|
||||
{"input": ""},
|
||||
agent_tool_policies=[["safe-*"]],
|
||||
)
|
||||
|
||||
assert success is False
|
||||
resolver.assert_not_called()
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
"""Tests for conservative ToolStep JSON Schema compatibility."""
|
||||
|
||||
from unittest.mock import patch
|
||||
|
||||
from cmdforge.preflight import (
|
||||
PreflightReport, _check_toolstep_compatibility, analyze_tool,
|
||||
)
|
||||
from cmdforge.schema_compat import compare_json_schemas
|
||||
from cmdforge.tool import McpStep, PromptStep, Tool, ToolStep
|
||||
|
||||
|
||||
class TestSchemaCompatibility:
|
||||
def test_type_mismatch_is_incompatible(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "string"}, {"type": "integer"}
|
||||
)
|
||||
assert result.state == "incompatible"
|
||||
|
||||
def test_integer_is_accepted_by_number(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "integer"}, {"type": "number"}
|
||||
)
|
||||
assert result.state == "compatible"
|
||||
|
||||
def test_missing_required_property_is_incompatible(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "object", "properties": {}},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
},
|
||||
)
|
||||
assert result.state == "incompatible"
|
||||
assert "required" in result.detail
|
||||
|
||||
def test_matching_object_property_is_compatible(self):
|
||||
schema = {
|
||||
"type": "object",
|
||||
"properties": {"name": {"type": "string"}},
|
||||
"required": ["name"],
|
||||
}
|
||||
assert compare_json_schemas(schema, schema).state == "compatible"
|
||||
|
||||
def test_literal_enum_value_is_checked(self):
|
||||
assert compare_json_schemas(
|
||||
{"const": "fast"}, {"type": "string", "enum": ["fast", "slow"]}
|
||||
).state == "compatible"
|
||||
assert compare_json_schemas(
|
||||
{"const": "invalid"}, {"type": "string", "enum": ["fast"]}
|
||||
).state == "incompatible"
|
||||
|
||||
def test_unconstrained_producer_does_not_satisfy_consumer_enum(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "string"}, {"type": "string", "enum": ["fast"]}
|
||||
)
|
||||
assert result.state == "incompatible"
|
||||
|
||||
def test_closed_consumer_rejects_open_producer_object(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "object", "properties": {}},
|
||||
{"type": "object", "properties": {}, "additionalProperties": False},
|
||||
)
|
||||
assert result.state == "incompatible"
|
||||
|
||||
def test_open_producer_cannot_guarantee_typed_optional_consumer_field(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "object", "properties": {}},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {"limit": {"type": "integer"}},
|
||||
},
|
||||
)
|
||||
assert result.state == "incompatible"
|
||||
assert "limit" in result.detail
|
||||
|
||||
def test_unproven_pattern_is_unknown(self):
|
||||
result = compare_json_schemas(
|
||||
{"type": "string"}, {"type": "string", "pattern": "^[a-z]+$"}
|
||||
)
|
||||
assert result.state == "unknown"
|
||||
|
||||
|
||||
class TestToolStepCompatibility:
|
||||
@staticmethod
|
||||
def child(input_schema):
|
||||
return Tool(name="child", input_schema=input_schema)
|
||||
|
||||
def analyze(self, parent, child):
|
||||
report = PreflightReport()
|
||||
with patch("cmdforge.tool.load_tool", return_value=child):
|
||||
_check_toolstep_compatibility(parent, report)
|
||||
return report
|
||||
|
||||
def test_compatible_stdin_contract(self):
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
steps=[ToolStep(tool="child", output_var="result")],
|
||||
)
|
||||
child = self.child({
|
||||
"type": "object",
|
||||
"properties": {"input": {"type": "string"}},
|
||||
"required": ["input"],
|
||||
})
|
||||
report = self.analyze(parent, child)
|
||||
assert report.compatibility[0]["state"] == "compatible"
|
||||
|
||||
def test_upstream_serialization_mismatch_is_reported(self):
|
||||
producer = PromptStep(
|
||||
prompt="produce", provider="mock", output_var="value",
|
||||
output_schema={"type": "string"},
|
||||
)
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
steps=[
|
||||
producer,
|
||||
ToolStep(
|
||||
tool="child", output_var="result", input_template="{value}"
|
||||
),
|
||||
],
|
||||
)
|
||||
child = self.child({
|
||||
"type": "object",
|
||||
"properties": {"input": {"type": "integer"}},
|
||||
"required": ["input"],
|
||||
})
|
||||
report = self.analyze(parent, child)
|
||||
finding = report.compatibility[0]
|
||||
assert finding["state"] == "incompatible"
|
||||
assert finding["upstream_schema"] == {"type": "string"}
|
||||
assert any("incompatible" in warning for warning in report.warnings)
|
||||
|
||||
def test_missing_required_argument_is_incompatible(self):
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
steps=[ToolStep(tool="child", output_var="result")],
|
||||
)
|
||||
child = self.child({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"input": {"type": "string"},
|
||||
"limit": {"type": "integer"},
|
||||
},
|
||||
"required": ["input", "limit"],
|
||||
})
|
||||
report = self.analyze(parent, child)
|
||||
assert report.compatibility[0]["state"] == "incompatible"
|
||||
assert "limit" in report.compatibility[0]["detail"]
|
||||
|
||||
def test_explicit_argument_reflects_string_transport(self):
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
steps=[ToolStep(
|
||||
tool="child", output_var="result", args={"limit": "{input}"}
|
||||
)],
|
||||
)
|
||||
child = self.child({
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"input": {"type": "string"},
|
||||
"limit": {"type": "integer"},
|
||||
},
|
||||
"required": ["input", "limit"],
|
||||
})
|
||||
assert self.analyze(parent, child).compatibility[0]["state"] == "incompatible"
|
||||
|
||||
def test_mcp_steps_are_ignored(self):
|
||||
parent = Tool(
|
||||
name="parent",
|
||||
steps=[McpStep(server="remote", tool="same-name", output_var="result")],
|
||||
)
|
||||
report = PreflightReport()
|
||||
with patch("cmdforge.tool.load_tool") as loader:
|
||||
_check_toolstep_compatibility(parent, report)
|
||||
loader.assert_not_called()
|
||||
assert report.compatibility == []
|
||||
|
||||
def test_server_mode_does_not_resolve_local_tools(self):
|
||||
parent = Tool(
|
||||
name="published",
|
||||
steps=[ToolStep(tool="child", output_var="result")],
|
||||
)
|
||||
with patch("cmdforge.tool.load_tool") as loader:
|
||||
report = analyze_tool(parent, check_local_dependencies=False)
|
||||
loader.assert_not_called()
|
||||
assert report.compatibility == []
|
||||
@@ -0,0 +1,11 @@
|
||||
import stat
|
||||
|
||||
from cmdforge.signing import initialize_signing_key, load_signing_key
|
||||
|
||||
|
||||
def test_signing_key_is_persistent_and_private(tmp_path):
|
||||
path = tmp_path / "release-key.json"
|
||||
created = initialize_signing_key(path)
|
||||
assert load_signing_key(path) == created
|
||||
assert stat.S_IMODE(path.stat().st_mode) == 0o600
|
||||
assert initialize_signing_key(path) == created
|
||||
@@ -0,0 +1,438 @@
|
||||
"""Tests for provider-attached skills system."""
|
||||
|
||||
import textwrap
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from cmdforge.skills import (
|
||||
Skill,
|
||||
SkillSet,
|
||||
_parse_frontmatter,
|
||||
_validate_provider_name,
|
||||
_validate_skill_name,
|
||||
load_provider_skills,
|
||||
inject_skills,
|
||||
)
|
||||
from cmdforge.providers import ProviderResult
|
||||
from cmdforge.runner import execute_prompt_step
|
||||
from cmdforge.tool import PromptStep
|
||||
|
||||
|
||||
def write_skill(root, provider, name, content="Skill instructions.", description="A useful skill"):
|
||||
directory = root / provider / "skills" / name
|
||||
directory.mkdir(parents=True)
|
||||
(directory / "SKILL.md").write_text(
|
||||
f"---\nname: {name}\ndescription: {description}\n---\n{content}",
|
||||
encoding="utf-8",
|
||||
)
|
||||
return directory
|
||||
|
||||
|
||||
class TestValidation:
|
||||
def test_valid_provider_name(self):
|
||||
_validate_provider_name("claude")
|
||||
_validate_provider_name("openrouter")
|
||||
_validate_provider_name("my-provider_1")
|
||||
|
||||
def test_invalid_provider_name_slash(self):
|
||||
with pytest.raises(ValueError):
|
||||
_validate_provider_name("claude/evil")
|
||||
|
||||
def test_invalid_provider_name_dots(self):
|
||||
with pytest.raises(ValueError):
|
||||
_validate_provider_name("../../etc")
|
||||
|
||||
def test_invalid_provider_name_empty(self):
|
||||
with pytest.raises(ValueError):
|
||||
_validate_provider_name("")
|
||||
|
||||
def test_valid_skill_name(self):
|
||||
_validate_skill_name("python-utils", "claude")
|
||||
|
||||
@pytest.mark.parametrize("name", ["Python", "python_utils", "-python", "python-", "python--utils"])
|
||||
def test_invalid_standard_skill_names(self, name):
|
||||
with pytest.raises(ValueError, match="Invalid skill name"):
|
||||
_validate_skill_name(name, "claude")
|
||||
|
||||
def test_skill_name_max_length(self):
|
||||
_validate_skill_name("a" * 64, "claude")
|
||||
with pytest.raises(ValueError, match="1-64"):
|
||||
_validate_skill_name("a" * 65, "claude")
|
||||
|
||||
def test_invalid_skill_name_slash(self):
|
||||
with pytest.raises(ValueError):
|
||||
_validate_skill_name("a/b", "claude")
|
||||
|
||||
def test_invalid_skill_name_dotdot(self):
|
||||
with pytest.raises(ValueError):
|
||||
_validate_skill_name("..", "claude")
|
||||
|
||||
|
||||
class TestFrontmatter:
|
||||
def test_basic(self):
|
||||
text = textwrap.dedent("""\
|
||||
---
|
||||
name: python-utils
|
||||
description: Python helpers
|
||||
---
|
||||
Use list comprehensions.
|
||||
""")
|
||||
meta, content = _parse_frontmatter(text)
|
||||
assert meta["name"] == "python-utils"
|
||||
assert meta["description"] == "Python helpers"
|
||||
assert content == "Use list comprehensions."
|
||||
|
||||
def test_no_frontmatter(self):
|
||||
with pytest.raises(ValueError, match="must begin"):
|
||||
_parse_frontmatter("Just raw content")
|
||||
|
||||
def test_unterminated_frontmatter(self):
|
||||
with pytest.raises(ValueError, match="not terminated"):
|
||||
_parse_frontmatter("---\nname: test\ndescription: Test")
|
||||
|
||||
def test_invalid_yaml_frontmatter(self):
|
||||
text = "---\n: invalid yaml\n---\ncontent"
|
||||
with pytest.raises(ValueError, match="Invalid YAML"):
|
||||
_parse_frontmatter(text)
|
||||
|
||||
def test_non_dict_frontmatter(self):
|
||||
text = "---\n- list item\n---\ncontent"
|
||||
with pytest.raises(ValueError, match="mapping"):
|
||||
_parse_frontmatter(text)
|
||||
|
||||
|
||||
class TestLoadProviderSkills:
|
||||
@pytest.fixture
|
||||
def temp_providers_dir(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
|
||||
return tmp_path
|
||||
|
||||
def test_no_directory(self, temp_providers_dir):
|
||||
skills = load_provider_skills("nonexistent")
|
||||
assert isinstance(skills, SkillSet)
|
||||
assert skills.skills == []
|
||||
|
||||
def test_empty_skills_dir(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "claude" / "skills"
|
||||
d.mkdir(parents=True)
|
||||
skills = load_provider_skills("claude")
|
||||
assert skills.skills == []
|
||||
|
||||
def test_loads_skill(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "claude" / "skills" / "python"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(textwrap.dedent("""\
|
||||
---
|
||||
name: python
|
||||
description: Python coding patterns
|
||||
---
|
||||
Use type hints always.
|
||||
"""))
|
||||
|
||||
skillset = load_provider_skills("claude")
|
||||
assert len(skillset.skills) == 1
|
||||
skill = skillset.skills[0]
|
||||
assert skill.name == "python"
|
||||
assert skill.description == "Python coding patterns"
|
||||
assert "type hints" in skill.content
|
||||
|
||||
def test_skips_non_md_files(self, temp_providers_dir):
|
||||
d = write_skill(temp_providers_dir, "claude", "python")
|
||||
(d / "readme.txt").write_text("ignore me")
|
||||
|
||||
skillset = load_provider_skills("claude")
|
||||
assert len(skillset.skills) == 1
|
||||
|
||||
def test_sorted_by_name(self, temp_providers_dir):
|
||||
for name in ("z-skill", "a-skill", "m-skill"):
|
||||
write_skill(temp_providers_dir, "claude", name)
|
||||
|
||||
skillset = load_provider_skills("claude")
|
||||
names = [s.name for s in skillset.skills]
|
||||
assert names == ["a-skill", "m-skill", "z-skill"]
|
||||
|
||||
def test_frontmatter_name_must_match_directory(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "claude" / "skills" / "python"
|
||||
d.mkdir(parents=True)
|
||||
(d / "SKILL.md").write_text(
|
||||
"---\nname: wrong-name\ndescription: Wrong\n---\ncontent"
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="does not match"):
|
||||
load_provider_skills("claude")
|
||||
|
||||
def test_invalid_provider_name_traversal(self, temp_providers_dir):
|
||||
with pytest.raises(ValueError):
|
||||
load_provider_skills("../secrets")
|
||||
|
||||
def test_skips_non_directories(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "claude" / "skills"
|
||||
d.mkdir(parents=True)
|
||||
(d / "file.txt").write_text("not a skill dir")
|
||||
skillset = load_provider_skills("claude")
|
||||
assert skillset.skills == []
|
||||
|
||||
def test_no_skill_md(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "claude" / "skills" / "empty-dir"
|
||||
d.mkdir(parents=True)
|
||||
skillset = load_provider_skills("claude")
|
||||
assert skillset.skills == []
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("frontmatter", "message"),
|
||||
[
|
||||
("description: Useful", "required 'name'"),
|
||||
("name: python", "required 'description'"),
|
||||
("name: python\ndescription: ''", "non-empty"),
|
||||
],
|
||||
)
|
||||
def test_required_metadata(self, temp_providers_dir, frontmatter, message):
|
||||
directory = temp_providers_dir / "claude" / "skills" / "python"
|
||||
directory.mkdir(parents=True)
|
||||
(directory / "SKILL.md").write_text(
|
||||
f"---\n{frontmatter}\n---\ncontent"
|
||||
)
|
||||
with pytest.raises(ValueError, match=message):
|
||||
load_provider_skills("claude")
|
||||
|
||||
def test_description_max_length(self, temp_providers_dir):
|
||||
write_skill(
|
||||
temp_providers_dir,
|
||||
"claude",
|
||||
"python",
|
||||
description="x" * 1025,
|
||||
)
|
||||
with pytest.raises(ValueError, match="1024"):
|
||||
load_provider_skills("claude")
|
||||
|
||||
def test_rejects_symlinked_provider_directory(self, temp_providers_dir, tmp_path):
|
||||
outside = tmp_path / "outside-provider"
|
||||
outside.mkdir()
|
||||
(temp_providers_dir / "claude").symlink_to(outside, target_is_directory=True)
|
||||
with pytest.raises(ValueError, match="symlinked provider"):
|
||||
load_provider_skills("claude")
|
||||
|
||||
def test_rejects_symlinked_skill_directory(self, temp_providers_dir, tmp_path):
|
||||
skills_dir = temp_providers_dir / "claude" / "skills"
|
||||
skills_dir.mkdir(parents=True)
|
||||
outside = tmp_path / "outside-skill"
|
||||
outside.mkdir()
|
||||
(skills_dir / "python").symlink_to(outside, target_is_directory=True)
|
||||
with pytest.raises(ValueError, match="symlinked skill directory"):
|
||||
load_provider_skills("claude")
|
||||
|
||||
def test_rejects_symlinked_skill_file(self, temp_providers_dir, tmp_path):
|
||||
directory = temp_providers_dir / "claude" / "skills" / "python"
|
||||
directory.mkdir(parents=True)
|
||||
outside = tmp_path / "outside-skill.md"
|
||||
outside.write_text("---\nname: python\ndescription: Test\n---\ncontent")
|
||||
(directory / "SKILL.md").symlink_to(outside)
|
||||
with pytest.raises(ValueError, match="symlinked SKILL.md"):
|
||||
load_provider_skills("claude")
|
||||
|
||||
|
||||
class TestSkillSetSelection:
|
||||
def test_get_all_when_none(self):
|
||||
skills = SkillSet("test", [
|
||||
Skill("a", "a-dir", "desc A", "content A"),
|
||||
Skill("b", "b-dir", "desc B", "content B"),
|
||||
])
|
||||
result = skills.get(None)
|
||||
assert len(result) == 2
|
||||
|
||||
def test_get_specific(self):
|
||||
skills = SkillSet("test", [
|
||||
Skill("a", "a-dir", "desc A", "content A"),
|
||||
Skill("b", "b-dir", "desc B", "content B"),
|
||||
])
|
||||
result = skills.get(["a"])
|
||||
assert len(result) == 1
|
||||
assert result[0].name == "a"
|
||||
|
||||
def test_get_empty_list(self):
|
||||
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
|
||||
assert skills.get([]) == []
|
||||
|
||||
def test_warn_on_missing(self, capsys):
|
||||
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
|
||||
result = skills.get(["missing"], on_missing="warn")
|
||||
assert result == []
|
||||
captured = capsys.readouterr()
|
||||
assert "not found" in captured.err
|
||||
|
||||
def test_error_on_missing(self):
|
||||
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
|
||||
with pytest.raises(KeyError, match="not found"):
|
||||
skills.get(["missing"], on_missing="error")
|
||||
|
||||
def test_ignore_on_missing(self):
|
||||
skills = SkillSet("test", [Skill("a", "a-dir", "desc", "c")])
|
||||
result = skills.get(["missing"], on_missing="ignore")
|
||||
assert result == []
|
||||
|
||||
def test_invalid_missing_policy(self):
|
||||
skills = SkillSet("test")
|
||||
with pytest.raises(ValueError, match="missing-skill policy"):
|
||||
skills.get([], on_missing="silent")
|
||||
|
||||
|
||||
class TestInjectSkills:
|
||||
@pytest.fixture
|
||||
def temp_providers_dir(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
|
||||
return tmp_path
|
||||
|
||||
def test_no_skills_no_injection(self, temp_providers_dir):
|
||||
d = temp_providers_dir / "empty" / "skills"
|
||||
d.mkdir(parents=True)
|
||||
result = inject_skills("Hello", "empty")
|
||||
assert result == "Hello"
|
||||
|
||||
def test_injects_skills_before_prompt(self, temp_providers_dir):
|
||||
write_skill(temp_providers_dir, "claude", "python", "Use type hints.")
|
||||
|
||||
result = inject_skills("Do X", "claude")
|
||||
assert result.startswith("## Skill: python")
|
||||
assert "Do X" in result
|
||||
assert result.index("## Skill") < result.index("Do X")
|
||||
|
||||
def test_specific_skill_names(self, temp_providers_dir):
|
||||
for name in ("python", "git", "testing"):
|
||||
write_skill(temp_providers_dir, "claude", name, f"{name} content")
|
||||
|
||||
result = inject_skills("Prompt", "claude", ["python"])
|
||||
assert "python content" in result
|
||||
assert "git content" not in result
|
||||
assert "testing content" not in result
|
||||
|
||||
def test_invalid_provider_name(self, temp_providers_dir):
|
||||
with pytest.raises(ValueError, match="Invalid provider"):
|
||||
inject_skills("Prompt", "../../etc")
|
||||
|
||||
|
||||
class TestPromptStepSkills:
|
||||
def test_to_dict_excludes_none(self):
|
||||
step = PromptStep(
|
||||
prompt="Test",
|
||||
provider="claude",
|
||||
output_var="out",
|
||||
)
|
||||
d = step.to_dict()
|
||||
assert "skills" not in d
|
||||
|
||||
def test_to_dict_includes_skills(self):
|
||||
step = PromptStep(
|
||||
prompt="Test",
|
||||
provider="claude",
|
||||
output_var="out",
|
||||
skills=["python", "git"],
|
||||
)
|
||||
d = step.to_dict()
|
||||
assert d["skills"] == ["python", "git"]
|
||||
|
||||
def test_roundtrip_with_skills(self):
|
||||
step = PromptStep(
|
||||
prompt="Test",
|
||||
provider="claude",
|
||||
output_var="out",
|
||||
skills=["python"],
|
||||
)
|
||||
restored = PromptStep.from_dict(step.to_dict())
|
||||
assert restored.skills == ["python"]
|
||||
|
||||
def test_roundtrip_without_skills(self):
|
||||
step = PromptStep(
|
||||
prompt="Test",
|
||||
provider="claude",
|
||||
output_var="out",
|
||||
)
|
||||
restored = PromptStep.from_dict(step.to_dict())
|
||||
assert restored.skills is None
|
||||
|
||||
def test_roundtrip_with_explicit_empty_skills(self):
|
||||
step = PromptStep("Test", "claude", "out", skills=[])
|
||||
serialized = step.to_dict()
|
||||
assert serialized["skills"] == []
|
||||
assert PromptStep.from_dict(serialized).skills == []
|
||||
|
||||
@pytest.mark.parametrize("skills", ["python", [1], ["*", "python"], ["python", "python"]])
|
||||
def test_rejects_invalid_skill_lists(self, skills):
|
||||
with pytest.raises(ValueError):
|
||||
PromptStep("Test", "claude", "out", skills=skills)
|
||||
|
||||
def test_rejects_invalid_skill_identifier(self):
|
||||
with pytest.raises(ValueError, match="Invalid skill name"):
|
||||
PromptStep("Test", "claude", "out", skills=["Python_Utils"])
|
||||
|
||||
|
||||
class TestRunnerSkillIntegration:
|
||||
@pytest.fixture
|
||||
def configured_skill(self, tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("cmdforge.skills.PROVIDERS_DIR", tmp_path)
|
||||
write_skill(tmp_path, "override", "python", "SKILL INSTRUCTIONS")
|
||||
return tmp_path
|
||||
|
||||
def test_plain_text_uses_effective_provider_and_context_order(
|
||||
self, configured_skill, monkeypatch
|
||||
):
|
||||
captured = {}
|
||||
|
||||
def fake_call(provider, prompt, max_tokens=None, **kwargs):
|
||||
captured.update(provider=provider, prompt=prompt)
|
||||
return ProviderResult(text="done", success=True)
|
||||
|
||||
monkeypatch.setattr("cmdforge.runner.call_provider", fake_call)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.runner.load_profile",
|
||||
lambda name: SimpleNamespace(system_prompt="PROFILE INSTRUCTIONS"),
|
||||
)
|
||||
step = PromptStep(
|
||||
prompt="USER PROMPT",
|
||||
provider="original",
|
||||
output_var="out",
|
||||
profile="reviewer",
|
||||
plain_text=True,
|
||||
skills=["python"],
|
||||
)
|
||||
|
||||
output, success = execute_prompt_step(
|
||||
step, {}, provider_override="override"
|
||||
)
|
||||
|
||||
assert success is True
|
||||
assert output == "done"
|
||||
assert captured["provider"] == "override"
|
||||
prompt = captured["prompt"]
|
||||
assert prompt.index("PROFILE INSTRUCTIONS") < prompt.index("## Skill: python")
|
||||
assert prompt.index("## Skill: python") < prompt.index("USER PROMPT")
|
||||
|
||||
def test_structured_path_preserves_context_order(
|
||||
self, configured_skill, monkeypatch
|
||||
):
|
||||
captured = {}
|
||||
|
||||
def fake_call(provider, prompt, max_tokens=None, **kwargs):
|
||||
captured.update(provider=provider, prompt=prompt)
|
||||
return ProviderResult(text='{"output": "done"}', success=True)
|
||||
|
||||
monkeypatch.setattr("cmdforge.runner.call_provider", fake_call)
|
||||
monkeypatch.setattr(
|
||||
"cmdforge.runner.load_profile",
|
||||
lambda name: SimpleNamespace(system_prompt="PROFILE INSTRUCTIONS"),
|
||||
)
|
||||
step = PromptStep(
|
||||
prompt="USER PROMPT",
|
||||
provider="override",
|
||||
output_var="out",
|
||||
profile="reviewer",
|
||||
skills=["python"],
|
||||
)
|
||||
|
||||
output, success = execute_prompt_step(step, {})
|
||||
|
||||
assert success is True
|
||||
assert output == {"output": "done"}
|
||||
prompt = captured["prompt"]
|
||||
assert prompt.index("PROFILE INSTRUCTIONS") < prompt.index("## Skill: python")
|
||||
assert prompt.index("## Skill: python") < prompt.index("USER PROMPT")
|
||||
+195
-1
@@ -8,7 +8,7 @@ import pytest
|
||||
import yaml
|
||||
|
||||
from cmdforge.tool import (
|
||||
Tool, ToolArgument, PromptStep, CodeStep,
|
||||
Tool, ToolArgument, PromptStep, CodeStep, ToolStep,
|
||||
validate_tool_name, load_tool, save_tool, delete_tool,
|
||||
list_tools, tool_exists, create_wrapper_script,
|
||||
DEFAULT_CATEGORIES
|
||||
@@ -80,6 +80,50 @@ class TestToolArgument:
|
||||
assert restored.default == original.default
|
||||
assert restored.description == original.description
|
||||
|
||||
def test_typed_metadata_roundtrip(self):
|
||||
original = ToolArgument(
|
||||
flag="--limit",
|
||||
variable="limit",
|
||||
default=10,
|
||||
description="Maximum results",
|
||||
type="integer",
|
||||
enum=[10, 25, 50],
|
||||
required=True,
|
||||
)
|
||||
assert ToolArgument.from_dict(original.to_dict()) == original
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"values",
|
||||
[
|
||||
{"type": "object"},
|
||||
{"enum": []},
|
||||
{"enum": "fast"},
|
||||
{"required": "yes"},
|
||||
],
|
||||
)
|
||||
def test_rejects_invalid_typed_metadata(self, values):
|
||||
with pytest.raises(ValueError):
|
||||
ToolArgument(flag="--value", variable="value", **values)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("legacy_type", "canonical_type"),
|
||||
[
|
||||
("str", "string"),
|
||||
("int", "integer"),
|
||||
("float", "number"),
|
||||
("bool", "boolean"),
|
||||
],
|
||||
)
|
||||
def test_legacy_scalar_type_aliases_are_normalized(
|
||||
self, legacy_type, canonical_type
|
||||
):
|
||||
argument = ToolArgument(
|
||||
flag="--value", variable="value", type=legacy_type
|
||||
)
|
||||
|
||||
assert argument.type == canonical_type
|
||||
assert argument.to_dict().get("type", "string") == canonical_type
|
||||
|
||||
|
||||
class TestPromptStep:
|
||||
"""Tests for PromptStep dataclass."""
|
||||
@@ -554,3 +598,153 @@ class TestDefaultCategories:
|
||||
assert "Developer" in DEFAULT_CATEGORIES
|
||||
assert "Data" in DEFAULT_CATEGORIES
|
||||
assert "Other" in DEFAULT_CATEGORIES
|
||||
|
||||
|
||||
class TestToolContracts:
|
||||
def test_config_must_be_mapping(self):
|
||||
with pytest.raises(ValueError, match="YAML mapping"):
|
||||
Tool.from_dict(["not", "a", "mapping"])
|
||||
|
||||
def test_legacy_source_string_is_normalized(self):
|
||||
tool = Tool.from_dict({"name": "legacy-source", "source": "old/tool"})
|
||||
assert tool.source is not None
|
||||
assert tool.source.type == "imported"
|
||||
assert tool.source.original_tool == "old/tool"
|
||||
|
||||
def test_contracts_roundtrip(self):
|
||||
tool = Tool(
|
||||
name="contract-tool",
|
||||
input_schema={"type": "string"},
|
||||
output_schema={"type": "array", "items": {"type": "integer"}},
|
||||
)
|
||||
restored = Tool.from_dict(tool.to_dict())
|
||||
assert restored.input_schema == {"type": "string"}
|
||||
assert restored.output_schema["type"] == "array"
|
||||
|
||||
def test_empty_contracts_are_preserved(self):
|
||||
tool = Tool(name="open-contract", input_schema={}, output_schema={})
|
||||
serialized = tool.to_dict()
|
||||
assert serialized["input_schema"] == {}
|
||||
assert serialized["output_schema"] == {}
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("field", "schema"),
|
||||
[
|
||||
("input_schema", "string"),
|
||||
("output_schema", {"type": "not-a-json-schema-type"}),
|
||||
("output_schema", {"required": "name"}),
|
||||
],
|
||||
)
|
||||
def test_invalid_contract_is_rejected(self, field, schema):
|
||||
with pytest.raises(ValueError, match=field):
|
||||
Tool(name="bad-contract", **{field: schema})
|
||||
|
||||
|
||||
class TestAgentContext:
|
||||
def test_toolstep_profile_roundtrip(self):
|
||||
step = ToolStep(tool="my-tool", output_var="out", profile="architect")
|
||||
d = step.to_dict()
|
||||
assert d["profile"] == "architect"
|
||||
restored = ToolStep.from_dict(d)
|
||||
assert restored.profile == "architect"
|
||||
|
||||
def test_toolstep_skills_roundtrip(self):
|
||||
step = ToolStep(tool="my-tool", output_var="out", skills=["python"])
|
||||
d = step.to_dict()
|
||||
assert d["skills"] == ["python"]
|
||||
restored = ToolStep.from_dict(d)
|
||||
assert restored.skills == ["python"]
|
||||
|
||||
def test_toolstep_empty_skills_serialized(self):
|
||||
step = ToolStep(tool="my-tool", output_var="out", skills=[])
|
||||
d = step.to_dict()
|
||||
assert d["skills"] == []
|
||||
restored = ToolStep.from_dict(d)
|
||||
assert restored.skills == []
|
||||
|
||||
def test_toolstep_no_skills_not_serialized(self):
|
||||
step = ToolStep(tool="my-tool", output_var="out")
|
||||
d = step.to_dict()
|
||||
assert "skills" not in d
|
||||
assert "profile" not in d
|
||||
|
||||
def test_toolstep_tools_roundtrip(self):
|
||||
step = ToolStep(
|
||||
tool="my-tool", output_var="out", tools=["safe-*", "official/lint"]
|
||||
)
|
||||
assert ToolStep.from_dict(step.to_dict()).tools == [
|
||||
"safe-*", "official/lint"
|
||||
]
|
||||
|
||||
def test_toolstep_empty_tools_roundtrip(self):
|
||||
step = ToolStep(tool="my-tool", output_var="out", tools=[])
|
||||
serialized = step.to_dict()
|
||||
assert serialized["tools"] == []
|
||||
assert ToolStep.from_dict(serialized).tools == []
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"data",
|
||||
[
|
||||
{"skills": "python"},
|
||||
{"skills": ["*", "python"]},
|
||||
{"skills": ["Python"]},
|
||||
{"tools": "safe-tool"},
|
||||
{"tools": [""]},
|
||||
{"profile": ""},
|
||||
],
|
||||
)
|
||||
def test_toolstep_rejects_invalid_agent_context(self, data):
|
||||
with pytest.raises(ValueError):
|
||||
ToolStep(tool="my-tool", output_var="out", **data)
|
||||
|
||||
def test_promptstep_skills_roundtrip(self):
|
||||
step = PromptStep(
|
||||
prompt="Test", provider="claude", output_var="out",
|
||||
skills=["python"],
|
||||
)
|
||||
restored = PromptStep.from_dict(step.to_dict())
|
||||
assert restored.skills == ["python"]
|
||||
|
||||
def test_promptstep_deny_fallback_roundtrip(self):
|
||||
step = PromptStep(
|
||||
prompt="Private", provider="local", output_var="out",
|
||||
fallback_policy="deny",
|
||||
)
|
||||
|
||||
restored = PromptStep.from_dict(step.to_dict())
|
||||
|
||||
assert restored.fallback_policy == "deny"
|
||||
|
||||
def test_promptstep_rejects_invalid_fallback_policy(self):
|
||||
with pytest.raises(ValueError, match="fallback_policy"):
|
||||
PromptStep(
|
||||
prompt="Test", provider="local", output_var="out",
|
||||
fallback_policy="sometimes",
|
||||
)
|
||||
|
||||
|
||||
class TestDeprecationFields:
|
||||
def test_round_trip_and_warning(self):
|
||||
from cmdforge.tool import deprecation_warning
|
||||
|
||||
tool = Tool(
|
||||
name="old", deprecated=True, deprecated_message="Moved.",
|
||||
replacement="official/new",
|
||||
)
|
||||
restored = Tool.from_dict(tool.to_dict())
|
||||
assert restored.deprecated is True
|
||||
assert restored.replacement == "official/new"
|
||||
assert "official/new" in deprecation_warning(restored)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"values",
|
||||
[
|
||||
{"deprecated": "false"},
|
||||
{"deprecated_message": 123},
|
||||
{"replacement": "../../escape"},
|
||||
{"replacement": "owner/name/extra"},
|
||||
],
|
||||
)
|
||||
def test_invalid_deprecation_metadata_is_rejected(self, values):
|
||||
with pytest.raises(ValueError):
|
||||
Tool(name="old", **values)
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
import json
|
||||
import stat
|
||||
|
||||
from cmdforge.usage import (
|
||||
build_composite_tool,
|
||||
clear_usage,
|
||||
get_suggestions,
|
||||
is_enabled,
|
||||
record_invocation,
|
||||
set_enabled,
|
||||
)
|
||||
|
||||
|
||||
class Stream:
|
||||
def __init__(self, inode=None):
|
||||
self.inode = inode
|
||||
|
||||
|
||||
def configure_paths(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("cmdforge.usage.CONFIG_DIR", tmp_path)
|
||||
monkeypatch.setattr("cmdforge.usage.USAGE_FILE", tmp_path / "usage.json")
|
||||
monkeypatch.setattr("cmdforge.usage.USAGE_LOCK_FILE", tmp_path / ".usage.lock")
|
||||
|
||||
|
||||
def test_tracking_is_disabled_by_default(tmp_path, monkeypatch):
|
||||
configure_paths(tmp_path, monkeypatch)
|
||||
assert not is_enabled()
|
||||
monkeypatch.setattr("cmdforge.usage._pipe_inode", lambda stream: stream.inode)
|
||||
record_invocation("one", Stream(), Stream())
|
||||
assert not (tmp_path / "usage.json").exists()
|
||||
|
||||
|
||||
def test_detects_only_pipe_link_and_stores_no_content(tmp_path, monkeypatch):
|
||||
configure_paths(tmp_path, monkeypatch)
|
||||
set_enabled(True)
|
||||
monkeypatch.setattr("cmdforge.usage._pipe_inode", lambda stream: stream.inode)
|
||||
for inode in range(44, 47):
|
||||
record_invocation("tool-a", Stream(), Stream(inode))
|
||||
record_invocation("tool-b", Stream(inode), Stream())
|
||||
|
||||
assert get_suggestions() == [{
|
||||
"tools": ["tool-a", "tool-b"],
|
||||
"count": 3,
|
||||
"last_seen": get_suggestions()[0]["last_seen"],
|
||||
}]
|
||||
data = json.loads((tmp_path / "usage.json").read_text())
|
||||
serialized = json.dumps(data)
|
||||
for forbidden in ("input_text", "output", "arguments", "cwd", "environment"):
|
||||
assert forbidden not in serialized
|
||||
assert stat.S_IMODE((tmp_path / "usage.json").stat().st_mode) == 0o600
|
||||
|
||||
|
||||
def test_disable_and_clear_preserve_consent_state(tmp_path, monkeypatch):
|
||||
configure_paths(tmp_path, monkeypatch)
|
||||
set_enabled(True)
|
||||
clear_usage()
|
||||
assert is_enabled()
|
||||
set_enabled(False)
|
||||
assert not is_enabled()
|
||||
|
||||
|
||||
def test_build_composite_tool_preserves_pipeline_order():
|
||||
tool = build_composite_tool("combined", ["tool-a", "tool-b"])
|
||||
assert tool.dependencies == ["tool-a", "tool-b"]
|
||||
assert tool.steps[0].input_template == "{input}"
|
||||
assert tool.steps[1].input_template == "{pipeline_1}"
|
||||
assert tool.output == "{pipeline_2}"
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Regression tests for user-facing documentation embedded in the web app."""
|
||||
|
||||
import re
|
||||
|
||||
from cmdforge.web.docs_content import DOCS, get_doc, get_toc
|
||||
|
||||
|
||||
def test_getting_started_documents_agent_first_workflow():
|
||||
content = get_doc("getting-started")["content"]
|
||||
|
||||
expected_commands = (
|
||||
"cmdforge list --json --filter",
|
||||
"cmdforge registry search",
|
||||
"cmdforge run-once",
|
||||
"cmdforge create classify-docs --project",
|
||||
"forge-tool --name classify-docs --project",
|
||||
"cmdforge mcp configure codex --dry-run",
|
||||
"cmdforge mcp configure claude-code --dry-run",
|
||||
)
|
||||
for command in expected_commands:
|
||||
assert command in content
|
||||
|
||||
|
||||
def test_first_tool_explains_project_ownership():
|
||||
content = get_doc("first-tool")["content"]
|
||||
|
||||
assert "cmdforge create explain --project" in content
|
||||
assert "./.cmdforge/" in content
|
||||
|
||||
|
||||
def test_every_document_appears_once_in_the_book_navigation():
|
||||
slugs = []
|
||||
for chapter in get_toc():
|
||||
slugs.append(chapter.slug)
|
||||
slugs.extend(page.slug for page in chapter.children)
|
||||
|
||||
assert len(slugs) == len(set(slugs))
|
||||
assert set(slugs) == set(DOCS)
|
||||
|
||||
|
||||
def test_internal_documentation_links_resolve():
|
||||
links = set()
|
||||
for page in DOCS.values():
|
||||
links.update(re.findall(r'href="/docs/([^"#?]+)', page["content"]))
|
||||
|
||||
assert links <= set(DOCS)
|
||||
|
||||
|
||||
def test_modern_architecture_has_dedicated_chapters():
|
||||
expected = {
|
||||
"mcp-overview",
|
||||
"mcp-client",
|
||||
"mcp-server",
|
||||
"agent-integration",
|
||||
"contracts-quality",
|
||||
"provider-policy",
|
||||
"skills-delegation",
|
||||
"optimization-usage",
|
||||
"forge-tool",
|
||||
"trust-publishing",
|
||||
}
|
||||
assert expected <= set(DOCS)
|
||||
|
||||
|
||||
def test_mcp_documentation_covers_both_directions_and_transports():
|
||||
content = " ".join(
|
||||
get_doc(slug)["content"]
|
||||
for slug in ("mcp-overview", "mcp-client", "mcp-server")
|
||||
)
|
||||
|
||||
for term in (
|
||||
"type: mcp",
|
||||
"transport: stdio",
|
||||
"transport: streamable-http",
|
||||
"result_mode",
|
||||
"cmdforge mcp connect",
|
||||
"cmdforge mcp serve",
|
||||
"expose",
|
||||
"deny",
|
||||
):
|
||||
assert term in content
|
||||
|
||||
|
||||
def test_provider_documentation_covers_strict_execution_contract():
|
||||
content = get_doc("providers")["content"] + get_doc("provider-policy")["content"]
|
||||
|
||||
for term in (
|
||||
"subprocess",
|
||||
"API",
|
||||
"PTY",
|
||||
"--no-fallback",
|
||||
"--require-local",
|
||||
"--require-capability",
|
||||
"--data-classification",
|
||||
"--require-model-identity",
|
||||
"--require-model-digest",
|
||||
"--result-envelope json",
|
||||
):
|
||||
assert term in content
|
||||
|
||||
|
||||
def test_command_atlas_covers_every_top_level_cli_family():
|
||||
content = get_doc("cli-reference")["content"]
|
||||
commands = (
|
||||
"list", "create", "edit", "delete", "test", "run", "run-once", "ui",
|
||||
"refresh", "docs", "inspect", "optimize", "check", "providers", "registry",
|
||||
"collections", "deps", "install", "lock", "verify", "add", "remove", "init",
|
||||
"config", "settings", "system-deps", "mcp", "usage",
|
||||
)
|
||||
|
||||
for command in commands:
|
||||
assert f"cmdforge {command}" in content
|
||||
|
||||
|
||||
def test_installation_matches_runtime_and_current_provider_onboarding():
|
||||
installation = get_doc("installation")["content"]
|
||||
provider_setup = get_doc("provider-setup")["content"]
|
||||
|
||||
assert "Python 3.10" in installation
|
||||
assert "Python 3.8" not in installation
|
||||
assert "cmdforge[mcp,pty]" in installation
|
||||
assert "cmdforge providers discover --add" in installation
|
||||
assert "cmdforge providers discover --add" in provider_setup
|
||||
assert "--type api" in provider_setup
|
||||
assert "pip install claude-cli" not in provider_setup
|
||||
Reference in New Issue
Block a user