CmdForge/tests

CmdForge Test Suite

Overview

The test suite covers the core CmdForge modules with 205 unit tests organized into test files:

Test File	Module Tested	Test Count	Description
test_tool.py	tool.py	35	Tool definitions, YAML loading/saving
test_runner.py	runner.py	31	Variable substitution, step execution
test_providers.py	providers.py	35	Provider management, AI calls
test_cli.py	cli/	20	CLI commands and arguments
test_collection.py	collection.py	27	Collection CRUD, tool resolution
test_collection_api.py	Registry API	20	Collection API endpoints, client methods
test_registry_integration.py	Registry	37	Registry client, resolution (12 require server)

Running Tests

# Run all unit tests (excluding integration tests)
pytest tests/ -m "not integration"

# Run all tests with verbose output
pytest tests/ -v

# Run a specific test file
pytest tests/test_tool.py -v

# Run a specific test class
pytest tests/test_runner.py::TestSubstituteVariables -v

# Run with coverage
pytest tests/ --cov=cmdforge --cov-report=html

Test Categories

Unit Tests (run without external dependencies)

test_tool.py - Tool data structures and persistence

• TestToolArgument - Custom argument definition and serialization
• TestPromptStep - AI prompt step configuration
• TestCodeStep - Python code step configuration
• TestTool - Full tool configuration and roundtrip serialization
• TestValidateToolName - Tool name validation rules
• TestToolPersistence - Save/load/delete operations
• TestLegacyFormat - Backward compatibility with old format

test_runner.py - Tool execution engine

• TestSubstituteVariables - Variable placeholder substitution
  • Simple substitution: {name} → value
  • Escaped braces: {{literal}} → {literal}
  • Multiple variables, multiline templates
• TestExecutePromptStep - AI provider calls with mocking
• TestExecuteCodeStep - Python code execution via exec()
• TestRunTool - Full tool execution with steps
• TestCreateArgumentParser - CLI argument parsing

test_providers.py - AI provider abstraction

• TestProvider - Provider dataclass and serialization
• TestProviderResult - Success/error result handling
• TestMockProvider - Built-in mock for testing
• TestProviderPersistence - YAML save/load operations
• TestCallProvider - Subprocess execution with mocking
• TestProviderCommandParsing - Shell command parsing with shlex

test_cli.py - Command-line interface

• TestCLIBasics - Help, version flags
• TestListCommand - List available tools
• TestCreateCommand - Create new tools
• TestDeleteCommand - Delete tools
• TestRunCommand - Execute tools
• TestTestCommand - Test tools with mock provider
• TestProvidersCommand - Manage AI providers
• TestRefreshCommand - Regenerate wrapper scripts
• TestDocsCommand - View/edit documentation

test_collection.py - Collection management

• TestCollection - Collection dataclass and CRUD operations
  • Create, save, load, delete collections
  • Registry URL generation
  • YAML serialization roundtrip
• TestListCollections - List local collections
• TestGetCollection - Get collection by name
• TestClassifyToolReference - Classify local vs registry tool refs
• TestResolveToolReferences - Transform tool refs for publishing
  • Local tool name expansion (name → user/name)
  • Visibility issue detection
  • Registry tool validation
  • Pinned version transformation
• TestToolResolutionResult - Resolution result dataclass

test_collection_api.py - Collection API and client

• TestGetMeEndpoint - GET /api/v1/me endpoint (requires Flask)
• TestToolApprovedEndpoint - GET /api/v1/tools/.../approved (requires Flask)
• TestPostCollectionsEndpoint - POST /api/v1/collections (requires Flask)
• TestRegistryClientMethods - Client methods with mocked requests
  • get_me() - Get authenticated user info
  • has_approved_public_tool() - Check tool approval status
  • publish_collection() - Publish collection to registry

Integration Tests (require running server)

test_registry_integration.py - Registry API tests (marked with @pytest.mark.integration)

• TestRegistryIntegration - List, search, categories, index
• TestAuthIntegration - Registration, login, tokens
• TestPublishIntegration - Tool publishing workflow

Run with a local registry server:

# Start registry server
python -m cmdforge.registry.app

# Run integration tests
pytest tests/test_registry_integration.py -v -m integration

Test Fixtures

Common fixtures used across tests:

@pytest.fixture
def temp_tools_dir(tmp_path):
    """Redirect TOOLS_DIR and BIN_DIR to temp directory."""
    with patch('cmdforge.tool.TOOLS_DIR', tmp_path / ".cmdforge"):
        with patch('cmdforge.tool.BIN_DIR', tmp_path / ".local" / "bin"):
            yield tmp_path

@pytest.fixture
def temp_providers_file(tmp_path):
    """Redirect providers.yaml to temp directory."""
    providers_file = tmp_path / ".cmdforge" / "providers.yaml"
    with patch('cmdforge.providers.PROVIDERS_FILE', providers_file):
        yield providers_file

Mocking Strategy

• File system: Use tmp_path fixture and patch module-level paths
• Subprocess calls: Mock subprocess.run and shutil.which
• Stdin: Use patch('sys.stdin', StringIO("input"))
• Provider calls: Mock call_provider to return ProviderResult

Known Limitations

1. Tool name validation - CLI doesn't validate names before creation (test skipped)
2. Nested braces - {{{x}}} breaks due to overlapping escape sequences (documented behavior)
3. Integration tests - Require local registry server at localhost:5000

Adding New Tests

1. Create test functions in the appropriate test file
2. Use descriptive names: test_<what>_<scenario>
3. Add docstrings explaining the test purpose
4. Use fixtures for common setup
5. Mock external dependencies (subprocess, network, file system)

Example:

def test_tool_with_multiple_code_steps(self):
    """Multiple code steps should pass variables between them."""
    tool = Tool(
        name="multi-code",
        steps=[
            CodeStep(code="x = 1", output_var="x"),
            CodeStep(code="y = int(x) + 1", output_var="y")
        ],
        output="{y}"
    )
    output, code = run_tool(tool, "", {})
    assert code == 0
    assert output == "2"