smarttools/tests/README.md

SmartTools Test Suite

Overview

The test suite covers the core SmartTools modules with 158 unit tests organized into four main test files:

Test File	Module Tested	Test Count	Description
test_tool.py	tool.py	35	Tool definitions, YAML loading/saving
test_runner.py	runner.py	31	Variable substitution, step execution
test_providers.py	providers.py	35	Provider management, AI calls
test_cli.py	cli/	20	CLI commands and arguments
test_registry_integration.py	Registry	37	Registry client, resolution (12 require server)

Running Tests

# Run all unit tests (excluding integration tests)
pytest tests/ -m "not integration"

# Run all tests with verbose output
pytest tests/ -v

# Run a specific test file
pytest tests/test_tool.py -v

# Run a specific test class
pytest tests/test_runner.py::TestSubstituteVariables -v

# Run with coverage
pytest tests/ --cov=smarttools --cov-report=html

Test Categories

Unit Tests (run without external dependencies)

test_tool.py - Tool data structures and persistence

• TestToolArgument - Custom argument definition and serialization
• TestPromptStep - AI prompt step configuration
• TestCodeStep - Python code step configuration
• TestTool - Full tool configuration and roundtrip serialization
• TestValidateToolName - Tool name validation rules
• TestToolPersistence - Save/load/delete operations
• TestLegacyFormat - Backward compatibility with old format

test_runner.py - Tool execution engine

• TestSubstituteVariables - Variable placeholder substitution
  • Simple substitution: {name} → value
  • Escaped braces: {{literal}} → {literal}
  • Multiple variables, multiline templates
• TestExecutePromptStep - AI provider calls with mocking
• TestExecuteCodeStep - Python code execution via exec()
• TestRunTool - Full tool execution with steps
• TestCreateArgumentParser - CLI argument parsing

test_providers.py - AI provider abstraction

• TestProvider - Provider dataclass and serialization
• TestProviderResult - Success/error result handling
• TestMockProvider - Built-in mock for testing
• TestProviderPersistence - YAML save/load operations
• TestCallProvider - Subprocess execution with mocking
• TestProviderCommandParsing - Shell command parsing with shlex

test_cli.py - Command-line interface

• TestCLIBasics - Help, version flags
• TestListCommand - List available tools
• TestCreateCommand - Create new tools
• TestDeleteCommand - Delete tools
• TestRunCommand - Execute tools
• TestTestCommand - Test tools with mock provider
• TestProvidersCommand - Manage AI providers
• TestRefreshCommand - Regenerate wrapper scripts
• TestDocsCommand - View/edit documentation

Integration Tests (require running server)

test_registry_integration.py - Registry API tests (marked with @pytest.mark.integration)

• TestRegistryIntegration - List, search, categories, index
• TestAuthIntegration - Registration, login, tokens
• TestPublishIntegration - Tool publishing workflow

Run with a local registry server:

# Start registry server
python -m smarttools.registry.app

# Run integration tests
pytest tests/test_registry_integration.py -v -m integration

Test Fixtures

Common fixtures used across tests:

@pytest.fixture
def temp_tools_dir(tmp_path):
    """Redirect TOOLS_DIR and BIN_DIR to temp directory."""
    with patch('smarttools.tool.TOOLS_DIR', tmp_path / ".smarttools"):
        with patch('smarttools.tool.BIN_DIR', tmp_path / ".local" / "bin"):
            yield tmp_path

@pytest.fixture
def temp_providers_file(tmp_path):
    """Redirect providers.yaml to temp directory."""
    providers_file = tmp_path / ".smarttools" / "providers.yaml"
    with patch('smarttools.providers.PROVIDERS_FILE', providers_file):
        yield providers_file

Mocking Strategy

• File system: Use tmp_path fixture and patch module-level paths
• Subprocess calls: Mock subprocess.run and shutil.which
• Stdin: Use patch('sys.stdin', StringIO("input"))
• Provider calls: Mock call_provider to return ProviderResult

Known Limitations

1. Tool name validation - CLI doesn't validate names before creation (test skipped)
2. Nested braces - {{{x}}} breaks due to overlapping escape sequences (documented behavior)
3. Integration tests - Require local registry server at localhost:5000

Adding New Tests

1. Create test functions in the appropriate test file
2. Use descriptive names: test_<what>_<scenario>
3. Add docstrings explaining the test purpose
4. Use fixtures for common setup
5. Mock external dependencies (subprocess, network, file system)

Example:

def test_tool_with_multiple_code_steps(self):
    """Multiple code steps should pass variables between them."""
    tool = Tool(
        name="multi-code",
        steps=[
            CodeStep(code="x = 1", output_var="x"),
            CodeStep(code="y = int(x) + 1", output_var="y")
        ],
        output="{y}"
    )
    output, code = run_tool(tool, "", {})
    assert code == 0
    assert output == "2"