# SmartTools Test Suite

## Overview

The test suite covers the core SmartTools modules with **158 unit tests** organized into four main test files:

| Test File | Module Tested | Test Count | Description |
|-----------|---------------|------------|-------------|
| `test_tool.py` | `tool.py` | 35 | Tool definitions, YAML loading/saving |
| `test_runner.py` | `runner.py` | 31 | Variable substitution, step execution |
| `test_providers.py` | `providers.py` | 35 | Provider management, AI calls |
| `test_cli.py` | `cli/` | 20 | CLI commands and arguments |
| `test_registry_integration.py` | Registry | 37 | Registry client, resolution (12 require server) |

## Running Tests

```bash
# Run all unit tests (excluding integration tests)
pytest tests/ -m "not integration"

# Run all tests with verbose output
pytest tests/ -v

# Run a specific test file
pytest tests/test_tool.py -v

# Run a specific test class
pytest tests/test_runner.py::TestSubstituteVariables -v

# Run with coverage
pytest tests/ --cov=smarttools --cov-report=html
```

## Test Categories

### Unit Tests (run without external dependencies)

**test_tool.py** - Tool data structures and persistence
- `TestToolArgument` - Custom argument definition and serialization
- `TestPromptStep` - AI prompt step configuration
- `TestCodeStep` - Python code step configuration
- `TestTool` - Full tool configuration and roundtrip serialization
- `TestValidateToolName` - Tool name validation rules
- `TestToolPersistence` - Save/load/delete operations
- `TestLegacyFormat` - Backward compatibility with old format

**test_runner.py** - Tool execution engine
- `TestSubstituteVariables` - Variable placeholder substitution
  - Simple substitution: `{name}` → value
  - Escaped braces: `{{literal}}` → `{literal}`
  - Multiple variables, multiline templates
- `TestExecutePromptStep` - AI provider calls with mocking
- `TestExecuteCodeStep` - Python code execution via `exec()`
- `TestRunTool` - Full tool execution with steps
- `TestCreateArgumentParser` - CLI argument parsing

**test_providers.py** - AI provider abstraction
- `TestProvider` - Provider dataclass and serialization
- `TestProviderResult` - Success/error result handling
- `TestMockProvider` - Built-in mock for testing
- `TestProviderPersistence` - YAML save/load operations
- `TestCallProvider` - Subprocess execution with mocking
- `TestProviderCommandParsing` - Shell command parsing with shlex

**test_cli.py** - Command-line interface
- `TestCLIBasics` - Help, version flags
- `TestListCommand` - List available tools
- `TestCreateCommand` - Create new tools
- `TestDeleteCommand` - Delete tools
- `TestRunCommand` - Execute tools
- `TestTestCommand` - Test tools with mock provider
- `TestProvidersCommand` - Manage AI providers
- `TestRefreshCommand` - Regenerate wrapper scripts
- `TestDocsCommand` - View/edit documentation

### Integration Tests (require running server)

**test_registry_integration.py** - Registry API tests (marked with `@pytest.mark.integration`)
- `TestRegistryIntegration` - List, search, categories, index
- `TestAuthIntegration` - Registration, login, tokens
- `TestPublishIntegration` - Tool publishing workflow

Run with a local registry server:
```bash
# Start registry server
python -m smarttools.registry.app

# Run integration tests
pytest tests/test_registry_integration.py -v -m integration
```

## Test Fixtures

Common fixtures used across tests:

```python
@pytest.fixture
def temp_tools_dir(tmp_path):
    """Redirect TOOLS_DIR and BIN_DIR to temp directory."""
    with patch('smarttools.tool.TOOLS_DIR', tmp_path / ".smarttools"):
        with patch('smarttools.tool.BIN_DIR', tmp_path / ".local" / "bin"):
            yield tmp_path

@pytest.fixture
def temp_providers_file(tmp_path):
    """Redirect providers.yaml to temp directory."""
    providers_file = tmp_path / ".smarttools" / "providers.yaml"
    with patch('smarttools.providers.PROVIDERS_FILE', providers_file):
        yield providers_file
```

## Mocking Strategy

- **File system**: Use `tmp_path` fixture and patch module-level paths
- **Subprocess calls**: Mock `subprocess.run` and `shutil.which`
- **Stdin**: Use `patch('sys.stdin', StringIO("input"))`
- **Provider calls**: Mock `call_provider` to return `ProviderResult`

## Known Limitations

1. **Tool name validation** - CLI doesn't validate names before creation (test skipped)
2. **Nested braces** - `{{{x}}}` breaks due to overlapping escape sequences (documented behavior)
3. **Integration tests** - Require local registry server at `localhost:5000`

## Adding New Tests

1. Create test functions in the appropriate test file
2. Use descriptive names: `test_<what>_<scenario>`
3. Add docstrings explaining the test purpose
4. Use fixtures for common setup
5. Mock external dependencies (subprocess, network, file system)

Example:
```python
def test_tool_with_multiple_code_steps(self):
    """Multiple code steps should pass variables between them."""
    tool = Tool(
        name="multi-code",
        steps=[
            CodeStep(code="x = 1", output_var="x"),
            CodeStep(code="y = int(x) + 1", output_var="y")
        ],
        output="{y}"
    )
    output, code = run_tool(tool, "", {})
    assert code == 0
    assert output == "2"
```