149 lines
5.1 KiB
Markdown
149 lines
5.1 KiB
Markdown
# SmartTools Test Suite
|
|
|
|
## Overview
|
|
|
|
The test suite covers the core SmartTools modules with **158 unit tests** organized into four main test files:
|
|
|
|
| Test File | Module Tested | Test Count | Description |
|
|
|-----------|---------------|------------|-------------|
|
|
| `test_tool.py` | `tool.py` | 35 | Tool definitions, YAML loading/saving |
|
|
| `test_runner.py` | `runner.py` | 31 | Variable substitution, step execution |
|
|
| `test_providers.py` | `providers.py` | 35 | Provider management, AI calls |
|
|
| `test_cli.py` | `cli/` | 20 | CLI commands and arguments |
|
|
| `test_registry_integration.py` | Registry | 37 | Registry client, resolution (12 require server) |
|
|
|
|
## Running Tests
|
|
|
|
```bash
|
|
# Run all unit tests (excluding integration tests)
|
|
pytest tests/ -m "not integration"
|
|
|
|
# Run all tests with verbose output
|
|
pytest tests/ -v
|
|
|
|
# Run a specific test file
|
|
pytest tests/test_tool.py -v
|
|
|
|
# Run a specific test class
|
|
pytest tests/test_runner.py::TestSubstituteVariables -v
|
|
|
|
# Run with coverage
|
|
pytest tests/ --cov=smarttools --cov-report=html
|
|
```
|
|
|
|
## Test Categories
|
|
|
|
### Unit Tests (run without external dependencies)
|
|
|
|
**test_tool.py** - Tool data structures and persistence
|
|
- `TestToolArgument` - Custom argument definition and serialization
|
|
- `TestPromptStep` - AI prompt step configuration
|
|
- `TestCodeStep` - Python code step configuration
|
|
- `TestTool` - Full tool configuration and roundtrip serialization
|
|
- `TestValidateToolName` - Tool name validation rules
|
|
- `TestToolPersistence` - Save/load/delete operations
|
|
- `TestLegacyFormat` - Backward compatibility with old format
|
|
|
|
**test_runner.py** - Tool execution engine
|
|
- `TestSubstituteVariables` - Variable placeholder substitution
|
|
- Simple substitution: `{name}` → value
|
|
- Escaped braces: `{{literal}}` → `{literal}`
|
|
- Multiple variables, multiline templates
|
|
- `TestExecutePromptStep` - AI provider calls with mocking
|
|
- `TestExecuteCodeStep` - Python code execution via `exec()`
|
|
- `TestRunTool` - Full tool execution with steps
|
|
- `TestCreateArgumentParser` - CLI argument parsing
|
|
|
|
**test_providers.py** - AI provider abstraction
|
|
- `TestProvider` - Provider dataclass and serialization
|
|
- `TestProviderResult` - Success/error result handling
|
|
- `TestMockProvider` - Built-in mock for testing
|
|
- `TestProviderPersistence` - YAML save/load operations
|
|
- `TestCallProvider` - Subprocess execution with mocking
|
|
- `TestProviderCommandParsing` - Shell command parsing with shlex
|
|
|
|
**test_cli.py** - Command-line interface
|
|
- `TestCLIBasics` - Help, version flags
|
|
- `TestListCommand` - List available tools
|
|
- `TestCreateCommand` - Create new tools
|
|
- `TestDeleteCommand` - Delete tools
|
|
- `TestRunCommand` - Execute tools
|
|
- `TestTestCommand` - Test tools with mock provider
|
|
- `TestProvidersCommand` - Manage AI providers
|
|
- `TestRefreshCommand` - Regenerate wrapper scripts
|
|
- `TestDocsCommand` - View/edit documentation
|
|
|
|
### Integration Tests (require running server)
|
|
|
|
**test_registry_integration.py** - Registry API tests (marked with `@pytest.mark.integration`)
|
|
- `TestRegistryIntegration` - List, search, categories, index
|
|
- `TestAuthIntegration` - Registration, login, tokens
|
|
- `TestPublishIntegration` - Tool publishing workflow
|
|
|
|
Run with a local registry server:
|
|
```bash
|
|
# Start registry server
|
|
python -m smarttools.registry.app
|
|
|
|
# Run integration tests
|
|
pytest tests/test_registry_integration.py -v -m integration
|
|
```
|
|
|
|
## Test Fixtures
|
|
|
|
Common fixtures used across tests:
|
|
|
|
```python
|
|
@pytest.fixture
|
|
def temp_tools_dir(tmp_path):
|
|
"""Redirect TOOLS_DIR and BIN_DIR to temp directory."""
|
|
with patch('smarttools.tool.TOOLS_DIR', tmp_path / ".smarttools"):
|
|
with patch('smarttools.tool.BIN_DIR', tmp_path / ".local" / "bin"):
|
|
yield tmp_path
|
|
|
|
@pytest.fixture
|
|
def temp_providers_file(tmp_path):
|
|
"""Redirect providers.yaml to temp directory."""
|
|
providers_file = tmp_path / ".smarttools" / "providers.yaml"
|
|
with patch('smarttools.providers.PROVIDERS_FILE', providers_file):
|
|
yield providers_file
|
|
```
|
|
|
|
## Mocking Strategy
|
|
|
|
- **File system**: Use `tmp_path` fixture and patch module-level paths
|
|
- **Subprocess calls**: Mock `subprocess.run` and `shutil.which`
|
|
- **Stdin**: Use `patch('sys.stdin', StringIO("input"))`
|
|
- **Provider calls**: Mock `call_provider` to return `ProviderResult`
|
|
|
|
## Known Limitations
|
|
|
|
1. **Tool name validation** - CLI doesn't validate names before creation (test skipped)
|
|
2. **Nested braces** - `{{{x}}}` breaks due to overlapping escape sequences (documented behavior)
|
|
3. **Integration tests** - Require local registry server at `localhost:5000`
|
|
|
|
## Adding New Tests
|
|
|
|
1. Create test functions in the appropriate test file
|
|
2. Use descriptive names: `test_<what>_<scenario>`
|
|
3. Add docstrings explaining the test purpose
|
|
4. Use fixtures for common setup
|
|
5. Mock external dependencies (subprocess, network, file system)
|
|
|
|
Example:
|
|
```python
|
|
def test_tool_with_multiple_code_steps(self):
|
|
"""Multiple code steps should pass variables between them."""
|
|
tool = Tool(
|
|
name="multi-code",
|
|
steps=[
|
|
CodeStep(code="x = 1", output_var="x"),
|
|
CodeStep(code="y = int(x) + 1", output_var="y")
|
|
],
|
|
output="{y}"
|
|
)
|
|
output, code = run_tool(tool, "", {})
|
|
assert code == 0
|
|
assert output == "2"
|
|
```
|