Refactor CLI and UI into modular packages, add registry scrutiny
Major refactoring:
- Split cli.py (1465 lines) into cli/ package with 6 modules
- Split ui_urwid.py (2313 lines) into ui_urwid/ package with 4 modules
- Maintain backwards compatibility via thin wrapper modules
New features:
- Add tool scrutiny system for registry publishing (honesty, transparency, scope, efficiency checks)
- Add optimization suggestions for AI calls that could be pure code
Bug fixes:
- Fix variable substitution escaping ({{literal}} now works)
- Fix provider command parsing with shlex for quoted paths
- Add error logging even without --verbose flag
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
This commit is contained in:
Claude Opus 4.5
@@ -0,0 +1,264 @@
|
||||
# SmartTools Deployment Guide
|
||||
|
||||
This guide covers deploying the SmartTools Registry web application.
|
||||
|
||||
## Quick Start (Development)
|
||||
|
||||
```bash
|
||||
# Clone and install
|
||||
git clone https://gitea.brrd.tech/rob/SmartTools.git
|
||||
cd SmartTools
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Initialize database
|
||||
cd src/smarttools/web
|
||||
python -c "from app import create_app; create_app()"
|
||||
|
||||
# Run development server
|
||||
flask run --port 5050 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Access at `http://localhost:5050`
|
||||
|
||||
## Production Deployment
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.8+
|
||||
- pip/pipx
|
||||
- nginx (recommended for reverse proxy)
|
||||
- systemd (for process management)
|
||||
|
||||
### 1. Install Application
|
||||
|
||||
```bash
|
||||
# Create application directory
|
||||
sudo mkdir -p /opt/smarttools
|
||||
sudo chown $USER:$USER /opt/smarttools
|
||||
cd /opt/smarttools
|
||||
|
||||
# Clone repository
|
||||
git clone https://gitea.brrd.tech/rob/SmartTools.git .
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -e ".[prod]"
|
||||
pip install gunicorn
|
||||
```
|
||||
|
||||
### 2. Configure Environment
|
||||
|
||||
Create `/opt/smarttools/.env`:
|
||||
|
||||
```bash
|
||||
# Required
|
||||
FLASK_ENV=production
|
||||
SECRET_KEY=your-secret-key-here # Generate with: python -c "import secrets; print(secrets.token_hex(32))"
|
||||
|
||||
# Optional
|
||||
SENTRY_DSN=https://[email protected]/xxx # Error monitoring
|
||||
SITE_URL=https://registry.smarttools.dev # For sitemap/SEO
|
||||
```
|
||||
|
||||
### 3. Initialize Database
|
||||
|
||||
```bash
|
||||
cd /opt/smarttools/src/smarttools/web
|
||||
python -c "from app import create_app; create_app()"
|
||||
```
|
||||
|
||||
Database will be created at `data/smarttools.db`.
|
||||
|
||||
### 4. systemd Service
|
||||
|
||||
Create `/etc/systemd/system/smarttools-web.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=SmartTools Registry Web Application
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=www-data
|
||||
Group=www-data
|
||||
WorkingDirectory=/opt/smarttools
|
||||
Environment="PATH=/opt/smarttools/venv/bin"
|
||||
EnvironmentFile=/opt/smarttools/.env
|
||||
ExecStart=/opt/smarttools/venv/bin/gunicorn \
|
||||
--workers 4 \
|
||||
--bind 127.0.0.1:5050 \
|
||||
--access-logfile /var/log/smarttools/access.log \
|
||||
--error-logfile /var/log/smarttools/error.log \
|
||||
'smarttools.web.app:create_app()'
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable and start:
|
||||
|
||||
```bash
|
||||
# Create log directory
|
||||
sudo mkdir -p /var/log/smarttools
|
||||
sudo chown www-data:www-data /var/log/smarttools
|
||||
|
||||
# Enable service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable smarttools-web
|
||||
sudo systemctl start smarttools-web
|
||||
|
||||
# Check status
|
||||
sudo systemctl status smarttools-web
|
||||
```
|
||||
|
||||
### 5. nginx Configuration
|
||||
|
||||
Create `/etc/nginx/sites-available/smarttools`:
|
||||
|
||||
```nginx
|
||||
server {
|
||||
listen 80;
|
||||
server_name registry.smarttools.dev;
|
||||
|
||||
# Redirect HTTP to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name registry.smarttools.dev;
|
||||
|
||||
# SSL certificates (use certbot for Let's Encrypt)
|
||||
ssl_certificate /etc/letsencrypt/live/registry.smarttools.dev/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/registry.smarttools.dev/privkey.pem;
|
||||
|
||||
# SSL settings
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_prefer_server_ciphers on;
|
||||
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
|
||||
|
||||
# Static files (optional - Flask can serve these)
|
||||
location /static/ {
|
||||
alias /opt/smarttools/src/smarttools/web/static/;
|
||||
expires 1d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
|
||||
# Proxy to gunicorn
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:5050;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Enable site:
|
||||
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/smarttools /etc/nginx/sites-enabled/
|
||||
sudo nginx -t
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
### 6. SSL with Let's Encrypt
|
||||
|
||||
```bash
|
||||
sudo apt install certbot python3-certbot-nginx
|
||||
sudo certbot --nginx -d registry.smarttools.dev
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `FLASK_ENV` | Yes | `production` or `development` |
|
||||
| `SECRET_KEY` | Yes | Secret key for sessions (32+ chars) |
|
||||
| `SENTRY_DSN` | No | Sentry error monitoring DSN |
|
||||
| `SITE_URL` | No | Public URL for sitemap generation |
|
||||
| `DATABASE_PATH` | No | Override database path (default: `data/smarttools.db`) |
|
||||
|
||||
## Database Backup
|
||||
|
||||
```bash
|
||||
# Backup
|
||||
cp /opt/smarttools/data/smarttools.db /backup/smarttools-$(date +%Y%m%d).db
|
||||
|
||||
# Or use SQLite backup command for consistency
|
||||
sqlite3 /opt/smarttools/data/smarttools.db ".backup '/backup/smarttools.db'"
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Logs
|
||||
|
||||
```bash
|
||||
# Application logs
|
||||
tail -f /var/log/smarttools/error.log
|
||||
tail -f /var/log/smarttools/access.log
|
||||
|
||||
# systemd logs
|
||||
journalctl -u smarttools-web -f
|
||||
```
|
||||
|
||||
### Sentry Integration
|
||||
|
||||
The application includes Sentry integration for error monitoring. Set `SENTRY_DSN` environment variable to enable.
|
||||
|
||||
### Health Check
|
||||
|
||||
```bash
|
||||
curl http://localhost:5050/api/v1/tools
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service won't start
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
journalctl -u smarttools-web -n 50
|
||||
|
||||
# Verify Python path
|
||||
/opt/smarttools/venv/bin/python -c "import smarttools"
|
||||
|
||||
# Test gunicorn manually
|
||||
cd /opt/smarttools
|
||||
source venv/bin/activate
|
||||
gunicorn --bind 127.0.0.1:5050 'smarttools.web.app:create_app()'
|
||||
```
|
||||
|
||||
### Database errors
|
||||
|
||||
```bash
|
||||
# Check database exists and is readable
|
||||
ls -la /opt/smarttools/data/smarttools.db
|
||||
|
||||
# Verify permissions
|
||||
chown www-data:www-data /opt/smarttools/data/
|
||||
chown www-data:www-data /opt/smarttools/data/smarttools.db
|
||||
```
|
||||
|
||||
### Static files not loading
|
||||
|
||||
```bash
|
||||
# Rebuild Tailwind CSS
|
||||
cd /opt/smarttools
|
||||
npm install
|
||||
npm run css:build
|
||||
|
||||
# Or use development CSS
|
||||
# (static/css/output.css should exist)
|
||||
```
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
See `docs/diagrams/deployment-architecture.svg` for visual representation.
|
||||
+180
@@ -0,0 +1,180 @@
|
||||
# SmartTools Project Overview
|
||||
|
||||
This document provides a comprehensive overview of the SmartTools project structure, components, and how everything fits together.
|
||||
|
||||
## What is SmartTools?
|
||||
|
||||
SmartTools is a personal tool builder for AI-powered CLI commands. It consists of three main components:
|
||||
|
||||
1. **CLI Tool** (`smarttools`) - Create, manage, and run AI-powered command-line tools
|
||||
2. **Registry API** - REST API for publishing and discovering tools
|
||||
3. **Web UI** - Browser interface for the registry with docs, search, and user dashboard
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| CLI | `src/smarttools/cli.py` | Main entry point for `smarttools` command |
|
||||
| Registry API | `src/smarttools/registry/` | REST API for tool registry |
|
||||
| Web UI | `src/smarttools/web/` | Flask web application |
|
||||
| Tool Storage | `~/.smarttools/` | User's installed tools |
|
||||
| Database | `data/smarttools.db` | SQLite with FTS5 search |
|
||||
|
||||
## Architecture Diagrams
|
||||
|
||||
See `docs/diagrams/` for visual representations:
|
||||
|
||||
- **system-architecture.svg** - How components interact
|
||||
- **deployment-architecture.svg** - Production deployment setup
|
||||
- **source-organization.svg** - Code structure
|
||||
|
||||
## Source Code Structure
|
||||
|
||||
```
|
||||
src/smarttools/
|
||||
├── cli.py # Entry point, subcommand routing
|
||||
├── tool.py # Tool dataclasses, YAML loading
|
||||
├── runner.py # Step execution engine
|
||||
├── providers.py # AI provider abstraction
|
||||
├── config.py # Global configuration
|
||||
├── manifest.py # Tool manifest handling
|
||||
│
|
||||
├── registry/ # Registry API Server
|
||||
│ ├── app.py # Flask API application
|
||||
│ ├── db.py # Database operations, FTS5 search
|
||||
│ ├── rate_limit.py # API rate limiting
|
||||
│ ├── sync.py # Git sync for tool submissions
|
||||
│ ├── categorize.py # Auto-categorization
|
||||
│ └── similarity.py # Similar tool recommendations
|
||||
│
|
||||
├── web/ # Web UI Application
|
||||
│ ├── app.py # Flask web app, Sentry integration
|
||||
│ ├── routes.py # Page routes and handlers
|
||||
│ ├── auth.py # Password hashing, login/register
|
||||
│ ├── sessions.py # Session management
|
||||
│ ├── docs_content.py # Documentation text content
|
||||
│ ├── seo.py # Sitemap, robots.txt
|
||||
│ ├── filters.py # Jinja template filters
|
||||
│ ├── templates/ # Jinja HTML templates
|
||||
│ └── static/ # CSS, JS, images
|
||||
│
|
||||
├── registry_client.py # CLI client for registry API
|
||||
├── resolver.py # Dependency resolution
|
||||
│
|
||||
├── ui.py # TUI dispatcher
|
||||
├── ui_urwid.py # urwid TUI implementation
|
||||
└── ui_snack.py # newt/snack TUI fallback
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
SQLite database at `data/smarttools.db`:
|
||||
|
||||
| Table | Purpose |
|
||||
|-------|---------|
|
||||
| `publishers` | User accounts (username, email, password_hash) |
|
||||
| `tools` | Published tools (name, owner, description, config) |
|
||||
| `api_tokens` | Authentication tokens for API access |
|
||||
| `download_stats` | Tool download tracking |
|
||||
| `pageviews` | Analytics for tool detail pages |
|
||||
| `tools_fts` | Full-text search index (FTS5) |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
Base URL: `/api/v1/`
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/tools` | GET | List/search tools |
|
||||
| `/tools/<owner>/<name>` | GET | Get tool details |
|
||||
| `/tools/<owner>/<name>` | POST | Publish tool (auth required) |
|
||||
| `/tools/<owner>/<name>/download` | POST | Record download |
|
||||
| `/me` | GET | Current user info |
|
||||
| `/tokens` | GET/POST | Manage API tokens |
|
||||
|
||||
## Web Routes
|
||||
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `/` | Homepage with featured tools |
|
||||
| `/tools` | Tool browser with search |
|
||||
| `/tools/<owner>/<name>` | Tool detail page |
|
||||
| `/docs/*` | Documentation pages |
|
||||
| `/login`, `/register` | Authentication |
|
||||
| `/dashboard/*` | User dashboard, token management |
|
||||
|
||||
## Key Files to Know
|
||||
|
||||
### Configuration
|
||||
- `~/.smarttools/providers.yaml` - AI provider configurations
|
||||
- `~/.smarttools/config.yaml` - Global settings
|
||||
- `~/.smarttools/<tool>/config.yaml` - Individual tool configs
|
||||
|
||||
### Development
|
||||
- `pyproject.toml` - Package configuration
|
||||
- `pytest.ini` - Test configuration
|
||||
- `tailwind.config.js` - Tailwind CSS configuration
|
||||
|
||||
### Documentation
|
||||
- `docs/REGISTRY.md` - Registry design document (comprehensive)
|
||||
- `docs/WEB_UI.md` - Web UI design document (comprehensive)
|
||||
- `wiki/Home.md` - User-facing wiki documentation
|
||||
- `CLAUDE.md` - Development guidance for AI assistants
|
||||
|
||||
## Running the Project
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Install with dev dependencies
|
||||
pip install -e ".[dev]"
|
||||
|
||||
# Run CLI
|
||||
python -m smarttools.cli
|
||||
|
||||
# Run web server (development)
|
||||
cd src/smarttools/web
|
||||
flask run --port 5050
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Using gunicorn
|
||||
gunicorn -w 4 -b 0.0.0.0:5050 'smarttools.web.app:create_app()'
|
||||
|
||||
# Environment variables needed:
|
||||
# FLASK_ENV=production
|
||||
# SECRET_KEY=<your-secret-key>
|
||||
# SENTRY_DSN=<optional-sentry-dsn>
|
||||
# SITE_URL=https://your-domain.com
|
||||
```
|
||||
|
||||
### systemd Service
|
||||
|
||||
See `docs/DEPLOYMENT.md` for systemd service configuration.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run specific test
|
||||
pytest tests/test_name.py::test_function
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=smarttools
|
||||
```
|
||||
|
||||
## Design Documents
|
||||
|
||||
For detailed design decisions and specifications:
|
||||
|
||||
- **docs/REGISTRY.md** - Complete registry API design (Phases 1-8)
|
||||
- **docs/WEB_UI.md** - Complete web UI design (Phase 7)
|
||||
- **wiki/Home.md** - User documentation for CLI usage
|
||||
|
||||
## Archive
|
||||
|
||||
Old development discussions and diagrams are preserved in `archive/discussions/` for historical reference.
|
||||
Reference in New Issue
Block a user