rob
/
smarttools
1
0
Fork 0
Code Issues Pull Requests Packages Projects 1 Releases Wiki Activity
smarttools/docs/REGISTRY.md

1897 lines
58 KiB
Markdown
Raw Blame History

# SmartTools Registry Design
## Purpose
Build a centralized registry for SmartTools to enable discovery, publishing, dependency management, and future curation at scale.
## Terminology
| Term | Definition |
|------|------------|
| **Tool definition** | The full YAML file in the registry (`config.yaml`) containing name, steps, arguments, etc. |
| **Tool config** | The configuration within a tool definition (arguments, steps, provider settings) |
| **smarttools.yaml** | Project manifest file declaring tool dependencies and overrides |
| **config.yaml** | The tool definition file, both in registry and when installed locally |
| **Owner** | Immutable namespace slug identifying the publisher (e.g., `rob`, `alice`) |
| **Publisher** | A registered user who can publish tools to the registry |
| **Wrapper script** | Auto-generated bash script in `~/.local/bin/` that invokes a tool |
**Canonical naming:** Use `SmartTools-Registry` (capitalized, hyphenated) for the repository name.
## Diagram References
- System overview: `discussions/diagrams/smarttools-registry_rob_1.puml`
- Data flows: `discussions/diagrams/smarttools-registry_rob_5.puml`
## System Overview
Users interact via the CLI and a future Web UI. Both call a Registry API hosted at `https://gitea.brrd.tech/api/v1` (future alias: `registry.smarttools.dev/api/v1`). The API syncs from a Gitea-backed registry repo and maintains a SQLite cache/search index.
**Canonical API base path:** `https://gitea.brrd.tech/api/v1`
All API endpoints are versioned under `/api/v1`. When breaking changes are needed, a new version (`/api/v2`) will be introduced with deprecation notices.
Core API endpoints:
- `GET /api/v1/tools`
- `GET /api/v1/tools/search?q=...`
- `GET /api/v1/tools/{owner}/{name}`
- `GET /api/v1/tools/{owner}/{name}/versions`
- `GET /api/v1/tools/{owner}/{name}/download?version=...`
- `POST /api/v1/tools` (publish)
- `GET /api/v1/categories`
- `GET /api/v1/stats/popular`
- `POST /api/v1/webhook/gitea`
### Pagination
All list endpoints support pagination:
| Parameter | Default | Max | Description |
|-----------|---------|-----|-------------|
| `page` | 1 | - | Page number (1-indexed) |
| `per_page` | 20 | 100 | Items per page |
| `sort` | `downloads` | - | Sort field |
| `order` | `desc` | - | Sort order (asc/desc) |
**Stable ordering:** To ensure deterministic results across pages, sorting includes a secondary key:
- Primary: requested field (e.g., `downloads`)
- Secondary: `published_at` (desc)
- Tertiary: `id` (for absolute stability)
```sql
ORDER BY downloads DESC, published_at DESC, id DESC
LIMIT 20 OFFSET 0
```
**Response pagination metadata:**
```json
{
"data": [...],
"meta": {
"page": 1,
"per_page": 20,
"total": 142,
"total_pages": 8
}
}
```
### Input Constraints
Size limits to prevent oversized uploads:
| Field | Max Size | Notes |
|-------|----------|-------|
| `config.yaml` | 64 KB | Tool definition |
| `README.md` | 256 KB | Documentation |
| Request body | 512 KB | Total POST payload |
| Tool name | 64 chars | Alphanumeric + hyphen |
| Description | 500 chars | Short summary |
| Tag | 32 chars | Individual tag |
| Tags array | 10 items | Maximum tags per tool |
**Validation errors:**
```json
{
"error": {
"code": "PAYLOAD_TOO_LARGE",
"message": "config.yaml exceeds 64KB limit",
"details": {
"field": "config",
"size": 72000,
"limit": 65536
}
}
}
```
### Sort Fields and Indexes
**Allowed sort fields:**
| Endpoint | Allowed `sort` values |
|----------|----------------------|
| `GET /tools` | `downloads`, `published_at`, `name` |
| `GET /tools/search` | `relevance`, `downloads`, `published_at` |
| `GET /categories` | `name`, `tool_count` |
Invalid sort values return 400:
```json
{"error": {"code": "INVALID_SORT", "message": "Unknown sort field 'foo'. Allowed: downloads, published_at, name"}}
```
**Database indexes:**
```sql
-- Frequent query patterns
CREATE INDEX idx_tools_owner_name ON tools(owner, name);
CREATE INDEX idx_tools_category ON tools(category);
CREATE INDEX idx_tools_published_at ON tools(published_at DESC);
CREATE INDEX idx_tools_downloads ON tools(downloads DESC);
CREATE INDEX idx_tools_owner_name_version ON tools(owner, name, version);
-- For pagination stability
CREATE INDEX idx_tools_sort_stable ON tools(downloads DESC, published_at DESC, id DESC);
-- Publisher lookups
CREATE INDEX idx_publishers_slug ON publishers(slug);
CREATE INDEX idx_publishers_email ON publishers(email);
-- Token lookups
CREATE INDEX idx_api_tokens_hash ON api_tokens(token_hash);
CREATE INDEX idx_api_tokens_publisher ON api_tokens(publisher_id);
```
### API Version Compatibility
**Forward compatibility:** Clients should ignore unknown fields in API responses:
```python
# Good: ignore unknown fields
tool = response['data']
name = tool.get('name')
# Don't fail if 'new_field' exists but client doesn't know about it
# Bad: strict parsing that fails on unknown fields
tool = ToolSchema.parse(response['data']) # May fail on new fields
```
**Backward compatibility:** The API will:
- Never remove fields in a version (only deprecate)
- Never change field types
- Add new optional fields without version bump
- Use new version (`/api/v2`) for breaking changes
**Deprecation process:**
1. Add `X-Deprecated-Field: old_field` header
2. Document in changelog
3. Remove after 6 months minimum
4. Major version bump if widely used
**Client version header:**
```
X-SmartTools-Client: cli/1.2.0
```
Helps server track client versions for deprecation decisions.
## Source of Truth
- Gitea registry repo is the source of truth.
- API syncs repo content into SQLite for fast queries, stats, and FTS5 search.
- `index.json` remains useful for offline CLI search and as a fallback.
If the cache is stale, the API can fall back to repo reads; a warning header may be emitted.
## Namespacing and Paths
Support owner/name from day one:
- Registry path: `tools/{owner}/{name}/config.yaml`
- API URL: `/tools/{owner}/{name}`
- Install: `smarttools registry install rob/summarize`
- Shorthand: `smarttools registry install summarize` resolves to the official namespace.
PR branches: `submit/{owner}/{name}/{version}`.
### Namespace Identity
The `owner` is an **immutable slug**, not the display name:
```sql
-- In publishers table
slug TEXT UNIQUE NOT NULL, -- immutable: "rob", "alice-dev"
display_name TEXT NOT NULL, -- mutable: "Rob", "Alice Developer"
```
**Slug rules:**
- Lowercase alphanumeric + hyphens only: `^[a-z0-9][a-z0-9-]*[a-z0-9]$`
- 2-39 characters
- Cannot start/end with hyphen
- Set once at registration, cannot be changed
- Reserved slugs: `official`, `admin`, `system`, `api`, `registry`
**Rename policy:**
- `display_name` can be changed anytime via dashboard
- `slug` (owner) is permanent to preserve URLs and tool references
- If a publisher absolutely must change slug (legal reasons, etc.):
1. Create new account with new slug
2. Republish tools under new namespace
3. Mark old tools as deprecated with `replacement` pointing to new namespace
4. Old namespace remains reserved (cannot be reused by others)
**Why immutable:**
- `rob/summarize@1.0.0` must always resolve to the same tool
- Prevents namespace hijacking after rename
- Simplifies caching and CDN strategies
## Tool Format (Registry == Local)
Registry tool folders mirror local tools:
```
tools/
rob/
summarize/
config.yaml
README.md
```
Tool files match the existing SmartTools format. Registry-specific metadata is kept under `registry:`. Deprecation is tool-defined and top-level:
```yaml
name: summarize
version: "1.2.0"
deprecated: true
deprecated_message: "Security issue. Use v1.2.1"
replacement: "rob/summarize@1.2.1"
registry:
published_at: "2025-01-15T10:30:00Z"
downloads: 142
```
**Schema compatibility note:** The current SmartTools config parser may reject unknown top-level keys like `deprecated`, `replacement`, and `registry`. Before implementing registry features:
1. Update the YAML parser to ignore unknown keys (permissive mode)
2. Or explicitly define these fields in the Tool dataclass with defaults
3. Validate registry-specific fields only when publishing, not when running locally
This ensures local tools continue to work even if they don't have registry fields.
## Versioning and Immutability
- Unique key: `owner/name + version`.
- Published versions are immutable.
- Deprecation uses `deprecated`, `deprecated_message`, and `replacement`.
- CLI warns on install if a version is deprecated.
### Yank Policy
Yanking allows removing a version from resolution without deleting it (for auditability):
```yaml
# In tool config
yanked: true
yanked_reason: "Critical security vulnerability CVE-2025-1234"
yanked_at: "2025-01-20T15:00:00Z"
```
**Yanked version behavior:**
| Operation | Behavior |
|-----------|----------|
| `install foo@1.0.0` (exact) | Warns but allows install |
| `install foo@^1.0.0` (constraint) | Excludes yanked, resolves to next valid |
| `search` / `browse` | Hidden by default, shown with `--include-yanked` |
| Direct URL access | Returns tool with `yanked: true` in response |
| Already installed | Continues to work, no forced removal |
**Database schema addition:**
```sql
-- Add to tools table
yanked BOOLEAN DEFAULT FALSE,
yanked_reason TEXT,
yanked_at TIMESTAMP
```
**Yank vs Delete:**
- **Yank**: Version remains in DB, excluded from resolution, auditable
- **Delete**: Reserved for DMCA/legal, requires admin action, leaves tombstone record
### Version Format
Tools use semantic versioning (semver):
```
MAJOR.MINOR.PATCH[-PRERELEASE][+BUILD]
Examples:
1.0.0 # stable release
1.2.3 # stable release
2.0.0-alpha.1 # prerelease
2.0.0-beta.2 # prerelease
2.0.0-rc.1 # release candidate
```
### Version Constraints
Manifest files support these constraint formats:
| Constraint | Meaning | Example Match |
|------------|---------|---------------|
| `1.2.3` | Exact version | `1.2.3` only |
| `>=1.2.0` | Minimum version | `1.2.0`, `1.3.0`, `2.0.0` |
| `<2.0.0` | Below version | `1.9.9`, `1.0.0` |
| `>=1.0.0,<2.0.0` | Range | `1.0.0` to `1.9.9` |
| `^1.2.3` | Compatible (same major) | `1.2.3` to `1.9.9` |
| `~1.2.3` | Approximately (same minor) | `1.2.3` to `1.2.9` |
| `*` | Any version | latest stable |
### Version Resolution Rules
When resolving a version constraint:
1. **Filter**: Get all versions matching the constraint
2. **Exclude prereleases**: Unless constraint explicitly includes them (e.g., `>=2.0.0-alpha.1`)
3. **Sort**: By semver precedence (descending)
4. **Select**: Highest matching version
**Tie-breakers:**
- Stable versions preferred over prereleases
- Later publish date wins if versions are equal (shouldn't happen with immutability)
**Unsatisfiable constraints:**
```json
// API Response: 404
{
"error": {
"code": "VERSION_NOT_FOUND",
"message": "No version of 'rob/summarize' satisfies constraint '>=5.0.0'",
"details": {
"tool": "rob/summarize",
"constraint": ">=5.0.0",
"available_versions": ["1.0.0", "1.1.0", "1.2.0"],
"latest_stable": "1.2.0"
}
}
}
```
### Prerelease Handling
- Prereleases are **not** returned for `*` or range constraints by default
- To install prerelease: `smarttools registry install rob/summarize@2.0.0-beta.1`
- To allow prereleases in manifest: `version: ">=2.0.0-0"` (the `-0` suffix includes prereleases)
### Download Endpoint Version Selection
The `/api/v1/tools/{owner}/{name}/download` endpoint accepts version parameters:
| Parameter | Behavior | Example |
|-----------|----------|---------|
| (none) | Returns latest stable version | `/download` → `1.2.0` |
| `version=1.2.0` | Exact version (must exist) | `/download?version=1.2.0` |
| `version=^1.0.0` | Server resolves constraint | `/download?version=^1.0.0` → `1.2.0` |
| `version=latest` | Alias for latest stable | `/download?version=latest` |
**Server-side resolution:** The API server resolves version constraints, not the client. This ensures consistent resolution and allows the server to apply policies (e.g., exclude yanked versions).
```
GET /api/v1/tools/rob/summarize/download?version=^1.0.0&install=true
Response (200):
{
"data": {
"owner": "rob",
"name": "summarize",
"resolved_version": "1.2.0",
"config": "... YAML content ..."
},
"meta": {
"constraint": "^1.0.0",
"available_versions": ["1.0.0", "1.1.0", "1.2.0"]
}
}
```
**Invalid/unsatisfiable constraint:**
```
GET /api/v1/tools/rob/summarize/download?version=^5.0.0
Response (404):
{
"error": {
"code": "CONSTRAINT_UNSATISFIABLE",
"message": "No version matches constraint '^5.0.0'",
"details": {
"constraint": "^5.0.0",
"latest_stable": "1.2.0",
"available_versions": ["1.0.0", "1.1.0", "1.2.0"]
}
}
}
```
## Tool Resolution Order
When a tool is invoked, the CLI searches in this order:
1. **Local project**: `./.smarttools/<owner>/<name>/config.yaml` (or `./.smarttools/<name>/` for unnamespaced)
2. **Global user**: `~/.smarttools/<owner>/<name>/config.yaml`
3. **Registry**: Fetch from API, install to global, then run
4. **Error**: `Tool '<toolname>' not found`
Step 3 only occurs if `auto_fetch_from_registry: true` in config (default: true).
**Path convention:** Use `.smarttools/` (with leading dot) for both local and global to maintain consistency.
Resolution also respects namespacing:
- `summarize` → searches for any tool named `summarize`, prefers `official/summarize` if exists
- `rob/summarize` → searches for exactly `rob/summarize`
### Official Namespace
The slug `official` is reserved for curated, high-quality tools maintained by the registry administrators.
- Shorthand `summarize` resolves to `official/summarize` if it exists
- If no `official/summarize`, falls back to most-downloaded tool named `summarize`
- To avoid ambiguity, always use full `owner/name` in manifests
Reserved slugs that cannot be registered: `official`, `admin`, `system`, `api`, `registry`, `smarttools`
## Auto-Fetch Behavior
When enabled (`auto_fetch_from_registry: true`), missing tools are automatically fetched:
```bash
$ summarize < file.txt
# Tool 'summarize' not found locally.
# Fetching from registry...
# Installed: official/summarize@1.2.0
# Running...
```
Behavior details:
- Fetches latest stable version unless pinned in `smarttools.yaml`
- Installs to `~/.smarttools/<owner>/<name>/`
- Generates wrapper script in `~/.local/bin/`
- Subsequent runs use local copy (no re-fetch)
To disable (require explicit install):
```yaml
# ~/.smarttools/config.yaml
auto_fetch_from_registry: false
```
### Wrapper Script Collisions
When two tools from different owners have the same name:
| Scenario | Behavior |
|----------|----------|
| Install `official/summarize` | Creates wrapper `~/.local/bin/summarize` |
| Install `rob/summarize` (collision) | Creates wrapper `~/.local/bin/rob-summarize` |
| Uninstall `official/summarize` | Removes `summarize` wrapper, promotes `rob-summarize` → `summarize` if desired |
The first-installed tool with a given name gets the short wrapper. Subsequent tools use `owner-name` format.
To invoke a specific owner's tool:
```bash
# Short form (whichever was installed first)
summarize < file.txt
# Explicit owner form (always works)
rob-summarize < file.txt
# Or via smarttools run
smarttools run rob/summarize < file.txt
```
## Project Manifest (smarttools.yaml)
Defines tool dependencies with optional runtime overrides:
```
name: my-ai-project
version: "1.0.0"
dependencies:
- name: rob/summarize
version: ">=1.0.0"
overrides:
rob/summarize:
provider: ollama
```
Overrides are applied at runtime and do not mutate installed tool configs.
## CLI Config and Tokens
Global config lives in `~/.smarttools/config.yaml`:
```yaml
registry:
url: https://gitea.brrd.tech/api/v1 # Must match canonical base path
token: "reg_xxxxxxxxxxxx"
client_id: "anon_abc123def456"
auto_fetch_from_registry: true
```
`client_id` is generated locally and used for anonymous install dedupe.
## Publishing and Auth
Publishing uses registry accounts, not Gitea accounts:
- Public endpoints require no auth.
- `POST /tools` requires a registry token.
- The API server uses a private Gitea service account to open PRs.
### Publish Idempotency and Edge Cases
**Idempotency key:** `owner/name@version`
| Scenario | API Response | HTTP Code |
|----------|--------------|-----------|
| New version, no PR exists | Create PR, return URL | `201 Created` |
| PR already exists (pending) | Return existing PR URL | `200 OK` |
| Version already published | Error: version exists | `409 Conflict` |
| PR was closed without merge | Allow new PR | `201 Created` |
| PR was merged, then tool deleted | Error: version exists (tombstone) | `409 Conflict` |
**Version immutability enforcement:**
```json
// Attempt to publish existing version
// Response: 409 Conflict
{
"error": {
"code": "VERSION_EXISTS",
"message": "Version 1.2.0 of 'rob/summarize' already exists and cannot be overwritten",
"details": {
"published_at": "2025-01-15T10:30:00Z",
"action": "Bump version number to publish changes"
}
}
}
```
**Closed PR handling:**
- Track PR state in database: `pending`, `merged`, `closed`
- If PR was closed (rejected/abandoned), allow new submission for same version
- If PR was merged, version is immutable forever
**Update flow (new version, not overwrite):**
1. Developer modifies tool locally
2. Bumps version in `config.yaml` (e.g., `1.2.0` → `1.3.0`)
3. Runs `smarttools registry publish`
4. New PR created for `1.3.0`
5. Old version `1.2.0` remains available
## Publisher Registration
Publishers register on the registry website, not Gitea:
**Registration flow:**
1. User visits `https://gitea.brrd.tech/registry/register` (or future `registry.smarttools.dev`)
2. Creates account with email + password + slug
3. Receives verification email (optional in v1, but track `verified` status)
4. Logs into dashboard at `/dashboard`
5. Generates API token from dashboard
6. Uses token in CLI for publishing
### Authentication Security
**Password hashing:**
- Algorithm: Argon2id (memory-hard, recommended by OWASP)
- Parameters: `memory=65536, iterations=3, parallelism=4`
- Library: `argon2-cffi` for Python
```python
from argon2 import PasswordHasher
ph = PasswordHasher(memory_cost=65536, time_cost=3, parallelism=4)
hash = ph.hash(password)
ph.verify(hash, password) # raises on mismatch
```
**API token format:**
```
reg_<random-32-bytes-base62>
Example: reg_7kX9mPqR2sT4vW6xY8zA1bC3dE5fG7hJ
```
- Prefix `reg_` for easy identification in logs/configs
- 32 bytes of cryptographically random data
- Base62 encoded (alphanumeric, no special chars)
- Total length: ~47 characters
- Stored as SHA-256 hash in database (never plain text)
**Token lifecycle:**
| Action | Behavior |
|--------|----------|
| Generate | Create new token, return once, store hash |
| List | Show token name, created date, last used (not the token itself) |
| Revoke | Set `revoked_at` timestamp, reject future uses |
| Rotate | Generate new token, optionally revoke old |
**Rate limits:**
| Endpoint | Limit | Window | Scope | Retry-After |
|----------|-------|--------|-------|-------------|
| `POST /register` | 5 | 1 hour | IP | 3600 |
| `POST /login` | 10 | 15 min | IP | 900 |
| `POST /login` (failed) | 5 | 15 min | IP + email | 900 |
| `POST /tokens` | 10 | 1 hour | Token | 3600 |
| `POST /tools` | 20 | 1 hour | Token | 3600 |
| `GET /tools/*` | 100 | 1 min | IP | 60 |
| `GET /download` | 60 | 1 min | IP | 60 |
**Rate limit response (429):**
```json
{
"error": {
"code": "RATE_LIMITED",
"message": "Too many requests. Try again in 60 seconds.",
"details": {
"limit": 100,
"window": "1 minute",
"retry_after": 60
}
}
}
```
**Headers on rate-limited response:**
```
HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1705766400
```
**Scope priority:** For authenticated requests, both IP and token limits apply. The more restrictive limit wins.
**Account lockout:**
- After 5 failed login attempts: 15-minute lockout for that email
- After 10 failed attempts: 1-hour lockout
- Lockout clears on successful password reset
**Password reset flow (deferred to v1.1):**
1. User requests reset via email
2. Server generates time-limited token (1 hour expiry)
3. Email contains reset link with token
4. User sets new password
5. All existing sessions/tokens optionally invalidated
**Email verification flow (deferred to v1.1):**
1. On registration, send verification email
2. User clicks link with verification token
3. Set `verified = true` in database
4. Unverified accounts can browse but not publish
### Token Scopes and Authorization
Tokens have scopes that limit their capabilities:
| Scope | Permissions |
|-------|-------------|
| `read` | View own published tools, download stats |
| `publish` | Submit new tools, update own tool metadata |
| `admin` | Yank tools, manage categories (registry admins only) |
**Default scope:** New tokens get `read,publish` by default.
**Ownership enforcement:**
```python
@app.route('/api/v1/tools', methods=['POST'])
@require_token(scopes=['publish'])
def publish_tool():
token = get_current_token()
tool_data = request.json
# Enforce owner == token holder's slug
if tool_data['owner'] != token.publisher.slug:
return {
"error": {
"code": "FORBIDDEN",
"message": f"Cannot publish to namespace '{tool_data['owner']}'. "
f"Your namespace is '{token.publisher.slug}'."
}
}, 403
# Proceed with publish...
```
**`GET /api/v1/me/tools` authorization:**
- Requires valid token with `read` scope
- Returns only tools where `owner == token.publisher.slug`
- Includes pending PRs and all versions (including yanked)
### Web Session Security
Dashboard login uses session cookies (not tokens) for browser auth:
**Cookie settings:**
```python
SESSION_COOKIE_NAME = 'smarttools_session'
SESSION_COOKIE_HTTPONLY = True # Prevent JS access
SESSION_COOKIE_SECURE = True # HTTPS only in production
SESSION_COOKIE_SAMESITE = 'Lax' # CSRF protection
SESSION_COOKIE_MAX_AGE = 86400 * 7 # 7 days
```
**CSRF protection:**
- All POST/PUT/DELETE forms include `csrf_token` hidden field
- Token validated server-side before processing
- 403 Forbidden if token missing or invalid
**Session lifecycle:**
| Event | Action |
|-------|--------|
| Login | Create session, set cookie |
| Logout | Delete session, clear cookie |
| Idle 24h | Session expires, re-login required |
| Password change | Invalidate all sessions |
| Token revocation | Existing sessions continue (token != session) |
**Secure session storage:**
```python
# Store sessions in DB, not filesystem
from flask_session import Session
app.config['SESSION_TYPE'] = 'sqlalchemy'
app.config['SESSION_SQLALCHEMY_TABLE'] = 'sessions'
```
**Database schema:**
```sql
-- Publishers
CREATE TABLE publishers (
id INTEGER PRIMARY KEY AUTOINCREMENT,
email TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
slug TEXT UNIQUE NOT NULL, -- immutable namespace: "rob", "alice-dev"
display_name TEXT NOT NULL, -- mutable: "Rob", "Alice Developer"
bio TEXT,
website TEXT,
verified BOOLEAN DEFAULT FALSE,
locked_until TIMESTAMP, -- account lockout
failed_login_attempts INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- API tokens (one publisher can have multiple)
CREATE TABLE api_tokens (
id INTEGER PRIMARY KEY AUTOINCREMENT,
publisher_id INTEGER NOT NULL REFERENCES publishers(id),
token_hash TEXT NOT NULL,
name TEXT NOT NULL, -- "CLI token", "CI token"
last_used_at TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
revoked_at TIMESTAMP -- NULL if active
);
-- Tools (links to publisher)
CREATE TABLE tools (
id INTEGER PRIMARY KEY AUTOINCREMENT,
owner TEXT NOT NULL, -- namespace slug (immutable, from publisher.slug)
name TEXT NOT NULL,
version TEXT NOT NULL,
description TEXT,
category TEXT,
tags TEXT, -- JSON array
config_yaml TEXT NOT NULL, -- Full tool config
readme TEXT,
publisher_id INTEGER NOT NULL REFERENCES publishers(id),
deprecated BOOLEAN DEFAULT FALSE,
deprecated_message TEXT,
replacement TEXT,
downloads INTEGER DEFAULT 0,
published_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(owner, name, version)
);
-- Download stats (for deduplication)
CREATE TABLE download_stats (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tool_id INTEGER NOT NULL REFERENCES tools(id),
client_id TEXT NOT NULL,
downloaded_at DATE NOT NULL,
UNIQUE(tool_id, client_id, downloaded_at)
);
-- Search index (FTS5)
CREATE VIRTUAL TABLE tools_fts USING fts5(
name, description, tags, readme,
content='tools',
content_rowid='id'
);
-- FTS5 sync triggers (required for external content tables)
CREATE TRIGGER tools_ai AFTER INSERT ON tools BEGIN
INSERT INTO tools_fts(rowid, name, description, tags, readme)
VALUES (new.id, new.name, new.description, new.tags, new.readme);
END;
CREATE TRIGGER tools_ad AFTER DELETE ON tools BEGIN
INSERT INTO tools_fts(tools_fts, rowid, name, description, tags, readme)
VALUES ('delete', old.id, old.name, old.description, old.tags, old.readme);
END;
CREATE TRIGGER tools_au AFTER UPDATE ON tools BEGIN
INSERT INTO tools_fts(tools_fts, rowid, name, description, tags, readme)
VALUES ('delete', old.id, old.name, old.description, old.tags, old.readme);
INSERT INTO tools_fts(rowid, name, description, tags, readme)
VALUES (new.id, new.name, new.description, new.tags, new.readme);
END;
-- Pending PRs (track publish state)
CREATE TABLE pending_prs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
publisher_id INTEGER NOT NULL REFERENCES publishers(id),
owner TEXT NOT NULL,
name TEXT NOT NULL,
version TEXT NOT NULL,
pr_number INTEGER NOT NULL,
pr_url TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending, merged, closed
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(owner, name, version)
);
-- Webhook sync log (idempotency)
CREATE TABLE webhook_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
delivery_id TEXT UNIQUE NOT NULL, -- Gitea delivery ID
event_type TEXT NOT NULL,
processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
**Note on tags indexing:** The `tags` column stores JSON arrays as text. For v1, FTS5 will search within the JSON string. If tag filtering becomes a bottleneck, normalize to a `tool_tags` junction table:
```sql
-- Future: normalized tags (if needed)
CREATE TABLE tags (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT UNIQUE NOT NULL
);
CREATE TABLE tool_tags (
tool_id INTEGER REFERENCES tools(id),
tag_id INTEGER REFERENCES tags(id),
PRIMARY KEY (tool_id, tag_id)
);
```
**CLI first-time publish flow:**
```bash
$ smarttools registry publish
No registry account configured.
1. Register at: https://gitea.brrd.tech/registry/register
2. Generate a token from your dashboard
3. Enter your token below
Registry token: ********
Token saved to ~/.smarttools/config.yaml
Validating tool...
✓ config.yaml is valid
✓ README.md exists (2.3 KB)
✓ Version 1.0.0 not yet published
Publishing rob/my-tool@1.0.0...
✓ PR created: https://gitea.brrd.tech/rob/SmartTools-Registry/pulls/42
Your tool is pending review. You'll receive an email when it's approved.
```
## CLI Commands Reference
Full mapping of CLI commands to API calls:
### Registry Commands
```bash
# Search for tools
$ smarttools registry search <query> [--category=<cat>] [--limit=20]
→ GET /api/v1/tools/search?q=<query>&category=<cat>&limit=20
# Browse tools (TUI)
$ smarttools registry browse [--category=<cat>]
→ GET /api/v1/tools?category=<cat>&page=1
→ GET /api/v1/categories
# View tool details
$ smarttools registry info <owner/name>
→ GET /api/v1/tools/<owner>/<name>
# Install a tool
$ smarttools registry install <owner/name> [--version=<ver>]
→ GET /api/v1/tools/<owner>/<name>/download?version=<ver>&install=true
→ Writes to ~/.smarttools/<owner>/<name>/config.yaml
→ Generates ~/.local/bin/<name> wrapper (or <owner>-<name> if collision)
# Uninstall a tool
$ smarttools registry uninstall <owner/name>
→ Removes ~/.smarttools/<owner>/<name>/
→ Removes wrapper script
# Publish a tool
$ smarttools registry publish [path] [--dry-run]
→ POST /api/v1/tools (with registry token)
→ Returns PR URL
# List my published tools
$ smarttools registry my-tools
→ GET /api/v1/me/tools (with registry token)
# Update index cache
$ smarttools registry update
→ GET /api/v1/index.json
→ Writes to ~/.smarttools/registry/index.json
```
### Project Commands
```bash
# Install project dependencies from smarttools.yaml
$ smarttools install
→ Reads ./smarttools.yaml
→ For each dependency:
GET /api/v1/tools/<owner>/<name>/download?version=<constraint>&install=true
→ Installs to ~/.smarttools/<owner>/<name>/
# Add a dependency to smarttools.yaml
$ smarttools add <owner/name> [--version=<constraint>]
→ Adds to ./smarttools.yaml dependencies
→ Runs install for that tool
# Show project dependencies status
$ smarttools deps
→ Reads ./smarttools.yaml
→ Shows installed status for each dependency
→ Note: "smarttools list" is reserved for listing installed tools
```
**Command naming note:** `smarttools list` already exists to list locally installed tools. Use `smarttools deps` to show project manifest dependencies.
### Flags available on most commands
| Flag | Description |
|------|-------------|
| `--offline` | Use cached index only, don't fetch |
| `--refresh` | Force refresh of cached data |
| `--json` | Output in JSON format |
| `--verbose` | Show detailed output |
## Webhooks and Security
### HMAC Verification
All Gitea webhooks are verified using HMAC-SHA256:
```python
import hmac
import hashlib
def verify_webhook(request, secret):
signature = request.headers.get('X-Gitea-Signature')
if not signature:
return False
expected = hmac.new(
secret.encode(),
request.body,
hashlib.sha256
).hexdigest()
return hmac.compare_digest(signature, expected)
```
### Replay Protection
While sync is idempotent, implement basic replay protection:
```python
def process_webhook(request):
delivery_id = request.headers.get('X-Gitea-Delivery')
# Check if already processed
if db.webhook_log.exists(delivery_id=delivery_id):
return {"status": "already_processed"}, 200
# Verify signature
if not verify_webhook(request, WEBHOOK_SECRET):
return {"error": "invalid_signature"}, 401
# Process with lock to prevent concurrent processing
with db.lock(f"webhook:{delivery_id}"):
# Double-check after acquiring lock
if db.webhook_log.exists(delivery_id=delivery_id):
return {"status": "already_processed"}, 200
# Process the webhook
result = sync_from_repo()
# Log successful processing
db.webhook_log.insert(
delivery_id=delivery_id,
event_type=request.json.get('action'),
processed_at=datetime.utcnow()
)
return {"status": "processed"}, 200
```
### Sync Job Locking
Prevent concurrent sync operations:
```python
# Using file lock or database advisory lock
SYNC_LOCK_TIMEOUT = 300 # 5 minutes max
def sync_from_repo():
try:
with acquire_lock("registry_sync", timeout=SYNC_LOCK_TIMEOUT):
# Pull latest from Gitea
repo.fetch()
repo.reset('origin/main', hard=True)
# Parse and update database
for tool_path in glob('tools/*/*/config.yaml'):
update_tool_in_db(tool_path)
# Rebuild FTS index if needed
rebuild_fts_index()
except LockTimeout:
logger.warning("Sync already in progress, skipping")
return {"status": "skipped", "reason": "sync_in_progress"}
```
### Atomic Sync Strategy
To avoid partially updated DB during webhook sync, use transactional table swap:
```python
def sync_from_repo_atomic():
with acquire_lock("registry_sync", timeout=SYNC_LOCK_TIMEOUT):
# 1. Pull latest from Gitea
repo.fetch()
repo.reset('origin/main', hard=True)
# 2. Parse all tools into memory
new_tools = []
for tool_path in glob('tools/*/*/config.yaml'):
tool_data = parse_tool(tool_path)
if tool_data:
new_tools.append(tool_data)
# 3. Atomic swap using transaction
with db.transaction():
# Create temp table
db.execute("CREATE TABLE tools_new AS SELECT * FROM tools WHERE 0")
# Bulk insert into temp table
for tool in new_tools:
db.execute("INSERT INTO tools_new ...", tool)
# Swap tables atomically
db.execute("ALTER TABLE tools RENAME TO tools_old")
db.execute("ALTER TABLE tools_new RENAME TO tools")
db.execute("DROP TABLE tools_old")
# Rebuild FTS index
db.execute("INSERT INTO tools_fts(tools_fts) VALUES('rebuild')")
# Update sync timestamp
db.execute("UPDATE sync_status SET last_sync = ?", [datetime.utcnow()])
```
**Why atomic:** Per-row updates with FTS triggers can yield inconsistent reads under load. Readers may see partial state mid-sync. Table swap ensures all-or-nothing visibility.
### Error Handling
| Error Scenario | Behavior |
|----------------|----------|
| Repo fetch fails | Log error, retry in 5 min, alert if 3 failures |
| YAML parse error | Skip tool, log error, continue with others |
| Database write fails | Rollback transaction, retry once, then alert |
| Lock timeout | Skip this sync, next webhook will retry |
## Automated CI Validation
PRs are validated automatically using SmartTools (dogfooding):
```
PR Submitted
│
▼
┌─────────────────────────────────────┐
│ Gitea CI runs validation tools: │
│ • schema-validator │
│ • security-scanner │
│ • duplicate-detector │
└───────────────┬─────────────────────┘
│
┌───────┴───────┐
│ │
All pass Any fail
│ │
▼ ▼
Auto-merge or Add comment,
flag for review request changes
```
Validation checks:
1. **Schema validation**: config.yaml matches expected format
2. **Security scan**: No dangerous shell commands, no secrets in prompts
3. **Duplicate detection**: AI-powered similarity check against existing tools
4. **README check**: README.md exists and is non-empty
CI workflow (`.gitea/workflows/validate.yaml`):
```yaml
name: Validate Tool Submission
on: [pull_request]
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Validate schema
run: python scripts/validate_tool.py ${{ github.event.pull_request.head.sha }}
- name: Security scan
run: smarttools run security-scanner < changed_files.txt
- name: Check duplicates
run: smarttools run duplicate-detector < changed_files.txt
```
## Registry Repository Structure
Full structure of the SmartTools-Registry repo:
```
SmartTools-Registry/
├── README.md # Registry overview
├── CONTRIBUTING.md # How to submit tools
├── LICENSE
│
├── tools/ # All published tools
│ ├── rob/
│ │ ├── summarize/
│ │ │ ├── config.yaml
│ │ │ └── README.md
│ │ └── translate/
│ │ ├── config.yaml
│ │ └── README.md
│ └── alice/
│ └── code-review/
│ ├── config.yaml
│ └── README.md
│
├── categories/
│ └── categories.yaml # Category definitions
│
├── index.json # Auto-generated search index
│
├── .gitea/
│ └── workflows/
│ ├── validate.yaml # PR validation
│ ├── build-index.yaml # Rebuild index on merge
│ └── notify-api.yaml # Webhook to API server
│
└── scripts/
├── validate_tool.py # Schema validation
├── build_index.py # Generate index.json
├── check_duplicates.py # Similarity detection
└── security_scan.py # Security checks
```
`categories.yaml` format:
```yaml
categories:
- name: text-processing
description: Tools for manipulating and analyzing text
icon: 📝
- name: code
description: Tools for code review, generation, and analysis
icon: 💻
- name: data
description: Tools for data transformation and analysis
icon: 📊
- name: media
description: Tools for image, audio, and video processing
icon: 🎨
- name: productivity
description: General productivity and automation tools
icon: ⚡
```
## Download Stats
### Counting Methodology
- Count installs only, not views or searches
- Increment **after** successful download (response sent)
- Dedupe by `client_id + tool_id + date`
```python
def download_tool(owner, name, version, install=False, client_id=None):
tool = get_tool(owner, name, version)
if not tool:
return {"error": "not_found"}, 404
config_yaml = tool.config_yaml
# Only count if this is an install (not just viewing)
if install:
record_download(tool.id, client_id)
return {"config": config_yaml}, 200
def record_download(tool_id, client_id):
today = date.today()
# Use client_id if provided, otherwise generate anonymous fallback
effective_client_id = client_id or f"anon_{hash(request.remote_addr)}"
# Dedupe: only count once per client per tool per day
try:
db.download_stats.insert(
tool_id=tool_id,
client_id=effective_client_id,
downloaded_at=today
)
# Increment counter (can be async/batch updated)
db.execute("UPDATE tools SET downloads = downloads + 1 WHERE id = ?", [tool_id])
except IntegrityError:
pass # Already counted today, ignore
```
### Client ID Generation
CLI generates a persistent anonymous ID on first run:
```python
# In CLI, on first run
import uuid
import os
CONFIG_PATH = os.path.expanduser("~/.smarttools/config.yaml")
def get_or_create_client_id():
config = load_config()
if 'client_id' not in config:
config['client_id'] = f"anon_{uuid.uuid4().hex[:16]}"
save_config(config)
return config['client_id']
```
**Fallback when client_id missing:**
- If header `X-Client-ID` not sent, use IP hash as fallback
- This still provides some dedupe for anonymous users
- Logged users' downloads are attributed to their account instead
### Privacy Considerations
- No IP addresses stored in database
- `client_id` is client-controlled and can be regenerated
- Stats are aggregated (total count), not individual tracking
### Async Stats Strategy
To avoid DB contention on the hot download path:
```python
from queue import Queue
from threading import Thread
# In-memory queue for stats
stats_queue = Queue()
def record_download_async(tool_id, client_id):
"""Non-blocking: enqueue for background processing"""
stats_queue.put({
'tool_id': tool_id,
'client_id': client_id,
'date': date.today()
})
def stats_worker():
"""Background thread: batch process stats every 5 seconds"""
batch = []
while True:
try:
item = stats_queue.get(timeout=5)
batch.append(item)
except Empty:
if batch:
flush_batch(batch)
batch = []
def flush_batch(batch):
"""Bulk insert with conflict ignore"""
with db.transaction():
for item in batch:
try:
db.execute("""
INSERT INTO download_stats (tool_id, client_id, downloaded_at)
VALUES (?, ?, ?)
ON CONFLICT DO NOTHING
""", [item['tool_id'], item['client_id'], item['date']])
except Exception as e:
logger.warning(f"Stats insert failed: {e}")
# Don't fail downloads for stats errors
```
**Failure behavior:** If stats DB write fails, log the error but don't fail the download. Stats are "best effort" - the download must succeed.
## Search
- Primary search: SQLite FTS5 inside the API.
- `index.json` provides offline CLI search and backup.
- If FTS5 is stale, return results with `X-Search-Index-Stale: true`.
## API Caching Strategy
### Cache Headers
| Endpoint | Cache-Control | ETag | Notes |
|----------|---------------|------|-------|
| `GET /index.json` | `max-age=300, stale-while-revalidate=60` | Yes | 5 min cache, background refresh |
| `GET /tools/{owner}/{name}` | `max-age=60` | Yes | 1 min cache |
| `GET /tools/{owner}/{name}/download` | `max-age=3600, immutable` | Yes | Immutable versions, 1 hour |
| `GET /tools/search` | `no-cache` | No | Always fresh |
| `GET /categories` | `max-age=3600` | Yes | Categories change rarely |
### ETag Implementation
```python
import hashlib
from datetime import datetime
def get_tool_etag(tool):
"""Generate ETag from tool identity (immutable versions don't change)"""
# Since versions are immutable, owner/name@version is stable
# Use published_at for extra safety (not updated_at, which doesn't exist)
content = f"{tool.owner}/{tool.name}@{tool.version}:{tool.published_at.isoformat()}"
return hashlib.md5(content.encode()).hexdigest()
def get_index_etag():
"""Generate ETag from last sync timestamp"""
last_sync = db.get_last_sync_time()
return hashlib.md5(last_sync.isoformat().encode()).hexdigest()
@app.route('/api/v1/tools/<owner>/<name>/download')
def download_tool(owner, name):
version = request.args.get('version', 'latest')
tool = resolve_and_get_tool(owner, name, version)
etag = get_tool_etag(tool)
# Check If-None-Match header
if request.headers.get('If-None-Match') == etag:
return '', 304 # Not Modified
response = jsonify({
"data": {
"owner": tool.owner,
"name": tool.name,
"resolved_version": tool.version,
"config": tool.config_yaml
}
})
response.headers['ETag'] = etag
response.headers['Cache-Control'] = 'max-age=3600, immutable'
return response
```
**Note:** Since tool versions are immutable, the ETag based on `owner/name@version` is permanently stable. The `published_at` timestamp is included for defense-in-depth but won't change.
### DB vs Repo Read Strategy
| Scenario | Read From | Reason |
|----------|-----------|--------|
| Normal operation | SQLite DB | Fast, indexed, FTS |
| DB empty/corrupted | Gitea repo | Fallback/recovery |
| Webhook sync in progress | DB (stale OK) | Avoid blocking reads |
| Search query | SQLite FTS5 | Full-text search |
| Download specific version | DB, fallback to repo | DB is cache, repo is truth |
### Staleness Detection
```python
STALE_THRESHOLD = timedelta(minutes=10)
def is_db_stale():
last_sync = db.get_last_sync_time()
return datetime.utcnow() - last_sync > STALE_THRESHOLD
@app.route('/tools/search')
def search_tools(q):
results = db.search_fts(q)
response = jsonify({"results": results})
if is_db_stale():
response.headers['X-Search-Index-Stale'] = 'true'
response.headers['X-Last-Sync'] = db.get_last_sync_time().isoformat()
return response
```
## Error Model
### Response Envelopes
**Success response:**
```json
{
"data": { ... },
"meta": {
"page": 1,
"per_page": 20,
"total": 42,
"total_pages": 3
}
}
```
**Error response:**
```json
{
"error": {
"code": "TOOL_NOT_FOUND",
"message": "Tool 'foo/bar' does not exist",
"details": {
"owner": "foo",
"name": "bar",
"suggestion": "Did you mean 'rob/bar'?"
},
"docs_url": "https://registry.smarttools.dev/docs/errors#TOOL_NOT_FOUND"
}
}
```
### Error Codes
| Code | HTTP | Description |
|------|------|-------------|
| `TOOL_NOT_FOUND` | 404 | Tool does not exist |
| `VERSION_NOT_FOUND` | 404 | Requested version doesn't exist |
| `VERSION_EXISTS` | 409 | Cannot overwrite published version |
| `INVALID_VERSION` | 400 | Version string is not valid semver |
| `INVALID_CONSTRAINT` | 400 | Version constraint syntax error |
| `CONSTRAINT_UNSATISFIABLE` | 404 | No version matches constraint |
| `VALIDATION_ERROR` | 400 | Tool config validation failed |
| `UNAUTHORIZED` | 401 | Missing or invalid auth token |
| `FORBIDDEN` | 403 | Token valid but lacks permission |
| `RATE_LIMITED` | 429 | Too many requests |
| `SLUG_TAKEN` | 409 | Namespace slug already registered |
| `ACCOUNT_LOCKED` | 403 | Too many failed login attempts |
| `SERVER_ERROR` | 500 | Internal error (logged for debugging) |
## Error Scenarios and Fallbacks
### CLI Error Handling
| Scenario | CLI Behavior | User Message |
|----------|--------------|--------------|
| Registry offline | Use cached tools if available | "Registry unavailable. Using cached version." |
| Tool not found | Check cache, then fail | "Tool 'foo/bar' not found in registry or cache." |
| Version constraint unsatisfiable | Show available versions | "No version matches '>=5.0.0'. Available: 1.0.0, 1.1.0, 1.2.0" |
| Auth token expired | Prompt for new token | "Token expired. Please re-authenticate." |
| Rate limited | Wait and retry (backoff) | "Rate limited. Retrying in 30 seconds..." |
| Network timeout | Retry with backoff, then fail | "Connection timed out. Check your network." |
### Validation Failure Details
When `VALIDATION_ERROR` occurs, provide specific field errors:
```json
{
"error": {
"code": "VALIDATION_ERROR",
"message": "Tool configuration is invalid",
"details": {
"errors": [
{
"path": "steps[0].provider",
"message": "Provider 'gpt5' is not recognized",
"allowed": ["claude", "openai", "ollama", "mock"]
},
{
"path": "version",
"message": "Version '1.0' is not valid semver (use '1.0.0')"
}
]
},
"docs_url": "https://registry.smarttools.dev/docs/tool-format"
}
}
```
### Dependency Resolution Failures
When `smarttools install` fails on a manifest:
```bash
$ smarttools install
Error: Could not resolve all dependencies
rob/summarize@^2.0.0
✗ No matching version (latest: 1.2.0)
alice/translate@>=1.0.0
✓ Found 1.3.0
Suggestions:
- Update rob/summarize constraint to "^1.0.0"
- Contact the tool author for a v2 release
```
### Graceful Degradation
| Component Down | Fallback Behavior |
|----------------|-------------------|
| API server | CLI uses `~/.smarttools/registry/index.json` for search |
| Gitea repo | API serves from DB cache (may be stale) |
| FTS5 index | Fall back to LIKE queries (slower but works) |
| Network | Use locally installed tools, skip registry features |
## UX Requirements (CLI/TUI)
### Publishing UX
- `smarttools registry publish --dry-run` validates locally and shows what would be submitted:
```bash
$ smarttools registry publish --dry-run
Validating tool...
✓ config.yaml is valid
✓ README.md exists (2.3 KB)
✓ Version 1.1.0 not yet published
Would submit:
Owner: rob
Name: summarize
Version: 1.1.0
Category: text-processing
Tags: summarization, ai, text
Config preview:
─────────────────────────────
name: summarize
version: "1.1.0"
description: Summarize text using AI
...
─────────────────────────────
Run without --dry-run to submit for review.
```
- **Version bump reminder:** CLI warns if version hasn't changed from published:
```
⚠ Version 1.0.0 is already published. Bump version in config.yaml to publish changes.
```
- First-time publishing flow prompts for token and saves it to config.
### Progress Indicators
Long-running operations show progress:
```bash
$ smarttools install
Installing project dependencies...
[1/3] rob/summarize@^1.0.0
Resolving version... 1.2.0
Downloading... done
Installing... done ✓
[2/3] alice/translate@>=2.0.0
Resolving version... 2.1.0
Downloading... done
Installing... done ✓
[3/3] official/code-review@*
Resolving version... 1.0.0
Downloading... done
Installing... done ✓
✓ Installed 3 tools
```
```bash
$ smarttools registry publish
Submitting rob/summarize@1.1.0...
Validating... done ✓
Uploading... done ✓
Creating PR... done ✓
✓ PR created: https://gitea.brrd.tech/rob/SmartTools-Registry/pulls/42
Your tool is pending review. You'll receive an email when it's approved.
```
### TUI Browse
`smarttools registry browse` opens a full-screen terminal UI:
```
┌─ SmartTools Registry ───────────────────────────────────────┐
│ Search: [________________] [All Categories ▼] [Sort: Popular ▼] │
├─────────────────────────────────────────────────────────────┤
│ │
│ ▶ rob/summarize v1.2.0 ⬇ 142 │
│ Summarize text using AI │
│ [text-processing] [ai] [summarization] │
│ │
│ alice/translate v2.1.0 ⬇ 98 │
│ Translate text between languages │
│ [text-processing] [translation] │
│ │
│ official/code-review v1.0.0 ⬇ 87 │
│ AI-powered code review │
│ [code] [review] [ai] │
│ │
├─────────────────────────────────────────────────────────────┤
│ ↑↓ Navigate Enter: Details i: Install /: Search q: Quit │
└─────────────────────────────────────────────────────────────┘
```
**Keyboard shortcuts:**
| Key | Action |
|-----|--------|
| `↑/↓` or `j/k` | Navigate list |
| `Enter` | View tool details |
| `i` | Install selected tool |
| `/` | Focus search box |
| `c` | Change category filter |
| `s` | Change sort order |
| `?` | Show help |
| `q` | Quit |
**Virtual scrolling:** For large tool lists (>100), use virtual scrolling to maintain performance.
### Project Initialization
```bash
$ smarttools init
Creating smarttools.yaml...
Project name [my-project]: my-ai-project
Version [1.0.0]:
Would you like to add any tools? (search with 's', skip with Enter)
> s
Search: summ
1. rob/summarize v1.2.0 - Summarize text using AI
2. alice/summary v1.0.0 - Generate summaries
Add tool (number, or Enter to finish): 1
Added rob/summarize@^1.2.0
Add tool (number, or Enter to finish):
✓ Created smarttools.yaml
name: my-ai-project
version: "1.0.0"
dependencies:
- name: rob/summarize
version: "^1.2.0"
Run 'smarttools install' to install dependencies.
```
### Accessibility
- **CLI:** All output works with screen readers, no color-only information
- **TUI:** Full keyboard navigation, high-contrast mode support
- **Web UI:** WCAG 2.1 AA compliance target
- Semantic HTML
- ARIA labels for interactive elements
- Focus management in modals
- Skip links for navigation
## Offline Cache
Cache registry index locally:
```
~/.smarttools/registry/index.json
```
Refresh when older than 24 hours; support `--offline` and `--refresh` flags.
### Index Integrity
The cached `index.json` includes integrity metadata:
```json
{
"version": "1.0",
"generated_at": "2025-01-20T12:00:00Z",
"checksum": "sha256:abc123...",
"tool_count": 142,
"tools": [...]
}
```
**API response headers:**
```
ETag: "abc123def456"
X-Index-Checksum: sha256:abc123...
X-Index-Generated: 2025-01-20T12:00:00Z
```
**CLI verification:**
```python
def verify_cached_index():
"""Verify cached index integrity on load"""
cached = load_cached_index()
if not cached:
return None
# Verify checksum
content = json.dumps(cached['tools'], sort_keys=True)
computed = hashlib.sha256(content.encode()).hexdigest()
if computed != cached.get('checksum', '').replace('sha256:', ''):
logger.warning("Cached index checksum mismatch, will refresh")
return None
return cached
```
**Corruption handling:**
- If checksum fails, discard cache and fetch fresh
- If partial write detected (missing fields), discard and refresh
- CLI shows warning: "Cached index corrupted, fetching fresh copy..."
## Web UI Vision
The registry includes a full website, not just an API:
**Site structure:**
```
registry.smarttools.dev (or gitea.brrd.tech/registry)
├── / # Landing page
├── /tools # Browse all tools
├── /tools/{owner}/{name} # Tool detail page
├── /categories # Browse by category
├── /categories/{name} # Tools in category
├── /search?q=... # Search results
├── /docs # Documentation
│ ├── /docs/getting-started
│ ├── /docs/creating-tools
│ ├── /docs/publishing
│ └── /docs/best-practices
├── /tutorials # Step-by-step guides
│ ├── /tutorials/first-tool
│ ├── /tutorials/chaining-steps
│ └── /tutorials/code-steps
├── /examples # Example projects
├── /blog # Updates, announcements (optional)
├── /register # Publisher registration
├── /login # Publisher login
├── /dashboard # Publisher dashboard
│ ├── /dashboard/tools # My published tools
│ ├── /dashboard/tokens # API tokens
│ └── /dashboard/settings # Account settings
└── /api/v1/... # API endpoints
```
**Landing page content:**
- Hero: "Share and discover AI-powered CLI tools"
- Quick install example
- Featured/popular tools
- Category highlights
- "Get Started" CTA
**Tool detail page:**
- Name, description, version, author
- README rendered as markdown (sanitized)
- Install command (copy-to-clipboard)
- Version history
- Download stats
- Category/tags
- "Report" button for abuse
### README Security
When rendering README markdown, apply XSS sanitization:
```python
import bleach
from markdown import markdown
ALLOWED_TAGS = [
'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
'p', 'br', 'hr',
'ul', 'ol', 'li',
'strong', 'em', 'code', 'pre',
'blockquote',
'a', 'img',
'table', 'thead', 'tbody', 'tr', 'th', 'td'
]
ALLOWED_ATTRS = {
'a': ['href', 'title'],
'img': ['src', 'alt', 'title'],
'code': ['class'], # for syntax highlighting
}
def render_readme_safe(readme_raw: str) -> str:
"""Convert markdown to sanitized HTML"""
# Convert markdown to HTML
html = markdown(readme_raw, extensions=['fenced_code', 'tables'])
# Sanitize to prevent XSS
safe_html = bleach.clean(
html,
tags=ALLOWED_TAGS,
attributes=ALLOWED_ATTRS,
strip=True
)
# Linkify URLs
safe_html = bleach.linkify(safe_html)
return safe_html
```
**Storage strategy:**
- Store raw README in `tools.readme`
- Render and sanitize on request (or cache rendered HTML)
- Never trust client-submitted HTML directly
**Tech stack options:**
| Option | Pros | Cons |
|--------|------|------|
| Flask + Jinja + Tailwind | Simple, Python-only, fast to build | Less interactive |
| FastAPI + Vue/React SPA | Modern, interactive | More complex, separate build |
| Astro/Next.js | Great SEO, static-first | Different stack (Node.js) |
**Recommendation:** Flask + Jinja + Tailwind for v1
- Keeps everything in Python
- Server-rendered is fine for a registry
- Good SEO out of the box
- Can add interactivity with Alpine.js or htmx if needed
**Monetization considerations:**
- AdSense-compatible (server-rendered pages)
- Analytics tracking for traffic insights
- Future: sponsored tools, featured placements
- Future: premium publisher tiers (more tools, priority review)
## Implementation Phases
### Phase 1: Foundation
- Define `smarttools.yaml` manifest format
- Implement tool resolution order (local → global → registry)
- Create SmartTools-Registry repo on Gitea (bootstrap)
- Add 3-5 example tools to seed the registry
### Phase 2: Core Backend
- Set up Flask/FastAPI project structure
- Implement SQLite database schema
- Build core API endpoints (list, search, get, download)
- Implement webhook receiver for Gitea sync
- Set up HMAC verification
### Phase 3: CLI Commands
- `smarttools registry search`
- `smarttools registry install`
- `smarttools registry info`
- `smarttools registry browse` (TUI)
- Local index caching
### Phase 4: Publishing
- Publisher registration (web UI)
- Token management
- `smarttools registry publish` command
- PR creation via Gitea API
- CI validation workflows
### Phase 5: Project Dependencies
- `smarttools install` (from manifest)
- `smarttools add` command
- Runtime override application
- Dependency resolution
### Phase 6: Smart Features
- SQLite FTS5 search index
- AI-powered auto-categorization
- Duplicate/similarity detection
- Security scanning
### Phase 7: Full Web UI
- Landing page
- Tool browsing/search pages
- Tool detail pages with README rendering
- Publisher dashboard
- Documentation/tutorials section
### Phase 8: Polish & Scale
- Rate limiting
- Abuse reporting
- Analytics integration
- Performance optimization
- Monitoring/alerting
Reference in New Issue View Git Blame Copy Permalink