Add SEO files and real documentation content
- Add permissive robots.txt welcoming all crawlers including AI - Add dynamic sitemap.xml with all pages and tools - Add real documentation content (Getting Started, Installation, First Tool, Publishing, Providers) - Update docs route to use content from docs_content.py 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
1ac3054aaa
commit
4318b6cec5
|
|
@ -0,0 +1,400 @@
|
||||||
|
"""Documentation content for SmartTools web UI.
|
||||||
|
|
||||||
|
This module contains the actual documentation text that gets rendered
|
||||||
|
on the /docs pages. Content is stored as markdown-ish HTML for simplicity.
|
||||||
|
"""
|
||||||
|
|
||||||
|
DOCS = {
|
||||||
|
"getting-started": {
|
||||||
|
"title": "Getting Started",
|
||||||
|
"description": "Learn how to install SmartTools and create your first AI-powered CLI tool",
|
||||||
|
"content": """
|
||||||
|
<p class="lead">SmartTools lets you build custom AI-powered CLI commands using simple YAML configuration.
|
||||||
|
Create tools that work with any AI provider and compose them like Unix pipes.</p>
|
||||||
|
|
||||||
|
<h2 id="what-is-smarttools">What is SmartTools?</h2>
|
||||||
|
<p>SmartTools is a lightweight personal tool builder that lets you:</p>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Create custom CLI commands</strong> that call AI providers</li>
|
||||||
|
<li><strong>Chain prompts with Python code</strong> for complex workflows</li>
|
||||||
|
<li><strong>Use tools like Unix pipes</strong> - read from stdin, write to stdout</li>
|
||||||
|
<li><strong>Share and discover tools</strong> through the registry</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="quick-start">Quick Start</h2>
|
||||||
|
<p>Get up and running in under a minute:</p>
|
||||||
|
|
||||||
|
<pre><code class="language-bash"># Install SmartTools
|
||||||
|
pip install smarttools
|
||||||
|
|
||||||
|
# Create your first tool interactively
|
||||||
|
smarttools create
|
||||||
|
|
||||||
|
# Or install a tool from the registry
|
||||||
|
smarttools registry install official/summarize
|
||||||
|
|
||||||
|
# Use it!
|
||||||
|
cat article.txt | summarize</code></pre>
|
||||||
|
|
||||||
|
<h2 id="how-it-works">How It Works</h2>
|
||||||
|
<p>Each tool is a YAML file that defines:</p>
|
||||||
|
<ol>
|
||||||
|
<li><strong>Arguments</strong> - Custom flags your tool accepts</li>
|
||||||
|
<li><strong>Steps</strong> - Prompts to send to AI or Python code to run</li>
|
||||||
|
<li><strong>Output</strong> - How to format the final result</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<p>Here's a simple example:</p>
|
||||||
|
<pre><code class="language-yaml">name: summarize
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Summarize text using AI
|
||||||
|
|
||||||
|
arguments:
|
||||||
|
- flag: --max-length
|
||||||
|
variable: max_length
|
||||||
|
default: "200"
|
||||||
|
description: Maximum summary length in words
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- type: prompt
|
||||||
|
provider: claude
|
||||||
|
prompt: |
|
||||||
|
Summarize the following text in {max_length} words or less:
|
||||||
|
|
||||||
|
{input}
|
||||||
|
output_var: summary
|
||||||
|
|
||||||
|
output: "{summary}"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="next-steps">Next Steps</h2>
|
||||||
|
<ul>
|
||||||
|
<li><a href="/docs/installation">Installation Guide</a> - Detailed setup instructions</li>
|
||||||
|
<li><a href="/docs/first-tool">Your First Tool</a> - Step-by-step tutorial</li>
|
||||||
|
<li><a href="/docs/providers">Providers</a> - Configure AI providers</li>
|
||||||
|
<li><a href="/tools">Browse Tools</a> - Discover community tools</li>
|
||||||
|
</ul>
|
||||||
|
""",
|
||||||
|
"headings": [
|
||||||
|
("what-is-smarttools", "What is SmartTools?"),
|
||||||
|
("quick-start", "Quick Start"),
|
||||||
|
("how-it-works", "How It Works"),
|
||||||
|
("next-steps", "Next Steps"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
"installation": {
|
||||||
|
"title": "Installation",
|
||||||
|
"description": "How to install SmartTools on your system",
|
||||||
|
"parent": "getting-started",
|
||||||
|
"content": """
|
||||||
|
<p class="lead">SmartTools requires Python 3.8+ and works on Linux, macOS, and Windows.</p>
|
||||||
|
|
||||||
|
<h2 id="pip-install">Install with pip</h2>
|
||||||
|
<p>The simplest way to install SmartTools:</p>
|
||||||
|
<pre><code class="language-bash">pip install smarttools</code></pre>
|
||||||
|
|
||||||
|
<p>Or with pipx for isolated installation:</p>
|
||||||
|
<pre><code class="language-bash">pipx install smarttools</code></pre>
|
||||||
|
|
||||||
|
<h2 id="verify">Verify Installation</h2>
|
||||||
|
<pre><code class="language-bash">smarttools --version
|
||||||
|
smarttools --help</code></pre>
|
||||||
|
|
||||||
|
<h2 id="configure-provider">Configure a Provider</h2>
|
||||||
|
<p>SmartTools needs at least one AI provider configured. The easiest is Claude CLI:</p>
|
||||||
|
|
||||||
|
<pre><code class="language-bash"># Install Claude CLI (if you have an Anthropic API key)
|
||||||
|
pip install claude-cli
|
||||||
|
|
||||||
|
# Or use OpenAI
|
||||||
|
pip install openai
|
||||||
|
|
||||||
|
# Configure your provider
|
||||||
|
smarttools config</code></pre>
|
||||||
|
|
||||||
|
<h2 id="wrapper-scripts">Wrapper Scripts Location</h2>
|
||||||
|
<p>SmartTools installs wrapper scripts to <code>~/.local/bin/</code>. Make sure this is in your PATH:</p>
|
||||||
|
<pre><code class="language-bash"># Add to ~/.bashrc or ~/.zshrc
|
||||||
|
export PATH="$HOME/.local/bin:$PATH"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="development-install">Development Installation</h2>
|
||||||
|
<p>To contribute or modify SmartTools:</p>
|
||||||
|
<pre><code class="language-bash">git clone https://gitea.brrd.tech/rob/SmartTools.git
|
||||||
|
cd SmartTools
|
||||||
|
pip install -e ".[dev]"</code></pre>
|
||||||
|
""",
|
||||||
|
"headings": [
|
||||||
|
("pip-install", "Install with pip"),
|
||||||
|
("verify", "Verify Installation"),
|
||||||
|
("configure-provider", "Configure a Provider"),
|
||||||
|
("wrapper-scripts", "Wrapper Scripts Location"),
|
||||||
|
("development-install", "Development Installation"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
"first-tool": {
|
||||||
|
"title": "Your First Tool",
|
||||||
|
"description": "Create your first SmartTools command step by step",
|
||||||
|
"parent": "getting-started",
|
||||||
|
"content": """
|
||||||
|
<p class="lead">Let's create a simple tool that explains code. You'll learn the basics of tool configuration.</p>
|
||||||
|
|
||||||
|
<h2 id="create-tool">Create the Tool</h2>
|
||||||
|
<p>Run the interactive creator:</p>
|
||||||
|
<pre><code class="language-bash">smarttools create</code></pre>
|
||||||
|
|
||||||
|
<p>Or create the file manually at <code>~/.smarttools/explain/config.yaml</code>:</p>
|
||||||
|
<pre><code class="language-yaml">name: explain
|
||||||
|
version: "1.0.0"
|
||||||
|
description: Explain code or concepts in simple terms
|
||||||
|
category: code
|
||||||
|
|
||||||
|
arguments:
|
||||||
|
- flag: --level
|
||||||
|
variable: level
|
||||||
|
default: "beginner"
|
||||||
|
description: "Explanation level: beginner, intermediate, or expert"
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- type: prompt
|
||||||
|
provider: claude
|
||||||
|
prompt: |
|
||||||
|
Explain the following in simple terms suitable for a {level}:
|
||||||
|
|
||||||
|
{input}
|
||||||
|
|
||||||
|
Be concise but thorough. Use examples where helpful.
|
||||||
|
output_var: explanation
|
||||||
|
|
||||||
|
output: "{explanation}"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="test-it">Test Your Tool</h2>
|
||||||
|
<pre><code class="language-bash"># Explain some code
|
||||||
|
echo "def fib(n): return n if n < 2 else fib(n-1) + fib(n-2)" | explain
|
||||||
|
|
||||||
|
# Explain for an expert
|
||||||
|
cat complex_algorithm.py | explain --level expert</code></pre>
|
||||||
|
|
||||||
|
<h2 id="understanding-config">Understanding the Config</h2>
|
||||||
|
|
||||||
|
<h3>Arguments</h3>
|
||||||
|
<p>Each argument becomes a CLI flag. The <code>variable</code> name is used in templates:</p>
|
||||||
|
<pre><code class="language-yaml">arguments:
|
||||||
|
- flag: --level # CLI flag: --level beginner
|
||||||
|
variable: level # Use as {level} in prompts
|
||||||
|
default: "beginner" # Default if not specified</code></pre>
|
||||||
|
|
||||||
|
<h3>Steps</h3>
|
||||||
|
<p>Steps run in order. Each step can be a prompt or Python code:</p>
|
||||||
|
<pre><code class="language-yaml">steps:
|
||||||
|
- type: prompt
|
||||||
|
provider: claude # Which AI to use
|
||||||
|
prompt: "..." # The prompt template
|
||||||
|
output_var: result # Store response in {result}</code></pre>
|
||||||
|
|
||||||
|
<h3>Output</h3>
|
||||||
|
<p>The output template formats the final result:</p>
|
||||||
|
<pre><code class="language-yaml">output: "{explanation}" # Print the explanation variable</code></pre>
|
||||||
|
|
||||||
|
<h2 id="next">Next Steps</h2>
|
||||||
|
<ul>
|
||||||
|
<li><a href="/docs/publishing">Publish your tool</a> to the registry</li>
|
||||||
|
<li>Learn about <a href="/docs/providers">different providers</a></li>
|
||||||
|
<li>See <a href="/tools">example tools</a> for inspiration</li>
|
||||||
|
</ul>
|
||||||
|
""",
|
||||||
|
"headings": [
|
||||||
|
("create-tool", "Create the Tool"),
|
||||||
|
("test-it", "Test Your Tool"),
|
||||||
|
("understanding-config", "Understanding the Config"),
|
||||||
|
("next", "Next Steps"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
"publishing": {
|
||||||
|
"title": "Publishing Tools",
|
||||||
|
"description": "Share your tools with the SmartTools community",
|
||||||
|
"content": """
|
||||||
|
<p class="lead">Share your tools with the community by publishing to the SmartTools Registry.</p>
|
||||||
|
|
||||||
|
<h2 id="before-publishing">Before Publishing</h2>
|
||||||
|
<p>Make sure your tool has:</p>
|
||||||
|
<ul>
|
||||||
|
<li>A descriptive <code>name</code> and <code>description</code></li>
|
||||||
|
<li>A proper <code>version</code> (semver format: 1.0.0)</li>
|
||||||
|
<li>A <code>README.md</code> file with usage examples</li>
|
||||||
|
<li>Tested and working functionality</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="create-account">Create an Account</h2>
|
||||||
|
<p>Register at <a href="/register">the registry</a> to get your publisher namespace.</p>
|
||||||
|
|
||||||
|
<h2 id="get-token">Get an API Token</h2>
|
||||||
|
<ol>
|
||||||
|
<li>Go to your <a href="/dashboard/tokens">Dashboard → Tokens</a></li>
|
||||||
|
<li>Click "Create New Token"</li>
|
||||||
|
<li>Copy the token (shown only once!)</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<h2 id="publish">Publish Your Tool</h2>
|
||||||
|
<pre><code class="language-bash"># Navigate to your tool directory
|
||||||
|
cd ~/.smarttools/my-tool/
|
||||||
|
|
||||||
|
# First time: enter your token when prompted
|
||||||
|
smarttools registry publish
|
||||||
|
|
||||||
|
# Dry run to validate without publishing
|
||||||
|
smarttools registry publish --dry-run</code></pre>
|
||||||
|
|
||||||
|
<h2 id="versioning">Versioning</h2>
|
||||||
|
<p>Published versions are <strong>immutable</strong>. To update a tool:</p>
|
||||||
|
<ol>
|
||||||
|
<li>Make your changes</li>
|
||||||
|
<li>Bump the version in <code>config.yaml</code></li>
|
||||||
|
<li>Run <code>smarttools registry publish</code></li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<h2 id="best-practices">Best Practices</h2>
|
||||||
|
<ul>
|
||||||
|
<li><strong>Clear names</strong> - Use descriptive, lowercase names with hyphens</li>
|
||||||
|
<li><strong>Good descriptions</strong> - Explain what the tool does in one sentence</li>
|
||||||
|
<li><strong>Document arguments</strong> - Describe each flag in the help text</li>
|
||||||
|
<li><strong>Include examples</strong> - Show real usage in your README</li>
|
||||||
|
<li><strong>Choose the right category</strong> - Help users discover your tool</li>
|
||||||
|
</ul>
|
||||||
|
""",
|
||||||
|
"headings": [
|
||||||
|
("before-publishing", "Before Publishing"),
|
||||||
|
("create-account", "Create an Account"),
|
||||||
|
("get-token", "Get an API Token"),
|
||||||
|
("publish", "Publish Your Tool"),
|
||||||
|
("versioning", "Versioning"),
|
||||||
|
("best-practices", "Best Practices"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
|
||||||
|
"providers": {
|
||||||
|
"title": "AI Providers",
|
||||||
|
"description": "Configure different AI providers for your tools",
|
||||||
|
"content": """
|
||||||
|
<p class="lead">SmartTools works with any AI provider that has a CLI interface. Configure providers in
|
||||||
|
<code>~/.smarttools/providers.yaml</code>.</p>
|
||||||
|
|
||||||
|
<h2 id="provider-config">Provider Configuration</h2>
|
||||||
|
<p>Create or edit <code>~/.smarttools/providers.yaml</code>:</p>
|
||||||
|
|
||||||
|
<pre><code class="language-yaml">providers:
|
||||||
|
- name: claude
|
||||||
|
command: "claude -p"
|
||||||
|
|
||||||
|
- name: openai
|
||||||
|
command: "openai-cli"
|
||||||
|
|
||||||
|
- name: ollama
|
||||||
|
command: "ollama run llama2"
|
||||||
|
|
||||||
|
- name: mock
|
||||||
|
command: "echo '[MOCK RESPONSE]'"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="using-providers">Using Providers in Tools</h2>
|
||||||
|
<p>Specify the provider in your step:</p>
|
||||||
|
<pre><code class="language-yaml">steps:
|
||||||
|
- type: prompt
|
||||||
|
provider: claude # Uses the "claude" provider from config
|
||||||
|
prompt: "..."
|
||||||
|
output_var: response</code></pre>
|
||||||
|
|
||||||
|
<h2 id="popular-providers">Popular Providers</h2>
|
||||||
|
|
||||||
|
<h3>Claude (Anthropic)</h3>
|
||||||
|
<pre><code class="language-bash"># Install
|
||||||
|
pip install claude-cli
|
||||||
|
|
||||||
|
# Configure with your API key
|
||||||
|
export ANTHROPIC_API_KEY="sk-ant-..."</code></pre>
|
||||||
|
|
||||||
|
<pre><code class="language-yaml"># providers.yaml
|
||||||
|
providers:
|
||||||
|
- name: claude
|
||||||
|
command: "claude -p"</code></pre>
|
||||||
|
|
||||||
|
<h3>OpenAI</h3>
|
||||||
|
<pre><code class="language-bash"># Install
|
||||||
|
pip install openai-cli
|
||||||
|
|
||||||
|
# Configure
|
||||||
|
export OPENAI_API_KEY="sk-..."</code></pre>
|
||||||
|
|
||||||
|
<h3>Ollama (Local)</h3>
|
||||||
|
<pre><code class="language-bash"># Install Ollama from ollama.ai
|
||||||
|
# Pull a model
|
||||||
|
ollama pull llama2</code></pre>
|
||||||
|
|
||||||
|
<pre><code class="language-yaml"># providers.yaml
|
||||||
|
providers:
|
||||||
|
- name: ollama
|
||||||
|
command: "ollama run llama2"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="testing">Testing with Mock Provider</h2>
|
||||||
|
<p>Use the mock provider to test tools without API calls:</p>
|
||||||
|
<pre><code class="language-yaml">providers:
|
||||||
|
- name: mock
|
||||||
|
command: "echo 'This is a mock response for testing'"</code></pre>
|
||||||
|
|
||||||
|
<h2 id="provider-selection">Choosing a Provider</h2>
|
||||||
|
<table class="table">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Provider</th>
|
||||||
|
<th>Best For</th>
|
||||||
|
<th>Cost</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>Claude</td>
|
||||||
|
<td>Complex reasoning, long context</td>
|
||||||
|
<td>Pay per token</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>OpenAI</td>
|
||||||
|
<td>General purpose, fast</td>
|
||||||
|
<td>Pay per token</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>Ollama</td>
|
||||||
|
<td>Privacy, offline use</td>
|
||||||
|
<td>Free (local)</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
""",
|
||||||
|
"headings": [
|
||||||
|
("provider-config", "Provider Configuration"),
|
||||||
|
("using-providers", "Using Providers in Tools"),
|
||||||
|
("popular-providers", "Popular Providers"),
|
||||||
|
("testing", "Testing with Mock Provider"),
|
||||||
|
("provider-selection", "Choosing a Provider"),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_doc(path: str) -> dict:
|
||||||
|
"""Get documentation content by path."""
|
||||||
|
# Normalize path
|
||||||
|
path = path.strip("/").replace("docs/", "") or "getting-started"
|
||||||
|
return DOCS.get(path, None)
|
||||||
|
|
||||||
|
|
||||||
|
def get_toc():
|
||||||
|
"""Get table of contents structure."""
|
||||||
|
from types import SimpleNamespace
|
||||||
|
return [
|
||||||
|
SimpleNamespace(slug="getting-started", title="Getting Started", children=[
|
||||||
|
SimpleNamespace(slug="installation", title="Installation"),
|
||||||
|
SimpleNamespace(slug="first-tool", title="Your First Tool"),
|
||||||
|
]),
|
||||||
|
SimpleNamespace(slug="publishing", title="Publishing", children=[]),
|
||||||
|
SimpleNamespace(slug="providers", title="Providers", children=[]),
|
||||||
|
]
|
||||||
|
|
@ -455,22 +455,30 @@ def dashboard_settings():
|
||||||
@web_bp.route("/docs", defaults={"path": ""}, endpoint="docs")
|
@web_bp.route("/docs", defaults={"path": ""}, endpoint="docs")
|
||||||
@web_bp.route("/docs/<path:path>", endpoint="docs")
|
@web_bp.route("/docs/<path:path>", endpoint="docs")
|
||||||
def docs(path: str):
|
def docs(path: str):
|
||||||
toc = [
|
from .docs_content import get_doc, get_toc
|
||||||
SimpleNamespace(slug="getting-started", title="Getting Started", children=[
|
|
||||||
SimpleNamespace(slug="installation", title="Installation"),
|
toc = get_toc()
|
||||||
SimpleNamespace(slug="first-tool", title="Your First Tool"),
|
|
||||||
]),
|
|
||||||
SimpleNamespace(slug="publishing", title="Publishing", children=[]),
|
|
||||||
SimpleNamespace(slug="providers", title="Providers", children=[]),
|
|
||||||
]
|
|
||||||
current = path or "getting-started"
|
current = path or "getting-started"
|
||||||
page = SimpleNamespace(
|
|
||||||
title=_title_case(current),
|
# Get content from docs_content module
|
||||||
description="SmartTools documentation",
|
doc = get_doc(current)
|
||||||
content_html=f"<p>Documentation for <strong>{escape(current)}</strong> is coming soon.</p>",
|
if doc:
|
||||||
headings=[],
|
page = SimpleNamespace(
|
||||||
parent=None,
|
title=doc["title"],
|
||||||
)
|
description=doc["description"],
|
||||||
|
content_html=Markup(doc["content"]),
|
||||||
|
headings=doc.get("headings", []),
|
||||||
|
parent=doc.get("parent"),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
page = SimpleNamespace(
|
||||||
|
title=_title_case(current),
|
||||||
|
description="SmartTools documentation",
|
||||||
|
content_html=Markup(f"<p>Documentation for <strong>{escape(current)}</strong> is coming soon.</p>"),
|
||||||
|
headings=[],
|
||||||
|
parent=None,
|
||||||
|
)
|
||||||
|
|
||||||
show_ads = current_app.config.get("SHOW_ADS", False)
|
show_ads = current_app.config.get("SHOW_ADS", False)
|
||||||
return render_template(
|
return render_template(
|
||||||
"pages/docs.html",
|
"pages/docs.html",
|
||||||
|
|
@ -538,3 +546,92 @@ def forgot_password():
|
||||||
title="Reset Password",
|
title="Reset Password",
|
||||||
body="Password resets are not available yet. Please contact support if needed.",
|
body="Password resets are not available yet. Please contact support if needed.",
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@web_bp.route("/robots.txt", endpoint="robots")
|
||||||
|
def robots():
|
||||||
|
"""Serve robots.txt - we welcome all crawlers including AI."""
|
||||||
|
from flask import send_from_directory
|
||||||
|
return send_from_directory(
|
||||||
|
web_bp.static_folder,
|
||||||
|
"robots.txt",
|
||||||
|
mimetype="text/plain"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@web_bp.route("/sitemap.xml", endpoint="sitemap")
|
||||||
|
def sitemap():
|
||||||
|
"""Generate dynamic sitemap.xml with all public pages and tools."""
|
||||||
|
from flask import Response
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
conn = connect_db()
|
||||||
|
try:
|
||||||
|
tools = query_all(
|
||||||
|
conn,
|
||||||
|
"""
|
||||||
|
WITH latest AS (
|
||||||
|
SELECT owner, name, MAX(id) AS max_id
|
||||||
|
FROM tools
|
||||||
|
WHERE version NOT LIKE '%-%'
|
||||||
|
GROUP BY owner, name
|
||||||
|
)
|
||||||
|
SELECT t.owner, t.name, t.published_at
|
||||||
|
FROM tools t
|
||||||
|
JOIN latest ON t.owner = latest.owner AND t.name = latest.name AND t.id = latest.max_id
|
||||||
|
ORDER BY t.published_at DESC
|
||||||
|
""",
|
||||||
|
[],
|
||||||
|
)
|
||||||
|
tools = [dict(row) for row in tools]
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
base_url = request.url_root.rstrip("/")
|
||||||
|
today = datetime.utcnow().strftime("%Y-%m-%d")
|
||||||
|
|
||||||
|
# Static pages
|
||||||
|
static_pages = [
|
||||||
|
("/", "1.0", "daily"),
|
||||||
|
("/tools", "0.9", "daily"),
|
||||||
|
("/docs", "0.8", "weekly"),
|
||||||
|
("/docs/getting-started", "0.8", "weekly"),
|
||||||
|
("/docs/installation", "0.7", "weekly"),
|
||||||
|
("/docs/first-tool", "0.7", "weekly"),
|
||||||
|
("/docs/publishing", "0.7", "weekly"),
|
||||||
|
("/docs/providers", "0.7", "weekly"),
|
||||||
|
("/tutorials", "0.7", "weekly"),
|
||||||
|
("/about", "0.5", "monthly"),
|
||||||
|
("/community", "0.5", "weekly"),
|
||||||
|
("/donate", "0.4", "monthly"),
|
||||||
|
("/privacy", "0.3", "monthly"),
|
||||||
|
("/terms", "0.3", "monthly"),
|
||||||
|
]
|
||||||
|
|
||||||
|
xml_parts = ['<?xml version="1.0" encoding="UTF-8"?>']
|
||||||
|
xml_parts.append('<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">')
|
||||||
|
|
||||||
|
# Add static pages
|
||||||
|
for path, priority, freq in static_pages:
|
||||||
|
xml_parts.append(f""" <url>
|
||||||
|
<loc>{base_url}{path}</loc>
|
||||||
|
<lastmod>{today}</lastmod>
|
||||||
|
<changefreq>{freq}</changefreq>
|
||||||
|
<priority>{priority}</priority>
|
||||||
|
</url>""")
|
||||||
|
|
||||||
|
# Add tool pages
|
||||||
|
for tool in tools:
|
||||||
|
pub_date = tool.get("published_at", today)
|
||||||
|
if pub_date and "T" in str(pub_date):
|
||||||
|
pub_date = str(pub_date).split("T")[0]
|
||||||
|
xml_parts.append(f""" <url>
|
||||||
|
<loc>{base_url}/tools/{tool['owner']}/{tool['name']}</loc>
|
||||||
|
<lastmod>{pub_date or today}</lastmod>
|
||||||
|
<changefreq>weekly</changefreq>
|
||||||
|
<priority>0.6</priority>
|
||||||
|
</url>""")
|
||||||
|
|
||||||
|
xml_parts.append("</urlset>")
|
||||||
|
|
||||||
|
return Response("\n".join(xml_parts), mimetype="application/xml")
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,47 @@
|
||||||
|
# SmartTools Registry - robots.txt
|
||||||
|
# We welcome all crawlers! Index everything to help users find us
|
||||||
|
# and enable AI assistants to help solve SmartTools issues.
|
||||||
|
|
||||||
|
User-agent: *
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
# Encourage full indexing of documentation and tutorials
|
||||||
|
Allow: /docs/
|
||||||
|
Allow: /tutorials/
|
||||||
|
Allow: /tools/
|
||||||
|
Allow: /categories/
|
||||||
|
|
||||||
|
# Allow AI crawlers explicitly (we want AI to learn about SmartTools)
|
||||||
|
User-agent: GPTBot
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: ChatGPT-User
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: Claude-Web
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: Anthropic-AI
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: Google-Extended
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: CCBot
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
User-agent: PerplexityBot
|
||||||
|
Allow: /
|
||||||
|
|
||||||
|
# Only block private user areas (not useful to index)
|
||||||
|
Disallow: /dashboard/
|
||||||
|
Disallow: /api/v1/me/
|
||||||
|
Disallow: /api/v1/tokens
|
||||||
|
|
||||||
|
# Allow API docs to be indexed (helpful for developers)
|
||||||
|
Allow: /api/
|
||||||
|
|
||||||
|
# Sitemap location
|
||||||
|
Sitemap: https://registry.smarttools.dev/sitemap.xml
|
||||||
|
|
||||||
|
# Crawl-delay: none - crawl as fast as you want!
|
||||||
Loading…
Reference in New Issue