orchestrated-discussions/docs/PIPELINE_SCHEMA.md

# Turn Pipeline Schema

## Overview

Each template can define a `turn_pipeline` that specifies which SmartTools to run and how to pipe data between them. The runner is thin orchestration - it just executes tools and pipes data, with no business logic.

## Design Philosophy

The pipeline should mirror what you could do manually in a shell:

```bash
# Parse the discussion
cat discussion.md | discussion-parser > parsed.json

# Route mentions
cat parsed.json | discussion-mention-router > routing.json

# Call each participant (could be parallel)
cat discussion.md | discussion-architect --callout "Review" > resp_architect.json
cat discussion.md | discussion-security --callout "Review" > resp_security.json

# Append responses
cat discussion.md | discussion-turn-appender --responses '[...]' > updated.md

# Count votes (if voting phase)
cat updated.md | discussion-parser | discussion-vote-counter > votes.json
```

## Schema

```yaml
turn_pipeline:
  steps:
    - tool: <tool-name>           # Required: SmartTool to execute
      input: <variable>           # What to pipe to stdin (default: $discussion)
      output: <variable>          # Where to store stdout (default: discarded)
      args:                       # CLI arguments
        --arg-name: <value>       # Can reference variables: $varname
      when: <condition>           # Optional: condition to run this step
      for_each: <variable>        # Optional: iterate over array
      parallel: <bool>            # Optional: run for_each in parallel
```

## Variables

Variables are referenced with `$` prefix:

### Built-in Variables
- `$discussion` - Current discussion markdown content (updated as pipeline runs)
- `$discussion_path` - Path to the discussion file
- `$callout` - Callout/question passed to the turn
- `$participants` - List of participant aliases from discussion metadata
- `$templates_dir` - Path to templates directory

### Step Output Variables
Each step can capture its output:
```yaml
- tool: discussion-parser
  input: $discussion
  output: $parsed        # Now $parsed contains the JSON output
```

### Accessing JSON Fields
For JSON outputs, access fields with dot notation:
```yaml
- tool: discussion-mention-router
  input: $parsed
  output: $routing

- tool: discussion-{participant}
  for_each: $routing.participants_to_call   # Iterate over array in JSON
```

### Array Collection
When using `for_each`, outputs are collected into an array:
```yaml
- tool: discussion-{participant}
  for_each: $routing.participants_to_call
  output: $responses[]    # [] means append to array
```

## Conditions

- `always` - Always run (default)
- `$variable` - Run if variable is truthy
- `$variable.field` - Run if JSON field is truthy
- `not $variable` - Run if variable is falsy

Examples:
```yaml
- tool: discussion-vote-counter
  when: $phase.voting                    # Only if phase has voting enabled

- tool: discussion-mention-router
  when: not $participants_specified      # Only if no participants given
```

## Complete Example

```yaml
turn_pipeline:
  steps:
    # Step 1: Parse discussion to get metadata
    - tool: discussion-parser
      input: $discussion
      output: $parsed

    # Step 2: Route mentions (optional - skip if participants specified)
    - tool: discussion-mention-router
      input: $parsed
      output: $routing
      when: not $participants_specified
      args:
        --default-participants: $participants

    # Step 3: Call each participant
    - tool: discussion-{participant}
      for_each: $routing.participants_to_call
      parallel: true
      input: $discussion
      output: $responses[]
      args:
        --callout: $callout
        --templates-dir: $templates_dir

    # Step 4: Append responses to discussion
    - tool: discussion-turn-appender
      input: $discussion
      output: $discussion              # Update the discussion variable
      args:
        --responses-json: $responses

    # Step 5: Count votes (only in voting phases)
    - tool: discussion-parser
      input: $discussion
      output: $updated_parsed
      when: $phase.voting

    - tool: discussion-vote-counter
      input: $updated_parsed
      output: $votes
      when: $phase.voting
      args:
        --threshold-ready: $phase.threshold_ready
        --human-required: $phase.human_required

    # Step 6: Check status promotion
    - tool: discussion-status-promoter
      input: $votes
      output: $promotion
      when: $phase.voting
      args:
        --current-status: $parsed.metadata.status
        --current-phase: $parsed.metadata.phase
```

## For Each Expansion

When `for_each` is used, the step runs once per item:

```yaml
- tool: discussion-{participant}
  for_each: $routing.participants_to_call   # e.g., ["architect", "security"]
  input: $discussion
  output: $responses[]
```

This expands to:
```bash
cat discussion.md | discussion-architect > resp_0.json
cat discussion.md | discussion-security > resp_1.json
# $responses = [resp_0.json contents, resp_1.json contents]
```

With `parallel: true`, these run concurrently.

## Special Tool Patterns

### `{participant}` Substitution
The pattern `{participant}` in tool names is replaced with the current iteration value:

```yaml
- tool: discussion-{participant}
  for_each: ["architect", "security"]
```

Becomes calls to `discussion-architect` and `discussion-security`.

## Minimal Example

A bare-bones pipeline that just calls participants and appends:

```yaml
turn_pipeline:
  steps:
    - tool: discussion-{participant}
      for_each: $participants
      parallel: true
      input: $discussion
      output: $responses[]

    - tool: discussion-turn-appender
      input: $discussion
      output: $discussion
      args:
        --responses-json: $responses
```

## Phase Artifacts

Phases can define `artifacts` to enable collaboration on diagrams, sketches, and models.
Participants read this config to know what formats to output and where to save files.

### Artifact Schema

```yaml
phases:
  sketch:
    goal: Rough out the selected idea
    artifacts:
      formats:                    # List of accepted formats
        - plantuml                # PlantUML diagrams (.puml)
        - svg                     # SVG vector graphics (.svg)
        - mermaid                 # Mermaid diagrams (.mmd)
        - openscad                # OpenSCAD 3D models (.scad)
        - solidpython             # SolidPython 3D models (.py)
      output_dir: diagrams/       # Where to save artifacts
      file_pattern: "{title}_{participant}_{n}"  # Naming pattern
    instructions: |
      - Create diagrams in any supported format
      - Review existing artifacts in the output directory
      - Build on or modify others' work as needed      
```

### Supported Formats

| Format | Extension | Use Case |
|--------|-----------|----------|
| `plantuml` | `.puml` | Sequence, component, class, state diagrams |
| `mermaid` | `.mmd` | Flowcharts, sequence, ER, Gantt charts |
| `svg` | `.svg` | UI mockups, icons, 2D layouts, sketches |
| `openscad` | `.scad` | 3D models, mechanical parts, enclosures |
| `solidpython` | `.py` | Programmatic 3D models (Python-based) |
| `dot` | `.dot` | Graph visualizations (Graphviz) |
| `ascii` | `.txt` | Simple inline text diagrams |

### How Participants Use Artifacts

Participants receive artifact config via the `--artifacts-json` argument:

```json
{
  "formats": ["plantuml", "svg"],
  "output_dir": "diagrams/",
  "file_pattern": "{title}_{participant}_{n}"
}
```

They then:
1. Know what formats are acceptable for this phase
2. Save artifacts to the correct directory
3. Reference artifacts in comments with `ARTIFACT: path/to/file.ext`
4. Can read existing artifacts in `output_dir` to build upon them

### Response Format with Artifacts

Participants output artifacts in their JSON response:

```json
{
  "comment": "Here's my proposed architecture...\n\nARTIFACT: diagrams/auth-flow_architect_1.puml",
  "vote": null,
  "artifacts": [
    {
      "format": "plantuml",
      "filename": "auth-flow_architect_1.puml",
      "content": "@startuml\nUser -> Auth: login\nAuth -> DB: validate\n@enduml"
    }
  ]
}
```

### Example: Hardware Design Template

```yaml
name: hardware_design
description: Physical product design workflow

phases:
  requirements:
    goal: Define physical constraints and requirements
    artifacts: null  # No artifacts in this phase
    voting: false

  enclosure_design:
    goal: Design the physical enclosure
    artifacts:
      formats: [openscad, solidpython]
      output_dir: models/
    instructions: |
      - Create 3D models considering manufacturing constraints
      - Review existing models in models/ folder
      - Consider assembly, thermal, and structural requirements      
    voting: false

  pcb_layout:
    goal: Plan PCB component placement
    artifacts:
      formats: [svg, plantuml]
      output_dir: pcb/
    instructions: |
      - Create component layout diagrams
      - Consider signal routing and thermal zones      
    voting: false

  final_review:
    goal: Approve the complete design
    artifacts:
      formats: [openscad, svg, plantuml]
      output_dir: final/
    voting: true
    threshold_ready: 0.67
```

### Example: UI Design Template

```yaml
name: ui_design
description: User interface design workflow

phases:
  wireframe:
    goal: Sketch rough layouts
    artifacts:
      formats: [svg, ascii]
      output_dir: wireframes/
    instructions: |
      - Create rough wireframes - boxes and labels only
      - Focus on layout and flow, not aesthetics
      - One SVG per screen/view      
    voting: false

  mockup:
    goal: Detailed visual mockups
    artifacts:
      formats: [svg]
      output_dir: mockups/
    instructions: |
      - Build on approved wireframes
      - Add visual hierarchy and styling
      - Consider responsive breakpoints      
    voting: true
    threshold_ready: 0.5
```