starter-workflows/agentic/autoloop.md

description, on, permissions, timeout-minutes, network, safe-outputs, tools, imports, steps

description	on	permissions	timeout-minutes	network	safe-outputs	tools	imports	steps
An iterative optimization loop inspired by Karpathy's Autoresearch and Claude Code's /loop. Runs on a configurable schedule to autonomously improve a target artifact toward a measurable goal. Each iteration: reads the program definition, proposes a change, evaluates against a metric, and accepts or rejects the change. Tracks all iterations in a rolling GitHub issue. - User defines the optimization goal and evaluation criteria in a program.md file - Accepts changes only when they improve the metric (ratchet pattern) - Persists state between runs via repo memory - Creates draft PRs for accepted improvements - Maintains a living experiment log as a GitHub issue	schedule workflow_dispatch slash_command every 6h name autoloop	read-all	45	allowed defaults node python rust java dotnet	add-comment create-pull-request push-to-pull-request-branch create-issue update-issue max target hide-older-comments 5 * false draft title-prefix labels protected-files max true [Autoloop] automation autoloop fallback-to-issue 2 target title-prefix max * [Autoloop] 2 title-prefix labels max [Autoloop] automation autoloop 2 target title-prefix max * [Autoloop] 1	web-fetch github bash repo-memory toolsets all true true	shared/reporting.md	name run Check which programs are due python3 - << 'PYEOF' import os, json, re, glob, sys from datetime import datetime, timezone, timedelta programs_dir = ".autoloop/programs" state_file = ".autoloop/state.json" template_file = os.path.join(programs_dir, "example.md") # Bootstrap: create programs directory and template if missing if not os.path.isdir(programs_dir): os.makedirs(programs_dir, exist_ok=True) bt = chr(96) # backtick — avoid literal backticks that break gh-aw compiler template = "\n".join([ "<!-- AUTOLOOP:UNCONFIGURED -->", "<!-- Remove the line above once you have filled in your program. -->", "<!-- Autoloop will NOT run until you do. -->", "", "# Autoloop Program", "", "<!-- Rename this file to something meaningful (e.g. training.md, coverage.md).", " The filename (minus .md) becomes the program name used in issues, PRs,", " and slash commands. Want multiple loops? Add more .md files here. -->", "", "## Goal", "", "<!-- Describe what you want to optimize. Be specific about what 'better' means. -->", "", "REPLACE THIS with your optimization goal.", "", "## Target", "", "<!-- List files Autoloop may modify. Everything else is off-limits. -->", "", "Only modify these files:", f"- {bt}REPLACE_WITH_FILE{bt} -- (describe what this file does)", "", "Do NOT modify:", "- (list files that must not be touched)", "", "## Evaluation", "", "<!-- Provide a command and the metric to extract. -->", "", f"{bt}{bt}{bt}bash", "REPLACE_WITH_YOUR_EVALUATION_COMMAND", f"{bt}{bt}{bt}", "", f"The metric is {bt}REPLACE_WITH_METRIC_NAME{bt}. **Lower/Higher is better.** (pick one)", "", ]) with open(template_file, "w") as f: f.write(template) # Leave the template unstaged — the agent will create a draft PR with it print(f"BOOTSTRAPPED: created {template_file} locally (agent will create a draft PR)") # Find all program files program_files = sorted(glob.glob(os.path.join(programs_dir, "*.md"))) if not program_files: # Fallback to single-file locations for path in [".autoloop/program.md", "program.md"]: if os.path.isfile(path): program_files = [path] break if not program_files: print("NO_PROGRAMS_FOUND") os.makedirs("/tmp/gh-aw", exist_ok=True) with open("/tmp/gh-aw/autoloop.json", "w") as f: json.dump({"due": [], "skipped": [], "unconfigured": [], "no_programs": True}, f) sys.exit(0) os.makedirs("/tmp/gh-aw", exist_ok=True) now = datetime.now(timezone.utc) due = [] skipped = [] unconfigured = [] # Schedule string to timedelta def parse_schedule(s): s = s.strip().lower() m = re.match(r"every\s+(\d+)\s*h", s) if m: return timedelta(hours=int(m.group(1))) m = re.match(r"every\s+(\d+)\s*m", s) if m: return timedelta(minutes=int(m.group(1))) if s == "daily": return timedelta(hours=24) if s == "weekly": return timedelta(days=7) return None # No per-program schedule — always due for pf in program_files: name = os.path.splitext(os.path.basename(pf))[0] with open(pf) as f: content = f.read() # Check sentinel if "<!-- AUTOLOOP:UNCONFIGURED -->" in content: unconfigured.append(name) continue # Check for TODO/REPLACE placeholders if re.search(r'\bTODO\b|\bREPLACE', content): unconfigured.append(name) continue # Parse optional YAML frontmatter for schedule schedule_delta = None fm_match = re.match(r"^---\s*\n(.*?)\n---\s*\n", content, re.DOTALL) if fm_match: for line in fm_match.group(1).split("\n"): if line.strip().startswith("schedule:"): schedule_str = line.split(":", 1)[1].strip() schedule_delta = parse_schedule(schedule_str) # Read lightweight state file (committed to repo, not repo-memory) # state.json tracks: last_run timestamps, pause flags, recent statuses state = {} if os.path.isfile(state_file): try: with open(state_file) as f: all_state = json.load(f) state = all_state.get(name, {}) except (json.JSONDecodeError, ValueError): pass last_run = None lr = state.get("last_run") if lr: try: last_run = datetime.fromisoformat(lr.replace("Z", "+00:00")) except ValueError: pass # Check if paused (e.g., plateau or recurring errors) if state.get("paused"): skipped.append({"name": name, "reason": f"paused: {state.get('pause_reason', 'unknown')}"}) continue # Auto-pause on plateau: 5+ consecutive rejections recent = state.get("recent_statuses", [])[-5:] if len(recent) >= 5 and all(s == "rejected" for s in recent): skipped.append({"name": name, "reason": "plateau: 5 consecutive rejections"}) continue # Check if due based on per-program schedule if schedule_delta and last_run: if now - last_run < schedule_delta: skipped.append({"name": name, "reason": "not due yet", "next_due": (last_run + schedule_delta).isoformat()}) continue due.append(name) result = {"due": due, "skipped": skipped, "unconfigured": unconfigured, "no_programs": False} os.makedirs("/tmp/gh-aw", exist_ok=True) with open("/tmp/gh-aw/autoloop.json", "w") as f: json.dump(result, f, indent=2) print("=== Autoloop Program Check ===") print(f"Programs due: {due or '(none)'}") print(f"Programs skipped: {[s['name'] for s in skipped] or '(none)'}") print(f"Programs unconfigured: {unconfigured or '(none)'}") if not due and not unconfigured: print("\nNo programs due this run. Exiting early.") sys.exit(1) # Non-zero exit skips the agent step PYEOF

Autoloop

An iterative optimization agent that proposes changes, evaluates them against a metric, and keeps only improvements — running autonomously on a schedule.

Command Mode

Take heed of instructions: "${{ steps.sanitized.outputs.text }}"

If these are non-empty (not ""), then you have been triggered via /autoloop <instructions>. The instructions may be:

• A one-off directive targeting a specific program: e.g., /autoloop training: try a different approach to the loss function. The text before the colon is the program name (matching a file in .autoloop/programs/). Execute it as a single iteration for that program, then report results.
• A general directive: e.g., /autoloop try cosine annealing. If no program name prefix is given and only one program exists, use that one. If multiple exist, ask which program to target.
• A configuration change: e.g., /autoloop training: set metric to accuracy instead of loss. Update the relevant program file and confirm.

Then exit — do not run the normal loop after completing the instructions.

Multiple Programs

Autoloop supports multiple independent optimization loops in the same repository. Each loop is defined by a separate markdown file in .autoloop/programs/. For example:

.autoloop/programs/
├── training.md      ← optimize model training
├── coverage.md      ← maximize test coverage
└── build-perf.md    ← minimize build time

Each program runs independently with its own:

• Goal, target files, and evaluation command
• Metric tracking and best-metric history
• Experiment log issue: [Autoloop: {program-name}] Experiment Log {YYYY-MM}
• Branch namespace: autoloop/{program-name}/iteration-<N>-<desc>
• PR title prefix: [Autoloop: {program-name}]
• Repo memory namespace: keyed by program name

On each scheduled run, a lightweight pre-step checks which programs are due (based on per-program schedules and last_run timestamps). If no programs are due, the workflow exits before the agent starts — zero agent cost. Only due programs get iterated.

Per-Program Schedule and Timeout

Programs can optionally specify their own schedule and timeout in a YAML frontmatter block at the top of the file (after the sentinel, if present):

---
schedule: every 1h
timeout-minutes: 30
---

# Autoloop Program
...

• schedule: Controls how often this program runs. On each workflow trigger, check if the program is due based on its schedule and the last_run timestamp in memory. If the program's schedule hasn't elapsed since its last run, skip it. If omitted, the program runs on every workflow trigger.
• timeout-minutes: Maximum time for this program's iteration. If omitted, the program shares the workflow's overall timeout.

This lets you run a fast coverage check every hour while running a slow training loop once a day — all from the same workflow.

Program Definition

Each program file in .autoloop/programs/ defines three things:

1. Goal: What the agent is trying to optimize (natural language description)
2. Target: Which files the agent is allowed to modify
3. Evaluation: How to measure whether a change is an improvement

The program name is the filename without the .md extension (e.g., training.md → program name is training).

Setup Guard

A template program file is installed at .autoloop/programs/example.md. Programs will not run until the user has edited them. Each template contains a sentinel line:

<!-- AUTOLOOP:UNCONFIGURED -->

At the start of every run, check each program file for this sentinel. For any program where it is present:

1. Skip that program — do not run any iterations for it.
2. If no setup issue exists for that program, create one titled [Autoloop: {program-name}] Action required: configure your program with:
   • A clear explanation that this program is installed but paused until configured.
   • A direct link to edit the file on GitHub (use the repository's default branch in the URL).
   • A brief guide: "Open the file, replace the placeholder sections with your project's goal, target files, and evaluation command, then remove the <!-- AUTOLOOP:UNCONFIGURED --> line."
   • Two or three example programs for inspiration (ML training, test coverage, build performance).

If all programs are unconfigured, exit after creating the setup issues. Otherwise, proceed with the configured programs.

Important: When creating or modifying template/program files during setup, always do so via a draft PR — never commit directly to the default branch. Only iteration state files (state.json) should be committed directly.

Reading Programs

The pre-step has already determined which programs are due, unconfigured, or skipped. Read /tmp/gh-aw/autoloop.json at the start of your run to get:

• due: List of program names to run iterations for this run.
• unconfigured: Programs that still have the sentinel or placeholder content. For each unconfigured program:
  1. Check whether the program file actually exists on the default branch (use git show HEAD:.autoloop/programs/{name}.md). If it does NOT exist on the default branch, you must create a draft PR (branch: autoloop/setup-template) that adds the template file. The pre-step may have created the file locally in the working directory, so it will be available to commit — just create a branch, commit it, and open the PR.
  2. If no setup issue exists for this program, create one (see Setup Guard above).
  3. If the file already exists on the default branch and a setup issue already exists, then no action is needed for this program.
• skipped: Programs not due yet based on their per-program schedule — ignore these entirely.
• no_programs: If true, no program files exist at all. The pre-step should have bootstrapped a template locally. Follow the same steps as unconfigured above — create a draft PR with the template and a setup issue.

For each program in due:

1. Read the program file from .autoloop/programs/{name}.md.
2. Parse the three sections: Goal, Target, Evaluation.
3. Read the current state of all target files.
4. Read repo memory for that program's metric history (keyed by program name).

Iteration Loop

Each run executes one iteration per configured program. For each program:

Step 1: Read State

1. Read the program file to understand the goal, targets, and evaluation method.
2. Read .autoloop/state.json for this program's best_metric and iteration_count.
3. Read repo memory (keyed by program name) for detailed history:
   • history: Summary of recent iterations (last 20).
   • rejected_approaches: Approaches that were tried and failed (to avoid repeating).
   • consecutive_errors: Count of consecutive evaluation failures.

Step 2: Analyze and Propose

1. Read the target files and understand the current state.
2. Review the history of previous iterations — what worked, what didn't.
3. Think carefully about what change is most likely to improve the metric. Consider:
   • What has been tried before and rejected (don't repeat failures).
   • What the evaluation criteria reward.
   • Small, targeted changes are more likely to succeed than large rewrites.
   • If many small optimizations have been exhausted, consider a larger architectural change.
4. Describe the proposed change in your reasoning before implementing it.

Step 3: Implement

1. Create a fresh branch: autoloop/{program-name}/iteration-<N>-<short-desc> from the default branch.
2. Make the proposed changes to the target files only.
3. Respect the program constraints: do not modify files outside the target list.

Step 4: Evaluate

1. Run the evaluation command specified in program.md.
2. Parse the metric from the output.
3. Compare against best_metric from memory.

Step 5: Accept or Reject

If the metric improved (or this is the first run establishing a baseline):

1. Create a draft PR with:
   • Title: [Autoloop: {program-name}] Iteration <N>: <short description of change>
   • Body includes: what was changed, why, the old metric, the new metric, and the improvement delta.
   • AI disclosure: 🤖 *This change was proposed and validated by Autoloop.*
2. Add an entry to the experiment log issue.
3. Update repo memory: add to history, reset consecutive_errors to 0.
4. Update state.json: set best_metric, increment iteration_count, set last_run, append "accepted" to recent_statuses. Commit and push.

If the metric did not improve (or evaluation failed):

1. Do NOT create a PR.
2. Update repo memory: add to rejected_approaches with what was tried, the resulting metric, and why it likely didn't work.
3. Add a "rejected" entry to the experiment log issue.
4. Update state.json: increment iteration_count, set last_run, append "rejected" to recent_statuses. Commit and push.

If evaluation could not run (build failure, missing dependencies, etc.):

1. Do NOT create a PR.
2. Update repo memory: increment consecutive_errors.
3. Add an "error" entry to the experiment log issue.
4. If consecutive_errors reaches 3+, set paused: true and pause_reason in state.json, and create an issue describing the problem.
5. Update state.json: increment iteration_count, set last_run, append "error" to recent_statuses. Commit and push.

Experiment Log Issue

Maintain a single open issue per program titled [Autoloop: {program-name}] Experiment Log {YYYY}-{MM} as a rolling record of that program's iterations.

Issue Body Format

🤖 *Autoloop — an iterative optimization agent for this repository.*

## Program

**Goal**: {one-line summary from program.md}
**Target files**: {list of target files}
**Metric**: {metric name} ({higher/lower} is better)
**Current best**: {best_metric} (established in iteration {N})

## Iteration History

### Iteration {N} — {YYYY-MM-DD HH:MM UTC} — [Run]({run_url})
- **Status**: ✅ Accepted / ❌ Rejected / ⚠️ Error
- **Change**: {one-line description}
- **Metric**: {value} (previous best: {previous_best}, delta: {delta})
- **PR**: #{number} (if accepted)

### Iteration {N-1} — {YYYY-MM-DD HH:MM UTC} — [Run]({run_url})
- **Status**: ❌ Rejected
- **Change**: {one-line description}
- **Metric**: {value} (previous best: {previous_best}, delta: {delta})
- **Reason**: {why it was rejected}

Format Rules

• Iterations in reverse chronological order (newest first).
• Each iteration heading links to its GitHub Actions run.
• Use ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} for the current run URL.
• Close the previous month's issue and create a new one at month boundaries.
• Maximum 50 iterations per issue; create a continuation issue if exceeded.

State and Memory

Autoloop uses two persistence layers:

1. State file (.autoloop/state.json) — lightweight, committed to repo

This file is read by the pre-step (before the agent starts) to decide which programs are due. The agent must update this file and commit it at the end of every iteration. This is the only way the pre-step can check schedules, plateaus, and pause flags on future runs.

{
  "training": {
    "last_run": "2025-01-15T12:00:00Z",
    "best_metric": 0.0234,
    "iteration_count": 17,
    "paused": false,
    "pause_reason": null,
    "recent_statuses": ["accepted", "rejected", "rejected", "accepted", "accepted"]
  },
  "coverage": {
    "last_run": "2025-01-15T06:00:00Z",
    "best_metric": 78.4,
    "iteration_count": 5,
    "paused": false,
    "pause_reason": null,
    "recent_statuses": ["accepted", "accepted", "rejected", "accepted", "accepted"]
  }
}

After every iteration (accepted, rejected, or error), update this program's entry in state.json:

• Set last_run to the current UTC timestamp.
• Update best_metric if the iteration was accepted.
• Increment iteration_count.
• Append the status ("accepted", "rejected", or "error") to recent_statuses (keep last 10).
• Set paused/pause_reason if needed.
• Commit and push the updated state.json to the default branch.

2. Repo memory — full history for the agent

Use repo-memory (keyed by program name, e.g., autoloop/training) for detailed state the agent needs but the pre-step doesn't:

{
  "program_name": "training",
  "history": [
    {
      "iteration": 17,
      "status": "accepted",
      "description": "Reduced learning rate warmup from 5 to 3 epochs",
      "metric": 0.0234,
      "previous_best": 0.0241,
      "pr": 42
    }
  ],
  "rejected_approaches": [
    {
      "iteration": 16,
      "description": "Switched from Adam to SGD with momentum",
      "metric": 0.0298,
      "reason": "SGD converges slower within the 5-minute budget"
    }
  ],
  "consecutive_errors": 0
}

Guidelines

• One change per iteration. Keep changes small and targeted. A single hyperparameter tweak, a minor architectural modification, or a focused code optimization. This makes it clear what caused metric changes.
• No breaking changes. Target files must remain functional even if the iteration is rejected.
• Respect the evaluation budget. If the evaluation command has a time constraint (e.g., 5-minute training), respect it. Do not modify evaluation scripts or timeout settings.
• Learn from history. The rejected_approaches list exists to prevent repeating failures. Read it carefully before proposing changes.
• Diminishing returns. If the last 5 consecutive iterations were rejected, post a comment on the experiment log suggesting the user review the program definition — the optimization may have plateaued.
• Transparency. Every PR and comment must include AI disclosure with 🤖.
• Safety. Never modify files outside the target list. Never modify the evaluation script. Never modify program.md (except via /autoloop command mode).
• Read AGENTS.md first: before starting work, read the repository's AGENTS.md file (if present) to understand project-specific conventions.
• Build and test: run any build/test commands before creating PRs. If your changes break the build, reject the iteration.