# Usage Guide

Step-by-step instructions for running the migration, handling failures, rolling back, and day-to-day operations.

---

## Table of Contents

1. [Before You Start](#before-you-start)
2. [Happy Path: Full Migration](#happy-path-full-migration)
3. [Resuming After Failure](#resuming-after-failure)
4. [Edge Cases](#edge-cases)
5. [Rollback Procedures](#rollback-procedures)
6. [Verifying Rollback Success](#verifying-rollback-success)
7. [Day-to-Day Operations](#day-to-day-operations)
8. [Backup and Restore](#backup-and-restore)
9. [Runner Management](#runner-management)
10. [Troubleshooting](#troubleshooting)

---

## Before You Start

### 1. Network

Before running anything, confirm:

- MacBook can SSH to Unraid: `ssh user@UNRAID_IP`
- MacBook can SSH to Fedora: `ssh user@FEDORA_IP`
- Macvlan container IPs are available on the LAN (not already in use)

DNS and TLS are only needed for Phase 8 (Caddy reverse proxy). You can set these up later:

- A DNS A record for your Gitea domain pointing to `UNRAID_IP`
- If using `TLS_MODE=cloudflare`: a Cloudflare API token with Zone:DNS:Edit permission

### 2. Passwordless sudo on remote hosts

The setup and phase scripts run `sudo` commands on Unraid and Fedora over SSH (non-interactive, no TTY). If `sudo` requires a password, it will fail with: `a terminal is required to read the password`.

**Enable temporarily** (on each remote host):

```bash
# SSH in interactively
ssh user@HOST_IP

# Create a drop-in sudoers file
sudo visudo -f /etc/sudoers.d/temp-nopasswd
# Add this line (replace YOUR_USER with the SSH username):
YOUR_USER ALL=(ALL) NOPASSWD: ALL
# Save and exit, then verify:
sudo -n true && echo "OK — passwordless sudo works"
```

**Disable after migration is complete:**

```bash
sudo rm /etc/sudoers.d/temp-nopasswd
```

### 3. GitHub Tokens

You need one GitHub Personal Access Token:

| Token | Scope | Used By |
|-------|-------|---------|
| `GITHUB_TOKEN` | `repo` (read+write) | Migration, push mirrors, preflight validation |

### 4. Configuration

```bash
# Option A: Interactive wizard (recommended for first time)
./setup/configure_env.sh

# Option B: Manual
cp .env.example .env
cp runners.conf.example runners.conf
# Edit both files with your values

# Optional: interactive runner definition wizard
./setup/configure_runners.sh
```

The wizard validates every input (IP format, port ranges, path format, password length) and shows your current values in brackets so you can press Enter to keep them.
Internal Gitea API URLs are derived automatically from `UNRAID_GITEA_IP` and `FEDORA_GITEA_IP`.

---

## Happy Path: Full Migration

### Automated (recommended)

```bash
./run_all.sh
```

This runs the full pipeline: setup scripts, preflight checks, and all 9 phases with post-checks after each phase. It stops on the first failure and prints a summary of what completed.

Expected output on success:

```
>>> Running: Setup MacBook
>>> Running: Setup Unraid
>>> Running: Setup Fedora
>>> Running: Cross-host SSH trust
>>> Running: Preflight checks
>>> Running: Phase 1: Gitea on Unraid
>>> Running: Phase 1 — post-check
>>> Running: Phase 2: Gitea on Fedora
>>> Running: Phase 2 — post-check
...
>>> Running: Phase 9: Security
>>> Running: Phase 9 — post-check

===== Execution Summary =====
  Setup MacBook
  Setup Unraid
  Setup Fedora
  Cross-host SSH trust
  Preflight checks
  Phase 1: Gitea on Unraid
  Phase 1 — post-check
  ...
  Phase 9: Security
  Phase 9 — post-check

Migration complete! Gitea is live.
```

### Manual (phase by phase)

If you prefer to run each phase individually and inspect results:

```bash
# 1. Setup (once)
./setup/configure_env.sh
./setup/macbook.sh
./setup/unraid.sh
./setup/fedora.sh
./setup/cross_host_ssh.sh

# 2. Validate
./preflight.sh

# 3. Run each phase
./phase1_gitea_unraid.sh && ./phase1_post_check.sh
./phase2_gitea_fedora.sh && ./phase2_post_check.sh
./phase3_runners.sh       && ./phase3_post_check.sh
./phase4_migrate_repos.sh && ./phase4_post_check.sh
./phase5_migrate_pipelines.sh && ./phase5_post_check.sh
./phase6_github_mirrors.sh    && ./phase6_post_check.sh
./phase7_branch_protection.sh && ./phase7_post_check.sh
./phase8_cutover.sh           && ./phase8_post_check.sh
./phase9_security.sh          && ./phase9_post_check.sh
```

### Skip setup (already done)

```bash
./run_all.sh --skip-setup
```

### What to verify when it's done

After the full migration completes, run the post-migration check:

```bash
./post-migration-check.sh
# or equivalently:
./run_all.sh --dry-run
```

This probes all live infrastructure and reports the state of every phase — what's done, what's pending, and any errors. See [Post-Migration Check](#post-migration-check) below for details.

You can also verify manually:

1. **HTTPS access**: Open `https://YOUR_DOMAIN` in a browser — you should see the Gitea login page with a valid SSL certificate.
2. **Repository content**: Log in as admin, navigate to your org, confirm all repos have commits, branches, and (if enabled) issues/labels.
3. **CI pipelines**: Create a test PR in one repo. Gitea Actions should run the migrated workflows. If Phase 9 ran, security scans (Semgrep, Trivy, Gitleaks) should also trigger.
4. **GitHub mirrors**: Check your GitHub repos — the description should start with `[MIRROR]` and the latest commits should match Gitea.
5. **Fedora mirrors**: SSH into the Fedora Gitea instance and verify repos exist under the admin user namespace.
6. **Runners**: Run `./manage_runner.sh list` — all runners should show status `online` or `idle`.

---

## Resuming After Failure

### Starting from a specific phase

If Phase 5 failed but Phases 1-4 completed successfully:

```bash
./run_all.sh --start-from=5
```

This runs preflight (with `--skip-port-checks` since Gitea is already running on those ports), then phases 5 through 9.

### Re-running a single phase

Every phase is idempotent. If Phase 4 failed on the second repo, just re-run it:

```bash
./phase4_migrate_repos.sh
```

It will skip the first repo (already exists) and retry the second.

### Re-running a post-check

Post-checks are read-only and can be run at any time:

```bash
./phase4_post_check.sh
```

### Preflight after partial migration

When resuming from a later phase, Gitea is already running on ports 3000. Use:

```bash
./preflight.sh --skip-port-checks
```

`run_all.sh --start-from=N` does this automatically when N > 1.

---

## Post-Migration Check

A standalone read-only script that probes live infrastructure and reports the state of every migration phase. No mutations — safe to run at any time, before, during, or after migration.

```bash
./post-migration-check.sh
# or:
./run_all.sh --dry-run
```

### What it checks

- **Connectivity**: SSH to Unraid/Fedora, Docker daemons, GitHub API token validity
- **Phase 1-2**: Docker networks, compose files, app.ini, container health, admin auth, API tokens, organization
- **Phase 3**: runners.conf, registration token, per-runner online/offline status
- **Phase 4**: GitHub source repos accessible, Gitea repos migrated, Fedora mirrors active
- **Phase 5**: Workflow directories present in Gitea repos
- **Phase 6**: Push mirrors configured, GitHub Actions disabled
- **Phase 7**: Branch protection rules with approval counts
- **Phase 8**: DNS resolution, Caddy container, HTTPS end-to-end, TLS cert, GitHub `[MIRROR]` marking
- **Phase 9**: Security scan workflows deployed

### Output format

Three states:

| State | Meaning |
|-------|---------|
| `[DONE]` | Already exists/running — phase would skip this step |
| `[TODO]` | Not done yet — phase would execute this step |
| `[ERROR]` | Something is broken — needs attention |

`[TODO]` is normal for phases you haven't run yet. Only `[ERROR]` indicates a problem.

The script exits 0 if no errors, 1 if any `[ERROR]` found. A summary at the end shows per-phase counts.

---

## Edge Cases

### GitHub API rate limit hit during migration

**Symptom**: Phase 4 or Phase 6 fails with HTTP 403 and a message about rate limits.

**Fix**: Wait for the rate limit window to reset (usually 1 hour), then re-run the failed phase. The phase is idempotent — repos that already migrated will be skipped.

### GitHub token expires mid-migration

**Symptom**: API calls fail with HTTP 401.

**Fix**: Generate a new token on GitHub, update `GITHUB_TOKEN` in `.env`, then re-run the failed phase.

### Large repos time out during migration

**Symptom**: Phase 4 logs "Timeout waiting for REPO migration to complete".

**Fix**: Increase the timeout in `.env`:

```bash
MIGRATION_POLL_TIMEOUT_SEC=1800    # 30 minutes instead of default 10
```

Then re-run Phase 4. Already-migrated repos will be skipped.

### Container IP already in use

**Symptom**: Preflight check 13 warns that a macvlan IP is already responding to ping.

**Fix**: Either release the conflicting IP or change the container IP in `.env` (`UNRAID_GITEA_IP`, `UNRAID_CADDY_IP`, or `FEDORA_GITEA_IP`).

### DNS doesn't resolve to Unraid IP

**Symptom**: Preflight check 14 fails.

**Fix**: Add or update your DNS A record. If using a local DNS server or `/etc/hosts`, ensure the record points to `UNRAID_IP`. DNS propagation can take minutes to hours.

### Caddy fails to start or obtain TLS certificate in Phase 8

**Symptom**: Phase 8 times out waiting for HTTPS to become available.

Common causes (TLS_MODE=cloudflare):
- `CLOUDFLARE_API_TOKEN` is invalid or lacks Zone:DNS:Edit permission
- DNS zone is not managed by Cloudflare
- Caddy container cannot reach Cloudflare API (firewall/proxy)

Common causes (TLS_MODE=existing):
- `SSL_CERT_PATH` or `SSL_KEY_PATH` doesn't exist on the Unraid host
- Certificate is expired or doesn't match `GITEA_DOMAIN`

**Fix**: Check Caddy container logs: `ssh USER@UNRAID_IP "docker logs caddy"`. For Cloudflare issues, verify the API token at `dash.cloudflare.com`. For existing certs, verify the paths exist and the cert matches the domain.

### SSH to remote host fails

**Symptom**: Preflight check 7 or 8 fails, or any phase fails with "SSH config incomplete".

**Fix**: Verify SSH key-based auth works manually:

```bash
ssh -o BatchMode=yes USER@IP "echo ok"
```

If this fails, add your public key to the remote host's `~/.ssh/authorized_keys`.

### Cross-host SSH fails (Unraid cannot reach Fedora or vice versa)

**Symptom**: Preflight checks 21-22 fail, or `backup/backup_primary.sh` fails on the SCP step.

**Fix**: Run the cross-host SSH setup:

```bash
./setup/cross_host_ssh.sh
```

This generates ed25519 keys on each host and distributes public keys to the other host's `authorized_keys`.

### Docker group membership not active on Fedora

**Symptom**: `setup/fedora.sh` warns "Docker requires sudo" or Phase 2/3 fails with permission errors.

**Fix**: Log out and back in to the Fedora machine (SSH session), then re-run the failed script. Adding a user to the `docker` group requires a new login session to take effect.

### Runner shows as offline after deployment

**Symptom**: Phase 3 post-check reports a runner as offline.

**Fix**:
- For Docker runners: `ssh USER@HOST "docker logs gitea-runner-RUNNER_NAME"` to check logs
- For native runners: `cat ~/gitea-runner/runner.err.log`
- For boot-mode native runners: service management requires `sudo launchctl list | grep com.gitea.runner`
- Common cause: the runner registration token expired. Clear it and re-run Phase 3:

```bash
# In .env, clear:
GITEA_RUNNER_REGISTRATION_TOKEN=
# Then:
./phase3_runners.sh
```

### Workflow files have invalid YAML after migration

**Symptom**: Gitea Actions fails to parse workflows in Phase 5 post-check or when triggering a PR.

**Fix**: Phase 5's `sed` replacements are syntactic — they can't break valid YAML, but the original file may have had GitHub-specific syntax that Gitea doesn't support. Clone the repo and fix the workflow files manually:

```bash
git clone https://YOUR_DOMAIN/ORG/REPO.git
cd REPO
# Edit .gitea/workflows/*.yml
git commit -am "Fix workflow compatibility"
git push
```

---

## Rollback Procedures

### Full rollback (everything)

```bash
./teardown_all.sh
```

Tears down all phases from 9 to 1, prompting for confirmation at each phase. This:
- Removes security workflows from repos
- Restores GitHub repo settings (description, homepage, wiki, projects, Pages)
- Removes Caddy container, config, and TLS data
- Removes branch protection rules
- Removes push mirrors and re-enables GitHub Actions
- Removes `.gitea/workflows/` from repos
- Deletes repos from primary and mirrors from Fedora
- Stops and removes runner containers
- Stops Gitea on both Unraid and Fedora, optionally removes all data

### Non-interactive full rollback

```bash
./teardown_all.sh --yes
```

Answers "yes" to all confirmation prompts. Use for automated testing.

### Partial rollback (keep infrastructure, remove config)

To tear down Phases 5-9 but keep Gitea and repos (Phases 1-4):

```bash
./teardown_all.sh --through=5
```

This removes security workflows, Caddy, branch protection, push mirrors, and Gitea workflows — but leaves the Gitea instances running with the migrated repos.

### Teardown a single phase

```bash
./phase6_teardown.sh
```

Each teardown script is safe to run independently and prompts before destructive actions by default.
For non-interactive execution, use `--yes` (or `-y`):

```bash
./phase6_teardown.sh --yes
```

### Full teardown including prerequisites

```bash
./teardown_all.sh --cleanup
```

After tearing down all phases, this also runs `setup/cleanup.sh` which reads `.manifests/` and uninstalls everything the setup scripts installed (brew packages on macOS, dnf packages on Fedora, static binaries on Unraid, SSH keys from cross-host setup).

### Preview what cleanup would do

```bash
./setup/cleanup.sh --dry-run
```

### Cleanup a specific host

```bash
./setup/cleanup.sh --host=fedora
./setup/cleanup.sh --host=unraid
./setup/cleanup.sh --host=macbook
```

---

## Verifying Rollback Success

After a rollback, run these checks to confirm everything was reversed cleanly.

### After Phase 1-2 teardown (Gitea removed)

```bash
# Verify Gitea is not running
ssh USER@UNRAID_IP "docker ps | grep gitea"     # Should return nothing
ssh USER@FEDORA_IP "docker ps | grep gitea"     # Should return nothing

# Verify data is removed (if you confirmed deletion)
ssh USER@UNRAID_IP "ls /path/to/gitea/data"     # Should fail: No such file
ssh USER@FEDORA_IP "ls /path/to/gitea/data"     # Should fail: No such file

# Verify tokens are cleared
grep "GITEA_ADMIN_TOKEN=" .env                   # Should be empty
grep "GITEA_BACKUP_ADMIN_TOKEN=" .env            # Should be empty
```

### After Phase 3 teardown (runners removed)

```bash
# Docker runners
ssh USER@UNRAID_IP "docker ps | grep gitea-runner"   # Should return nothing
ssh USER@FEDORA_IP "docker ps | grep gitea-runner"   # Should return nothing

# Native runner (macOS) — login agent
launchctl list | grep com.gitea.runner                # Should return nothing
# Native runner (macOS) — boot daemon (if boot=true was used)
sudo launchctl list | grep com.gitea.runner           # Should return nothing

# Registration token cleared
grep "GITEA_RUNNER_REGISTRATION_TOKEN=" .env          # Should be empty
```

### After Phase 4 teardown (repos deleted)

If Gitea is still running (you only tore down Phase 4+), verify via API:

```bash
# Should all return 404
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO_NAME

curl -sf -H "Authorization: token $BACKUP_TOKEN" \
  http://FEDORA_IP:3000/api/v1/repos/ADMIN/REPO_NAME
```

### After Phase 6 teardown (push mirrors removed)

```bash
# Should return empty array []
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO_NAME/push_mirrors

# GitHub Actions should be re-enabled
# Check at: https://github.com/USERNAME/REPO/settings/actions
```

### After Phase 8 teardown (HTTPS removed, GitHub restored)

```bash
# HTTPS should no longer work
curl -sf https://YOUR_DOMAIN/api/v1/version       # Should fail

# Caddy container and config removed
ssh USER@UNRAID_IP "docker ps --filter name=caddy --format '{{.Status}}'"  # Should be empty
ssh USER@UNRAID_IP "test -f /path/to/caddy/Caddyfile && echo exists || echo gone"
# Should print: gone

# GitHub repos should be restored
# Check that description no longer starts with [MIRROR]
curl -sf -H "Authorization: token $GH_TOKEN" \
  https://api.github.com/repos/USERNAME/REPO | jq '.description'
```

### After full teardown + cleanup

```bash
# Verify setup manifests are cleared
ls .manifests/    # Directory should be empty or not exist

# Verify brew packages removed (macOS)
brew list jq      # Should fail if jq was installed by setup

# Verify cross-host SSH keys removed
ssh USER@UNRAID_IP "test -f ~/.ssh/id_ed25519 && echo exists || echo gone"
ssh USER@FEDORA_IP "test -f ~/.ssh/id_ed25519 && echo exists || echo gone"
```

---

## Day-to-Day Operations

### Updating Gitea version

1. Edit `GITEA_VERSION` in `.env`
2. SSH to Unraid: `cd $UNRAID_COMPOSE_DIR/gitea && docker compose pull && docker compose up -d`
3. Repeat on Fedora if you want the backup instance to match

### Rotating API tokens

```bash
# Delete old token in Gitea web UI (Site Administration > Applications)
# Then clear and regenerate:
sed -i 's/^GITEA_ADMIN_TOKEN=.*/GITEA_ADMIN_TOKEN=/' .env
./phase1_gitea_unraid.sh    # Step 7 will generate a new token
```

### Adding a new repository

1. Append the new repo name to `REPO_NAMES` in `.env` (space-separated): `REPO_NAMES="repo1 repo2 new-repo"`
2. Run phases 4-9 (existing repos will be skipped, only the new one gets processed)

### Checking mirror sync status

```bash
# Push mirrors (Gitea to GitHub)
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO/push_mirrors | jq '.[].last_update'

# Trigger manual push sync
curl -sf -X POST -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO/push_mirrors-sync
```

### TLS certificate renewal

When `TLS_MODE=cloudflare`, Caddy handles certificate renewal automatically — no manual action needed. Caddy renews certificates 30 days before expiry.

When `TLS_MODE=existing`, replace the cert/key files at `SSL_CERT_PATH` and `SSL_KEY_PATH` on the Unraid host, then restart Caddy:

```bash
ssh USER@UNRAID_IP "cd $UNRAID_COMPOSE_DIR/caddy && docker compose restart"
```

---

## Backup and Restore

### Creating a backup

```bash
./backup/backup_primary.sh
```

This:
1. Runs `gitea dump` inside the Gitea container on Unraid
2. SCPs the dump directly from Unraid to Fedora (not through MacBook)
3. Verifies archive integrity on Fedora
4. Cleans up the dump from Unraid
5. Prunes old backups beyond `BACKUP_RETENTION_COUNT`

The dump contains the SQLite database (users, tokens, SSH keys, org structure, issues), all git repositories, and the `app.ini` config.

### Restoring from backup

```bash
# From a backup stored on Fedora:
./backup/restore_to_primary.sh --archive /path/on/fedora/gitea-dump-20260228-120000.zip

# From a local file on the MacBook:
./backup/restore_to_primary.sh --archive ./gitea-dump-20260228-120000.zip
```

This is destructive — it replaces all Gitea data on Unraid. The script:
1. Stops the Gitea container
2. Renames the current `data/` directory to `data.pre-restore-TIMESTAMP` as a safety net
3. Extracts the archive
4. Restarts Gitea
5. Verifies admin login
6. Regenerates the API token (old tokens from the dump may be stale)

### Verifying a restore

```bash
./phase1_post_check.sh       # Verify Gitea is running with correct admin
./phase4_post_check.sh       # Verify all repos exist with commits
```

### Scheduling automatic backups

Add a cron job on the MacBook (or any machine with SSH access to both servers):

```bash
# Daily at 2 AM
0 2 * * * cd /path/to/gitea-migration && ./backup/backup_primary.sh >> /var/log/gitea-backup.log 2>&1
```

### Backing up .env to Bitwarden

The `.env` file is gitignored (it contains passwords and tokens) but is needed for ongoing operations — teardown, runner management, backup/restore, and token rotation. If you lose it, you lose the ability to manage the migration.

**Export .env to Bitwarden:**

```bash
./setup/env_to_bitwarden.sh -o bitwarden-import.json
```

This creates a Bitwarden-importable JSON file with a secure note called `gitea-migration-env`. Each `.env` variable becomes a custom field — passwords and tokens are marked as hidden fields.

Import it: Bitwarden Web Vault → Tools → Import Data → Format: Bitwarden (json) → Upload the file. Delete `bitwarden-import.json` from disk afterward.

**Restore .env from Bitwarden:**

```bash
bw unlock                              # unlock your vault first
./setup/bitwarden_to_env.sh --bw       # fetches just the one item
./preflight.sh                         # validate the restored .env
```

The script uses `.env.example` as a template to preserve section headers and comments. It automatically validates all restored values (IP formats, port ranges, email addresses, booleans, etc.) and warns about any failures. Run preflight afterward to confirm connectivity works.

**Important**: Do not use `bw export` (full vault export) to get the data — it dumps your entire vault to a plaintext JSON file on disk. The `--bw` flag fetches only the `gitea-migration-env` item.

**After cleanup**: `teardown_all.sh --cleanup` uninstalls the `bw` CLI from your Mac, but the secure note remains in your Bitwarden vault. Reinstall with `brew install bitwarden-cli` if you need to restore later.

---

## Runner Management

### Listing all runners

```bash
./manage_runner.sh list
```

Output:

```
NAME                 HOST             LABELS     TYPE     CAP    STATUS
----                 ----             ------     ----     ---    ------
unraid-runner        192.168.1.10     linux      docker   2      online
fedora-runner        192.168.1.20     linux      docker   2      idle
macbook-runner       local            macos      native   1      online
```

### Adding a runner

Define it in `runners.conf` (INI format):

```ini
[new-runner]
host     = custom
type     = docker
data_path = /opt/gitea-runner
labels   = linux
default_image = catthehacker/ubuntu:act-latest
repos    = all
capacity = 2
ssh_host = 192.168.1.30
ssh_user = user
ssh_port = 22
```

Then deploy:

```bash
./manage_runner.sh add --name new-runner
```

### Removing a runner

```bash
./manage_runner.sh remove --name new-runner
```

For Docker runners, this stops and removes the container. For native runners, this unloads the launchd service, removes the plist, and optionally deletes the runner data directory. If the native runner was deployed with `boot = true`, removal uses `sudo` for `launchctl unload` and plist deletion.

### Scoping runners to specific repos

By default, runners use `repos = all` (instance-level — available to every repo). To scope a runner to specific repos, use comma-separated names in `runners.conf`:

```ini
[ci-runner]
host     = unraid
type     = docker
repos    = augur,periodvault
# ... other fields
```

`configure_runners.sh` automatically expands this into separate sections (one per repo), since each `act_runner` process can only register to a single repo:

```ini
[ci-runner-augur]
repos    = augur
# ... same config

[ci-runner-periodvault]
repos    = periodvault
# ... same config
```

### Runner types

| Type | Deployed On | How It Works |
|------|------------|--------------|
| `docker` | Linux hosts | Docker Compose container, auto-registers via environment variables |
| `native` | macOS (login) | Binary + launchd agent in `~/Library/LaunchAgents/` — starts when user logs in |
| `native` + `boot=true` | macOS (boot) | Binary + launchd daemon in `/Library/LaunchDaemons/` — starts at boot before login |

### Boot vs login startup (native runners)

Native macOS runners support two startup modes controlled by the `boot` field in `runners.conf`:

| Setting | Plist Location | Starts | Sudo Required | Use Case |
|---------|---------------|--------|---------------|----------|
| `boot = false` (default) | `~/Library/LaunchAgents/` | At user login | No | Developer workstation, shared Mac |
| `boot = true` | `/Library/LaunchDaemons/` | At boot (before login) | Yes | Headless Mac mini, dedicated CI machine |

When `boot = true`, `manage_runner.sh` uses `sudo` for:
- Copying the plist to `/Library/LaunchDaemons/`
- Running `launchctl load` / `launchctl unload`
- Removing the plist on teardown

The plist includes a `<key>UserName</key>` entry so the daemon runs as the deploying user, not root. Log rotation via newsyslog always requires `sudo` regardless of boot mode.

---

## Troubleshooting

### "Missing required var: X"

A required `.env` variable is empty. Open `.env` and fill in the missing value, or re-run `./setup/configure_env.sh`.

### "SSH config incomplete for HOST"

The `*_IP` or `*_SSH_USER` variables for the target host are empty in `.env`.

### "API POST /path returned HTTP 409"

The resource already exists (conflict). This usually means idempotency is working — the script should skip this operation and continue. If it doesn't, check for a bug in the idempotency check.

### "API POST /path returned HTTP 422"

Validation error from Gitea. Common causes:
- Token name already exists (Phase 1/2 token generation after a partial teardown)
- Branch protection rule name conflict
- Invalid JSON payload

Check the error response body in the log output for details.

### "Timeout waiting for HTTP 200 at URL"

Gitea didn't start within the timeout window (120 seconds). Check:
- Docker container logs: `ssh USER@HOST "docker logs gitea"`
- Disk space: `ssh USER@HOST "df -h"`
- Port conflicts: `ssh USER@HOST "ss -tlnp | grep 3000"`

### "curl failed for METHOD URL"

Network-level failure (connection refused, DNS resolution failed, timeout). Verify:
- The target service is running
- Firewall rules allow the connection
- The corresponding container IP in `.env` is correct (`UNRAID_GITEA_IP` or `FEDORA_GITEA_IP`)

### Reading logs

All script output goes to stderr with colored prefixes:
- `[INFO]` — progress messages (blue)
- `[OK]` — success confirmations (green)
- `[WARN]` — non-fatal issues (yellow)
- `[ERROR]` — failures (red)

To capture logs to a file while still seeing them:

```bash
./run_all.sh 2>&1 | tee migration.log
```

### Checking what scripts installed

```bash
# See all install manifests
ls .manifests/

# See what was installed on a specific host
cat .manifests/macbook.manifest
cat .manifests/unraid.manifest
cat .manifests/fedora.manifest

# Preview cleanup without doing it
./setup/cleanup.sh --dry-run
```