gitea-migration/USAGE_GUIDE.md

# Usage Guide

Step-by-step instructions for running the migration, handling failures, rolling back, and day-to-day operations.

---

## Table of Contents

1. [Before You Start](#before-you-start)
2. [Happy Path: Full Migration](#happy-path-full-migration)
3. [Resuming After Failure](#resuming-after-failure)
4. [Edge Cases](#edge-cases)
5. [Rollback Procedures](#rollback-procedures)
6. [Verifying Rollback Success](#verifying-rollback-success)
7. [Day-to-Day Operations](#day-to-day-operations)
8. [Backup and Restore](#backup-and-restore)
9. [Runner Management](#runner-management)
10. [Troubleshooting](#troubleshooting)

---

## Before You Start

### 1. Network and DNS

Before running anything, confirm:

- MacBook can SSH to Unraid: `ssh user@UNRAID_IP`
- MacBook can SSH to Fedora: `ssh user@FEDORA_IP`
- A DNS A record exists for your Gitea domain pointing to `UNRAID_IP`
- Ports 3000 (Gitea web) and 2222 (Gitea SSH) are free on both servers
- An Nginx Docker container is already running on Unraid

### 2. GitHub Tokens

You need two GitHub Personal Access Tokens:

| Token | Scope | Used By |
|-------|-------|---------|
| `GITHUB_TOKEN` | `repo` (read) | Preflight validation, repo migration |
| `GITHUB_MIRROR_TOKEN` | `repo` (read+write) | Push mirrors from Gitea to GitHub |

These can be the same token if it has write scope.

### 3. Configuration

```bash
# Option A: Interactive wizard (recommended for first time)
./setup/configure_env.sh

# Option B: Manual
cp .env.example .env
cp runners.conf.example runners.conf
# Edit both files with your values
```

The wizard validates every input (IP format, port ranges, URL format, password length) and shows your current values in brackets so you can press Enter to keep them.

---

## Happy Path: Full Migration

### Automated (recommended)

```bash
./run_all.sh
```

This runs the full pipeline: setup scripts, preflight checks, and all 9 phases with post-checks after each phase. It stops on the first failure and prints a summary of what completed.

Expected output on success:

```
>>> Running: Setup MacBook
>>> Running: Setup Unraid
>>> Running: Setup Fedora
>>> Running: Cross-host SSH trust
>>> Running: Preflight checks
>>> Running: Phase 1: Gitea on Unraid
>>> Running: Phase 1 — post-check
>>> Running: Phase 2: Gitea on Fedora
>>> Running: Phase 2 — post-check
...
>>> Running: Phase 9: Security
>>> Running: Phase 9 — post-check

===== Execution Summary =====
  Setup MacBook
  Setup Unraid
  Setup Fedora
  Cross-host SSH trust
  Preflight checks
  Phase 1: Gitea on Unraid
  Phase 1 — post-check
  ...
  Phase 9: Security
  Phase 9 — post-check

Migration complete! Gitea is live.
```

### Manual (phase by phase)

If you prefer to run each phase individually and inspect results:

```bash
# 1. Setup (once)
./setup/configure_env.sh
./setup/macbook.sh
./setup/unraid.sh
./setup/fedora.sh
./setup/cross_host_ssh.sh

# 2. Validate
./preflight.sh

# 3. Run each phase
./phase1_gitea_unraid.sh && ./phase1_post_check.sh
./phase2_gitea_fedora.sh && ./phase2_post_check.sh
./phase3_runners.sh       && ./phase3_post_check.sh
./phase4_migrate_repos.sh && ./phase4_post_check.sh
./phase5_migrate_pipelines.sh && ./phase5_post_check.sh
./phase6_github_mirrors.sh    && ./phase6_post_check.sh
./phase7_branch_protection.sh && ./phase7_post_check.sh
./phase8_cutover.sh           && ./phase8_post_check.sh
./phase9_security.sh          && ./phase9_post_check.sh
```

### Skip setup (already done)

```bash
./run_all.sh --skip-setup
```

### What to verify when it's done

After the full migration completes:

1. **HTTPS access**: Open `https://YOUR_DOMAIN` in a browser — you should see the Gitea login page with a valid SSL certificate.
2. **Repository content**: Log in as admin, navigate to your org, confirm all 3 repos have commits, branches, and (if enabled) issues/labels.
3. **CI pipelines**: Create a test PR in one repo. Gitea Actions should run the migrated workflows. If Phase 9 ran, security scans (Semgrep, Trivy, Gitleaks) should also trigger.
4. **GitHub mirrors**: Check your GitHub repos — the description should start with `[MIRROR]` and the latest commits should match Gitea.
5. **Fedora mirrors**: SSH into the Fedora Gitea instance and verify repos exist under the admin user namespace.
6. **Runners**: Run `./manage_runner.sh list` — all runners should show status `online` or `idle`.

---

## Resuming After Failure

### Starting from a specific phase

If Phase 5 failed but Phases 1-4 completed successfully:

```bash
./run_all.sh --start-from=5
```

This runs preflight (with `--skip-port-checks` since Gitea is already running on those ports), then phases 5 through 9.

### Re-running a single phase

Every phase is idempotent. If Phase 4 failed on the second repo, just re-run it:

```bash
./phase4_migrate_repos.sh
```

It will skip the first repo (already exists) and retry the second.

### Re-running a post-check

Post-checks are read-only and can be run at any time:

```bash
./phase4_post_check.sh
```

### Preflight after partial migration

When resuming from a later phase, Gitea is already running on ports 3000. Use:

```bash
./preflight.sh --skip-port-checks
```

`run_all.sh --start-from=N` does this automatically when N > 1.

---

## Edge Cases

### GitHub API rate limit hit during migration

**Symptom**: Phase 4 or Phase 6 fails with HTTP 403 and a message about rate limits.

**Fix**: Wait for the rate limit window to reset (usually 1 hour), then re-run the failed phase. The phase is idempotent — repos that already migrated will be skipped.

### GitHub token expires mid-migration

**Symptom**: API calls fail with HTTP 401.

**Fix**: Generate a new token on GitHub, update `GITHUB_TOKEN` (and/or `GITHUB_MIRROR_TOKEN`) in `.env`, then re-run the failed phase.

### Large repos time out during migration

**Symptom**: Phase 4 logs "Timeout waiting for REPO migration to complete".

**Fix**: Increase the timeout in `.env`:

```bash
MIGRATION_POLL_TIMEOUT_SEC=1800    # 30 minutes instead of default 10
```

Then re-run Phase 4. Already-migrated repos will be skipped.

### Port already in use on Unraid/Fedora

**Symptom**: Preflight check 13 or 14 fails with "Port 3000 already in use".

**Fix**: Either stop whatever is using port 3000, or change `UNRAID_GITEA_PORT` / `FEDORA_GITEA_PORT` in `.env` to use a different port.

### DNS doesn't resolve to Unraid IP

**Symptom**: Preflight check 15 fails.

**Fix**: Add or update your DNS A record. If using a local DNS server or `/etc/hosts`, ensure the record points to `UNRAID_IP`. DNS propagation can take minutes to hours.

### Nginx config test fails in Phase 8

**Symptom**: Phase 8 aborts with "Nginx config test failed".

**Fix**: The script automatically removes the bad config file. Check that `NGINX_CONF_PATH` in `.env` points to the correct `conf.d` directory inside the Nginx container's volume mount. Verify the Nginx container has `/etc/letsencrypt` mounted if using Let's Encrypt.

### Certbot fails (Let's Encrypt)

**Symptom**: Phase 8 fails at "Running Certbot".

Common causes:
- Port 80 is not reachable from the internet (firewall, router, ISP blocking)
- DNS doesn't resolve to the server's public IP
- Nginx container doesn't mount `/etc/letsencrypt` and `/var/www/html`
- Rate limit: Let's Encrypt allows 5 certs per domain per week

**Fix**: Verify port 80 is open (`curl http://YOUR_DOMAIN/.well-known/acme-challenge/test`), fix the DNS, or add the volume mounts to the Nginx container. If rate limited, wait or use `SSL_MODE=existing` with a cert from another provider.

### SSH to remote host fails

**Symptom**: Preflight check 7 or 8 fails, or any phase fails with "SSH config incomplete".

**Fix**: Verify SSH key-based auth works manually:

```bash
ssh -o BatchMode=yes USER@IP "echo ok"
```

If this fails, add your public key to the remote host's `~/.ssh/authorized_keys`.

### Cross-host SSH fails (Unraid cannot reach Fedora or vice versa)

**Symptom**: Preflight checks 23-24 fail, or `backup/backup_primary.sh` fails on the SCP step.

**Fix**: Run the cross-host SSH setup:

```bash
./setup/cross_host_ssh.sh
```

This generates ed25519 keys on each host and distributes public keys to the other host's `authorized_keys`.

### Docker group membership not active on Fedora

**Symptom**: `setup/fedora.sh` warns "Docker requires sudo" or Phase 2/3 fails with permission errors.

**Fix**: Log out and back in to the Fedora machine (SSH session), then re-run the failed script. Adding a user to the `docker` group requires a new login session to take effect.

### Runner shows as offline after deployment

**Symptom**: Phase 3 post-check reports a runner as offline.

**Fix**:
- For Docker runners: `ssh USER@HOST "docker logs gitea-runner-RUNNER_NAME"` to check logs
- For native runners: `cat ~/gitea-runner/runner.err.log`
- Common cause: the runner registration token expired. Clear it and re-run Phase 3:

```bash
# In .env, clear:
GITEA_RUNNER_REGISTRATION_TOKEN=
# Then:
./phase3_runners.sh
```

### Workflow files have invalid YAML after migration

**Symptom**: Gitea Actions fails to parse workflows in Phase 5 post-check or when triggering a PR.

**Fix**: Phase 5's `sed` replacements are syntactic — they can't break valid YAML, but the original file may have had GitHub-specific syntax that Gitea doesn't support. Clone the repo and fix the workflow files manually:

```bash
git clone https://YOUR_DOMAIN/ORG/REPO.git
cd REPO
# Edit .gitea/workflows/*.yml
git commit -am "Fix workflow compatibility"
git push
```

---

## Rollback Procedures

### Full rollback (everything)

```bash
./teardown_all.sh
```

Tears down all phases from 9 to 1, prompting for confirmation at each phase. This:
- Removes security workflows from repos
- Restores GitHub repo settings (description, homepage, wiki, projects, Pages)
- Removes Nginx config and SSL certs
- Removes branch protection rules
- Removes push mirrors and re-enables GitHub Actions
- Removes `.gitea/workflows/` from repos
- Deletes repos from primary and mirrors from Fedora
- Stops and removes runner containers
- Stops Gitea on both Unraid and Fedora, optionally removes all data

### Non-interactive full rollback

```bash
./teardown_all.sh --yes
```

Answers "yes" to all confirmation prompts. Use for automated testing.

### Partial rollback (keep infrastructure, remove config)

To tear down Phases 5-9 but keep Gitea and repos (Phases 1-4):

```bash
./teardown_all.sh --through=5
```

This removes security workflows, Nginx, branch protection, push mirrors, and Gitea workflows — but leaves the Gitea instances running with the migrated repos.

### Teardown a single phase

```bash
./phase6_teardown.sh
```

Each teardown script is safe to run independently and prompts before destructive actions.

### Full teardown including prerequisites

```bash
./teardown_all.sh --cleanup
```

After tearing down all phases, this also runs `setup/cleanup.sh` which reads `.manifests/` and uninstalls everything the setup scripts installed (brew packages on macOS, dnf packages on Fedora, static binaries on Unraid, SSH keys from cross-host setup).

### Preview what cleanup would do

```bash
./setup/cleanup.sh --dry-run
```

### Cleanup a specific host

```bash
./setup/cleanup.sh --host=fedora
./setup/cleanup.sh --host=unraid
./setup/cleanup.sh --host=macbook
```

---

## Verifying Rollback Success

After a rollback, run these checks to confirm everything was reversed cleanly.

### After Phase 1-2 teardown (Gitea removed)

```bash
# Verify Gitea is not running
ssh USER@UNRAID_IP "docker ps | grep gitea"     # Should return nothing
ssh USER@FEDORA_IP "docker ps | grep gitea"     # Should return nothing

# Verify data is removed (if you confirmed deletion)
ssh USER@UNRAID_IP "ls /path/to/gitea/data"     # Should fail: No such file
ssh USER@FEDORA_IP "ls /path/to/gitea/data"     # Should fail: No such file

# Verify tokens are cleared
grep "GITEA_ADMIN_TOKEN=" .env                   # Should be empty
grep "GITEA_BACKUP_ADMIN_TOKEN=" .env            # Should be empty
```

### After Phase 3 teardown (runners removed)

```bash
# Docker runners
ssh USER@UNRAID_IP "docker ps | grep gitea-runner"   # Should return nothing
ssh USER@FEDORA_IP "docker ps | grep gitea-runner"   # Should return nothing

# Native runner (macOS)
launchctl list | grep com.gitea.runner                # Should return nothing

# Registration token cleared
grep "GITEA_RUNNER_REGISTRATION_TOKEN=" .env          # Should be empty
```

### After Phase 4 teardown (repos deleted)

If Gitea is still running (you only tore down Phase 4+), verify via API:

```bash
# Should all return 404
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO_NAME

curl -sf -H "Authorization: token $BACKUP_TOKEN" \
  http://FEDORA_IP:3000/api/v1/repos/ADMIN/REPO_NAME
```

### After Phase 6 teardown (push mirrors removed)

```bash
# Should return empty array []
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO_NAME/push_mirrors

# GitHub Actions should be re-enabled
# Check at: https://github.com/USERNAME/REPO/settings/actions
```

### After Phase 8 teardown (HTTPS removed, GitHub restored)

```bash
# HTTPS should no longer work
curl -sf https://YOUR_DOMAIN/api/v1/version       # Should fail

# Nginx config removed
ssh USER@UNRAID_IP "test -f /path/to/nginx/conf.d/gitea.conf && echo exists || echo gone"
# Should print: gone

# GitHub repos should be restored
# Check that description no longer starts with [MIRROR]
curl -sf -H "Authorization: token $GH_TOKEN" \
  https://api.github.com/repos/USERNAME/REPO | jq '.description'
```

### After full teardown + cleanup

```bash
# Verify setup manifests are cleared
ls .manifests/    # Directory should be empty or not exist

# Verify brew packages removed (macOS)
brew list jq      # Should fail if jq was installed by setup

# Verify cross-host SSH keys removed
ssh USER@UNRAID_IP "test -f ~/.ssh/id_ed25519 && echo exists || echo gone"
ssh USER@FEDORA_IP "test -f ~/.ssh/id_ed25519 && echo exists || echo gone"
```

---

## Day-to-Day Operations

### Updating Gitea version

1. Edit `GITEA_VERSION` in `.env`
2. SSH to Unraid: `cd /path/to/gitea && docker compose pull && docker compose up -d`
3. Repeat on Fedora if you want the backup instance to match

### Rotating API tokens

```bash
# Delete old token in Gitea web UI (Site Administration > Applications)
# Then clear and regenerate:
sed -i 's/^GITEA_ADMIN_TOKEN=.*/GITEA_ADMIN_TOKEN=/' .env
./phase1_gitea_unraid.sh    # Step 7 will generate a new token
```

### Adding a new repository

1. Add `REPO_4_NAME=new-repo` to `.env`
2. Add `"$REPO_4_NAME"` to the `REPOS=()` array in phases 4-9
3. Run phases 4-9 for the new repo (existing repos will be skipped)

### Checking mirror sync status

```bash
# Push mirrors (Gitea to GitHub)
curl -sf -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO/push_mirrors | jq '.[].last_update'

# Trigger manual push sync
curl -sf -X POST -H "Authorization: token $TOKEN" \
  http://UNRAID_IP:3000/api/v1/repos/ORG/REPO/push_mirrors-sync
```

### Renewing SSL certificate manually

```bash
ssh USER@UNRAID_IP "docker run --rm \
  -v /etc/letsencrypt:/etc/letsencrypt \
  -v /var/www/html:/var/www/html \
  certbot/certbot renew --quiet && \
  docker exec NGINX_CONTAINER nginx -s reload"
```

---

## Backup and Restore

### Creating a backup

```bash
./backup/backup_primary.sh
```

This:
1. Runs `gitea dump` inside the Gitea container on Unraid
2. SCPs the dump directly from Unraid to Fedora (not through MacBook)
3. Verifies archive integrity on Fedora
4. Cleans up the dump from Unraid
5. Prunes old backups beyond `BACKUP_RETENTION_COUNT`

The dump contains the SQLite database (users, tokens, SSH keys, org structure, issues), all git repositories, and the `app.ini` config.

### Restoring from backup

```bash
# From a backup stored on Fedora:
./backup/restore_to_primary.sh --archive /path/on/fedora/gitea-dump-20260228-120000.zip

# From a local file on the MacBook:
./backup/restore_to_primary.sh --archive ./gitea-dump-20260228-120000.zip
```

This is destructive — it replaces all Gitea data on Unraid. The script:
1. Stops the Gitea container
2. Renames the current `data/` directory to `data.pre-restore-TIMESTAMP` as a safety net
3. Extracts the archive
4. Restarts Gitea
5. Verifies admin login
6. Regenerates the API token (old tokens from the dump may be stale)

### Verifying a restore

```bash
./phase1_post_check.sh       # Verify Gitea is running with correct admin
./phase4_post_check.sh       # Verify all repos exist with commits
```

### Scheduling automatic backups

Add a cron job on the MacBook (or any machine with SSH access to both servers):

```bash
# Daily at 2 AM
0 2 * * * cd /path/to/gitea-migration && ./backup/backup_primary.sh >> /var/log/gitea-backup.log 2>&1
```

### Backing up .env to Bitwarden

The `.env` file is gitignored (it contains passwords and tokens) but is needed for ongoing operations — teardown, runner management, backup/restore, and token rotation. If you lose it, you lose the ability to manage the migration.

**Export .env to Bitwarden:**

```bash
./setup/env_to_bitwarden.sh -o bitwarden-import.json
```

This creates a Bitwarden-importable JSON file with a secure note called `gitea-migration-env`. Each `.env` variable becomes a custom field — passwords and tokens are marked as hidden fields.

Import it: Bitwarden Web Vault → Tools → Import Data → Format: Bitwarden (json) → Upload the file. Delete `bitwarden-import.json` from disk afterward.

**Restore .env from Bitwarden:**

```bash
bw unlock                              # unlock your vault first
./setup/bitwarden_to_env.sh --bw       # fetches just the one item
./preflight.sh                         # validate the restored .env
```

The script uses `.env.example` as a template to preserve section headers and comments. Run preflight afterward to confirm all variables are correct and connectivity works.

**Important**: Do not use `bw export` (full vault export) to get the data — it dumps your entire vault to a plaintext JSON file on disk. The `--bw` flag fetches only the `gitea-migration-env` item.

**After cleanup**: `teardown_all.sh --cleanup` uninstalls the `bw` CLI from your Mac, but the secure note remains in your Bitwarden vault. Reinstall with `brew install bitwarden-cli` if you need to restore later.

---

## Runner Management

### Listing all runners

```bash
./manage_runner.sh list
```

Output:

```
NAME                 HOST             LABELS     TYPE     STATUS
----                 ----             ------     ----     ------
unraid-runner        192.168.1.10     linux      docker   online
fedora-runner        192.168.1.20     linux      docker   idle
macbook-runner       local            macos      native   online
```

### Adding a runner

Define it in `runners.conf`:

```
new-runner|192.168.1.30|user|22|/opt/gitea-runner|linux|docker
```

Then deploy:

```bash
./manage_runner.sh add --name new-runner
```

### Removing a runner

```bash
./manage_runner.sh remove --name new-runner
```

For Docker runners, this stops and removes the container. For native runners, this unloads the launchd service, removes the plist, and optionally deletes the runner data directory.

### Runner types

| Type | Deployed On | How It Works |
|------|------------|--------------|
| `docker` | Linux hosts | Docker Compose container, auto-registers via environment variables |
| `native` | macOS | Binary downloaded from Gitea, registered manually, managed via launchd plist |

---

## Troubleshooting

### "Missing required var: X"

A required `.env` variable is empty. Open `.env` and fill in the missing value, or re-run `./setup/configure_env.sh`.

### "SSH config incomplete for HOST"

The `*_IP` or `*_SSH_USER` variables for the target host are empty in `.env`.

### "API POST /path returned HTTP 409"

The resource already exists (conflict). This usually means idempotency is working — the script should skip this operation and continue. If it doesn't, check for a bug in the idempotency check.

### "API POST /path returned HTTP 422"

Validation error from Gitea. Common causes:
- Token name already exists (Phase 1/2 token generation after a partial teardown)
- Branch protection rule name conflict
- Invalid JSON payload

Check the error response body in the log output for details.

### "Timeout waiting for HTTP 200 at URL"

Gitea didn't start within the timeout window (120 seconds). Check:
- Docker container logs: `ssh USER@HOST "docker logs gitea"`
- Disk space: `ssh USER@HOST "df -h"`
- Port conflicts: `ssh USER@HOST "ss -tlnp | grep 3000"`

### "curl failed for METHOD URL"

Network-level failure (connection refused, DNS resolution failed, timeout). Verify:
- The target service is running
- Firewall rules allow the connection
- The URL in `.env` is correct (check for typos in IP/port)

### Reading logs

All script output goes to stderr with colored prefixes:
- `[INFO]` — progress messages (blue)
- `[OK]` — success confirmations (green)
- `[WARN]` — non-fatal issues (yellow)
- `[ERROR]` — failures (red)

To capture logs to a file while still seeing them:

```bash
./run_all.sh 2>&1 | tee migration.log
```

### Checking what scripts installed

```bash
# See all install manifests
ls .manifests/

# See what was installed on a specific host
cat .manifests/macbook.manifest
cat .manifests/unraid.manifest
cat .manifests/fedora.manifest

# Preview cleanup without doing it
./setup/cleanup.sh --dry-run
```