diff --git a/README.md b/README.md index cef9bba..805a6fd 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # Gitea Migration Toolkit -Automated migration of GitHub repositories to self-hosted Gitea, with backup mirroring and push-mirror offsite redundancy. 40 shell scripts, 7 config templates, ~6,100 lines of bash. +Automated migration of GitHub repositories to self-hosted Gitea, with backup mirroring and push-mirror offsite redundancy. 40+ shell scripts, 10 config templates, ~6,500 lines of bash. ## What This Does -Moves 3 GitHub repos to a self-hosted Gitea instance on Unraid, sets up a backup Gitea mirror on Fedora, and keeps GitHub as an offsite push mirror. After migration, Gitea is the primary git host — all CI runs on Gitea Actions, GitHub receives automatic push mirrors, and Fedora pulls from Unraid on a schedule. +Moves GitHub repos to a self-hosted Gitea instance on Unraid, sets up a backup Gitea mirror on Fedora, and keeps GitHub as an offsite push mirror. After migration, Gitea is the primary git host — all CI runs on Gitea Actions, GitHub receives automatic push mirrors, and Fedora pulls from Unraid on a schedule. Supports any number of repos via space-delimited `REPO_NAMES` in `.env`. The entire process is driven from a MacBook over SSH. Nothing is installed on the remote machines beyond what the setup scripts explicitly provision. @@ -19,9 +19,9 @@ The entire process is driven from a MacBook over SSH. Nothing is installed on th │ SSH │ SSH ┌──────────▼──────────┐ ┌────▼───────────────┐ │ Unraid (Primary) │ │ Fedora (Backup) │ - │ Gitea + Nginx │ │ Gitea (mirror) │ + │ Gitea + Caddy │ │ Gitea (mirror) │ │ Docker runners │ │ Docker runners │ - │ Let's Encrypt │ │ Backup storage │ + │ macvlan networking │ │ Backup storage │ └──────────┬──────────┘ └────▲───────────────┘ │ │ │ pull mirror │ @@ -54,7 +54,7 @@ The entire process is driven from a MacBook over SSH. Nothing is installed on th | 5 | `phase5_migrate_pipelines.sh` | Copy `.github/workflows/` to `.gitea/workflows/`, apply context variable fixes | | 6 | `phase6_github_mirrors.sh` | Configure push mirrors from Gitea to GitHub, disable GitHub Actions | | 7 | `phase7_branch_protection.sh` | Apply branch protection rules to all repos | -| 8 | `phase8_cutover.sh` | Deploy Nginx HTTPS reverse proxy, obtain SSL cert, mark GitHub repos as mirrors | +| 8 | `phase8_cutover.sh` | Deploy Caddy HTTPS reverse proxy (Cloudflare DNS-01 or existing certs), mark GitHub repos as mirrors | | 9 | `phase9_security.sh` | Deploy Semgrep + Trivy + Gitleaks security scanning workflows | Each phase has three scripts: the main script, a `_post_check.sh` that independently verifies success, and a `_teardown.sh` that cleanly reverses the phase. @@ -79,9 +79,11 @@ gitea-migration/ │ ├── app.ini.tpl │ ├── docker-compose-gitea.yml.tpl │ ├── docker-compose-runner.yml.tpl -│ ├── nginx-gitea.conf.tpl +│ ├── Caddyfile.tpl +│ ├── docker-compose-caddy.yml.tpl │ ├── runner-config.yaml.tpl │ ├── com.gitea.runner.plist.tpl +│ ├── com.gitea.runner.newsyslog.conf.tpl │ └── workflows/security-scan.yml.tpl ├── contracts/gitea-api.md # API contract documentation ├── backup/ @@ -100,7 +102,7 @@ gitea-migration/ ### Why bash scripts instead of Ansible/Terraform/Pulumi? -The migration targets 3 repos across 3 machines with a one-time execution path. Ansible requires installing agents or running a control node; Terraform manages ongoing state that doesn't apply to a one-shot migration; Pulumi requires a runtime. Bash scripts with SSH are zero-dependency beyond what's already on a Mac, run anywhere, are readable without framework knowledge, and produce no ongoing state to manage. The downside is more verbose error handling and no built-in parallelism, but for a sequential 9-phase pipeline that's acceptable. +The migration targets a handful of repos across 3 machines with a one-time execution path. Ansible requires installing agents or running a control node; Terraform manages ongoing state that doesn't apply to a one-shot migration; Pulumi requires a runtime. Bash scripts with SSH are zero-dependency beyond what's already on a Mac, run anywhere, are readable without framework knowledge, and produce no ongoing state to manage. The downside is more verbose error handling and no built-in parallelism, but for a sequential 9-phase pipeline that's acceptable. ### Why a single MacBook control plane? @@ -116,7 +118,7 @@ Docker Desktop on macOS is heavyweight (~4 GB), requires a commercial license fo ### Why `envsubst` templates instead of Jinja2/Helm/gomplate? -`envsubst` is a single binary from GNU gettext with zero dependencies. Templates are plain config files with `${VAR}` placeholders — anyone can read them without learning a template language. The trade-off is no conditionals or loops in templates. The scripts work around this by generating template variants in bash (e.g., HTTP-only vs HTTPS Nginx configs use marker-block stripping with `sed`). +`envsubst` is a single binary from GNU gettext with zero dependencies. Templates are plain config files with `${VAR}` placeholders — anyone can read them without learning a template language. The trade-off is no conditionals or loops in templates. The scripts work around this by using marker-block stripping with `sed` (e.g., sqlite3 vs external DB blocks in the docker-compose template). ### Why check-before-act idempotency instead of desired-state? @@ -126,9 +128,9 @@ Every operation checks if its target already exists before creating it. This is All four Gitea-supported database backends are available: `sqlite3`, `mysql`, `postgres`, and `mssql`. Set `GITEA_DB_TYPE` in `.env` — sqlite3 is the default and needs no additional configuration. For external databases, the toolkit deploys a containerized database alongside Gitea (PostgreSQL 16, MySQL 8.0, or MSSQL 2022) with health checks, and the wizard prompts for connection details (host, port, name, user, password) only when needed. Backup/restore handles SQL dump import into the correct database engine. -### Why Nginx reverse proxy instead of Caddy/Traefik? +### Why Caddy reverse proxy? -An Nginx Docker container was already running on Unraid. Adding a server block and SSL cert to an existing Nginx is simpler than deploying a new reverse proxy. Caddy has simpler cert management but would require replacing the existing proxy stack. +Caddy with the Cloudflare DNS plugin handles wildcard TLS certificates automatically via DNS-01 challenge — no port 80 exposure needed, no certbot cron jobs, and zero-touch renewal. The `slothcroissant/caddy-cloudflaredns` Docker image bundles the plugin. For environments without Cloudflare, `TLS_MODE=existing` supports manual cert/key paths. Each host gets its own Caddy container on a dedicated macvlan IP. ### Why mark GitHub repos as mirrors instead of archiving them? @@ -166,7 +168,7 @@ Phase 5 copies workflow files and does a `sed` replacement of `github.*` context - Migrate secrets, OIDC providers, or environment configurations - Handle composite actions or reusable workflows -Full semantic migration would require parsing YAML, understanding the GitHub Actions schema, and mapping every action to a Gitea equivalent. For 3 repos, manual review after automated migration is faster than building a full converter. +Full semantic migration would require parsing YAML, understanding the GitHub Actions schema, and mapping every action to a Gitea equivalent. For a small number of repos, manual review after automated migration is faster than building a full converter. ### No automatic rollback on failure @@ -181,7 +183,7 @@ Phase 4 polls the Gitea API every N seconds to check if a migration completed, w ### No parallel phase execution -Phases run strictly sequentially. Phase 4 could potentially import all 3 repos in parallel, and Phase 3 could deploy runners concurrently. Sequential execution was chosen because: +Phases run strictly sequentially. Phase 4 could potentially import repos in parallel, and Phase 3 could deploy runners concurrently. Sequential execution was chosen because: - Bash parallelism (`&` + `wait`) makes error handling complex - The total migration time is dominated by network transfers, not script execution - Sequential execution produces readable, linear logs @@ -202,9 +204,9 @@ When `boot = true` is set in `runners.conf`, `manage_runner.sh` uses `sudo` for The JSON file that records pre-cutover GitHub repo settings is stored alongside install manifests in `.manifests/`. This directory is gitignored (machine-specific state). If the user deletes `.manifests/` before running Phase 8 teardown, the teardown falls back to parsing the original description from the `[MIRROR] ... — was: ORIGINAL` format, but cannot restore homepage, wiki, projects, or Pages settings. -### SSL renewal cron on Unraid may not survive reboots +### TLS certificate renewal -The Let's Encrypt renewal cron is added via `crontab` on Unraid. Unraid is not designed for persistent user crontabs — they can be lost on reboot depending on the Unraid version and configuration. A more robust approach would be a dedicated Certbot Docker container with a restart policy, but that adds deployment complexity. +When `TLS_MODE=cloudflare`, Caddy handles certificate renewal automatically via the Cloudflare DNS-01 challenge — no cron jobs or manual intervention needed. Caddy renews certificates 30 days before expiry and persists them in `$CADDY_DATA_PATH/data`. When `TLS_MODE=existing`, cert renewal is the user's responsibility. ## Security Notes @@ -220,9 +222,9 @@ The Let's Encrypt renewal cron is added via `crontab` on Unraid. Unraid is not d | Machine | Requirements | |---------|-------------| | MacBook | macOS, Homebrew, jq >= 1.6, curl >= 7.70, git >= 2.30, shellcheck >= 0.8, gh >= 2.0, bw >= 2.0 | -| Unraid | Linux, Docker >= 20.0, docker-compose >= 2.0, jq >= 1.6, existing Nginx container | +| Unraid | Linux, Docker >= 20.0, docker-compose >= 2.0, jq >= 1.6 | | Fedora | Linux with dnf, Docker CE >= 20.0, docker-compose >= 2.0, jq >= 1.6 | -| Network | MacBook can SSH to both servers, DNS A record pointing to Unraid for HTTPS | +| Network | MacBook can SSH to both servers, DNS A record pointing to Unraid for HTTPS, Cloudflare API token (if using `TLS_MODE=cloudflare`) | ## Quick Start