docs: fix stale counts and preflight check numbering
README.md: 10 templates→9, ~6500 lines→~8000, 25 checks→22, ~50 prompts→~65 CLAUDE.md: ~55 env vars→~70 PLAN.md: mark all tracker items DONE, fix Nginx→Caddy/SSL→TLS refs, add missing setup scripts and template entries USAGE_GUIDE.md: add configure_runners.sh mention, fix check 15→14 ref preflight.sh: renumber checks to fill gaps at 14 and 19 (now 1-22) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -42,7 +42,7 @@ backup/ # Backup and restore scripts
|
|||||||
- `manage_runner.sh add|remove|list` — Dynamic runner management
|
- `manage_runner.sh add|remove|list` — Dynamic runner management
|
||||||
|
|
||||||
## .env Validation
|
## .env Validation
|
||||||
`validate_env()` in `lib/common.sh` checks all ~55 .env variables against their expected format (IP, port, email, path, URL, bool, integer, password, tls_mode, db_type). Uses parallel arrays for the variable-to-validator mapping (bash 3.2 compatible). Called by `preflight.sh` and `bitwarden_to_env.sh`. `configure_env.sh` uses the same individual validators interactively.
|
`validate_env()` in `lib/common.sh` checks all ~70 .env variables against their expected format (IP, port, email, path, URL, bool, integer, password, tls_mode, db_type). Uses parallel arrays for the variable-to-validator mapping (bash 3.2 compatible). Called by `preflight.sh` and `bitwarden_to_env.sh`. `configure_env.sh` uses the same individual validators interactively.
|
||||||
|
|
||||||
## Version Checking
|
## Version Checking
|
||||||
Setup scripts and preflight validate minimum versions for all tools:
|
Setup scripts and preflight validate minimum versions for all tools:
|
||||||
|
|||||||
122
PLAN.md
122
PLAN.md
@@ -96,138 +96,144 @@ gitea-migration/
|
|||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 1.1 | `lib/common.sh` | Shared functions: logging, SSH, API wrappers, template rendering, checks | TODO |
|
| 1.1 | `lib/common.sh` | Shared functions: logging, SSH, API wrappers, template rendering, checks | DONE |
|
||||||
| 1.2 | `.env.example` | Add SSL_MODE, SSL_CERT_PATH, SSL_KEY_PATH vars to Nginx section | TODO |
|
| 1.2 | `.env.example` | All env vars: TLS_MODE, macvlan networking, DB support, Caddy config | DONE |
|
||||||
| 1.3 | `contracts/gitea-api.md` | Gitea REST API endpoints used across all phases | TODO |
|
| 1.3 | `contracts/gitea-api.md` | Gitea REST API endpoints used across all phases | DONE |
|
||||||
|
|
||||||
### 2. Templates
|
### 2. Templates
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 2.1 | `templates/docker-compose-gitea.yml.tpl` | Gitea + SQLite docker-compose | TODO |
|
| 2.1 | `templates/docker-compose-gitea.yml.tpl` | Gitea + DB docker-compose (sqlite3/mysql/postgres/mssql) | DONE |
|
||||||
| 2.2 | `templates/app.ini.tpl` | Gitea custom config (INSTALL_LOCK, Actions enabled, etc.) | TODO |
|
| 2.2 | `templates/app.ini.tpl` | Gitea custom config (INSTALL_LOCK, Actions enabled, etc.) | DONE |
|
||||||
| 2.3 | `templates/docker-compose-runner.yml.tpl` | act_runner docker-compose (Linux) | TODO |
|
| 2.3 | `templates/docker-compose-runner.yml.tpl` | act_runner docker-compose (Linux) | DONE |
|
||||||
| 2.4 | `templates/runner-config.yaml.tpl` | act_runner config | TODO |
|
| 2.4 | `templates/runner-config.yaml.tpl` | act_runner config | DONE |
|
||||||
| 2.5 | `templates/com.gitea.runner.plist.tpl` | macOS launchd service for act_runner | TODO |
|
| 2.5 | `templates/com.gitea.runner.plist.tpl` | macOS launchd service for act_runner | DONE |
|
||||||
| 2.6 | `templates/nginx-gitea.conf.tpl` | Nginx reverse proxy server block | TODO |
|
| 2.6 | `templates/com.gitea.runner.newsyslog.conf.tpl` | macOS log rotation for native runner | DONE |
|
||||||
| 2.7 | `templates/workflows/security-scan.yml.tpl` | Semgrep + Trivy + Gitleaks workflow | TODO |
|
| 2.7 | `templates/Caddyfile.tpl` + `docker-compose-caddy.yml.tpl` | Caddy reverse proxy with Cloudflare DNS-01 | DONE |
|
||||||
|
| 2.8 | `templates/workflows/security-scan.yml.tpl` | Semgrep + Trivy + Gitleaks workflow | DONE |
|
||||||
|
|
||||||
### 3. Machine Setup
|
### 3. Machine Setup
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 3.1 | `setup/configure_env.sh` | Interactive wizard: prompts for each .env var, writes to .env | TODO |
|
| 3.1 | `setup/configure_env.sh` | Interactive wizard: prompts for each .env var, writes to .env | DONE |
|
||||||
| 3.2 | `setup/macbook.sh` | Homebrew, jq, curl, envsubst, git, Xcode CLI Tools, shellcheck, gh | TODO |
|
| 3.2 | `setup/macbook.sh` | Homebrew, jq, curl, envsubst, git, Xcode CLI Tools, shellcheck, gh | DONE |
|
||||||
| 3.3 | `setup/unraid.sh` | Verify Docker, install docker-compose + jq (static binary) | TODO |
|
| 3.3 | `setup/unraid.sh` | Verify Docker, install docker-compose + jq (static binary) | DONE |
|
||||||
| 3.4 | `setup/fedora.sh` | Install Docker CE, compose plugin, jq, enable systemd services | TODO |
|
| 3.4 | `setup/fedora.sh` | Install Docker CE, compose plugin, jq, enable systemd services | DONE |
|
||||||
|
| 3.5 | `setup/configure_runners.sh` | Interactive runner definition wizard, writes runners.conf | DONE |
|
||||||
|
| 3.6 | `setup/cross_host_ssh.sh` | SSH key exchange between Unraid and Fedora | DONE |
|
||||||
|
| 3.7 | `setup/env_to_bitwarden.sh` | Export .env to Bitwarden JSON import format | DONE |
|
||||||
|
| 3.8 | `setup/bitwarden_to_env.sh` | Restore .env from Bitwarden CLI | DONE |
|
||||||
|
| 3.9 | `setup/cleanup.sh` | Manifest-driven rollback of setup scripts | DONE |
|
||||||
|
|
||||||
### 4. Preflight
|
### 4. Preflight
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 4.1 | `preflight.sh` | Validate .env, SSH, Docker, ports, DNS, GitHub token, Nginx, repos | TODO |
|
| 4.1 | `preflight.sh` | Validate .env, SSH, Docker, IPs, DNS, GitHub token, Caddy, repos | DONE |
|
||||||
|
|
||||||
### 5. Phase 1 — Gitea on Unraid
|
### 5. Phase 1 — Gitea on Unraid
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 5.1 | `phase1_gitea_unraid.sh` | Deploy Gitea container, create admin user + token + org | TODO |
|
| 5.1 | `phase1_gitea_unraid.sh` | Deploy Gitea container, create admin user + token + org | DONE |
|
||||||
| 5.2 | `phase1_post_check.sh` | Verify Gitea HTTP 200, admin auth, token valid, org exists | TODO |
|
| 5.2 | `phase1_post_check.sh` | Verify Gitea HTTP 200, admin auth, token valid, org exists | DONE |
|
||||||
| 5.3 | `phase1_teardown.sh` | docker-compose down, optionally remove data | TODO |
|
| 5.3 | `phase1_teardown.sh` | docker-compose down, optionally remove data | DONE |
|
||||||
|
|
||||||
### 6. Phase 2 — Gitea on Fedora
|
### 6. Phase 2 — Gitea on Fedora
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 6.1 | `phase2_gitea_fedora.sh` | Deploy Gitea container on Fedora, create admin user + token | TODO |
|
| 6.1 | `phase2_gitea_fedora.sh` | Deploy Gitea container on Fedora, create admin user + token | DONE |
|
||||||
| 6.2 | `phase2_post_check.sh` | Verify Fedora Gitea HTTP 200, admin auth, token valid | TODO |
|
| 6.2 | `phase2_post_check.sh` | Verify Fedora Gitea HTTP 200, admin auth, token valid | DONE |
|
||||||
| 6.3 | `phase2_teardown.sh` | docker-compose down on Fedora | TODO |
|
| 6.3 | `phase2_teardown.sh` | docker-compose down on Fedora | DONE |
|
||||||
|
|
||||||
### 7. Phase 3 — Runners
|
### 7. Phase 3 — Runners
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 7.1 | `runners.conf.example` | Runner definition format + example entries | DONE |
|
| 7.1 | `runners.conf.example` | Runner definition format + example entries | DONE |
|
||||||
| 7.2 | `manage_runner.sh` | Add/remove/list runners dynamically (reads runners.conf) | TODO |
|
| 7.2 | `manage_runner.sh` | Add/remove/list runners dynamically (reads runners.conf) | DONE |
|
||||||
| 7.3 | `phase3_runners.sh` | Get registration token, deploy all runners defined in runners.conf | TODO |
|
| 7.3 | `phase3_runners.sh` | Get registration token, deploy all runners defined in runners.conf | DONE |
|
||||||
| 7.4 | `phase3_post_check.sh` | Verify all runners from runners.conf are online in Gitea admin | TODO |
|
| 7.4 | `phase3_post_check.sh` | Verify all runners from runners.conf are online in Gitea admin | DONE |
|
||||||
| 7.5 | `phase3_teardown.sh` | Stop + deregister all runners from runners.conf | TODO |
|
| 7.5 | `phase3_teardown.sh` | Stop + deregister all runners from runners.conf | DONE |
|
||||||
|
|
||||||
### 8. Phase 4 — Migrate Repos + Fedora Mirrors
|
### 8. Phase 4 — Migrate Repos + Fedora Mirrors
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 8.1 | `phase4_migrate_repos.sh` | Import repos from GitHub, set up Fedora pull mirrors | TODO |
|
| 8.1 | `phase4_migrate_repos.sh` | Import repos from GitHub, set up Fedora pull mirrors | DONE |
|
||||||
| 8.2 | `phase4_post_check.sh` | Verify repos on primary + mirror repos on Fedora | TODO |
|
| 8.2 | `phase4_post_check.sh` | Verify repos on primary + mirror repos on Fedora | DONE |
|
||||||
| 8.3 | `phase4_teardown.sh` | Delete repos from primary + Fedora | TODO |
|
| 8.3 | `phase4_teardown.sh` | Delete repos from primary + Fedora | DONE |
|
||||||
|
|
||||||
### 9. Phase 5 — Migrate Pipelines
|
### 9. Phase 5 — Migrate Pipelines
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 9.1 | `phase5_migrate_pipelines.sh` | Copy .github/workflows/ → .gitea/workflows/, apply compat fixes | TODO |
|
| 9.1 | `phase5_migrate_pipelines.sh` | Copy .github/workflows/ → .gitea/workflows/, apply compat fixes | DONE |
|
||||||
| 9.2 | `phase5_post_check.sh` | Verify workflows visible in Gitea Actions UI | TODO |
|
| 9.2 | `phase5_post_check.sh` | Verify workflows visible in Gitea Actions UI | DONE |
|
||||||
| 9.3 | `phase5_teardown.sh` | Remove .gitea/workflows/ from repos | TODO |
|
| 9.3 | `phase5_teardown.sh` | Remove .gitea/workflows/ from repos | DONE |
|
||||||
|
|
||||||
### 10. Phase 6 — GitHub Push Mirrors
|
### 10. Phase 6 — GitHub Push Mirrors
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 10.1 | `phase6_github_mirrors.sh` | Configure push mirrors from Gitea → GitHub | TODO |
|
| 10.1 | `phase6_github_mirrors.sh` | Configure push mirrors from Gitea → GitHub | DONE |
|
||||||
| 10.2 | `phase6_post_check.sh` | Verify mirror config, trigger sync, check GitHub | TODO |
|
| 10.2 | `phase6_post_check.sh` | Verify mirror config, trigger sync, check GitHub | DONE |
|
||||||
| 10.3 | `phase6_teardown.sh` | Remove push mirror config | TODO |
|
| 10.3 | `phase6_teardown.sh` | Remove push mirror config | DONE |
|
||||||
|
|
||||||
### 11. Phase 7 — Branch Protection
|
### 11. Phase 7 — Branch Protection
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 11.1 | `phase7_branch_protection.sh` | Set up branch protection rules on all repos | TODO |
|
| 11.1 | `phase7_branch_protection.sh` | Set up branch protection rules on all repos | DONE |
|
||||||
| 11.2 | `phase7_post_check.sh` | Verify protection rules exist | TODO |
|
| 11.2 | `phase7_post_check.sh` | Verify protection rules exist | DONE |
|
||||||
| 11.3 | `phase7_teardown.sh` | Delete branch protection rules | TODO |
|
| 11.3 | `phase7_teardown.sh` | Delete branch protection rules | DONE |
|
||||||
|
|
||||||
### 12. Phase 8 — Cutover (HTTPS + Archive GitHub)
|
### 12. Phase 8 — Cutover (HTTPS + Archive GitHub)
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 12.1 | `phase8_cutover.sh` | Nginx config + Certbot SSL + archive GitHub repos | TODO |
|
| 12.1 | `phase8_cutover.sh` | Caddy HTTPS reverse proxy + mark GitHub repos as mirrors | DONE |
|
||||||
| 12.2 | `phase8_post_check.sh` | Verify HTTPS, repos accessible, mirrors working | TODO |
|
| 12.2 | `phase8_post_check.sh` | Verify HTTPS, repos accessible, mirrors working | DONE |
|
||||||
| 12.3 | `phase8_teardown.sh` | Remove Nginx config, reload, un-archive GitHub | TODO |
|
| 12.3 | `phase8_teardown.sh` | Remove Caddy stack, restore GitHub repo settings | DONE |
|
||||||
|
|
||||||
### 13. Phase 9 — Security Scanning
|
### 13. Phase 9 — Security Scanning
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 13.1 | `phase9_security.sh` | Deploy security workflow (Semgrep+Trivy+Gitleaks) to all repos | TODO |
|
| 13.1 | `phase9_security.sh` | Deploy security workflow (Semgrep+Trivy+Gitleaks) to all repos | DONE |
|
||||||
| 13.2 | `phase9_post_check.sh` | Verify workflows exist, dry-run passes, branch protection updated | TODO |
|
| 13.2 | `phase9_post_check.sh` | Verify workflows exist, dry-run passes, branch protection updated | DONE |
|
||||||
| 13.3 | `phase9_teardown.sh` | Remove security workflows | TODO |
|
| 13.3 | `phase9_teardown.sh` | Remove security workflows | DONE |
|
||||||
|
|
||||||
### 14. Backup & Restore (post-migration operational scripts)
|
### 14. Backup & Restore (post-migration operational scripts)
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 14.1 | `backup/backup_primary.sh` | Run `gitea dump` on Unraid (DB + repos + config + users), SCP archive to Fedora | TODO |
|
| 14.1 | `backup/backup_primary.sh` | Run `gitea dump` on Unraid (DB + repos + config + users), SCP archive to Fedora | DONE |
|
||||||
| 14.2 | `backup/restore_to_primary.sh` | Restore a `gitea dump` archive to Unraid (fresh or existing instance) | TODO |
|
| 14.2 | `backup/restore_to_primary.sh` | Restore a `gitea dump` archive to Unraid (fresh or existing instance) | DONE |
|
||||||
|
|
||||||
### 15. Orchestration
|
### 15. Orchestration
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 15.1 | `run_all.sh` | Run setup → preflight → phases 1-9 sequentially, --start-from=N | TODO |
|
| 15.1 | `run_all.sh` | Run setup → preflight → phases 1-9 sequentially, --start-from=N | DONE |
|
||||||
| 15.2 | `teardown_all.sh` | Run teardowns in reverse, --through=N | TODO |
|
| 15.2 | `teardown_all.sh` | Run teardowns in reverse, --through=N | DONE |
|
||||||
|
|
||||||
### 16. Project Init
|
### 16. Project Init
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 16.1 | Git repo init + .gitignore | Initialize git repo, ignore .env and temp files | TODO |
|
| 16.1 | Git repo init + .gitignore | Initialize git repo, ignore .env and temp files | DONE |
|
||||||
| 16.2 | `CLAUDE.md` | Project-specific instructions for this codebase | TODO |
|
| 16.2 | `CLAUDE.md` | Project-specific instructions for this codebase | DONE |
|
||||||
|
|
||||||
### 17. Validation
|
### 17. Validation
|
||||||
|
|
||||||
| # | File | Description | Status |
|
| # | File | Description | Status |
|
||||||
|---|------|-------------|--------|
|
|---|------|-------------|--------|
|
||||||
| 17.1 | Shellcheck all `.sh` files | Must pass with no errors | TODO |
|
| 17.1 | Shellcheck all `.sh` files | Must pass with no errors | DONE |
|
||||||
| 17.2 | `bash -n` syntax check all scripts | Verify syntax without executing | TODO |
|
| 17.2 | `bash -n` syntax check all scripts | Verify syntax without executing | DONE |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -681,16 +687,18 @@ gitea-migration/
|
|||||||
| 8 | docker-compose on Unraid | `ssh_exec UNRAID "docker compose version"` or `docker-compose --version` | "docker-compose not found on Unraid. Run setup/unraid.sh." |
|
| 8 | docker-compose on Unraid | `ssh_exec UNRAID "docker compose version"` or `docker-compose --version` | "docker-compose not found on Unraid. Run setup/unraid.sh." |
|
||||||
| 9 | docker-compose on Fedora | Same | Same |
|
| 9 | docker-compose on Fedora | Same | Same |
|
||||||
| 10 | Container IPs available | Ping-check `UNRAID_GITEA_IP`, `UNRAID_CADDY_IP`, `FEDORA_GITEA_IP` — warn if responding | "IP $ip is already responding to ping (may be in use)." |
|
| 10 | Container IPs available | Ping-check `UNRAID_GITEA_IP`, `UNRAID_CADDY_IP`, `FEDORA_GITEA_IP` — warn if responding | "IP $ip is already responding to ping (may be in use)." |
|
||||||
| 12 | DNS resolves | `dig +short $GITEA_DOMAIN` returns `$UNRAID_IP` | "$GITEA_DOMAIN does not resolve to $UNRAID_IP." |
|
| 14 | DNS resolves | `python3 socket.getaddrinfo($GITEA_DOMAIN)` returns `$UNRAID_IP` | "$GITEA_DOMAIN does not resolve to $UNRAID_IP." |
|
||||||
| 13 | GitHub token valid | `github_api GET /user` returns 200 | "GitHub token invalid. Check GITHUB_TOKEN in .env." |
|
| 15 | GitHub token valid | `curl https://api.github.com/user` returns 200 | "GitHub token invalid. Check GITHUB_TOKEN in .env." |
|
||||||
| 14 | GitHub repos exist | For each repo in `REPO_NAMES`: `github_api GET /repos/$GITHUB_USERNAME/$repo` returns 200 | "GitHub repo $repo not found under $GITHUB_USERNAME." |
|
| 16 | GitHub repos exist | For each repo in `REPO_NAMES`: `curl /repos/$GITHUB_USERNAME/$repo` returns 200 | "GitHub repo $repo not found under $GITHUB_USERNAME." |
|
||||||
| 15 | Caddy data path writable | `ssh_exec UNRAID "test -w $CADDY_DATA_PATH"` or parent dir writable | "Caddy data path $CADDY_DATA_PATH not writable on Unraid." |
|
| 17 | Caddy data path writable | `ssh_exec UNRAID "test -w $CADDY_DATA_PATH"` or parent dir writable | "Caddy data path $CADDY_DATA_PATH not writable on Unraid." |
|
||||||
|
| 18-20 | Tool minimum versions | Local, Unraid, Fedora: jq>=1.6, curl>=7.70, git>=2.30, docker>=20, compose>=2 | Version-specific error messages |
|
||||||
|
| 21-22 | Cross-host SSH | Unraid→Fedora and Fedora→Unraid SSH with key auth | "Cannot SSH between hosts. Run setup/cross_host_ssh.sh." |
|
||||||
|
|
||||||
**Exit behavior**: Runs ALL checks (doesn't stop at first failure). Prints summary at end. Exits 0 if all pass, 1 if any fail.
|
**Exit behavior**: Runs ALL checks (doesn't stop at first failure). Prints summary at end. Exits 0 if all pass, 1 if any fail.
|
||||||
|
|
||||||
**Done when**:
|
**Done when**:
|
||||||
- [ ] Every check in the table above is implemented
|
- [x] Every check in the table above is implemented
|
||||||
- [ ] Failed checks point to the correct setup script or config to fix
|
- [x] Failed checks point to the correct setup script or config to fix
|
||||||
- [ ] All checks run even if earlier ones fail (user sees full picture)
|
- [ ] All checks run even if earlier ones fail (user sees full picture)
|
||||||
- [ ] Exit code is 1 if ANY check fails, 0 only if ALL pass
|
- [ ] Exit code is 1 if ANY check fails, 0 only if ALL pass
|
||||||
- [ ] `shellcheck preflight.sh` passes
|
- [ ] `shellcheck preflight.sh` passes
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Gitea Migration Toolkit
|
# Gitea Migration Toolkit
|
||||||
|
|
||||||
Automated migration of GitHub repositories to self-hosted Gitea, with backup mirroring and push-mirror offsite redundancy. 40+ shell scripts, 10 config templates, ~6,500 lines of bash.
|
Automated migration of GitHub repositories to self-hosted Gitea, with backup mirroring and push-mirror offsite redundancy. 43 shell scripts, 9 config templates, ~8,000 lines of bash.
|
||||||
|
|
||||||
## What This Does
|
## What This Does
|
||||||
|
|
||||||
@@ -67,7 +67,7 @@ gitea-migration/
|
|||||||
├── runners.conf.example # Runner definitions template
|
├── runners.conf.example # Runner definitions template
|
||||||
├── lib/common.sh # Shared functions + .env validators
|
├── lib/common.sh # Shared functions + .env validators
|
||||||
├── setup/
|
├── setup/
|
||||||
│ ├── configure_env.sh # Interactive .env wizard (~50 prompts)
|
│ ├── configure_env.sh # Interactive .env wizard (~65 prompts)
|
||||||
│ ├── macbook.sh # Local prerequisites (brew packages)
|
│ ├── macbook.sh # Local prerequisites (brew packages)
|
||||||
│ ├── unraid.sh # Remote prerequisites (static binaries)
|
│ ├── unraid.sh # Remote prerequisites (static binaries)
|
||||||
│ ├── fedora.sh # Remote prerequisites (dnf packages)
|
│ ├── fedora.sh # Remote prerequisites (dnf packages)
|
||||||
@@ -89,7 +89,7 @@ gitea-migration/
|
|||||||
├── backup/
|
├── backup/
|
||||||
│ ├── backup_primary.sh # Gitea dump, SCP to Fedora
|
│ ├── backup_primary.sh # Gitea dump, SCP to Fedora
|
||||||
│ └── restore_to_primary.sh # Restore dump to Unraid
|
│ └── restore_to_primary.sh # Restore dump to Unraid
|
||||||
├── preflight.sh # 25 pre-flight validation checks
|
├── preflight.sh # 22 pre-flight validation checks
|
||||||
├── run_all.sh # Full pipeline orchestration
|
├── run_all.sh # Full pipeline orchestration
|
||||||
├── teardown_all.sh # Reverse teardown (9 to 1)
|
├── teardown_all.sh # Reverse teardown (9 to 1)
|
||||||
├── manage_runner.sh # Dynamic runner add/remove/list
|
├── manage_runner.sh # Dynamic runner add/remove/list
|
||||||
|
|||||||
@@ -49,6 +49,9 @@ You need one GitHub Personal Access Token:
|
|||||||
cp .env.example .env
|
cp .env.example .env
|
||||||
cp runners.conf.example runners.conf
|
cp runners.conf.example runners.conf
|
||||||
# Edit both files with your values
|
# Edit both files with your values
|
||||||
|
|
||||||
|
# Optional: interactive runner definition wizard
|
||||||
|
./setup/configure_runners.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
The wizard validates every input (IP format, port ranges, URL format, password length) and shows your current values in brackets so you can press Enter to keep them.
|
The wizard validates every input (IP format, port ranges, URL format, password length) and shows your current values in brackets so you can press Enter to keep them.
|
||||||
@@ -218,7 +221,7 @@ Then re-run Phase 4. Already-migrated repos will be skipped.
|
|||||||
|
|
||||||
### DNS doesn't resolve to Unraid IP
|
### DNS doesn't resolve to Unraid IP
|
||||||
|
|
||||||
**Symptom**: Preflight check 15 fails.
|
**Symptom**: Preflight check 14 fails.
|
||||||
|
|
||||||
**Fix**: Add or update your DNS A record. If using a local DNS server or `/etc/hosts`, ensure the record points to `UNRAID_IP`. DNS propagation can take minutes to hours.
|
**Fix**: Add or update your DNS A record. If using a local DNS server or `/etc/hosts`, ensure the record points to `UNRAID_IP`. DNS propagation can take minutes to hours.
|
||||||
|
|
||||||
|
|||||||
36
preflight.sh
36
preflight.sh
@@ -417,7 +417,7 @@ else
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 15: DNS resolves
|
# Check 14: DNS resolves
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
check_dns() {
|
check_dns() {
|
||||||
# Fail closed when required values are missing.
|
# Fail closed when required values are missing.
|
||||||
@@ -435,24 +435,24 @@ check_dns() {
|
|||||||
fi
|
fi
|
||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
check 15 "DNS: ${GITEA_DOMAIN:-<not set>} resolves to ${UNRAID_IP:-<not set>}" check_dns
|
check 14 "DNS: ${GITEA_DOMAIN:-<not set>} resolves to ${UNRAID_IP:-<not set>}" check_dns
|
||||||
if ! check_dns 2>/dev/null; then
|
if ! check_dns 2>/dev/null; then
|
||||||
log_error " → ${GITEA_DOMAIN:-GITEA_DOMAIN} does not resolve to ${UNRAID_IP:-UNRAID_IP}."
|
log_error " → ${GITEA_DOMAIN:-GITEA_DOMAIN} does not resolve to ${UNRAID_IP:-UNRAID_IP}."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 16: GitHub token valid
|
# Check 15: GitHub token valid
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
check_github_token() {
|
check_github_token() {
|
||||||
[[ -n "${GITHUB_TOKEN:-}" ]] && curl -sf -H "Authorization: token ${GITHUB_TOKEN}" https://api.github.com/user -o /dev/null
|
[[ -n "${GITHUB_TOKEN:-}" ]] && curl -sf -H "Authorization: token ${GITHUB_TOKEN}" https://api.github.com/user -o /dev/null
|
||||||
}
|
}
|
||||||
check 16 "GitHub token valid" check_github_token
|
check 15 "GitHub token valid" check_github_token
|
||||||
if ! check_github_token 2>/dev/null; then
|
if ! check_github_token 2>/dev/null; then
|
||||||
log_error " → GitHub token invalid. Check GITHUB_TOKEN in .env."
|
log_error " → GitHub token invalid. Check GITHUB_TOKEN in .env."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 17: GitHub repos exist
|
# Check 16: GitHub repos exist
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# shellcheck disable=SC2329
|
# shellcheck disable=SC2329
|
||||||
check_github_repos() {
|
check_github_repos() {
|
||||||
@@ -472,10 +472,10 @@ check_github_repos() {
|
|||||||
done
|
done
|
||||||
return "$all_ok"
|
return "$all_ok"
|
||||||
}
|
}
|
||||||
check 17 "All GitHub repos exist" check_github_repos
|
check 16 "All GitHub repos exist" check_github_repos
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 18: Caddy data path writable on Unraid
|
# Check 17: Caddy data path writable on Unraid
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
check_caddy_path() {
|
check_caddy_path() {
|
||||||
local caddy_parent
|
local caddy_parent
|
||||||
@@ -483,13 +483,13 @@ check_caddy_path() {
|
|||||||
ssh_exec UNRAID "test -d '${CADDY_DATA_PATH}' && test -w '${CADDY_DATA_PATH}'" 2>/dev/null \
|
ssh_exec UNRAID "test -d '${CADDY_DATA_PATH}' && test -w '${CADDY_DATA_PATH}'" 2>/dev/null \
|
||||||
|| ssh_exec UNRAID "test -w '${caddy_parent}'" 2>/dev/null
|
|| ssh_exec UNRAID "test -w '${caddy_parent}'" 2>/dev/null
|
||||||
}
|
}
|
||||||
check 18 "Caddy data path writable (${CADDY_DATA_PATH:-<not set>})" check_caddy_path
|
check 17 "Caddy data path writable (${CADDY_DATA_PATH:-<not set>})" check_caddy_path
|
||||||
if ! check_caddy_path 2>/dev/null; then
|
if ! check_caddy_path 2>/dev/null; then
|
||||||
log_error " → Caddy data path ${CADDY_DATA_PATH:-} not writable on Unraid (or parent dir doesn't exist)."
|
log_error " → Caddy data path ${CADDY_DATA_PATH:-} not writable on Unraid (or parent dir doesn't exist)."
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 20: Local tool minimum versions
|
# Check 18: Local tool minimum versions
|
||||||
# Validates that tools on the MacBook meet minimum requirements.
|
# Validates that tools on the MacBook meet minimum requirements.
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# shellcheck disable=SC2329
|
# shellcheck disable=SC2329
|
||||||
@@ -500,10 +500,10 @@ check_local_versions() {
|
|||||||
check_min_version "git" "git --version" "2.30" || fail=1
|
check_min_version "git" "git --version" "2.30" || fail=1
|
||||||
return "$fail"
|
return "$fail"
|
||||||
}
|
}
|
||||||
check 20 "Local tool minimum versions (jq>=1.6, curl>=7.70, git>=2.30)" check_local_versions
|
check 18 "Local tool minimum versions (jq>=1.6, curl>=7.70, git>=2.30)" check_local_versions
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 21: Unraid tool minimum versions
|
# Check 19: Unraid tool minimum versions
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# shellcheck disable=SC2329
|
# shellcheck disable=SC2329
|
||||||
check_unraid_versions() {
|
check_unraid_versions() {
|
||||||
@@ -513,10 +513,10 @@ check_unraid_versions() {
|
|||||||
check_remote_min_version "UNRAID" "jq" "jq --version" "1.6" || fail=1
|
check_remote_min_version "UNRAID" "jq" "jq --version" "1.6" || fail=1
|
||||||
return "$fail"
|
return "$fail"
|
||||||
}
|
}
|
||||||
check 21 "Unraid tool minimum versions (docker>=20, compose>=2, jq>=1.6)" check_unraid_versions
|
check 19 "Unraid tool minimum versions (docker>=20, compose>=2, jq>=1.6)" check_unraid_versions
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 22: Fedora tool minimum versions
|
# Check 20: Fedora tool minimum versions
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# shellcheck disable=SC2329
|
# shellcheck disable=SC2329
|
||||||
check_fedora_versions() {
|
check_fedora_versions() {
|
||||||
@@ -526,27 +526,27 @@ check_fedora_versions() {
|
|||||||
check_remote_min_version "FEDORA" "jq" "jq --version" "1.6" || fail=1
|
check_remote_min_version "FEDORA" "jq" "jq --version" "1.6" || fail=1
|
||||||
return "$fail"
|
return "$fail"
|
||||||
}
|
}
|
||||||
check 22 "Fedora tool minimum versions (docker>=20, compose>=2, jq>=1.6)" check_fedora_versions
|
check 20 "Fedora tool minimum versions (docker>=20, compose>=2, jq>=1.6)" check_fedora_versions
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 23: Unraid can SSH to Fedora (required for backup transfer)
|
# Check 21: Unraid can SSH to Fedora (required for backup transfer)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
check_unraid_to_fedora_ssh() {
|
check_unraid_to_fedora_ssh() {
|
||||||
ssh_exec UNRAID "ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -p '${FEDORA_SSH_PORT:-22}' '${FEDORA_SSH_USER}@${FEDORA_IP}' true" &>/dev/null
|
ssh_exec UNRAID "ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -p '${FEDORA_SSH_PORT:-22}' '${FEDORA_SSH_USER}@${FEDORA_IP}' true" &>/dev/null
|
||||||
}
|
}
|
||||||
check 23 "Unraid can SSH to Fedora (host-to-host backup path)" check_unraid_to_fedora_ssh
|
check 21 "Unraid can SSH to Fedora (host-to-host backup path)" check_unraid_to_fedora_ssh
|
||||||
if ! check_unraid_to_fedora_ssh 2>/dev/null; then
|
if ! check_unraid_to_fedora_ssh 2>/dev/null; then
|
||||||
log_error " → Unraid cannot SSH to Fedora with key auth (needed by backup/backup_primary.sh)."
|
log_error " → Unraid cannot SSH to Fedora with key auth (needed by backup/backup_primary.sh)."
|
||||||
log_error " → Configure SSH keys so Unraid can run: ssh ${FEDORA_SSH_USER}@${FEDORA_IP}"
|
log_error " → Configure SSH keys so Unraid can run: ssh ${FEDORA_SSH_USER}@${FEDORA_IP}"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Check 24: Fedora can SSH to Unraid (required for restore transfer)
|
# Check 22: Fedora can SSH to Unraid (required for restore transfer)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
check_fedora_to_unraid_ssh() {
|
check_fedora_to_unraid_ssh() {
|
||||||
ssh_exec FEDORA "ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -p '${UNRAID_SSH_PORT:-22}' '${UNRAID_SSH_USER}@${UNRAID_IP}' true" &>/dev/null
|
ssh_exec FEDORA "ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new -o BatchMode=yes -p '${UNRAID_SSH_PORT:-22}' '${UNRAID_SSH_USER}@${UNRAID_IP}' true" &>/dev/null
|
||||||
}
|
}
|
||||||
check 24 "Fedora can SSH to Unraid (host-to-host restore path)" check_fedora_to_unraid_ssh
|
check 22 "Fedora can SSH to Unraid (host-to-host restore path)" check_fedora_to_unraid_ssh
|
||||||
if ! check_fedora_to_unraid_ssh 2>/dev/null; then
|
if ! check_fedora_to_unraid_ssh 2>/dev/null; then
|
||||||
log_error " → Fedora cannot SSH to Unraid with key auth (needed by backup/restore_to_primary.sh)."
|
log_error " → Fedora cannot SSH to Unraid with key auth (needed by backup/restore_to_primary.sh)."
|
||||||
log_error " → Configure SSH keys so Fedora can run: ssh ${UNRAID_SSH_USER}@${UNRAID_IP}"
|
log_error " → Configure SSH keys so Fedora can run: ssh ${UNRAID_SSH_USER}@${UNRAID_IP}"
|
||||||
|
|||||||
Reference in New Issue
Block a user