diff --git a/README.md b/README.md index 0e274a0..bdcdf22 100644 --- a/README.md +++ b/README.md @@ -10,15 +10,15 @@ Mirrors **all** GitHub repositories of a user into a self-hosted Gitea instance as pull mirrors — including repositories created later. Gitea can already mirror a single repository through its web UI. What it cannot -do is watch an account and pick up new repositories on its own. That gap is the -only reason this script exists: a daily cron job asks the GitHub API which -repositories exist and creates the mirrors that are missing. Keeping their -*content* up to date remains Gitea's job. +do is watch an account and pick up new repositories on its own, or continuously +mirror GitHub release metadata and uploaded assets. A cron job closes both gaps. +Keeping the repositories' *git content* up to date remains Gitea's job. | Task | Handled by | Frequency | | --- | --- | --- | | Discover **new** repositories, create mirrors | this script (cron) | daily | | Pull new commits into **existing** mirrors | Gitea itself | `MIRROR_INTERVAL` | +| Synchronize release metadata and uploaded assets | this script (cron) | daily | ### Requirements @@ -74,6 +74,8 @@ python3 sync_mirrors.py # run for real | `MIRROR_INTERVAL` | Go duration, e.g. `24h`, `8h`, `30m` | | `INCLUDE_FORKS` | Mirror forks too (default `false`) | | `INCLUDE_ARCHIVED` | Mirror archived repositories (default `true`) | +| `SYNC_RELEASES` | Synchronize GitHub releases on every run (default `true`) | +| `SYNC_RELEASE_ASSETS` | Download uploaded release assets too (default `true`) | | `DRY_RUN` | `true` = show what would happen, change nothing | Every key can be overridden by an environment variable of the same name, which @@ -107,13 +109,20 @@ remaining validity, warns from 14 days out and exits non-zero once expired: ### Scope and limits -- Mirrored is **git data only**: all branches and tags. Issues, pull requests, - releases and the wiki are excluded on purpose — a pull mirror never syncs - those, so importing them once would leave a snapshot that silently goes - stale. To change that, flip the flags in `create_mirror()`. +- Gitea mirrors **git data** itself: all branches and tags. The script separately + synchronizes GitHub release titles, notes, draft/prerelease state and uploaded + assets on every run. Issues, pull requests and the wiki remain excluded. +- A release is synchronized only after its git tag has reached Gitea. If the tag + is still missing, the release is logged and retried by the next cron run. +- Source archives generated automatically for a tag are not copied because + Gitea generates its own. Assets explicitly uploaded to GitHub are copied. - The script **never deletes** anything in Gitea. If a repository disappears or is renamed on GitHub, its mirror stays and is reported as an orphan in the - log — for a backup that is the right direction. + log — for a backup that is the right direction. The same applies to releases + and assets deleted upstream. An asset with the same name but a different size + is reported, while the existing Gitea copy is retained. +- GitHub author identities, original release timestamps and download counters + cannot be reproduced through Gitea's regular release API. - A Gitea repository that exists but is **not** a mirror is left untouched, so a real working repository can never be overwritten. - Mirrors are read-only in Gitea. Push to GitHub, not to the mirror. @@ -122,13 +131,22 @@ remaining validity, warns from 14 days out and exits non-zero once expired: | File | Purpose | | --- | --- | -| `sync_mirrors.py` | Creates missing mirrors, aligns the mirror interval | +| `sync_mirrors.py` | Creates mirrors, aligns intervals and synchronizes releases | +| `tests/` | Unit tests for repeatable release synchronization | | `run_sync.sh` | Cron wrapper: appends to `sync.log`, trims it to 1000 lines | | `install.sh` | Installs scripts, config and cron entry | | `mirror.env.example` | Configuration template | | `mirror.env` | Real config with secrets — git-ignored, `chmod 600` | | `sync.log` | Last ~2000 log lines | +### Tests + +Run the unit tests from the repository root: + +```bash +python3 -m unittest discover -s tests -v +``` + ### Troubleshooting ```bash @@ -141,6 +159,8 @@ tail -f ~/gitea-github-mirror/sync.log | `GitHub API 403` | Rate limit, or token lacks repository access | | `migrate failed (HTTP 409)` | Name already taken in Gitea | | `... exists in Gitea but is not a mirror` | Real repo with that name — rename one of them | +| `release ... waits for its tag` | Gitea has not fetched the GitHub tag yet; the next run retries it | +| `asset ... differs` | An upstream asset changed; the retained Gitea backup is not overwritten | | `Missing config value(s)` | `mirror.env` not filled in | | `no private repositories` despite a token | Token's repository access is "Public repositories", or no repository permission is set at all | @@ -167,14 +187,15 @@ selbst gehostete Gitea-Instanz — auch Repositories, die erst später entstehen Ein einzelnes Repository kann Gitea bereits über die Weboberfläche spiegeln. Was Gitea nicht kann: einen Account beobachten und neue Repositories von selbst -aufnehmen. Genau dafür gibt es dieses Skript — ein täglicher Cron-Job fragt die -GitHub-API, welche Repositories existieren, und legt die fehlenden Mirrors an. -Deren *Inhalt* aktuell zu halten bleibt Giteas Aufgabe. +aufnehmen oder GitHub-Release-Metadaten und hochgeladene Assets fortlaufend +spiegeln. Ein Cron-Job schließt beide Lücken. Die *Git-Inhalte* aktuell zu halten +bleibt Giteas Aufgabe. | Aufgabe | Erledigt von | Häufigkeit | | --- | --- | --- | | **Neue** Repositories finden, Mirror anlegen | dieses Skript (Cron) | täglich | | Commits in **bestehende** Mirrors holen | Gitea selbst | `MIRROR_INTERVAL` | +| Release-Metadaten und hochgeladene Assets synchronisieren | dieses Skript (Cron) | täglich | ### Voraussetzungen @@ -230,6 +251,8 @@ python3 sync_mirrors.py # echter Lauf | `MIRROR_INTERVAL` | Go-Duration, z. B. `24h`, `8h`, `30m` | | `INCLUDE_FORKS` | Forks mitspiegeln (Standard `false`) | | `INCLUDE_ARCHIVED` | Archivierte Repositories spiegeln (Standard `true`) | +| `SYNC_RELEASES` | GitHub-Releases bei jedem Lauf synchronisieren (Standard `true`) | +| `SYNC_RELEASE_ASSETS` | Auch hochgeladene Release-Assets laden (Standard `true`) | | `DRY_RUN` | `true` = nur anzeigen, nichts ändern | Jeder Schlüssel lässt sich durch eine gleichnamige Umgebungsvariable @@ -266,13 +289,22 @@ Ablauf mit Exit-Code 1: ### Umfang und Grenzen -- Gespiegelt werden **nur Git-Daten**: alle Branches und Tags. Issues, Pull - Requests, Releases und Wiki bleiben bewusst außen vor — ein Pull-Mirror - synchronisiert das nie, ein einmaliger Import würde also stillschweigend - veralten. Umstellbar über die Flags in `create_mirror()`. +- Gitea selbst spiegelt die **Git-Daten**: alle Branches und Tags. Das Skript + synchronisiert zusätzlich bei jedem Lauf GitHub-Release-Titel, Beschreibung, + Draft-/Prerelease-Status und hochgeladene Assets. Issues, Pull Requests und + Wiki bleiben außen vor. +- Ein Release wird erst synchronisiert, wenn sein Git-Tag in Gitea angekommen + ist. Fehlt der Tag noch, wird das protokolliert und beim nächsten Cron-Lauf + erneut versucht. +- Automatisch für Tags erzeugte Quellcodearchive werden nicht kopiert, weil + Gitea eigene erzeugt. Explizit auf GitHub hochgeladene Assets werden kopiert. - Das Skript **löscht nie** etwas in Gitea. Verschwindet ein Repository auf GitHub oder wird umbenannt, bleibt der Mirror bestehen und wird im Log als - verwaist gemeldet — für ein Backup ist das die richtige Richtung. + verwaist gemeldet — für ein Backup ist das die richtige Richtung. Gleiches + gilt für upstream gelöschte Releases und Assets. Hat ein gleichnamiges Asset + eine andere Größe, wird dies gemeldet und die Gitea-Kopie behalten. +- GitHub-Autoren, ursprüngliche Release-Zeitpunkte und Downloadzähler können + über die reguläre Gitea-Release-API nicht identisch übernommen werden. - Ein Gitea-Repository, das existiert, aber **kein** Mirror ist, wird nicht angefasst; ein echtes Arbeits-Repository kann so nie überschrieben werden. - Mirrors sind in Gitea schreibgeschützt. Gepusht wird nach GitHub, nicht in @@ -282,13 +314,22 @@ Ablauf mit Exit-Code 1: | Datei | Zweck | | --- | --- | -| `sync_mirrors.py` | Legt fehlende Mirrors an, gleicht das Mirror-Intervall ab | +| `sync_mirrors.py` | Legt Mirrors an, gleicht Intervalle ab und synchronisiert Releases | +| `tests/` | Unit-Tests für die wiederholbare Release-Synchronisierung | | `run_sync.sh` | Cron-Wrapper: schreibt nach `sync.log`, kürzt es auf 1000 Zeilen | | `install.sh` | Installiert Skripte, Konfiguration und Cron-Eintrag | | `mirror.env.example` | Konfigurationsvorlage | | `mirror.env` | Echte Konfiguration mit Secrets — git-ignoriert, `chmod 600` | | `sync.log` | Die letzten ~2000 Log-Zeilen | +### Tests + +Die Unit-Tests im Wurzelverzeichnis des Repositorys starten: + +```bash +python3 -m unittest discover -s tests -v +``` + ### Fehlersuche ```bash @@ -301,6 +342,8 @@ tail -f ~/gitea-github-mirror/sync.log | `GitHub API 403` | Rate Limit, oder Token hat keinen Repository-Zugriff | | `migrate failed (HTTP 409)` | Name in Gitea bereits vergeben | | `... exists in Gitea but is not a mirror` | Echtes Repo gleichen Namens — eines umbenennen | +| `release ... waits for its tag` | Gitea hat den GitHub-Tag noch nicht geholt; der nächste Lauf versucht es erneut | +| `asset ... differs` | Ein Upstream-Asset wurde geändert; das Gitea-Backup wird nicht überschrieben | | `Missing config value(s)` | `mirror.env` nicht ausgefüllt | | `no private repositories` trotz Token | Repository access steht auf „Public repositories", oder es ist gar keine Repository-Permission gesetzt | diff --git a/mirror.env.example b/mirror.env.example index c16e63c..76e2518 100644 --- a/mirror.env.example +++ b/mirror.env.example @@ -22,5 +22,9 @@ GITEA_OWNER=your-gitea-username MIRROR_INTERVAL=24h INCLUDE_FORKS=false INCLUDE_ARCHIVED=true +# Gitea's native mirror only pulls git data. The script separately keeps +# GitHub release metadata and uploaded assets in sync on every cron run. +SYNC_RELEASES=true +SYNC_RELEASE_ASSETS=true # Set to true to preview without changing anything. DRY_RUN=false diff --git a/sync_mirrors.py b/sync_mirrors.py index d8ff212..e2f49a3 100644 --- a/sync_mirrors.py +++ b/sync_mirrors.py @@ -1,18 +1,23 @@ #!/usr/bin/env python3 """Mirror all GitHub repositories of a user into Gitea as pull mirrors. -Existing mirrors are left to Gitea's own cron; this script only creates the -ones that are missing and keeps the mirror interval aligned with the config. -It never deletes anything on the Gitea side -- a mirror that vanished upstream -is exactly what a backup is supposed to keep. +Existing git mirrors are left to Gitea's own cron. This script creates the ones +that are missing, aligns their interval, and synchronizes GitHub release +metadata and assets because Gitea's native mirror does not. It never deletes +anything on the Gitea side -- retained history is the point of a backup. """ import json +import http.client +import io import os import re import sys +import tempfile import urllib.error +import urllib.parse import urllib.request +import uuid from datetime import datetime, timezone KNOWN_KEYS = ( @@ -24,6 +29,8 @@ KNOWN_KEYS = ( "MIRROR_INTERVAL", "INCLUDE_FORKS", "INCLUDE_ARCHIVED", + "SYNC_RELEASES", + "SYNC_RELEASE_ASSETS", "DRY_RUN", ) @@ -54,6 +61,8 @@ def load_config(): "MIRROR_INTERVAL": "24h", "INCLUDE_FORKS": "false", "INCLUDE_ARCHIVED": "true", + "SYNC_RELEASES": "true", + "SYNC_RELEASE_ASSETS": "true", "DRY_RUN": "false", } if os.path.exists(CONFIG_PATH): @@ -102,6 +111,109 @@ def api(url, token=None, scheme="token", method="GET", payload=None): return resp.status, (json.loads(body) if body.strip() else None) +def path_part(value): + """Quote one URL path component, including slashes used inside tag names.""" + return urllib.parse.quote(str(value), safe="") + + +class SafeRedirectHandler(urllib.request.HTTPRedirectHandler): + """Do not forward API credentials to a cross-host download redirect.""" + + def redirect_request(self, req, fp, code, msg, headers, newurl): + redirected = super().redirect_request(req, fp, code, msg, headers, newurl) + if redirected is None: + return None + old_host = urllib.parse.urlsplit(req.full_url).netloc.lower() + new_host = urllib.parse.urlsplit(newurl).netloc.lower() + if old_host != new_host: + redirected.remove_header("Authorization") + return redirected + + +safe_download_opener = urllib.request.build_opener(SafeRedirectHandler()) + + +def download_github_asset(cfg, asset): + """Download a release asset into a temporary file without buffering it in RAM.""" + headers = { + "Accept": "application/octet-stream", + "User-Agent": "gitea-github-mirror/1.0", + } + if cfg["GITHUB_TOKEN"]: + headers["Authorization"] = "Bearer %s" % cfg["GITHUB_TOKEN"] + req = urllib.request.Request(asset["url"], headers=headers) + tmp = tempfile.TemporaryFile() + size = 0 + try: + with safe_download_opener.open(req, timeout=180) as resp: + while True: + chunk = resp.read(1024 * 1024) + if not chunk: + break + tmp.write(chunk) + size += len(chunk) + expected = int(asset.get("size") or 0) + if expected and size != expected: + raise IOError("downloaded %d bytes, expected %d" % (size, expected)) + tmp.seek(0) + return tmp, size + except Exception: + tmp.close() + raise + + +def upload_release_asset(cfg, repo_name, release_id, asset, source, size): + """Stream a temporary file to Gitea as multipart/form-data.""" + query = urllib.parse.urlencode({"name": asset["name"]}) + url = ("%s/api/v1/repos/%s/%s/releases/%s/assets?%s" + % (cfg["GITEA_URL"], path_part(cfg["GITEA_OWNER"]), path_part(repo_name), + release_id, query)) + parsed = urllib.parse.urlsplit(url) + boundary = "----gitea-github-mirror-%s" % uuid.uuid4().hex + safe_name = str(asset["name"]).replace("\\", "_").replace('"', "_") + safe_name = safe_name.replace("\r", "_").replace("\n", "_") + content_type = asset.get("content_type") or "application/octet-stream" + content_type = str(content_type).replace("\r", "").replace("\n", "") + preamble = ( + "--%s\r\n" + "Content-Disposition: form-data; name=\"attachment\"; filename=\"%s\"\r\n" + "Content-Type: %s\r\n\r\n" % (boundary, safe_name, content_type) + ).encode("utf-8") + ending = ("\r\n--%s--\r\n" % boundary).encode("ascii") + request_path = parsed.path + (("?" + parsed.query) if parsed.query else "") + + if parsed.scheme == "https": + conn = http.client.HTTPSConnection(parsed.hostname, parsed.port, timeout=180) + elif parsed.scheme == "http": + conn = http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=180) + else: + raise ValueError("Unsupported Gitea URL scheme: %s" % parsed.scheme) + + try: + conn.putrequest("POST", request_path) + conn.putheader("Authorization", "token %s" % cfg["GITEA_TOKEN"]) + conn.putheader("Accept", "application/json") + conn.putheader("Content-Type", "multipart/form-data; boundary=%s" % boundary) + conn.putheader("Content-Length", str(len(preamble) + size + len(ending))) + conn.putheader("User-Agent", "gitea-github-mirror/1.0") + conn.endheaders() + conn.send(preamble) + while True: + chunk = source.read(1024 * 1024) + if not chunk: + break + conn.send(chunk) + conn.send(ending) + response = conn.getresponse() + body = response.read() + if response.status >= 400: + raise urllib.error.HTTPError( + url, response.status, response.reason, response.headers, io.BytesIO(body)) + return json.loads(body.decode("utf-8")) if body.strip() else None + finally: + conn.close() + + def github_repos(cfg): """All repos owned by the user; private ones included when a token is set.""" token = cfg["GITHUB_TOKEN"] @@ -229,6 +341,229 @@ def align_interval(cfg, name, existing, dry_run): log("OK", "interval of %s set to %s" % (name, cfg["MIRROR_INTERVAL"])) +def github_releases(cfg, repo_name): + releases = [] + page = 1 + while True: + url = ("https://api.github.com/repos/%s/%s/releases?per_page=100&page=%d" + % (path_part(cfg["GITHUB_USER"]), path_part(repo_name), page)) + _, batch = api(url, token=cfg["GITHUB_TOKEN"], scheme="Bearer") + if not batch: + break + releases.extend(batch) + if len(batch) < 100: + break + page += 1 + return releases + + +def github_release_assets(cfg, repo_name, release_id): + assets = [] + page = 1 + while True: + url = ("https://api.github.com/repos/%s/%s/releases/%s/assets" + "?per_page=100&page=%d" + % (path_part(cfg["GITHUB_USER"]), path_part(repo_name), release_id, page)) + _, batch = api(url, token=cfg["GITHUB_TOKEN"], scheme="Bearer") + if not batch: + break + assets.extend(batch) + if len(batch) < 100: + break + page += 1 + return assets + + +def gitea_releases(cfg, repo_name): + releases = [] + page = 1 + while True: + url = ("%s/api/v1/repos/%s/%s/releases?limit=50&page=%d" + % (cfg["GITEA_URL"], path_part(cfg["GITEA_OWNER"]), + path_part(repo_name), page)) + _, batch = api(url, token=cfg["GITEA_TOKEN"]) + if not batch: + break + releases.extend(batch) + if len(batch) < 50: + break + page += 1 + return releases + + +def gitea_tags(cfg, repo_name): + tags = set() + page = 1 + while True: + url = ("%s/api/v1/repos/%s/%s/tags?limit=50&page=%d" + % (cfg["GITEA_URL"], path_part(cfg["GITEA_OWNER"]), + path_part(repo_name), page)) + _, batch = api(url, token=cfg["GITEA_TOKEN"]) + if not batch: + break + tags.update(tag["name"] for tag in batch) + if len(batch) < 50: + break + page += 1 + return tags + + +def release_payload(release): + return { + "tag_name": release["tag_name"], + "target_commitish": release.get("target_commitish") or "", + "name": release.get("name") or "", + "body": release.get("body") or "", + "draft": bool(release.get("draft")), + "prerelease": bool(release.get("prerelease")), + } + + +def release_needs_update(source, target): + """Return true for fields Gitea's PATCH endpoint can actually change.""" + wanted = release_payload(source) + if bool(target.get("draft")) != wanted["draft"]: + return True + if bool(target.get("prerelease")) != wanted["prerelease"]: + return True + # Gitea ignores empty strings in release PATCH requests. Retaining old text + # also follows this script's policy of never deleting backup information. + if wanted["name"] and (target.get("name") or "") != wanted["name"]: + return True + if wanted["body"] and (target.get("body") or "") != wanted["body"]: + return True + return False + + +def release_update_payload(source): + wanted = release_payload(source) + payload = { + "draft": wanted["draft"], + "prerelease": wanted["prerelease"], + } + for key in ("target_commitish", "name", "body"): + if wanted[key]: + payload[key] = wanted[key] + return payload + + +def sync_release_assets(cfg, repo_name, source_release, target_release, dry_run): + copied = preserved = 0 + source_assets = github_release_assets(cfg, repo_name, source_release["id"]) + target_assets = {asset["name"]: asset for asset in target_release.get("assets", [])} + + for asset in source_assets: + name = asset["name"] + existing = target_assets.get(name) + if existing is not None: + source_size = int(asset.get("size") or 0) + target_size = int(existing.get("size") or 0) + if source_size and target_size and source_size != target_size: + preserved += 1 + log("WARN", "%s release %s asset %s differs (%d vs %d bytes) - " + "Gitea copy left untouched" + % (repo_name, source_release["tag_name"], name, + source_size, target_size)) + continue + + if dry_run: + log("DRY", "would copy %s release %s asset %s (%d bytes)" + % (repo_name, source_release["tag_name"], name, + int(asset.get("size") or 0))) + copied += 1 + continue + + tmp = None + try: + tmp, size = download_github_asset(cfg, asset) + upload_release_asset(cfg, repo_name, target_release["id"], asset, tmp, size) + copied += 1 + log("OK", "copied %s release %s asset %s (%d bytes)" + % (repo_name, source_release["tag_name"], name, size)) + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", "replace")[:300] + fail("%s release %s asset %s failed (HTTP %s): %s" + % (repo_name, source_release["tag_name"], name, exc.code, detail)) + except Exception as exc: + fail("%s release %s asset %s failed: %s" + % (repo_name, source_release["tag_name"], name, exc)) + finally: + if tmp is not None: + tmp.close() + return copied, preserved + + +def sync_releases(cfg, repo_name, dry_run): + stats = {"created": 0, "updated": 0, "assets": 0, "preserved": 0, "waiting": 0} + source_releases = github_releases(cfg, repo_name) + target_releases = gitea_releases(cfg, repo_name) + by_tag = {release["tag_name"]: release for release in target_releases} + tags_to_check = [ + release["tag_name"] for release in source_releases + if not release.get("draft") + and (release["tag_name"] not in by_tag + or by_tag[release["tag_name"]].get("draft")) + ] + available_tags = gitea_tags(cfg, repo_name) if tags_to_check else set() + + source_tags = {release["tag_name"] for release in source_releases} + orphaned = sorted(tag for tag in by_tag if tag not in source_tags) + if orphaned: + log("WARN", "%s Gitea release(s) with no GitHub counterpart left untouched: %s" + % (repo_name, ", ".join(orphaned))) + + for source in source_releases: + tag = source["tag_name"] + target = by_tag.get(tag) + needs_mirrored_tag = (not source.get("draft") + and (target is None or target.get("draft"))) + if needs_mirrored_tag and tag not in available_tags: + stats["waiting"] += 1 + log("WARN", "%s release %s waits for its tag to reach the Gitea mirror" + % (repo_name, tag)) + continue + + try: + if target is None: + if dry_run: + target = {"id": source["id"], "tag_name": tag, "assets": []} + log("DRY", "would create %s release %s" % (repo_name, tag)) + else: + _, target = api( + "%s/api/v1/repos/%s/%s/releases" + % (cfg["GITEA_URL"], path_part(cfg["GITEA_OWNER"]), + path_part(repo_name)), + token=cfg["GITEA_TOKEN"], method="POST", + payload=release_payload(source)) + log("OK", "created %s release %s" % (repo_name, tag)) + stats["created"] += 1 + elif release_needs_update(source, target): + if dry_run: + log("DRY", "would update %s release %s" % (repo_name, tag)) + else: + _, target = api( + "%s/api/v1/repos/%s/%s/releases/%s" + % (cfg["GITEA_URL"], path_part(cfg["GITEA_OWNER"]), + path_part(repo_name), target["id"]), + token=cfg["GITEA_TOKEN"], method="PATCH", + payload=release_update_payload(source)) + log("OK", "updated %s release %s" % (repo_name, tag)) + stats["updated"] += 1 + + if truthy(cfg["SYNC_RELEASE_ASSETS"]): + copied, preserved = sync_release_assets( + cfg, repo_name, source, target, dry_run) + stats["assets"] += copied + stats["preserved"] += preserved + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", "replace")[:300] + fail("%s release %s failed (HTTP %s): %s" + % (repo_name, tag, exc.code, detail)) + except Exception as exc: + fail("%s release %s failed: %s" % (repo_name, tag, exc)) + return stats + + def main(): cfg = load_config() dry_run = truthy(cfg["DRY_RUN"]) @@ -272,6 +607,8 @@ def main(): "not 'Public repositories'") created = skipped = 0 + release_totals = {"created": 0, "updated": 0, "assets": 0, + "preserved": 0, "waiting": 0} for repo in sorted(repos, key=lambda r: r["name"].lower()): name = repo["name"] try: @@ -281,14 +618,30 @@ def main(): continue if existing is None: + mirror_created = False try: create_mirror(cfg, repo, dry_run) created += 1 + mirror_created = True except urllib.error.HTTPError as exc: detail = exc.read().decode("utf-8", "replace")[:300] fail("%s: migrate failed (HTTP %s): %s" % (name, exc.code, detail)) except Exception as exc: fail("%s: migrate failed: %s" % (name, exc)) + + if mirror_created and truthy(cfg["SYNC_RELEASES"]): + try: + if dry_run: + count = len(github_releases(cfg, name)) + if count: + log("DRY", "would synchronize %d release(s) after creating %s" + % (count, name)) + else: + stats = sync_releases(cfg, name, dry_run) + for key in release_totals: + release_totals[key] += stats[key] + except Exception as exc: + fail("%s: could not synchronize releases: %s" % (name, exc)) continue skipped += 1 @@ -300,6 +653,18 @@ def main(): except Exception as exc: fail("%s: could not update mirror interval: %s" % (name, exc)) + if truthy(cfg["SYNC_RELEASES"]): + try: + stats = sync_releases(cfg, name, dry_run) + for key in release_totals: + release_totals[key] += stats[key] + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", "replace")[:300] + fail("%s: release sync failed (HTTP %s): %s" + % (name, exc.code, detail)) + except Exception as exc: + fail("%s: release sync failed: %s" % (name, exc)) + github_names = {r["name"].lower() for r in repos} try: page, orphans = 1, [] @@ -323,7 +688,12 @@ def main(): except Exception as exc: log("WARN", "Could not check for orphaned mirrors: %s" % exc) - log("INFO", "Done: %d created, %d already present, %d error(s)" % (created, skipped, errors)) + log("INFO", "Done: %d mirror(s) created, %d already present; releases: " + "%d created, %d updated, %d asset(s) copied, %d preserved, " + "%d waiting for tags; %d error(s)" + % (created, skipped, release_totals["created"], release_totals["updated"], + release_totals["assets"], release_totals["preserved"], + release_totals["waiting"], errors)) return 1 if errors else 0 diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..7918cc8 --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1 @@ +"""Tests for gitea-github-mirror.""" diff --git a/tests/test_sync_mirrors.py b/tests/test_sync_mirrors.py new file mode 100644 index 0000000..c8c531c --- /dev/null +++ b/tests/test_sync_mirrors.py @@ -0,0 +1,188 @@ +import io +import unittest +from unittest import mock + +import sync_mirrors as mirror + + +def config(**overrides): + values = { + "GITHUB_USER": "octocat", + "GITHUB_TOKEN": "github-token", + "GITEA_URL": "http://gitea.invalid", + "GITEA_TOKEN": "gitea-token", + "GITEA_OWNER": "backup", + "SYNC_RELEASE_ASSETS": "true", + } + values.update(overrides) + return values + + +def github_release(**overrides): + values = { + "id": 101, + "tag_name": "v1.0.0", + "target_commitish": "main", + "name": "Version 1.0.0", + "body": "Release notes", + "draft": False, + "prerelease": False, + } + values.update(overrides) + return values + + +class ReleaseSyncTests(unittest.TestCase): + def setUp(self): + mirror.errors = 0 + + @mock.patch.object(mirror, "upload_release_asset") + @mock.patch.object(mirror, "download_github_asset") + @mock.patch.object(mirror, "github_release_assets") + @mock.patch.object(mirror, "gitea_tags") + @mock.patch.object(mirror, "gitea_releases") + @mock.patch.object(mirror, "github_releases") + @mock.patch.object(mirror, "api") + def test_creates_release_and_copies_missing_asset( + self, api, github_releases, gitea_releases, gitea_tags, + github_assets, download_asset, upload_asset): + source = github_release() + asset = { + "id": 501, + "name": "program.zip", + "size": 3, + "url": "https://api.github.com/assets/501", + "content_type": "application/zip", + } + github_releases.return_value = [source] + gitea_releases.return_value = [] + gitea_tags.return_value = {"v1.0.0"} + github_assets.return_value = [asset] + download_asset.return_value = (io.BytesIO(b"zip"), 3) + api.return_value = (201, {"id": 901, "tag_name": "v1.0.0", "assets": []}) + + stats = mirror.sync_releases(config(), "project", False) + + self.assertEqual(1, stats["created"]) + self.assertEqual(1, stats["assets"]) + self.assertEqual(0, mirror.errors) + self.assertEqual("POST", api.call_args.kwargs["method"]) + self.assertEqual("v1.0.0", api.call_args.kwargs["payload"]["tag_name"]) + upload_asset.assert_called_once() + + @mock.patch.object(mirror, "github_release_assets") + @mock.patch.object(mirror, "gitea_tags") + @mock.patch.object(mirror, "gitea_releases") + @mock.patch.object(mirror, "github_releases") + @mock.patch.object(mirror, "api") + def test_second_run_does_not_write_matching_release( + self, api, github_releases, gitea_releases, gitea_tags, github_assets): + source = github_release() + target = { + "id": 901, + "tag_name": "v1.0.0", + "name": "Version 1.0.0", + "body": "Release notes", + "draft": False, + "prerelease": False, + "assets": [{"name": "program.zip", "size": 3}], + } + github_releases.return_value = [source] + gitea_releases.return_value = [target] + github_assets.return_value = [{"name": "program.zip", "size": 3}] + + stats = mirror.sync_releases(config(), "project", False) + + self.assertEqual(0, stats["created"]) + self.assertEqual(0, stats["updated"]) + self.assertEqual(0, stats["assets"]) + api.assert_not_called() + gitea_tags.assert_not_called() + + @mock.patch.object(mirror, "gitea_tags", return_value=set()) + @mock.patch.object(mirror, "gitea_releases", return_value=[]) + @mock.patch.object(mirror, "github_releases", return_value=[github_release()]) + @mock.patch.object(mirror, "api") + def test_waits_when_mirror_has_not_fetched_release_tag( + self, api, github_releases, gitea_releases, gitea_tags): + stats = mirror.sync_releases(config(), "project", False) + + self.assertEqual(1, stats["waiting"]) + self.assertEqual(0, stats["created"]) + api.assert_not_called() + + @mock.patch.object(mirror, "gitea_tags") + @mock.patch.object(mirror, "gitea_releases", return_value=[]) + @mock.patch.object( + mirror, "github_releases", return_value=[github_release(draft=True)]) + @mock.patch.object(mirror, "api") + def test_creates_draft_without_requiring_mirrored_tag( + self, api, github_releases, gitea_releases, gitea_tags): + api.return_value = ( + 201, + {"id": 901, "tag_name": "v1.0.0", "draft": True, "assets": []}, + ) + + stats = mirror.sync_releases( + config(SYNC_RELEASE_ASSETS="false"), "project", False) + + self.assertEqual(1, stats["created"]) + self.assertEqual(0, stats["waiting"]) + gitea_tags.assert_not_called() + self.assertTrue(api.call_args.kwargs["payload"]["draft"]) + + @mock.patch.object(mirror, "gitea_releases") + @mock.patch.object(mirror, "github_releases") + @mock.patch.object(mirror, "api") + def test_updates_metadata_without_touching_assets( + self, api, github_releases, gitea_releases): + source = github_release(body="New notes", prerelease=True) + target = { + "id": 901, + "tag_name": "v1.0.0", + "name": "Version 1.0.0", + "body": "Old notes", + "draft": False, + "prerelease": False, + "assets": [], + } + github_releases.return_value = [source] + gitea_releases.return_value = [target] + api.return_value = (200, dict(target, body="New notes", prerelease=True)) + + stats = mirror.sync_releases( + config(SYNC_RELEASE_ASSETS="false"), "project", False) + + self.assertEqual(1, stats["updated"]) + self.assertEqual("PATCH", api.call_args.kwargs["method"]) + self.assertEqual("New notes", api.call_args.kwargs["payload"]["body"]) + self.assertTrue(api.call_args.kwargs["payload"]["prerelease"]) + + @mock.patch.object(mirror, "download_github_asset") + @mock.patch.object(mirror, "github_release_assets") + @mock.patch.object(mirror, "gitea_releases") + @mock.patch.object(mirror, "github_releases") + def test_keeps_changed_asset_in_gitea( + self, github_releases, gitea_releases, github_assets, download_asset): + source = github_release() + target = { + "id": 901, + "tag_name": "v1.0.0", + "name": "Version 1.0.0", + "body": "Release notes", + "draft": False, + "prerelease": False, + "assets": [{"name": "program.zip", "size": 2}], + } + github_releases.return_value = [source] + gitea_releases.return_value = [target] + github_assets.return_value = [{"name": "program.zip", "size": 3}] + + stats = mirror.sync_releases(config(), "project", False) + + self.assertEqual(1, stats["preserved"]) + download_asset.assert_not_called() + + +if __name__ == "__main__": + unittest.main()