mirror of
https://github.com/Andre-Bzk/github-to-gitea-mirror.git
synced 2026-11-15 09:33:13 +01:00
git init
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
# Real configuration -- contains GitHub and Gitea tokens.
|
||||
mirror.env
|
||||
|
||||
# Runtime output
|
||||
sync.log
|
||||
sync.log.tmp
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -0,0 +1,321 @@
|
||||
# GitHub → Gitea Mirror
|
||||
|
||||
[English](#english) | [Deutsch](#deutsch)
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
Mirrors **all** GitHub repositories of a user into a self-hosted Gitea instance
|
||||
as pull mirrors — including repositories created later.
|
||||
|
||||
Gitea can already mirror a single repository through its web UI. What it cannot
|
||||
do is watch an account and pick up new repositories on its own. That gap is the
|
||||
only reason this script exists: a daily cron job asks the GitHub API which
|
||||
repositories exist and creates the mirrors that are missing. Keeping their
|
||||
*content* up to date remains Gitea's job.
|
||||
|
||||
| Task | Handled by | Frequency |
|
||||
| --- | --- | --- |
|
||||
| Discover **new** repositories, create mirrors | this script (cron) | daily |
|
||||
| Pull new commits into **existing** mirrors | Gitea itself | `MIRROR_INTERVAL` |
|
||||
|
||||
### Requirements
|
||||
|
||||
- Gitea 1.20+ with an API token (scopes `write:repository,read:user`)
|
||||
- Python 3.7+ — standard library only, no `pip install`, no `jq`
|
||||
- `cron`
|
||||
|
||||
Deliberately dependency-free so it runs on a small box: on a Raspberry Pi 3B
|
||||
with 1 GB RAM the sync is invisible in `free -h`.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
git clone <this-repo> gitea-github-mirror
|
||||
cd gitea-github-mirror
|
||||
./install.sh
|
||||
```
|
||||
|
||||
`install.sh` copies the scripts to `~/gitea-github-mirror`, creates `mirror.env`
|
||||
from the example, fixes permissions and adds the cron entry. It is safe to
|
||||
re-run — an existing `mirror.env` is never overwritten.
|
||||
|
||||
```bash
|
||||
TARGET=/opt/gitea-mirror ./install.sh # different directory
|
||||
CRON_TIME="0 3" ./install.sh # different schedule
|
||||
```
|
||||
|
||||
Create the Gitea token on the Gitea host:
|
||||
|
||||
```bash
|
||||
docker exec -u git <container> gitea admin user generate-access-token \
|
||||
--username <user> --token-name github-mirror-sync \
|
||||
--scopes write:repository,read:user
|
||||
```
|
||||
|
||||
Then fill in `mirror.env` and verify:
|
||||
|
||||
```bash
|
||||
cd ~/gitea-github-mirror
|
||||
DRY_RUN=true python3 sync_mirrors.py # preview, changes nothing
|
||||
python3 sync_mirrors.py # run for real
|
||||
```
|
||||
|
||||
### Configuration (`mirror.env`)
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `GITHUB_USER` | GitHub account to mirror |
|
||||
| `GITHUB_TOKEN` | See below. Empty = public repositories only |
|
||||
| `GITEA_URL` | Gitea base URL as seen from this machine |
|
||||
| `GITEA_TOKEN` | Gitea API token |
|
||||
| `GITEA_OWNER` | Gitea user the mirrors belong to |
|
||||
| `MIRROR_INTERVAL` | Go duration, e.g. `24h`, `8h`, `30m` |
|
||||
| `INCLUDE_FORKS` | Mirror forks too (default `false`) |
|
||||
| `INCLUDE_ARCHIVED` | Mirror archived repositories (default `true`) |
|
||||
| `DRY_RUN` | `true` = show what would happen, change nothing |
|
||||
|
||||
Every key can be overridden by an environment variable of the same name, which
|
||||
is what `DRY_RUN=true python3 sync_mirrors.py` uses.
|
||||
|
||||
### GitHub token
|
||||
|
||||
Needed only for private repositories. Use a **fine-grained** token
|
||||
(<https://github.com/settings/personal-access-tokens>):
|
||||
|
||||
- Repository access: **All repositories** — required, otherwise repositories
|
||||
created later are invisible to the script and auto-discovery is pointless.
|
||||
- Permissions: **Contents: Read-only**. Metadata read is added automatically,
|
||||
everything else stays on "No access".
|
||||
- Expiration: **No expiration** is the sensible choice for an unattended cron
|
||||
job — an expiring token stalls the mirror without anyone noticing. The token
|
||||
can only read, so an unlimited lifetime costs little here.
|
||||
|
||||
A classic token with scope `repo` works too, but grants full write and delete
|
||||
access to every repository — far more than a read-only mirror needs, for a
|
||||
token that sits in plaintext on the server.
|
||||
|
||||
If you do set an expiry (or your organization enforces one), each run logs the
|
||||
remaining validity, warns from 14 days out and exits non-zero once expired:
|
||||
|
||||
```
|
||||
[INFO] GITHUB_TOKEN valid until 2026-12-31 (145 days)
|
||||
[WARN] GITHUB_TOKEN expires in 7 day(s), on 2026-08-15 - renew it
|
||||
[ERROR] GITHUB_TOKEN expired on 2026-08-01 - renew it
|
||||
```
|
||||
|
||||
### Scope and limits
|
||||
|
||||
- Mirrored is **git data only**: all branches and tags. Issues, pull requests,
|
||||
releases and the wiki are excluded on purpose — a pull mirror never syncs
|
||||
those, so importing them once would leave a snapshot that silently goes
|
||||
stale. To change that, flip the flags in `create_mirror()`.
|
||||
- The script **never deletes** anything in Gitea. If a repository disappears or
|
||||
is renamed on GitHub, its mirror stays and is reported as an orphan in the
|
||||
log — for a backup that is the right direction.
|
||||
- A Gitea repository that exists but is **not** a mirror is left untouched, so
|
||||
a real working repository can never be overwritten.
|
||||
- Mirrors are read-only in Gitea. Push to GitHub, not to the mirror.
|
||||
|
||||
### Files
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `sync_mirrors.py` | Creates missing mirrors, aligns the mirror interval |
|
||||
| `run_sync.sh` | Cron wrapper: appends to `sync.log`, trims it to 1000 lines |
|
||||
| `install.sh` | Installs scripts, config and cron entry |
|
||||
| `mirror.env.example` | Configuration template |
|
||||
| `mirror.env` | Real config with secrets — git-ignored, `chmod 600` |
|
||||
| `sync.log` | Last ~2000 log lines |
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
```bash
|
||||
tail -f ~/gitea-github-mirror/sync.log
|
||||
```
|
||||
|
||||
| Log line | Cause |
|
||||
| --- | --- |
|
||||
| `GitHub API 401` | Token invalid or expired |
|
||||
| `GitHub API 403` | Rate limit, or token lacks repository access |
|
||||
| `migrate failed (HTTP 409)` | Name already taken in Gitea |
|
||||
| `... exists in Gitea but is not a mirror` | Real repo with that name — rename one of them |
|
||||
| `Missing config value(s)` | `mirror.env` not filled in |
|
||||
| `no private repositories` despite a token | Token's repository access is "Public repositories", or no repository permission is set at all |
|
||||
|
||||
**Fine-grained token shows no private repositories.** Repository access and
|
||||
permissions are two separate settings and both are needed: access decides
|
||||
*which* repositories the token covers, permissions decide *what* it may do with
|
||||
them. With "All repositories" but an empty permission list the token can see
|
||||
nothing, and private repositories do not even appear in the listing. Set
|
||||
**Contents: Read-only**; Metadata read is added automatically.
|
||||
|
||||
**After rotating the GitHub token**, updating `mirror.env` is not enough. Gitea
|
||||
stores the credentials inside each private mirror's remote URL when the mirror
|
||||
is created, so existing private mirrors keep using the old token and start
|
||||
failing silently. Either update *Settings → Mirror Settings* per repository in
|
||||
Gitea, or delete the private mirrors and let the next run recreate them. Public
|
||||
mirrors are unaffected — they clone without credentials.
|
||||
|
||||
---
|
||||
|
||||
## Deutsch
|
||||
|
||||
Spiegelt **alle** GitHub-Repositories eines Benutzers als Pull-Mirror in eine
|
||||
selbst gehostete Gitea-Instanz — auch Repositories, die erst später entstehen.
|
||||
|
||||
Ein einzelnes Repository kann Gitea bereits über die Weboberfläche spiegeln.
|
||||
Was Gitea nicht kann: einen Account beobachten und neue Repositories von selbst
|
||||
aufnehmen. Genau dafür gibt es dieses Skript — ein täglicher Cron-Job fragt die
|
||||
GitHub-API, welche Repositories existieren, und legt die fehlenden Mirrors an.
|
||||
Deren *Inhalt* aktuell zu halten bleibt Giteas Aufgabe.
|
||||
|
||||
| Aufgabe | Erledigt von | Häufigkeit |
|
||||
| --- | --- | --- |
|
||||
| **Neue** Repositories finden, Mirror anlegen | dieses Skript (Cron) | täglich |
|
||||
| Commits in **bestehende** Mirrors holen | Gitea selbst | `MIRROR_INTERVAL` |
|
||||
|
||||
### Voraussetzungen
|
||||
|
||||
- Gitea 1.20+ mit API-Token (Scopes `write:repository,read:user`)
|
||||
- Python 3.7+ — nur Standardbibliothek, kein `pip install`, kein `jq`
|
||||
- `cron`
|
||||
|
||||
Bewusst ohne Abhängigkeiten, damit es auf kleiner Hardware läuft: Auf einem
|
||||
Raspberry Pi 3B mit 1 GB RAM ist der Sync in `free -h` nicht zu sehen.
|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
git clone <dieses-repo> gitea-github-mirror
|
||||
cd gitea-github-mirror
|
||||
./install.sh
|
||||
```
|
||||
|
||||
`install.sh` kopiert die Skripte nach `~/gitea-github-mirror`, legt `mirror.env`
|
||||
aus der Vorlage an, setzt die Rechte und trägt den Cron-Job ein. Mehrfaches
|
||||
Ausführen ist unbedenklich — eine vorhandene `mirror.env` wird nie überschrieben.
|
||||
|
||||
```bash
|
||||
TARGET=/opt/gitea-mirror ./install.sh # anderes Verzeichnis
|
||||
CRON_TIME="0 3" ./install.sh # andere Uhrzeit
|
||||
```
|
||||
|
||||
Den Gitea-Token auf dem Gitea-Host erzeugen:
|
||||
|
||||
```bash
|
||||
docker exec -u git <container> gitea admin user generate-access-token \
|
||||
--username <user> --token-name github-mirror-sync \
|
||||
--scopes write:repository,read:user
|
||||
```
|
||||
|
||||
Danach `mirror.env` ausfüllen und prüfen:
|
||||
|
||||
```bash
|
||||
cd ~/gitea-github-mirror
|
||||
DRY_RUN=true python3 sync_mirrors.py # Vorschau, ändert nichts
|
||||
python3 sync_mirrors.py # echter Lauf
|
||||
```
|
||||
|
||||
### Konfiguration (`mirror.env`)
|
||||
|
||||
| Schlüssel | Bedeutung |
|
||||
| --- | --- |
|
||||
| `GITHUB_USER` | Zu spiegelnder GitHub-Account |
|
||||
| `GITHUB_TOKEN` | Siehe unten. Leer = nur öffentliche Repositories |
|
||||
| `GITEA_URL` | Gitea-Basis-URL, von dieser Maschine aus gesehen |
|
||||
| `GITEA_TOKEN` | Gitea-API-Token |
|
||||
| `GITEA_OWNER` | Gitea-Benutzer, dem die Mirrors gehören |
|
||||
| `MIRROR_INTERVAL` | Go-Duration, z. B. `24h`, `8h`, `30m` |
|
||||
| `INCLUDE_FORKS` | Forks mitspiegeln (Standard `false`) |
|
||||
| `INCLUDE_ARCHIVED` | Archivierte Repositories spiegeln (Standard `true`) |
|
||||
| `DRY_RUN` | `true` = nur anzeigen, nichts ändern |
|
||||
|
||||
Jeder Schlüssel lässt sich durch eine gleichnamige Umgebungsvariable
|
||||
überschreiben — genau das nutzt `DRY_RUN=true python3 sync_mirrors.py`.
|
||||
|
||||
### GitHub-Token
|
||||
|
||||
Nur für private Repositories nötig. Empfohlen ist ein **fine-grained** Token
|
||||
(<https://github.com/settings/personal-access-tokens>):
|
||||
|
||||
- Repository access: **All repositories** — zwingend, sonst sind später
|
||||
angelegte Repositories für das Skript unsichtbar und die automatische
|
||||
Erkennung wäre sinnlos.
|
||||
- Permissions: **Contents: Read-only**. Metadata-Lesen kommt automatisch dazu,
|
||||
alles andere bleibt auf „No access".
|
||||
- Expiration: **No expiration** ist für einen unbeaufsichtigten Cron-Job die
|
||||
sinnvolle Wahl — ein ablaufender Token legt den Spiegel still, ohne dass es
|
||||
jemand merkt. Der Token kann ohnehin nur lesen, unbegrenzte Laufzeit kostet
|
||||
hier also wenig.
|
||||
|
||||
Ein Classic-Token mit Scope `repo` funktioniert ebenfalls, gibt aber Vollzugriff
|
||||
zum Schreiben und Löschen auf jedes Repository — deutlich mehr, als ein
|
||||
Lese-Spiegel braucht, für einen Token, der im Klartext auf dem Server liegt.
|
||||
|
||||
Wird doch ein Ablaufdatum gesetzt (oder von der Organisation erzwungen),
|
||||
protokolliert jeder Lauf die Restlaufzeit, warnt ab 14 Tagen und endet nach
|
||||
Ablauf mit Exit-Code 1:
|
||||
|
||||
```
|
||||
[INFO] GITHUB_TOKEN valid until 2026-12-31 (145 days)
|
||||
[WARN] GITHUB_TOKEN expires in 7 day(s), on 2026-08-15 - renew it
|
||||
[ERROR] GITHUB_TOKEN expired on 2026-08-01 - renew it
|
||||
```
|
||||
|
||||
### Umfang und Grenzen
|
||||
|
||||
- Gespiegelt werden **nur Git-Daten**: alle Branches und Tags. Issues, Pull
|
||||
Requests, Releases und Wiki bleiben bewusst außen vor — ein Pull-Mirror
|
||||
synchronisiert das nie, ein einmaliger Import würde also stillschweigend
|
||||
veralten. Umstellbar über die Flags in `create_mirror()`.
|
||||
- Das Skript **löscht nie** etwas in Gitea. Verschwindet ein Repository auf
|
||||
GitHub oder wird umbenannt, bleibt der Mirror bestehen und wird im Log als
|
||||
verwaist gemeldet — für ein Backup ist das die richtige Richtung.
|
||||
- Ein Gitea-Repository, das existiert, aber **kein** Mirror ist, wird nicht
|
||||
angefasst; ein echtes Arbeits-Repository kann so nie überschrieben werden.
|
||||
- Mirrors sind in Gitea schreibgeschützt. Gepusht wird nach GitHub, nicht in
|
||||
den Spiegel.
|
||||
|
||||
### Dateien
|
||||
|
||||
| Datei | Zweck |
|
||||
| --- | --- |
|
||||
| `sync_mirrors.py` | Legt fehlende Mirrors an, gleicht das Mirror-Intervall ab |
|
||||
| `run_sync.sh` | Cron-Wrapper: schreibt nach `sync.log`, kürzt es auf 1000 Zeilen |
|
||||
| `install.sh` | Installiert Skripte, Konfiguration und Cron-Eintrag |
|
||||
| `mirror.env.example` | Konfigurationsvorlage |
|
||||
| `mirror.env` | Echte Konfiguration mit Secrets — git-ignoriert, `chmod 600` |
|
||||
| `sync.log` | Die letzten ~2000 Log-Zeilen |
|
||||
|
||||
### Fehlersuche
|
||||
|
||||
```bash
|
||||
tail -f ~/gitea-github-mirror/sync.log
|
||||
```
|
||||
|
||||
| Log-Zeile | Ursache |
|
||||
| --- | --- |
|
||||
| `GitHub API 401` | Token ungültig oder abgelaufen |
|
||||
| `GitHub API 403` | Rate Limit, oder Token hat keinen Repository-Zugriff |
|
||||
| `migrate failed (HTTP 409)` | Name in Gitea bereits vergeben |
|
||||
| `... exists in Gitea but is not a mirror` | Echtes Repo gleichen Namens — eines umbenennen |
|
||||
| `Missing config value(s)` | `mirror.env` nicht ausgefüllt |
|
||||
| `no private repositories` trotz Token | Repository access steht auf „Public repositories", oder es ist gar keine Repository-Permission gesetzt |
|
||||
|
||||
**Fine-grained Token zeigt keine privaten Repos.** Repository access und
|
||||
Permissions sind zwei getrennte Einstellungen, beide werden gebraucht: Der
|
||||
Access bestimmt, *welche* Repositories der Token betrifft, die Permissions,
|
||||
*was* er damit darf. Mit „All repositories", aber leerer Permission-Liste sieht
|
||||
der Token nichts — private Repositories erscheinen dann nicht einmal in der
|
||||
Auflistung. **Contents: Read-only** setzen, Metadata-Lesen kommt automatisch
|
||||
dazu.
|
||||
|
||||
**Nach dem Wechsel des GitHub-Tokens** genügt es nicht, `mirror.env`
|
||||
anzupassen. Gitea hinterlegt die Zugangsdaten beim Anlegen eines privaten
|
||||
Mirrors fest in dessen Remote-URL — bestehende private Mirrors arbeiten also
|
||||
weiter mit dem alten Token und scheitern still. Entweder pro Repository unter
|
||||
*Einstellungen → Mirror-Einstellungen* in Gitea nachziehen, oder die privaten
|
||||
Mirrors löschen und vom nächsten Lauf neu anlegen lassen. Öffentliche Mirrors
|
||||
sind nicht betroffen, sie klonen ohne Zugangsdaten.
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
#!/bin/sh
|
||||
# Install the GitHub -> Gitea mirror sync on this machine.
|
||||
#
|
||||
# Safe to re-run: an existing mirror.env is never overwritten and the cron
|
||||
# entry is only added once.
|
||||
#
|
||||
# TARGET=/opt/gitea-mirror ./install.sh # override install directory
|
||||
# CRON_TIME="0 3" ./install.sh # override schedule (minute hour)
|
||||
set -eu
|
||||
|
||||
SRC="$(dirname "$(readlink -f "$0")")"
|
||||
TARGET="${TARGET:-$HOME/gitea-github-mirror}"
|
||||
CRON_TIME="${CRON_TIME:-30 4}"
|
||||
|
||||
say() { printf '%s\n' "$*"; }
|
||||
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
# --- prerequisites ---
|
||||
command -v python3 >/dev/null 2>&1 || die "python3 not found"
|
||||
python3 -c 'import json, urllib.request' 2>/dev/null \
|
||||
|| die "python3 is missing the standard library modules json/urllib"
|
||||
command -v crontab >/dev/null 2>&1 || die "crontab not found (install cron)"
|
||||
say "python3 $(python3 -c 'import platform; print(platform.python_version())') ok"
|
||||
|
||||
# --- files ---
|
||||
mkdir -p "$TARGET"
|
||||
for f in sync_mirrors.py run_sync.sh README.md; do
|
||||
[ -f "$SRC/$f" ] || die "missing source file: $f"
|
||||
cp "$SRC/$f" "$TARGET/$f"
|
||||
# Files may arrive from Windows; CRLF would break the shebang.
|
||||
sed -i 's/\r$//' "$TARGET/$f"
|
||||
done
|
||||
chmod 700 "$TARGET/sync_mirrors.py" "$TARGET/run_sync.sh"
|
||||
say "installed scripts in $TARGET"
|
||||
|
||||
if [ -f "$TARGET/mirror.env" ]; then
|
||||
chmod 600 "$TARGET/mirror.env"
|
||||
say "kept existing mirror.env"
|
||||
CONFIGURED=1
|
||||
else
|
||||
cp "$SRC/mirror.env.example" "$TARGET/mirror.env"
|
||||
sed -i 's/\r$//' "$TARGET/mirror.env"
|
||||
chmod 600 "$TARGET/mirror.env"
|
||||
say "created $TARGET/mirror.env from the example"
|
||||
CONFIGURED=0
|
||||
fi
|
||||
|
||||
# --- cron ---
|
||||
CRON_LINE="$CRON_TIME * * * $TARGET/run_sync.sh"
|
||||
if crontab -l 2>/dev/null | grep -Fq "$TARGET/run_sync.sh"; then
|
||||
say "cron entry already present"
|
||||
else
|
||||
{ crontab -l 2>/dev/null || true; printf '%s\n' "$CRON_LINE"; } | crontab -
|
||||
say "cron entry added: $CRON_LINE"
|
||||
fi
|
||||
|
||||
say ""
|
||||
if [ "$CONFIGURED" = "0" ]; then
|
||||
say "Next: edit $TARGET/mirror.env (GITHUB_USER, GITEA_TOKEN, GITEA_OWNER),"
|
||||
say "then preview with:"
|
||||
else
|
||||
say "Next: verify with"
|
||||
fi
|
||||
say " cd $TARGET && DRY_RUN=true python3 sync_mirrors.py"
|
||||
@@ -0,0 +1,26 @@
|
||||
# Copy to mirror.env and fill in. Contains secrets -- keep at chmod 600.
|
||||
# mirror.env is git-ignored on purpose; never commit the real file.
|
||||
|
||||
# --- GitHub (source) ---
|
||||
GITHUB_USER=your-github-username
|
||||
# Fine-grained token: "All repositories" + Contents: Read-only + No expiration.
|
||||
# Leave empty to mirror public repositories only.
|
||||
GITHUB_TOKEN=
|
||||
|
||||
# --- Gitea (target) ---
|
||||
# As seen from the machine running this script.
|
||||
GITEA_URL=http://127.0.0.1:3000
|
||||
# Create with:
|
||||
# docker exec -u git <container> gitea admin user generate-access-token \
|
||||
# --username <user> --token-name github-mirror-sync \
|
||||
# --scopes write:repository,read:user
|
||||
GITEA_TOKEN=
|
||||
GITEA_OWNER=your-gitea-username
|
||||
|
||||
# --- Behaviour ---
|
||||
# How often Gitea pulls from GitHub (Go duration: 24h, 8h, 30m, ...).
|
||||
MIRROR_INTERVAL=24h
|
||||
INCLUDE_FORKS=false
|
||||
INCLUDE_ARCHIVED=true
|
||||
# Set to true to preview without changing anything.
|
||||
DRY_RUN=false
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
# Cron wrapper: runs the mirror sync and keeps its log from growing unbounded
|
||||
# on the Pi's SD card. All output goes to the log, so cron stays silent.
|
||||
set -u
|
||||
|
||||
DIR="$(dirname "$(readlink -f "$0")")"
|
||||
LOG="$DIR/sync.log"
|
||||
|
||||
/usr/bin/python3 "$DIR/sync_mirrors.py" >>"$LOG" 2>&1
|
||||
status=$?
|
||||
|
||||
if [ -f "$LOG" ] && [ "$(wc -l <"$LOG")" -gt 2000 ]; then
|
||||
tail -n 1000 "$LOG" >"$LOG.tmp" && mv "$LOG.tmp" "$LOG"
|
||||
fi
|
||||
|
||||
exit $status
|
||||
+331
@@ -0,0 +1,331 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Mirror all GitHub repositories of a user into Gitea as pull mirrors.
|
||||
|
||||
Existing mirrors are left to Gitea's own cron; this script only creates the
|
||||
ones that are missing and keeps the mirror interval aligned with the config.
|
||||
It never deletes anything on the Gitea side -- a mirror that vanished upstream
|
||||
is exactly what a backup is supposed to keep.
|
||||
"""
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
KNOWN_KEYS = (
|
||||
"GITHUB_USER",
|
||||
"GITHUB_TOKEN",
|
||||
"GITEA_URL",
|
||||
"GITEA_TOKEN",
|
||||
"GITEA_OWNER",
|
||||
"MIRROR_INTERVAL",
|
||||
"INCLUDE_FORKS",
|
||||
"INCLUDE_ARCHIVED",
|
||||
"DRY_RUN",
|
||||
)
|
||||
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
CONFIG_PATH = os.environ.get("MIRROR_ENV", os.path.join(SCRIPT_DIR, "mirror.env"))
|
||||
|
||||
errors = 0
|
||||
last_headers = {}
|
||||
|
||||
|
||||
def log(level, msg):
|
||||
print("%s [%s] %s" % (datetime.now().strftime("%Y-%m-%d %H:%M:%S"), level, msg), flush=True)
|
||||
|
||||
|
||||
def fail(msg):
|
||||
global errors
|
||||
errors += 1
|
||||
log("ERROR", msg)
|
||||
|
||||
|
||||
def load_config():
|
||||
cfg = {
|
||||
"GITHUB_USER": "",
|
||||
"GITHUB_TOKEN": "",
|
||||
"GITEA_URL": "http://127.0.0.1:3000",
|
||||
"GITEA_TOKEN": "",
|
||||
"GITEA_OWNER": "",
|
||||
"MIRROR_INTERVAL": "24h",
|
||||
"INCLUDE_FORKS": "false",
|
||||
"INCLUDE_ARCHIVED": "true",
|
||||
"DRY_RUN": "false",
|
||||
}
|
||||
if os.path.exists(CONFIG_PATH):
|
||||
with open(CONFIG_PATH, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, _, value = line.partition("=")
|
||||
key = key.strip()
|
||||
if key in KNOWN_KEYS:
|
||||
cfg[key] = value.strip().strip('"').strip("'")
|
||||
else:
|
||||
log("WARN", "No config file at %s, relying on environment" % CONFIG_PATH)
|
||||
|
||||
for key in KNOWN_KEYS:
|
||||
if os.environ.get(key):
|
||||
cfg[key] = os.environ[key]
|
||||
|
||||
cfg["GITEA_URL"] = cfg["GITEA_URL"].rstrip("/")
|
||||
return cfg
|
||||
|
||||
|
||||
def truthy(value):
|
||||
return str(value).strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def api(url, token=None, scheme="token", method="GET", payload=None):
|
||||
"""Return (status, parsed_body). Raises urllib.error.HTTPError on >=400."""
|
||||
data = None
|
||||
headers = {
|
||||
"Accept": "application/json",
|
||||
"User-Agent": "gitea-github-mirror/1.0",
|
||||
}
|
||||
if payload is not None:
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
headers["Content-Type"] = "application/json"
|
||||
if token:
|
||||
headers["Authorization"] = "%s %s" % (scheme, token)
|
||||
|
||||
req = urllib.request.Request(url, data=data, headers=headers, method=method)
|
||||
with urllib.request.urlopen(req, timeout=180) as resp:
|
||||
body = resp.read().decode("utf-8")
|
||||
last_headers.clear()
|
||||
last_headers.update({k.lower(): v for k, v in resp.headers.items()})
|
||||
return resp.status, (json.loads(body) if body.strip() else None)
|
||||
|
||||
|
||||
def github_repos(cfg):
|
||||
"""All repos owned by the user; private ones included when a token is set."""
|
||||
token = cfg["GITHUB_TOKEN"]
|
||||
repos = []
|
||||
page = 1
|
||||
while True:
|
||||
if token:
|
||||
url = ("https://api.github.com/user/repos"
|
||||
"?per_page=100&affiliation=owner&page=%d" % page)
|
||||
else:
|
||||
url = ("https://api.github.com/users/%s/repos"
|
||||
"?per_page=100&type=owner&page=%d" % (cfg["GITHUB_USER"], page))
|
||||
_, batch = api(url, token=token, scheme="Bearer")
|
||||
if not batch:
|
||||
break
|
||||
repos.extend(batch)
|
||||
if len(batch) < 100:
|
||||
break
|
||||
page += 1
|
||||
|
||||
if token:
|
||||
# affiliation=owner still returns repos owned by orgs the user belongs to
|
||||
repos = [r for r in repos
|
||||
if r["owner"]["login"].lower() == cfg["GITHUB_USER"].lower()]
|
||||
return repos
|
||||
|
||||
|
||||
def warn_token_expiry():
|
||||
"""Fine-grained PATs always expire; a silent expiry would stall the mirror.
|
||||
|
||||
GitHub reports the date in a response header, so every run can check it.
|
||||
"""
|
||||
raw = last_headers.get("github-authentication-token-expiration")
|
||||
if not raw:
|
||||
return # classic token without expiry, or no token at all
|
||||
text = raw.strip().replace(" UTC", "").replace("Z", "").replace("T", " ")
|
||||
parsed = None
|
||||
for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%d %H:%M:%S %z", "%Y-%m-%d"):
|
||||
try:
|
||||
parsed = datetime.strptime(text.split(".")[0], fmt)
|
||||
break
|
||||
except ValueError:
|
||||
continue
|
||||
if parsed is None:
|
||||
return
|
||||
|
||||
now_utc = datetime.now(timezone.utc).replace(tzinfo=None)
|
||||
days = (parsed.replace(tzinfo=None) - now_utc).days
|
||||
if days < 0:
|
||||
fail("GITHUB_TOKEN expired on %s - renew it" % parsed.date())
|
||||
elif days <= 14:
|
||||
log("WARN", "GITHUB_TOKEN expires in %d day(s), on %s - renew it"
|
||||
% (days, parsed.date()))
|
||||
else:
|
||||
log("INFO", "GITHUB_TOKEN valid until %s (%d days)" % (parsed.date(), days))
|
||||
|
||||
|
||||
def duration_seconds(value):
|
||||
"""Parse a Go-style duration ('24h', '8h0m0s') into seconds. -1 if unparseable."""
|
||||
if not value:
|
||||
return -1
|
||||
total = 0
|
||||
matched = False
|
||||
for amount, unit in re.findall(r"(\d+)\s*([hms])", str(value)):
|
||||
matched = True
|
||||
total += int(amount) * {"h": 3600, "m": 60, "s": 1}[unit]
|
||||
return total if matched else -1
|
||||
|
||||
|
||||
def gitea_repo(cfg, name):
|
||||
try:
|
||||
_, repo = api("%s/api/v1/repos/%s/%s" % (cfg["GITEA_URL"], cfg["GITEA_OWNER"], name),
|
||||
token=cfg["GITEA_TOKEN"])
|
||||
return repo
|
||||
except urllib.error.HTTPError as exc:
|
||||
if exc.code == 404:
|
||||
return None
|
||||
raise
|
||||
|
||||
|
||||
def create_mirror(cfg, repo, dry_run):
|
||||
name = repo["name"]
|
||||
payload = {
|
||||
"clone_addr": repo["clone_url"],
|
||||
"repo_name": name,
|
||||
"repo_owner": cfg["GITEA_OWNER"],
|
||||
"mirror": True,
|
||||
"mirror_interval": cfg["MIRROR_INTERVAL"],
|
||||
"private": bool(repo.get("private")),
|
||||
"description": (repo.get("description") or "")[:255],
|
||||
"service": "github",
|
||||
# Pull mirrors only ever sync git data, so importing the rest would be a
|
||||
# one-off snapshot that silently goes stale. Keep it lean on the Pi.
|
||||
"wiki": False,
|
||||
"issues": False,
|
||||
"labels": False,
|
||||
"milestones": False,
|
||||
"releases": False,
|
||||
"pull_requests": False,
|
||||
}
|
||||
if cfg["GITHUB_TOKEN"]:
|
||||
payload["auth_token"] = cfg["GITHUB_TOKEN"]
|
||||
|
||||
if dry_run:
|
||||
log("DRY", "would mirror %s (private=%s)" % (name, payload["private"]))
|
||||
return
|
||||
|
||||
api("%s/api/v1/repos/migrate" % cfg["GITEA_URL"], token=cfg["GITEA_TOKEN"],
|
||||
method="POST", payload=payload)
|
||||
log("OK", "mirrored %s (private=%s)" % (name, payload["private"]))
|
||||
|
||||
|
||||
def align_interval(cfg, name, existing, dry_run):
|
||||
wanted = duration_seconds(cfg["MIRROR_INTERVAL"])
|
||||
current = duration_seconds(existing.get("mirror_interval"))
|
||||
if wanted < 0 or current < 0 or wanted == current:
|
||||
return
|
||||
if dry_run:
|
||||
log("DRY", "would set %s interval %s -> %s"
|
||||
% (name, existing.get("mirror_interval"), cfg["MIRROR_INTERVAL"]))
|
||||
return
|
||||
api("%s/api/v1/repos/%s/%s" % (cfg["GITEA_URL"], cfg["GITEA_OWNER"], name),
|
||||
token=cfg["GITEA_TOKEN"], method="PATCH",
|
||||
payload={"mirror_interval": cfg["MIRROR_INTERVAL"]})
|
||||
log("OK", "interval of %s set to %s" % (name, cfg["MIRROR_INTERVAL"]))
|
||||
|
||||
|
||||
def main():
|
||||
cfg = load_config()
|
||||
dry_run = truthy(cfg["DRY_RUN"])
|
||||
|
||||
missing = [k for k in ("GITHUB_USER", "GITEA_URL", "GITEA_TOKEN", "GITEA_OWNER")
|
||||
if not cfg[k]]
|
||||
if missing:
|
||||
log("ERROR", "Missing config value(s): %s" % ", ".join(missing))
|
||||
return 2
|
||||
|
||||
if not cfg["GITHUB_TOKEN"]:
|
||||
log("WARN", "GITHUB_TOKEN is empty - private repositories will be skipped")
|
||||
|
||||
try:
|
||||
repos = github_repos(cfg)
|
||||
except urllib.error.HTTPError as exc:
|
||||
log("ERROR", "GitHub API %s: %s" % (exc.code, exc.read().decode("utf-8", "replace")[:300]))
|
||||
return 2
|
||||
except Exception as exc:
|
||||
log("ERROR", "GitHub API unreachable: %s" % exc)
|
||||
return 2
|
||||
|
||||
warn_token_expiry() # must run before any Gitea call overwrites the headers
|
||||
|
||||
if not truthy(cfg["INCLUDE_FORKS"]):
|
||||
repos = [r for r in repos if not r.get("fork")]
|
||||
if not truthy(cfg["INCLUDE_ARCHIVED"]):
|
||||
repos = [r for r in repos if not r.get("archived")]
|
||||
|
||||
private_count = sum(1 for r in repos if r.get("private"))
|
||||
log("INFO", "GitHub: %d repositor%s to consider (%d public, %d private)%s"
|
||||
% (len(repos), "y" if len(repos) == 1 else "ies",
|
||||
len(repos) - private_count, private_count,
|
||||
" (DRY RUN)" if dry_run else ""))
|
||||
|
||||
# A fine-grained token defaults to "Public repositories", which looks exactly
|
||||
# like an account without private repos. Name the likely cause explicitly.
|
||||
if cfg["GITHUB_TOKEN"] and private_count == 0:
|
||||
log("WARN", "Token is set but GitHub reports no private repositories - "
|
||||
"check that its repository access is 'All repositories', "
|
||||
"not 'Public repositories'")
|
||||
|
||||
created = skipped = 0
|
||||
for repo in sorted(repos, key=lambda r: r["name"].lower()):
|
||||
name = repo["name"]
|
||||
try:
|
||||
existing = gitea_repo(cfg, name)
|
||||
except Exception as exc:
|
||||
fail("%s: cannot query Gitea: %s" % (name, exc))
|
||||
continue
|
||||
|
||||
if existing is None:
|
||||
try:
|
||||
create_mirror(cfg, repo, dry_run)
|
||||
created += 1
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:300]
|
||||
fail("%s: migrate failed (HTTP %s): %s" % (name, exc.code, detail))
|
||||
except Exception as exc:
|
||||
fail("%s: migrate failed: %s" % (name, exc))
|
||||
continue
|
||||
|
||||
skipped += 1
|
||||
if not existing.get("mirror"):
|
||||
log("WARN", "%s exists in Gitea but is not a mirror - left untouched" % name)
|
||||
continue
|
||||
try:
|
||||
align_interval(cfg, name, existing, dry_run)
|
||||
except Exception as exc:
|
||||
fail("%s: could not update mirror interval: %s" % (name, exc))
|
||||
|
||||
github_names = {r["name"].lower() for r in repos}
|
||||
try:
|
||||
page, orphans = 1, []
|
||||
while True:
|
||||
_, batch = api("%s/api/v1/user/repos?limit=50&page=%d" % (cfg["GITEA_URL"], page),
|
||||
token=cfg["GITEA_TOKEN"])
|
||||
if not batch:
|
||||
break
|
||||
orphans.extend(
|
||||
r["name"] for r in batch
|
||||
if r.get("mirror")
|
||||
and r["owner"]["login"].lower() == cfg["GITEA_OWNER"].lower()
|
||||
and r["name"].lower() not in github_names
|
||||
)
|
||||
if len(batch) < 50:
|
||||
break
|
||||
page += 1
|
||||
if orphans:
|
||||
log("WARN", "Mirrors with no GitHub counterpart (renamed or deleted upstream): %s"
|
||||
% ", ".join(sorted(orphans)))
|
||||
except Exception as exc:
|
||||
log("WARN", "Could not check for orphaned mirrors: %s" % exc)
|
||||
|
||||
log("INFO", "Done: %d created, %d already present, %d error(s)" % (created, skipped, errors))
|
||||
return 1 if errors else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user