fix cron

Add SQLITE_TIMEOUT, BACKUP_RETRIES, and BACKUP_RETRY_DELAY
environment variables to handle "database is locked" errors during
backup. Retry the sqlite3 .backup command with configurable timeouts
and delays, and clean up any WAL/SHM files that may interfere with
subsequent attempts. Bump version to 1.2.2.

.env.example

@@ -1,7 +1,11 @@
 DB_FILE=/data/db.sqlite
-CRONTAB="00 06 * * *"
+CRONTAB=00 06 * * *
 DELETE_AFTER=10
+KEEP_LAST=10
 TZ=Asia/Novosibirsk
+SQLITE_TIMEOUT=60000
+BACKUP_RETRIES=5
+BACKUP_RETRY_DELAY=30
 MINIO_PATH=myminio://sqlite/master/
 MINIO_ACCOUNT_ID=account
 MINIO_APPLICATION_KEY=key

.gitea/workflows/docker-build.yml

@@ -2,8 +2,11 @@ name: docker-build
 
 on:
   push:
-    tags:
-      - "*"
+    branches: [main, master]
+
+env:
+  REGISTRY: git.realmanual.ru
+  IMAGE_PREFIX: ${{ gitea.repository }}
 
 permissions:
   contents: read
@@ -20,12 +23,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
-
-      - name: Log in to the Container registry
+      - name: Read Version
+        id: version
+        run: echo "VERSION=$(cat VERSION)" >> $GITHUB_OUTPUT
+      - name: Log in to Gitea Registry
         uses: docker/login-action@v3
         with:
           registry: ${{ env.REGISTRY }}
-          username: ${{ gitea.actor }}
+          username: ${{ github.actor }}
           password: ${{ secrets.PUSH_TOKEN }}
 
       - name: Build and push Docker image
@@ -35,5 +40,5 @@ jobs:
           context: .
           push: true
           tags: |
-            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ gitea.ref_name }}
+            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.version.outputs.VERSION }}
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest

Dockerfile

@@ -19,8 +19,6 @@ RUN wget https://github.com/rclone/rclone/releases/download/v1.71.0/rclone-v1.71
     chown root:root /usr/bin/rclone && \
     chmod 755 /usr/bin/rclone
 
-RUN export TZ=/usr/share/zoneinfo/${TZ}
-
 RUN mkdir -p scripts
 COPY scripts/ /scripts
 

README.md

@@ -12,7 +12,11 @@ services:
       - DB_FILE
       - CRONTAB
       - DELETE_AFTER
+      - KEEP_LAST
       - TZ
+      - SQLITE_TIMEOUT
+      - BACKUP_RETRIES
+      - BACKUP_RETRY_DELAY
       - MINIO_PATH
       - MINIO_ACCOUNT_ID
       - MINIO_APPLICATION_KEY
@@ -26,9 +30,13 @@ services:
 
 ```bash
 DB_FILE=/data/db.sqlite
-CRONTAB="00 06 * * *"
+CRONTAB=00 06 * * *
 DELETE_AFTER=10
+KEEP_LAST=10
 TZ=Asia/Novosibirsk
+SQLITE_TIMEOUT=60000
+BACKUP_RETRIES=5
+BACKUP_RETRY_DELAY=30
 MINIO_PATH=myminio://sqlite/master/
 MINIO_ACCOUNT_ID=account
 MINIO_APPLICATION_KEY=key
@@ -43,6 +51,34 @@ HEALTHCHECK_UUID=00000000-0000-0000-0000-000000000000
 1. myminio - системно! не менять
 2. проверьте настройки локации, не должно быть пусто
 3. DELETE_AFTER - в днях
+4. KEEP_LAST - минимальное число копий, которые сохраняются всегда
+
+## Ротация
+
+Удаление старых бекапов работает по двум условиям одновременно:
+
+1. бекап старше `DELETE_AFTER` дней
+2. и при этом не входит в `KEEP_LAST` самых свежих копий
+
+То есть последние `KEEP_LAST` бекапов не удаляются никогда, даже если они старше `DELETE_AFTER` дней. Это защищает от потери всех копий, если контейнер долго стоял и новые бекапы не создавались.
+
+`KEEP_LAST` по умолчанию `10`, если переменная не задана.
+
+## Заблокированная база
+
+Если базу пишет живой процесс (grafana, или не добитый старый под), `sqlite3 .backup` падает с `Error: database is locked`.
+
+Бекап это переживает: соединение открывается с `.timeout SQLITE_TIMEOUT` (мс) — sqlite ждёт освобождения блокировки вместо мгновенной ошибки. Если за это время писатель не отпустил базу, попытка повторяется `BACKUP_RETRIES` раз с паузой `BACKUP_RETRY_DELAY` секунд.
+
+| Переменная | Default | Что делает |
+|---|---|---|
+| `SQLITE_TIMEOUT` | `60000` | сколько мс ждать блокировку в рамках одной попытки |
+| `BACKUP_RETRIES` | `5` | сколько попыток снять дамп |
+| `BACKUP_RETRY_DELAY` | `30` | пауза между попытками, сек |
+
+Максимальное время ожидания с дефолтами: `60s * 5 + 30s * 4 = 7 минут`. После этого — `exit 1` и пинг `/fail`.
+
+Если база залочена *постоянно* — это не про бекап, а про то, что старый под не умер. Ретраи только прикрывают короткие окна.
 
 ## Healthchecks
 
@@ -58,10 +94,38 @@ HEALTHCHECK_UUID=00000000-0000-0000-0000-000000000000
 
 Для self-hosted инстанса `HEALTHCHECK_URL` — это адрес его ping-эндпоинта, например `https://hc.domain.ru/ping`.
 
-## Как передаётся окружение в cron
+## Планировщик
 
-`crond` из busybox запускает джобы с пустым окружением, поэтому `entrypoint.sh`
-сохраняет env контейнера в `/tmp/container.env` (права 600), а cron-запись
-вызывает `/scripts/backup.sh --from-cron`, который его подгружает.
-Ручной запуск (`docker exec ... /scripts/backup.sh`) использует то окружение,
+`crond` из busybox не используется: под non-root uid (обычная ситуация в
+kubernetes с `securityContext.runAsUser`) он не загружает ни одного crontab и
+не сообщает об этом — в логе бесконечно идёт `wakeup dt=60`, а бекапов нет.
+Вместо него `entrypoint.sh` содержит свой планировщик на sh.
+
+Что это даёт:
+
+* работает под любым uid;
+* окружение наследуется напрямую, без выгрузки env в файл;
+* бекапы никогда не накладываются друг на друга — следующий запуск возможен
+  только после завершения предыдущего;
+* некорректный `CRONTAB` роняет контейнер на старте с внятным сообщением
+  (`[scheduler] FATAL: minute: '60' is outside 0-59`), а не приводит к тихому
+  простою.
+
+Синтаксис `CRONTAB` — стандартные пять полей: `*`, `5`, `1,15`, `2-6`, `*/15`,
+`8-18/2`, `30/5`. В месяцах и днях недели понимаются имена (`jan`, `mon`).
+Воскресенье — и `0`, и `7`. Если ограничены одновременно день месяца и день
+недели, запуск происходит при совпадении любого из них — как в `crontab(5)`.
+
+При старте контейнера бекап делается сразу, эта минута считается отработанной.
+
+Расписание видно в логе при запуске:
+
+```
+[scheduler] schedule: 30 07 * * * (TZ=Asia/Novosibirsk, now 2026-09-02 16:45:21 +0700)
+[scheduler] expanded: min=[30] hour=[7] dom=[1 2 3 ...] mon=[1 2 ...] dow=[0 1 ...] day-match=and
+```
+
+Ручной запуск: `docker exec ... /scripts/backup.sh` — использует то окружение,
 с которым его вызвали.
+
+Период опроса часов — 20 секунд, меняется через `POLL_INTERVAL` (секунды).

VERSION

@@ -0,0 +1 @@
+1.3.0

docker-compose.yml

@@ -12,7 +12,11 @@ services:
       - DB_FILE
       - CRONTAB
       - DELETE_AFTER
+      - KEEP_LAST
       - TZ
+      - SQLITE_TIMEOUT
+      - BACKUP_RETRIES
+      - BACKUP_RETRY_DELAY
       - MINIO_PATH
       - MINIO_ACCOUNT_ID
       - MINIO_APPLICATION_KEY

scripts/backup.sh

@@ -1,12 +1,5 @@
 #!/bin/sh
 
-# crond gives the job a bare environment (HOME/PATH/SHELL only), so the cron
-# entry passes --from-cron and the env dumped by entrypoint.sh is restored here.
-# A manual run keeps the environment it was invoked with.
-if [ "${1:-}" = "--from-cron" ] && [ -f /tmp/container.env ]; then
-  . /tmp/container.env
-fi
-
 set -u
 
 BACKUP_FILE="sqlite_$(date "+%F-%H%M%S").tar.gz"
@@ -27,7 +20,7 @@ hc_ping() {
     || log "WARN: healthcheck ping failed: ${url}"
 }
 
-cleanup() { rm -f "$DUMP" "$ARCHIVE"; }
+cleanup() { rm -f "$DUMP" "${DUMP}-journal" "${DUMP}-wal" "${DUMP}-shm" "$ARCHIVE"; }
 
 fail() {
   log "ERROR: $*"
@@ -43,10 +36,34 @@ fail() {
 hc_ping start
 log "backup started: $DB_FILE -> ${DEST}${BACKUP_FILE}"
 
-# 1. Consistent snapshot
-rm -f "$DUMP"
-sqlite3 "$DB_FILE" ".backup '$DUMP'" || fail "sqlite3 .backup failed"
-[ -s "$DUMP" ] || fail "dump is empty"
+# 1. Consistent snapshot.
+# A live writer (grafana, or a previous pod that has not died yet) holds the
+# write lock, so `.backup` gets "database is locked". `.timeout` makes sqlite
+# wait instead of failing instantly; the loop covers longer write bursts.
+SQLITE_TIMEOUT="${SQLITE_TIMEOUT:-60000}"
+BACKUP_RETRIES="${BACKUP_RETRIES:-5}"
+BACKUP_RETRY_DELAY="${BACKUP_RETRY_DELAY:-30}"
+
+ATTEMPT=1
+while : ; do
+  rm -f "$DUMP" "${DUMP}-journal" "${DUMP}-wal" "${DUMP}-shm"
+  ERR=$(sqlite3 -cmd ".timeout $SQLITE_TIMEOUT" "$DB_FILE" ".backup '$DUMP'" 2>&1)
+  RC=$?
+
+  if [ "$RC" -eq 0 ] && [ -s "$DUMP" ]; then
+    break
+  fi
+
+  [ -n "$ERR" ] || ERR="dump is empty"
+
+  if [ "$ATTEMPT" -ge "$BACKUP_RETRIES" ]; then
+    fail "sqlite3 .backup failed after ${ATTEMPT} attempts: $ERR"
+  fi
+
+  log "WARN: .backup attempt ${ATTEMPT}/${BACKUP_RETRIES} failed: ${ERR}; retrying in ${BACKUP_RETRY_DELAY}s"
+  sleep "$BACKUP_RETRY_DELAY"
+  ATTEMPT=$((ATTEMPT + 1))
+done
 
 # 2. Verify the snapshot before shipping it anywhere
 INTEGRITY=$(sqlite3 "$DUMP" "PRAGMA integrity_check;" 2>&1) \

scripts/deleteold.sh

@@ -2,14 +2,38 @@
 
 set -u
 
-# Seconds since epoch for current time
+# Retention: delete backups older than DELETE_AFTER days, but always keep
+# at least KEEP_LAST most recent copies regardless of their age.
 DATE_NOW=$(date +%s)
 DEST="${MINIO_PATH%/}/"
+KEEP_LAST="${KEEP_LAST:-10}"
 
-/scripts/minio_uploader.sh list "$DEST" | grep "sqlite_" | while read -r LINE
+case "$KEEP_LAST" in
+  ''|*[!0-9]*) echo "WARN: KEEP_LAST is not a number ('$KEEP_LAST'), falling back to 10"; KEEP_LAST=10 ;;
+esac
+
+LIST=$(mktemp)
+trap 'rm -f "$LIST"' EXIT
+
+# `rclone ls` prints "<size> <path>". Filenames are sqlite_<YYYY-MM-DD-HHMMSS>.tar.gz,
+# so a reverse lexicographic sort is a reverse chronological sort.
+/scripts/minio_uploader.sh list "$DEST" \
+  | awk '{ print $2 }' \
+  | grep "^sqlite_" \
+  | sort -r > "$LIST"
+
+TOTAL=$(wc -l < "$LIST" | tr -d ' ')
+echo "Found $TOTAL backups, keeping at least $KEEP_LAST most recent, max age $DELETE_AFTER days"
+
+if [ "$TOTAL" -le "$KEEP_LAST" ]; then
+  echo "Nothing to rotate: $TOTAL backups <= KEEP_LAST=$KEEP_LAST"
+  exit 0
+fi
+
+# Only the tail beyond the protected window is a candidate for deletion.
+tail -n "+$((KEEP_LAST + 1))" "$LIST" | while read -r BACKUP_FILENAME
 do
-  # `rclone ls` prints "<size> <path>"
-  BACKUP_FILENAME=$(echo "$LINE" | awk '{ print $2 }')
+  [ -n "$BACKUP_FILENAME" ] || continue
 
   BACKUP_DATE=$(echo "$BACKUP_FILENAME" | awk 'BEGIN { FS = "[_-]" } ; { printf "%s-%s-%s",$2,$3,$4 }')
 

scripts/entrypoint.sh

@@ -1,22 +1,193 @@
 #!/bin/sh
 
-set -e
+# Planner for periodic backups.
+#
+# busybox crond is deliberately not used here: when the container runs under a
+# non-root uid (a very common k8s securityContext) crond loads no crontab at
+# all and never says why - it just keeps printing "wakeup dt=60" forever while
+# no backup is ever made. This loop does the same job, works under any uid and
+# fails loudly on a bad schedule.
 
-# crond runs jobs with a bare environment (HOME/PATH/SHELL only), so the
-# container env is dumped here and sourced back by backup.sh on every run.
-ENV_FILE=/tmp/container.env
-export -p | grep -v -E "^export (PWD|SHLVL|OLDPWD|_)=" > "$ENV_FILE"
-chmod 600 "$ENV_FILE"
+# -f matters: the schedule is full of '*' and word splitting would otherwise
+# expand them against the files in the working directory.
+set -euf
 
-# Create crontab in a writable location and set proper permissions
-mkdir -p /tmp/crontabs
-echo "${CRONTAB:-"0 * * * *"} /scripts/backup.sh --from-cron >> /proc/1/fd/1 2>&1" > /tmp/crontabs/root
-chmod 644 /tmp/crontabs/root
+SCHEDULE_RAW="${CRONTAB:-0 * * * *}"
+POLL_INTERVAL="${POLL_INTERVAL:-20}"
 
-echo "[entrypoint] schedule: ${CRONTAB:-"0 * * * *"} (TZ=${TZ:-UTC})"
+log()  { echo "[scheduler] $*"; }
+die()  { echo "[scheduler] FATAL: $*" >&2; exit 1; }
 
-# Run initial backup (do not abort the container if it fails)
-/scripts/backup.sh || echo "[entrypoint] initial backup failed, continuing to crond"
+# A value coming from .env, a configmap or a helm chart may carry quotes or a
+# trailing CR; both silently break the schedule, so strip them up front.
+SCHEDULE=$(printf '%s' "$SCHEDULE_RAW" | tr -d '\r' \
+  | sed -e 's/^"\(.*\)"$/\1/' -e "s/^'\(.*\)'\$/\1/")
 
-# Start crond in foreground with debug output
-exec crond -f -c /tmp/crontabs -d 0
+MONTH_NAMES="jan feb mar apr may jun jul aug sep oct nov dec"
+DOW_NAMES="sun mon tue wed thu fri sat"
+
+# Resolve a three-letter month/weekday name to its number, echo the input back
+# when it is not a name.
+resolve_name() { # $1=token $2=names $3=base
+  _tok=$(echo "$1" | tr '[:upper:]' '[:lower:]')
+  _n=0
+  for _w in $2; do
+    if [ "$_tok" = "$_w" ]; then
+      echo $((_n + $3))
+      return 0
+    fi
+    _n=$((_n + 1))
+  done
+  echo "$1"
+}
+
+# Drop a leading zero so that "07" is not read as an invalid octal number.
+strip_zero() {
+  _v=${1#0}
+  [ -n "$_v" ] || _v=0
+  echo "$_v"
+}
+
+# Expand one crontab field into the explicit list of values it matches.
+# Supports *, N, a-b, a,b, */n, a-b/n and N/n.
+expand_field() { # $1=spec $2=lo $3=hi $4=field name $5=names $6=names base
+  _spec=$1; _lo=$2; _hi=$3; _name=$4; _names=$5; _base=$6
+  _out=''
+  _oifs=$IFS
+  IFS=,
+  for _part in $_spec; do
+    IFS=$_oifs
+
+    _step=1
+    case $_part in
+      */*)
+        _step=${_part##*/}
+        _part=${_part%%/*}
+        ;;
+    esac
+    case $_step in
+      ''|*[!0-9]*) die "$_name: bad step in '$_spec'" ;;
+    esac
+    _step=$(strip_zero "$_step")
+    [ "$_step" -ge 1 ] || die "$_name: step must be >= 1 in '$_spec'"
+
+    case $_part in
+      '*')
+        _s=$_lo; _e=$_hi
+        ;;
+      *-*)
+        _s=$(resolve_name "${_part%%-*}" "$_names" "$_base")
+        _e=$(resolve_name "${_part##*-}" "$_names" "$_base")
+        ;;
+      *)
+        _s=$(resolve_name "$_part" "$_names" "$_base")
+        # "N/step" means "from N to the end of the range", like vixie cron.
+        if [ "$_step" -gt 1 ]; then _e=$_hi; else _e=$_s; fi
+        ;;
+    esac
+
+    for _v in "$_s" "$_e"; do
+      case $_v in
+        ''|*[!0-9]*) die "$_name: '$_part' is not a number or a known name" ;;
+      esac
+    done
+    _s=$(strip_zero "$_s")
+    _e=$(strip_zero "$_e")
+    [ "$_s" -ge "$_lo" ] && [ "$_e" -le "$_hi" ] \
+      || die "$_name: '$_part' is outside ${_lo}-${_hi}"
+    [ "$_s" -le "$_e" ] || die "$_name: reversed range '$_part'"
+
+    _i=$_s
+    while [ "$_i" -le "$_e" ]; do
+      _out="$_out $_i"
+      _i=$((_i + _step))
+    done
+
+    IFS=,
+  done
+  IFS=$_oifs
+  [ -n "$_out" ] || die "$_name: empty field"
+  echo "$_out"
+}
+
+in_list() { # $1=list $2=value
+  case " $1 " in
+    *" $2 "*) return 0 ;;
+  esac
+  return 1
+}
+
+# Word splitting is the point here, and globbing is off (set -f).
+# shellcheck disable=SC2086
+set -- $SCHEDULE
+[ $# -eq 5 ] || die "CRONTAB must have 5 fields, got $#: '$SCHEDULE_RAW'"
+SPEC_MIN=$1 SPEC_HOUR=$2 SPEC_DOM=$3 SPEC_MON=$4 SPEC_DOW=$5
+
+LIST_MIN=$(expand_field  "$SPEC_MIN"  0 59 minute  '' 0)
+LIST_HOUR=$(expand_field "$SPEC_HOUR" 0 23 hour    '' 0)
+LIST_DOM=$(expand_field  "$SPEC_DOM"  1 31 day     '' 0)
+LIST_MON=$(expand_field  "$SPEC_MON"  1 12 month   "$MONTH_NAMES" 1)
+LIST_DOW=$(expand_field  "$SPEC_DOW"  0  7 weekday "$DOW_NAMES"   0)
+# Sunday is both 0 and 7.
+if in_list "$LIST_DOW" 7 && ! in_list "$LIST_DOW" 0; then
+  LIST_DOW="$LIST_DOW 0"
+fi
+
+# crontab(5): when both day-of-month and day-of-week are restricted, a run
+# happens if either of them matches, not both.
+if [ "$SPEC_DOM" != '*' ] && [ "$SPEC_DOW" != '*' ]; then
+  DAY_MATCH=or
+else
+  DAY_MATCH=and
+fi
+
+log "schedule: $SCHEDULE (TZ=${TZ:-UTC}, now $(date '+%F %T %z'))"
+log "expanded: min=[${LIST_MIN# }] hour=[${LIST_HOUR# }] dom=[${LIST_DOM# }] mon=[${LIST_MON# }] dow=[${LIST_DOW# }] day-match=$DAY_MATCH"
+
+matches_now() { # $1=minute $2=hour $3=dom $4=month $5=dow
+  in_list "$LIST_MIN"  "$1" || return 1
+  in_list "$LIST_HOUR" "$2" || return 1
+  in_list "$LIST_MON"  "$4" || return 1
+
+  if [ "$DAY_MATCH" = or ]; then
+    in_list "$LIST_DOM" "$3" || in_list "$LIST_DOW" "$5" || return 1
+  else
+    in_list "$LIST_DOM" "$3" || return 1
+    in_list "$LIST_DOW" "$5" || return 1
+  fi
+  return 0
+}
+
+TERMINATE=0
+trap 'TERMINATE=1' TERM INT
+
+# Initial backup: do not abort the container if it fails, the schedule matters
+# more than this one run.
+/scripts/backup.sh || log "initial backup failed, continuing to the schedule"
+
+# The startup backup counts as this minute's run, otherwise a schedule that
+# matches the minute the container came up would immediately back up twice.
+LAST_RUN=$(date '+%Y-%m-%dT%H:%M')
+while [ "$TERMINATE" -eq 0 ]; do
+  # Polling instead of sleeping to the minute boundary: a backup can take
+  # minutes and the clock can jump, LAST_RUN is what keeps a minute from
+  # firing twice.
+  # shellcheck disable=SC2046
+  set -- $(date '+%Y-%m-%dT%H:%M %M %H %d %m %w')
+  KEY=$1
+  MIN=$(strip_zero "$2"); HOUR=$(strip_zero "$3")
+  DOM=$(strip_zero "$4"); MON=$(strip_zero "$5"); DOW=$(strip_zero "$6")
+
+  if [ "$KEY" != "$LAST_RUN" ] && matches_now "$MIN" "$HOUR" "$DOM" "$MON" "$DOW"; then
+    LAST_RUN=$KEY
+    log "triggered at $KEY"
+    /scripts/backup.sh || log "backup failed, waiting for the next run"
+  fi
+
+  # Backgrounded so that SIGTERM is handled right away instead of after the
+  # whole interval.
+  sleep "$POLL_INTERVAL" &
+  wait $! || true
+done
+
+log "terminating"