Compare commits
50
Commits
4b9c806719
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0b12d2ed2d
|
||
|
|
039ac2f1da
|
||
|
|
c077b368f3
|
||
|
|
76ada57dd7
|
||
|
|
bb3635e517
|
||
|
|
8bc7ff026d
|
||
|
|
94fb410c59
|
||
|
|
6bf3a6c4ca
|
||
|
|
c741cd19a0
|
||
|
|
b6e68bdd90
|
||
|
|
d125320667
|
||
|
|
b352cda166
|
||
|
|
5d296c39b1
|
||
|
|
fc335dee3f
|
||
|
|
b439eee164
|
||
|
|
2e5036aad5
|
||
|
|
599c66189c
|
||
|
|
315e56046b
|
||
|
|
692a83a468
|
||
|
|
6d8471efdd
|
||
|
|
3eac1cea7d
|
||
|
|
b076ac92d6
|
||
|
|
e84366eb0c
|
||
|
|
95ddbf5619
|
||
|
|
2bdb4904d6
|
||
|
|
7ad326fa36
|
||
|
|
45b0ff2358
|
||
|
|
2623bc8815
|
||
|
|
b9cc7749c8
|
||
|
|
d5192e1cf7
|
||
|
|
dca1363ec9
|
||
|
|
8656f46674
|
||
|
|
e104a77de1
|
||
|
|
64bcb1d60b
|
||
|
|
e8acab6920
|
||
|
|
d2c69c6a5e
|
||
|
|
be777dc908
|
||
|
|
b9421d388c
|
||
|
|
8f93dcd97b
|
||
|
|
e8f29064fb
|
||
|
|
1810497c2a
|
||
|
|
b59ce67497
|
||
|
|
964e4cfc33
|
||
|
|
1ba7b54e05
|
||
|
|
bdb3b72f38
|
||
|
|
bc72f8172f
|
||
|
|
d6d17ee544
|
||
|
|
e7870c6aa4
|
||
|
|
79fd200e57
|
||
|
|
d909618ced
|
@@ -6,3 +6,8 @@
|
|||||||
!/internal/httpapi/webdist/index.html
|
!/internal/httpapi/webdist/index.html
|
||||||
|
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|
||||||
|
# local cache of the impeccable design hook
|
||||||
|
.impeccable/
|
||||||
|
**/.impeccable/
|
||||||
|
*.csv
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
{
|
||||||
|
"files": ["web/index.html"],
|
||||||
|
"insertBefore": "</body>",
|
||||||
|
"commentSyntax": "html",
|
||||||
|
"cspChecked": true
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
# Design
|
||||||
|
|
||||||
|
Terminal / ops-console visual language for an IMAP migration tool. Near-black,
|
||||||
|
green-tinted dark surface; a single amber functional accent; monospaced body with
|
||||||
|
a condensed display face for headings. Dense, instrument-panel layouts where the
|
||||||
|
system state is always legible. Captured from `web/src/index.css` and
|
||||||
|
`web/src/app.css` — these tokens are canonical; new work extends them, it does not
|
||||||
|
replace them.
|
||||||
|
|
||||||
|
## Theme
|
||||||
|
|
||||||
|
Dark, committed. Not "dark to look cool" — the scene is an operator running a
|
||||||
|
long-lived migration and watching thousands of counters tick; a dark console
|
||||||
|
keeps the amber accent and the ok/fail/pending signal colors doing the reading
|
||||||
|
work with minimal eye strain. Color strategy: **restrained** — tinted neutrals
|
||||||
|
carry the surface, one amber accent stays under ~10% of the pixels, and the
|
||||||
|
green/red/yellow/blue roles appear only as status semantics.
|
||||||
|
|
||||||
|
Signature textures (keep, don't multiply):
|
||||||
|
- A faint amber radial glow at the top of the page (`radial-gradient` at 50% -10%).
|
||||||
|
- A 3px repeating-linear scanline overlay at ~1.2% white — the CRT tell.
|
||||||
|
- Panels wear a floating uppercase tab label (`.panel-label`) punched through the
|
||||||
|
border; the log panel mirrors it on the right (`.log-clear`).
|
||||||
|
|
||||||
|
## Color
|
||||||
|
|
||||||
|
OKLCH is the target space for any new color; existing tokens are hex and stay hex.
|
||||||
|
|
||||||
|
### Surfaces (green-tinted neutrals, darkest → raised)
|
||||||
|
- `--bg` `#0a0d0b` — page base
|
||||||
|
- `--bg-inset` `#070a08` — insets: inputs, progress track, log pane
|
||||||
|
- `--bg-panel` `#0f1512` — panels, cards, login card
|
||||||
|
- `--bg-panel-raised` `#141b17` — topbar, table headers, modal dialog
|
||||||
|
|
||||||
|
### Borders
|
||||||
|
- `--border` `#23342b` — default hairline (1px)
|
||||||
|
- `--border-bright` `#3a5443` — raised/interactive edges, dashed underlines
|
||||||
|
|
||||||
|
### Ink
|
||||||
|
- `--fg` `#dbe8de` — primary text
|
||||||
|
- `--fg-dim` `#6f8478` — labels, secondary text, nav idle
|
||||||
|
- `--fg-faint` `#4a5c50` — hints, empty states, faint indices
|
||||||
|
|
||||||
|
### Accent (amber — attention & primary action)
|
||||||
|
- `--accent` `#ffb238` — primary buttons, active nav, panel labels, progress fill
|
||||||
|
- `--accent-strong` `#ffd27a` — hover/emphasis
|
||||||
|
- `--accent-dim` `#7a5a26` — active borders, tinted glows
|
||||||
|
- Primary-button ink is near-black `#1a1200` on amber, never white.
|
||||||
|
|
||||||
|
### Status semantics (color always carries meaning)
|
||||||
|
- `--ok` `#52e6a0` / `--ok-dim` `#234a37` — success (glowing dot)
|
||||||
|
- `--fail` `#ff5d5d` / `--fail-dim` `#4a2323` — error
|
||||||
|
- `--pending` `#f0c419` — waiting/queued
|
||||||
|
- `--info` `#57c2ff` — in-progress / scan / new-folder marker (pulsing dot)
|
||||||
|
|
||||||
|
> Contrast watch: `--fg-dim` on `--bg-panel` is the thinnest pairing — for real
|
||||||
|
> body copy (not just uppercase micro-labels) prefer `--fg`. Placeholder and
|
||||||
|
> small print must still clear AA; don't push text below `--fg-faint`.
|
||||||
|
|
||||||
|
## Typography
|
||||||
|
|
||||||
|
Two families on a real contrast axis (mono vs. condensed display) — never two
|
||||||
|
similar sans.
|
||||||
|
|
||||||
|
- **Body / UI / data:** `--font-mono` = JetBrains Mono (400/500/700), 14px base,
|
||||||
|
line-height 1.5. Numbers use `font-variant-numeric: tabular-nums`
|
||||||
|
(`.mono-num`, `.num-cell`, stat values) so columns align.
|
||||||
|
- **Display / headings / brand:** `--font-display` = Big Shoulders Display
|
||||||
|
(600/800), uppercase, letter-spacing ~0.3–0.5px. Page titles 34px, brand 22px,
|
||||||
|
login brand 30px.
|
||||||
|
- **Micro-labels:** uppercase, 10–12px, letter-spacing 0.08–0.14em, `--fg-dim`.
|
||||||
|
This is the workhorse label style (nav, field labels, badges, table headers,
|
||||||
|
panel tabs). Used as a deliberate system, not a per-section eyebrow.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
- **Panels** (`.panel`): bordered surface, 3px radius, floating uppercase tab
|
||||||
|
label. The primary content container — used instead of stacked cards.
|
||||||
|
- **Buttons**: `.btn` (ghost outline, uppercase, 2px radius) · `.btn-primary`
|
||||||
|
(solid amber, dark ink, amber glow on hover) · `.btn-danger` · `.btn-ghost` ·
|
||||||
|
`.link-btn` (dashed-underline text button, `.danger` variant).
|
||||||
|
- **Status badges** (`.badge` + `.badge-ok/-fail/-pending/-info`): dot + uppercase
|
||||||
|
label, tinted 6%-alpha background, colored dot (glowing/pulsing per state).
|
||||||
|
- **Tables** (`table.tbl` in `.tbl-wrap`): dense, hairline row borders, raised
|
||||||
|
uppercase header, right-aligned `.num-cell`, subtle row-hover, `.empty-row` for
|
||||||
|
empty states. Horizontal scroll lives on `.tbl-wrap`.
|
||||||
|
- **Forms** (`.field`, `.field-row`): uppercase label above inset input; focus =
|
||||||
|
amber border + 3px amber-12% ring.
|
||||||
|
- **Modals** (`.modal-overlay` + `.modal-dialog`, `.modal-md/-lg`): fixed overlay,
|
||||||
|
backdrop-blur(2px), fade + 6px rise entrance. Folder-mapping grid
|
||||||
|
(`.map-row`: src → select). z-index scale: topbar 10 → overlay 100.
|
||||||
|
- **Live progress** (`.acct-progress`, `.pbar`/`.pbar-fill`, `.pmeta`, `.pscan`,
|
||||||
|
`.acct-error`): thin amber bar + mono meta; scan phase in `--info`; persisted
|
||||||
|
last-error in `--fail`, ellipsis-truncated.
|
||||||
|
- **Log pane** (`.log-pane`): reversed-column inset console, tagged lines
|
||||||
|
(`.tag` amber + `.payload`).
|
||||||
|
- **Chrome**: sticky `.topbar` with bracketed `.brand`, uppercase `.topnav`
|
||||||
|
(active = amber), `.session-indicator` with pulsing `.pulse-dot`.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- App shell: sticky topbar (56px) over a centered `.main`, `max-width: 1180px`,
|
||||||
|
padding `32px 24px 64px`.
|
||||||
|
- Grids: `repeat(auto-fit, minmax(340px, 1fr))` for panel grids — responsive
|
||||||
|
without breakpoints. Flexbox for 1D rows (`.btn-row`, `.stat-row`, `.upload-row`).
|
||||||
|
- Radii: tight — 2px (controls) / 3px (surfaces). Nothing pill-shaped.
|
||||||
|
- Dividers: centered uppercase `.divider-label` with rule lines both sides.
|
||||||
|
|
||||||
|
## Motion
|
||||||
|
|
||||||
|
Restrained, functional. Ease-out only; no bounce/elastic.
|
||||||
|
- Entrances: modal fade 0.12s + rise 0.14s.
|
||||||
|
- Feedback: 0.15s color/border transitions on nav, buttons, inputs.
|
||||||
|
- Ambient: `pulse` (2.4s) on session/info dots; progress-bar width 0.3s ease-out.
|
||||||
|
- Amber glow bloom on primary-button hover.
|
||||||
|
- **Owed:** a `prefers-reduced-motion: reduce` block — the pulsing dots and
|
||||||
|
progress transitions need a static fallback. Add before shipping motion work.
|
||||||
+63
@@ -0,0 +1,63 @@
|
|||||||
|
# Product
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
product
|
||||||
|
|
||||||
|
## Users
|
||||||
|
|
||||||
|
Системные администраторы, DevOps- и хостинг-инженеры, которые массово мигрируют
|
||||||
|
почтовые ящики между IMAP-серверами (source → destination). Контекст использования —
|
||||||
|
рабочая сессия миграции: пользователь заводит задачу, добавляет десятки аккаунтов
|
||||||
|
(в т.ч. импортом из CSV), запускает копирование и следит за живым прогрессом по
|
||||||
|
тысячам писем на аккаунт. Работают с плотными таблицами, читают счётчики
|
||||||
|
Copied / Skipped / Errors и статусы, ждут точности и предсказуемости, а не «красоты».
|
||||||
|
|
||||||
|
_(Выведено из README и кода; пользователь был недоступен при уточнении — подтвердить/скорректировать.)_
|
||||||
|
|
||||||
|
## Product Purpose
|
||||||
|
|
||||||
|
Однобинарный Go-сервер (встраивает React-SPA), который **копирует** содержимое
|
||||||
|
IMAP-ящиков между аккаунтами: неразрушающе (только APPEND, ничего не удаляет),
|
||||||
|
с дедупликацией по Message-ID, идемпотентно и с возобновляемостью. Успех =
|
||||||
|
инженер запускает миграцию, видит достоверный прогресс в реальном времени, а
|
||||||
|
повторный запуск честно скипает уже перенесённое и показывает ровно то, что
|
||||||
|
произошло в этом прогоне. Ценность — доверие к цифрам и отсутствие сюрпризов
|
||||||
|
на больших объёмах.
|
||||||
|
|
||||||
|
## Brand Personality
|
||||||
|
|
||||||
|
Три слова: **технично, точно, спокойно**. Голос — инструмент оператора, а не
|
||||||
|
маркетинговый продукт: краткие подписи, честные статусы, никакого хайпа.
|
||||||
|
Эмоциональная цель — уверенность и контроль. Интерфейс должен читаться как
|
||||||
|
приборная панель / консоль: плотная информация, монохром с одним функциональным
|
||||||
|
акцентом, состояние системы всегда очевидно.
|
||||||
|
|
||||||
|
## Anti-references
|
||||||
|
|
||||||
|
- Яркие SaaS-лендинги с hero-метриками, градиентным текстом и карточками-плитками.
|
||||||
|
- «Дружелюбные» дашборды в стиле генеративного AI: cream/paper-фон, скруглённые
|
||||||
|
пастельные карточки, иконка+заголовок+текст в бесконечной сетке.
|
||||||
|
- Любой декор ради декора (glassmorphism, тени-свечения без функции).
|
||||||
|
- Разреженные «воздушные» лейауты — здесь плотность данных это фича, а не проблема.
|
||||||
|
|
||||||
|
## Design Principles
|
||||||
|
|
||||||
|
1. **Числам верят.** Счётчики и статусы обязаны быть достоверны и однозначны;
|
||||||
|
визуальная неоднозначность статуса — это баг, а не косметика.
|
||||||
|
2. **Плотность — это уважение к оператору.** Компактные таблицы и моноширинные
|
||||||
|
табличные цифры важнее «воздуха».
|
||||||
|
3. **Один акцент, одна семантика.** Янтарь = внимание/действие; зелёный = успех;
|
||||||
|
красный = ошибка; жёлтый = ожидание. Цвет всегда несёт смысл.
|
||||||
|
4. **Состояние системы всегда видно.** Прогресс, скип, ошибка и последняя причина
|
||||||
|
сбоя должны переживать перезагрузку страницы и не теряться.
|
||||||
|
5. **Спокойная инженерная сдержанность.** Никакого хайпа и лишнего движения;
|
||||||
|
анимация только там, где отражает реальный процесс.
|
||||||
|
|
||||||
|
## Accessibility & Inclusion
|
||||||
|
|
||||||
|
Базовый уровень: читаемый контраст (цель — WCAG AA для основного текста, особенно
|
||||||
|
приглушённого `--fg-dim` на тёмных панелях), корректный `prefers-reduced-motion`
|
||||||
|
для live-прогресса, различимость статусов не только цветом (иконка/подпись рядом
|
||||||
|
с цветом для дальтоников). Формальной сертификации не требуется — инструмент для
|
||||||
|
узкого технического круга, но контраст и reduced-motion соблюдаем осознанно.
|
||||||
@@ -4,6 +4,16 @@ Single-binary Go server (embeds the React SPA) that copies IMAP mailboxes
|
|||||||
between a source and a destination account. Non-destructive (copy only,
|
between a source and a destination account. Non-destructive (copy only,
|
||||||
never deletes), deduplicated by Message-ID, resumable/idempotent re-runs.
|
never deletes), deduplicated by Message-ID, resumable/idempotent re-runs.
|
||||||
|
|
||||||
|
## Screenshots
|
||||||
|
|
||||||
|
Task list — create migration jobs and track their status:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Task detail — run control, per-account credentials, live event log and progress:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
## Quick start (plain HTTP on :80)
|
## Quick start (plain HTTP on :80)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
_ "net/http/pprof" // registers /debug/pprof on http.DefaultServeMux (served only on PprofAddr)
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
"github.com/golang-migrate/migrate/v4"
|
"github.com/golang-migrate/migrate/v4"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"github.com/vasyansk/imap-copier/internal/config"
|
"github.com/vasyansk/imap-copier/internal/config"
|
||||||
"github.com/vasyansk/imap-copier/internal/httpapi"
|
"github.com/vasyansk/imap-copier/internal/httpapi"
|
||||||
"github.com/vasyansk/imap-copier/internal/orchestrator"
|
"github.com/vasyansk/imap-copier/internal/orchestrator"
|
||||||
|
"github.com/vasyansk/imap-copier/internal/scheduler"
|
||||||
"github.com/vasyansk/imap-copier/internal/store"
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
"github.com/vasyansk/imap-copier/internal/wshub"
|
"github.com/vasyansk/imap-copier/internal/wshub"
|
||||||
)
|
)
|
||||||
@@ -45,6 +47,20 @@ func main() {
|
|||||||
orch := orchestrator.New(st, hub, cfg.EncKey, cfg.WorkerConcurrency)
|
orch := orchestrator.New(st, hub, cfg.EncKey, cfg.WorkerConcurrency)
|
||||||
srv := httpapi.NewServer(cfg, st, orch, hub)
|
srv := httpapi.NewServer(cfg, st, orch, hub)
|
||||||
|
|
||||||
|
go scheduler.New(st, orch).Start(context.Background())
|
||||||
|
|
||||||
|
// Optional pprof endpoint on a separate, non-published listener so we can
|
||||||
|
// pull live goroutine dumps (docker compose exec app wget -qO-
|
||||||
|
// localhost:6060/debug/pprof/goroutine?debug=2) without killing the process.
|
||||||
|
if cfg.PprofAddr != "" {
|
||||||
|
go func() {
|
||||||
|
slog.Info("pprof listening", "addr", cfg.PprofAddr)
|
||||||
|
if err := http.ListenAndServe(cfg.PprofAddr, nil); err != nil {
|
||||||
|
slog.Error("pprof serve", "err", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
slog.Info("listening", "addr", cfg.HTTPAddr)
|
slog.Info("listening", "addr", cfg.HTTPAddr)
|
||||||
if err := http.ListenAndServe(cfg.HTTPAddr, srv.Router()); err != nil {
|
if err := http.ListenAndServe(cfg.HTTPAddr, srv.Router()); err != nil {
|
||||||
slog.Error("serve", "err", err)
|
slog.Error("serve", "err", err)
|
||||||
|
|||||||
@@ -0,0 +1,69 @@
|
|||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:18-alpine
|
||||||
|
environment:
|
||||||
|
POSTGRES_USER: imap
|
||||||
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||||
|
POSTGRES_DB: imapcopier
|
||||||
|
volumes:
|
||||||
|
- pgdata:/var/lib/postgresql
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 5
|
||||||
|
networks:
|
||||||
|
- imap-copier
|
||||||
|
|
||||||
|
app:
|
||||||
|
build: .
|
||||||
|
environment:
|
||||||
|
DATABASE_URL: postgres://imap:${POSTGRES_PASSWORD}@postgres:5432/imapcopier?sslmode=disable
|
||||||
|
AUTH_USER: ${AUTH_USER}
|
||||||
|
AUTH_PASS: ${AUTH_PASS}
|
||||||
|
ENC_KEY: ${ENC_KEY}
|
||||||
|
SESSION_SECRET: ${SESSION_SECRET}
|
||||||
|
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||||
|
PPROF_ADDR: ${PPROF_ADDR:-:6060}
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- imap-copier
|
||||||
|
|
||||||
|
caddy:
|
||||||
|
image: caddy:2-alpine
|
||||||
|
labels:
|
||||||
|
- "traefik.enable=true"
|
||||||
|
- "traefik.http.routers.imap-copier.rule=Host(`${DOMAIN}`)"
|
||||||
|
- "traefik.http.routers.imap-copier.entrypoints=https"
|
||||||
|
- "traefik.http.routers.imap-copier.tls=true"
|
||||||
|
- "traefik.http.routers.imap-copier.tls.certresolver=letsEncrypt"
|
||||||
|
- "traefik.http.services.imap-copier-service.loadbalancer.server.port=80"
|
||||||
|
- "traefik.docker.network=webproxy"
|
||||||
|
expose:
|
||||||
|
- "${HTTP_PORT:-80}"
|
||||||
|
- "443"
|
||||||
|
environment:
|
||||||
|
DOMAIN: ${DOMAIN:-}
|
||||||
|
ACME_EMAIL: ${ACME_EMAIL:-}
|
||||||
|
volumes:
|
||||||
|
- ./Caddyfile:/etc/caddy/Caddyfile:ro
|
||||||
|
- caddy_data:/data
|
||||||
|
- caddy_config:/config
|
||||||
|
depends_on:
|
||||||
|
- app
|
||||||
|
networks:
|
||||||
|
- imap-copier
|
||||||
|
- webproxy
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
pgdata:
|
||||||
|
caddy_data:
|
||||||
|
caddy_config:
|
||||||
|
|
||||||
|
networks:
|
||||||
|
imap-copier:
|
||||||
|
name: imap-copier
|
||||||
|
webproxy:
|
||||||
|
external: true
|
||||||
+3
-2
@@ -8,7 +8,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- pgdata:/var/lib/postgresql
|
- pgdata:/var/lib/postgresql
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "pg_isready -U imap"]
|
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
|
||||||
interval: 5s
|
interval: 5s
|
||||||
timeout: 3s
|
timeout: 3s
|
||||||
retries: 5
|
retries: 5
|
||||||
@@ -21,7 +21,8 @@ services:
|
|||||||
AUTH_PASS: ${AUTH_PASS}
|
AUTH_PASS: ${AUTH_PASS}
|
||||||
ENC_KEY: ${ENC_KEY}
|
ENC_KEY: ${ENC_KEY}
|
||||||
SESSION_SECRET: ${SESSION_SECRET}
|
SESSION_SECRET: ${SESSION_SECRET}
|
||||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-4}
|
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||||
|
PPROF_ADDR: ${PPROF_ADDR:-:6060}
|
||||||
depends_on:
|
depends_on:
|
||||||
postgres:
|
postgres:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|||||||
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 326 KiB |
@@ -0,0 +1,873 @@
|
|||||||
|
# Per-account Folder Mapping & Exclusion Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Give every account its own source→destination folder mapping and let the operator exclude folders from the copy, editable per account via a "folders" button.
|
||||||
|
|
||||||
|
**Architecture:** Move folder config from `tasks.folder_mapping` to two new per-account JSONB columns (`folder_mapping`, `excluded_folders`), migrating existing task data onto accounts. The orchestrator builds each account's copy plan from its own config via a pure, unit-tested `planFolders` helper. New HTTP endpoints re-probe an existing account (server-side password decrypt) and persist its mapping. The React `FolderMappingModal` gains per-folder sync checkboxes; a per-row "folders" button opens it in edit mode.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.x (net/http, pgx v5, golang-migrate), Postgres 18, React 19 + Vite + TypeScript.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Non-destructive: never drop `tasks.folder_mapping`; migration copies out of it.
|
||||||
|
- Default behavior must stay backward-compatible: empty mapping + empty exclusions = sync every folder name-matched.
|
||||||
|
- New source folders (not in an account's config) default to **synced**, never silently excluded.
|
||||||
|
- pgx v5 auto-(de)serializes JSONB into Go `map[string]string` / `[]string` — scan/pass the Go value directly (see `store/tasks.go` `FolderMapping`).
|
||||||
|
- Store tests require `TEST_DATABASE_URL`; they `t.Skip` without it. Migrations are applied by `psql` against the test DB before running (see Task 1 verify).
|
||||||
|
- Backend verification commands: `go build ./...`, `go vet ./...`, `go test ./...`.
|
||||||
|
- Frontend verification: `cd web && npm run build` (tsc + vite), `npx oxlint src/`.
|
||||||
|
- Commit after each task.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Per-account folder-config storage (migration + store)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `migrations/0003_account_folder_mapping.up.sql`
|
||||||
|
- Create: `migrations/0003_account_folder_mapping.down.sql`
|
||||||
|
- Modify: `internal/store/accounts.go` (Account struct ~L8-22; `ListAccountsByTask` SELECT/Scan ~L39-58; add `SetAccountFolderMapping`)
|
||||||
|
- Test: `internal/store/accounts_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: `store.Account.FolderMapping map[string]string`, `store.Account.ExcludedFolders []string`; `func (s *Store) SetAccountFolderMapping(ctx context.Context, id int64, mapping map[string]string, excluded []string) error`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the migration up/down**
|
||||||
|
|
||||||
|
`migrations/0003_account_folder_mapping.up.sql`:
|
||||||
|
```sql
|
||||||
|
ALTER TABLE accounts ADD COLUMN folder_mapping JSONB NOT NULL DEFAULT '{}';
|
||||||
|
ALTER TABLE accounts ADD COLUMN excluded_folders JSONB NOT NULL DEFAULT '[]';
|
||||||
|
|
||||||
|
-- Carry the existing task-wide mapping onto its accounts so behavior is preserved.
|
||||||
|
UPDATE accounts a SET folder_mapping = t.folder_mapping
|
||||||
|
FROM tasks t WHERE a.task_id = t.id AND t.folder_mapping <> '{}'::jsonb;
|
||||||
|
```
|
||||||
|
|
||||||
|
`migrations/0003_account_folder_mapping.down.sql`:
|
||||||
|
```sql
|
||||||
|
ALTER TABLE accounts DROP COLUMN excluded_folders;
|
||||||
|
ALTER TABLE accounts DROP COLUMN folder_mapping;
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Write the failing store test**
|
||||||
|
|
||||||
|
Add to `internal/store/accounts_test.go`:
|
||||||
|
```go
|
||||||
|
func TestSetAccountFolderMapping(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
|
||||||
|
|
||||||
|
// Fresh account defaults: empty map, empty exclusions (not nil after scan).
|
||||||
|
accs, _ := s.ListAccountsByTask(ctx, taskID)
|
||||||
|
if len(accs) != 1 || len(accs[0].FolderMapping) != 0 || len(accs[0].ExcludedFolders) != 0 {
|
||||||
|
t.Fatalf("defaults: map=%v excl=%v", accs[0].FolderMapping, accs[0].ExcludedFolders)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.SetAccountFolderMapping(ctx, accID,
|
||||||
|
map[string]string{"Спам": "Spam"}, []string{"Trash"}); err != nil {
|
||||||
|
t.Fatalf("set: %v", err)
|
||||||
|
}
|
||||||
|
accs, _ = s.ListAccountsByTask(ctx, taskID)
|
||||||
|
a := accs[0]
|
||||||
|
if a.FolderMapping["Спам"] != "Spam" {
|
||||||
|
t.Fatalf("mapping not persisted: %v", a.FolderMapping)
|
||||||
|
}
|
||||||
|
if len(a.ExcludedFolders) != 1 || a.ExcludedFolders[0] != "Trash" {
|
||||||
|
t.Fatalf("excluded not persisted: %v", a.ExcludedFolders)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test ./internal/store/ -run TestSetAccountFolderMapping` (with a migrated `TEST_DATABASE_URL`).
|
||||||
|
Expected: FAIL to **compile** — `a.FolderMapping`/`ExcludedFolders` and `SetAccountFolderMapping` undefined.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add struct fields**
|
||||||
|
|
||||||
|
In `internal/store/accounts.go`, extend the `Account` struct (after `LastError string`):
|
||||||
|
```go
|
||||||
|
LastError string
|
||||||
|
FolderMapping map[string]string
|
||||||
|
ExcludedFolders []string
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Select + scan the new columns**
|
||||||
|
|
||||||
|
In `ListAccountsByTask`, add both columns to the SELECT and the Scan. New SELECT:
|
||||||
|
```go
|
||||||
|
rows, err := s.Pool.Query(ctx,
|
||||||
|
`SELECT id, task_id, src_login, src_pass_enc, dst_login, dst_pass_enc,
|
||||||
|
test_src_status, test_dst_status, status, copied_count, skipped_count,
|
||||||
|
error_count, last_error, folder_mapping, excluded_folders
|
||||||
|
FROM accounts WHERE task_id=$1 ORDER BY id`, taskID)
|
||||||
|
```
|
||||||
|
New Scan (add the two trailing pointers):
|
||||||
|
```go
|
||||||
|
if err := rows.Scan(&a.ID, &a.TaskID, &a.SrcLogin, &a.SrcPassEnc, &a.DstLogin, &a.DstPassEnc,
|
||||||
|
&a.TestSrcStatus, &a.TestDstStatus, &a.Status, &a.Copied, &a.Skipped, &a.Errors, &a.LastError,
|
||||||
|
&a.FolderMapping, &a.ExcludedFolders); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Add the setter**
|
||||||
|
|
||||||
|
Append to `internal/store/accounts.go`:
|
||||||
|
```go
|
||||||
|
// SetAccountFolderMapping persists an account's per-folder rename map and the
|
||||||
|
// set of source folders to skip. nil is normalized to empty so JSONB stays
|
||||||
|
// '{}' / '[]' rather than null.
|
||||||
|
func (s *Store) SetAccountFolderMapping(ctx context.Context, id int64, mapping map[string]string, excluded []string) error {
|
||||||
|
if mapping == nil {
|
||||||
|
mapping = map[string]string{}
|
||||||
|
}
|
||||||
|
if excluded == nil {
|
||||||
|
excluded = []string{}
|
||||||
|
}
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE accounts SET folder_mapping=$2, excluded_folders=$3 WHERE id=$1`,
|
||||||
|
id, mapping, excluded)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run the test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test ./internal/store/ -run TestSetAccountFolderMapping -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Verify build + full store tests**
|
||||||
|
|
||||||
|
Run: `go build ./... && go vet ./... && go test ./internal/store/`
|
||||||
|
Expected: all pass.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add migrations/0003_account_folder_mapping.up.sql migrations/0003_account_folder_mapping.down.sql \
|
||||||
|
internal/store/accounts.go internal/store/accounts_test.go
|
||||||
|
git commit -m "feat(store): per-account folder_mapping + excluded_folders columns"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Orchestrator plan honors per-account mapping + exclusions
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `internal/orchestrator/orchestrator.go` (lift `folderPlan` to package scope; add `planFolders`; replace the plan-building loop in `runAccount` ~L271-293)
|
||||||
|
- Test: Create `internal/orchestrator/planfolders_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `store.Account.FolderMapping`, `store.Account.ExcludedFolders` (Task 1).
|
||||||
|
- Produces: package-level `type folderPlan struct { src, dst string; total int64 }`; `func planFolders(folders []string, mapping map[string]string, excluded []string) []folderPlan`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing unit test**
|
||||||
|
|
||||||
|
`internal/orchestrator/planfolders_test.go`:
|
||||||
|
```go
|
||||||
|
package orchestrator
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func names(ps []folderPlan) []string {
|
||||||
|
out := make([]string, len(ps))
|
||||||
|
for i, p := range ps {
|
||||||
|
out[i] = p.src + "->" + p.dst
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPlanFolders(t *testing.T) {
|
||||||
|
folders := []string{"INBOX", "Спам", "Trash", "Sent"}
|
||||||
|
|
||||||
|
// Empty config: every folder syncs to its own name, in order.
|
||||||
|
got := planFolders(folders, nil, nil)
|
||||||
|
if len(got) != 4 || got[0].src != "INBOX" || got[0].dst != "INBOX" {
|
||||||
|
t.Fatalf("empty config: %v", names(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rename + exclusion.
|
||||||
|
got = planFolders(folders,
|
||||||
|
map[string]string{"Спам": "Spam"},
|
||||||
|
[]string{"Trash"})
|
||||||
|
if len(got) != 3 {
|
||||||
|
t.Fatalf("want 3 plans (Trash excluded), got %v", names(got))
|
||||||
|
}
|
||||||
|
for _, p := range got {
|
||||||
|
if p.src == "Trash" {
|
||||||
|
t.Fatalf("excluded folder present: %v", names(got))
|
||||||
|
}
|
||||||
|
if p.src == "Спам" && p.dst != "Spam" {
|
||||||
|
t.Fatalf("rename not applied: %v", names(got))
|
||||||
|
}
|
||||||
|
if p.src == "INBOX" && p.dst != "INBOX" {
|
||||||
|
t.Fatalf("unmapped folder should keep its name: %v", names(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test ./internal/orchestrator/ -run TestPlanFolders`
|
||||||
|
Expected: FAIL to compile — `folderPlan` is a local type inside `runAccount`, `planFolders` undefined.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Lift `folderPlan` to package scope and add `planFolders`**
|
||||||
|
|
||||||
|
In `internal/orchestrator/orchestrator.go`, add near the top (package level, e.g. above `runAccount`):
|
||||||
|
```go
|
||||||
|
// folderPlan is one source folder scheduled for copy and its destination name.
|
||||||
|
type folderPlan struct {
|
||||||
|
src, dst string
|
||||||
|
total int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// planFolders decides which of an account's source folders to copy and where.
|
||||||
|
// Excluded source folders are dropped; a folder absent from mapping keeps its
|
||||||
|
// own name. Order follows the input folder list.
|
||||||
|
func planFolders(folders []string, mapping map[string]string, excluded []string) []folderPlan {
|
||||||
|
skip := make(map[string]struct{}, len(excluded))
|
||||||
|
for _, e := range excluded {
|
||||||
|
skip[e] = struct{}{}
|
||||||
|
}
|
||||||
|
plan := make([]folderPlan, 0, len(folders))
|
||||||
|
for _, f := range folders {
|
||||||
|
if _, ok := skip[f]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
df := f
|
||||||
|
if m, ok := mapping[f]; ok && m != "" {
|
||||||
|
df = m
|
||||||
|
}
|
||||||
|
plan = append(plan, folderPlan{src: f, dst: df})
|
||||||
|
}
|
||||||
|
return plan
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Rewrite the plan-building loop in `runAccount`**
|
||||||
|
|
||||||
|
Replace the existing block (the local `type folderPlan struct {...}`, `plan := make(...)`, and the `for _, folder := range folders { ... df := folder; if m, ok := task.FolderMapping[folder]; ... }` loop, currently ~L271-293) with:
|
||||||
|
```go
|
||||||
|
// Planning pass: decide folders from the account's own config, then EXAMINE
|
||||||
|
// each to learn message counts for the overall progress bar.
|
||||||
|
plan := planFolders(folders, a.FolderMapping, a.ExcludedFolders)
|
||||||
|
var grandTotal int64
|
||||||
|
for i := range plan {
|
||||||
|
if actx.Err() != nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
n, cerr := imapx.FolderMessageCount(src, plan[i].src)
|
||||||
|
if cerr != nil && actx.Err() == nil {
|
||||||
|
slog.Warn("count folder failed", "account", a.ID, "folder", plan[i].src, "err", cerr)
|
||||||
|
}
|
||||||
|
plan[i].total = n
|
||||||
|
grandTotal += n
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Leave the subsequent `o.hub.Publish(... "plan" ...)` and the copy loop (`for _, fp := range plan`) unchanged — they already read `fp.src`, `fp.dst`, `fp.total`.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run the unit test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test ./internal/orchestrator/ -run TestPlanFolders -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Verify build + vet + full test suite**
|
||||||
|
|
||||||
|
Run: `go build ./... && go vet ./... && go test ./...`
|
||||||
|
Expected: all pass (store tests skip without `TEST_DATABASE_URL`; that's fine here).
|
||||||
|
|
||||||
|
- [ ] **Step 7: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/orchestrator/orchestrator.go internal/orchestrator/planfolders_test.go
|
||||||
|
git commit -m "feat(orchestrator): build copy plan from per-account mapping + exclusions"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: HTTP — probe existing account + persist its mapping
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `internal/httpapi/accounts.go` (AccountView + accountDTO; add `handleProbeAccountFolders`, `handleSetAccountFolderMapping`)
|
||||||
|
- Modify: `internal/httpapi/router.go` (register two routes)
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `store.SetAccountFolderMapping`, `store.Account.FolderMapping/ExcludedFolders`, existing `crypto.Decrypt`, `imapx.TestLogin`, `s.store.ListAccountsByTask`.
|
||||||
|
- Produces: routes `POST /api/tasks/{id}/accounts/{accountId}/probe`, `PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping`; `AccountView.FolderMapping`, `AccountView.ExcludedFolders`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Expose the config on AccountView**
|
||||||
|
|
||||||
|
In `internal/httpapi/accounts.go`, add to `AccountView` (after `LastError`):
|
||||||
|
```go
|
||||||
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
FolderMapping map[string]string `json:"folder_mapping"`
|
||||||
|
ExcludedFolders []string `json:"excluded_folders"`
|
||||||
|
```
|
||||||
|
And in `accountDTO`, populate them:
|
||||||
|
```go
|
||||||
|
LastError: a.LastError,
|
||||||
|
FolderMapping: a.FolderMapping,
|
||||||
|
ExcludedFolders: a.ExcludedFolders,
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add a helper to fetch one account by id**
|
||||||
|
|
||||||
|
The store exposes `ListAccountsByTask` only. Add a small lookup inside the handler (no new store method needed) — a private helper in `accounts.go`:
|
||||||
|
```go
|
||||||
|
// findAccount returns the account with accID under taskID, or ok=false.
|
||||||
|
func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) {
|
||||||
|
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
return store.Account{}, false
|
||||||
|
}
|
||||||
|
for _, a := range accs {
|
||||||
|
if a.ID == accID {
|
||||||
|
return a, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return store.Account{}, false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add the probe-by-account handler**
|
||||||
|
|
||||||
|
Append to `internal/httpapi/accounts.go`. It reuses the same probe shape as `handleProbeFolders` but decrypts stored passwords:
|
||||||
|
```go
|
||||||
|
// handleProbeAccountFolders re-probes an existing account using its stored
|
||||||
|
// (encrypted) credentials, so the operator can (re)map an account added via CSV.
|
||||||
|
func (s *Server) handleProbeAccountFolders(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
task, err := s.store.GetTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a, ok := s.findAccount(r, taskID, accID)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "account not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
srcEP, err := s.store.GetEndpoint(r.Context(), task.SrcEndpointID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dstEP, err := s.store.GetEndpoint(r.Context(), task.DstEndpointID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
srcPass, err := crypto.Decrypt(s.cfg.EncKey, a.SrcPassEnc)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "decrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dstPass, err := crypto.Decrypt(s.cfg.EncKey, a.DstPassEnc)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "decrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
probe := func(ep store.Endpoint, login string, pass []byte) map[string]any {
|
||||||
|
folders, err := imapx.TestLogin(r.Context(),
|
||||||
|
imapx.Endpoint{Host: ep.Host, Port: ep.Port, TLSMode: ep.TLSMode},
|
||||||
|
strings.TrimSpace(login), strings.TrimSpace(string(pass)))
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"ok": false, "error": err.Error(), "folders": []string{}}
|
||||||
|
}
|
||||||
|
return map[string]any{"ok": true, "folders": folders}
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{
|
||||||
|
"src": probe(srcEP, a.SrcLogin, srcPass),
|
||||||
|
"dst": probe(dstEP, a.DstLogin, dstPass),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add the per-account mapping setter handler**
|
||||||
|
|
||||||
|
Append to `internal/httpapi/accounts.go`:
|
||||||
|
```go
|
||||||
|
// handleSetAccountFolderMapping persists one account's rename map + excluded set.
|
||||||
|
func (s *Server) handleSetAccountFolderMapping(w http.ResponseWriter, r *http.Request) {
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
Mapping map[string]string `json:"mapping"`
|
||||||
|
Excluded []string `json:"excluded"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||||
|
http.Error(w, "bad json", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.store.SetAccountFolderMapping(r.Context(), accID, body.Mapping, body.Excluded); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Register the routes**
|
||||||
|
|
||||||
|
In `internal/httpapi/router.go`, after the existing `.../accounts/{accountId}/cancel` line:
|
||||||
|
```go
|
||||||
|
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders)
|
||||||
|
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Verify build + vet**
|
||||||
|
|
||||||
|
Run: `go build ./... && go vet ./...`
|
||||||
|
Expected: pass (confirms `s.cfg.EncKey`, `crypto.Decrypt`, `imapx.TestLogin`, `writeJSON` all resolve).
|
||||||
|
|
||||||
|
- [ ] **Step 7: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/httpapi/accounts.go internal/httpapi/router.go
|
||||||
|
git commit -m "feat(api): probe existing account + persist per-account folder mapping"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Frontend API client + exclusion checkboxes in the mapping modal
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/api.ts` (Account fields; `probeAccountFolders`; `setAccountFolderMapping`)
|
||||||
|
- Modify: `web/src/components/FolderMappingModal.tsx` (checkbox column, `initialExcluded`, `onConfirm(mapping, excluded)`)
|
||||||
|
- Modify: `web/src/app.css` (checkbox + excluded-row styles)
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: backend routes from Task 3.
|
||||||
|
- Produces: `probeAccountFolders(taskId, accId): Promise<ProbeResult>`; `setAccountFolderMapping(taskId, accId, mapping, excluded): Promise<unknown>`; `FolderMappingModal` prop `initialExcluded: string[]` and `onConfirm: (mapping: Record<string,string>, excluded: string[]) => void`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Extend the API client**
|
||||||
|
|
||||||
|
In `web/src/api.ts`, add to `Account` (after `last_error?`):
|
||||||
|
```ts
|
||||||
|
folder_mapping?: Record<string, string>
|
||||||
|
excluded_folders?: string[]
|
||||||
|
```
|
||||||
|
Add near the existing probe/mapping helpers:
|
||||||
|
```ts
|
||||||
|
export const probeAccountFolders = (taskId: number, accId: number) =>
|
||||||
|
api<ProbeResult>(`/api/tasks/${taskId}/accounts/${accId}/probe`, { method: 'POST' })
|
||||||
|
|
||||||
|
export const setAccountFolderMapping = (
|
||||||
|
taskId: number,
|
||||||
|
accId: number,
|
||||||
|
mapping: Record<string, string>,
|
||||||
|
excluded: string[],
|
||||||
|
) =>
|
||||||
|
api(`/api/tasks/${taskId}/accounts/${accId}/folder-mapping`, {
|
||||||
|
...jsonBody({ mapping, excluded }),
|
||||||
|
method: 'PUT',
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add exclusion to FolderMappingModal — props + state**
|
||||||
|
|
||||||
|
In `web/src/components/FolderMappingModal.tsx`, change the `Props` type and signature:
|
||||||
|
```ts
|
||||||
|
type Props = {
|
||||||
|
open: boolean
|
||||||
|
srcFolders: string[]
|
||||||
|
dstFolders: string[]
|
||||||
|
initialMapping: Record<string, string>
|
||||||
|
initialExcluded: string[]
|
||||||
|
onConfirm: (mapping: Record<string, string>, excluded: string[]) => void
|
||||||
|
onCancel: () => void
|
||||||
|
}
|
||||||
|
|
||||||
|
export function FolderMappingModal({
|
||||||
|
open, srcFolders, dstFolders, initialMapping, initialExcluded, onConfirm, onCancel,
|
||||||
|
}: Props) {
|
||||||
|
const [choice, setChoice] = useState<Record<string, string>>({})
|
||||||
|
const [synced, setSynced] = useState<Record<string, boolean>>(() => {
|
||||||
|
const excl = new Set(initialExcluded)
|
||||||
|
return Object.fromEntries(srcFolders.map((f) => [f, !excl.has(f)]))
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Update `confirm()` to emit excluded**
|
||||||
|
|
||||||
|
Replace the `confirm()` body:
|
||||||
|
```ts
|
||||||
|
function confirm() {
|
||||||
|
const mapping: Record<string, string> = { ...initialMapping }
|
||||||
|
const excluded: string[] = []
|
||||||
|
for (const src of srcFolders) {
|
||||||
|
if (synced[src] === false) {
|
||||||
|
excluded.push(src)
|
||||||
|
delete mapping[src]
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
const dst = valueFor(src)
|
||||||
|
if (dst === src) delete mapping[src]
|
||||||
|
else mapping[src] = dst
|
||||||
|
}
|
||||||
|
onConfirm(mapping, excluded)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Add the checkbox + "sync all" to the JSX**
|
||||||
|
|
||||||
|
Add a select-all control above `.map-grid` and a checkbox per row. Replace the `.map-grid` block:
|
||||||
|
```tsx
|
||||||
|
<label className="map-all">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={srcFolders.every((f) => synced[f] !== false)}
|
||||||
|
onChange={(e) => {
|
||||||
|
const on = e.target.checked
|
||||||
|
setSynced(Object.fromEntries(srcFolders.map((f) => [f, on])))
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
sync all folders
|
||||||
|
</label>
|
||||||
|
<div className="map-grid">
|
||||||
|
{srcFolders.map((src) => {
|
||||||
|
const on = synced[src] !== false
|
||||||
|
const val = valueFor(src)
|
||||||
|
const creates = !dstFolders.includes(val)
|
||||||
|
return (
|
||||||
|
<div className={`map-row${on ? '' : ' map-row-off'}`} key={src}>
|
||||||
|
<label className="map-check">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={on}
|
||||||
|
aria-label={`Sync folder ${src}`}
|
||||||
|
onChange={(e) => setSynced((s) => ({ ...s, [src]: e.target.checked }))}
|
||||||
|
/>
|
||||||
|
</label>
|
||||||
|
<span className="map-src" title={src}>
|
||||||
|
{src}
|
||||||
|
</span>
|
||||||
|
<span className="map-arrow">→</span>
|
||||||
|
<select
|
||||||
|
className="map-select"
|
||||||
|
aria-label={`Destination folder for ${src}`}
|
||||||
|
value={val}
|
||||||
|
disabled={!on}
|
||||||
|
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
|
||||||
|
>
|
||||||
|
{options(src).map((f) => (
|
||||||
|
<option key={f} value={f}>
|
||||||
|
{f}
|
||||||
|
{f === src && !dstFolders.includes(src) ? ' (create)' : ''}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
{on ? (
|
||||||
|
creates && <span className="map-new">new</span>
|
||||||
|
) : (
|
||||||
|
<span className="map-excluded">excluded</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
})}
|
||||||
|
</div>
|
||||||
|
```
|
||||||
|
Note: `.map-row` grid now has a leading checkbox column — update its `grid-template-columns` in Step 5.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add CSS for the checkbox column + excluded row**
|
||||||
|
|
||||||
|
In `web/src/app.css`, replace the `.map-row` rule and add new rules:
|
||||||
|
```css
|
||||||
|
.map-row {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: auto 1fr auto 1fr auto;
|
||||||
|
align-items: center;
|
||||||
|
gap: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-row-off .map-src,
|
||||||
|
.map-row-off .map-arrow {
|
||||||
|
color: var(--fg-faint);
|
||||||
|
text-decoration: line-through;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-check {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-check input,
|
||||||
|
.map-all input {
|
||||||
|
accent-color: var(--accent);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-all {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
margin-bottom: 12px;
|
||||||
|
font-size: 11px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-excluded {
|
||||||
|
font-size: 10px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
color: var(--fg-faint);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Verify build + lint**
|
||||||
|
|
||||||
|
Run: `cd web && npm run build && npx oxlint src/`
|
||||||
|
Expected: build succeeds; only the two pre-existing oxlint warnings (`ConfirmProvider` fast-refresh, `TaskDetail` exhaustive-deps). This step will still show a type error at the `FolderMappingModal` call site in `TaskDetail.tsx` (missing `initialExcluded`, new `onConfirm` arity) — that is fixed in Task 5. If building Task 4 alone, temporarily expect that call-site error; it resolves once Task 5 lands. (Preferred: run Task 5 before the final build.)
|
||||||
|
|
||||||
|
- [ ] **Step 7: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add web/src/api.ts web/src/components/FolderMappingModal.tsx web/src/app.css
|
||||||
|
git commit -m "feat(web): folder-sync checkboxes + per-account mapping API client"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: Frontend TaskDetail — "folders" button, edit mode, per-account save
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/pages/TaskDetail.tsx`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `probeAccountFolders`, `setAccountFolderMapping` (Task 4); `FolderMappingModal` new props (Task 4); `Account.folder_mapping/excluded_folders` (Task 4).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Import the new API functions**
|
||||||
|
|
||||||
|
In `web/src/pages/TaskDetail.tsx`, extend the import from `../api` to add `probeAccountFolders` and `setAccountFolderMapping` (keep the rest). Remove `setFolderMapping` from the import (single-add stops using it — see Step 4):
|
||||||
|
```ts
|
||||||
|
import { cancelAccount, createAccount, deleteAccount, getTask, importCSV, probeAccountFolders, probeFolders, runTask, setAccountFolderMapping, setFolderMapping, testAccounts, type TaskDetail as TaskDetailData } from '../api'
|
||||||
|
```
|
||||||
|
(Keep `setFolderMapping` in the import only if still referenced; after Step 4 it is not — drop it to avoid an unused-import lint error.)
|
||||||
|
|
||||||
|
- [ ] **Step 2: Add edit-modal state**
|
||||||
|
|
||||||
|
Next to `const [mapState, setMapState] = useState<...>(null)`, add:
|
||||||
|
```ts
|
||||||
|
const [editMap, setEditMap] = useState<{
|
||||||
|
accId: number
|
||||||
|
src: string[]
|
||||||
|
dst: string[]
|
||||||
|
mapping: Record<string, string>
|
||||||
|
excluded: string[]
|
||||||
|
} | null>(null)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add the open-editor + save handlers**
|
||||||
|
|
||||||
|
Add two functions inside the component (near `onDeleteAccount`):
|
||||||
|
```ts
|
||||||
|
async function onEditFolders(a: { id: number; src_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) {
|
||||||
|
setBusy('probe')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
const res = await probeAccountFolders(id, a.id)
|
||||||
|
if (!res.src.ok || !res.dst.ok) {
|
||||||
|
const parts: string[] = []
|
||||||
|
if (!res.src.ok) parts.push(`source: ${res.src.error ?? 'login failed'}`)
|
||||||
|
if (!res.dst.ok) parts.push(`destination: ${res.dst.error ?? 'login failed'}`)
|
||||||
|
setError(`Folder probe failed — ${parts.join('; ')}`)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setEditMap({
|
||||||
|
accId: a.id,
|
||||||
|
src: res.src.folders ?? [],
|
||||||
|
dst: res.dst.folders ?? [],
|
||||||
|
mapping: a.folder_mapping ?? {},
|
||||||
|
excluded: a.excluded_folders ?? [],
|
||||||
|
})
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to probe folders')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveEditMapping(mapping: Record<string, string>, excluded: string[]) {
|
||||||
|
if (!editMap) return
|
||||||
|
setBusy('add')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await setAccountFolderMapping(id, editMap.accId, mapping, excluded)
|
||||||
|
setEditMap(null)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to save mapping')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Switch single-add to per-account save**
|
||||||
|
|
||||||
|
In `confirmMapping`, replace the task-level call. Change `onConfirm` signature to accept `excluded` and persist to the new account:
|
||||||
|
```ts
|
||||||
|
async function confirmMapping(mapping: Record<string, string>, excluded: string[]) {
|
||||||
|
if (!mapState) return
|
||||||
|
setBusy('add')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
const { id: accId } = await createAccount(id, mapState.creds)
|
||||||
|
await setAccountFolderMapping(id, accId, mapping, excluded)
|
||||||
|
setForm(emptyAccount)
|
||||||
|
setMapState(null)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to add account')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
(`createAccount` already returns `{ id }`.) After this, `setFolderMapping` is unused — ensure it is removed from the Step 1 import.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add the "folders" button to the actions cell**
|
||||||
|
|
||||||
|
In the accounts table actions cell, put a `folders` button left of remove/cancel. Replace the last `<td className="num-cell">…</td>` action cell with:
|
||||||
|
```tsx
|
||||||
|
<td className="num-cell">
|
||||||
|
<div className="row-actions">
|
||||||
|
{a.status !== 'running' && (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="link-btn"
|
||||||
|
onClick={() => onEditFolders(a)}
|
||||||
|
disabled={busy !== null}
|
||||||
|
>
|
||||||
|
folders
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
{a.status === 'running' ? (
|
||||||
|
<button type="button" className="link-btn danger" onClick={() => onCancelAccount(a.id)}>
|
||||||
|
cancel
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="link-btn danger"
|
||||||
|
onClick={() => onDeleteAccount(a.id, a.src_login)}
|
||||||
|
disabled={busy !== null || data?.task.status === 'running'}
|
||||||
|
>
|
||||||
|
remove
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Pass `initialExcluded` to the add-modal and render the edit-modal**
|
||||||
|
|
||||||
|
Update the existing add-modal usage to pass `initialExcluded={[]}` (new accounts sync all) — `onConfirm={confirmMapping}` now matches the `(mapping, excluded)` arity:
|
||||||
|
```tsx
|
||||||
|
{mapState && (
|
||||||
|
<FolderMappingModal
|
||||||
|
key={`${mapState.creds.src_login}|${mapState.creds.dst_login}`}
|
||||||
|
open
|
||||||
|
srcFolders={mapState.src}
|
||||||
|
dstFolders={mapState.dst}
|
||||||
|
initialMapping={task.folder_mapping ?? {}}
|
||||||
|
initialExcluded={[]}
|
||||||
|
onCancel={() => setMapState(null)}
|
||||||
|
onConfirm={confirmMapping}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
{editMap && (
|
||||||
|
<FolderMappingModal
|
||||||
|
key={`edit-${editMap.accId}`}
|
||||||
|
open
|
||||||
|
srcFolders={editMap.src}
|
||||||
|
dstFolders={editMap.dst}
|
||||||
|
initialMapping={editMap.mapping}
|
||||||
|
initialExcluded={editMap.excluded}
|
||||||
|
onCancel={() => setEditMap(null)}
|
||||||
|
onConfirm={saveEditMapping}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Add `.row-actions` CSS**
|
||||||
|
|
||||||
|
In `web/src/app.css`, add:
|
||||||
|
```css
|
||||||
|
.row-actions {
|
||||||
|
display: inline-flex;
|
||||||
|
gap: 12px;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: flex-end;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Verify build + lint**
|
||||||
|
|
||||||
|
Run: `cd web && npm run build && npx oxlint src/`
|
||||||
|
Expected: build succeeds; only the two pre-existing oxlint warnings. No type errors.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add web/src/pages/TaskDetail.tsx web/src/app.css
|
||||||
|
git commit -m "feat(web): per-account folders button + edit-mapping modal"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 6: End-to-end verification on prod
|
||||||
|
|
||||||
|
Per project rules, functional verification runs on prod after deploy (Web via playwright-cli). This task has no code; it gates "done".
|
||||||
|
|
||||||
|
- [ ] **Step 1: Build + apply migration + deploy**
|
||||||
|
|
||||||
|
Run: `make build && make up` (migration 0003 auto-applies at startup via golang-migrate). Wait for `curl -fsS http://localhost/healthz`.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Migration sanity — existing accounts preserved**
|
||||||
|
|
||||||
|
Open an existing task that had a task-level mapping. Confirm accounts still run and copy the same folders (migrated mapping carried over).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Per-account exclusion + rename E2E**
|
||||||
|
|
||||||
|
1. CSV-import (or add) an account. In the accounts table, click **folders**.
|
||||||
|
2. Modal opens with the account's freshly probed source folders, all checked.
|
||||||
|
3. Uncheck one folder (e.g. `Trash`) → its select greys out, shows "excluded".
|
||||||
|
4. Rename another (e.g. `Спам → Spam`). Save.
|
||||||
|
5. Reopen **folders** on the same account → the excluded folder is unchecked and the rename is preserved (state re-read from DB).
|
||||||
|
6. Run the task. Verify: the excluded folder is NOT copied (absent from progress/log), the renamed folder lands under its new name on the destination.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Record result**
|
||||||
|
|
||||||
|
Save an E2E note under `./swarm-report/per-account-folder-mapping-<YYYY-MM-DD>.md` with pass/fail per step.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review
|
||||||
|
|
||||||
|
- **Spec coverage:** per-account model (Task 1) ✓; migration of task mapping (Task 1 up.sql) ✓; exclusion list w/ safe default (Task 1 defaults + Task 2 planFolders) ✓; orchestrator honors config (Task 2) ✓; probe-by-account decrypt (Task 3) ✓; PUT per-account mapping (Task 3) ✓; AccountView exposes config (Task 3) ✓; api.ts + modal checkboxes (Task 4) ✓; folders button + edit mode + single-add switch (Task 5) ✓; tests: store round-trip + defaults (Task 1), planFolders unit (Task 2), E2E (Task 6) ✓.
|
||||||
|
- **Placeholder scan:** none — all steps carry real code/commands.
|
||||||
|
- **Type consistency:** `folderPlan{src,dst,total}` package-level (Task 2) matches copy-loop usage; `SetAccountFolderMapping(id, mapping, excluded)` identical across Tasks 1/3; `onConfirm(mapping, excluded)` arity consistent across modal (Task 4) and both call sites (Task 5); `probeAccountFolders`/`setAccountFolderMapping` signatures match between api.ts (Task 4) and TaskDetail (Task 5).
|
||||||
|
- **Note:** Task 4 built in isolation leaves a transient type error at the modal call site until Task 5; documented in Task 4 Step 6. Prefer running Tasks 4→5 back-to-back before a clean build.
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,546 @@
|
|||||||
|
# Selective Account Run Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** Let the user run a migration for only the checkbox-selected accounts; with no selection, run all (unchanged).
|
||||||
|
|
||||||
|
**Architecture:** A pure `selectAccounts` filter narrows the orchestrator's account slice before the test gate; the run HTTP handler parses an optional `account_ids` body and passes it through. The accounts table gains row checkboxes and a "select all" header, and the single "Run migration" button becomes "Run selected (N)" when a selection exists. Copy is already idempotent (dedup by message key), so a re-run of selected accounts skips copied messages and delivers the missing tail.
|
||||||
|
|
||||||
|
**Tech Stack:** Go 1.x (net/http, encoding/json, standard testing), React + TypeScript + Vite, existing REST/WS client.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Backend module path: `github.com/vasyansk/imap-copier`.
|
||||||
|
- `Run` with an empty/nil account-ID set MUST behave exactly as today (all accounts) — the scheduler depends on it.
|
||||||
|
- Preserve existing `handleRun` error mapping: `ErrNotTested` → 409 "accounts must pass connection tests first"; `ErrAlreadyRunning` → 409 "task is already running".
|
||||||
|
- Frontend has no unit-test harness; frontend tasks verify via `npm run build` + `tsc --noEmit` + manual app check.
|
||||||
|
- Follow existing code style: no new dependencies.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Orchestrator account filter + `Run` signature
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `internal/orchestrator/orchestrator.go`
|
||||||
|
- Test: `internal/orchestrator/orchestrator_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces:
|
||||||
|
- `func selectAccounts(accs []store.Account, ids []int64) []store.Account` — returns `accs` unchanged when `ids` is empty/nil; otherwise returns only the accounts whose `ID` is in `ids`, preserving input order.
|
||||||
|
- `var ErrNoAccountsSelected = errors.New("no matching accounts selected")`
|
||||||
|
- `func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)` — new trailing `accountIDs` param.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Add to `internal/orchestrator/orchestrator_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestSelectAccounts(t *testing.T) {
|
||||||
|
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
|
||||||
|
|
||||||
|
if got := selectAccounts(accs, nil); len(got) != 3 {
|
||||||
|
t.Fatalf("nil ids must return all, got %d", len(got))
|
||||||
|
}
|
||||||
|
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
|
||||||
|
t.Fatalf("empty ids must return all, got %d", len(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
got := selectAccounts(accs, []int64{3, 1})
|
||||||
|
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
|
||||||
|
t.Fatalf("must keep matching ids in input order, got %+v", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
|
||||||
|
t.Fatalf("unknown ids must yield empty, got %d", len(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
|
||||||
|
Expected: FAIL — `undefined: selectAccounts`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Add the filter helper and sentinel error**
|
||||||
|
|
||||||
|
In `internal/orchestrator/orchestrator.go`, add the sentinel next to the existing `var ErrNotTested`/`ErrAlreadyRunning`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
|
||||||
|
```
|
||||||
|
|
||||||
|
Add the helper (place it near `gateOK`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
// selectAccounts narrows accs to those whose ID is in ids, preserving input
|
||||||
|
// order. An empty or nil ids means "all accounts" — the scheduler and the
|
||||||
|
// unfiltered manual run rely on this.
|
||||||
|
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return accs
|
||||||
|
}
|
||||||
|
want := make(map[int64]struct{}, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
want[id] = struct{}{}
|
||||||
|
}
|
||||||
|
out := make([]store.Account, 0, len(ids))
|
||||||
|
for _, a := range accs {
|
||||||
|
if _, ok := want[a.ID]; ok {
|
||||||
|
out = append(out, a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Thread `accountIDs` through `Run`**
|
||||||
|
|
||||||
|
In `internal/orchestrator/orchestrator.go`, change the `Run` signature and apply the filter after loading accounts. Replace:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
|
||||||
|
task, err := o.store.GetTask(ctx, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if !gateOK(accs) {
|
||||||
|
return 0, ErrNotTested
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
with:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
|
||||||
|
task, err := o.store.GetTask(ctx, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
accs = selectAccounts(accs, accountIDs)
|
||||||
|
// A non-empty request that matched nothing is a client error, distinct
|
||||||
|
// from "not tested".
|
||||||
|
if len(accountIDs) > 0 && len(accs) == 0 {
|
||||||
|
return 0, ErrNoAccountsSelected
|
||||||
|
}
|
||||||
|
if !gateOK(accs) {
|
||||||
|
return 0, ErrNotTested
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The rest of `Run` (endpoints, `TryMarkTaskRunning`, `CreateRun`, `go o.runAll(...)`) is unchanged — it already operates over the `accs` slice.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Update the two `Run` call sites so the package compiles**
|
||||||
|
|
||||||
|
In `internal/scheduler/scheduler.go`, change:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
|
||||||
|
```
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
```go
|
||||||
|
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
|
||||||
|
```
|
||||||
|
|
||||||
|
In `internal/httpapi/run.go`, inside `handleRun`, change:
|
||||||
|
|
||||||
|
```go
|
||||||
|
runID, err := s.orch.Run(r.Context(), taskID, "manual")
|
||||||
|
```
|
||||||
|
|
||||||
|
to (IDs wired in Task 2; for now pass `nil` to keep it compiling):
|
||||||
|
|
||||||
|
```go
|
||||||
|
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Verify the whole backend builds and tests pass**
|
||||||
|
|
||||||
|
Run: `go build ./... && go test ./internal/orchestrator/ ./internal/scheduler/ -v`
|
||||||
|
Expected: build succeeds; all tests PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 8: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/orchestrator/orchestrator.go internal/orchestrator/orchestrator_test.go internal/scheduler/scheduler.go internal/httpapi/run.go
|
||||||
|
git commit -m "Add selectAccounts filter and accountIDs param to orchestrator.Run"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: `handleRun` parses optional `account_ids`
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `internal/httpapi/run.go`
|
||||||
|
- Test: `internal/httpapi/run_test.go`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `orchestrator.Run(ctx, taskID, "manual", accountIDs)`, `orchestrator.ErrNoAccountsSelected` (Task 1).
|
||||||
|
- Produces:
|
||||||
|
- `func parseRunAccountIDs(r *http.Request) ([]int64, error)` — returns `nil` for an empty body; otherwise decodes `{"account_ids":[...]}` and returns the slice (possibly empty). Returns an error on malformed JSON.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Add to `internal/httpapi/run_test.go` (the `strings` import is already present):
|
||||||
|
|
||||||
|
```go
|
||||||
|
func TestParseRunAccountIDs(t *testing.T) {
|
||||||
|
// empty body => nil (run all)
|
||||||
|
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
|
||||||
|
ids, err := parseRunAccountIDs(req)
|
||||||
|
if err != nil || ids != nil {
|
||||||
|
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// explicit selection
|
||||||
|
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
|
||||||
|
ids, err = parseRunAccountIDs(req)
|
||||||
|
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
|
||||||
|
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// malformed JSON => error
|
||||||
|
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
|
||||||
|
if _, err := parseRunAccountIDs(req); err == nil {
|
||||||
|
t.Fatal("malformed body must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails**
|
||||||
|
|
||||||
|
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
|
||||||
|
Expected: FAIL — `undefined: parseRunAccountIDs`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Implement `parseRunAccountIDs` and wire it into `handleRun`**
|
||||||
|
|
||||||
|
In `internal/httpapi/run.go`, add the imports `encoding/json` and `io` to the existing import block, then add the helper:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
|
||||||
|
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
|
||||||
|
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
|
||||||
|
raw, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(bytes.TrimSpace(raw)) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
AccountIDs []int64 `json:"account_ids"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &body); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return body.AccountIDs, nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Add `"bytes"` to the import block as well (used by `bytes.TrimSpace`).
|
||||||
|
|
||||||
|
Then replace the body of `handleRun`'s run call. Change:
|
||||||
|
|
||||||
|
```go
|
||||||
|
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
|
||||||
|
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||||
|
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
|
||||||
|
http.Error(w, "task is already running", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
to:
|
||||||
|
|
||||||
|
```go
|
||||||
|
accountIDs, err := parseRunAccountIDs(r)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad request body", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
|
||||||
|
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
|
||||||
|
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||||
|
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
|
||||||
|
http.Error(w, "task is already running", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes**
|
||||||
|
|
||||||
|
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Verify the whole backend builds and the httpapi suite passes**
|
||||||
|
|
||||||
|
Run: `go build ./... && go test ./internal/httpapi/ -v`
|
||||||
|
Expected: build succeeds; all tests PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add internal/httpapi/run.go internal/httpapi/run_test.go
|
||||||
|
git commit -m "Parse optional account_ids in run handler"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Frontend `runTask` accepts account IDs
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/api.ts:141`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: `runTask(id: number, accountIds?: number[]) => Promise<...>` — sends `{account_ids}` JSON only when a non-empty list is given; otherwise a bare POST (all accounts).
|
||||||
|
|
||||||
|
- [ ] **Step 1: Update `runTask`**
|
||||||
|
|
||||||
|
In `web/src/api.ts`, replace:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
|
||||||
|
```
|
||||||
|
|
||||||
|
with:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export const runTask = (id: number, accountIds?: number[]) =>
|
||||||
|
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Verify typecheck**
|
||||||
|
|
||||||
|
Run: `cd web && npx tsc --noEmit`
|
||||||
|
Expected: no errors from `api.ts` (pre-existing warnings elsewhere, if any, are unrelated).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add web/src/api.ts
|
||||||
|
git commit -m "runTask accepts optional accountIds"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Accounts table checkboxes + selective Run button
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `web/src/pages/TaskDetail.tsx`
|
||||||
|
- Modify: `web/src/app.css`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `runTask(id, accountIds?)` (Task 3).
|
||||||
|
- Produces: no exported interface; internal UI state only.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add selection state and helpers**
|
||||||
|
|
||||||
|
In `web/src/pages/TaskDetail.tsx`, add a state hook next to the other `useState` declarations (near line 90):
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
const [selected, setSelected] = useState<Set<number>>(new Set())
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Compute selectable rows and the effective run set**
|
||||||
|
|
||||||
|
After the existing `const { task, accounts } = data` / `allTested` block (near line 369), add:
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
const isRunning = task.status === 'running'
|
||||||
|
// A row is selectable only when both connection tests pass and no run is live.
|
||||||
|
const selectableIds = accounts
|
||||||
|
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||||
|
.map((a) => a.id)
|
||||||
|
const selectableSet = new Set(selectableIds)
|
||||||
|
// Effective set: the checked accounts, or all accounts when nothing is checked.
|
||||||
|
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
|
||||||
|
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
|
||||||
|
const runReady =
|
||||||
|
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||||
|
const allSelectableChecked =
|
||||||
|
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
|
||||||
|
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
|
||||||
|
|
||||||
|
function toggleOne(accId: number, checked: boolean) {
|
||||||
|
setSelected((prev) => {
|
||||||
|
const next = new Set(prev)
|
||||||
|
if (checked) next.add(accId)
|
||||||
|
else next.delete(accId)
|
||||||
|
return next
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
function toggleAll(checked: boolean) {
|
||||||
|
setSelected(checked ? new Set(selectableIds) : new Set())
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Make `onRun` send the selection**
|
||||||
|
|
||||||
|
Replace the existing `onRun` function (near line 331):
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
async function onRun() {
|
||||||
|
setBusy('run')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await runTask(id)
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
with:
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
async function onRun() {
|
||||||
|
setBusy('run')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
|
||||||
|
await runTask(id, ids.length ? ids : undefined)
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Update the Run button label and gate**
|
||||||
|
|
||||||
|
Replace the run button (near line 421):
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
|
||||||
|
{busy === 'run' ? 'Starting…' : 'Run migration'}
|
||||||
|
</button>
|
||||||
|
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
|
||||||
|
```
|
||||||
|
|
||||||
|
with:
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !runReady || isRunning}>
|
||||||
|
{busy === 'run'
|
||||||
|
? 'Starting…'
|
||||||
|
: effectiveSelected.length > 0
|
||||||
|
? `Run selected (${effectiveSelected.length})`
|
||||||
|
: 'Run migration'}
|
||||||
|
</button>
|
||||||
|
{!runReady && accounts.length > 0 && (
|
||||||
|
<span className="hint">
|
||||||
|
{effectiveSelected.length > 0
|
||||||
|
? 'selected accounts must pass both connection tests'
|
||||||
|
: 'run unlocks once every account tests OK on both sides'}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add the checkbox header column**
|
||||||
|
|
||||||
|
In the accounts table `<thead><tr>` (near line 542), add a leading `<th>` before `<th>Account</th>`:
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
<th className="chk-col">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
aria-label="Select all accounts"
|
||||||
|
checked={allSelectableChecked}
|
||||||
|
ref={(el) => {
|
||||||
|
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
|
||||||
|
}}
|
||||||
|
disabled={isRunning || selectableIds.length === 0}
|
||||||
|
onChange={(e) => toggleAll(e.target.checked)}
|
||||||
|
/>
|
||||||
|
</th>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 6: Add the per-row checkbox cell and fix the empty-row colspan**
|
||||||
|
|
||||||
|
In the empty-row branch (near line 557), change `colSpan={9}` to `colSpan={10}`.
|
||||||
|
|
||||||
|
In the account `<tr>` map (near line 561), add a leading `<td>` as the first child, before the `<td>` with `acct-ident`:
|
||||||
|
|
||||||
|
```tsx
|
||||||
|
<td className="chk-col">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
aria-label={`Select ${a.src_login}`}
|
||||||
|
checked={selected.has(a.id)}
|
||||||
|
disabled={isRunning || !selectableSet.has(a.id)}
|
||||||
|
onChange={(e) => toggleOne(a.id, e.target.checked)}
|
||||||
|
/>
|
||||||
|
</td>
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 7: Add checkbox column CSS**
|
||||||
|
|
||||||
|
In `web/src/app.css`, append:
|
||||||
|
|
||||||
|
```css
|
||||||
|
.tbl .chk-col {
|
||||||
|
width: 32px;
|
||||||
|
text-align: center;
|
||||||
|
padding-right: 0;
|
||||||
|
}
|
||||||
|
.tbl .chk-col input[type='checkbox'] {
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.tbl .chk-col input[type='checkbox']:disabled {
|
||||||
|
cursor: not-allowed;
|
||||||
|
opacity: 0.4;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Build and typecheck**
|
||||||
|
|
||||||
|
Run: `cd web && npx tsc --noEmit && npm run build`
|
||||||
|
Expected: typecheck clean (any pre-existing oxlint/fast-refresh warnings are unrelated); Vite build succeeds.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Manual verification in the running app**
|
||||||
|
|
||||||
|
Start the app, open a task with ≥2 accounts that have passed both tests:
|
||||||
|
1. With nothing checked, the button reads "Run migration" and runs all accounts (confirm `account_started` WS events for every account).
|
||||||
|
2. Check a subset; the button reads "Run selected (N)". Run it; confirm only the checked accounts start, and the unchecked accounts' Copied/Skipped/Status are untouched.
|
||||||
|
3. Confirm checkboxes for untested accounts are disabled, and the header checkbox toggles only the selectable rows.
|
||||||
|
|
||||||
|
- [ ] **Step 10: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add web/src/pages/TaskDetail.tsx web/src/app.css
|
||||||
|
git commit -m "Add account selection checkboxes and selective run button"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes for the executor
|
||||||
|
|
||||||
|
- Tasks 1 and 2 are backend and must land in order (Task 2 depends on the new `Run` signature and `ErrNoAccountsSelected`). Task 3 and 4 are frontend; Task 4 depends on Task 3.
|
||||||
|
- The orchestrator's `Run` is integration-heavy (spawns goroutines, needs a real store/hub), so it is not unit-tested directly; the pure `selectAccounts` filter and the `parseRunAccountIDs` helper carry the backend test coverage, and Task 4 Step 9 covers the end-to-end behavior manually.
|
||||||
@@ -0,0 +1,160 @@
|
|||||||
|
# Per-account folder mapping & folder exclusion — design
|
||||||
|
|
||||||
|
**Date:** 2026-07-03
|
||||||
|
**Status:** awaiting user review
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Two gaps in the current folder-mapping feature:
|
||||||
|
|
||||||
|
1. **No per-account mapping for bulk (CSV) accounts.** Folder mapping is collected
|
||||||
|
only in the single "Add account" flow. Accounts imported from CSV inherit the
|
||||||
|
task-wide mapping and cannot be mapped individually. The user wants a **folders**
|
||||||
|
button per account row (left of remove) that re-probes that account's live
|
||||||
|
folders and opens the mapping modal for it.
|
||||||
|
2. **No exclusion.** Mapping can only rename `src → dst`; it cannot skip folders
|
||||||
|
the user doesn't want to copy (Spam, Trash, per-client junk). The user wants
|
||||||
|
**checkboxes** to choose which source folders are synced.
|
||||||
|
|
||||||
|
### Current architecture (as-is)
|
||||||
|
|
||||||
|
- `tasks.folder_mapping` (JSONB `map[src]dst`) — **one mapping for the whole task**,
|
||||||
|
applied to every account by the orchestrator: `df := task.FolderMapping[folder]`
|
||||||
|
(`internal/orchestrator/orchestrator.go:283-286`).
|
||||||
|
- Single-add flow: `probeFolders` (form creds) → `FolderMappingModal` → `createAccount`
|
||||||
|
then `setFolderMapping` (task-level) (`TaskDetail.tsx:167-204`).
|
||||||
|
- `handleProbeFolders` takes **plaintext** creds from the request body
|
||||||
|
(`internal/httpapi/accounts.go:39-83`). CSV accounts store only encrypted
|
||||||
|
passwords, so re-probing an existing account needs a server-side decrypt path.
|
||||||
|
- `store.Account` has **no** mapping field.
|
||||||
|
|
||||||
|
## Decision: fully per-account model
|
||||||
|
|
||||||
|
Folder configuration moves from the task to the account. `tasks.folder_mapping`
|
||||||
|
is retired as the source of truth (column kept, no longer read) and its data is
|
||||||
|
migrated onto each account. This matches the intent ("each account its own
|
||||||
|
folders") and removes the current oddity where adding one account rewrites the
|
||||||
|
whole task's mapping.
|
||||||
|
|
||||||
|
Exclusion is stored as a **separate `excluded_folders` list** (not a sentinel in
|
||||||
|
the mapping). Rationale: a source folder that is *not configured* (new folder
|
||||||
|
appearing after mapping, or a CSV account never opened) must **default to synced**
|
||||||
|
— a copy tool must never silently drop mail. An exclusion list gives that safe
|
||||||
|
default; an inclusion list would silently skip new folders.
|
||||||
|
|
||||||
|
### Semantics
|
||||||
|
|
||||||
|
For an account, given its live source folders:
|
||||||
|
|
||||||
|
- `excluded_folders: string[]` — source folder names to **skip** entirely.
|
||||||
|
- `folder_mapping: map[src]dst` — rename for synced folders; a src absent from the
|
||||||
|
map syncs to its own name (existing behavior).
|
||||||
|
- Default (empty mapping, empty exclusions) = sync every folder name-matched =
|
||||||
|
current behavior. Backward compatible.
|
||||||
|
|
||||||
|
## Data model
|
||||||
|
|
||||||
|
**Migration `0003_account_folder_mapping`:**
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- up
|
||||||
|
ALTER TABLE accounts ADD COLUMN folder_mapping JSONB NOT NULL DEFAULT '{}';
|
||||||
|
ALTER TABLE accounts ADD COLUMN excluded_folders JSONB NOT NULL DEFAULT '[]';
|
||||||
|
-- carry the existing task-wide mapping onto its accounts
|
||||||
|
UPDATE accounts a SET folder_mapping = t.folder_mapping
|
||||||
|
FROM tasks t WHERE a.task_id = t.id AND t.folder_mapping <> '{}';
|
||||||
|
-- down
|
||||||
|
ALTER TABLE accounts DROP COLUMN excluded_folders;
|
||||||
|
ALTER TABLE accounts DROP COLUMN folder_mapping;
|
||||||
|
```
|
||||||
|
|
||||||
|
**`store.Account`** gains:
|
||||||
|
- `FolderMapping map[string]string`
|
||||||
|
- `ExcludedFolders []string`
|
||||||
|
|
||||||
|
`ListAccountsByTask` selects/scans both (JSONB). New helper:
|
||||||
|
- `SetAccountFolderMapping(ctx, id int64, mapping map[string]string, excluded []string) error`
|
||||||
|
|
||||||
|
`tasks.folder_mapping` column and `SetTaskFolderMapping` remain but are no longer
|
||||||
|
read by the orchestrator (kept for the migration source and to avoid a destructive
|
||||||
|
change; single-add stops writing it).
|
||||||
|
|
||||||
|
## Orchestrator
|
||||||
|
|
||||||
|
Extract the per-folder plan decision into a small **pure, testable** helper so the
|
||||||
|
exclusion/mapping logic can be unit-tested without IMAP:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// planFolders returns the (src,dst) pairs to copy for an account, dropping
|
||||||
|
// excluded source folders and applying renames.
|
||||||
|
func planFolders(folders []string, mapping map[string]string, excluded []string) []folderPlan
|
||||||
|
```
|
||||||
|
|
||||||
|
`runAccount` replaces the inline `task.FolderMapping[folder]` loop with
|
||||||
|
`planFolders(folders, a.FolderMapping, a.ExcludedFolders)`, then counts messages
|
||||||
|
per surviving folder as today. Excluded folders never enter the plan (not counted,
|
||||||
|
not scanned, not copied).
|
||||||
|
|
||||||
|
## HTTP API
|
||||||
|
|
||||||
|
- **`POST /api/tasks/{id}/accounts/{accId}/probe`** — new. Loads the account,
|
||||||
|
decrypts its stored src/dst passwords, reuses the existing `imapx.TestLogin`
|
||||||
|
probe, returns `{src:{ok,folders,error}, dst:{...}}`. No creds in the request.
|
||||||
|
- **`PUT /api/tasks/{id}/accounts/{accId}/folder-mapping`** — new. Body
|
||||||
|
`{mapping: map[string]string, excluded: string[]}` → `SetAccountFolderMapping`.
|
||||||
|
- **`AccountView` / `accountDTO`** expose `folder_mapping` and `excluded_folders`
|
||||||
|
so the modal can seed saved state when reopened.
|
||||||
|
- Existing task-level `POST /probe` stays (single-add probes before the account
|
||||||
|
exists). Task-level `PUT /folder-mapping` handler stays but is unused by the UI.
|
||||||
|
|
||||||
|
## Frontend
|
||||||
|
|
||||||
|
**`api.ts`**
|
||||||
|
- `Account` interface gains `folder_mapping?: Record<string,string>` and
|
||||||
|
`excluded_folders?: string[]`.
|
||||||
|
- `probeAccountFolders(taskId, accId)` → `ProbeResult`.
|
||||||
|
- `setAccountFolderMapping(taskId, accId, mapping, excluded)`.
|
||||||
|
|
||||||
|
**`FolderMappingModal`** — add exclusion:
|
||||||
|
- New props `initialExcluded: string[]`, and `onConfirm(mapping, excluded)`.
|
||||||
|
- Each row gets a leading **checkbox** (checked = sync). Unchecking greys/disables
|
||||||
|
that row's dst `<select>` and the "new/create" marker, and adds the src to the
|
||||||
|
excluded set.
|
||||||
|
- A header "sync all" checkbox toggles every row.
|
||||||
|
- `confirm()` emits `mapping` (only for synced rows) and `excluded` (unchecked rows).
|
||||||
|
|
||||||
|
**`TaskDetail`**
|
||||||
|
- Actions cell: add a `folders` link-btn **left of** remove/cancel, shown for every
|
||||||
|
account that is not currently running. Click → `probeAccountFolders(accId)` →
|
||||||
|
open the modal in **edit-account** mode seeded with the account's saved
|
||||||
|
`folder_mapping` + `excluded_folders` and the freshly probed folders.
|
||||||
|
- New state `editMap: { accId; src: string[]; dst: string[]; mapping; excluded } | null`,
|
||||||
|
mounted like the existing add-modal with a `key` per account for clean remount.
|
||||||
|
- On confirm in edit mode → `setAccountFolderMapping(accId, mapping, excluded)` → reload.
|
||||||
|
- Single-add `confirmMapping` switches from task-level `setFolderMapping` to
|
||||||
|
`setAccountFolderMapping(newAccId, mapping, excluded)` after `createAccount`.
|
||||||
|
- Probe button shows a busy state (`busy: 'probe'`) while re-probing an account.
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- Probe-by-account failures (login fails, IMAP down) surface via the existing
|
||||||
|
`error` banner (now `role="alert"`), same as single-add probe failures.
|
||||||
|
- Saving mapping while a run is in progress is blocked: the `folders` button is
|
||||||
|
hidden/disabled for `running` accounts (mapping is read at run start).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **store**: `SetAccountFolderMapping` round-trips mapping+excluded via
|
||||||
|
`ListAccountsByTask`; migration 0003 copies `tasks.folder_mapping` onto accounts.
|
||||||
|
- **orchestrator**: unit-test the extracted `planFolders` — excluded folders
|
||||||
|
dropped, renames applied, absent-src keeps its name, empty config = all folders.
|
||||||
|
- **httpapi**: probe-by-account decrypts and returns folders; PUT persists.
|
||||||
|
- **frontend / e2e**: `/run` the app — CSV-import an account, click **folders**,
|
||||||
|
uncheck a folder + rename another, save, run, verify the excluded folder is not
|
||||||
|
copied and the rename lands. Reduced to a manual E2E checklist per project rules.
|
||||||
|
|
||||||
|
## Out of scope (YAGNI)
|
||||||
|
|
||||||
|
- Task-wide "apply to all accounts" bulk mapping editor.
|
||||||
|
- Regex/glob folder matching.
|
||||||
|
- Removing the `tasks.folder_mapping` column (kept to avoid a destructive migration).
|
||||||
@@ -0,0 +1,207 @@
|
|||||||
|
# Scheduled (recurring) task runs — design
|
||||||
|
|
||||||
|
**Date:** 2026-07-03
|
||||||
|
**Status:** awaiting user review
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Migrations are run manually today (the operator clicks "Run migration"). For
|
||||||
|
ongoing source→destination sync, the operator wants a task to run itself on a
|
||||||
|
recurring interval (e.g. every 1h or 6h) without babysitting it. Requirements
|
||||||
|
from the user:
|
||||||
|
|
||||||
|
- Per-task recurrence interval; the task runs automatically.
|
||||||
|
- Never start a run while the previous one is still running; measure the interval
|
||||||
|
from the **completion** of the last run (not its start).
|
||||||
|
- Show when the next run is due, in the **browser's local time**.
|
||||||
|
- A separate modal showing a log of runs with their status and totals.
|
||||||
|
- Deleting a task must clean up all this data along with the task.
|
||||||
|
|
||||||
|
Decisions confirmed with the user:
|
||||||
|
|
||||||
|
- **Run-log detail:** per-run totals only (reuse `runs`); no per-account snapshot.
|
||||||
|
- **Enable gate:** a schedule can be enabled only when **all accounts test OK**.
|
||||||
|
- **Breaker:** if a *scheduled* run finishes with errors, disable the schedule and
|
||||||
|
mark the task **broken** (red icon in the tasks list and the task detail).
|
||||||
|
Manual runs never trip the breaker.
|
||||||
|
- **First run:** one interval after enabling (`schedule_anchor + interval`).
|
||||||
|
- **Intervals:** presets 1 / 3 / 6 / 12 / 24 h, plus Off.
|
||||||
|
|
||||||
|
### Current architecture (as-is)
|
||||||
|
|
||||||
|
- `runs` table already has `started_at`, `finished_at`, `status`, and totals
|
||||||
|
(`migrations/0001_init.up.sql:35-44`). `CreateRun`/`FinishRun` in
|
||||||
|
`internal/store/runs.go`. No list-by-task query yet.
|
||||||
|
- `Orchestrator.Run(ctx, taskID)` gates on `gateOK` (all accounts test ok) and
|
||||||
|
`TryMarkTaskRunning` (atomic single-run), creates a run, and launches `runAll`
|
||||||
|
asynchronously; `runAll` sets the task status and calls `FinishRun`
|
||||||
|
(`internal/orchestrator/orchestrator.go`).
|
||||||
|
- `ON DELETE CASCADE` from `tasks` already removes `runs`, `accounts`, and
|
||||||
|
`migrated_messages` when a task is deleted.
|
||||||
|
- `main.go` wires store → orchestrator → hub → server and calls
|
||||||
|
`ResetRunningOnStartup` to clear phantom "running" after a restart. No scheduler.
|
||||||
|
- Task DTO / TS `Task` interface: `internal/httpapi/tasks.go`, `web/src/api.ts`.
|
||||||
|
|
||||||
|
## Decision: in-process polling scheduler
|
||||||
|
|
||||||
|
A single background goroutine (a `Scheduler`) started from `main.go`, ticking
|
||||||
|
every **30 s**. Each tick queries the DB for schedulable tasks and triggers the
|
||||||
|
due ones through the existing `Orchestrator.Run`. The DB is the source of truth,
|
||||||
|
so the scheduler is stateless and restart-safe (paired with the existing
|
||||||
|
`ResetRunningOnStartup`). Chosen over per-task timers (restart-fragile, in-memory
|
||||||
|
state) and external cron (breaks the single-binary model, needs auth wiring).
|
||||||
|
|
||||||
|
30 s poll precision is ample for hour-scale intervals.
|
||||||
|
|
||||||
|
## Data model
|
||||||
|
|
||||||
|
**Migration `0004_task_scheduling`:**
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- up
|
||||||
|
ALTER TABLE tasks ADD COLUMN schedule_interval_seconds INT NOT NULL DEFAULT 0;
|
||||||
|
ALTER TABLE tasks ADD COLUMN schedule_anchor TIMESTAMPTZ;
|
||||||
|
ALTER TABLE tasks ADD COLUMN broken BOOLEAN NOT NULL DEFAULT false;
|
||||||
|
ALTER TABLE runs ADD COLUMN trigger TEXT NOT NULL DEFAULT 'manual';
|
||||||
|
-- down
|
||||||
|
ALTER TABLE runs DROP COLUMN trigger;
|
||||||
|
ALTER TABLE tasks DROP COLUMN broken;
|
||||||
|
ALTER TABLE tasks DROP COLUMN schedule_anchor;
|
||||||
|
ALTER TABLE tasks DROP COLUMN schedule_interval_seconds;
|
||||||
|
```
|
||||||
|
|
||||||
|
- `schedule_interval_seconds` — 0 = Off. Presets map to seconds (3600, 10800,
|
||||||
|
21600, 43200, 86400).
|
||||||
|
- `schedule_anchor` — set to `now()` when a schedule is enabled; the baseline for
|
||||||
|
the first run when no finished run exists yet.
|
||||||
|
- `broken` — the schedule breaker tripped; task needs operator attention.
|
||||||
|
- `runs.trigger` — `'manual' | 'scheduled'`; drives the breaker and enriches the
|
||||||
|
run log.
|
||||||
|
|
||||||
|
`store.Task` gains `ScheduleIntervalSeconds int64`, `ScheduleAnchor *time.Time`,
|
||||||
|
`Broken bool`. `store.Run` gains `Trigger string`, plus `StartedAt time.Time` and
|
||||||
|
`FinishedAt *time.Time` (needed for the log + due computation; not currently
|
||||||
|
scanned).
|
||||||
|
|
||||||
|
### Cascade / deletion
|
||||||
|
|
||||||
|
No new cleanup code. `runs` cascade-delete on task delete already; the schedule
|
||||||
|
columns live on `tasks` and vanish with the row. The in-memory scheduler polls
|
||||||
|
the DB, so a deleted task simply stops appearing. (Confirmed against
|
||||||
|
`migrations/0001_init.up.sql` FKs.)
|
||||||
|
|
||||||
|
## Store additions
|
||||||
|
|
||||||
|
- `SetTaskSchedule(ctx, id, intervalSeconds int64) error` — sets interval, sets
|
||||||
|
`schedule_anchor = now()` when enabling (interval>0) / leaves anchor when
|
||||||
|
disabling, and clears `broken` on any call (any explicit schedule change
|
||||||
|
un-breaks the task; enabling is additionally gated on all-accounts-OK in the
|
||||||
|
HTTP layer, disabling is always allowed).
|
||||||
|
- `SetTaskBroken(ctx, id) error` — `broken = true, schedule_interval_seconds = 0`
|
||||||
|
in one UPDATE (the breaker).
|
||||||
|
- `CreateRun(ctx, taskID, trigger string)` — extend the existing signature to
|
||||||
|
record the trigger.
|
||||||
|
- `ListRunsByTask(ctx, taskID) ([]Run, error)` — newest first, for the modal;
|
||||||
|
scans `id, started_at, finished_at, status, totals, trigger`.
|
||||||
|
- `LastFinishedRunAt(ctx, taskID) (*time.Time, error)` — most recent run with a
|
||||||
|
non-null `finished_at`, for due computation. (Or fold into `ListSchedulable`.)
|
||||||
|
- `ListSchedulableTasks(ctx) ([]ScheduledTask, error)` — tasks with
|
||||||
|
`schedule_interval_seconds > 0 AND NOT broken AND status <> 'running'`, joined
|
||||||
|
with their last finished run's `finished_at`, so the scheduler decides in one
|
||||||
|
query per tick. Returns the fields `dueAt` needs.
|
||||||
|
|
||||||
|
`GetTask`/`ListTasks` SELECT + Scan extend for the three new task columns.
|
||||||
|
|
||||||
|
## Scheduler (`internal/scheduler/`)
|
||||||
|
|
||||||
|
New package, one focused responsibility.
|
||||||
|
|
||||||
|
- `type Scheduler struct { store *store.Store; orch *orchestrator.Orchestrator }`
|
||||||
|
- `func (s *Scheduler) Start(ctx context.Context)` — `time.NewTicker(30s)` loop;
|
||||||
|
on each tick calls `s.tick(ctx)`; stops on `ctx.Done()`.
|
||||||
|
- `func (s *Scheduler) tick(ctx)` — `ListSchedulableTasks`, then for each due task
|
||||||
|
calls `s.orch.Run(ctx, taskID)` with `trigger = "scheduled"`. `Run` errors
|
||||||
|
(`ErrAlreadyRunning`, `ErrNotTested`) are logged, not fatal.
|
||||||
|
- **Pure, unit-tested decision function:**
|
||||||
|
`func dueAt(interval time.Duration, anchor time.Time, lastFinished *time.Time) time.Time`
|
||||||
|
→ `lastFinished + interval` if a finished run exists, else `anchor + interval`.
|
||||||
|
A task is due when `now >= dueAt(...)`. Kept pure so the tick logic is testable
|
||||||
|
without a DB or clock injection at the call site.
|
||||||
|
|
||||||
|
`main.go` starts it: `go scheduler.New(st, orch).Start(context.Background())`.
|
||||||
|
|
||||||
|
### Trigger threading + breaker
|
||||||
|
|
||||||
|
`Orchestrator.Run` gains a `trigger string` parameter (manual callers pass
|
||||||
|
`"manual"`; the scheduler passes `"scheduled"`). `Run` forwards it to
|
||||||
|
`CreateRun` and into `runAll`. In `runAll`, after the final status is computed:
|
||||||
|
|
||||||
|
```
|
||||||
|
if trigger == "scheduled" && totErr > 0 {
|
||||||
|
_ = o.store.SetTaskBroken(ctx, task.ID)
|
||||||
|
o.hub.Publish(Event{Type: "task_broken", TaskID: task.ID, ...})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The HTTP `handleRun` passes `"manual"`. This keeps the breaker precise: only
|
||||||
|
scheduled runs trip it.
|
||||||
|
|
||||||
|
## HTTP API
|
||||||
|
|
||||||
|
- **`PUT /api/tasks/{id}/schedule`** — body `{interval_seconds: int}`. Enabling
|
||||||
|
(interval>0) first checks `gateOK` (all accounts test OK) via
|
||||||
|
`ListAccountsByTask`; returns **409** with a clear message if not. Then
|
||||||
|
`SetTaskSchedule`. Disabling (0) always allowed.
|
||||||
|
- **`GET /api/tasks/{id}/runs`** — `ListRunsByTask` → run-log rows.
|
||||||
|
- **Task DTO** (`tasks.go`) gains `schedule_interval_seconds`, `broken`, and a
|
||||||
|
computed `next_run_at *string` (RFC3339, UTC; null when off / running / broken).
|
||||||
|
`next_run_at` is computed server-side from the same `dueAt` logic so the client
|
||||||
|
only formats it. `ListTasks` DTO also includes `broken` (for the red icon in the
|
||||||
|
list).
|
||||||
|
|
||||||
|
## Frontend
|
||||||
|
|
||||||
|
- **`api.ts`**: `Task` gains `schedule_interval_seconds?`, `broken?`,
|
||||||
|
`next_run_at?`. New `setTaskSchedule(taskId, intervalSeconds)` and
|
||||||
|
`listRuns(taskId)` (+ a `Run` interface).
|
||||||
|
- **`TaskDetail.tsx`**: in the run-control panel, an interval `<select>`
|
||||||
|
(Off/1h/3h/6h/12h/24h) bound to `setTaskSchedule`; a "Next run: <local time>"
|
||||||
|
line formatting `next_run_at` with `toLocaleString()` (shows "running…" when a
|
||||||
|
run is active, nothing when off); a red **broken** badge when `broken`. A
|
||||||
|
**Runs** button opens a new modal.
|
||||||
|
- **`RunLogModal.tsx`** (new): lists `listRuns(taskId)` rows — started/finished in
|
||||||
|
browser-local time, `trigger`, a `StatusBadge`, and copied/skipped/errors.
|
||||||
|
Reuses the existing `Modal` + `StatusBadge` + `.tbl` patterns.
|
||||||
|
- **`Tasks.tsx`**: red icon/indicator on rows where `broken`.
|
||||||
|
- WS: on `task_broken`/`run_done`, `TaskDetail` reloads (existing reload wiring);
|
||||||
|
`Tasks` list refreshes.
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- Enable-while-not-tested → 409, surfaced in the existing `role="alert"` banner.
|
||||||
|
- Scheduler `Run` errors are logged and skipped; a broken task is skipped every
|
||||||
|
tick until the operator re-enables (which clears `broken`).
|
||||||
|
- Restart: `ResetRunningOnStartup` clears phantom running; the scheduler recomputes
|
||||||
|
due times from persisted `finished_at`/`anchor`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **store**: `SetTaskSchedule` (sets interval+anchor, clears broken),
|
||||||
|
`SetTaskBroken` (interval→0, broken→true), `CreateRun` records trigger,
|
||||||
|
`ListRunsByTask` ordering + fields, `GetTask` returns new columns.
|
||||||
|
- **scheduler (pure)**: `dueAt` / due decision — not-yet-due, due-from-finished,
|
||||||
|
first-run-from-anchor, running skipped, broken skipped.
|
||||||
|
- **orchestrator**: scheduled run with errors trips `SetTaskBroken`; manual run
|
||||||
|
with errors does not.
|
||||||
|
- **cascade**: deleting a task removes its runs (extend existing cascade test).
|
||||||
|
- **E2E** (prod, per project rules): enable a schedule on a tested task → "Next
|
||||||
|
run" shows correct local time; (with a short test interval) the run auto-starts;
|
||||||
|
the Runs modal shows the entry with `scheduled` trigger; break an account → the
|
||||||
|
next scheduled run disables the schedule and marks the task broken (red icon in
|
||||||
|
list + detail).
|
||||||
|
|
||||||
|
## Out of scope (YAGNI)
|
||||||
|
|
||||||
|
- Arbitrary cron expressions; catch-up/backfill of missed runs (run once when due);
|
||||||
|
notifications; per-account breakdown in the run log (totals chosen); configurable
|
||||||
|
poll interval.
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# Account Errors Modal — Design
|
||||||
|
|
||||||
|
Date: 2026-07-05
|
||||||
|
Status: approved
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
When an account finishes `done_with_errors`, the UI shows only an error **count**
|
||||||
|
(e.g. `2`) and a single persisted `last_error`. There is no way to see the
|
||||||
|
individual errors — which folder, which message, and the actual error text. The
|
||||||
|
event log holds them live but is lost on reload, and message-level errors
|
||||||
|
(`res.Errors`) currently record only a counter, discarding their text entirely.
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Click the ERRORS count of an account (when `> 0`) to open a modal listing the
|
||||||
|
concrete errors of that account's **most recent run**: folder, kind, message
|
||||||
|
reference, error text, and timestamp.
|
||||||
|
|
||||||
|
## Scope decisions
|
||||||
|
|
||||||
|
- **Coverage:** errors of the latest run only. The list is cleared at the start
|
||||||
|
of each run (like the per-account counters).
|
||||||
|
- **Granularity:** every error is its own record — folder-level, message-level,
|
||||||
|
and account-level (connect/login) are distinct rows.
|
||||||
|
- **Cap:** at most 500 error rows per account per run (guard against a corrupt
|
||||||
|
mailbox producing thousands). On overflow a final synthetic row records
|
||||||
|
"… N more errors suppressed".
|
||||||
|
|
||||||
|
## Data model
|
||||||
|
|
||||||
|
New table (chosen over a JSONB column on `accounts`): a table gives atomic
|
||||||
|
`INSERT` per error with no read-modify-write races, trivial per-run clearing,
|
||||||
|
`ON DELETE CASCADE` with the account, and matches the existing `runs` /
|
||||||
|
`migrated_messages` shape.
|
||||||
|
|
||||||
|
Migration `0005_account_errors`:
|
||||||
|
|
||||||
|
```
|
||||||
|
account_errors(
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
account_id BIGINT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||||
|
run_id BIGINT, -- run the error belongs to (context)
|
||||||
|
kind TEXT NOT NULL, -- 'folder' | 'message' | 'account'
|
||||||
|
folder TEXT NOT NULL DEFAULT '',
|
||||||
|
message_ref TEXT NOT NULL DEFAULT '', -- e.g. "UID 42: <subject>"
|
||||||
|
error TEXT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
)
|
||||||
|
CREATE INDEX ON account_errors(account_id);
|
||||||
|
```
|
||||||
|
|
||||||
|
Down migration: `DROP TABLE account_errors`.
|
||||||
|
|
||||||
|
## Backend
|
||||||
|
|
||||||
|
**Store** (`internal/store/account_errors.go`):
|
||||||
|
- `AddAccountError(ctx, accountID, runID int64, kind, folder, ref, msg string) error`
|
||||||
|
- `ClearAccountErrors(ctx, accountID int64) error`
|
||||||
|
- `ListAccountErrors(ctx, accountID int64) ([]AccountError, error)` — ordered by id.
|
||||||
|
- `AccountError` struct mirrors the row.
|
||||||
|
|
||||||
|
**Orchestrator** (`internal/orchestrator/orchestrator.go`):
|
||||||
|
- At account start (next to `ResetAccountCounters`): `ClearAccountErrors`.
|
||||||
|
- New `CopyDeps.OnError(kind, folder, ref, msg string)` callback. `CopyFolder`
|
||||||
|
invokes it on every message-level error (the `res.Errors++` sites in
|
||||||
|
`copy.go`) and folder-level error, passing UID/subject where available. The
|
||||||
|
orchestrator's `OnError` impl persists via `AddAccountError`, enforcing the
|
||||||
|
500-row cap with an in-worker counter.
|
||||||
|
- Folder-level copy error (already persisted to `last_error`) also emits
|
||||||
|
`OnError(kind="folder", ...)`.
|
||||||
|
- `accountFailed` (connect/login/decrypt) emits `OnError(kind="account", ...)`.
|
||||||
|
- `last_error` behavior is unchanged (still the latest single error for the
|
||||||
|
inline row hint).
|
||||||
|
|
||||||
|
**HTTP** (`internal/httpapi`):
|
||||||
|
- `GET /api/tasks/{id}/accounts/{accountId}/errors` → JSON `[]AccountError`.
|
||||||
|
|
||||||
|
## Frontend
|
||||||
|
|
||||||
|
- `web/src/api.ts`: `listAccountErrors(taskId, accountId)`.
|
||||||
|
- ERRORS cell becomes a button when the count `> 0`; opens `AccountErrorsModal`.
|
||||||
|
- `web/src/components/AccountErrorsModal.tsx` (built on existing `Modal`,
|
||||||
|
styled like `RunLogModal`): fetches on open, renders rows
|
||||||
|
`time · folder · kind · message_ref · error`, with loading / empty /
|
||||||
|
fetch-error states.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- Store: `AddAccountError` / `ClearAccountErrors` / `ListAccountErrors` against
|
||||||
|
Postgres (existing store test harness).
|
||||||
|
- `CopyFolder`: `OnError` is invoked when a per-message step fails (e.g.
|
||||||
|
`IsMigrated` returns an error) — unit test with fakes, no server needed.
|
||||||
|
- HTTP: handler returns the account's errors as JSON.
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Cross-run error history.
|
||||||
|
- Retrying individual failed messages from the modal.
|
||||||
|
- Exporting errors.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# Selective Account Run — Design
|
||||||
|
|
||||||
|
**Date:** 2026-07-08
|
||||||
|
**Status:** Approved (pending spec review)
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
When a migration run fails for some accounts (e.g. transient `use of closed
|
||||||
|
network connection`), the user wants to re-run **only the affected accounts**,
|
||||||
|
not the whole task.
|
||||||
|
|
||||||
|
An earlier idea — per-error "retry" buttons inside the errors modal — was
|
||||||
|
rejected: on a `closed network connection` the copy aborts the whole folder
|
||||||
|
(`copy.go`, the `net.ErrClosed` branch), so the undelivered tail of the folder
|
||||||
|
never appears in the error list. A per-UID retry would replay only the few
|
||||||
|
visible errors and leave the invisible tail uncopied, creating a false sense of
|
||||||
|
completion.
|
||||||
|
|
||||||
|
The correct primitive is a **re-run of the selected accounts**. Copy is
|
||||||
|
idempotent — `IsMigrated`/`MarkMigrated` dedup by message key — so a re-run
|
||||||
|
skips everything already copied and delivers only the missing tail, covering the
|
||||||
|
network-drop case fully.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
Add checkboxes to the accounts table. The main "Run migration" button runs the
|
||||||
|
selected accounts; with no selection it runs all accounts (current behavior,
|
||||||
|
preserved).
|
||||||
|
|
||||||
|
## Backend
|
||||||
|
|
||||||
|
### `orchestrator.Run`
|
||||||
|
|
||||||
|
Change signature to:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)
|
||||||
|
```
|
||||||
|
|
||||||
|
- `accountIDs == nil || len == 0` → all accounts (unchanged behavior).
|
||||||
|
- Otherwise, filter the `ListAccountsByTask` result down to the requested IDs
|
||||||
|
**before** `gateOK`, so only the selected accounts are gate-checked and run.
|
||||||
|
- IDs not belonging to the task are silently dropped by the filter (defense in
|
||||||
|
depth; the handler also validates).
|
||||||
|
- If the filter yields zero accounts, return `ErrNotTested` (nothing runnable) —
|
||||||
|
reuses the existing 409 path; the handler maps empty selection separately (see
|
||||||
|
below).
|
||||||
|
|
||||||
|
Call sites updated:
|
||||||
|
- `scheduler.go` → `Run(ctx, id, "scheduled", nil)` (always all accounts).
|
||||||
|
- `run.go` (`handleRun`) → passes the parsed IDs.
|
||||||
|
|
||||||
|
`runAll`, `runAccount`, counters, `FinishRun`, and task-status reconciliation are
|
||||||
|
unchanged: they already operate over whatever `accs` slice they are handed.
|
||||||
|
|
||||||
|
### `handleRun`
|
||||||
|
|
||||||
|
- Parse an **optional** JSON body `{"account_ids": [1,2,3]}`. Empty/absent body
|
||||||
|
→ `nil` → all accounts.
|
||||||
|
- Validate parsed IDs against the task's accounts; if a non-empty selection
|
||||||
|
resolves to zero valid IDs → `400 Bad Request`.
|
||||||
|
- Existing error mapping preserved: `ErrNotTested` → 409 "accounts must pass
|
||||||
|
connection tests first"; `ErrAlreadyRunning` → 409.
|
||||||
|
|
||||||
|
## Frontend
|
||||||
|
|
||||||
|
### Accounts table (`TaskDetail.tsx`)
|
||||||
|
|
||||||
|
- New leading column with a per-row checkbox; a "select all" checkbox in the
|
||||||
|
header (selects only selectable rows).
|
||||||
|
- State: `selected: Set<number>` of account IDs.
|
||||||
|
- A row checkbox is disabled when the account has not passed **both** tests
|
||||||
|
(`test_src_status`/`test_dst_status` !== 'ok') or the task is `running`.
|
||||||
|
- "select all" reflects indeterminate/checked state over the selectable rows.
|
||||||
|
|
||||||
|
### Run button
|
||||||
|
|
||||||
|
- With a non-empty selection: label "Run selected (N)"; calls
|
||||||
|
`runTask(id, [...selected])`.
|
||||||
|
- With no selection: label "Run migration"; calls `runTask(id)` (all).
|
||||||
|
- Enable gate computed over the **effective set** (selected, or all when none
|
||||||
|
selected): every account in that set must be tested OK, and the task must not
|
||||||
|
be `running`.
|
||||||
|
- Side benefit: one untested account no longer blocks running the ready ones —
|
||||||
|
select only the ready accounts and run them.
|
||||||
|
|
||||||
|
### `api.ts`
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export const runTask = (id: number, accountIds?: number[]) =>
|
||||||
|
api(`/api/tasks/${id}/run`, accountIds?.length
|
||||||
|
? jsonBody({ account_ids: accountIds })
|
||||||
|
: { method: 'POST' })
|
||||||
|
```
|
||||||
|
|
||||||
|
## Out of scope (YAGNI)
|
||||||
|
|
||||||
|
- The errors modal (`AccountErrorsModal`) is untouched; no retry buttons there.
|
||||||
|
- No new "partial" task status. After a partial run the task status reflects the
|
||||||
|
last run's outcome over the accounts that ran — same as today.
|
||||||
|
- No selection persistence across reloads.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Backend unit:** `orchestrator.Run` with a subset filters correctly; empty
|
||||||
|
filter runs all; unknown IDs dropped; empty-after-filter → `ErrNotTested`.
|
||||||
|
- **Backend handler:** `handleRun` parses body, validates IDs, maps empty
|
||||||
|
selection → 400, absent body → all.
|
||||||
|
- **Frontend:** manual verification in the running app — select a subset, run,
|
||||||
|
confirm only those accounts start (WS `account_started` events), and the
|
||||||
|
unselected accounts' counters/status are untouched.
|
||||||
@@ -15,6 +15,7 @@ type Config struct {
|
|||||||
EncKey []byte
|
EncKey []byte
|
||||||
SessionSecret []byte
|
SessionSecret []byte
|
||||||
WorkerConcurrency int
|
WorkerConcurrency int
|
||||||
|
PprofAddr string // if non-empty, serve net/http/pprof here (e.g. ":6060")
|
||||||
}
|
}
|
||||||
|
|
||||||
func Load() (Config, error) {
|
func Load() (Config, error) {
|
||||||
@@ -24,7 +25,8 @@ func Load() (Config, error) {
|
|||||||
AuthUser: os.Getenv("AUTH_USER"),
|
AuthUser: os.Getenv("AUTH_USER"),
|
||||||
AuthPass: os.Getenv("AUTH_PASS"),
|
AuthPass: os.Getenv("AUTH_PASS"),
|
||||||
SessionSecret: []byte(os.Getenv("SESSION_SECRET")),
|
SessionSecret: []byte(os.Getenv("SESSION_SECRET")),
|
||||||
WorkerConcurrency: 4,
|
WorkerConcurrency: 2,
|
||||||
|
PprofAddr: os.Getenv("PPROF_ADDR"),
|
||||||
}
|
}
|
||||||
if v := os.Getenv("WORKER_CONCURRENCY"); v != "" {
|
if v := os.Getenv("WORKER_CONCURRENCY"); v != "" {
|
||||||
n, err := strconv.Atoi(v)
|
n, err := strconv.Atoi(v)
|
||||||
|
|||||||
@@ -31,7 +31,7 @@ func TestLoadDefaults(t *testing.T) {
|
|||||||
if cfg.HTTPAddr != ":8080" {
|
if cfg.HTTPAddr != ":8080" {
|
||||||
t.Errorf("HTTPAddr = %q, want :8080", cfg.HTTPAddr)
|
t.Errorf("HTTPAddr = %q, want :8080", cfg.HTTPAddr)
|
||||||
}
|
}
|
||||||
if cfg.WorkerConcurrency != 4 {
|
if cfg.WorkerConcurrency != 2 {
|
||||||
t.Errorf("WorkerConcurrency = %d, want 4", cfg.WorkerConcurrency)
|
t.Errorf("WorkerConcurrency = %d, want 2", cfg.WorkerConcurrency)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
package csvimport
|
package csvimport
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bufio"
|
||||||
"encoding/csv"
|
"encoding/csv"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -52,3 +54,110 @@ func Parse(r io.Reader) ([]Row, error) {
|
|||||||
}
|
}
|
||||||
return rows, nil
|
return rows, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Kerio Connect exports users as a semicolon-separated file whose first line is
|
||||||
|
// a header. Only these columns matter; the rest (quotas, last login, …) is
|
||||||
|
// ignored. The file carries no domain — the caller supplies it.
|
||||||
|
const (
|
||||||
|
kerioColName = 0 // login without the domain part
|
||||||
|
kerioColDescription = 2 // Kerio keeps the plaintext password here
|
||||||
|
kerioColEnable = 3 // "Yes" / "No"
|
||||||
|
kerioMinColumns = 4
|
||||||
|
)
|
||||||
|
|
||||||
|
// domainRe accepts a bare DNS domain: labels of alphanumerics/hyphens with at
|
||||||
|
// least one dot and no scheme, user part, or whitespace.
|
||||||
|
var domainRe = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$`)
|
||||||
|
|
||||||
|
func normalizeDomain(domain string) (string, error) {
|
||||||
|
d := strings.ToLower(strings.TrimSpace(domain))
|
||||||
|
if d == "" {
|
||||||
|
return "", fmt.Errorf("domain is required")
|
||||||
|
}
|
||||||
|
if !domainRe.MatchString(d) {
|
||||||
|
return "", fmt.Errorf("invalid domain %q", domain)
|
||||||
|
}
|
||||||
|
return d, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// skipBOM consumes a leading UTF-8 byte-order mark, which Kerio writes into its
|
||||||
|
// exports and encoding/csv would otherwise glue onto the first header field.
|
||||||
|
func skipBOM(br *bufio.Reader) error {
|
||||||
|
b, err := br.Peek(3)
|
||||||
|
if err != nil && err != io.EOF {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(b) == 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
|
||||||
|
_, _ = br.Discard(3)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseKerio reads a Kerio Connect user export and maps every enabled, non-admin
|
||||||
|
// account onto both sides of a migration: the login and password are identical
|
||||||
|
// on source and destination, only the server differs. Rows for disabled accounts
|
||||||
|
// and the built-in admin are skipped.
|
||||||
|
func ParseKerio(r io.Reader, domain string) ([]Row, error) {
|
||||||
|
d, err := normalizeDomain(domain)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
br := bufio.NewReader(r)
|
||||||
|
if err := skipBOM(br); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
cr := csv.NewReader(br)
|
||||||
|
cr.Comma = ';'
|
||||||
|
cr.FieldsPerRecord = -1 // проверяем сами
|
||||||
|
cr.LazyQuotes = true // FullName нередко содержит одиночную кавычку
|
||||||
|
|
||||||
|
header, err := cr.Read()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("cannot read header: %w", err)
|
||||||
|
}
|
||||||
|
if len(header) < kerioMinColumns || !strings.EqualFold(strings.TrimSpace(header[kerioColName]), "Name") {
|
||||||
|
return nil, fmt.Errorf("not a Kerio export: expected a header starting with Name;FullName;Description;Enable")
|
||||||
|
}
|
||||||
|
|
||||||
|
var rows []Row
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for {
|
||||||
|
rec, err := cr.Read()
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
line, _ := cr.FieldPos(0)
|
||||||
|
if len(rec) == 1 && strings.TrimSpace(rec[0]) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if len(rec) < kerioMinColumns {
|
||||||
|
return nil, fmt.Errorf("line %d: expected at least %d columns, got %d", line, kerioMinColumns, len(rec))
|
||||||
|
}
|
||||||
|
name := strings.ToLower(strings.TrimSpace(rec[kerioColName]))
|
||||||
|
if name == "" {
|
||||||
|
return nil, fmt.Errorf("line %d: Name is empty", line)
|
||||||
|
}
|
||||||
|
if !strings.EqualFold(strings.TrimSpace(rec[kerioColEnable]), "Yes") || name == "admin" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pass := strings.TrimSpace(rec[kerioColDescription])
|
||||||
|
if pass == "" {
|
||||||
|
return nil, fmt.Errorf("line %d: no password in the Description column for %q", line, name)
|
||||||
|
}
|
||||||
|
if seen[name] {
|
||||||
|
return nil, fmt.Errorf("line %d: duplicate Name %q", line, name)
|
||||||
|
}
|
||||||
|
seen[name] = true
|
||||||
|
login := name + "@" + d
|
||||||
|
rows = append(rows, Row{SrcLogin: login, SrcPass: pass, DstLogin: login, DstPass: pass})
|
||||||
|
}
|
||||||
|
if len(rows) == 0 {
|
||||||
|
return nil, fmt.Errorf("no enabled accounts found in the export")
|
||||||
|
}
|
||||||
|
return rows, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -49,3 +49,100 @@ func TestParseZeroRowsErrors(t *testing.T) {
|
|||||||
t.Fatal("expected error when no rows parsed")
|
t.Fatal("expected error when no rows parsed")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const kerioHeader = "Name;FullName;Description;Enable;DataSource;Authentication;Role;Groups;MailAddress\n"
|
||||||
|
|
||||||
|
func TestParseKerioOK(t *testing.T) {
|
||||||
|
in := kerioHeader +
|
||||||
|
"j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n" +
|
||||||
|
"k.smith;Kim Smith;SrcPass22;Yes;Internal;Internal;No rights;all;k.smith\n"
|
||||||
|
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(rows) != 2 {
|
||||||
|
t.Fatalf("want 2 rows, got %d: %+v", len(rows), rows)
|
||||||
|
}
|
||||||
|
want := Row{
|
||||||
|
SrcLogin: "j.doe@example.test", SrcPass: "SrcPass11",
|
||||||
|
DstLogin: "j.doe@example.test", DstPass: "SrcPass11",
|
||||||
|
}
|
||||||
|
if rows[0] != want {
|
||||||
|
t.Fatalf("row 0: got %+v, want %+v", rows[0], want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioSkipsDisabledAndAdmin(t *testing.T) {
|
||||||
|
in := kerioHeader +
|
||||||
|
"admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n" +
|
||||||
|
"o.disabled;;OffPass111;No;Internal;Internal;No rights;all;o.disabled\n" +
|
||||||
|
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" {
|
||||||
|
t.Fatalf("only the enabled non-admin row must survive, got %+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioStripsBOM(t *testing.T) {
|
||||||
|
in := "\ufeff" + kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(rows) != 1 {
|
||||||
|
t.Fatalf("want 1 row, got %+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioRejectsMissingHeader(t *testing.T) {
|
||||||
|
in := "j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n"
|
||||||
|
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||||
|
t.Fatal("a file without the Kerio header must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioRejectsBadDomain(t *testing.T) {
|
||||||
|
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
for _, domain := range []string{"", " ", "@example.test", "exam ple.test", "example", "info@example.test"} {
|
||||||
|
if _, err := ParseKerio(strings.NewReader(in), domain); err == nil {
|
||||||
|
t.Fatalf("domain %q must be rejected", domain)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioTrimsAndLowercasesDomain(t *testing.T) {
|
||||||
|
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
rows, err := ParseKerio(strings.NewReader(in), " Example.TEST ")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if rows[0].SrcLogin != "info@example.test" {
|
||||||
|
t.Fatalf("domain must be trimmed and lowercased, got %q", rows[0].SrcLogin)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioRejectsEmptyPassword(t *testing.T) {
|
||||||
|
in := kerioHeader + "info;;;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||||
|
t.Fatal("an enabled account without a password must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioRejectsDuplicateName(t *testing.T) {
|
||||||
|
in := kerioHeader +
|
||||||
|
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n" +
|
||||||
|
"info;;aaaaaaAa1;Yes;Internal;Internal;No rights;all;info\n"
|
||||||
|
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||||
|
t.Fatal("duplicate Name must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseKerioZeroRowsErrors(t *testing.T) {
|
||||||
|
in := kerioHeader + "admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n"
|
||||||
|
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||||
|
t.Fatal("expected error when every row is filtered out")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
+219
-11
@@ -12,16 +12,18 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type AccountView struct {
|
type AccountView struct {
|
||||||
ID int64 `json:"id"`
|
ID int64 `json:"id"`
|
||||||
SrcLogin string `json:"src_login"`
|
SrcLogin string `json:"src_login"`
|
||||||
DstLogin string `json:"dst_login"`
|
DstLogin string `json:"dst_login"`
|
||||||
TestSrcStatus string `json:"test_src_status"`
|
TestSrcStatus string `json:"test_src_status"`
|
||||||
TestDstStatus string `json:"test_dst_status"`
|
TestDstStatus string `json:"test_dst_status"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
Copied int64 `json:"copied"`
|
Copied int64 `json:"copied"`
|
||||||
Skipped int64 `json:"skipped"`
|
Skipped int64 `json:"skipped"`
|
||||||
Errors int64 `json:"errors"`
|
Errors int64 `json:"errors"`
|
||||||
LastError string `json:"last_error,omitempty"`
|
LastError string `json:"last_error,omitempty"`
|
||||||
|
FolderMapping map[string]string `json:"folder_mapping"`
|
||||||
|
ExcludedFolders []string `json:"excluded_folders"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func accountDTO(a store.Account) AccountView {
|
func accountDTO(a store.Account) AccountView {
|
||||||
@@ -29,7 +31,9 @@ func accountDTO(a store.Account) AccountView {
|
|||||||
ID: a.ID, SrcLogin: a.SrcLogin, DstLogin: a.DstLogin,
|
ID: a.ID, SrcLogin: a.SrcLogin, DstLogin: a.DstLogin,
|
||||||
TestSrcStatus: a.TestSrcStatus, TestDstStatus: a.TestDstStatus,
|
TestSrcStatus: a.TestSrcStatus, TestDstStatus: a.TestDstStatus,
|
||||||
Status: a.Status, Copied: a.Copied, Skipped: a.Skipped, Errors: a.Errors,
|
Status: a.Status, Copied: a.Copied, Skipped: a.Skipped, Errors: a.Errors,
|
||||||
LastError: a.LastError,
|
LastError: a.LastError,
|
||||||
|
FolderMapping: a.FolderMapping,
|
||||||
|
ExcludedFolders: a.ExcludedFolders,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,3 +153,207 @@ func (s *Server) handleCreateAccount(w http.ResponseWriter, r *http.Request) {
|
|||||||
}
|
}
|
||||||
writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
|
writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleUpdateAccountCredentials fixes the logins/passwords of an existing
|
||||||
|
// account — typically after an import brought in a wrong password and the
|
||||||
|
// connection test failed. An empty password field keeps the stored one, so the
|
||||||
|
// operator can correct one side without retyping the other.
|
||||||
|
func (s *Server) handleUpdateAccountCredentials(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
task, err := s.store.GetTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
acc, ok := s.findAccount(r, taskID, accID)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "account not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if task.Status == "running" || acc.Status == "running" {
|
||||||
|
http.Error(w, "cannot change credentials while the account is running", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
SrcLogin string `json:"src_login"`
|
||||||
|
SrcPass string `json:"src_pass"`
|
||||||
|
DstLogin string `json:"dst_login"`
|
||||||
|
DstPass string `json:"dst_pass"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||||
|
http.Error(w, "bad json", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Same trimming as account creation: pasted logins/passwords often carry a
|
||||||
|
// stray space or newline that the IMAP server rejects.
|
||||||
|
body.SrcLogin = strings.TrimSpace(body.SrcLogin)
|
||||||
|
body.DstLogin = strings.TrimSpace(body.DstLogin)
|
||||||
|
body.SrcPass = strings.TrimSpace(body.SrcPass)
|
||||||
|
body.DstPass = strings.TrimSpace(body.DstPass)
|
||||||
|
if body.SrcLogin == "" || body.DstLogin == "" {
|
||||||
|
http.Error(w, "src_login and dst_login are required", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
encrypt := func(pass string) (*string, error) {
|
||||||
|
if pass == "" {
|
||||||
|
return nil, nil // keep the stored password
|
||||||
|
}
|
||||||
|
enc, err := crypto.Encrypt(s.cfg.EncKey, []byte(pass))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &enc, nil
|
||||||
|
}
|
||||||
|
srcEnc, err := encrypt(body.SrcPass)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "encrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dstEnc, err := encrypt(body.DstPass)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "encrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.store.UpdateAccountCredentials(r.Context(), accID, body.SrcLogin, body.DstLogin, srcEnc, dstEnc); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
|
||||||
|
// findAccount returns the account with accID under taskID, or ok=false.
|
||||||
|
func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) {
|
||||||
|
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
return store.Account{}, false
|
||||||
|
}
|
||||||
|
for _, a := range accs {
|
||||||
|
if a.ID == accID {
|
||||||
|
return a, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return store.Account{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleProbeAccountFolders re-probes an existing account using its stored
|
||||||
|
// (encrypted) credentials, so the operator can (re)map an account added via CSV.
|
||||||
|
func (s *Server) handleProbeAccountFolders(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
task, err := s.store.GetTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a, ok := s.findAccount(r, taskID, accID)
|
||||||
|
if !ok {
|
||||||
|
http.Error(w, "account not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
srcEP, err := s.store.GetEndpoint(r.Context(), task.SrcEndpointID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dstEP, err := s.store.GetEndpoint(r.Context(), task.DstEndpointID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
srcPass, err := crypto.Decrypt(s.cfg.EncKey, a.SrcPassEnc)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "decrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
dstPass, err := crypto.Decrypt(s.cfg.EncKey, a.DstPassEnc)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "decrypt", http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
probe := func(ep store.Endpoint, login string, pass []byte) map[string]any {
|
||||||
|
folders, err := imapx.TestLogin(r.Context(),
|
||||||
|
imapx.Endpoint{Host: ep.Host, Port: ep.Port, TLSMode: ep.TLSMode},
|
||||||
|
strings.TrimSpace(login), strings.TrimSpace(string(pass)))
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"ok": false, "error": err.Error(), "folders": []string{}}
|
||||||
|
}
|
||||||
|
return map[string]any{"ok": true, "folders": folders}
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{
|
||||||
|
"src": probe(srcEP, a.SrcLogin, srcPass),
|
||||||
|
"dst": probe(dstEP, a.DstLogin, dstPass),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleListAccountErrors returns the individual errors recorded for an account
|
||||||
|
// during its most recent run, for the per-account error modal.
|
||||||
|
func (s *Server) handleListAccountErrors(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, ok := s.findAccount(r, taskID, accID); !ok {
|
||||||
|
http.Error(w, "account not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
errs, err := s.store.ListAccountErrors(r.Context(), accID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, errs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleSetAccountFolderMapping persists one account's rename map + excluded set.
|
||||||
|
func (s *Server) handleSetAccountFolderMapping(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
accID, err := pathID(r, "accountId")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, ok := s.findAccount(r, taskID, accID); !ok {
|
||||||
|
http.Error(w, "account not found", http.StatusNotFound)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
Mapping map[string]string `json:"mapping"`
|
||||||
|
Excluded []string `json:"excluded"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||||
|
http.Error(w, "bad json", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.store.SetAccountFolderMapping(r.Context(), accID, body.Mapping, body.Excluded); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
|||||||
@@ -2,8 +2,11 @@ package httpapi
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgconn"
|
||||||
"github.com/vasyansk/imap-copier/internal/store"
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -54,6 +57,36 @@ func (s *Server) handleUpdateEndpoint(w http.ResponseWriter, r *http.Request) {
|
|||||||
w.WriteHeader(http.StatusNoContent)
|
w.WriteHeader(http.StatusNoContent)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handleDeleteEndpoint removes an endpoint that no task references. A referenced
|
||||||
|
// endpoint is refused with 409 rather than a foreign-key error, and the same
|
||||||
|
// status covers the race where a task is created between check and delete.
|
||||||
|
func (s *Server) handleDeleteEndpoint(w http.ResponseWriter, r *http.Request) {
|
||||||
|
id, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
used, err := s.store.CountTasksUsingEndpoint(r.Context(), id)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if used > 0 {
|
||||||
|
http.Error(w, fmt.Sprintf("endpoint is used by %d task(s) — delete them first", used), http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.store.DeleteEndpoint(r.Context(), id); err != nil {
|
||||||
|
var pgErr *pgconn.PgError
|
||||||
|
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
|
||||||
|
http.Error(w, "endpoint is used by a task — delete it first", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) {
|
||||||
eps, err := s.store.ListEndpoints(r.Context())
|
eps, err := s.store.ListEndpoints(r.Context())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ func (s *Server) Router() http.Handler {
|
|||||||
api.HandleFunc("GET /api/endpoints", s.handleListEndpoints)
|
api.HandleFunc("GET /api/endpoints", s.handleListEndpoints)
|
||||||
api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint)
|
api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint)
|
||||||
api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint)
|
api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint)
|
||||||
|
api.HandleFunc("DELETE /api/endpoints/{id}", s.handleDeleteEndpoint)
|
||||||
api.HandleFunc("GET /api/tasks", s.handleListTasks)
|
api.HandleFunc("GET /api/tasks", s.handleListTasks)
|
||||||
api.HandleFunc("POST /api/tasks", s.handleCreateTask)
|
api.HandleFunc("POST /api/tasks", s.handleCreateTask)
|
||||||
api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask)
|
api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask)
|
||||||
@@ -22,11 +23,20 @@ func (s *Server) Router() http.Handler {
|
|||||||
api.HandleFunc("POST /api/tasks/{id}/accounts", s.handleCreateAccount)
|
api.HandleFunc("POST /api/tasks/{id}/accounts", s.handleCreateAccount)
|
||||||
api.HandleFunc("POST /api/tasks/{id}/probe", s.handleProbeFolders)
|
api.HandleFunc("POST /api/tasks/{id}/probe", s.handleProbeFolders)
|
||||||
api.HandleFunc("PUT /api/tasks/{id}/folder-mapping", s.handleSetFolderMapping)
|
api.HandleFunc("PUT /api/tasks/{id}/folder-mapping", s.handleSetFolderMapping)
|
||||||
|
api.HandleFunc("PUT /api/tasks/{id}/schedule", s.handleSetSchedule)
|
||||||
|
api.HandleFunc("GET /api/tasks/{id}/runs", s.handleListRuns)
|
||||||
|
api.HandleFunc("GET /api/tasks/{id}/accounts/{accountId}/errors", s.handleListAccountErrors)
|
||||||
api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount)
|
api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount)
|
||||||
|
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/credentials", s.handleUpdateAccountCredentials)
|
||||||
api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV)
|
api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV)
|
||||||
api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts)
|
api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts)
|
||||||
api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun)
|
api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun)
|
||||||
|
api.HandleFunc("POST /api/tasks/{id}/pause", s.handlePauseRun)
|
||||||
|
api.HandleFunc("POST /api/tasks/{id}/cancel", s.handleCancelRun)
|
||||||
|
api.HandleFunc("POST /api/tasks/{id}/resume", s.handleResumeRun)
|
||||||
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount)
|
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount)
|
||||||
|
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders)
|
||||||
|
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping)
|
||||||
mux.Handle("/api/", s.requireAuth(api))
|
mux.Handle("/api/", s.requireAuth(api))
|
||||||
mux.Handle("/ws", s.requireAuth(http.HandlerFunc(s.handleWS)))
|
mux.Handle("/ws", s.requireAuth(http.HandlerFunc(s.handleWS)))
|
||||||
|
|
||||||
|
|||||||
+94
-2
@@ -1,8 +1,11 @@
|
|||||||
package httpapi
|
package httpapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"io"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
|
||||||
"github.com/vasyansk/imap-copier/internal/crypto"
|
"github.com/vasyansk/imap-copier/internal/crypto"
|
||||||
@@ -11,6 +14,17 @@ import (
|
|||||||
"github.com/vasyansk/imap-copier/internal/store"
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// parseImportRows picks the CSV dialect from the "format" form field. A Kerio
|
||||||
|
// Connect export holds one login/password pair and no domain, so the operator
|
||||||
|
// supplies the domain alongside the file; anything else is the plain 4-column
|
||||||
|
// src/dst format.
|
||||||
|
func parseImportRows(r *http.Request, file io.Reader) ([]csvimport.Row, error) {
|
||||||
|
if r.FormValue("format") == "kerio" {
|
||||||
|
return csvimport.ParseKerio(file, r.FormValue("domain"))
|
||||||
|
}
|
||||||
|
return csvimport.Parse(file)
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
|
||||||
taskID, err := pathID(r, "id")
|
taskID, err := pathID(r, "id")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -23,7 +37,7 @@ func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
defer file.Close()
|
defer file.Close()
|
||||||
rows, err := csvimport.Parse(file)
|
rows, err := parseImportRows(r, file)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||||
return
|
return
|
||||||
@@ -63,13 +77,41 @@ func (s *Server) handleTestAccounts(w http.ResponseWriter, r *http.Request) {
|
|||||||
w.WriteHeader(http.StatusAccepted)
|
w.WriteHeader(http.StatusAccepted)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
|
||||||
|
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
|
||||||
|
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
|
||||||
|
raw, err := io.ReadAll(r.Body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(bytes.TrimSpace(raw)) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
AccountIDs []int64 `json:"account_ids"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &body); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return body.AccountIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
|
||||||
taskID, err := pathID(r, "id")
|
taskID, err := pathID(r, "id")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(w, "bad id", http.StatusBadRequest)
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
runID, err := s.orch.Run(r.Context(), taskID)
|
accountIDs, err := parseRunAccountIDs(r)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad request body", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
|
||||||
|
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
|
||||||
|
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
if errors.Is(err, orchestrator.ErrNotTested) {
|
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||||
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||||
return
|
return
|
||||||
@@ -85,6 +127,56 @@ func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
|
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handlePauseRun stops the live run but keeps the unfinished accounts
|
||||||
|
// resumable; handleCancelRun stops it for good. Both are no-ops (409) when the
|
||||||
|
// task has no run in flight.
|
||||||
|
func (s *Server) handlePauseRun(w http.ResponseWriter, r *http.Request) {
|
||||||
|
s.stopRun(w, r, s.orch.PauseTask)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) handleCancelRun(w http.ResponseWriter, r *http.Request) {
|
||||||
|
s.stopRun(w, r, s.orch.CancelTask)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) stopRun(w http.ResponseWriter, r *http.Request, stop func(int64) bool) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !stop(taskID) {
|
||||||
|
http.Error(w, "task is not running", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusAccepted)
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleResumeRun restarts a paused task with the accounts its pause left
|
||||||
|
// unfinished; already-copied messages are skipped by the migration journal.
|
||||||
|
func (s *Server) handleResumeRun(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
runID, err := s.orch.ResumeTask(r.Context(), taskID)
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, orchestrator.ErrNothingToResume):
|
||||||
|
http.Error(w, "no paused accounts to resume", http.StatusConflict)
|
||||||
|
return
|
||||||
|
case errors.Is(err, orchestrator.ErrNotTested):
|
||||||
|
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||||
|
return
|
||||||
|
case errors.Is(err, orchestrator.ErrAlreadyRunning):
|
||||||
|
http.Error(w, "task is already running", http.StatusConflict)
|
||||||
|
return
|
||||||
|
case err != nil:
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) {
|
||||||
taskID, err := pathID(r, "id")
|
taskID, err := pathID(r, "id")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
package httpapi
|
package httpapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"io"
|
||||||
"mime/multipart"
|
"mime/multipart"
|
||||||
|
"net/http"
|
||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
@@ -26,3 +28,79 @@ func TestImportCSVFailsOnBadEncKey(t *testing.T) {
|
|||||||
t.Fatalf("import must fail on bad EncKey, got %d", rw.Code)
|
t.Fatalf("import must fail on bad EncKey, got %d", rw.Code)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// importReq builds a multipart import request with the given CSV payload and
|
||||||
|
// extra form fields, then returns it with the uploaded file ready to read.
|
||||||
|
func importReq(t *testing.T, csv string, fields map[string]string) (*http.Request, io.Reader) {
|
||||||
|
t.Helper()
|
||||||
|
body := &strings.Builder{}
|
||||||
|
mw := multipart.NewWriter(body)
|
||||||
|
for k, v := range fields {
|
||||||
|
_ = mw.WriteField(k, v)
|
||||||
|
}
|
||||||
|
fw, _ := mw.CreateFormFile("file", "a.csv")
|
||||||
|
fw.Write([]byte(csv))
|
||||||
|
mw.Close()
|
||||||
|
req := httptest.NewRequest("POST", "/api/tasks/1/import", strings.NewReader(body.String()))
|
||||||
|
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||||
|
req.SetPathValue("id", "1")
|
||||||
|
file, _, err := req.FormFile("file")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("form file: %v", err)
|
||||||
|
}
|
||||||
|
return req, file
|
||||||
|
}
|
||||||
|
|
||||||
|
const kerioCSV = "Name;FullName;Description;Enable;DataSource\n" +
|
||||||
|
"info;;InfoPass11;Yes;Internal\n"
|
||||||
|
|
||||||
|
func TestParseImportRowsUsesKerioParserWithDomain(t *testing.T) {
|
||||||
|
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio", "domain": "example.test"})
|
||||||
|
rows, err := parseImportRows(req, file)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" || rows[0].DstLogin != "info@example.test" {
|
||||||
|
t.Fatalf("kerio rows must carry the supplied domain on both sides, got %+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseImportRowsKerioRequiresDomain(t *testing.T) {
|
||||||
|
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio"})
|
||||||
|
if _, err := parseImportRows(req, file); err == nil {
|
||||||
|
t.Fatal("kerio import without a domain must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseImportRowsDefaultsToPlainFormat(t *testing.T) {
|
||||||
|
req, file := importReq(t, "a@x,p1,a@y,p2\n", nil)
|
||||||
|
rows, err := parseImportRows(req, file)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if len(rows) != 1 || rows[0].SrcLogin != "a@x" || rows[0].DstPass != "p2" {
|
||||||
|
t.Fatalf("no format field must keep the 4-column parser, got %+v", rows)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRunAccountIDs(t *testing.T) {
|
||||||
|
// empty body => nil (run all)
|
||||||
|
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
|
||||||
|
ids, err := parseRunAccountIDs(req)
|
||||||
|
if err != nil || ids != nil {
|
||||||
|
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// explicit selection
|
||||||
|
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
|
||||||
|
ids, err = parseRunAccountIDs(req)
|
||||||
|
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
|
||||||
|
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// malformed JSON => error
|
||||||
|
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
|
||||||
|
if _, err := parseRunAccountIDs(req); err == nil {
|
||||||
|
t.Fatal("malformed body must error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,9 +1,12 @@
|
|||||||
package httpapi
|
package httpapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/vasyansk/imap-copier/internal/scheduler"
|
||||||
"github.com/vasyansk/imap-copier/internal/store"
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -50,5 +53,95 @@ func (s *Server) handleGetTask(w http.ResponseWriter, r *http.Request) {
|
|||||||
for _, a := range accs {
|
for _, a := range accs {
|
||||||
views = append(views, accountDTO(a))
|
views = append(views, accountDTO(a))
|
||||||
}
|
}
|
||||||
writeJSON(w, http.StatusOK, map[string]any{"task": task, "accounts": views})
|
tv, err := s.taskViewFor(r.Context(), task)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{"task": tv, "accounts": views})
|
||||||
|
}
|
||||||
|
|
||||||
|
// taskView augments the stored task with the server-computed next scheduled run
|
||||||
|
// (RFC3339 UTC; null when the schedule is off, the task is running, or broken).
|
||||||
|
type taskView struct {
|
||||||
|
store.Task
|
||||||
|
NextRunAt *string `json:"next_run_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) taskViewFor(ctx context.Context, t store.Task) (taskView, error) {
|
||||||
|
if t.ScheduleIntervalSeconds <= 0 || t.Broken || t.Status == "running" || t.ScheduleAnchor == nil {
|
||||||
|
return taskView{Task: t}, nil
|
||||||
|
}
|
||||||
|
last, err := s.store.LastFinishedRunAt(ctx, t.ID)
|
||||||
|
if err != nil {
|
||||||
|
return taskView{}, err
|
||||||
|
}
|
||||||
|
n := scheduler.NextRun(time.Duration(t.ScheduleIntervalSeconds)*time.Second, *t.ScheduleAnchor, last)
|
||||||
|
str := n.UTC().Format(time.RFC3339)
|
||||||
|
return taskView{Task: t, NextRunAt: &str}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleSetSchedule enables/changes/disables a task's recurring schedule.
|
||||||
|
// Enabling (interval>0) requires every account to pass its connection tests.
|
||||||
|
func (s *Server) handleSetSchedule(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var body struct {
|
||||||
|
IntervalSeconds int64 `json:"interval_seconds"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||||
|
http.Error(w, "bad json", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if body.IntervalSeconds < 0 {
|
||||||
|
http.Error(w, "interval_seconds must be >= 0", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if body.IntervalSeconds > 0 {
|
||||||
|
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !allAccountsTested(accs) {
|
||||||
|
http.Error(w, "all accounts must pass connection tests before scheduling", http.StatusConflict)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := s.store.SetTaskSchedule(r.Context(), taskID, body.IntervalSeconds); err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.WriteHeader(http.StatusNoContent)
|
||||||
|
}
|
||||||
|
|
||||||
|
// allAccountsTested mirrors the orchestrator's run gate: non-empty and every
|
||||||
|
// account OK on both sides.
|
||||||
|
func allAccountsTested(accs []store.Account) bool {
|
||||||
|
if len(accs) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, a := range accs {
|
||||||
|
if a.TestSrcStatus != "ok" || a.TestDstStatus != "ok" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) handleListRuns(w http.ResponseWriter, r *http.Request) {
|
||||||
|
taskID, err := pathID(r, "id")
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, "bad id", http.StatusBadRequest)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
runs, err := s.store.ListRunsByTask(r.Context(), taskID)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, runs)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,13 +4,12 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
|
|
||||||
"github.com/emersion/go-imap/v2"
|
"github.com/emersion/go-imap/v2"
|
||||||
"github.com/emersion/go-imap/v2/imapclient"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// FolderMessageCount opens a folder read-only (EXAMINE) and returns how many
|
// FolderMessageCount opens a folder read-only (EXAMINE) and returns how many
|
||||||
// messages it holds — used to plan an accurate overall progress total before
|
// messages it holds — used to plan an accurate overall progress total before
|
||||||
// copying begins. It does not fetch any message bodies.
|
// copying begins. It does not fetch any message bodies.
|
||||||
func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
|
func FolderMessageCount(c *Client, folder string) (int64, error) {
|
||||||
sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
@@ -19,7 +18,7 @@ func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ListFolders returns the mailbox names visible on an already-connected, logged-in client.
|
// ListFolders returns the mailbox names visible on an already-connected, logged-in client.
|
||||||
func ListFolders(c *imapclient.Client) ([]string, error) {
|
func ListFolders(c *Client) ([]string, error) {
|
||||||
mboxes, err := c.List("", "*", nil).Collect()
|
mboxes, err := c.List("", "*", nil).Collect()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
|
|||||||
@@ -0,0 +1,54 @@
|
|||||||
|
package imapx
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/emersion/go-imap/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// metaBatches must tile the sequence 1..total into contiguous, non-overlapping
|
||||||
|
// windows of at most batchSize, covering every message exactly once. A single
|
||||||
|
// unbounded FETCH 1:* is what wedges large mailboxes; batching keeps each
|
||||||
|
// command short so the server stays responsive and ctx can be checked between
|
||||||
|
// windows.
|
||||||
|
func TestMetaBatches(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
total, size uint32
|
||||||
|
want []imap.SeqRange
|
||||||
|
}{
|
||||||
|
{0, 1000, nil},
|
||||||
|
{1, 1000, []imap.SeqRange{{Start: 1, Stop: 1}}},
|
||||||
|
{1000, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}}},
|
||||||
|
{1001, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}, {Start: 1001, Stop: 1001}}},
|
||||||
|
{2500, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}, {Start: 1001, Stop: 2000}, {Start: 2001, Stop: 2500}}},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
got := metaBatches(c.total, c.size)
|
||||||
|
if len(got) != len(c.want) {
|
||||||
|
t.Fatalf("total=%d size=%d: got %d windows %v, want %d %v", c.total, c.size, len(got), got, len(c.want), c.want)
|
||||||
|
}
|
||||||
|
for i := range got {
|
||||||
|
if got[i] != c.want[i] {
|
||||||
|
t.Fatalf("total=%d size=%d window %d: got %+v want %+v", c.total, c.size, i, got[i], c.want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every window must be within 1..total and the windows must be gap-free so no
|
||||||
|
// message is skipped or fetched twice.
|
||||||
|
func TestMetaBatchesCoverage(t *testing.T) {
|
||||||
|
const total, size = 4321, 1000
|
||||||
|
got := metaBatches(total, size)
|
||||||
|
if got[0].Start != 1 {
|
||||||
|
t.Fatalf("first window must start at 1, got %d", got[0].Start)
|
||||||
|
}
|
||||||
|
if last := got[len(got)-1]; last.Stop != total {
|
||||||
|
t.Fatalf("last window must stop at total=%d, got %d", total, last.Stop)
|
||||||
|
}
|
||||||
|
for i := 1; i < len(got); i++ {
|
||||||
|
if got[i].Start != got[i-1].Stop+1 {
|
||||||
|
t.Fatalf("gap/overlap between window %d (%+v) and %d (%+v)", i-1, got[i-1], i, got[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+249
-49
@@ -3,8 +3,13 @@ package imapx
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/emersion/go-imap/v2"
|
"github.com/emersion/go-imap/v2"
|
||||||
@@ -27,8 +32,42 @@ type CopyDeps struct {
|
|||||||
// folder's messages have been examined so far — so the UI shows movement
|
// folder's messages have been examined so far — so the UI shows movement
|
||||||
// while dedup decisions are made, before bodies start copying.
|
// while dedup decisions are made, before bodies start copying.
|
||||||
OnScan func(scanned, total int64)
|
OnScan func(scanned, total int64)
|
||||||
|
// OnError is called for each message-level error, with a message reference
|
||||||
|
// ("UID N: subject", empty when the envelope is unavailable) and the error
|
||||||
|
// text — so the orchestrator can persist individual errors for the
|
||||||
|
// per-account error modal. Folder-level errors are reported by the caller.
|
||||||
|
OnError func(ref, msg string)
|
||||||
|
// OnActivity is called repeatedly WHILE a single message body is streamed
|
||||||
|
// (each FETCH read chunk and each APPEND write chunk). Copying one large
|
||||||
|
// message can take longer than the orchestrator's stall timeout; without an
|
||||||
|
// in-body signal the watchdog can't tell a slow-but-live transfer from a
|
||||||
|
// wedged connection and cancels a healthy copy. May be nil.
|
||||||
|
OnActivity func()
|
||||||
|
// ReconnectSrc dials and logs in a FRESH source client, returning it. It is
|
||||||
|
// called when a body read times out (server under-delivered a literal),
|
||||||
|
// which leaves the current src connection desynced and unusable. CopyFolder
|
||||||
|
// swaps to the returned client, re-EXAMINEs the folder, and resumes. The
|
||||||
|
// implementation is expected to also update any external reference to the
|
||||||
|
// live src client (e.g. so a cancel path closes the right connection). If
|
||||||
|
// nil, a body-read timeout aborts the folder instead of recovering.
|
||||||
|
ReconnectSrc func() (*Client, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ErrBodyTimeout means a message body did not finish transferring within the
|
||||||
|
// idle deadline — the server stopped sending mid-literal. It is almost always a
|
||||||
|
// server announcing a BODY[] literal larger than the bytes it actually sends,
|
||||||
|
// which makes go-imap wait forever for bytes that never come. Distinct from a
|
||||||
|
// closed connection so the caller can skip just this one message and resume.
|
||||||
|
var ErrBodyTimeout = errors.New("message body read timed out")
|
||||||
|
|
||||||
|
// bodyIdleTimeout bounds how long a body read may go with NO bytes arriving
|
||||||
|
// before it is abandoned. It is generous enough for legitimately slow servers
|
||||||
|
// (even ~15 KB/s links keep bytes flowing far more often than this) yet well
|
||||||
|
// under the orchestrator's multi-minute stall watchdog, so an under-delivered
|
||||||
|
// literal is caught quickly and locally instead of stalling the whole account.
|
||||||
|
// A var (not const) so tests can shorten it.
|
||||||
|
var bodyIdleTimeout = 30 * time.Second
|
||||||
|
|
||||||
// CopyResult summarizes the outcome of one CopyFolder run.
|
// CopyResult summarizes the outcome of one CopyFolder run.
|
||||||
type CopyResult struct {
|
type CopyResult struct {
|
||||||
Copied int
|
Copied int
|
||||||
@@ -36,6 +75,43 @@ type CopyResult struct {
|
|||||||
Errors int
|
Errors int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// metaScanBatch bounds how many messages one Pass-1 metadata FETCH covers. A
|
||||||
|
// single unbounded FETCH 1:* over a large mailbox keeps one command open for
|
||||||
|
// the entire scan; under parallel load the server can stop responding and,
|
||||||
|
// since go-imap has no per-command deadline, the worker wedges forever. Short
|
||||||
|
// windows keep each command brief so the server stays responsive and ctx is
|
||||||
|
// checked between windows.
|
||||||
|
const metaScanBatch = 1000
|
||||||
|
|
||||||
|
// msgRef builds a human-readable reference for a message error: its UID plus
|
||||||
|
// subject when known, e.g. "UID 42: Invoice". Falls back to just the UID.
|
||||||
|
func msgRef(uid imap.UID, subject string) string {
|
||||||
|
if subject == "" {
|
||||||
|
return fmt.Sprintf("UID %d", uid)
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("UID %d: %s", uid, subject)
|
||||||
|
}
|
||||||
|
|
||||||
|
// metaBatches tiles 1..total into contiguous, non-overlapping windows of at
|
||||||
|
// most batchSize, covering every sequence number exactly once.
|
||||||
|
func metaBatches(total, batchSize uint32) []imap.SeqRange {
|
||||||
|
if total == 0 || batchSize == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var out []imap.SeqRange
|
||||||
|
for start := uint32(1); start <= total; start += batchSize {
|
||||||
|
stop := start + batchSize - 1
|
||||||
|
if stop > total {
|
||||||
|
stop = total
|
||||||
|
}
|
||||||
|
out = append(out, imap.SeqRange{Start: start, Stop: stop})
|
||||||
|
if stop == total {
|
||||||
|
break // guard against uint32 overflow when total is near max
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// CopyFolder streams messages from srcFolder on src to dstFolder on dst.
|
// CopyFolder streams messages from srcFolder on src to dstFolder on dst.
|
||||||
//
|
//
|
||||||
// The source folder is opened read-only (EXAMINE) and is never mutated:
|
// The source folder is opened read-only (EXAMINE) and is never mutated:
|
||||||
@@ -43,7 +119,7 @@ type CopyResult struct {
|
|||||||
// held in memory only for the duration of a single FETCH->APPEND and is
|
// held in memory only for the duration of a single FETCH->APPEND and is
|
||||||
// never written to disk. Messages already migrated (per deps.IsMigrated)
|
// never written to disk. Messages already migrated (per deps.IsMigrated)
|
||||||
// are skipped without re-fetching their bodies.
|
// are skipped without re-fetching their bodies.
|
||||||
func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
|
func CopyFolder(ctx context.Context, src, dst *Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
|
||||||
var res CopyResult
|
var res CopyResult
|
||||||
|
|
||||||
sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
||||||
@@ -67,48 +143,67 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
|
|||||||
type queued struct {
|
type queued struct {
|
||||||
uid imap.UID
|
uid imap.UID
|
||||||
key string
|
key string
|
||||||
|
subject string
|
||||||
flags []imap.Flag
|
flags []imap.Flag
|
||||||
internalDate time.Time
|
internalDate time.Time
|
||||||
}
|
}
|
||||||
var todo []queued
|
var todo []queued
|
||||||
metaSet := imap.SeqSet{imap.SeqRange{Start: 1, Stop: sel.NumMessages}}
|
|
||||||
fc := src.Fetch(metaSet, &imap.FetchOptions{
|
|
||||||
UID: true, Envelope: true, RFC822Size: true, Flags: true, InternalDate: true,
|
|
||||||
})
|
|
||||||
var scanned int64
|
var scanned int64
|
||||||
for {
|
reportErr := func(ref, msg string) {
|
||||||
if err := ctx.Err(); err != nil {
|
if deps.OnError != nil {
|
||||||
_ = fc.Close()
|
deps.OnError(ref, msg)
|
||||||
return res, err
|
|
||||||
}
|
|
||||||
msg := fc.Next()
|
|
||||||
if msg == nil {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
buf, err := msg.Collect()
|
|
||||||
if err != nil {
|
|
||||||
res.Errors++
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
scanned++
|
|
||||||
key := MessageKey(buf.Envelope, buf.RFC822Size)
|
|
||||||
already, err := deps.IsMigrated(key)
|
|
||||||
if err != nil {
|
|
||||||
res.Errors++
|
|
||||||
} else if already {
|
|
||||||
res.Skipped++
|
|
||||||
if deps.OnProgress != nil {
|
|
||||||
deps.OnProgress(res.Copied, res.Skipped)
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
todo = append(todo, queued{uid: buf.UID, key: key, flags: buf.Flags, internalDate: buf.InternalDate})
|
|
||||||
}
|
|
||||||
if deps.OnScan != nil {
|
|
||||||
deps.OnScan(scanned, total)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if err := fc.Close(); err != nil {
|
// Scan metadata in bounded windows instead of one FETCH 1:*, so each
|
||||||
return res, fmt.Errorf("fetch meta %q: %w", srcFolder, err)
|
// command is short (the server stays responsive) and ctx is checked on
|
||||||
|
// every window boundary — not just between messages of one giant command.
|
||||||
|
for _, win := range metaBatches(sel.NumMessages, metaScanBatch) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
fc := src.Fetch(imap.SeqSet{win}, &imap.FetchOptions{
|
||||||
|
UID: true, Envelope: true, RFC822Size: true, Flags: true, InternalDate: true,
|
||||||
|
})
|
||||||
|
for {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
_ = fc.Close()
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
msg := fc.Next()
|
||||||
|
if msg == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
buf, err := msg.Collect()
|
||||||
|
if err != nil {
|
||||||
|
res.Errors++
|
||||||
|
reportErr("", "read message metadata: "+err.Error())
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
scanned++
|
||||||
|
key := MessageKey(buf.Envelope, buf.RFC822Size)
|
||||||
|
subject := ""
|
||||||
|
if buf.Envelope != nil {
|
||||||
|
subject = buf.Envelope.Subject
|
||||||
|
}
|
||||||
|
already, err := deps.IsMigrated(key)
|
||||||
|
if err != nil {
|
||||||
|
res.Errors++
|
||||||
|
reportErr(msgRef(buf.UID, subject), "dedup lookup: "+err.Error())
|
||||||
|
} else if already {
|
||||||
|
res.Skipped++
|
||||||
|
if deps.OnProgress != nil {
|
||||||
|
deps.OnProgress(res.Copied, res.Skipped)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
todo = append(todo, queued{uid: buf.UID, key: key, subject: subject, flags: buf.Flags, internalDate: buf.InternalDate})
|
||||||
|
}
|
||||||
|
if deps.OnScan != nil {
|
||||||
|
deps.OnScan(scanned, total)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := fc.Close(); err != nil {
|
||||||
|
return res, fmt.Errorf("fetch meta %q: %w", srcFolder, err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Pass 2: fetch bodies for the queued (new) messages, one at a time.
|
// Pass 2: fetch bodies for the queued (new) messages, one at a time.
|
||||||
@@ -116,12 +211,39 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
|
|||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return res, err
|
return res, err
|
||||||
}
|
}
|
||||||
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate); err != nil {
|
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate, deps.OnActivity); err != nil {
|
||||||
res.Errors++
|
res.Errors++
|
||||||
|
reportErr(msgRef(q.uid, q.subject), "copy message: "+err.Error())
|
||||||
|
// A body-read timeout means the server under-delivered this message's
|
||||||
|
// literal; the src connection is now desynced. Mark the message
|
||||||
|
// migrated so this and future runs skip it (it is un-fetchable via a
|
||||||
|
// conforming client), then reconnect src and resume the folder with
|
||||||
|
// the remaining queued messages.
|
||||||
|
if errors.Is(err, ErrBodyTimeout) && deps.ReconnectSrc != nil {
|
||||||
|
if merr := deps.MarkMigrated(dstFolder, q.key); merr != nil {
|
||||||
|
reportErr(msgRef(q.uid, q.subject), "mark skipped: "+merr.Error())
|
||||||
|
}
|
||||||
|
newSrc, rerr := deps.ReconnectSrc()
|
||||||
|
if rerr != nil {
|
||||||
|
return res, fmt.Errorf("reconnect src after body timeout in %q: %w", srcFolder, rerr)
|
||||||
|
}
|
||||||
|
src = newSrc
|
||||||
|
if _, serr := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait(); serr != nil {
|
||||||
|
return res, fmt.Errorf("re-examine %q after reconnect: %w", srcFolder, serr)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// A closed/broken connection won't recover: every remaining APPEND
|
||||||
|
// would fail identically. Abort the folder instead of logging
|
||||||
|
// thousands of the same error; a re-run resumes via dedup.
|
||||||
|
if errors.Is(err, net.ErrClosed) || strings.Contains(err.Error(), "use of closed network connection") {
|
||||||
|
return res, fmt.Errorf("dst connection lost in %q: %w", dstFolder, err)
|
||||||
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if err := deps.MarkMigrated(dstFolder, q.key); err != nil {
|
if err := deps.MarkMigrated(dstFolder, q.key); err != nil {
|
||||||
res.Errors++
|
res.Errors++
|
||||||
|
reportErr(msgRef(q.uid, q.subject), "mark migrated: "+err.Error())
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
res.Copied++
|
res.Copied++
|
||||||
@@ -132,11 +254,63 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// slowMessage marks how long one message's FETCH or APPEND phase may take
|
||||||
|
// before it is logged as anomalous. Well below the orchestrator's 3-minute
|
||||||
|
// stall timeout, so a message that trips the watchdog always leaves a log line
|
||||||
|
// naming the phase (FETCH vs APPEND) and size — turning a silent stall into
|
||||||
|
// evidence of which side and which message is the culprit.
|
||||||
|
const slowMessage = 20 * time.Second
|
||||||
|
|
||||||
|
// deadlineReader arms an idle read deadline on the source socket before every
|
||||||
|
// read of a message body, so a server that stops sending mid-literal — having
|
||||||
|
// announced a larger BODY[] size than it actually delivers — trips the deadline
|
||||||
|
// instead of blocking go-imap forever waiting for bytes that never arrive. It
|
||||||
|
// also pings onActivity as bytes arrive, feeding the orchestrator's stall
|
||||||
|
// watchdog. The deadline is refreshed on each read, so it bounds IDLE time
|
||||||
|
// (no bytes) rather than total transfer time — a legitimately slow but steady
|
||||||
|
// download never trips it.
|
||||||
|
type deadlineReader struct {
|
||||||
|
c *Client
|
||||||
|
r io.Reader
|
||||||
|
idle time.Duration
|
||||||
|
on func()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (d deadlineReader) Read(p []byte) (int, error) {
|
||||||
|
_ = d.c.SetReadDeadline(time.Now().Add(d.idle))
|
||||||
|
n, err := d.r.Read(p)
|
||||||
|
if n > 0 && d.on != nil {
|
||||||
|
d.on()
|
||||||
|
}
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// touchWriter wraps the APPEND write stream and pings onActivity on every
|
||||||
|
// non-empty write, so a long upload keeps the stall watchdog fed byte-by-byte.
|
||||||
|
type touchWriter struct {
|
||||||
|
w io.Writer
|
||||||
|
on func()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t touchWriter) Write(p []byte) (int, error) {
|
||||||
|
n, err := t.w.Write(p)
|
||||||
|
if n > 0 && t.on != nil {
|
||||||
|
t.on()
|
||||||
|
}
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
// streamOne FETCHes BODY[] for one message and APPENDs it into dst without
|
// streamOne FETCHes BODY[] for one message and APPENDs it into dst without
|
||||||
// spooling to disk. The body is buffered in RAM only for the duration of
|
// spooling to disk. The body is buffered in RAM only for the duration of
|
||||||
// this single FETCH->APPEND round trip.
|
// this single FETCH->APPEND round trip. onActivity (may be nil) fires as bytes
|
||||||
func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time) error {
|
// move in either direction, feeding the orchestrator's stall watchdog.
|
||||||
|
//
|
||||||
|
// The body read is guarded by an idle deadline on src: if the server goes
|
||||||
|
// silent mid-literal, streamOne returns ErrBodyTimeout rather than hanging, so
|
||||||
|
// CopyFolder can skip the message and reconnect.
|
||||||
|
func streamOne(src, dst *Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time, onActivity func()) error {
|
||||||
bodySection := &imap.FetchItemBodySection{}
|
bodySection := &imap.FetchItemBodySection{}
|
||||||
|
fetchStart := time.Now()
|
||||||
fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{
|
fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{
|
||||||
BodySection: []*imap.FetchItemBodySection{bodySection},
|
BodySection: []*imap.FetchItemBodySection{bodySection},
|
||||||
})
|
})
|
||||||
@@ -147,35 +321,61 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
|
|||||||
return fmt.Errorf("no message for uid %v", uid)
|
return fmt.Errorf("no message for uid %v", uid)
|
||||||
}
|
}
|
||||||
var body []byte
|
var body []byte
|
||||||
|
var readErr error
|
||||||
for {
|
for {
|
||||||
item := msg.Next()
|
item := msg.Next()
|
||||||
if item == nil {
|
if item == nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
if d, ok := item.(imapclient.FetchItemDataBodySection); ok {
|
if d, ok := item.(imapclient.FetchItemDataBodySection); ok {
|
||||||
b, err := io.ReadAll(d.Literal)
|
body, readErr = io.ReadAll(deadlineReader{c: src, r: d.Literal, idle: bodyIdleTimeout, on: onActivity})
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
body = b
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Clear the deadline before any further I/O on src (fetchCmd.Close reads the
|
||||||
|
// command's completion off the same socket).
|
||||||
|
_ = src.SetReadDeadline(time.Time{})
|
||||||
|
if readErr != nil {
|
||||||
|
if errors.Is(readErr, os.ErrDeadlineExceeded) {
|
||||||
|
return fmt.Errorf("%w: uid %v (server sent fewer bytes than the announced literal)", ErrBodyTimeout, uid)
|
||||||
|
}
|
||||||
|
return readErr
|
||||||
|
}
|
||||||
if err := fetchCmd.Close(); err != nil {
|
if err := fetchCmd.Close(); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if body == nil {
|
if body == nil {
|
||||||
return fmt.Errorf("empty body uid %v", uid)
|
return fmt.Errorf("empty body uid %v", uid)
|
||||||
}
|
}
|
||||||
|
fetchDur := time.Since(fetchStart)
|
||||||
|
|
||||||
|
appendStart := time.Now()
|
||||||
appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate})
|
appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate})
|
||||||
if _, err := io.Copy(appendCmd, bytes.NewReader(body)); err != nil {
|
// Append acquires go-imap's per-client encoder mutex and holds it until
|
||||||
|
// Close() calls enc.end(). Close() MUST run on every path: if io.Copy
|
||||||
|
// fails mid-write (server stall, idle timeout), returning without Close()
|
||||||
|
// leaks the mutex and the NEXT Append on this client deadlocks forever on
|
||||||
|
// beginCommand. Close() is idempotent and always releases the lock.
|
||||||
|
_, copyErr := io.Copy(touchWriter{w: appendCmd, on: onActivity}, bytes.NewReader(body))
|
||||||
|
closeErr := appendCmd.Close()
|
||||||
|
if copyErr != nil {
|
||||||
|
return fmt.Errorf("append body uid %v: %w", uid, copyErr)
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
return closeErr
|
||||||
|
}
|
||||||
|
if _, err := appendCmd.Wait(); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := appendCmd.Close(); err != nil {
|
appendDur := time.Since(appendStart)
|
||||||
return err
|
|
||||||
|
// One message that individually eats a large slice of the stall budget is
|
||||||
|
// the prime suspect behind a "no progress" cancel; name it, its size, and
|
||||||
|
// which phase was slow so the culprit is visible in the logs.
|
||||||
|
if fetchDur > slowMessage || appendDur > slowMessage {
|
||||||
|
slog.Warn("slow message copy", "uid", uid, "bytes", len(body),
|
||||||
|
"fetch", fetchDur.Round(time.Millisecond), "append", appendDur.Round(time.Millisecond))
|
||||||
}
|
}
|
||||||
_, err := appendCmd.Wait()
|
return nil
|
||||||
return err
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8
|
// keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8
|
||||||
|
|||||||
@@ -169,6 +169,56 @@ func TestCopyFolderPreservesInternalDate(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCopyFolderReportsActivityDuringBody proves CopyFolder invokes OnActivity
|
||||||
|
// while a message body is being transferred (FETCH/APPEND), not only between
|
||||||
|
// messages. This is what keeps the orchestrator's stall watchdog from killing a
|
||||||
|
// single large-but-live message whose transfer legitimately exceeds the stall
|
||||||
|
// timeout: without an in-body activity signal, one slow message looks identical
|
||||||
|
// to a wedged connection.
|
||||||
|
func TestCopyFolderReportsActivityDuringBody(t *testing.T) {
|
||||||
|
ep := testEP(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
seedInbox(t, ep, "actsrc@localhost", "p", 1)
|
||||||
|
|
||||||
|
src, err := Connect(ctx, ep)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = src.Logout().Wait() }()
|
||||||
|
if err := src.Login("actsrc@localhost", "p").Wait(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dst, err := Connect(ctx, ep)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = dst.Logout().Wait() }()
|
||||||
|
if err := dst.Login("actdst@localhost", "p").Wait(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var activity int
|
||||||
|
deps := CopyDeps{
|
||||||
|
IsMigrated: func(string) (bool, error) { return false, nil },
|
||||||
|
MarkMigrated: func(_, _ string) error { return nil },
|
||||||
|
OnProgress: func(_, _ int) {},
|
||||||
|
OnActivity: func() { activity++ },
|
||||||
|
}
|
||||||
|
|
||||||
|
r, err := CopyFolder(ctx, src, dst, "INBOX", "INBOX", deps)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CopyFolder: %v", err)
|
||||||
|
}
|
||||||
|
if r.Copied != 1 {
|
||||||
|
t.Fatalf("copied=%d want 1", r.Copied)
|
||||||
|
}
|
||||||
|
if activity == 0 {
|
||||||
|
t.Fatal("OnActivity never called during body transfer")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped).
|
// Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped).
|
||||||
func TestCopyFolderIdempotent(t *testing.T) {
|
func TestCopyFolderIdempotent(t *testing.T) {
|
||||||
ep := testEP(t) // plain greenmail
|
ep := testEP(t) // plain greenmail
|
||||||
|
|||||||
+79
-10
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"crypto/tls"
|
"crypto/tls"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/emersion/go-imap/v2/imapclient"
|
"github.com/emersion/go-imap/v2/imapclient"
|
||||||
@@ -17,31 +18,99 @@ type Endpoint struct {
|
|||||||
|
|
||||||
func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) }
|
func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) }
|
||||||
|
|
||||||
func dialOnce(ep Endpoint) (*imapclient.Client, error) {
|
// Client wraps an imapclient.Client together with the raw network connection it
|
||||||
|
// runs over. The embedded *imapclient.Client provides the full IMAP API; the
|
||||||
|
// retained conn lets callers impose a read deadline on the socket for the
|
||||||
|
// duration of a body transfer.
|
||||||
|
//
|
||||||
|
// This defends against servers that announce a BODY[] literal LARGER than the
|
||||||
|
// bytes they actually send (a protocol violation observed on some webmail
|
||||||
|
// servers). go-imap reads a literal strictly by its announced size, so a short
|
||||||
|
// literal makes it block forever waiting for bytes that never arrive. A
|
||||||
|
// deadline around the body read turns that infinite hang into a timeout the
|
||||||
|
// copier can recover from.
|
||||||
|
type Client struct {
|
||||||
|
*imapclient.Client
|
||||||
|
conn net.Conn
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetReadDeadline sets (or, with the zero time, clears) a deadline on the
|
||||||
|
// underlying socket. Used to bound a single body read; always cleared again
|
||||||
|
// once the read completes so it never affects idle periods.
|
||||||
|
func (c *Client) SetReadDeadline(t time.Time) error {
|
||||||
|
return c.conn.SetReadDeadline(t)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dialTimeout bounds establishing the TCP connection.
|
||||||
|
const dialTimeout = 30 * time.Second
|
||||||
|
|
||||||
|
// dialOnce establishes one connection and returns a ready *Client. ctx bounds
|
||||||
|
// the TCP dial. We deliberately do NOT impose a socket-level read deadline:
|
||||||
|
// a blanket read deadline can't tell an idle connection (e.g. dst sitting idle
|
||||||
|
// during a long src scan) from one stuck mid-response, and would wrongly close
|
||||||
|
// idle connections. Stall detection is done at the orchestrator level via a
|
||||||
|
// progress watchdog; go-imap's own per-command timeouts bound active commands.
|
||||||
|
func dialOnce(ctx context.Context, ep Endpoint) (*Client, error) {
|
||||||
|
d := &net.Dialer{Timeout: dialTimeout}
|
||||||
|
raw, err := d.DialContext(ctx, "tcp", ep.addr())
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
switch ep.TLSMode {
|
switch ep.TLSMode {
|
||||||
case "ssl":
|
case "ssl":
|
||||||
return imapclient.DialTLS(ep.addr(), &imapclient.Options{
|
// NextProtos mirrors imapclient.DialTLS's ALPN advertisement.
|
||||||
TLSConfig: &tls.Config{ServerName: ep.Host},
|
tlsConn := tls.Client(raw, &tls.Config{ServerName: ep.Host, NextProtos: []string{"imap"}})
|
||||||
})
|
if err := tlsConn.HandshakeContext(ctx); err != nil {
|
||||||
|
_ = raw.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
c, err := waitGreeting(imapclient.New(tlsConn, nil))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &Client{Client: c, conn: tlsConn}, nil
|
||||||
case "starttls":
|
case "starttls":
|
||||||
return imapclient.DialStartTLS(ep.addr(), &imapclient.Options{
|
// Deadline goes on the raw TCP conn: it sits beneath the TLS layer that
|
||||||
TLSConfig: &tls.Config{ServerName: ep.Host},
|
// NewStartTLS negotiates, and a TCP read deadline still interrupts the
|
||||||
})
|
// TLS read above it.
|
||||||
|
opts := &imapclient.Options{TLSConfig: &tls.Config{ServerName: ep.Host}}
|
||||||
|
c, err := imapclient.NewStartTLS(raw, opts)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &Client{Client: c, conn: raw}, nil
|
||||||
case "plain":
|
case "plain":
|
||||||
return imapclient.DialInsecure(ep.addr(), nil)
|
c, err := waitGreeting(imapclient.New(raw, nil))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &Client{Client: c, conn: raw}, nil
|
||||||
default:
|
default:
|
||||||
|
_ = raw.Close()
|
||||||
return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode)
|
return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func Connect(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
|
// waitGreeting blocks for the server's initial greeting so a mute server is
|
||||||
|
// caught at connect time rather than at the first command. NewStartTLS already
|
||||||
|
// awaits the greeting during its STARTTLS upgrade.
|
||||||
|
func waitGreeting(c *imapclient.Client) (*imapclient.Client, error) {
|
||||||
|
if err := c.WaitGreeting(); err != nil {
|
||||||
|
_ = c.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return c, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Connect(ctx context.Context, ep Endpoint) (*Client, error) {
|
||||||
const attempts = 3
|
const attempts = 3
|
||||||
var lastErr error
|
var lastErr error
|
||||||
for i := 0; i < attempts; i++ {
|
for i := 0; i < attempts; i++ {
|
||||||
if err := ctx.Err(); err != nil {
|
if err := ctx.Err(); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
c, err := dialOnce(ep)
|
c, err := dialOnce(ctx, ep)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
package imapx
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// KeepaliveInterval is how often an otherwise-idle IMAP connection is pinged
|
||||||
|
// with NOOP so the server does not drop it.
|
||||||
|
//
|
||||||
|
// The destination connection sits completely idle for the entire duration of
|
||||||
|
// the source-side metadata scan (Pass 1 of CopyFolder), which on a large
|
||||||
|
// mailbox runs for many minutes across all folders. With no traffic, the
|
||||||
|
// server closes the idle connection; go-imap's reader then tears the client
|
||||||
|
// down, and every subsequent APPEND fails with "use of closed network
|
||||||
|
// connection" — aborting each folder and copying nothing. A periodic NOOP
|
||||||
|
// keeps the connection warm. 60s is well under the idle timeout of any common
|
||||||
|
// IMAP server.
|
||||||
|
const KeepaliveInterval = 60 * time.Second
|
||||||
|
|
||||||
|
// Keepalive pings c with a NOOP every interval until ctx is cancelled, keeping
|
||||||
|
// an idle connection from being dropped by the server. It is meant to run in
|
||||||
|
// its own goroutine.
|
||||||
|
//
|
||||||
|
// It is safe to run concurrently with other commands on c: go-imap serializes
|
||||||
|
// command submission and supports multiple in-flight commands over a single
|
||||||
|
// connection (RFC 9051 §5.5 pipelining). A NOOP issued while another command is
|
||||||
|
// in flight simply queues behind it and completes when the server responds.
|
||||||
|
//
|
||||||
|
// Keepalive returns when ctx is done or when a NOOP fails — a failed NOOP means
|
||||||
|
// the connection is already gone, so there is nothing left to keep alive.
|
||||||
|
func Keepalive(ctx context.Context, c *Client, interval time.Duration) {
|
||||||
|
t := time.NewTicker(interval)
|
||||||
|
defer t.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-t.C:
|
||||||
|
if err := c.Noop().Wait(); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
package imapx
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestKeepaliveReturnsOnContextCancel proves Keepalive is a well-behaved
|
||||||
|
// goroutine: it exits promptly when its context is cancelled instead of
|
||||||
|
// leaking.
|
||||||
|
func TestKeepaliveReturnsOnContextCancel(t *testing.T) {
|
||||||
|
ep := testEP(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
c, err := Connect(ctx, ep)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = c.Logout().Wait() }()
|
||||||
|
if err := c.Login("ka1@localhost", "p").Wait(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
kctx, cancel := context.WithCancel(ctx)
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { Keepalive(kctx, c, 10*time.Millisecond); close(done) }()
|
||||||
|
|
||||||
|
// Let a few NOOPs fire, then cancel and require a prompt return.
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
cancel()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("Keepalive did not return within 2s of context cancel")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestKeepaliveDoesNotDisruptCopy runs Keepalive on the destination connection
|
||||||
|
// at an aggressive interval while CopyFolder is APPENDing to it, proving the
|
||||||
|
// concurrent NOOPs do not corrupt in-flight commands (the real risk of pinging
|
||||||
|
// a connection that is also being used for real work).
|
||||||
|
func TestKeepaliveDoesNotDisruptCopy(t *testing.T) {
|
||||||
|
ep := testEP(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
const n = 8
|
||||||
|
seedInbox(t, ep, "kasrc@localhost", "p", n)
|
||||||
|
|
||||||
|
src, err := Connect(ctx, ep)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = src.Logout().Wait() }()
|
||||||
|
if err := src.Login("kasrc@localhost", "p").Wait(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
dst, err := Connect(ctx, ep)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() { _ = dst.Logout().Wait() }()
|
||||||
|
if err := dst.Login("kadst@localhost", "p").Wait(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
kctx, cancel := context.WithCancel(ctx)
|
||||||
|
defer cancel()
|
||||||
|
go Keepalive(kctx, dst, 1*time.Millisecond)
|
||||||
|
|
||||||
|
seen := map[string]bool{}
|
||||||
|
deps := CopyDeps{
|
||||||
|
IsMigrated: func(k string) (bool, error) { return seen[k], nil },
|
||||||
|
MarkMigrated: func(_, k string) error { seen[k] = true; return nil },
|
||||||
|
OnProgress: func(_, _ int) {},
|
||||||
|
}
|
||||||
|
|
||||||
|
r, err := CopyFolder(kctx, src, dst, "INBOX", "INBOX", deps)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CopyFolder with concurrent keepalive: %v", err)
|
||||||
|
}
|
||||||
|
if r.Copied != n {
|
||||||
|
t.Fatalf("copied=%d want %d", r.Copied, n)
|
||||||
|
}
|
||||||
|
if r.Errors != 0 {
|
||||||
|
t.Fatalf("errors=%d want 0", r.Errors)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,218 @@
|
|||||||
|
package imapx
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/emersion/go-imap/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// underflowServer is a minimal IMAP server that reproduces the amega.kz bug: it
|
||||||
|
// answers a BODY[] FETCH by announcing a literal LARGER than the bytes it then
|
||||||
|
// sends, and afterwards goes silent — exactly what makes go-imap's strict
|
||||||
|
// literal reader block forever. LOGIN/EXAMINE and the Pass-1 metadata FETCH are
|
||||||
|
// answered normally so a full CopyFolder can reach the poisoned message.
|
||||||
|
//
|
||||||
|
// It serves one connection per accept and keeps accepting, so a reconnect gets
|
||||||
|
// a fresh, well-behaved session (its second EXAMINE reports zero messages, so
|
||||||
|
// the resumed folder simply finishes).
|
||||||
|
type underflowServer struct {
|
||||||
|
ln net.Listener
|
||||||
|
mu sync.Mutex
|
||||||
|
accepts int // how many connections have been accepted
|
||||||
|
stop chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newUnderflowServer(t *testing.T) *underflowServer {
|
||||||
|
t.Helper()
|
||||||
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("listen: %v", err)
|
||||||
|
}
|
||||||
|
s := &underflowServer{ln: ln, stop: make(chan struct{})}
|
||||||
|
go s.serve()
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *underflowServer) addr() Endpoint {
|
||||||
|
a := s.ln.Addr().(*net.TCPAddr)
|
||||||
|
return Endpoint{Host: "127.0.0.1", Port: a.Port, TLSMode: "plain"}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *underflowServer) close() { close(s.stop); _ = s.ln.Close() }
|
||||||
|
|
||||||
|
func (s *underflowServer) serve() {
|
||||||
|
for {
|
||||||
|
conn, err := s.ln.Accept()
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.mu.Lock()
|
||||||
|
s.accepts++
|
||||||
|
first := s.accepts == 1
|
||||||
|
s.mu.Unlock()
|
||||||
|
go s.handle(conn, first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// handle drives one connection. On the FIRST connection the mailbox reports one
|
||||||
|
// message and its BODY[] fetch under-delivers; on any later connection (i.e.
|
||||||
|
// after a reconnect) the mailbox is empty so the resumed folder completes.
|
||||||
|
func (s *underflowServer) handle(conn net.Conn, first bool) {
|
||||||
|
defer func() { _ = conn.Close() }()
|
||||||
|
br := bufio.NewReader(conn)
|
||||||
|
fmt.Fprint(conn, "* OK IMAP4rev1 ready\r\n")
|
||||||
|
for {
|
||||||
|
line, err := br.ReadString('\n')
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
tag := fields[0]
|
||||||
|
up := strings.ToUpper(line)
|
||||||
|
switch {
|
||||||
|
case strings.Contains(up, "LOGIN"):
|
||||||
|
fmt.Fprintf(conn, "%s OK LOGIN completed\r\n", tag)
|
||||||
|
case strings.Contains(up, "EXAMINE"), strings.Contains(up, "SELECT"):
|
||||||
|
n := 0
|
||||||
|
if first {
|
||||||
|
n = 1
|
||||||
|
}
|
||||||
|
fmt.Fprintf(conn, "* %d EXISTS\r\n", n)
|
||||||
|
fmt.Fprint(conn, "* OK [UIDVALIDITY 1] ok\r\n")
|
||||||
|
fmt.Fprintf(conn, "%s OK [READ-ONLY] EXAMINE completed\r\n", tag)
|
||||||
|
case strings.Contains(up, "BODY["), strings.Contains(up, "BODY.PEEK"):
|
||||||
|
// Poison: announce 100000 bytes, send 10, then stall until shutdown.
|
||||||
|
fmt.Fprint(conn, "* 1 FETCH (UID 1 BODY[] {100000}\r\n")
|
||||||
|
fmt.Fprint(conn, "0123456789")
|
||||||
|
<-s.stop
|
||||||
|
return
|
||||||
|
case strings.Contains(up, "FETCH"):
|
||||||
|
// Pass-1 metadata fetch for the single message.
|
||||||
|
fmt.Fprint(conn, "* 1 FETCH (UID 1 RFC822.SIZE 100 FLAGS () "+
|
||||||
|
"INTERNALDATE \"01-Jan-2020 00:00:00 +0000\" "+
|
||||||
|
"ENVELOPE (\"Wed, 01 Jan 2020 00:00:00 +0000\" \"poison\" NIL NIL NIL NIL NIL NIL NIL \"<poison@x>\"))\r\n")
|
||||||
|
fmt.Fprintf(conn, "%s OK FETCH completed\r\n", tag)
|
||||||
|
case strings.Contains(up, "LOGOUT"):
|
||||||
|
fmt.Fprintf(conn, "* BYE\r\n%s OK LOGOUT completed\r\n", tag)
|
||||||
|
return
|
||||||
|
case strings.Contains(up, "CREATE"), strings.Contains(up, "NOOP"):
|
||||||
|
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
|
||||||
|
default:
|
||||||
|
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestStreamOneTimesOutOnLiteralUnderflow proves a server that announces a
|
||||||
|
// larger BODY[] literal than it sends makes streamOne return ErrBodyTimeout
|
||||||
|
// (bounded by bodyIdleTimeout) instead of hanging forever.
|
||||||
|
func TestStreamOneTimesOutOnLiteralUnderflow(t *testing.T) {
|
||||||
|
restore := shortenBodyIdle(200 * time.Millisecond)
|
||||||
|
defer restore()
|
||||||
|
|
||||||
|
srv := newUnderflowServer(t)
|
||||||
|
defer srv.close()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
src, err := Connect(ctx, srv.addr())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = src.Close() }()
|
||||||
|
if err := src.Login("u", "p").Wait(); err != nil {
|
||||||
|
t.Fatalf("login: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
done <- streamOne(src, src, "INBOX", imap.UID(1), nil, time.Time{}, nil)
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
if !errors.Is(err, ErrBodyTimeout) {
|
||||||
|
t.Fatalf("want ErrBodyTimeout, got %v", err)
|
||||||
|
}
|
||||||
|
case <-time.After(5 * time.Second):
|
||||||
|
t.Fatal("streamOne hung on literal underflow instead of timing out")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCopyFolderSkipsAndReconnectsOnUnderflow proves CopyFolder does not hang on
|
||||||
|
// a poisoned message: it marks the message migrated (so future runs skip it),
|
||||||
|
// invokes ReconnectSrc, and returns with the error counted — the folder is not
|
||||||
|
// wedged.
|
||||||
|
func TestCopyFolderSkipsAndReconnectsOnUnderflow(t *testing.T) {
|
||||||
|
restore := shortenBodyIdle(200 * time.Millisecond)
|
||||||
|
defer restore()
|
||||||
|
|
||||||
|
srv := newUnderflowServer(t)
|
||||||
|
defer srv.close()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
src, err := Connect(ctx, srv.addr())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = src.Close() }()
|
||||||
|
if err := src.Login("u", "p").Wait(); err != nil {
|
||||||
|
t.Fatalf("login: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var marked []string
|
||||||
|
var reconnected bool
|
||||||
|
deps := CopyDeps{
|
||||||
|
IsMigrated: func(string) (bool, error) { return false, nil },
|
||||||
|
MarkMigrated: func(_, k string) error { marked = append(marked, k); return nil },
|
||||||
|
OnProgress: func(_, _ int) {},
|
||||||
|
ReconnectSrc: func() (*Client, error) {
|
||||||
|
reconnected = true
|
||||||
|
nc, derr := Connect(ctx, srv.addr())
|
||||||
|
if derr != nil {
|
||||||
|
return nil, derr
|
||||||
|
}
|
||||||
|
if lerr := nc.Login("u", "p").Wait(); lerr != nil {
|
||||||
|
return nil, lerr
|
||||||
|
}
|
||||||
|
return nc, nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
done := make(chan CopyResult, 1)
|
||||||
|
go func() {
|
||||||
|
r, _ := CopyFolder(ctx, src, src, "INBOX", "INBOX", deps)
|
||||||
|
done <- r
|
||||||
|
}()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case r := <-done:
|
||||||
|
if r.Errors == 0 {
|
||||||
|
t.Fatalf("expected the poisoned message counted as an error, got %+v", r)
|
||||||
|
}
|
||||||
|
if !reconnected {
|
||||||
|
t.Fatal("ReconnectSrc was never called")
|
||||||
|
}
|
||||||
|
if len(marked) != 1 {
|
||||||
|
t.Fatalf("poisoned message should be marked migrated once, got %v", marked)
|
||||||
|
}
|
||||||
|
case <-time.After(8 * time.Second):
|
||||||
|
t.Fatal("CopyFolder hung on a poisoned message instead of skipping it")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func shortenBodyIdle(d time.Duration) func() {
|
||||||
|
old := bodyIdleTimeout
|
||||||
|
bodyIdleTimeout = d
|
||||||
|
return func() { bodyIdleTimeout = old }
|
||||||
|
}
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
package orchestrator
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestShouldBreak(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
trigger string
|
||||||
|
errs int64
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{"scheduled", 2, true}, // scheduled run with errors trips the breaker
|
||||||
|
{"scheduled", 0, false}, // scheduled run, clean — no break
|
||||||
|
{"manual", 5, false}, // manual run never trips the breaker
|
||||||
|
{"manual", 0, false},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if got := shouldBreak(c.trigger, c.errs); got != c.want {
|
||||||
|
t.Fatalf("shouldBreak(%q,%d)=%v want %v", c.trigger, c.errs, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/vasyansk/imap-copier/internal/crypto"
|
"github.com/vasyansk/imap-copier/internal/crypto"
|
||||||
@@ -15,6 +16,84 @@ import (
|
|||||||
|
|
||||||
var ErrNotTested = errors.New("accounts not fully tested")
|
var ErrNotTested = errors.New("accounts not fully tested")
|
||||||
var ErrAlreadyRunning = errors.New("task already running")
|
var ErrAlreadyRunning = errors.New("task already running")
|
||||||
|
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
|
||||||
|
var ErrNothingToResume = errors.New("no paused accounts to resume")
|
||||||
|
|
||||||
|
// maxAccountErrors caps how many individual error rows one account records per
|
||||||
|
// run, so a corrupt mailbox producing thousands of failures can't bloat the
|
||||||
|
// account_errors table. The cap'th row is a synthetic "further errors
|
||||||
|
// suppressed" note.
|
||||||
|
const maxAccountErrors = 500
|
||||||
|
|
||||||
|
// A running account that emits no scan/copy progress for stallTimeout is wedged
|
||||||
|
// (silent server mid-FETCH, stalled APPEND). The watchdog cancels it so the
|
||||||
|
// connections close and the worker unwinds instead of hanging forever. This
|
||||||
|
// replaces socket-level read deadlines, which can't tell an idle connection
|
||||||
|
// from a stuck one. The threshold is generous so slow-but-live runs aren't cut.
|
||||||
|
const (
|
||||||
|
stallTimeout = 3 * time.Minute
|
||||||
|
stallCheckInterval = 30 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
// folderPlan is one source folder scheduled for copy and its destination name.
|
||||||
|
type folderPlan struct {
|
||||||
|
src, dst string
|
||||||
|
total int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// planFolders decides which of an account's source folders to copy and where.
|
||||||
|
// Excluded source folders are dropped; a folder absent from mapping keeps its
|
||||||
|
// own name. Order follows the input folder list.
|
||||||
|
func planFolders(folders []string, mapping map[string]string, excluded []string) []folderPlan {
|
||||||
|
skip := make(map[string]struct{}, len(excluded))
|
||||||
|
for _, e := range excluded {
|
||||||
|
skip[e] = struct{}{}
|
||||||
|
}
|
||||||
|
plan := make([]folderPlan, 0, len(folders))
|
||||||
|
for _, f := range folders {
|
||||||
|
if _, ok := skip[f]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
df := f
|
||||||
|
if m, ok := mapping[f]; ok && m != "" {
|
||||||
|
df = m
|
||||||
|
}
|
||||||
|
plan = append(plan, folderPlan{src: f, dst: df})
|
||||||
|
}
|
||||||
|
return plan
|
||||||
|
}
|
||||||
|
|
||||||
|
// A run stops either on its own or because the operator intervened. Pausing and
|
||||||
|
// cancelling take the same path — stop the in-flight work — and differ only in
|
||||||
|
// the status left behind: paused accounts are what Resume picks up again.
|
||||||
|
type stopReason int32
|
||||||
|
|
||||||
|
const (
|
||||||
|
stopNone stopReason = iota
|
||||||
|
stopPaused
|
||||||
|
stopCancelled
|
||||||
|
)
|
||||||
|
|
||||||
|
func (r stopReason) accountStatus() string {
|
||||||
|
if r == stopPaused {
|
||||||
|
return "paused"
|
||||||
|
}
|
||||||
|
return "cancelled"
|
||||||
|
}
|
||||||
|
|
||||||
|
// runHandle is the live state of one task's run: the cancel that stops every
|
||||||
|
// account under it, plus why it was stopped.
|
||||||
|
type runHandle struct {
|
||||||
|
cancel context.CancelFunc
|
||||||
|
reason atomic.Int32
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *runHandle) stopWith(r stopReason) {
|
||||||
|
h.reason.CompareAndSwap(int32(stopNone), int32(r))
|
||||||
|
h.cancel()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *runHandle) stopReason() stopReason { return stopReason(h.reason.Load()) }
|
||||||
|
|
||||||
type Orchestrator struct {
|
type Orchestrator struct {
|
||||||
store *store.Store
|
store *store.Store
|
||||||
@@ -24,10 +103,66 @@ type Orchestrator struct {
|
|||||||
|
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy
|
cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy
|
||||||
|
runs map[int64]*runHandle // task_id -> live run
|
||||||
}
|
}
|
||||||
|
|
||||||
func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator {
|
func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator {
|
||||||
return &Orchestrator{store: s, hub: hub, encKey: encKey, concurrency: concurrency, cancels: map[int64]context.CancelFunc{}}
|
return &Orchestrator{
|
||||||
|
store: s, hub: hub, encKey: encKey, concurrency: concurrency,
|
||||||
|
cancels: map[int64]context.CancelFunc{},
|
||||||
|
runs: map[int64]*runHandle{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// PauseTask stops the task's live run, leaving every unfinished account
|
||||||
|
// "paused" so ResumeTask can pick them up. Returns false if nothing is running.
|
||||||
|
func (o *Orchestrator) PauseTask(taskID int64) bool { return o.stopRun(taskID, stopPaused) }
|
||||||
|
|
||||||
|
// CancelTask stops the task's live run and marks every unfinished account
|
||||||
|
// "cancelled". Returns false if nothing is running.
|
||||||
|
func (o *Orchestrator) CancelTask(taskID int64) bool { return o.stopRun(taskID, stopCancelled) }
|
||||||
|
|
||||||
|
func (o *Orchestrator) stopRun(taskID int64, reason stopReason) bool {
|
||||||
|
o.mu.Lock()
|
||||||
|
h, ok := o.runs[taskID]
|
||||||
|
o.mu.Unlock()
|
||||||
|
if !ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
h.stopWith(reason)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o *Orchestrator) registerRun(taskID int64, h *runHandle) {
|
||||||
|
o.mu.Lock()
|
||||||
|
o.runs[taskID] = h
|
||||||
|
o.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o *Orchestrator) unregisterRun(taskID int64) {
|
||||||
|
o.mu.Lock()
|
||||||
|
delete(o.runs, taskID)
|
||||||
|
o.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResumeTask restarts a paused task with exactly the accounts the pause left
|
||||||
|
// unfinished. Everything already copied is skipped by the migration journal, so
|
||||||
|
// each account continues where it stopped.
|
||||||
|
func (o *Orchestrator) ResumeTask(ctx context.Context, taskID int64) (int64, error) {
|
||||||
|
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
ids := make([]int64, 0, len(accs))
|
||||||
|
for _, a := range accs {
|
||||||
|
if a.Status == "paused" {
|
||||||
|
ids = append(ids, a.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return 0, ErrNothingToResume
|
||||||
|
}
|
||||||
|
return o.Run(ctx, taskID, "manual", ids)
|
||||||
}
|
}
|
||||||
|
|
||||||
// CancelAccount aborts the in-flight copy for one account, if it is running.
|
// CancelAccount aborts the in-flight copy for one account, if it is running.
|
||||||
@@ -54,6 +189,26 @@ func (o *Orchestrator) unregisterCancel(accountID int64) {
|
|||||||
o.mu.Unlock()
|
o.mu.Unlock()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// selectAccounts narrows accs to those whose ID is in ids, preserving input
|
||||||
|
// order. An empty or nil ids means "all accounts" — the scheduler and the
|
||||||
|
// unfiltered manual run rely on this.
|
||||||
|
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
|
||||||
|
if len(ids) == 0 {
|
||||||
|
return accs
|
||||||
|
}
|
||||||
|
want := make(map[int64]struct{}, len(ids))
|
||||||
|
for _, id := range ids {
|
||||||
|
want[id] = struct{}{}
|
||||||
|
}
|
||||||
|
out := make([]store.Account, 0, len(ids))
|
||||||
|
for _, a := range accs {
|
||||||
|
if _, ok := want[a.ID]; ok {
|
||||||
|
out = append(out, a)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func gateOK(accs []store.Account) bool {
|
func gateOK(accs []store.Account) bool {
|
||||||
if len(accs) == 0 {
|
if len(accs) == 0 {
|
||||||
return false
|
return false
|
||||||
@@ -122,7 +277,13 @@ func (o *Orchestrator) testSide(ctx context.Context, ep imapx.Endpoint, accID in
|
|||||||
}})
|
}})
|
||||||
}
|
}
|
||||||
|
|
||||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) {
|
// shouldBreak reports whether a completed run should trip the schedule breaker:
|
||||||
|
// only scheduled runs that ended with errors.
|
||||||
|
func shouldBreak(trigger string, totErr int64) bool {
|
||||||
|
return trigger == "scheduled" && totErr > 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
|
||||||
task, err := o.store.GetTask(ctx, taskID)
|
task, err := o.store.GetTask(ctx, taskID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
@@ -131,6 +292,12 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
|
accs = selectAccounts(accs, accountIDs)
|
||||||
|
// A non-empty request that matched nothing is a client error, distinct
|
||||||
|
// from "not tested".
|
||||||
|
if len(accountIDs) > 0 && len(accs) == 0 {
|
||||||
|
return 0, ErrNoAccountsSelected
|
||||||
|
}
|
||||||
if !gateOK(accs) {
|
if !gateOK(accs) {
|
||||||
return 0, ErrNotTested
|
return 0, ErrNotTested
|
||||||
}
|
}
|
||||||
@@ -146,23 +313,38 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) {
|
|||||||
_ = o.store.SetTaskStatus(ctx, taskID, "error")
|
_ = o.store.SetTaskStatus(ctx, taskID, "error")
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
runID, err := o.store.CreateRun(ctx, taskID)
|
runID, err := o.store.CreateRun(ctx, taskID, trigger)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
_ = o.store.SetTaskStatus(ctx, taskID, "error")
|
_ = o.store.SetTaskStatus(ctx, taskID, "error")
|
||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}})
|
o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}})
|
||||||
|
|
||||||
go o.runAll(context.WithoutCancel(ctx), task, runID, accs, srcEP, dstEP)
|
// dbCtx outlives the request so status/counter writes still land after a
|
||||||
|
// pause or cancel; runCtx is what Pause/Cancel actually stop, and every
|
||||||
|
// account's IMAP work hangs off it.
|
||||||
|
dbCtx := context.WithoutCancel(ctx)
|
||||||
|
runCtx, runCancel := context.WithCancel(dbCtx)
|
||||||
|
h := &runHandle{cancel: runCancel}
|
||||||
|
o.registerRun(taskID, h)
|
||||||
|
|
||||||
|
go o.runAll(dbCtx, runCtx, h, task, runID, accs, srcEP, dstEP, trigger)
|
||||||
return runID, nil
|
return runID, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint) {
|
func (o *Orchestrator) runAll(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
|
||||||
|
defer o.unregisterRun(task.ID)
|
||||||
|
defer h.cancel()
|
||||||
defer func() {
|
defer func() {
|
||||||
if r := recover(); r != nil {
|
if r := recover(); r != nil {
|
||||||
slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r)
|
slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r)
|
||||||
_ = o.store.FinishRun(ctx, runID, "error", 0, 0, 0)
|
_ = o.store.FinishRun(ctx, runID, "error", 0, 0, 0)
|
||||||
_ = o.store.SetTaskStatus(ctx, task.ID, "error")
|
_ = o.store.SetTaskStatus(ctx, task.ID, "error")
|
||||||
|
if trigger == "scheduled" {
|
||||||
|
_ = o.store.SetTaskBroken(ctx, task.ID)
|
||||||
|
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
|
||||||
|
Data: map[string]any{"task_id": task.ID, "errors": int64(0)}})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
@@ -173,7 +355,17 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
|||||||
sem := make(chan struct{}, o.concurrency)
|
sem := make(chan struct{}, o.concurrency)
|
||||||
var wg sync.WaitGroup
|
var wg sync.WaitGroup
|
||||||
|
|
||||||
for _, a := range accs {
|
for i, a := range accs {
|
||||||
|
// Stopped mid-queue: the accounts that never started are marked with the
|
||||||
|
// same status as the ones that were interrupted, so a pause leaves the
|
||||||
|
// whole remainder resumable and a cancel leaves it cancelled.
|
||||||
|
if runCtx.Err() != nil {
|
||||||
|
st := h.stopReason().accountStatus()
|
||||||
|
for _, rest := range accs[i:] {
|
||||||
|
_ = o.store.SetAccountStatus(ctx, rest.ID, st)
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
wg.Add(1)
|
wg.Add(1)
|
||||||
sem <- struct{}{}
|
sem <- struct{}{}
|
||||||
go func(a store.Account) {
|
go func(a store.Account) {
|
||||||
@@ -190,7 +382,7 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
|||||||
mu.Unlock()
|
mu.Unlock()
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
c, s, e := o.runAccount(ctx, task, runID, a, srcEP, dstEP)
|
c, s, e := o.runAccount(ctx, runCtx, h, task, runID, a, srcEP, dstEP)
|
||||||
mu.Lock()
|
mu.Lock()
|
||||||
totCopied += c
|
totCopied += c
|
||||||
totSkipped += s
|
totSkipped += s
|
||||||
@@ -200,29 +392,47 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
|||||||
}
|
}
|
||||||
wg.Wait()
|
wg.Wait()
|
||||||
|
|
||||||
|
reason := h.stopReason()
|
||||||
status := "done"
|
status := "done"
|
||||||
if totErr > 0 {
|
switch {
|
||||||
|
case reason == stopPaused:
|
||||||
|
status = "paused"
|
||||||
|
case reason == stopCancelled:
|
||||||
|
status = "cancelled"
|
||||||
|
case totErr > 0:
|
||||||
status = "done_with_errors"
|
status = "done_with_errors"
|
||||||
}
|
}
|
||||||
_ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr)
|
_ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr)
|
||||||
_ = o.store.SetTaskStatus(ctx, task.ID, status)
|
_ = o.store.SetTaskStatus(ctx, task.ID, status)
|
||||||
o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID,
|
o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID,
|
||||||
Data: map[string]any{"run_id": runID, "copied": totCopied, "skipped": totSkipped, "errors": totErr}})
|
Data: map[string]any{"run_id": runID, "status": status,
|
||||||
|
"copied": totCopied, "skipped": totSkipped, "errors": totErr}})
|
||||||
|
|
||||||
|
// An operator stopping the run is not a schedule failure, so leave the
|
||||||
|
// breaker alone even when the accounts that did run reported errors.
|
||||||
|
if reason == stopNone && shouldBreak(trigger, totErr) {
|
||||||
|
_ = o.store.SetTaskBroken(ctx, task.ID)
|
||||||
|
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
|
||||||
|
Data: map[string]any{"task_id": task.ID, "errors": totErr}})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
|
func (o *Orchestrator) runAccount(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
|
||||||
o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{
|
o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{
|
||||||
"account_id": a.ID,
|
"account_id": a.ID,
|
||||||
"src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port,
|
"src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port,
|
||||||
"dst_login": a.DstLogin, "dst_host": dstEP.Host, "dst_port": dstEP.Port,
|
"dst_login": a.DstLogin, "dst_host": dstEP.Host, "dst_port": dstEP.Port,
|
||||||
}})
|
}})
|
||||||
_ = o.store.SetAccountStatus(ctx, a.ID, "running")
|
_ = o.store.SetAccountStatus(ctx, a.ID, "running")
|
||||||
_ = o.store.SetAccountError(ctx, a.ID, "") // clear any error from a previous run
|
_ = o.store.SetAccountError(ctx, a.ID, "") // clear any error from a previous run
|
||||||
|
_ = o.store.ResetAccountCounters(ctx, a.ID) // start from zero; IncAccountCounters is additive
|
||||||
|
_ = o.store.ClearAccountErrors(ctx, a.ID) // drop last run's per-error rows
|
||||||
|
|
||||||
// Per-account cancellable context: IMAP work uses actx (so CancelAccount
|
// Per-account cancellable context: IMAP work uses actx, so both CancelAccount
|
||||||
// stops it); DB writes keep the parent ctx so status/counters persist even
|
// and a task-wide pause/cancel (which cancels runCtx) stop it. DB writes keep
|
||||||
// after cancellation. ctx is context.WithoutCancel from runAll.
|
// ctx — the uncancellable one from runAll — so status/counters persist even
|
||||||
actx, cancel := context.WithCancel(ctx)
|
// after cancellation.
|
||||||
|
actx, cancel := context.WithCancel(runCtx)
|
||||||
o.registerCancel(a.ID, cancel)
|
o.registerCancel(a.ID, cancel)
|
||||||
defer func() {
|
defer func() {
|
||||||
o.unregisterCancel(a.ID)
|
o.unregisterCancel(a.ID)
|
||||||
@@ -231,65 +441,124 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
|
|
||||||
srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc)
|
srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||||
}
|
}
|
||||||
dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc)
|
dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
src, err := imapx.Connect(actx, srcEP)
|
src, err := imapx.Connect(actx, srcEP)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||||
}
|
}
|
||||||
defer func() { _ = src.Logout().Wait() }()
|
|
||||||
if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
|
if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err)
|
_ = src.Logout().Wait()
|
||||||
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||||
}
|
}
|
||||||
|
// srcClient holds the LIVE source connection. A body-read timeout (server
|
||||||
|
// under-delivering a literal) forces a mid-run reconnect via reconnectSrc,
|
||||||
|
// which swaps this pointer. The cancel goroutine and the deferred logout
|
||||||
|
// below both read through it, so they always act on the current connection
|
||||||
|
// rather than a stale one that was already replaced and logged out.
|
||||||
|
var srcClient atomic.Pointer[imapx.Client]
|
||||||
|
srcClient.Store(src)
|
||||||
|
defer func() { _ = srcClient.Load().Logout().Wait() }()
|
||||||
|
|
||||||
dst, err := imapx.Connect(actx, dstEP)
|
dst, err := imapx.Connect(actx, dstEP)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||||
}
|
}
|
||||||
defer func() { _ = dst.Logout().Wait() }()
|
defer func() { _ = dst.Logout().Wait() }()
|
||||||
if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil {
|
if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// reconnectSrc dials and logs in a fresh source client, swaps it in as the
|
||||||
|
// live connection, and logs the old (desynced) one out. CopyFolder calls it
|
||||||
|
// to recover after a message body times out: the server left the connection
|
||||||
|
// mid-literal, so it can't be reused. Bound to actx, so a cancelled account
|
||||||
|
// fails the dial instead of reconnecting.
|
||||||
|
reconnectSrc := func() (*imapx.Client, error) {
|
||||||
|
nc, err := imapx.Connect(actx, srcEP)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := nc.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
|
||||||
|
_ = nc.Logout().Wait()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if old := srcClient.Swap(nc); old != nil {
|
||||||
|
_ = old.Logout().Wait()
|
||||||
|
}
|
||||||
|
slog.Info("reconnected src after message body timeout", "account", a.ID, "src_login", a.SrcLogin)
|
||||||
|
return nc, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// On cancel, close the connections so any in-flight network read (a slow
|
// On cancel, close the connections so any in-flight network read (a slow
|
||||||
// FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately.
|
// FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately.
|
||||||
go func() {
|
go func() {
|
||||||
<-actx.Done()
|
<-actx.Done()
|
||||||
_ = src.Close()
|
_ = srcClient.Load().Close()
|
||||||
_ = dst.Close()
|
_ = dst.Close()
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
// Keep the destination connection warm. It sits idle for the whole
|
||||||
|
// source-side metadata scan (Pass 1 of CopyFolder), which on a large
|
||||||
|
// mailbox runs for minutes; without traffic the server drops it and every
|
||||||
|
// subsequent APPEND fails with "use of closed network connection", copying
|
||||||
|
// nothing. Periodic NOOPs prevent that. Only dst needs it — src is
|
||||||
|
// continuously busy scanning/fetching. Bound to actx so it stops with the
|
||||||
|
// account.
|
||||||
|
go imapx.Keepalive(actx, dst, imapx.KeepaliveInterval)
|
||||||
|
|
||||||
|
// Progress watchdog: track the last time we saw scan/copy activity; if it
|
||||||
|
// goes quiet for stallTimeout, cancel the account so the connections close
|
||||||
|
// and this worker unwinds (it would otherwise block forever on a silent
|
||||||
|
// server). touch() is called on every progress signal below.
|
||||||
|
var lastActivity atomic.Int64
|
||||||
|
lastActivity.Store(time.Now().UnixNano())
|
||||||
|
touch := func() { lastActivity.Store(time.Now().UnixNano()) }
|
||||||
|
go func() {
|
||||||
|
t := time.NewTicker(stallCheckInterval)
|
||||||
|
defer t.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-actx.Done():
|
||||||
|
return
|
||||||
|
case <-t.C:
|
||||||
|
if time.Since(time.Unix(0, lastActivity.Load())) > stallTimeout {
|
||||||
|
slog.Warn("account stalled with no progress; cancelling",
|
||||||
|
"account", a.ID, "src_login", a.SrcLogin, "stall", stallTimeout)
|
||||||
|
_ = o.store.SetAccountError(ctx, a.ID, "stalled: no progress for "+stallTimeout.String()+", cancelled")
|
||||||
|
cancel()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
folders, err := imapx.ListFolders(src)
|
folders, err := imapx.ListFolders(src)
|
||||||
|
touch()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err)
|
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Planning pass: EXAMINE every folder up front to learn the total message
|
// Planning pass: decide folders from the account's own config, then EXAMINE
|
||||||
// count, so the UI can show an accurate overall bar / ETA before copying.
|
// each to learn message counts for the overall progress bar.
|
||||||
type folderPlan struct {
|
plan := planFolders(folders, a.FolderMapping, a.ExcludedFolders)
|
||||||
src, dst string
|
|
||||||
total int64
|
|
||||||
}
|
|
||||||
plan := make([]folderPlan, 0, len(folders))
|
|
||||||
var grandTotal int64
|
var grandTotal int64
|
||||||
for _, folder := range folders {
|
for i := range plan {
|
||||||
if actx.Err() != nil {
|
if actx.Err() != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
df := folder
|
n, cerr := imapx.FolderMessageCount(src, plan[i].src)
|
||||||
if m, ok := task.FolderMapping[folder]; ok {
|
|
||||||
df = m
|
|
||||||
}
|
|
||||||
n, cerr := imapx.FolderMessageCount(src, folder)
|
|
||||||
if cerr != nil && actx.Err() == nil {
|
if cerr != nil && actx.Err() == nil {
|
||||||
slog.Warn("count folder failed", "account", a.ID, "folder", folder, "err", cerr)
|
slog.Warn("count folder failed", "account", a.ID, "folder", plan[i].src, "err", cerr)
|
||||||
}
|
}
|
||||||
plan = append(plan, folderPlan{src: folder, dst: df, total: n})
|
plan[i].total = n
|
||||||
grandTotal += n
|
grandTotal += n
|
||||||
|
touch()
|
||||||
}
|
}
|
||||||
o.hub.Publish(wshub.Event{Type: "plan", TaskID: task.ID, Data: map[string]any{
|
o.hub.Publish(wshub.Event{Type: "plan", TaskID: task.ID, Data: map[string]any{
|
||||||
"account_id": a.ID, "src_login": a.SrcLogin, "folders": len(plan), "total": grandTotal,
|
"account_id": a.ID, "src_login": a.SrcLogin, "folders": len(plan), "total": grandTotal,
|
||||||
@@ -303,10 +572,34 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
var curFolder string
|
var curFolder string
|
||||||
var curTotal int64
|
var curTotal int64
|
||||||
var lastEmit, lastScanEmit time.Time
|
var lastEmit, lastScanEmit time.Time
|
||||||
|
// Persist individual errors for the per-account error modal, capped so a
|
||||||
|
// corrupt mailbox can't write unbounded rows. Runs on this goroutine, so
|
||||||
|
// the counter is race-free (same reasoning as the progress vars above).
|
||||||
|
var persistedErrs int
|
||||||
|
addErr := func(kind, folder, ref, msg string) {
|
||||||
|
persistedErrs++
|
||||||
|
switch {
|
||||||
|
case persistedErrs < maxAccountErrors:
|
||||||
|
_ = o.store.AddAccountError(ctx, a.ID, runID, kind, folder, ref, msg)
|
||||||
|
case persistedErrs == maxAccountErrors:
|
||||||
|
_ = o.store.AddAccountError(ctx, a.ID, runID, "account", "", "",
|
||||||
|
"too many errors — further errors suppressed")
|
||||||
|
}
|
||||||
|
}
|
||||||
deps := imapx.CopyDeps{
|
deps := imapx.CopyDeps{
|
||||||
IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) },
|
IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) },
|
||||||
MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) },
|
MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) },
|
||||||
|
OnError: func(ref, msg string) { addErr("message", curFolder, ref, msg) },
|
||||||
|
// Fires as bytes move within a single message's FETCH/APPEND, so the
|
||||||
|
// stall watchdog sees a large-but-live transfer as progress instead of
|
||||||
|
// cancelling it as a wedged connection.
|
||||||
|
OnActivity: touch,
|
||||||
|
// Recovers from a message body timeout (server under-delivering a
|
||||||
|
// literal) by swapping in a fresh source connection so the folder can
|
||||||
|
// resume with the remaining messages.
|
||||||
|
ReconnectSrc: reconnectSrc,
|
||||||
OnProgress: func(c, s int) {
|
OnProgress: func(c, s int) {
|
||||||
|
touch()
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
done := c + s
|
done := c + s
|
||||||
// throttle to ~3/sec per account, but always emit folder completion
|
// throttle to ~3/sec per account, but always emit folder completion
|
||||||
@@ -326,6 +619,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
},
|
},
|
||||||
// Fires after EXAMINE (before the long fetch) with the folder's message count.
|
// Fires after EXAMINE (before the long fetch) with the folder's message count.
|
||||||
OnFolder: func(srcFolder, dstFolder string, total int64) {
|
OnFolder: func(srcFolder, dstFolder string, total int64) {
|
||||||
|
touch()
|
||||||
curFolder, curTotal = srcFolder, total
|
curFolder, curTotal = srcFolder, total
|
||||||
o.hub.Publish(wshub.Event{Type: "folder", TaskID: task.ID, Data: map[string]any{
|
o.hub.Publish(wshub.Event{Type: "folder", TaskID: task.ID, Data: map[string]any{
|
||||||
"account_id": a.ID, "src_login": a.SrcLogin,
|
"account_id": a.ID, "src_login": a.SrcLogin,
|
||||||
@@ -335,6 +629,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
// Fires while streaming metadata (dedup scan) so the UI shows movement
|
// Fires while streaming metadata (dedup scan) so the UI shows movement
|
||||||
// before bodies start copying. Throttled to ~4/sec, always emit the last.
|
// before bodies start copying. Throttled to ~4/sec, always emit the last.
|
||||||
OnScan: func(scanned, total int64) {
|
OnScan: func(scanned, total int64) {
|
||||||
|
touch()
|
||||||
now := time.Now()
|
now := time.Now()
|
||||||
if now.Sub(lastScanEmit) < 250*time.Millisecond && scanned < total {
|
if now.Sub(lastScanEmit) < 250*time.Millisecond && scanned < total {
|
||||||
return
|
return
|
||||||
@@ -349,12 +644,13 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
if actx.Err() != nil {
|
if actx.Err() != nil {
|
||||||
break // cancelled — stop scheduling more folders
|
break // cancelled — stop scheduling more folders
|
||||||
}
|
}
|
||||||
res, err := imapx.CopyFolder(actx, src, dst, fp.src, fp.dst, deps)
|
res, err := imapx.CopyFolder(actx, srcClient.Load(), dst, fp.src, fp.dst, deps)
|
||||||
folderErr := int64(0)
|
folderErr := int64(0)
|
||||||
if err != nil && actx.Err() == nil {
|
if err != nil && actx.Err() == nil {
|
||||||
slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err)
|
slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err)
|
||||||
folderErr = 1
|
folderErr = 1
|
||||||
_ = o.store.SetAccountError(ctx, a.ID, "folder \""+fp.src+"\": "+err.Error())
|
_ = o.store.SetAccountError(ctx, a.ID, "folder \""+fp.src+"\": "+err.Error())
|
||||||
|
addErr("folder", fp.src, "", err.Error())
|
||||||
o.hub.Publish(wshub.Event{Type: "error", TaskID: task.ID, Data: map[string]any{
|
o.hub.Publish(wshub.Event{Type: "error", TaskID: task.ID, Data: map[string]any{
|
||||||
"account_id": a.ID, "src_login": a.SrcLogin, "folder": fp.src, "error": err.Error(),
|
"account_id": a.ID, "src_login": a.SrcLogin, "folder": fp.src, "error": err.Error(),
|
||||||
}})
|
}})
|
||||||
@@ -370,11 +666,18 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
}
|
}
|
||||||
|
|
||||||
if actx.Err() != nil {
|
if actx.Err() != nil {
|
||||||
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
|
// A task-wide pause leaves the account resumable; anything else (per-account
|
||||||
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: task.ID,
|
// cancel, stall watchdog, task-wide cancel) leaves it cancelled.
|
||||||
|
st := "cancelled"
|
||||||
|
if runCtx.Err() != nil {
|
||||||
|
st = h.stopReason().accountStatus()
|
||||||
|
}
|
||||||
|
_ = o.store.SetAccountStatus(ctx, a.ID, st)
|
||||||
|
o.hub.Publish(wshub.Event{Type: st, TaskID: task.ID,
|
||||||
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin,
|
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin,
|
||||||
"copied": copied, "skipped": skipped, "errors": errs}})
|
"copied": copied, "skipped": skipped, "errors": errs}})
|
||||||
slog.Info("account cancelled", "account", a.ID, "src_login", a.SrcLogin, "copied", copied, "skipped", skipped)
|
slog.Info("account stopped", "account", a.ID, "src_login", a.SrcLogin,
|
||||||
|
"status", st, "copied", copied, "skipped", skipped)
|
||||||
return copied, skipped, errs
|
return copied, skipped, errs
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -390,11 +693,16 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
|||||||
return copied, skipped, errs
|
return copied, skipped, errs
|
||||||
}
|
}
|
||||||
|
|
||||||
func (o *Orchestrator) accountFailed(ctx context.Context, taskID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
|
func (o *Orchestrator) accountFailed(ctx, runCtx context.Context, h *runHandle, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
|
||||||
// A cancellation surfacing as an error is a cancel, not a failure.
|
// A cancellation surfacing as an error is a stop, not a failure — and a
|
||||||
|
// task-wide pause must still leave the account resumable.
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
|
st := "cancelled"
|
||||||
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: taskID,
|
if runCtx.Err() != nil {
|
||||||
|
st = h.stopReason().accountStatus()
|
||||||
|
}
|
||||||
|
_ = o.store.SetAccountStatus(ctx, a.ID, st)
|
||||||
|
o.hub.Publish(wshub.Event{Type: st, TaskID: taskID,
|
||||||
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}})
|
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}})
|
||||||
return 0, 0, 0
|
return 0, 0, 0
|
||||||
}
|
}
|
||||||
@@ -404,7 +712,9 @@ func (o *Orchestrator) accountFailed(ctx context.Context, taskID int64, a store.
|
|||||||
}
|
}
|
||||||
slog.Error("account failed", "account", a.ID, "side", side, "login", login, "host", host, "port", port, "err", err)
|
slog.Error("account failed", "account", a.ID, "side", side, "login", login, "host", host, "port", port, "err", err)
|
||||||
_ = o.store.SetAccountStatus(ctx, a.ID, "error")
|
_ = o.store.SetAccountStatus(ctx, a.ID, "error")
|
||||||
_ = o.store.SetAccountError(ctx, a.ID, side+" "+login+"@"+host+": "+err.Error())
|
failMsg := side + " " + login + "@" + host + ": " + err.Error()
|
||||||
|
_ = o.store.SetAccountError(ctx, a.ID, failMsg)
|
||||||
|
_ = o.store.AddAccountError(ctx, a.ID, runID, "account", "", "", failMsg)
|
||||||
o.hub.Publish(wshub.Event{Type: "error", TaskID: taskID,
|
o.hub.Publish(wshub.Event{Type: "error", TaskID: taskID,
|
||||||
Data: map[string]any{"account_id": a.ID, "side": side, "login": login, "host": host, "port": port, "error": err.Error()}})
|
Data: map[string]any{"account_id": a.ID, "side": side, "login": login, "host": host, "port": port, "error": err.Error()}})
|
||||||
return 0, 0, 1
|
return 0, 0, 1
|
||||||
|
|||||||
@@ -22,3 +22,23 @@ func TestGateOK(t *testing.T) {
|
|||||||
t.Fatal("empty accounts must fail gate")
|
t.Fatal("empty accounts must fail gate")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSelectAccounts(t *testing.T) {
|
||||||
|
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
|
||||||
|
|
||||||
|
if got := selectAccounts(accs, nil); len(got) != 3 {
|
||||||
|
t.Fatalf("nil ids must return all, got %d", len(got))
|
||||||
|
}
|
||||||
|
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
|
||||||
|
t.Fatalf("empty ids must return all, got %d", len(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
got := selectAccounts(accs, []int64{3, 1})
|
||||||
|
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
|
||||||
|
t.Fatalf("must keep matching ids in input order, got %+v", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
|
||||||
|
t.Fatalf("unknown ids must yield empty, got %d", len(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package orchestrator
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func names(ps []folderPlan) []string {
|
||||||
|
out := make([]string, len(ps))
|
||||||
|
for i, p := range ps {
|
||||||
|
out[i] = p.src + "->" + p.dst
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPlanFolders(t *testing.T) {
|
||||||
|
folders := []string{"INBOX", "Спам", "Trash", "Sent"}
|
||||||
|
|
||||||
|
// Empty config: every folder syncs to its own name, in order.
|
||||||
|
got := planFolders(folders, nil, nil)
|
||||||
|
if len(got) != 4 || got[0].src != "INBOX" || got[0].dst != "INBOX" {
|
||||||
|
t.Fatalf("empty config: %v", names(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rename + exclusion.
|
||||||
|
got = planFolders(folders,
|
||||||
|
map[string]string{"Спам": "Spam"},
|
||||||
|
[]string{"Trash"})
|
||||||
|
if len(got) != 3 {
|
||||||
|
t.Fatalf("want 3 plans (Trash excluded), got %v", names(got))
|
||||||
|
}
|
||||||
|
for _, p := range got {
|
||||||
|
if p.src == "Trash" {
|
||||||
|
t.Fatalf("excluded folder present: %v", names(got))
|
||||||
|
}
|
||||||
|
if p.src == "Спам" && p.dst != "Spam" {
|
||||||
|
t.Fatalf("rename not applied: %v", names(got))
|
||||||
|
}
|
||||||
|
if p.src == "INBOX" && p.dst != "INBOX" {
|
||||||
|
t.Fatalf("unmapped folder should keep its name: %v", names(got))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
package orchestrator
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestStopReasonAccountStatus(t *testing.T) {
|
||||||
|
if got := stopPaused.accountStatus(); got != "paused" {
|
||||||
|
t.Fatalf("stopPaused = %q want paused", got)
|
||||||
|
}
|
||||||
|
if got := stopCancelled.accountStatus(); got != "cancelled" {
|
||||||
|
t.Fatalf("stopCancelled = %q want cancelled", got)
|
||||||
|
}
|
||||||
|
// A run stopped without an operator reason (per-account cancel, stall
|
||||||
|
// watchdog) must not look like a pause, or Resume would pick it up.
|
||||||
|
if got := stopNone.accountStatus(); got != "cancelled" {
|
||||||
|
t.Fatalf("stopNone = %q want cancelled", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The first stop wins: a cancel arriving after a pause must not downgrade the
|
||||||
|
// accounts a pause already promised to keep resumable, and vice versa.
|
||||||
|
func TestRunHandleFirstStopWins(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
first, later stopReason
|
||||||
|
}{
|
||||||
|
{"pause then cancel", stopPaused, stopCancelled},
|
||||||
|
{"cancel then pause", stopCancelled, stopPaused},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
_, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
h := &runHandle{cancel: cancel}
|
||||||
|
h.stopWith(tc.first)
|
||||||
|
h.stopWith(tc.later)
|
||||||
|
if got := h.stopReason(); got != tc.first {
|
||||||
|
t.Fatalf("reason = %v want %v", got, tc.first)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunHandleStopCancelsContext(t *testing.T) {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
h := &runHandle{cancel: cancel}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
t.Fatal("context cancelled before stop")
|
||||||
|
}
|
||||||
|
h.stopWith(stopPaused)
|
||||||
|
if ctx.Err() == nil {
|
||||||
|
t.Fatal("stop must cancel the run context")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Pause/Cancel report false for a task with no live run, which the HTTP layer
|
||||||
|
// turns into 409 instead of pretending it stopped something.
|
||||||
|
func TestStopRunWithoutLiveRun(t *testing.T) {
|
||||||
|
o := &Orchestrator{runs: map[int64]*runHandle{}}
|
||||||
|
if o.PauseTask(1) {
|
||||||
|
t.Fatal("PauseTask must report false with no live run")
|
||||||
|
}
|
||||||
|
if o.CancelTask(1) {
|
||||||
|
t.Fatal("CancelTask must report false with no live run")
|
||||||
|
}
|
||||||
|
|
||||||
|
_, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
h := &runHandle{cancel: cancel}
|
||||||
|
o.registerRun(1, h)
|
||||||
|
if !o.PauseTask(1) {
|
||||||
|
t.Fatal("PauseTask must report true for a live run")
|
||||||
|
}
|
||||||
|
if got := h.stopReason(); got != stopPaused {
|
||||||
|
t.Fatalf("reason = %v want stopPaused", got)
|
||||||
|
}
|
||||||
|
o.unregisterRun(1)
|
||||||
|
if o.CancelTask(1) {
|
||||||
|
t.Fatal("unregistered run must not be stoppable")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
// Package scheduler runs tasks on their recurring interval. A single goroutine
|
||||||
|
// polls the DB every pollInterval and triggers due tasks through the
|
||||||
|
// orchestrator. The DB is the source of truth, so the scheduler holds no state
|
||||||
|
// and is safe across restarts.
|
||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/vasyansk/imap-copier/internal/orchestrator"
|
||||||
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
const pollInterval = 30 * time.Second
|
||||||
|
|
||||||
|
// NextRun is when a task should next run: one interval after its last completed
|
||||||
|
// run, or one interval after the schedule anchor if it has never completed one.
|
||||||
|
func NextRun(interval time.Duration, anchor time.Time, lastFinished *time.Time) time.Time {
|
||||||
|
base := anchor
|
||||||
|
if lastFinished != nil && lastFinished.After(anchor) {
|
||||||
|
base = *lastFinished
|
||||||
|
}
|
||||||
|
return base.Add(interval)
|
||||||
|
}
|
||||||
|
|
||||||
|
// dueTaskIDs returns the IDs of tasks whose next run is at or before now.
|
||||||
|
func dueTaskIDs(tasks []store.SchedulableTask, now time.Time) []int64 {
|
||||||
|
var ids []int64
|
||||||
|
for _, t := range tasks {
|
||||||
|
next := NextRun(time.Duration(t.IntervalSeconds)*time.Second, t.Anchor, t.LastFinished)
|
||||||
|
if !now.Before(next) {
|
||||||
|
ids = append(ids, t.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
type Scheduler struct {
|
||||||
|
store *store.Store
|
||||||
|
orch *orchestrator.Orchestrator
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(st *store.Store, orch *orchestrator.Orchestrator) *Scheduler {
|
||||||
|
return &Scheduler{store: st, orch: orch}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start blocks, ticking until ctx is cancelled. Run it in a goroutine.
|
||||||
|
func (s *Scheduler) Start(ctx context.Context) {
|
||||||
|
ticker := time.NewTicker(pollInterval)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
s.tick(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Scheduler) tick(ctx context.Context) {
|
||||||
|
tasks, err := s.store.ListSchedulableTasks(ctx)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: list schedulable", "err", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, id := range dueTaskIDs(tasks, time.Now()) {
|
||||||
|
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
|
||||||
|
// ErrAlreadyRunning / ErrNotTested are expected races/edge cases, not fatal.
|
||||||
|
slog.Info("scheduler: run skipped", "task", id, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/vasyansk/imap-copier/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNextRun(t *testing.T) {
|
||||||
|
anchor := time.Date(2026, 7, 3, 10, 0, 0, 0, time.UTC)
|
||||||
|
fin := time.Date(2026, 7, 3, 11, 0, 0, 0, time.UTC)
|
||||||
|
|
||||||
|
// No finished run yet → anchor + interval.
|
||||||
|
if got := NextRun(time.Hour, anchor, nil); !got.Equal(anchor.Add(time.Hour)) {
|
||||||
|
t.Fatalf("first run: got %v", got)
|
||||||
|
}
|
||||||
|
// Finished run exists → last finished + interval (from completion, not start).
|
||||||
|
if got := NextRun(time.Hour, anchor, &fin); !got.Equal(fin.Add(time.Hour)) {
|
||||||
|
t.Fatalf("recurring: got %v", got)
|
||||||
|
}
|
||||||
|
// Re-enable: anchor is newer than a stale lastFinished from before re-enable →
|
||||||
|
// use anchor, not the stale lastFinished (which would fire immediately/in the past).
|
||||||
|
staleFin := anchor.Add(-2 * time.Hour)
|
||||||
|
if got := NextRun(time.Hour, anchor, &staleFin); !got.Equal(anchor.Add(time.Hour)) {
|
||||||
|
t.Fatalf("re-enable: got %v, want %v", got, anchor.Add(time.Hour))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDueTaskIDs(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 3, 12, 0, 0, 0, time.UTC)
|
||||||
|
anchorOld := now.Add(-2 * time.Hour) // enabled 2h ago
|
||||||
|
finRecent := now.Add(-30 * time.Minute)
|
||||||
|
|
||||||
|
tasks := []store.SchedulableTask{
|
||||||
|
{ID: 1, IntervalSeconds: 3600, Anchor: anchorOld, LastFinished: nil}, // due: anchor+1h < now
|
||||||
|
{ID: 2, IntervalSeconds: 3600, Anchor: anchorOld, LastFinished: &finRecent}, // not due: fin+1h > now
|
||||||
|
{ID: 3, IntervalSeconds: 21600, Anchor: anchorOld, LastFinished: nil}, // not due: anchor+6h > now
|
||||||
|
}
|
||||||
|
ids := dueTaskIDs(tasks, now)
|
||||||
|
if len(ids) != 1 || ids[0] != 1 {
|
||||||
|
t.Fatalf("due IDs = %v, want [1]", ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AccountError is one concrete error recorded during an account's run —
|
||||||
|
// folder-level, message-level, or account-level (connect/login).
|
||||||
|
type AccountError struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
AccountID int64 `json:"account_id"`
|
||||||
|
RunID int64 `json:"run_id"`
|
||||||
|
Kind string `json:"kind"` // folder | message | account
|
||||||
|
Folder string `json:"folder"`
|
||||||
|
MessageRef string `json:"message_ref"`
|
||||||
|
Error string `json:"error"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// AddAccountError appends one error row for an account's current run.
|
||||||
|
func (s *Store) AddAccountError(ctx context.Context, accountID, runID int64, kind, folder, ref, msg string) error {
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`INSERT INTO account_errors (account_id, run_id, kind, folder, message_ref, error)
|
||||||
|
VALUES ($1,$2,$3,$4,$5,$6)`,
|
||||||
|
accountID, runID, kind, folder, ref, msg)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClearAccountErrors removes an account's errors at the start of a run, so the
|
||||||
|
// list reflects only the current run (mirrors ResetAccountCounters).
|
||||||
|
func (s *Store) ClearAccountErrors(ctx context.Context, accountID int64) error {
|
||||||
|
_, err := s.Pool.Exec(ctx, `DELETE FROM account_errors WHERE account_id=$1`, accountID)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListAccountErrors returns an account's errors in insertion order, for the
|
||||||
|
// per-account error modal.
|
||||||
|
func (s *Store) ListAccountErrors(ctx context.Context, accountID int64) ([]AccountError, error) {
|
||||||
|
rows, err := s.Pool.Query(ctx,
|
||||||
|
`SELECT id, account_id, run_id, kind, folder, message_ref, error, created_at
|
||||||
|
FROM account_errors WHERE account_id=$1 ORDER BY id`, accountID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
out := []AccountError{}
|
||||||
|
for rows.Next() {
|
||||||
|
var e AccountError
|
||||||
|
var runID *int64
|
||||||
|
if err := rows.Scan(&e.ID, &e.AccountID, &runID, &e.Kind, &e.Folder,
|
||||||
|
&e.MessageRef, &e.Error, &e.CreatedAt); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if runID != nil {
|
||||||
|
e.RunID = *runID
|
||||||
|
}
|
||||||
|
out = append(out, e)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AddAccountError persists individual errors; ListAccountErrors returns them in
|
||||||
|
// insertion order; ClearAccountErrors wipes them for the next run.
|
||||||
|
func TestAccountErrorsAddListClear(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
runID, _ := s.CreateRun(ctx, taskID, "manual")
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
|
||||||
|
|
||||||
|
if err := s.AddAccountError(ctx, accID, runID, "folder", "INBOX", "", "examine failed"); err != nil {
|
||||||
|
t.Fatalf("add folder: %v", err)
|
||||||
|
}
|
||||||
|
if err := s.AddAccountError(ctx, accID, runID, "message", "INBOX", "UID 42: hi", "append rejected"); err != nil {
|
||||||
|
t.Fatalf("add message: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
errs, err := s.ListAccountErrors(ctx, accID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("list: %v", err)
|
||||||
|
}
|
||||||
|
if len(errs) != 2 {
|
||||||
|
t.Fatalf("len=%d want 2", len(errs))
|
||||||
|
}
|
||||||
|
if errs[0].Kind != "folder" || errs[0].Folder != "INBOX" || errs[0].Error != "examine failed" {
|
||||||
|
t.Fatalf("errs[0]=%+v", errs[0])
|
||||||
|
}
|
||||||
|
if errs[1].Kind != "message" || errs[1].MessageRef != "UID 42: hi" || errs[1].Error != "append rejected" {
|
||||||
|
t.Fatalf("errs[1]=%+v", errs[1])
|
||||||
|
}
|
||||||
|
if errs[0].ID >= errs[1].ID {
|
||||||
|
t.Fatalf("expected insertion order by id: %d then %d", errs[0].ID, errs[1].ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.ClearAccountErrors(ctx, accID); err != nil {
|
||||||
|
t.Fatalf("clear: %v", err)
|
||||||
|
}
|
||||||
|
errs, err = s.ListAccountErrors(ctx, accID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("list after clear: %v", err)
|
||||||
|
}
|
||||||
|
if len(errs) != 0 {
|
||||||
|
t.Fatalf("after clear len=%d want 0", len(errs))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Deleting an account cascades its errors (ON DELETE CASCADE).
|
||||||
|
func TestAccountErrorsCascadeOnAccountDelete(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
|
||||||
|
_ = s.AddAccountError(ctx, accID, 0, "account", "", "", "login failed")
|
||||||
|
|
||||||
|
if err := s.DeleteAccount(ctx, accID); err != nil {
|
||||||
|
t.Fatalf("delete: %v", err)
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if err := s.Pool.QueryRow(ctx, `SELECT count(*) FROM account_errors WHERE account_id=$1`, accID).Scan(&n); err != nil {
|
||||||
|
t.Fatalf("count: %v", err)
|
||||||
|
}
|
||||||
|
if n != 0 {
|
||||||
|
t.Fatalf("account_errors not cascaded: %d rows", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
+59
-15
@@ -6,19 +6,21 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type Account struct {
|
type Account struct {
|
||||||
ID int64
|
ID int64
|
||||||
TaskID int64
|
TaskID int64
|
||||||
SrcLogin string
|
SrcLogin string
|
||||||
SrcPassEnc string
|
SrcPassEnc string
|
||||||
DstLogin string
|
DstLogin string
|
||||||
DstPassEnc string
|
DstPassEnc string
|
||||||
TestSrcStatus string
|
TestSrcStatus string
|
||||||
TestDstStatus string
|
TestDstStatus string
|
||||||
Status string
|
Status string
|
||||||
Copied int64
|
Copied int64
|
||||||
Skipped int64
|
Skipped int64
|
||||||
Errors int64
|
Errors int64
|
||||||
LastError string
|
LastError string
|
||||||
|
FolderMapping map[string]string
|
||||||
|
ExcludedFolders []string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
|
func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
|
||||||
@@ -30,6 +32,21 @@ func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
|
|||||||
return id, err
|
return id, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// UpdateAccountCredentials replaces an account's logins and, when a new
|
||||||
|
// ciphertext is supplied, its passwords; a nil password keeps the stored one so
|
||||||
|
// the operator can fix only the side that failed. Both connection tests are
|
||||||
|
// reset to "unknown" because the previous verdicts no longer describe these
|
||||||
|
// credentials, which also forces a re-test before the account can run.
|
||||||
|
func (s *Store) UpdateAccountCredentials(ctx context.Context, id int64, srcLogin, dstLogin string, srcPassEnc, dstPassEnc *string) error {
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE accounts SET src_login=$2, dst_login=$3,
|
||||||
|
src_pass_enc=COALESCE($4, src_pass_enc), dst_pass_enc=COALESCE($5, dst_pass_enc),
|
||||||
|
test_src_status='unknown', test_dst_status='unknown', last_error=''
|
||||||
|
WHERE id=$1`,
|
||||||
|
id, srcLogin, dstLogin, srcPassEnc, dstPassEnc)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
// DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE).
|
// DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE).
|
||||||
func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
|
func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
|
||||||
_, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id)
|
_, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id)
|
||||||
@@ -39,7 +56,8 @@ func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
|
|||||||
func (s *Store) ListAccountsByTask(ctx context.Context, taskID int64) ([]Account, error) {
|
func (s *Store) ListAccountsByTask(ctx context.Context, taskID int64) ([]Account, error) {
|
||||||
rows, err := s.Pool.Query(ctx,
|
rows, err := s.Pool.Query(ctx,
|
||||||
`SELECT id, task_id, src_login, src_pass_enc, dst_login, dst_pass_enc,
|
`SELECT id, task_id, src_login, src_pass_enc, dst_login, dst_pass_enc,
|
||||||
test_src_status, test_dst_status, status, copied_count, skipped_count, error_count, last_error
|
test_src_status, test_dst_status, status, copied_count, skipped_count,
|
||||||
|
error_count, last_error, folder_mapping, excluded_folders
|
||||||
FROM accounts WHERE task_id=$1 ORDER BY id`, taskID)
|
FROM accounts WHERE task_id=$1 ORDER BY id`, taskID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -49,7 +67,8 @@ func (s *Store) ListAccountsByTask(ctx context.Context, taskID int64) ([]Account
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var a Account
|
var a Account
|
||||||
if err := rows.Scan(&a.ID, &a.TaskID, &a.SrcLogin, &a.SrcPassEnc, &a.DstLogin, &a.DstPassEnc,
|
if err := rows.Scan(&a.ID, &a.TaskID, &a.SrcLogin, &a.SrcPassEnc, &a.DstLogin, &a.DstPassEnc,
|
||||||
&a.TestSrcStatus, &a.TestDstStatus, &a.Status, &a.Copied, &a.Skipped, &a.Errors, &a.LastError); err != nil {
|
&a.TestSrcStatus, &a.TestDstStatus, &a.Status, &a.Copied, &a.Skipped, &a.Errors, &a.LastError,
|
||||||
|
&a.FolderMapping, &a.ExcludedFolders); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, a)
|
out = append(out, a)
|
||||||
@@ -79,6 +98,15 @@ func (s *Store) SetAccountStatus(ctx context.Context, id int64, status string) e
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ResetAccountCounters zeroes the per-account copied/skipped/error counts at the
|
||||||
|
// start of a run so a re-run reflects only the current run's totals instead of
|
||||||
|
// accumulating on top of the previous run (IncAccountCounters is additive).
|
||||||
|
func (s *Store) ResetAccountCounters(ctx context.Context, id int64) error {
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE accounts SET copied_count=0, skipped_count=0, error_count=0 WHERE id=$1`, id)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Store) IncAccountCounters(ctx context.Context, id, copied, skipped, errs int64) error {
|
func (s *Store) IncAccountCounters(ctx context.Context, id, copied, skipped, errs int64) error {
|
||||||
_, err := s.Pool.Exec(ctx,
|
_, err := s.Pool.Exec(ctx,
|
||||||
`UPDATE accounts SET copied_count=copied_count+$2,
|
`UPDATE accounts SET copied_count=copied_count+$2,
|
||||||
@@ -86,3 +114,19 @@ func (s *Store) IncAccountCounters(ctx context.Context, id, copied, skipped, err
|
|||||||
id, copied, skipped, errs)
|
id, copied, skipped, errs)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetAccountFolderMapping persists an account's per-folder rename map and the
|
||||||
|
// set of source folders to skip. nil is normalized to empty so JSONB stays
|
||||||
|
// '{}' / '[]' rather than null.
|
||||||
|
func (s *Store) SetAccountFolderMapping(ctx context.Context, id int64, mapping map[string]string, excluded []string) error {
|
||||||
|
if mapping == nil {
|
||||||
|
mapping = map[string]string{}
|
||||||
|
}
|
||||||
|
if excluded == nil {
|
||||||
|
excluded = []string{}
|
||||||
|
}
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE accounts SET folder_mapping=$2, excluded_folders=$3 WHERE id=$1`,
|
||||||
|
id, mapping, excluded)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|||||||
@@ -28,3 +28,108 @@ func TestMigratedIdempotency(t *testing.T) {
|
|||||||
t.Fatal("unknown key must be false")
|
t.Fatal("unknown key must be false")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A re-run must start from zero counters, not accumulate on top of the previous
|
||||||
|
// run's totals — otherwise a run that skips everything still shows the prior
|
||||||
|
// run's Copied count. ResetAccountCounters is called at the start of runAccount.
|
||||||
|
func TestResetAccountCounters(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
|
||||||
|
|
||||||
|
// First run: everything copied.
|
||||||
|
if err := s.IncAccountCounters(ctx, accID, 100, 0, 1); err != nil {
|
||||||
|
t.Fatalf("inc: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Second run starts: counters reset to zero.
|
||||||
|
if err := s.ResetAccountCounters(ctx, accID); err != nil {
|
||||||
|
t.Fatalf("reset: %v", err)
|
||||||
|
}
|
||||||
|
// Second run skips everything.
|
||||||
|
if err := s.IncAccountCounters(ctx, accID, 0, 100, 0); err != nil {
|
||||||
|
t.Fatalf("inc2: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
accs, err := s.ListAccountsByTask(ctx, taskID)
|
||||||
|
if err != nil || len(accs) != 1 {
|
||||||
|
t.Fatalf("list: %v len=%d", err, len(accs))
|
||||||
|
}
|
||||||
|
a := accs[0]
|
||||||
|
if a.Copied != 0 || a.Skipped != 100 || a.Errors != 0 {
|
||||||
|
t.Fatalf("after reset+rerun: copied=%d skipped=%d errors=%d want 0/100/0",
|
||||||
|
a.Copied, a.Skipped, a.Errors)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fixing an imported account's credentials must replace only what the operator
|
||||||
|
// supplied: a nil password keeps the stored ciphertext, and both test verdicts
|
||||||
|
// go back to unknown because they described the old credentials.
|
||||||
|
func TestUpdateAccountCredentials(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "oldsrc", DstLogin: "u2", DstPassEnc: "olddst"})
|
||||||
|
_ = s.SetAccountTestStatus(ctx, accID, "src", "fail")
|
||||||
|
_ = s.SetAccountTestStatus(ctx, accID, "dst", "ok")
|
||||||
|
_ = s.SetAccountError(ctx, accID, "authentication failed")
|
||||||
|
|
||||||
|
newSrc := "newsrc"
|
||||||
|
if err := s.UpdateAccountCredentials(ctx, accID, "u@src.example", "u@dst.example", &newSrc, nil); err != nil {
|
||||||
|
t.Fatalf("update: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
accs, _ := s.ListAccountsByTask(ctx, taskID)
|
||||||
|
if len(accs) != 1 {
|
||||||
|
t.Fatalf("len=%d want 1", len(accs))
|
||||||
|
}
|
||||||
|
a := accs[0]
|
||||||
|
if a.SrcLogin != "u@src.example" || a.DstLogin != "u@dst.example" {
|
||||||
|
t.Fatalf("logins not updated: %q / %q", a.SrcLogin, a.DstLogin)
|
||||||
|
}
|
||||||
|
if a.SrcPassEnc != "newsrc" {
|
||||||
|
t.Fatalf("src password not updated: %q", a.SrcPassEnc)
|
||||||
|
}
|
||||||
|
if a.DstPassEnc != "olddst" {
|
||||||
|
t.Fatalf("nil password must keep the stored one, got %q", a.DstPassEnc)
|
||||||
|
}
|
||||||
|
if a.TestSrcStatus != "unknown" || a.TestDstStatus != "unknown" {
|
||||||
|
t.Fatalf("test statuses not reset: %q / %q", a.TestSrcStatus, a.TestDstStatus)
|
||||||
|
}
|
||||||
|
if a.LastError != "" {
|
||||||
|
t.Fatalf("last_error not cleared: %q", a.LastError)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSetAccountFolderMapping(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
|
||||||
|
|
||||||
|
// Fresh account defaults: empty map, empty exclusions (not nil after scan).
|
||||||
|
accs, _ := s.ListAccountsByTask(ctx, taskID)
|
||||||
|
if len(accs) != 1 || len(accs[0].FolderMapping) != 0 || len(accs[0].ExcludedFolders) != 0 {
|
||||||
|
t.Fatalf("defaults: map=%v excl=%v", accs[0].FolderMapping, accs[0].ExcludedFolders)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := s.SetAccountFolderMapping(ctx, accID,
|
||||||
|
map[string]string{"Спам": "Spam"}, []string{"Trash"}); err != nil {
|
||||||
|
t.Fatalf("set: %v", err)
|
||||||
|
}
|
||||||
|
accs, _ = s.ListAccountsByTask(ctx, taskID)
|
||||||
|
a := accs[0]
|
||||||
|
if a.FolderMapping["Спам"] != "Spam" {
|
||||||
|
t.Fatalf("mapping not persisted: %v", a.FolderMapping)
|
||||||
|
}
|
||||||
|
if len(a.ExcludedFolders) != 1 || a.ExcludedFolders[0] != "Trash" {
|
||||||
|
t.Fatalf("excluded not persisted: %v", a.ExcludedFolders)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -18,6 +18,41 @@ func TestUpdateEndpoint(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDeleteEndpoint(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
id, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a.com", Port: 993, TLSMode: "ssl"})
|
||||||
|
if err := s.DeleteEndpoint(ctx, id); err != nil {
|
||||||
|
t.Fatalf("delete: %v", err)
|
||||||
|
}
|
||||||
|
eps, _ := s.ListEndpoints(ctx)
|
||||||
|
if len(eps) != 0 {
|
||||||
|
t.Fatalf("endpoint not deleted: %d remain", len(eps))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeleteEndpointUsedByTaskRefused(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
ep1, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "s", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
if _, err := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2}); err != nil {
|
||||||
|
t.Fatalf("create task: %v", err)
|
||||||
|
}
|
||||||
|
for _, id := range []int64{ep1, ep2} {
|
||||||
|
n, err := s.CountTasksUsingEndpoint(ctx, id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("count: %v", err)
|
||||||
|
}
|
||||||
|
if n != 1 {
|
||||||
|
t.Fatalf("count for ep %d = %d, want 1", id, n)
|
||||||
|
}
|
||||||
|
if err := s.DeleteEndpoint(ctx, id); err == nil {
|
||||||
|
t.Fatalf("delete of referenced endpoint %d succeeded, want FK violation", id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDeleteAccountCascadesJournal(t *testing.T) {
|
func TestDeleteAccountCascadesJournal(t *testing.T) {
|
||||||
s := testStore(t)
|
s := testStore(t)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
@@ -49,7 +84,7 @@ func TestDeleteTaskCascades(t *testing.T) {
|
|||||||
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
|
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2})
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2})
|
||||||
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "v", DstPassEnc: "y"})
|
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "v", DstPassEnc: "y"})
|
||||||
_, _ = s.CreateRun(ctx, taskID)
|
_, _ = s.CreateRun(ctx, taskID, "manual")
|
||||||
|
|
||||||
if err := s.DeleteTask(ctx, taskID); err != nil {
|
if err := s.DeleteTask(ctx, taskID); err != nil {
|
||||||
t.Fatalf("delete task: %v", err)
|
t.Fatalf("delete task: %v", err)
|
||||||
|
|||||||
@@ -26,6 +26,23 @@ func (s *Store) UpdateEndpoint(ctx context.Context, e Endpoint) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CountTasksUsingEndpoint reports how many tasks reference the endpoint on
|
||||||
|
// either side, so a delete can be refused with a meaningful message instead of
|
||||||
|
// surfacing a raw foreign-key violation.
|
||||||
|
func (s *Store) CountTasksUsingEndpoint(ctx context.Context, id int64) (int, error) {
|
||||||
|
var n int
|
||||||
|
err := s.Pool.QueryRow(ctx,
|
||||||
|
`SELECT count(*) FROM tasks WHERE src_endpoint_id=$1 OR dst_endpoint_id=$1`, id).Scan(&n)
|
||||||
|
return n, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteEndpoint removes an endpoint. Tasks reference endpoints without ON
|
||||||
|
// DELETE CASCADE, so Postgres rejects the delete while any task still uses it.
|
||||||
|
func (s *Store) DeleteEndpoint(ctx context.Context, id int64) error {
|
||||||
|
_, err := s.Pool.Exec(ctx, `DELETE FROM endpoints WHERE id=$1`, id)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) {
|
func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) {
|
||||||
var e Endpoint
|
var e Endpoint
|
||||||
err := s.Pool.QueryRow(ctx,
|
err := s.Pool.QueryRow(ctx,
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import "context"
|
|||||||
// process died mid-run (crash, container restart). A fresh process has no
|
// process died mid-run (crash, container restart). A fresh process has no
|
||||||
// in-flight goroutines, so any persisted "running" is stale and would otherwise
|
// in-flight goroutines, so any persisted "running" is stale and would otherwise
|
||||||
// wedge the task (the run-guard refuses to start, and accounts can't be edited).
|
// wedge the task (the run-guard refuses to start, and accounts can't be edited).
|
||||||
|
// It also closes any run rows left stuck in "running" (finished_at NULL), which
|
||||||
|
// would otherwise surface as perpetually "running" in the run-log modal.
|
||||||
// Returns how many task and account rows were reset.
|
// Returns how many task and account rows were reset.
|
||||||
func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, accounts int64, err error) {
|
func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, accounts int64, err error) {
|
||||||
ct, err := s.Pool.Exec(ctx, `UPDATE accounts SET status='idle' WHERE status='running'`)
|
ct, err := s.Pool.Exec(ctx, `UPDATE accounts SET status='idle' WHERE status='running'`)
|
||||||
@@ -18,6 +20,11 @@ func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, account
|
|||||||
return 0, accounts, err
|
return 0, accounts, err
|
||||||
}
|
}
|
||||||
tasks = ct.RowsAffected()
|
tasks = ct.RowsAffected()
|
||||||
|
_, err = s.Pool.Exec(ctx,
|
||||||
|
`UPDATE runs SET status='error', finished_at=now() WHERE finished_at IS NULL`)
|
||||||
|
if err != nil {
|
||||||
|
return tasks, accounts, err
|
||||||
|
}
|
||||||
return tasks, accounts, nil
|
return tasks, accounts, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ func TestResetRunningOnStartup(t *testing.T) {
|
|||||||
// simulate a crash mid-run
|
// simulate a crash mid-run
|
||||||
_ = s.SetTaskStatus(ctx, taskID, "running")
|
_ = s.SetTaskStatus(ctx, taskID, "running")
|
||||||
_ = s.SetAccountStatus(ctx, accID, "running")
|
_ = s.SetAccountStatus(ctx, accID, "running")
|
||||||
|
runID, _ := s.CreateRun(ctx, taskID, "manual") // phantom run: never finished
|
||||||
|
|
||||||
tn, an, err := s.ResetRunningOnStartup(ctx)
|
tn, an, err := s.ResetRunningOnStartup(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -31,6 +32,19 @@ func TestResetRunningOnStartup(t *testing.T) {
|
|||||||
if accs[0].Status == "running" {
|
if accs[0].Status == "running" {
|
||||||
t.Fatal("account still running after reset")
|
t.Fatal("account still running after reset")
|
||||||
}
|
}
|
||||||
|
runs, _ := s.ListRunsByTask(ctx, taskID)
|
||||||
|
var found bool
|
||||||
|
for _, r := range runs {
|
||||||
|
if r.ID == runID {
|
||||||
|
found = true
|
||||||
|
if r.Status == "running" || r.FinishedAt == nil {
|
||||||
|
t.Fatalf("phantom run %d still running: status=%s finished_at=%v", runID, r.Status, r.FinishedAt)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("run %d not found", runID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestClearStuckAccountAndReconcile(t *testing.T) {
|
func TestClearStuckAccountAndReconcile(t *testing.T) {
|
||||||
|
|||||||
+48
-9
@@ -1,20 +1,26 @@
|
|||||||
package store
|
package store
|
||||||
|
|
||||||
import "context"
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
type Run struct {
|
type Run struct {
|
||||||
ID int64
|
ID int64 `json:"id"`
|
||||||
TaskID int64
|
TaskID int64 `json:"task_id"`
|
||||||
Status string
|
Status string `json:"status"`
|
||||||
TotalCopied int64
|
StartedAt time.Time `json:"started_at"`
|
||||||
TotalSkipped int64
|
FinishedAt *time.Time `json:"finished_at"`
|
||||||
TotalErrors int64
|
TotalCopied int64 `json:"total_copied"`
|
||||||
|
TotalSkipped int64 `json:"total_skipped"`
|
||||||
|
TotalErrors int64 `json:"total_errors"`
|
||||||
|
Trigger string `json:"trigger"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) CreateRun(ctx context.Context, taskID int64) (int64, error) {
|
func (s *Store) CreateRun(ctx context.Context, taskID int64, trigger string) (int64, error) {
|
||||||
var id int64
|
var id int64
|
||||||
err := s.Pool.QueryRow(ctx,
|
err := s.Pool.QueryRow(ctx,
|
||||||
`INSERT INTO runs (task_id) VALUES ($1) RETURNING id`, taskID).Scan(&id)
|
`INSERT INTO runs (task_id, trigger) VALUES ($1,$2) RETURNING id`, taskID, trigger).Scan(&id)
|
||||||
return id, err
|
return id, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -25,3 +31,36 @@ func (s *Store) FinishRun(ctx context.Context, id int64, status string, copied,
|
|||||||
id, status, copied, skipped, errs)
|
id, status, copied, skipped, errs)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ListRunsByTask returns a task's runs, newest first, for the run-log modal.
|
||||||
|
func (s *Store) ListRunsByTask(ctx context.Context, taskID int64) ([]Run, error) {
|
||||||
|
rows, err := s.Pool.Query(ctx,
|
||||||
|
`SELECT id, task_id, started_at, finished_at, status,
|
||||||
|
total_copied, total_skipped, total_errors, trigger
|
||||||
|
FROM runs WHERE task_id=$1 ORDER BY id DESC`, taskID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
out := []Run{}
|
||||||
|
for rows.Next() {
|
||||||
|
var r Run
|
||||||
|
if err := rows.Scan(&r.ID, &r.TaskID, &r.StartedAt, &r.FinishedAt, &r.Status,
|
||||||
|
&r.TotalCopied, &r.TotalSkipped, &r.TotalErrors, &r.Trigger); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, r)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// LastFinishedRunAt returns the most recent finished run's timestamp, or nil if
|
||||||
|
// the task has never completed a run — the baseline for the next scheduled run.
|
||||||
|
// The same max(finished_at) rule is also inlined in ListSchedulableTasks's
|
||||||
|
// subquery — keep the two in sync if this changes.
|
||||||
|
func (s *Store) LastFinishedRunAt(ctx context.Context, taskID int64) (*time.Time, error) {
|
||||||
|
var t *time.Time
|
||||||
|
err := s.Pool.QueryRow(ctx,
|
||||||
|
`SELECT max(finished_at) FROM runs WHERE task_id=$1 AND finished_at IS NOT NULL`, taskID).Scan(&t)
|
||||||
|
return t, err
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,81 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTaskScheduleAndBroken(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
|
||||||
|
// Defaults: no schedule, not broken.
|
||||||
|
tk, _ := s.GetTask(ctx, taskID)
|
||||||
|
if tk.ScheduleIntervalSeconds != 0 || tk.Broken || tk.ScheduleAnchor != nil {
|
||||||
|
t.Fatalf("defaults: interval=%d broken=%v anchor=%v", tk.ScheduleIntervalSeconds, tk.Broken, tk.ScheduleAnchor)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enable → interval set, anchor stamped, appears in schedulable list.
|
||||||
|
if err := s.SetTaskSchedule(ctx, taskID, 3600); err != nil {
|
||||||
|
t.Fatalf("set schedule: %v", err)
|
||||||
|
}
|
||||||
|
tk, _ = s.GetTask(ctx, taskID)
|
||||||
|
if tk.ScheduleIntervalSeconds != 3600 || tk.ScheduleAnchor == nil {
|
||||||
|
t.Fatalf("after enable: interval=%d anchor=%v", tk.ScheduleIntervalSeconds, tk.ScheduleAnchor)
|
||||||
|
}
|
||||||
|
sch, _ := s.ListSchedulableTasks(ctx)
|
||||||
|
if len(sch) != 1 || sch[0].ID != taskID || sch[0].IntervalSeconds != 3600 || sch[0].LastFinished != nil {
|
||||||
|
t.Fatalf("schedulable: %+v", sch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Breaker → broken true, interval 0, drops out of schedulable list.
|
||||||
|
if err := s.SetTaskBroken(ctx, taskID); err != nil {
|
||||||
|
t.Fatalf("set broken: %v", err)
|
||||||
|
}
|
||||||
|
tk, _ = s.GetTask(ctx, taskID)
|
||||||
|
if !tk.Broken || tk.ScheduleIntervalSeconds != 0 {
|
||||||
|
t.Fatalf("after break: broken=%v interval=%d", tk.Broken, tk.ScheduleIntervalSeconds)
|
||||||
|
}
|
||||||
|
if sch, _ := s.ListSchedulableTasks(ctx); len(sch) != 0 {
|
||||||
|
t.Fatalf("broken task still schedulable: %+v", sch)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-enable clears broken.
|
||||||
|
_ = s.SetTaskSchedule(ctx, taskID, 21600)
|
||||||
|
if tk, _ = s.GetTask(ctx, taskID); tk.Broken {
|
||||||
|
t.Fatalf("re-enable did not clear broken")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestListRunsByTaskWithTrigger(t *testing.T) {
|
||||||
|
s := testStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||||
|
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||||
|
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||||
|
|
||||||
|
r1, _ := s.CreateRun(ctx, taskID, "manual")
|
||||||
|
_ = s.FinishRun(ctx, r1, "done", 5, 1, 0)
|
||||||
|
_, _ = s.CreateRun(ctx, taskID, "scheduled") // still running (no finish)
|
||||||
|
|
||||||
|
runs, err := s.ListRunsByTask(ctx, taskID)
|
||||||
|
if err != nil || len(runs) != 2 {
|
||||||
|
t.Fatalf("list runs: %v len=%d", err, len(runs))
|
||||||
|
}
|
||||||
|
// newest first
|
||||||
|
if runs[0].Trigger != "scheduled" || runs[0].FinishedAt != nil {
|
||||||
|
t.Fatalf("run[0]: %+v", runs[0])
|
||||||
|
}
|
||||||
|
if runs[1].Trigger != "manual" || runs[1].FinishedAt == nil || runs[1].TotalCopied != 5 {
|
||||||
|
t.Fatalf("run[1]: %+v", runs[1])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Last finished-at reflects the finished manual run.
|
||||||
|
lf, _ := s.LastFinishedRunAt(ctx, taskID)
|
||||||
|
if lf == nil {
|
||||||
|
t.Fatalf("LastFinishedRunAt nil, want the finished run's time")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -17,7 +17,7 @@ func testStore(t *testing.T) *Store {
|
|||||||
}
|
}
|
||||||
t.Cleanup(func() {
|
t.Cleanup(func() {
|
||||||
s.Pool.Exec(context.Background(),
|
s.Pool.Exec(context.Background(),
|
||||||
`TRUNCATE endpoints, tasks, accounts, runs, migrated_messages RESTART IDENTITY CASCADE`)
|
`TRUNCATE endpoints, tasks, accounts, runs, migrated_messages, account_errors RESTART IDENTITY CASCADE`)
|
||||||
s.Pool.Close()
|
s.Pool.Close()
|
||||||
})
|
})
|
||||||
return s
|
return s
|
||||||
|
|||||||
+75
-11
@@ -1,14 +1,20 @@
|
|||||||
package store
|
package store
|
||||||
|
|
||||||
import "context"
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
type Task struct {
|
type Task struct {
|
||||||
ID int64 `json:"id"`
|
ID int64 `json:"id"`
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
SrcEndpointID int64 `json:"src_endpoint_id"`
|
SrcEndpointID int64 `json:"src_endpoint_id"`
|
||||||
DstEndpointID int64 `json:"dst_endpoint_id"`
|
DstEndpointID int64 `json:"dst_endpoint_id"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
FolderMapping map[string]string `json:"folder_mapping"`
|
FolderMapping map[string]string `json:"folder_mapping"`
|
||||||
|
ScheduleIntervalSeconds int64 `json:"schedule_interval_seconds"`
|
||||||
|
ScheduleAnchor *time.Time `json:"-"`
|
||||||
|
Broken bool `json:"broken"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) {
|
func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) {
|
||||||
@@ -26,9 +32,11 @@ func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) {
|
|||||||
func (s *Store) GetTask(ctx context.Context, id int64) (Task, error) {
|
func (s *Store) GetTask(ctx context.Context, id int64) (Task, error) {
|
||||||
var t Task
|
var t Task
|
||||||
err := s.Pool.QueryRow(ctx,
|
err := s.Pool.QueryRow(ctx,
|
||||||
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping
|
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping,
|
||||||
|
schedule_interval_seconds, schedule_anchor, broken
|
||||||
FROM tasks WHERE id=$1`, id).
|
FROM tasks WHERE id=$1`, id).
|
||||||
Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping)
|
Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping,
|
||||||
|
&t.ScheduleIntervalSeconds, &t.ScheduleAnchor, &t.Broken)
|
||||||
return t, err
|
return t, err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -40,7 +48,8 @@ func (s *Store) DeleteTask(ctx context.Context, id int64) error {
|
|||||||
|
|
||||||
func (s *Store) ListTasks(ctx context.Context) ([]Task, error) {
|
func (s *Store) ListTasks(ctx context.Context) ([]Task, error) {
|
||||||
rows, err := s.Pool.Query(ctx,
|
rows, err := s.Pool.Query(ctx,
|
||||||
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping
|
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping,
|
||||||
|
schedule_interval_seconds, schedule_anchor, broken
|
||||||
FROM tasks ORDER BY id DESC`)
|
FROM tasks ORDER BY id DESC`)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -49,7 +58,8 @@ func (s *Store) ListTasks(ctx context.Context) ([]Task, error) {
|
|||||||
out := []Task{}
|
out := []Task{}
|
||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var t Task
|
var t Task
|
||||||
if err := rows.Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping); err != nil {
|
if err := rows.Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping,
|
||||||
|
&t.ScheduleIntervalSeconds, &t.ScheduleAnchor, &t.Broken); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, t)
|
out = append(out, t)
|
||||||
@@ -80,3 +90,57 @@ func (s *Store) TryMarkTaskRunning(ctx context.Context, id int64) (bool, error)
|
|||||||
}
|
}
|
||||||
return ct.RowsAffected() == 1, nil
|
return ct.RowsAffected() == 1, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SetTaskSchedule sets the recurrence interval (0 = off), stamps the anchor when
|
||||||
|
// enabling (the first run is one interval after this), and clears broken — any
|
||||||
|
// explicit schedule change un-breaks the task.
|
||||||
|
func (s *Store) SetTaskSchedule(ctx context.Context, id, intervalSeconds int64) error {
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE tasks SET schedule_interval_seconds=$2,
|
||||||
|
schedule_anchor = CASE WHEN $2 > 0 THEN now() ELSE schedule_anchor END,
|
||||||
|
broken = false
|
||||||
|
WHERE id=$1`, id, intervalSeconds)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetTaskBroken trips the schedule breaker: disables the schedule and flags the
|
||||||
|
// task for operator attention.
|
||||||
|
func (s *Store) SetTaskBroken(ctx context.Context, id int64) error {
|
||||||
|
_, err := s.Pool.Exec(ctx,
|
||||||
|
`UPDATE tasks SET broken=true, schedule_interval_seconds=0 WHERE id=$1`, id)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// SchedulableTask is the per-tick decision input for the scheduler.
|
||||||
|
type SchedulableTask struct {
|
||||||
|
ID int64
|
||||||
|
IntervalSeconds int64
|
||||||
|
Anchor time.Time
|
||||||
|
LastFinished *time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListSchedulableTasks returns tasks eligible to auto-run: schedule on, not
|
||||||
|
// broken, neither running nor paused — each joined with its last finished run
|
||||||
|
// time. A paused task waits for the operator to resume it; auto-starting a full
|
||||||
|
// run behind their back would defeat the pause.
|
||||||
|
func (s *Store) ListSchedulableTasks(ctx context.Context) ([]SchedulableTask, error) {
|
||||||
|
rows, err := s.Pool.Query(ctx,
|
||||||
|
`SELECT t.id, t.schedule_interval_seconds, t.schedule_anchor,
|
||||||
|
(SELECT max(finished_at) FROM runs r WHERE r.task_id=t.id AND r.finished_at IS NOT NULL)
|
||||||
|
FROM tasks t
|
||||||
|
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken
|
||||||
|
AND t.status <> 'running' AND t.status <> 'paused'`)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
out := []SchedulableTask{}
|
||||||
|
for rows.Next() {
|
||||||
|
var st SchedulableTask
|
||||||
|
if err := rows.Scan(&st.ID, &st.IntervalSeconds, &st.Anchor, &st.LastFinished); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, st)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,2 @@
|
|||||||
|
ALTER TABLE accounts DROP COLUMN excluded_folders;
|
||||||
|
ALTER TABLE accounts DROP COLUMN folder_mapping;
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
ALTER TABLE accounts ADD COLUMN folder_mapping JSONB NOT NULL DEFAULT '{}';
|
||||||
|
ALTER TABLE accounts ADD COLUMN excluded_folders JSONB NOT NULL DEFAULT '[]';
|
||||||
|
|
||||||
|
-- Carry the existing task-wide mapping onto its accounts so behavior is preserved.
|
||||||
|
UPDATE accounts a SET folder_mapping = t.folder_mapping
|
||||||
|
FROM tasks t WHERE a.task_id = t.id AND t.folder_mapping <> '{}'::jsonb;
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
ALTER TABLE runs DROP COLUMN trigger;
|
||||||
|
ALTER TABLE tasks DROP COLUMN broken;
|
||||||
|
ALTER TABLE tasks DROP COLUMN schedule_anchor;
|
||||||
|
ALTER TABLE tasks DROP COLUMN schedule_interval_seconds;
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
ALTER TABLE tasks ADD COLUMN schedule_interval_seconds INT NOT NULL DEFAULT 0;
|
||||||
|
ALTER TABLE tasks ADD COLUMN schedule_anchor TIMESTAMPTZ;
|
||||||
|
ALTER TABLE tasks ADD COLUMN broken BOOLEAN NOT NULL DEFAULT false;
|
||||||
|
ALTER TABLE runs ADD COLUMN trigger TEXT NOT NULL DEFAULT 'manual';
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
DROP TABLE account_errors;
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
CREATE TABLE account_errors (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
account_id BIGINT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||||
|
run_id BIGINT,
|
||||||
|
kind TEXT NOT NULL CHECK (kind IN ('folder','message','account')),
|
||||||
|
folder TEXT NOT NULL DEFAULT '',
|
||||||
|
message_ref TEXT NOT NULL DEFAULT '',
|
||||||
|
error TEXT NOT NULL,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX account_errors_account_id_idx ON account_errors (account_id);
|
||||||
+94
-9
@@ -158,7 +158,8 @@ wait_test_ok() {
|
|||||||
wait_test_ok
|
wait_test_ok
|
||||||
|
|
||||||
wait_run_done() {
|
wait_run_done() {
|
||||||
for ((i = 1; i <= 60; i++)); do
|
# Generous: the resume scenario re-scans and copies thousands of messages.
|
||||||
|
for ((i = 1; i <= 600; i++)); do
|
||||||
local status
|
local status
|
||||||
status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
|
status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
|
||||||
if [[ "$status" == "done" ]]; then
|
if [[ "$status" == "done" ]]; then
|
||||||
@@ -185,16 +186,100 @@ log "POST /run (second run, expect idempotency)"
|
|||||||
api POST "/api/tasks/${TASK_ID}/run" >/dev/null
|
api POST "/api/tasks/${TASK_ID}/run" >/dev/null
|
||||||
wait_run_done
|
wait_run_done
|
||||||
|
|
||||||
|
# Counters are reset at the start of every run, so run 2's row shows run 2
|
||||||
|
# alone: it must copy nothing and skip what run 1 already migrated.
|
||||||
RES2=$(api GET "/api/tasks/${TASK_ID}")
|
RES2=$(api GET "/api/tasks/${TASK_ID}")
|
||||||
RUN2_COPIED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].copied')
|
RUN2_COPIED=$(echo "$RES2" | jq -r '.accounts[0].copied')
|
||||||
RUN2_SKIPPED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].skipped')
|
RUN2_SKIPPED=$(echo "$RES2" | jq -r '.accounts[0].skipped')
|
||||||
RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors')
|
RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors')
|
||||||
RUN2_COPIED_DELTA=$((RUN2_COPIED_TOTAL - RUN1_COPIED))
|
log "run 2: copied=$RUN2_COPIED skipped=$RUN2_SKIPPED errors=$RUN2_ERRORS"
|
||||||
RUN2_SKIPPED_DELTA=$((RUN2_SKIPPED_TOTAL - RUN1_SKIPPED))
|
|
||||||
log "run 2: copied_delta=$RUN2_COPIED_DELTA skipped_delta=$RUN2_SKIPPED_DELTA errors=$RUN2_ERRORS"
|
|
||||||
|
|
||||||
[[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors"
|
[[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors"
|
||||||
[[ "$RUN2_COPIED_DELTA" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED_DELTA new messages (expected 0, not idempotent)"
|
[[ "$RUN2_COPIED" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED new messages (expected 0, not idempotent)"
|
||||||
[[ "$RUN2_SKIPPED_DELTA" -gt 0 ]] || fail "run 2 skipped delta is $RUN2_SKIPPED_DELTA (expected >0)"
|
[[ "$RUN2_SKIPPED" -eq "$RUN1_COPIED" ]] ||
|
||||||
|
fail "run 2 skipped $RUN2_SKIPPED of the $RUN1_COPIED messages run 1 copied"
|
||||||
|
|
||||||
log "PASS: run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED_DELTA skipped=$RUN2_SKIPPED_DELTA (idempotent)"
|
log "run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED skipped=$RUN2_SKIPPED (idempotent)"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Pause / resume: a second account with enough messages that the run is still
|
||||||
|
# in flight when the pause lands. Pausing must leave the account resumable, and
|
||||||
|
# resuming must finish it without re-copying what the first stretch already did.
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
SRC_USER2="src2@example.com"
|
||||||
|
DST_USER2="dst2@example.com"
|
||||||
|
# Large enough that the copy is still in flight when the pause lands — greenmail
|
||||||
|
# on a local socket copies well over a thousand small messages per second.
|
||||||
|
SEED_COUNT=3000
|
||||||
|
|
||||||
|
log "seeding ${SEED_COUNT} messages into ${SRC_USER2} INBOX (pause/resume scenario)"
|
||||||
|
python3 "$SEED_PY" 127.0.0.1 3143 "$SRC_USER2" "$MAIL_PASS" "$SEED_COUNT"
|
||||||
|
|
||||||
|
log "adding second account (src2 -> dst2)"
|
||||||
|
ACCOUNT2_ID=$(api POST "/api/tasks/${TASK_ID}/accounts" \
|
||||||
|
"{\"src_login\":\"${SRC_USER2}\",\"src_pass\":\"${MAIL_PASS}\",\"dst_login\":\"${DST_USER2}\",\"dst_pass\":\"${MAIL_PASS}\"}" | jq -r .id)
|
||||||
|
[[ "$ACCOUNT2_ID" =~ ^[0-9]+$ ]] || fail "bad second account id: $ACCOUNT2_ID"
|
||||||
|
log "account2_id=$ACCOUNT2_ID"
|
||||||
|
|
||||||
|
log "POST /test (both accounts)"
|
||||||
|
api POST "/api/tasks/${TASK_ID}/test" >/dev/null
|
||||||
|
for ((i = 1; i <= 30; i++)); do
|
||||||
|
BOTH_OK=$(api GET "/api/tasks/${TASK_ID}" |
|
||||||
|
jq -r '[.accounts[] | select(.test_src_status=="ok" and .test_dst_status=="ok")] | length')
|
||||||
|
[[ "$BOTH_OK" == "2" ]] && break
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
[[ "$BOTH_OK" == "2" ]] || fail "second account did not pass connection tests (ok count=$BOTH_OK)"
|
||||||
|
|
||||||
|
# Account view for account2, by id.
|
||||||
|
acct2() { api GET "/api/tasks/${TASK_ID}" | jq -r ".accounts[] | select(.id==${ACCOUNT2_ID}) | $1"; }
|
||||||
|
|
||||||
|
log "POST /run (account2 only)"
|
||||||
|
api POST "/api/tasks/${TASK_ID}/run" "{\"account_ids\":[${ACCOUNT2_ID}]}" >/dev/null
|
||||||
|
|
||||||
|
# Per-account counters are only written to the DB when a folder completes, so
|
||||||
|
# "copied so far" is invisible here — wait for the account to go running, give
|
||||||
|
# the copy a few seconds of real work, then pause mid-folder.
|
||||||
|
log "waiting for account2 to start running"
|
||||||
|
for ((i = 1; i <= 120; i++)); do
|
||||||
|
[[ "$(acct2 .status)" == "running" ]] && break
|
||||||
|
sleep 0.5
|
||||||
|
done
|
||||||
|
[[ "$(acct2 .status)" == "running" ]] || fail "account2 never reached running"
|
||||||
|
|
||||||
|
log "letting it copy for a few seconds, then pausing mid-folder"
|
||||||
|
sleep 5
|
||||||
|
curl -fsS -b "$COOKIE_JAR" -c "$COOKIE_JAR" -X POST "$BASE/api/tasks/${TASK_ID}/pause" >/dev/null ||
|
||||||
|
fail "pause rejected — the run finished before the pause landed, seed more messages"
|
||||||
|
|
||||||
|
log "waiting for the task to settle into paused"
|
||||||
|
for ((i = 1; i <= 60; i++)); do
|
||||||
|
TASK_STATUS=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
|
||||||
|
[[ "$TASK_STATUS" == "paused" ]] && break
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
[[ "$TASK_STATUS" == "paused" ]] || fail "task status=$TASK_STATUS after pause (expected paused)"
|
||||||
|
|
||||||
|
ACC2_STATUS=$(acct2 .status)
|
||||||
|
[[ "$ACC2_STATUS" == "paused" ]] || fail "account2 status=$ACC2_STATUS after pause (expected paused)"
|
||||||
|
log "paused (folder-level counters at copied=$(acct2 .copied) of $SEED_COUNT)"
|
||||||
|
|
||||||
|
log "POST /resume"
|
||||||
|
api POST "/api/tasks/${TASK_ID}/resume" >/dev/null
|
||||||
|
wait_run_done
|
||||||
|
|
||||||
|
RESUMED_COPIED=$(acct2 .copied)
|
||||||
|
RESUMED_SKIPPED=$(acct2 .skipped)
|
||||||
|
RESUMED_ERRORS=$(acct2 .errors)
|
||||||
|
RESUMED_TOTAL=$((RESUMED_COPIED + RESUMED_SKIPPED))
|
||||||
|
log "after resume: copied=$RESUMED_COPIED skipped=$RESUMED_SKIPPED errors=$RESUMED_ERRORS"
|
||||||
|
[[ "$RESUMED_ERRORS" == "0" ]] || fail "resumed run had errors"
|
||||||
|
# Counters reset per run, so the resumed run alone must account for every
|
||||||
|
# message: the ones it copied now plus the ones the paused stretch already did.
|
||||||
|
[[ "$RESUMED_TOTAL" -eq "$SEED_COUNT" ]] || fail "resumed run covered $RESUMED_TOTAL of $SEED_COUNT messages"
|
||||||
|
# Non-zero skipped is the proof that the paused stretch's work survived: those
|
||||||
|
# messages are in the migration journal, so the resume did not re-copy them.
|
||||||
|
[[ "$RESUMED_SKIPPED" -gt 0 ]] ||
|
||||||
|
fail "resumed run skipped nothing — the paused stretch's progress was lost"
|
||||||
|
|
||||||
|
log "PASS: idempotent re-run; pause was resumable, resume re-copied $RESUMED_COPIED and skipped $RESUMED_SKIPPED of $SEED_COUNT"
|
||||||
|
|||||||
+79
-3
@@ -18,6 +18,9 @@ export interface Task {
|
|||||||
dst_endpoint_id: number
|
dst_endpoint_id: number
|
||||||
status: string
|
status: string
|
||||||
folder_mapping?: Record<string, string>
|
folder_mapping?: Record<string, string>
|
||||||
|
schedule_interval_seconds?: number
|
||||||
|
broken?: boolean
|
||||||
|
next_run_at?: string | null
|
||||||
}
|
}
|
||||||
|
|
||||||
export type TestStatus = 'pending' | 'ok' | 'fail' | string
|
export type TestStatus = 'pending' | 'ok' | 'fail' | string
|
||||||
@@ -33,6 +36,8 @@ export interface Account {
|
|||||||
skipped: number
|
skipped: number
|
||||||
errors: number
|
errors: number
|
||||||
last_error?: string
|
last_error?: string
|
||||||
|
folder_mapping?: Record<string, string>
|
||||||
|
excluded_folders?: string[]
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface TaskDetail {
|
export interface TaskDetail {
|
||||||
@@ -77,11 +82,21 @@ export const updateEndpoint = (
|
|||||||
body: { role_label: string; host: string; port: number; tls_mode: TLSMode },
|
body: { role_label: string; host: string; port: number; tls_mode: TLSMode },
|
||||||
) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' })
|
) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' })
|
||||||
|
|
||||||
|
export const deleteEndpoint = (id: number) => api(`/api/endpoints/${id}`, { method: 'DELETE' })
|
||||||
|
|
||||||
export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' })
|
export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' })
|
||||||
|
|
||||||
export const deleteAccount = (taskId: number, accountId: number) =>
|
export const deleteAccount = (taskId: number, accountId: number) =>
|
||||||
api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' })
|
api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' })
|
||||||
|
|
||||||
|
// Empty password fields keep the stored ones; both connection tests reset to
|
||||||
|
// unknown server-side, so the account must be re-tested afterwards.
|
||||||
|
export const updateAccountCredentials = (
|
||||||
|
taskId: number,
|
||||||
|
accountId: number,
|
||||||
|
body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
|
||||||
|
) => api(`/api/tasks/${taskId}/accounts/${accountId}/credentials`, { ...jsonBody(body), method: 'PUT' })
|
||||||
|
|
||||||
export const cancelAccount = (taskId: number, accountId: number) =>
|
export const cancelAccount = (taskId: number, accountId: number) =>
|
||||||
api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' })
|
api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' })
|
||||||
|
|
||||||
@@ -101,8 +116,19 @@ export const probeFolders = (
|
|||||||
creds: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
|
creds: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
|
||||||
) => api<ProbeResult>(`/api/tasks/${taskId}/probe`, jsonBody(creds))
|
) => api<ProbeResult>(`/api/tasks/${taskId}/probe`, jsonBody(creds))
|
||||||
|
|
||||||
export const setFolderMapping = (taskId: number, mapping: Record<string, string>) =>
|
export const probeAccountFolders = (taskId: number, accId: number) =>
|
||||||
api(`/api/tasks/${taskId}/folder-mapping`, { ...jsonBody({ mapping }), method: 'PUT' })
|
api<ProbeResult>(`/api/tasks/${taskId}/accounts/${accId}/probe`, { method: 'POST' })
|
||||||
|
|
||||||
|
export const setAccountFolderMapping = (
|
||||||
|
taskId: number,
|
||||||
|
accId: number,
|
||||||
|
mapping: Record<string, string>,
|
||||||
|
excluded: string[],
|
||||||
|
) =>
|
||||||
|
api(`/api/tasks/${taskId}/accounts/${accId}/folder-mapping`, {
|
||||||
|
...jsonBody({ mapping, excluded }),
|
||||||
|
method: 'PUT',
|
||||||
|
})
|
||||||
|
|
||||||
export const listTasks = () => api<Task[]>('/api/tasks')
|
export const listTasks = () => api<Task[]>('/api/tasks')
|
||||||
|
|
||||||
@@ -122,10 +148,60 @@ export const createAccount = (
|
|||||||
|
|
||||||
export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' })
|
export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' })
|
||||||
|
|
||||||
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
|
export const runTask = (id: number, accountIds?: number[]) =>
|
||||||
|
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
|
||||||
|
|
||||||
|
// Pause stops the run but leaves its unfinished accounts resumable; cancel ends
|
||||||
|
// it and marks them cancelled. Resume re-runs exactly the paused accounts.
|
||||||
|
export const pauseTask = (id: number) => api(`/api/tasks/${id}/pause`, { method: 'POST' })
|
||||||
|
|
||||||
|
export const cancelTask = (id: number) => api(`/api/tasks/${id}/cancel`, { method: 'POST' })
|
||||||
|
|
||||||
|
export const resumeTask = (id: number) => api<{ run_id: number }>(`/api/tasks/${id}/resume`, { method: 'POST' })
|
||||||
|
|
||||||
|
export interface Run {
|
||||||
|
id: number
|
||||||
|
task_id: number
|
||||||
|
status: string
|
||||||
|
started_at: string
|
||||||
|
finished_at: string | null
|
||||||
|
total_copied: number
|
||||||
|
total_skipped: number
|
||||||
|
total_errors: number
|
||||||
|
trigger: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export const setTaskSchedule = (taskId: number, intervalSeconds: number) =>
|
||||||
|
api(`/api/tasks/${taskId}/schedule`, { ...jsonBody({ interval_seconds: intervalSeconds }), method: 'PUT' })
|
||||||
|
|
||||||
|
export const listRuns = (taskId: number) => api<Run[]>(`/api/tasks/${taskId}/runs`)
|
||||||
|
|
||||||
|
export interface AccountError {
|
||||||
|
id: number
|
||||||
|
account_id: number
|
||||||
|
run_id: number
|
||||||
|
kind: string // folder | message | account
|
||||||
|
folder: string
|
||||||
|
message_ref: string
|
||||||
|
error: string
|
||||||
|
created_at: string
|
||||||
|
}
|
||||||
|
|
||||||
|
export const listAccountErrors = (taskId: number, accountId: number) =>
|
||||||
|
api<AccountError[]>(`/api/tasks/${taskId}/accounts/${accountId}/errors`)
|
||||||
|
|
||||||
export const importCSV = (id: number, file: File) => {
|
export const importCSV = (id: number, file: File) => {
|
||||||
const fd = new FormData()
|
const fd = new FormData()
|
||||||
fd.append('file', file)
|
fd.append('file', file)
|
||||||
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
|
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A Kerio Connect export carries no domain, so the operator supplies it here;
|
||||||
|
// the login and password apply to both sides of the migration.
|
||||||
|
export const importKerioCSV = (id: number, file: File, domain: string) => {
|
||||||
|
const fd = new FormData()
|
||||||
|
fd.append('file', file)
|
||||||
|
fd.append('format', 'kerio')
|
||||||
|
fd.append('domain', domain)
|
||||||
|
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
|
||||||
|
}
|
||||||
|
|||||||
+337
-8
@@ -245,8 +245,8 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
.link-btn.danger:hover {
|
.link-btn.danger:hover {
|
||||||
color: var(--danger, #ff5c5c);
|
color: var(--fail);
|
||||||
border-bottom-color: var(--danger, #ff5c5c);
|
border-bottom-color: var(--fail);
|
||||||
}
|
}
|
||||||
|
|
||||||
.link-btn:disabled {
|
.link-btn:disabled {
|
||||||
@@ -255,11 +255,36 @@
|
|||||||
}
|
}
|
||||||
|
|
||||||
/* per-account live progress */
|
/* per-account live progress */
|
||||||
|
/* merged Source/Destination "Account" cell: single line when src == dst,
|
||||||
|
stacked (src over dst) when they differ. */
|
||||||
|
.acct-ident {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.acct-dst {
|
||||||
|
color: var(--fg-dim);
|
||||||
|
font-size: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
.acct-progress {
|
.acct-progress {
|
||||||
display: flex;
|
display: flex;
|
||||||
flex-direction: column;
|
flex-direction: column;
|
||||||
gap: 4px;
|
gap: 4px;
|
||||||
min-width: 160px;
|
min-width: 160px;
|
||||||
|
/* reserve room for the bar + two meta lines so toggling the scan line
|
||||||
|
during a run doesn't change the row height */
|
||||||
|
min-height: 40px;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Running rows get a fixed wide size up front so live progress updates
|
||||||
|
(which change text length and can wrap) never resize the row mid-scan.
|
||||||
|
The size reverts only when the run finishes and the class drops. */
|
||||||
|
.progress-cell--live {
|
||||||
|
min-width: 340px;
|
||||||
|
height: 48px;
|
||||||
|
vertical-align: top;
|
||||||
}
|
}
|
||||||
|
|
||||||
.pbar {
|
.pbar {
|
||||||
@@ -272,14 +297,19 @@
|
|||||||
.pbar-fill {
|
.pbar-fill {
|
||||||
display: block;
|
display: block;
|
||||||
height: 100%;
|
height: 100%;
|
||||||
|
width: 100%;
|
||||||
|
transform: scaleX(0);
|
||||||
|
transform-origin: left;
|
||||||
background: var(--accent);
|
background: var(--accent);
|
||||||
transition: width 0.3s ease-out;
|
transition: transform 0.3s ease-out;
|
||||||
}
|
}
|
||||||
|
|
||||||
.pmeta {
|
.pmeta {
|
||||||
font-size: 10px;
|
font-size: 10px;
|
||||||
color: var(--fg-dim);
|
color: var(--fg-dim);
|
||||||
letter-spacing: 0.04em;
|
letter-spacing: 0.04em;
|
||||||
|
/* keep progress text on one line so its length never reflows the row */
|
||||||
|
white-space: nowrap;
|
||||||
}
|
}
|
||||||
|
|
||||||
.pscan {
|
.pscan {
|
||||||
@@ -297,6 +327,77 @@
|
|||||||
white-space: nowrap;
|
white-space: nowrap;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* per-account errors modal: a paginated list so long messages wrap on the
|
||||||
|
full modal width instead of squeezing into narrow table columns. */
|
||||||
|
.err-empty {
|
||||||
|
padding: 24px 8px;
|
||||||
|
text-align: center;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-list {
|
||||||
|
list-style: none;
|
||||||
|
margin: 0;
|
||||||
|
padding: 0;
|
||||||
|
max-height: 60vh;
|
||||||
|
overflow-y: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-item {
|
||||||
|
padding: 10px 4px;
|
||||||
|
border-bottom: 1px solid var(--border);
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-item-head {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
align-items: baseline;
|
||||||
|
gap: 8px;
|
||||||
|
font-size: 11px;
|
||||||
|
letter-spacing: 0.04em;
|
||||||
|
margin-bottom: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-kind {
|
||||||
|
color: var(--accent);
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-folder {
|
||||||
|
color: var(--fg);
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-ref {
|
||||||
|
color: var(--fg-dim);
|
||||||
|
word-break: break-word;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-time {
|
||||||
|
margin-left: auto;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-text {
|
||||||
|
font-size: 12px;
|
||||||
|
line-height: 1.45;
|
||||||
|
color: var(--fail);
|
||||||
|
white-space: normal;
|
||||||
|
word-break: break-word;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-pager {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
gap: 14px;
|
||||||
|
padding-top: 12px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.err-pageinfo {
|
||||||
|
font-size: 12px;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
}
|
||||||
|
|
||||||
/* clear-log button: mirrors the .panel-label tab on the right edge */
|
/* clear-log button: mirrors the .panel-label tab on the right edge */
|
||||||
.log-clear {
|
.log-clear {
|
||||||
position: absolute;
|
position: absolute;
|
||||||
@@ -365,11 +466,59 @@
|
|||||||
|
|
||||||
.map-row {
|
.map-row {
|
||||||
display: grid;
|
display: grid;
|
||||||
grid-template-columns: 1fr auto 1fr auto;
|
grid-template-columns: auto 1fr auto 1fr auto;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
gap: 10px;
|
gap: 10px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.map-row-off .map-src,
|
||||||
|
.map-row-off .map-arrow {
|
||||||
|
color: var(--fg-faint);
|
||||||
|
text-decoration: line-through;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-check {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-check input,
|
||||||
|
.map-all input {
|
||||||
|
accent-color: var(--accent);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-toolbar {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 14px;
|
||||||
|
margin-bottom: 12px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-toolbar .btn {
|
||||||
|
padding: 6px 12px;
|
||||||
|
font-size: 11px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-all {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 8px;
|
||||||
|
font-size: 11px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.map-excluded {
|
||||||
|
font-size: 10px;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.08em;
|
||||||
|
color: var(--fg-faint);
|
||||||
|
}
|
||||||
|
|
||||||
.map-src {
|
.map-src {
|
||||||
font-family: var(--font-mono);
|
font-family: var(--font-mono);
|
||||||
font-size: 13px;
|
font-size: 13px;
|
||||||
@@ -473,11 +622,32 @@
|
|||||||
box-shadow: 0 0 16px -2px rgba(255, 178, 56, 0.6);
|
box-shadow: 0 0 16px -2px rgba(255, 178, 56, 0.6);
|
||||||
}
|
}
|
||||||
|
|
||||||
.btn:disabled {
|
.btn:disabled,
|
||||||
|
.btn.is-disabled {
|
||||||
opacity: 0.35;
|
opacity: 0.35;
|
||||||
cursor: not-allowed;
|
cursor: not-allowed;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* label styled as .btn (CSV upload) can't use :disabled — block interaction. */
|
||||||
|
.btn.is-disabled {
|
||||||
|
pointer-events: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- keyboard focus ---------- */
|
||||||
|
/* Custom-styled controls drop the faint UA outline; restore a visible, on-brand
|
||||||
|
focus ring for keyboard users (mouse clicks stay ring-free via :focus-visible). */
|
||||||
|
.btn:focus-visible,
|
||||||
|
.link-btn:focus-visible,
|
||||||
|
.file-btn:focus-within,
|
||||||
|
.topnav a:focus-visible,
|
||||||
|
.crumb:focus-visible,
|
||||||
|
table.tbl a.rowlink:focus-visible {
|
||||||
|
outline: 2px solid var(--accent);
|
||||||
|
outline-offset: 2px;
|
||||||
|
border-radius: 2px;
|
||||||
|
color: var(--accent-strong);
|
||||||
|
}
|
||||||
|
|
||||||
.btn-row {
|
.btn-row {
|
||||||
display: flex;
|
display: flex;
|
||||||
gap: 10px;
|
gap: 10px;
|
||||||
@@ -497,6 +667,25 @@
|
|||||||
|
|
||||||
/* ---------- status badges ---------- */
|
/* ---------- status badges ---------- */
|
||||||
|
|
||||||
|
/* A badge that opens a dialog: the badge keeps its own look, the button only
|
||||||
|
contributes the affordance. */
|
||||||
|
.badge-btn {
|
||||||
|
padding: 0;
|
||||||
|
border: 0;
|
||||||
|
background: none;
|
||||||
|
font: inherit;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge-btn:hover .badge {
|
||||||
|
filter: brightness(1.25);
|
||||||
|
}
|
||||||
|
|
||||||
|
.badge-btn:focus-visible {
|
||||||
|
outline: 1px solid var(--accent);
|
||||||
|
outline-offset: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
.badge {
|
.badge {
|
||||||
display: inline-flex;
|
display: inline-flex;
|
||||||
align-items: center;
|
align-items: center;
|
||||||
@@ -533,18 +722,51 @@
|
|||||||
|
|
||||||
.badge-pending {
|
.badge-pending {
|
||||||
color: var(--pending);
|
color: var(--pending);
|
||||||
border-color: #4a4423;
|
border-color: var(--pending-dim);
|
||||||
background: rgba(240, 196, 25, 0.06);
|
background: rgba(240, 196, 25, 0.06);
|
||||||
}
|
}
|
||||||
.badge-pending .dot { background: var(--pending); }
|
.badge-pending .dot { background: var(--pending); }
|
||||||
|
|
||||||
.badge-info {
|
.badge-info {
|
||||||
color: var(--info);
|
color: var(--info);
|
||||||
border-color: #234456;
|
border-color: var(--info-dim);
|
||||||
background: rgba(87, 194, 255, 0.06);
|
background: rgba(87, 194, 255, 0.06);
|
||||||
}
|
}
|
||||||
.badge-info .dot { background: var(--info); animation: pulse 1.4s ease-in-out infinite; }
|
.badge-info .dot { background: var(--info); animation: pulse 1.4s ease-in-out infinite; }
|
||||||
|
|
||||||
|
/* ---------- schedule row ---------- */
|
||||||
|
|
||||||
|
.sched-row {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 12px;
|
||||||
|
margin-top: 14px;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sched-row label {
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
letter-spacing: 0.1em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
}
|
||||||
|
|
||||||
|
.sched-row select {
|
||||||
|
background: var(--bg-inset);
|
||||||
|
border: 1px solid var(--border);
|
||||||
|
color: var(--fg);
|
||||||
|
padding: 7px 10px;
|
||||||
|
font-size: 13px;
|
||||||
|
border-radius: 2px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sched-next {
|
||||||
|
font-size: 12px;
|
||||||
|
color: var(--fg-dim);
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
}
|
||||||
|
|
||||||
/* ---------- tables ---------- */
|
/* ---------- tables ---------- */
|
||||||
|
|
||||||
.tbl-wrap {
|
.tbl-wrap {
|
||||||
@@ -599,6 +821,23 @@ table.tbl a.rowlink:hover {
|
|||||||
text-align: right;
|
text-align: right;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.row-actions {
|
||||||
|
display: inline-flex;
|
||||||
|
gap: 12px;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: flex-end;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Marks accounts whose folders have already been mapped/excluded. */
|
||||||
|
.map-dot {
|
||||||
|
width: 8px;
|
||||||
|
height: 8px;
|
||||||
|
border-radius: 50%;
|
||||||
|
flex: none;
|
||||||
|
background: var(--ok);
|
||||||
|
box-shadow: 0 0 5px var(--ok);
|
||||||
|
}
|
||||||
|
|
||||||
.empty-row td {
|
.empty-row td {
|
||||||
text-align: center;
|
text-align: center;
|
||||||
color: var(--fg-faint);
|
color: var(--fg-faint);
|
||||||
@@ -697,11 +936,24 @@ table.tbl a.rowlink:hover {
|
|||||||
|
|
||||||
.upload-row {
|
.upload-row {
|
||||||
display: flex;
|
display: flex;
|
||||||
align-items: center;
|
align-items: flex-start;
|
||||||
gap: 12px;
|
gap: 12px;
|
||||||
flex-wrap: wrap;
|
flex-wrap: wrap;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* One import route per column: the action on top, its sample file underneath. */
|
||||||
|
.upload-item {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
align-items: stretch;
|
||||||
|
gap: 6px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.upload-item .link-btn {
|
||||||
|
align-self: center;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
.file-btn {
|
.file-btn {
|
||||||
position: relative;
|
position: relative;
|
||||||
overflow: hidden;
|
overflow: hidden;
|
||||||
@@ -784,3 +1036,80 @@ table.tbl a.rowlink:hover {
|
|||||||
color: var(--fg-faint);
|
color: var(--fg-faint);
|
||||||
font-size: 12px;
|
font-size: 12px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* ---------- responsive ---------- */
|
||||||
|
/* The shell is desktop-first (density is a feature for operators). These
|
||||||
|
overrides keep the chrome usable on phones/tablets without thinning the
|
||||||
|
data-dense surfaces. Intrinsic grids (panel-grid auto-fit, stat-row/page-head
|
||||||
|
flex-wrap) already reflow; the topbar and paired fields need explicit help. */
|
||||||
|
@media (max-width: 640px) {
|
||||||
|
.topbar {
|
||||||
|
flex-wrap: wrap;
|
||||||
|
height: auto;
|
||||||
|
padding: 10px 16px;
|
||||||
|
gap: 12px 18px;
|
||||||
|
}
|
||||||
|
/* Secondary chrome — drop the ambient session pill to save the row on mobile. */
|
||||||
|
.session-indicator {
|
||||||
|
display: none;
|
||||||
|
}
|
||||||
|
.topnav {
|
||||||
|
order: 3;
|
||||||
|
flex-basis: 100%;
|
||||||
|
flex: 0 0 100%;
|
||||||
|
}
|
||||||
|
.main {
|
||||||
|
padding: 20px 16px 48px;
|
||||||
|
}
|
||||||
|
.page-title {
|
||||||
|
font-size: 26px;
|
||||||
|
}
|
||||||
|
/* Stacked src/dst fields — two password inputs side by side are too tight. */
|
||||||
|
.field-row {
|
||||||
|
grid-template-columns: 1fr;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- reduced motion ---------- */
|
||||||
|
/* Neutralize the infinite pulse dots, modal entrance, and progress transition
|
||||||
|
for users who opt out. State stays legible through color + presence, not
|
||||||
|
movement. */
|
||||||
|
@media (prefers-reduced-motion: reduce) {
|
||||||
|
*,
|
||||||
|
*::before,
|
||||||
|
*::after {
|
||||||
|
animation-duration: 0.01ms !important;
|
||||||
|
animation-iteration-count: 1 !important;
|
||||||
|
transition-duration: 0.01ms !important;
|
||||||
|
scroll-behavior: auto !important;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Coarse pointers (touch) get larger hit areas; fine pointers keep the dense
|
||||||
|
inline layout. WCAG 2.5.8 target size (AA) is 24px minimum. */
|
||||||
|
@media (pointer: coarse) {
|
||||||
|
.link-btn {
|
||||||
|
display: inline-block;
|
||||||
|
padding-block: 6px;
|
||||||
|
min-height: 24px;
|
||||||
|
}
|
||||||
|
.topnav a {
|
||||||
|
padding: 12px 16px;
|
||||||
|
}
|
||||||
|
.btn-ghost {
|
||||||
|
padding: 12px 14px;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
.tbl .chk-col {
|
||||||
|
width: 32px;
|
||||||
|
text-align: center;
|
||||||
|
padding-right: 0;
|
||||||
|
}
|
||||||
|
.tbl .chk-col input[type='checkbox'] {
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.tbl .chk-col input[type='checkbox']:disabled {
|
||||||
|
cursor: not-allowed;
|
||||||
|
opacity: 0.4;
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,116 @@
|
|||||||
|
import { useEffect, useState, type FormEvent } from 'react'
|
||||||
|
import { Modal } from './Modal'
|
||||||
|
import type { Account } from '../api'
|
||||||
|
|
||||||
|
type Props = {
|
||||||
|
open: boolean
|
||||||
|
busy: boolean
|
||||||
|
account: Account | null
|
||||||
|
onClose: () => void
|
||||||
|
onSubmit: (body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) => void
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fixes the credentials of an account that failed its connection test — usually
|
||||||
|
// a wrong password that came in through a CSV import. Passwords are never sent
|
||||||
|
// back to the browser, so the fields start empty and an empty field means
|
||||||
|
// "keep the stored password".
|
||||||
|
export function AccountCredentialsModal({ open, busy, account, onClose, onSubmit }: Props) {
|
||||||
|
const [srcLogin, setSrcLogin] = useState('')
|
||||||
|
const [dstLogin, setDstLogin] = useState('')
|
||||||
|
const [srcPass, setSrcPass] = useState('')
|
||||||
|
const [dstPass, setDstPass] = useState('')
|
||||||
|
const [error, setError] = useState<string | null>(null)
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open || !account) return
|
||||||
|
setSrcLogin(account.src_login)
|
||||||
|
setDstLogin(account.dst_login)
|
||||||
|
setSrcPass('')
|
||||||
|
setDstPass('')
|
||||||
|
setError(null)
|
||||||
|
}, [open, account])
|
||||||
|
|
||||||
|
function submit(e: FormEvent) {
|
||||||
|
e.preventDefault()
|
||||||
|
if (srcLogin.trim() === '' || dstLogin.trim() === '') {
|
||||||
|
setError('Both logins are required')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setError(null)
|
||||||
|
onSubmit({
|
||||||
|
src_login: srcLogin.trim(),
|
||||||
|
src_pass: srcPass,
|
||||||
|
dst_login: dstLogin.trim(),
|
||||||
|
dst_pass: dstPass,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Modal open={open} title={account ? `Edit credentials — ${account.src_login}` : 'Edit credentials'} onClose={onClose}>
|
||||||
|
<form onSubmit={submit}>
|
||||||
|
<p className="map-hint">
|
||||||
|
Leave a password field empty to keep the stored one. Saving resets both connection tests, so re-run{' '}
|
||||||
|
<strong>Test connections</strong> afterwards.
|
||||||
|
</p>
|
||||||
|
<div className="field-row">
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="edit_src_login">Source login</label>
|
||||||
|
<input
|
||||||
|
id="edit_src_login"
|
||||||
|
data-modal-autofocus
|
||||||
|
value={srcLogin}
|
||||||
|
onChange={(e) => setSrcLogin(e.target.value)}
|
||||||
|
disabled={busy}
|
||||||
|
required
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="edit_src_pass">Source password</label>
|
||||||
|
<input
|
||||||
|
id="edit_src_pass"
|
||||||
|
type="password"
|
||||||
|
value={srcPass}
|
||||||
|
onChange={(e) => setSrcPass(e.target.value)}
|
||||||
|
placeholder="unchanged"
|
||||||
|
autoComplete="new-password"
|
||||||
|
disabled={busy}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="field-row">
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="edit_dst_login">Destination login</label>
|
||||||
|
<input
|
||||||
|
id="edit_dst_login"
|
||||||
|
value={dstLogin}
|
||||||
|
onChange={(e) => setDstLogin(e.target.value)}
|
||||||
|
disabled={busy}
|
||||||
|
required
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="edit_dst_pass">Destination password</label>
|
||||||
|
<input
|
||||||
|
id="edit_dst_pass"
|
||||||
|
type="password"
|
||||||
|
value={dstPass}
|
||||||
|
onChange={(e) => setDstPass(e.target.value)}
|
||||||
|
placeholder="unchanged"
|
||||||
|
autoComplete="new-password"
|
||||||
|
disabled={busy}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{error && <div className="error-banner">{error}</div>}
|
||||||
|
<div className="modal-actions">
|
||||||
|
<button type="button" className="btn" onClick={onClose} disabled={busy}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-primary" disabled={busy}>
|
||||||
|
{busy ? 'Saving…' : 'Save credentials'}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</Modal>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -0,0 +1,91 @@
|
|||||||
|
import { useEffect, useState } from 'react'
|
||||||
|
import { Modal } from './Modal'
|
||||||
|
import { listAccountErrors, type AccountError } from '../api'
|
||||||
|
|
||||||
|
const fmt = (iso: string) => (iso ? new Date(iso).toLocaleString() : '—')
|
||||||
|
const PAGE_SIZE = 50
|
||||||
|
|
||||||
|
export function AccountErrorsModal({
|
||||||
|
taskId,
|
||||||
|
account,
|
||||||
|
onClose,
|
||||||
|
}: {
|
||||||
|
taskId: number
|
||||||
|
account: { id: number; src_login: string } | null
|
||||||
|
onClose: () => void
|
||||||
|
}) {
|
||||||
|
const [errors, setErrors] = useState<AccountError[] | null>(null)
|
||||||
|
const [failed, setFailed] = useState(false)
|
||||||
|
const [page, setPage] = useState(0)
|
||||||
|
const open = account !== null
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!account) return
|
||||||
|
setErrors(null)
|
||||||
|
setFailed(false)
|
||||||
|
setPage(0)
|
||||||
|
listAccountErrors(taskId, account.id)
|
||||||
|
.then((e) => setErrors(e ?? []))
|
||||||
|
.catch(() => setFailed(true))
|
||||||
|
}, [taskId, account])
|
||||||
|
|
||||||
|
const total = errors?.length ?? 0
|
||||||
|
const pageCount = Math.max(1, Math.ceil(total / PAGE_SIZE))
|
||||||
|
const current = Math.min(page, pageCount - 1)
|
||||||
|
const slice = errors?.slice(current * PAGE_SIZE, current * PAGE_SIZE + PAGE_SIZE) ?? []
|
||||||
|
|
||||||
|
const title = account
|
||||||
|
? `Errors — ${account.src_login}${total ? ` (${total})` : ''}`
|
||||||
|
: 'Errors'
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Modal open={open} title={title} onClose={onClose} size="lg">
|
||||||
|
{failed ? (
|
||||||
|
<div className="err-empty">failed to load errors</div>
|
||||||
|
) : errors === null ? (
|
||||||
|
<div className="err-empty">loading…</div>
|
||||||
|
) : total === 0 ? (
|
||||||
|
<div className="err-empty">no errors recorded</div>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<ul className="err-list">
|
||||||
|
{slice.map((e) => (
|
||||||
|
<li key={e.id} className="err-item">
|
||||||
|
<div className="err-item-head">
|
||||||
|
<span className="err-kind">{e.kind}</span>
|
||||||
|
{e.folder && <span className="err-folder">{e.folder}</span>}
|
||||||
|
{e.message_ref && <span className="err-ref">{e.message_ref}</span>}
|
||||||
|
<span className="err-time">{fmt(e.created_at)}</span>
|
||||||
|
</div>
|
||||||
|
<div className="err-text">{e.error}</div>
|
||||||
|
</li>
|
||||||
|
))}
|
||||||
|
</ul>
|
||||||
|
{pageCount > 1 && (
|
||||||
|
<div className="err-pager">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="btn"
|
||||||
|
disabled={current === 0}
|
||||||
|
onClick={() => setPage(current - 1)}
|
||||||
|
>
|
||||||
|
‹ prev
|
||||||
|
</button>
|
||||||
|
<span className="err-pageinfo">
|
||||||
|
page {current + 1} / {pageCount}
|
||||||
|
</span>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="btn"
|
||||||
|
disabled={current >= pageCount - 1}
|
||||||
|
onClick={() => setPage(current + 1)}
|
||||||
|
>
|
||||||
|
next ›
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</Modal>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -6,7 +6,9 @@ type Props = {
|
|||||||
srcFolders: string[]
|
srcFolders: string[]
|
||||||
dstFolders: string[]
|
dstFolders: string[]
|
||||||
initialMapping: Record<string, string>
|
initialMapping: Record<string, string>
|
||||||
onConfirm: (mapping: Record<string, string>) => void
|
initialExcluded: string[]
|
||||||
|
accountLabel?: string
|
||||||
|
onConfirm: (mapping: Record<string, string>, excluded: string[]) => void
|
||||||
onCancel: () => void
|
onCancel: () => void
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -18,62 +20,154 @@ function defaultDst(src: string, dstFolders: string[], initial: Record<string, s
|
|||||||
return src
|
return src
|
||||||
}
|
}
|
||||||
|
|
||||||
export function FolderMappingModal({ open, srcFolders, dstFolders, initialMapping, onConfirm, onCancel }: Props) {
|
// Exchange/Kerio special folders and their mailcow counterparts. Keyed by the
|
||||||
|
// lowercased source name so casing differences between servers don't matter.
|
||||||
|
const DEFAULT_TARGETS: Record<string, string> = {
|
||||||
|
'deleted items': 'Trash',
|
||||||
|
'deleted messages': 'Trash',
|
||||||
|
'junk e-mail': 'Junk',
|
||||||
|
'junk email': 'Junk',
|
||||||
|
spam: 'Junk',
|
||||||
|
'sent items': 'Sent',
|
||||||
|
'sent messages': 'Sent',
|
||||||
|
}
|
||||||
|
|
||||||
|
// Source folders with no counterpart on mailcow — unchecked by the defaults.
|
||||||
|
const DEFAULT_EXCLUDED = new Set(['public folders'])
|
||||||
|
|
||||||
|
export function FolderMappingModal({
|
||||||
|
open, srcFolders, dstFolders, initialMapping, initialExcluded, accountLabel, onConfirm, onCancel,
|
||||||
|
}: Props) {
|
||||||
const [choice, setChoice] = useState<Record<string, string>>({})
|
const [choice, setChoice] = useState<Record<string, string>>({})
|
||||||
|
const [synced, setSynced] = useState<Record<string, boolean>>(() => {
|
||||||
|
const excl = new Set(initialExcluded)
|
||||||
|
return Object.fromEntries(srcFolders.map((f) => [f, !excl.has(f)]))
|
||||||
|
})
|
||||||
|
|
||||||
// Options per select: all destination folders, plus the source name itself
|
// Options per select: all destination folders, plus the source name itself
|
||||||
// (marked "create") when it does not already exist on the destination.
|
// (marked "create") when it does not already exist on the destination.
|
||||||
|
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
|
||||||
|
|
||||||
|
// Options per select: all destination folders, plus any name not present there
|
||||||
|
// — the source folder itself and the current selection — marked "create".
|
||||||
const options = useMemo(() => {
|
const options = useMemo(() => {
|
||||||
const set = new Set(dstFolders)
|
const set = new Set(dstFolders)
|
||||||
return (src: string) => {
|
return (src: string, current: string) => {
|
||||||
const opts = [...dstFolders]
|
const opts = [...dstFolders]
|
||||||
if (!set.has(src)) opts.unshift(src)
|
if (!set.has(current)) opts.unshift(current)
|
||||||
|
if (!set.has(src) && src !== current) opts.unshift(src)
|
||||||
return opts
|
return opts
|
||||||
}
|
}
|
||||||
}, [dstFolders])
|
}, [dstFolders])
|
||||||
|
|
||||||
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
|
// Collapse the Exchange/Kerio folder layout onto mailcow's in one click: the
|
||||||
|
// per-account mapping is otherwise repetitive work when importing many users.
|
||||||
|
function applyDefaults() {
|
||||||
|
const nextChoice = { ...choice }
|
||||||
|
const nextSynced = { ...synced }
|
||||||
|
for (const src of srcFolders) {
|
||||||
|
const key = src.trim().toLowerCase()
|
||||||
|
if (DEFAULT_EXCLUDED.has(key)) {
|
||||||
|
nextSynced[src] = false
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
const target = DEFAULT_TARGETS[key]
|
||||||
|
if (target) {
|
||||||
|
nextChoice[src] = target
|
||||||
|
nextSynced[src] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
setChoice(nextChoice)
|
||||||
|
setSynced(nextSynced)
|
||||||
|
}
|
||||||
|
|
||||||
function confirm() {
|
function confirm() {
|
||||||
const mapping: Record<string, string> = { ...initialMapping }
|
const mapping: Record<string, string> = {}
|
||||||
|
const excluded: string[] = []
|
||||||
for (const src of srcFolders) {
|
for (const src of srcFolders) {
|
||||||
|
if (synced[src] === false) {
|
||||||
|
excluded.push(src)
|
||||||
|
delete mapping[src]
|
||||||
|
continue
|
||||||
|
}
|
||||||
const dst = valueFor(src)
|
const dst = valueFor(src)
|
||||||
if (dst === src) delete mapping[src]
|
if (dst === src) delete mapping[src]
|
||||||
else mapping[src] = dst
|
else mapping[src] = dst
|
||||||
}
|
}
|
||||||
onConfirm(mapping)
|
onConfirm(mapping, excluded)
|
||||||
}
|
}
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<Modal open={open} title="Map folders (source → destination)" onClose={onCancel} size="lg">
|
<Modal
|
||||||
|
open={open}
|
||||||
|
title={accountLabel ? `Map folders — ${accountLabel}` : 'Map folders (source → destination)'}
|
||||||
|
onClose={onCancel}
|
||||||
|
size="lg"
|
||||||
|
>
|
||||||
<div className="map-body">
|
<div className="map-body">
|
||||||
<p className="map-hint">
|
<p className="map-hint">
|
||||||
Route each source folder to an existing destination folder. Leaving a folder mapped to its own name
|
Route each source folder to an existing destination folder. Leaving a folder mapped to its own name
|
||||||
creates it on the destination if missing (e.g. map <code>Спам</code> → <code>Spam</code> to avoid duplicates).
|
creates it on the destination if missing (e.g. map <code>Спам</code> → <code>Spam</code> to avoid duplicates).
|
||||||
</p>
|
</p>
|
||||||
|
<div className="map-toolbar">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="btn btn-ghost"
|
||||||
|
onClick={applyDefaults}
|
||||||
|
title="Map Deleted Items → Trash, Junk E-mail → Junk, Sent Items → Sent and skip Public Folders"
|
||||||
|
>
|
||||||
|
By default
|
||||||
|
</button>
|
||||||
|
<label className="map-all">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={srcFolders.every((f) => synced[f] !== false)}
|
||||||
|
onChange={(e) => {
|
||||||
|
const on = e.target.checked
|
||||||
|
setSynced(Object.fromEntries(srcFolders.map((f) => [f, on])))
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
sync all folders
|
||||||
|
</label>
|
||||||
|
</div>
|
||||||
<div className="map-grid">
|
<div className="map-grid">
|
||||||
{srcFolders.map((src) => {
|
{srcFolders.map((src) => {
|
||||||
|
const on = synced[src] !== false
|
||||||
const val = valueFor(src)
|
const val = valueFor(src)
|
||||||
const creates = !dstFolders.includes(val)
|
const creates = !dstFolders.includes(val)
|
||||||
return (
|
return (
|
||||||
<div className="map-row" key={src}>
|
<div className={`map-row${on ? '' : ' map-row-off'}`} key={src}>
|
||||||
|
<label className="map-check">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
checked={on}
|
||||||
|
aria-label={`Sync folder ${src}`}
|
||||||
|
onChange={(e) => setSynced((s) => ({ ...s, [src]: e.target.checked }))}
|
||||||
|
/>
|
||||||
|
</label>
|
||||||
<span className="map-src" title={src}>
|
<span className="map-src" title={src}>
|
||||||
{src}
|
{src}
|
||||||
</span>
|
</span>
|
||||||
<span className="map-arrow">→</span>
|
<span className="map-arrow">→</span>
|
||||||
<select
|
<select
|
||||||
className="map-select"
|
className="map-select"
|
||||||
|
aria-label={`Destination folder for ${src}`}
|
||||||
value={val}
|
value={val}
|
||||||
|
disabled={!on}
|
||||||
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
|
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
|
||||||
>
|
>
|
||||||
{options(src).map((f) => (
|
{options(src, val).map((f) => (
|
||||||
<option key={f} value={f}>
|
<option key={f} value={f}>
|
||||||
{f}
|
{f}
|
||||||
{f === src && !dstFolders.includes(src) ? ' (create)' : ''}
|
{dstFolders.includes(f) ? '' : ' (create)'}
|
||||||
</option>
|
</option>
|
||||||
))}
|
))}
|
||||||
</select>
|
</select>
|
||||||
{creates && <span className="map-new">new</span>}
|
{on ? (
|
||||||
|
creates && <span className="map-new">new</span>
|
||||||
|
) : (
|
||||||
|
<span className="map-excluded">excluded</span>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
})}
|
})}
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
import { useEffect, useState, type FormEvent } from 'react'
|
||||||
|
import { Modal } from './Modal'
|
||||||
|
|
||||||
|
// A bare DNS domain: no scheme, no user part, no whitespace. Mirrors the
|
||||||
|
// server-side check in csvimport.normalizeDomain so bad input is caught here.
|
||||||
|
const domainRe = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/
|
||||||
|
|
||||||
|
type Props = {
|
||||||
|
open: boolean
|
||||||
|
busy: boolean
|
||||||
|
onClose: () => void
|
||||||
|
onSubmit: (file: File, domain: string) => void
|
||||||
|
}
|
||||||
|
|
||||||
|
export function KerioImportModal({ open, busy, onClose, onSubmit }: Props) {
|
||||||
|
const [domain, setDomain] = useState('')
|
||||||
|
const [file, setFile] = useState<File | null>(null)
|
||||||
|
const [error, setError] = useState<string | null>(null)
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return
|
||||||
|
setDomain('')
|
||||||
|
setFile(null)
|
||||||
|
setError(null)
|
||||||
|
}, [open])
|
||||||
|
|
||||||
|
function submit(e: FormEvent) {
|
||||||
|
e.preventDefault()
|
||||||
|
const d = domain.trim().toLowerCase()
|
||||||
|
if (!domainRe.test(d)) {
|
||||||
|
setError('Enter a bare domain, e.g. galaxyhotel.kz')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if (!file) {
|
||||||
|
setError('Choose the Kerio export file')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setError(null)
|
||||||
|
onSubmit(file, d)
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Modal open={open} title="Import from Kerio" onClose={onClose}>
|
||||||
|
<form onSubmit={submit}>
|
||||||
|
<p className="map-hint">
|
||||||
|
The Kerio user export lists a login and its password but no domain. The domain you enter is appended to every
|
||||||
|
login and used for both the source and the destination. Disabled accounts and <code>admin</code> are skipped.
|
||||||
|
</p>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="kerio_domain">Mail domain</label>
|
||||||
|
<input
|
||||||
|
id="kerio_domain"
|
||||||
|
data-modal-autofocus
|
||||||
|
value={domain}
|
||||||
|
onChange={(e) => setDomain(e.target.value)}
|
||||||
|
placeholder="galaxyhotel.kz"
|
||||||
|
disabled={busy}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="kerio_file">Export file</label>
|
||||||
|
<input
|
||||||
|
id="kerio_file"
|
||||||
|
type="file"
|
||||||
|
accept=".csv,text/csv"
|
||||||
|
onChange={(e) => setFile(e.target.files?.[0] ?? null)}
|
||||||
|
disabled={busy}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
{error && <div className="error-banner">{error}</div>}
|
||||||
|
<div className="modal-actions">
|
||||||
|
<button type="button" className="btn" onClick={onClose} disabled={busy}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-primary" disabled={busy}>
|
||||||
|
{busy ? 'Importing…' : 'Import'}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</Modal>
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
import { useEffect, useState } from 'react'
|
||||||
|
import { Modal } from './Modal'
|
||||||
|
import { StatusBadge } from './StatusBadge'
|
||||||
|
import { listRuns, type Run } from '../api'
|
||||||
|
|
||||||
|
const fmt = (iso: string | null) => (iso ? new Date(iso).toLocaleString() : '—')
|
||||||
|
|
||||||
|
export function RunLogModal({ taskId, open, onClose }: { taskId: number; open: boolean; onClose: () => void }) {
|
||||||
|
const [runs, setRuns] = useState<Run[] | null>(null)
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!open) return
|
||||||
|
setRuns(null)
|
||||||
|
listRuns(taskId).then((r) => setRuns(r ?? [])).catch(() => setRuns([]))
|
||||||
|
}, [open, taskId])
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Modal open={open} title="Run log" onClose={onClose} size="lg">
|
||||||
|
<div className="tbl-wrap">
|
||||||
|
<table className="tbl">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Started</th>
|
||||||
|
<th>Finished</th>
|
||||||
|
<th>Trigger</th>
|
||||||
|
<th>Status</th>
|
||||||
|
<th>Copied</th>
|
||||||
|
<th>Skipped</th>
|
||||||
|
<th>Errors</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{runs === null ? (
|
||||||
|
<tr className="empty-row"><td colSpan={7}>loading…</td></tr>
|
||||||
|
) : runs.length === 0 ? (
|
||||||
|
<tr className="empty-row"><td colSpan={7}>no runs yet</td></tr>
|
||||||
|
) : (
|
||||||
|
runs.map((r) => (
|
||||||
|
<tr key={r.id}>
|
||||||
|
<td>{fmt(r.started_at)}</td>
|
||||||
|
<td>{fmt(r.finished_at)}</td>
|
||||||
|
<td>{r.trigger}</td>
|
||||||
|
<td><StatusBadge status={r.status} /></td>
|
||||||
|
<td className="num-cell">{r.total_copied}</td>
|
||||||
|
<td className="num-cell">{r.total_skipped}</td>
|
||||||
|
<td className="num-cell">{r.total_errors}</td>
|
||||||
|
</tr>
|
||||||
|
))
|
||||||
|
)}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</Modal>
|
||||||
|
)
|
||||||
|
}
|
||||||
+4
-2
@@ -12,8 +12,8 @@
|
|||||||
--border: #23342b;
|
--border: #23342b;
|
||||||
--border-bright: #3a5443;
|
--border-bright: #3a5443;
|
||||||
--fg: #dbe8de;
|
--fg: #dbe8de;
|
||||||
--fg-dim: #6f8478;
|
--fg-dim: #7a8f82;
|
||||||
--fg-faint: #4a5c50;
|
--fg-faint: #6f8478;
|
||||||
--accent: #ffb238;
|
--accent: #ffb238;
|
||||||
--accent-strong: #ffd27a;
|
--accent-strong: #ffd27a;
|
||||||
--accent-dim: #7a5a26;
|
--accent-dim: #7a5a26;
|
||||||
@@ -22,7 +22,9 @@
|
|||||||
--fail: #ff5d5d;
|
--fail: #ff5d5d;
|
||||||
--fail-dim: #4a2323;
|
--fail-dim: #4a2323;
|
||||||
--pending: #f0c419;
|
--pending: #f0c419;
|
||||||
|
--pending-dim: #4a4423;
|
||||||
--info: #57c2ff;
|
--info: #57c2ff;
|
||||||
|
--info-dim: #234456;
|
||||||
|
|
||||||
--font-display: 'Big Shoulders Display', 'Arial Narrow', sans-serif;
|
--font-display: 'Big Shoulders Display', 'Arial Narrow', sans-serif;
|
||||||
--font-mono: 'JetBrains Mono', ui-monospace, monospace;
|
--font-mono: 'JetBrains Mono', ui-monospace, monospace;
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { useEffect, useState, type FormEvent } from 'react'
|
import { useEffect, useState, type FormEvent } from 'react'
|
||||||
import { createEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
|
import { createEndpoint, deleteEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
|
||||||
|
import { useConfirm } from '../components/ConfirmProvider'
|
||||||
|
|
||||||
const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode }
|
const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode }
|
||||||
|
|
||||||
@@ -9,6 +10,7 @@ export function Endpoints() {
|
|||||||
const [editingId, setEditingId] = useState<number | null>(null)
|
const [editingId, setEditingId] = useState<number | null>(null)
|
||||||
const [error, setError] = useState<string | null>(null)
|
const [error, setError] = useState<string | null>(null)
|
||||||
const [busy, setBusy] = useState(false)
|
const [busy, setBusy] = useState(false)
|
||||||
|
const confirm = useConfirm()
|
||||||
|
|
||||||
function startEdit(ep: Endpoint) {
|
function startEdit(ep: Endpoint) {
|
||||||
setEditingId(ep.id)
|
setEditingId(ep.id)
|
||||||
@@ -29,6 +31,25 @@ export function Endpoints() {
|
|||||||
|
|
||||||
useEffect(reload, [])
|
useEffect(reload, [])
|
||||||
|
|
||||||
|
async function onDelete(ep: Endpoint) {
|
||||||
|
const ok = await confirm({
|
||||||
|
title: 'Delete endpoint',
|
||||||
|
message: `Delete endpoint "${ep.role_label}" (${ep.host}:${ep.port})?`,
|
||||||
|
confirmLabel: 'Delete',
|
||||||
|
danger: true,
|
||||||
|
})
|
||||||
|
if (!ok) return
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await deleteEndpoint(ep.id)
|
||||||
|
// The form still edits a row that no longer exists — drop back to create mode.
|
||||||
|
if (editingId === ep.id) cancelEdit()
|
||||||
|
reload()
|
||||||
|
} catch (e) {
|
||||||
|
setError(e instanceof Error ? e.message : 'Failed to delete endpoint')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function submit(e: FormEvent) {
|
async function submit(e: FormEvent) {
|
||||||
e.preventDefault()
|
e.preventDefault()
|
||||||
setBusy(true)
|
setBusy(true)
|
||||||
@@ -111,7 +132,7 @@ export function Endpoints() {
|
|||||||
</select>
|
</select>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{error && <div className="error-banner">{error}</div>}
|
{error && <div className="error-banner" role="alert">{error}</div>}
|
||||||
<div className="btn-row">
|
<div className="btn-row">
|
||||||
<button className="btn btn-primary" disabled={busy}>
|
<button className="btn btn-primary" disabled={busy}>
|
||||||
{busy ? 'Saving…' : editingId !== null ? 'Save changes' : 'Add endpoint'}
|
{busy ? 'Saving…' : editingId !== null ? 'Save changes' : 'Add endpoint'}
|
||||||
@@ -159,6 +180,9 @@ export function Endpoints() {
|
|||||||
<td className="num-cell">
|
<td className="num-cell">
|
||||||
<button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}>
|
<button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}>
|
||||||
edit
|
edit
|
||||||
|
</button>{' '}
|
||||||
|
<button type="button" className="link-btn danger" onClick={() => onDelete(ep)} disabled={busy}>
|
||||||
|
delete
|
||||||
</button>
|
</button>
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ export function Login({ onSuccess }: { onSuccess: () => void }) {
|
|||||||
<button className="btn btn-primary" style={{ width: '100%' }} disabled={busy || !user || !pass}>
|
<button className="btn btn-primary" style={{ width: '100%' }} disabled={busy || !user || !pass}>
|
||||||
{busy ? 'Authenticating…' : 'Sign in'}
|
{busy ? 'Authenticating…' : 'Sign in'}
|
||||||
</button>
|
</button>
|
||||||
<div className="login-error">{error}</div>
|
<div className="login-error" role="alert">{error}</div>
|
||||||
</form>
|
</form>
|
||||||
</div>
|
</div>
|
||||||
)
|
)
|
||||||
|
|||||||
+425
-52
@@ -1,9 +1,13 @@
|
|||||||
import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react'
|
import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react'
|
||||||
import { cancelAccount, createAccount, deleteAccount, getTask, importCSV, probeFolders, runTask, setFolderMapping, testAccounts, type TaskDetail as TaskDetailData } from '../api'
|
import { cancelAccount, cancelTask, createAccount, deleteAccount, getTask, importCSV, importKerioCSV, pauseTask, probeAccountFolders, probeFolders, resumeTask, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, updateAccountCredentials, type Account, type TaskDetail as TaskDetailData } from '../api'
|
||||||
import { connectTaskWS, type TaskEvent } from '../ws'
|
import { connectTaskWS, type TaskEvent } from '../ws'
|
||||||
import { StatusBadge } from '../components/StatusBadge'
|
import { StatusBadge } from '../components/StatusBadge'
|
||||||
import { useConfirm } from '../components/ConfirmProvider'
|
import { useConfirm } from '../components/ConfirmProvider'
|
||||||
import { FolderMappingModal } from '../components/FolderMappingModal'
|
import { FolderMappingModal } from '../components/FolderMappingModal'
|
||||||
|
import { RunLogModal } from '../components/RunLogModal'
|
||||||
|
import { AccountErrorsModal } from '../components/AccountErrorsModal'
|
||||||
|
import { KerioImportModal } from '../components/KerioImportModal'
|
||||||
|
import { AccountCredentialsModal } from '../components/AccountCredentialsModal'
|
||||||
|
|
||||||
const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' }
|
const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' }
|
||||||
|
|
||||||
@@ -55,6 +59,8 @@ function describeEvent(ev: TaskEvent): string {
|
|||||||
}
|
}
|
||||||
case 'cancelled':
|
case 'cancelled':
|
||||||
return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}`
|
return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}`
|
||||||
|
case 'paused':
|
||||||
|
return `PAUSED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0} — resumable`
|
||||||
case 'error': {
|
case 'error': {
|
||||||
const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : ''
|
const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : ''
|
||||||
return `ERROR #${d.account_id}${where}: ${d.error}`
|
return `ERROR #${d.account_id}${where}: ${d.error}`
|
||||||
@@ -62,7 +68,7 @@ function describeEvent(ev: TaskEvent): string {
|
|||||||
case 'run_started':
|
case 'run_started':
|
||||||
return `RUN started (run #${d.run_id})`
|
return `RUN started (run #${d.run_id})`
|
||||||
case 'run_done':
|
case 'run_done':
|
||||||
return `RUN finished: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
|
return `RUN ${String(d.status ?? 'finished')}: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
|
||||||
default:
|
default:
|
||||||
return JSON.stringify(ev.data)
|
return JSON.stringify(ev.data)
|
||||||
}
|
}
|
||||||
@@ -75,9 +81,22 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
const [form, setForm] = useState(emptyAccount)
|
const [form, setForm] = useState(emptyAccount)
|
||||||
const [busy, setBusy] = useState<'test' | 'run' | 'add' | 'import' | 'delete' | 'probe' | null>(null)
|
const [busy, setBusy] = useState<'test' | 'run' | 'add' | 'import' | 'delete' | 'probe' | null>(null)
|
||||||
const [mapState, setMapState] = useState<{ src: string[]; dst: string[]; creds: typeof emptyAccount } | null>(null)
|
const [mapState, setMapState] = useState<{ src: string[]; dst: string[]; creds: typeof emptyAccount } | null>(null)
|
||||||
|
const [editMap, setEditMap] = useState<{
|
||||||
|
accId: number
|
||||||
|
label: string
|
||||||
|
src: string[]
|
||||||
|
dst: string[]
|
||||||
|
mapping: Record<string, string>
|
||||||
|
excluded: string[]
|
||||||
|
} | null>(null)
|
||||||
const confirm = useConfirm()
|
const confirm = useConfirm()
|
||||||
const [error, setError] = useState<string | null>(null)
|
const [error, setError] = useState<string | null>(null)
|
||||||
const [live, setLive] = useState<Record<number, LiveProgress>>({})
|
const [live, setLive] = useState<Record<number, LiveProgress>>({})
|
||||||
|
const [showRuns, setShowRuns] = useState(false)
|
||||||
|
const [errorsFor, setErrorsFor] = useState<{ id: number; src_login: string } | null>(null)
|
||||||
|
const [kerioOpen, setKerioOpen] = useState(false)
|
||||||
|
const [credsFor, setCredsFor] = useState<Account | null>(null)
|
||||||
|
const [selected, setSelected] = useState<Set<number>>(new Set())
|
||||||
const fileInputRef = useRef<HTMLInputElement>(null)
|
const fileInputRef = useRef<HTMLInputElement>(null)
|
||||||
|
|
||||||
function reload() {
|
function reload() {
|
||||||
@@ -146,7 +165,7 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || (ev.type === 'error' && d.folder == null))) {
|
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || ev.type === 'paused' || (ev.type === 'error' && d.folder == null))) {
|
||||||
// terminal/reset for this account — drop live overlay, fall back to DB
|
// terminal/reset for this account — drop live overlay, fall back to DB
|
||||||
setLive((prev) => {
|
setLive((prev) => {
|
||||||
if (!(accId in prev)) return prev
|
if (!(accId in prev)) return prev
|
||||||
@@ -157,7 +176,7 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Structural events refresh the persisted view; `progress` is covered by live state.
|
// Structural events refresh the persisted view; `progress` is covered by live state.
|
||||||
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'plan'].includes(ev.type)) {
|
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'paused', 'plan', 'task_broken'].includes(ev.type)) {
|
||||||
reload()
|
reload()
|
||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
@@ -186,13 +205,13 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function confirmMapping(mapping: Record<string, string>) {
|
async function confirmMapping(mapping: Record<string, string>, excluded: string[]) {
|
||||||
if (!mapState) return
|
if (!mapState) return
|
||||||
setBusy('add')
|
setBusy('add')
|
||||||
setError(null)
|
setError(null)
|
||||||
try {
|
try {
|
||||||
await createAccount(id, mapState.creds)
|
const { id: accId } = await createAccount(id, mapState.creds)
|
||||||
await setFolderMapping(id, mapping)
|
await setAccountFolderMapping(id, accId, mapping, excluded)
|
||||||
setForm(emptyAccount)
|
setForm(emptyAccount)
|
||||||
setMapState(null)
|
setMapState(null)
|
||||||
reload()
|
reload()
|
||||||
@@ -203,20 +222,82 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function downloadExampleCSV() {
|
async function onEditFolders(a: { id: number; src_login: string; dst_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) {
|
||||||
const sample = [
|
setBusy('probe')
|
||||||
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
|
setError(null)
|
||||||
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
|
try {
|
||||||
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
|
const res = await probeAccountFolders(id, a.id)
|
||||||
].join('\n') + '\n'
|
if (!res.src.ok || !res.dst.ok) {
|
||||||
const url = URL.createObjectURL(new Blob([sample], { type: 'text/csv' }))
|
const parts: string[] = []
|
||||||
|
if (!res.src.ok) parts.push(`source: ${res.src.error ?? 'login failed'}`)
|
||||||
|
if (!res.dst.ok) parts.push(`destination: ${res.dst.error ?? 'login failed'}`)
|
||||||
|
setError(`Folder probe failed — ${parts.join('; ')}`)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
setEditMap({
|
||||||
|
accId: a.id,
|
||||||
|
label: a.dst_login && a.dst_login !== a.src_login ? `${a.src_login} → ${a.dst_login}` : a.src_login,
|
||||||
|
src: res.src.folders ?? [],
|
||||||
|
dst: res.dst.folders ?? [],
|
||||||
|
mapping: a.folder_mapping ?? {},
|
||||||
|
excluded: a.excluded_folders ?? [],
|
||||||
|
})
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to probe folders')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveEditMapping(mapping: Record<string, string>, excluded: string[]) {
|
||||||
|
if (!editMap) return
|
||||||
|
setBusy('add')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await setAccountFolderMapping(id, editMap.accId, mapping, excluded)
|
||||||
|
setEditMap(null)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to save mapping')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function downloadCSV(name: string, content: string) {
|
||||||
|
const url = URL.createObjectURL(new Blob([content], { type: 'text/csv' }))
|
||||||
const a = document.createElement('a')
|
const a = document.createElement('a')
|
||||||
a.href = url
|
a.href = url
|
||||||
a.download = 'imap-copier-accounts-example.csv'
|
a.download = name
|
||||||
a.click()
|
a.click()
|
||||||
URL.revokeObjectURL(url)
|
URL.revokeObjectURL(url)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Plain import: comma-separated, no header, src_login,src_pass,dst_login,dst_pass.
|
||||||
|
function downloadExampleCSV() {
|
||||||
|
const sample =
|
||||||
|
[
|
||||||
|
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
|
||||||
|
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
|
||||||
|
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
|
||||||
|
].join('\n') + '\n'
|
||||||
|
downloadCSV('imap-copier-accounts-example.csv', sample)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Kerio Connect export: semicolon-separated with a Name;FullName;Description;Enable
|
||||||
|
// header. The password lives in Description; disabled rows and admin are skipped
|
||||||
|
// on import, and the domain is supplied in the import dialog.
|
||||||
|
function downloadKerioExampleCSV() {
|
||||||
|
const sample =
|
||||||
|
[
|
||||||
|
'Name;FullName;Description;Enable',
|
||||||
|
'alice;Alice Smith;SrcPass1;Yes',
|
||||||
|
'bob;Bob Jones;SrcPass2;Yes',
|
||||||
|
'carol;Carol White (disabled, skipped);SrcPass3;No',
|
||||||
|
].join('\n') + '\n'
|
||||||
|
downloadCSV('kerio-users-example.csv', sample)
|
||||||
|
}
|
||||||
|
|
||||||
async function onFileChosen(e: ChangeEvent<HTMLInputElement>) {
|
async function onFileChosen(e: ChangeEvent<HTMLInputElement>) {
|
||||||
const file = e.target.files?.[0]
|
const file = e.target.files?.[0]
|
||||||
if (!file) return
|
if (!file) return
|
||||||
@@ -233,6 +314,35 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function onKerioImport(file: File, domain: string) {
|
||||||
|
setBusy('import')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await importKerioCSV(id, file, domain)
|
||||||
|
setKerioOpen(false)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Kerio import failed')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveCredentials(body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) {
|
||||||
|
if (!credsFor) return
|
||||||
|
setBusy('add')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await updateAccountCredentials(id, credsFor.id, body)
|
||||||
|
setCredsFor(null)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to save credentials')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function onDeleteAccount(accId: number, login: string) {
|
async function onDeleteAccount(accId: number, login: string) {
|
||||||
const ok = await confirm({
|
const ok = await confirm({
|
||||||
title: 'Remove account',
|
title: 'Remove account',
|
||||||
@@ -278,7 +388,8 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
setBusy('run')
|
setBusy('run')
|
||||||
setError(null)
|
setError(null)
|
||||||
try {
|
try {
|
||||||
await runTask(id)
|
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
|
||||||
|
await runTask(id, ids.length ? ids : undefined)
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
setError(err instanceof Error ? err.message : 'Failed to start run')
|
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||||
} finally {
|
} finally {
|
||||||
@@ -286,6 +397,61 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function onPause() {
|
||||||
|
setBusy('run')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await pauseTask(id)
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to pause the run')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onCancelRun() {
|
||||||
|
const ok = await confirm({
|
||||||
|
title: 'Cancel migration',
|
||||||
|
message: 'Stop the run and mark every unfinished account as cancelled? Copied messages are kept.',
|
||||||
|
confirmLabel: 'Cancel migration',
|
||||||
|
cancelLabel: 'Keep running',
|
||||||
|
danger: true,
|
||||||
|
})
|
||||||
|
if (!ok) return
|
||||||
|
setBusy('run')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await cancelTask(id)
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to cancel the run')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onResume() {
|
||||||
|
setBusy('run')
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await resumeTask(id)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to resume the run')
|
||||||
|
} finally {
|
||||||
|
setBusy(null)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onSchedule(intervalSeconds: number) {
|
||||||
|
setError(null)
|
||||||
|
try {
|
||||||
|
await setTaskSchedule(id, intervalSeconds)
|
||||||
|
reload()
|
||||||
|
} catch (err) {
|
||||||
|
setError(err instanceof Error ? err.message : 'Failed to set schedule')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (notFound) {
|
if (notFound) {
|
||||||
return (
|
return (
|
||||||
<div className="panel">
|
<div className="panel">
|
||||||
@@ -302,7 +468,51 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
const { task, accounts } = data
|
const { task, accounts } = data
|
||||||
const allTested = accounts.length > 0 && accounts.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
const isRunning = task.status === 'running'
|
||||||
|
// Accounts a pause left unfinished — what Resume picks up.
|
||||||
|
const pausedCount = accounts.filter((a) => a.status === 'paused').length
|
||||||
|
|
||||||
|
// A failed connection test is the entry point for fixing the credentials that
|
||||||
|
// caused it — an imported account is otherwise only deletable.
|
||||||
|
const testCell = (a: Account, status: string) =>
|
||||||
|
status === 'fail' && !isRunning && a.status !== 'running' ? (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="badge-btn"
|
||||||
|
title="Edit credentials for this account"
|
||||||
|
onClick={() => setCredsFor(a)}
|
||||||
|
>
|
||||||
|
<StatusBadge status={status} />
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<StatusBadge status={status} />
|
||||||
|
)
|
||||||
|
// A row is selectable only when both connection tests pass and no run is live.
|
||||||
|
const selectableIds = accounts
|
||||||
|
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||||
|
.map((a) => a.id)
|
||||||
|
const selectableSet = new Set(selectableIds)
|
||||||
|
// Effective set: the checked accounts, or all accounts when nothing is checked.
|
||||||
|
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
|
||||||
|
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
|
||||||
|
const runReady =
|
||||||
|
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||||
|
const allSelectableChecked =
|
||||||
|
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
|
||||||
|
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
|
||||||
|
|
||||||
|
function toggleOne(accId: number, checked: boolean) {
|
||||||
|
setSelected((prev) => {
|
||||||
|
const next = new Set(prev)
|
||||||
|
if (checked) next.add(accId)
|
||||||
|
else next.delete(accId)
|
||||||
|
return next
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
function toggleAll(checked: boolean) {
|
||||||
|
setSelected(checked ? new Set(selectableIds) : new Set())
|
||||||
|
}
|
||||||
// Prefer live (WS) copied/skipped over the DB values, which only advance per
|
// Prefer live (WS) copied/skipped over the DB values, which only advance per
|
||||||
// folder — so the summary moves in real time during a large folder.
|
// folder — so the summary moves in real time during a large folder.
|
||||||
const totals = accounts.reduce(
|
const totals = accounts.reduce(
|
||||||
@@ -326,11 +536,12 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
</h1>
|
</h1>
|
||||||
</div>
|
</div>
|
||||||
<StatusBadge status={task.status} />
|
<StatusBadge status={task.status} />
|
||||||
|
{task.broken && <span className="badge badge-fail" style={{ marginLeft: 8 }}><span className="dot" />broken</span>}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="panel">
|
<div className="panel">
|
||||||
<span className="panel-label">Run control</span>
|
<span className="panel-label">Run control</span>
|
||||||
<div className="stat-row">
|
<div className="stat-row" aria-live="polite">
|
||||||
<div className="stat ok">
|
<div className="stat ok">
|
||||||
<span className="val mono-num">{totals.copied}</span>
|
<span className="val mono-num">{totals.copied}</span>
|
||||||
<span className="lbl">copied</span>
|
<span className="lbl">copied</span>
|
||||||
@@ -348,15 +559,70 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
<span className="lbl">accounts</span>
|
<span className="lbl">accounts</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
{error && <div className="error-banner">{error}</div>}
|
{error && <div className="error-banner" role="alert">{error}</div>}
|
||||||
<div className="btn-row" style={{ marginTop: 16 }}>
|
<div className="btn-row" style={{ marginTop: 16 }}>
|
||||||
<button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}>
|
<button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}>
|
||||||
{busy === 'test' ? 'Testing…' : 'Test connections'}
|
{busy === 'test' ? 'Testing…' : 'Test connections'}
|
||||||
</button>
|
</button>
|
||||||
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
|
{isRunning ? (
|
||||||
{busy === 'run' ? 'Starting…' : 'Run migration'}
|
<>
|
||||||
|
<button className="btn" onClick={onPause} disabled={busy !== null}>
|
||||||
|
{busy === 'run' ? 'Stopping…' : 'Pause'}
|
||||||
|
</button>
|
||||||
|
<button className="btn btn-danger" onClick={onCancelRun} disabled={busy !== null}>
|
||||||
|
Cancel
|
||||||
|
</button>
|
||||||
|
<span className="hint">pause keeps the unfinished accounts resumable</span>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
{pausedCount > 0 && (
|
||||||
|
<button className="btn btn-primary" onClick={onResume} disabled={busy !== null}>
|
||||||
|
{busy === 'run' ? 'Resuming…' : `Resume (${pausedCount})`}
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
<button
|
||||||
|
className={pausedCount > 0 ? 'btn' : 'btn btn-primary'}
|
||||||
|
onClick={onRun}
|
||||||
|
disabled={busy !== null || !runReady}
|
||||||
|
>
|
||||||
|
{busy === 'run'
|
||||||
|
? 'Starting…'
|
||||||
|
: effectiveSelected.length > 0
|
||||||
|
? `Run selected (${effectiveSelected.length})`
|
||||||
|
: 'Run migration'}
|
||||||
|
</button>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
{!isRunning && !runReady && accounts.length > 0 && (
|
||||||
|
<span className="hint">
|
||||||
|
{effectiveSelected.length > 0
|
||||||
|
? 'selected accounts must pass both connection tests'
|
||||||
|
: 'run unlocks once every account tests OK on both sides'}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
<div className="sched-row">
|
||||||
|
<label htmlFor="sched">Schedule</label>
|
||||||
|
<select
|
||||||
|
id="sched"
|
||||||
|
value={task.schedule_interval_seconds ?? 0}
|
||||||
|
onChange={(e) => onSchedule(Number(e.target.value))}
|
||||||
|
>
|
||||||
|
<option value={0}>Off</option>
|
||||||
|
<option value={3600}>Every 1h</option>
|
||||||
|
<option value={10800}>Every 3h</option>
|
||||||
|
<option value={21600}>Every 6h</option>
|
||||||
|
<option value={43200}>Every 12h</option>
|
||||||
|
<option value={86400}>Every 24h</option>
|
||||||
|
</select>
|
||||||
|
{task.next_run_at && (
|
||||||
|
<span className="sched-next">Next run: {new Date(task.next_run_at).toLocaleString()}</span>
|
||||||
|
)}
|
||||||
|
{task.broken && <span className="badge badge-fail"><span className="dot" />broken</span>}
|
||||||
|
<button type="button" className="link-btn" onClick={() => setShowRuns(true)}>
|
||||||
|
runs
|
||||||
</button>
|
</button>
|
||||||
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
|
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -415,13 +681,23 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
|
|
||||||
<div className="divider-label">or bulk import</div>
|
<div className="divider-label">or bulk import</div>
|
||||||
<div className="upload-row">
|
<div className="upload-row">
|
||||||
<button className="btn file-btn" disabled={busy !== null}>
|
<div className="upload-item">
|
||||||
{busy === 'import' ? 'Importing…' : 'Upload CSV'}
|
<label className={`btn file-btn${busy !== null ? ' is-disabled' : ''}`}>
|
||||||
<input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} />
|
{busy === 'import' ? 'Importing…' : 'Upload CSV'}
|
||||||
</button>
|
<input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} />
|
||||||
<button type="button" className="link-btn" onClick={downloadExampleCSV}>
|
</label>
|
||||||
download example.csv
|
<button type="button" className="link-btn" onClick={downloadExampleCSV}>
|
||||||
</button>
|
download example.csv
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
<div className="upload-item">
|
||||||
|
<button type="button" className="btn" onClick={() => setKerioOpen(true)} disabled={busy !== null}>
|
||||||
|
Import from Kerio
|
||||||
|
</button>
|
||||||
|
<button type="button" className="link-btn" onClick={downloadKerioExampleCSV}>
|
||||||
|
download kerio example.csv
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
@@ -432,7 +708,7 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
clear log
|
clear log
|
||||||
</button>
|
</button>
|
||||||
)}
|
)}
|
||||||
<div className="log-pane">
|
<div className="log-pane" role="log" aria-live="polite" aria-relevant="additions" aria-label="Event log">
|
||||||
{log.length === 0 ? (
|
{log.length === 0 ? (
|
||||||
<div className="log-empty">awaiting events over websocket…</div>
|
<div className="log-empty">awaiting events over websocket…</div>
|
||||||
) : (
|
) : (
|
||||||
@@ -453,8 +729,19 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
<table className="tbl">
|
<table className="tbl">
|
||||||
<thead>
|
<thead>
|
||||||
<tr>
|
<tr>
|
||||||
<th>Source</th>
|
<th className="chk-col">
|
||||||
<th>Destination</th>
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
aria-label="Select all accounts"
|
||||||
|
checked={allSelectableChecked}
|
||||||
|
ref={(el) => {
|
||||||
|
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
|
||||||
|
}}
|
||||||
|
disabled={isRunning || selectableIds.length === 0}
|
||||||
|
onChange={(e) => toggleAll(e.target.checked)}
|
||||||
|
/>
|
||||||
|
</th>
|
||||||
|
<th>Account</th>
|
||||||
<th>Src test</th>
|
<th>Src test</th>
|
||||||
<th>Dst test</th>
|
<th>Dst test</th>
|
||||||
<th>Status</th>
|
<th>Status</th>
|
||||||
@@ -473,25 +760,34 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
) : (
|
) : (
|
||||||
accounts.map((a) => (
|
accounts.map((a) => (
|
||||||
<tr key={a.id}>
|
<tr key={a.id}>
|
||||||
|
<td className="chk-col">
|
||||||
|
<input
|
||||||
|
type="checkbox"
|
||||||
|
aria-label={`Select ${a.src_login}`}
|
||||||
|
checked={selected.has(a.id)}
|
||||||
|
disabled={isRunning || !selectableSet.has(a.id)}
|
||||||
|
onChange={(e) => toggleOne(a.id, e.target.checked)}
|
||||||
|
/>
|
||||||
|
</td>
|
||||||
<td>
|
<td>
|
||||||
{a.src_login}
|
<div className="acct-ident">
|
||||||
|
<span>{a.src_login}</span>
|
||||||
|
{a.dst_login !== a.src_login && (
|
||||||
|
<span className="acct-dst">→ {a.dst_login}</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
{a.last_error && (
|
{a.last_error && (
|
||||||
<div className="acct-error" title={a.last_error}>
|
<div className="acct-error" title={a.last_error}>
|
||||||
{a.last_error}
|
{a.last_error}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
</td>
|
</td>
|
||||||
<td>{a.dst_login}</td>
|
<td>{testCell(a, a.test_src_status)}</td>
|
||||||
<td>
|
<td>{testCell(a, a.test_dst_status)}</td>
|
||||||
<StatusBadge status={a.test_src_status} />
|
|
||||||
</td>
|
|
||||||
<td>
|
|
||||||
<StatusBadge status={a.test_dst_status} />
|
|
||||||
</td>
|
|
||||||
<td>
|
<td>
|
||||||
<StatusBadge status={a.status} />
|
<StatusBadge status={a.status} />
|
||||||
</td>
|
</td>
|
||||||
<td className="progress-cell">
|
<td className={`progress-cell${a.status === 'running' ? ' progress-cell--live' : ''}`}>
|
||||||
{(() => {
|
{(() => {
|
||||||
const lv = live[a.id]
|
const lv = live[a.id]
|
||||||
if (!lv || !lv.total) return <span className="muted-note">—</span>
|
if (!lv || !lv.total) return <span className="muted-note">—</span>
|
||||||
@@ -501,8 +797,15 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
const scanning = lv.scanned != null && lv.scanTotal != null && lv.scanned < lv.scanTotal
|
const scanning = lv.scanned != null && lv.scanTotal != null && lv.scanned < lv.scanTotal
|
||||||
return (
|
return (
|
||||||
<div className="acct-progress">
|
<div className="acct-progress">
|
||||||
<div className="pbar">
|
<div
|
||||||
<span className="pbar-fill" style={{ width: `${pct}%` }} />
|
className="pbar"
|
||||||
|
role="progressbar"
|
||||||
|
aria-valuenow={pct}
|
||||||
|
aria-valuemin={0}
|
||||||
|
aria-valuemax={100}
|
||||||
|
aria-label={`Copy progress${lv.folder ? `: ${lv.folder}` : ''}`}
|
||||||
|
>
|
||||||
|
<span className="pbar-fill" style={{ transform: `scaleX(${pct / 100})` }} />
|
||||||
</div>
|
</div>
|
||||||
<span className="pmeta mono-num">
|
<span className="pmeta mono-num">
|
||||||
{done}/{lv.total} ({pct}%) · {lv.speed >= 1 ? Math.round(lv.speed) : lv.speed.toFixed(1)}/s · ETA {fmtDuration(eta)}
|
{done}/{lv.total} ({pct}%) · {lv.speed >= 1 ? Math.round(lv.speed) : lv.speed.toFixed(1)}/s · ETA {fmtDuration(eta)}
|
||||||
@@ -519,23 +822,59 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
</td>
|
</td>
|
||||||
<td className="num-cell">{live[a.id]?.copied ?? a.copied}</td>
|
<td className="num-cell">{live[a.id]?.copied ?? a.copied}</td>
|
||||||
<td className="num-cell">{live[a.id]?.skipped ?? a.skipped}</td>
|
<td className="num-cell">{live[a.id]?.skipped ?? a.skipped}</td>
|
||||||
<td className="num-cell">{a.errors}</td>
|
|
||||||
<td className="num-cell">
|
<td className="num-cell">
|
||||||
{a.status === 'running' ? (
|
{a.errors > 0 ? (
|
||||||
<button type="button" className="link-btn danger" onClick={() => onCancelAccount(a.id)}>
|
|
||||||
cancel
|
|
||||||
</button>
|
|
||||||
) : (
|
|
||||||
<button
|
<button
|
||||||
type="button"
|
type="button"
|
||||||
className="link-btn danger"
|
className="link-btn danger"
|
||||||
onClick={() => onDeleteAccount(a.id, a.src_login)}
|
onClick={() => setErrorsFor({ id: a.id, src_login: a.src_login })}
|
||||||
disabled={busy !== null || data?.task.status === 'running'}
|
|
||||||
>
|
>
|
||||||
remove
|
{a.errors}
|
||||||
</button>
|
</button>
|
||||||
|
) : (
|
||||||
|
a.errors
|
||||||
)}
|
)}
|
||||||
</td>
|
</td>
|
||||||
|
<td className="num-cell">
|
||||||
|
<div className="row-actions">
|
||||||
|
{(() => {
|
||||||
|
const mapped =
|
||||||
|
Object.keys(a.folder_mapping ?? {}).length > 0 ||
|
||||||
|
(a.excluded_folders?.length ?? 0) > 0
|
||||||
|
return mapped ? (
|
||||||
|
<span
|
||||||
|
className="map-dot"
|
||||||
|
title="Folders already mapped for this account"
|
||||||
|
aria-label="Folders mapped"
|
||||||
|
/>
|
||||||
|
) : null
|
||||||
|
})()}
|
||||||
|
{a.status !== 'running' && data?.task.status !== 'running' && (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="link-btn"
|
||||||
|
onClick={() => onEditFolders(a)}
|
||||||
|
disabled={busy !== null}
|
||||||
|
>
|
||||||
|
folders
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
{a.status === 'running' ? (
|
||||||
|
<button type="button" className="link-btn danger" onClick={() => onCancelAccount(a.id)}>
|
||||||
|
cancel
|
||||||
|
</button>
|
||||||
|
) : (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
className="link-btn danger"
|
||||||
|
onClick={() => onDeleteAccount(a.id, a.src_login)}
|
||||||
|
disabled={busy !== null || data?.task.status === 'running'}
|
||||||
|
>
|
||||||
|
remove
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
))
|
))
|
||||||
)}
|
)}
|
||||||
@@ -551,10 +890,44 @@ export function TaskDetail({ id }: { id: number }) {
|
|||||||
srcFolders={mapState.src}
|
srcFolders={mapState.src}
|
||||||
dstFolders={mapState.dst}
|
dstFolders={mapState.dst}
|
||||||
initialMapping={task.folder_mapping ?? {}}
|
initialMapping={task.folder_mapping ?? {}}
|
||||||
|
initialExcluded={[]}
|
||||||
|
accountLabel={
|
||||||
|
mapState.creds.dst_login && mapState.creds.dst_login !== mapState.creds.src_login
|
||||||
|
? `${mapState.creds.src_login} → ${mapState.creds.dst_login}`
|
||||||
|
: mapState.creds.src_login
|
||||||
|
}
|
||||||
onCancel={() => setMapState(null)}
|
onCancel={() => setMapState(null)}
|
||||||
onConfirm={confirmMapping}
|
onConfirm={confirmMapping}
|
||||||
/>
|
/>
|
||||||
)}
|
)}
|
||||||
|
{editMap && (
|
||||||
|
<FolderMappingModal
|
||||||
|
key={`edit-${editMap.accId}`}
|
||||||
|
open
|
||||||
|
srcFolders={editMap.src}
|
||||||
|
dstFolders={editMap.dst}
|
||||||
|
initialMapping={editMap.mapping}
|
||||||
|
initialExcluded={editMap.excluded}
|
||||||
|
accountLabel={editMap.label}
|
||||||
|
onCancel={() => setEditMap(null)}
|
||||||
|
onConfirm={saveEditMapping}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
<RunLogModal taskId={id} open={showRuns} onClose={() => setShowRuns(false)} />
|
||||||
|
<AccountErrorsModal taskId={id} account={errorsFor} onClose={() => setErrorsFor(null)} />
|
||||||
|
<KerioImportModal
|
||||||
|
open={kerioOpen}
|
||||||
|
busy={busy === 'import'}
|
||||||
|
onClose={() => setKerioOpen(false)}
|
||||||
|
onSubmit={onKerioImport}
|
||||||
|
/>
|
||||||
|
<AccountCredentialsModal
|
||||||
|
open={credsFor !== null}
|
||||||
|
busy={busy === 'add'}
|
||||||
|
account={credsFor}
|
||||||
|
onClose={() => setCredsFor(null)}
|
||||||
|
onSubmit={saveCredentials}
|
||||||
|
/>
|
||||||
</>
|
</>
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -163,6 +163,7 @@ export function Tasks() {
|
|||||||
</td>
|
</td>
|
||||||
<td>
|
<td>
|
||||||
<StatusBadge status={t.status} />
|
<StatusBadge status={t.status} />
|
||||||
|
{t.broken && <span className="badge badge-fail" style={{ marginLeft: 8 }}><span className="dot" />broken</span>}
|
||||||
</td>
|
</td>
|
||||||
<td className="num-cell">
|
<td className="num-cell">
|
||||||
<button
|
<button
|
||||||
|
|||||||
Reference in New Issue
Block a user