Compare commits

..
40 Commits
Author SHA1 Message Date
vasyanskandClaude Opus 5 0b12d2ed2d Cover pause and resume in the e2e run
The idempotency assertion compared counter deltas between the two runs,
but per-run counters are reset at the start of every run, so the second
run's row already showed that run alone — the delta was negative and the
script failed before reaching anything else.

With that fixed, a second account of 3000 messages exercises the new
stop path end to end: pause mid-folder, assert the task and the account
settle into paused, resume, and assert the resumed run covers every
message while skipping the ones the paused stretch had already copied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:21:02 +07:00
vasyanskandClaude Opus 5 039ac2f1da Add pause and cancel to a running migration
A live run could only be stopped one account at a time, and stopping it
at all meant losing the queue: the accounts that had not started yet
stayed idle with no record that they were meant to run.

A run now carries a handle holding the context that stops every account
under it plus the reason it was stopped. Pause and cancel take the same
path and differ only in the status left behind — paused accounts are
what Resume re-runs, and the migration journal makes each one continue
where it stopped instead of re-copying. Accounts still queued when the
stop lands get the same status as the interrupted ones, so the whole
remainder is resumable after a pause and cancelled after a cancel.
Database writes keep using the uncancellable context, so statuses and
counters survive the stop.

The scheduler skips paused tasks: auto-starting a full run would defeat
the pause. An operator stopping a run no longer trips the schedule
breaker either — that is for failures, not for intent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:20:52 +07:00
vasyanskandClaude Opus 5 c077b368f3 Allow fixing an account's credentials from a failed test
An imported account with a wrong password could only be deleted and
re-added, which loses its folder mapping and its migration journal.
Clicking either FAIL badge now opens a dialog for both logins and both
passwords.

Passwords are never sent to the browser, so the password fields start
empty and an empty field keeps the stored ciphertext — one side can be
corrected without retyping the other. Saving resets both test verdicts
to unknown: they described the previous credentials, and the run gate
requires a passing test on both sides, so the account cannot start on an
unverified password.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 12:08:28 +07:00
vasyanskandClaude Opus 5 76ada57dd7 Add a defaults button to the folder mapping dialog
Mapping the Exchange/Kerio special folders onto mailcow's by hand is
repetitive work that scales with the number of accounts. "By default"
maps Deleted Items to Trash, Junk E-mail to Junk, Sent Items to Sent and
unchecks Public Folders, which mailcow has no counterpart for.

The destination select only offered the source folder as a name to
create, so a target missing on the destination could not be selected at
all. It now also offers the current selection, and marks any name absent
from the destination as "(create)".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:55 +07:00
vasyanskandClaude Opus 5 bb3635e517 Add a Kerio sample file to the bulk import
Only the plain four-column format had a downloadable example, leaving
the Kerio route undocumented in the UI. Each import button now carries
its own sample link underneath: the plain comma-separated layout and a
Kerio export with the Name;FullName;Description;Enable header, a
disabled row included to show what the import skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:45 +07:00
vasyanskandClaude Opus 5 8bc7ff026d Add endpoint deletion
The endpoints screen could only create and edit servers, so a mistyped
or retired endpoint stayed in the list forever.

Tasks reference endpoints without ON DELETE CASCADE, so a referenced
endpoint is refused with 409 and a count of the tasks using it rather
than cascading away migration history. The foreign-key violation is
mapped to the same status to cover a task created between check and
delete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:12 +07:00
vasyanskandClaude Opus 5 94fb410c59 Fix postgres healthcheck probing a nonexistent database
pg_isready without -d connects to a database named after the user, but
the database is imapcopier, so every probe logged a FATAL and the
healthcheck only passed because pg_isready treats "server rejects the
connection" as reachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:02 +07:00
vasyansk 6bf3a6c4ca Add body read timeout recovery to IMAP client
Introduce Client wrapper with socket deadline support
Add reconnection logic for body read timeouts
Implement test cases for underflow scenarios
Update orchestrator to handle reconnections
2026-07-21 05:29:35 +07:00
vasyansk c741cd19a0 Update docker-compose.yml 2026-07-21 04:41:25 +07:00
vasyansk b6e68bdd90 Add activity tracking to prevent stall timeouts during message transfers
Add OnActivity callback to CopyDeps to prevent stall timeouts during large message transfers
Implement touchReader and touchWriter wrappers to call OnActivity during FETCH and APPEND operations
Add slow message logging to identify performance bottlenecks
Add test case to verify activity reporting during message transfers
Clean up orchestrator account reset code formatting
2026-07-21 04:38:32 +07:00
vasyansk d125320667 fix dtt 2026-07-18 12:21:58 +07:00
vasyansk b352cda166 add kerio format support 2026-07-17 11:42:34 +07:00
vasyansk 5d296c39b1 Merge feat/selective-account-run: selective account run via checkboxes 2026-07-08 13:56:28 +07:00
vasyanskandClaude Opus 4.8 fc335dee3f Add selective account run implementation plan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 13:55:02 +07:00
vasyansk b439eee164 Add account selection checkboxes and selective run button 2026-07-08 13:48:48 +07:00
vasyansk 2e5036aad5 runTask accepts optional accountIds 2026-07-08 13:44:53 +07:00
vasyansk 599c66189c Parse optional account_ids in run handler 2026-07-08 13:41:14 +07:00
vasyansk 315e56046b Add selectAccounts filter and accountIDs param to orchestrator.Run 2026-07-08 13:37:24 +07:00
vasyansk 692a83a468 fix domain 2026-07-08 13:34:53 +07:00
vasyansk 6d8471efdd add dick pics 2026-07-08 13:33:14 +07:00
vasyanskandClaude Opus 4.8 3eac1cea7d Add selective account run design spec
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 13:30:11 +07:00
vasyansk b076ac92d6 Add folder mapping status indicator to task list
Add account label to folder mapping modal

Update folder mapping logic to include destination login
2026-07-06 19:15:30 +07:00
vasyansk e84366eb0c Remove idle connection timeout handling
Remove error handling for closed connections

Add progress watchdog to detect stalled accounts

Improve error modal styling and pagination
2026-07-05 14:50:25 +07:00
vasyansk 95ddbf5619 Update copy.go 2026-07-05 13:59:56 +07:00
vasyansk 2bdb4904d6 Add stacked account display and fixed live progress row sizing
Improve account display by showing source/destination in a stacked format
when they differ, and add fixed sizing for live progress rows to prevent
resizing during scans. Also update the table header to use a single
"Account" column.
2026-07-05 12:20:33 +07:00
vasyansk 7ad326fa36 Update docker-compose-traefik.yml 2026-07-05 12:16:44 +07:00
vasyanskandClaude Opus 4.8 45b0ff2358 feat(errors): per-account error modal with persisted error list
Accounts finishing done_with_errors showed only a count and a single
last_error. This adds a modal listing every concrete error of the
account's most recent run.

- migration 0005: account_errors table (kind folder|message|account,
  folder, message_ref, error, created_at; ON DELETE CASCADE; indexed)
- store: AddAccountError / ClearAccountErrors / ListAccountErrors
- copy: OnError callback captures per-message error text (previously
  only counted), with a "UID N: subject" reference
- orchestrator: clear errors at run start; persist folder/message/
  account errors; cap 500 rows/account/run with a suppressed-note row
- api: GET /api/tasks/{id}/accounts/{accountId}/errors
- web: AccountErrorsModal, clickable ERRORS count, api + styles

Verified: migration applies on Postgres 18; store add/list/clear and
cascade tests pass against real pg; backend build/vet/test green; web
tsc+vite build and oxlint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9Eq4JtWjyNTv5qat3B3mM
2026-07-05 12:16:12 +07:00
vasyansk 2623bc8815 Reduce worker concurrency from 4 to 2
Reduce idle read timeout from 120s to 60s

Add batching for IMAP metadata fetches
2026-07-05 11:40:04 +07:00
vasyansk b9cc7749c8 Update docker-compose-traefik.yml 2026-07-05 11:24:44 +07:00
vasyansk d5192e1cf7 Add optional pprof endpoint for debugging 2026-07-04 21:31:53 +07:00
vasyansk dca1363ec9 Add IMAP connection timeout handling
Introduce idleConn wrapper to prevent wedged connections
Add test for silent server timeout behavior
Implement proper TLS handshake and greeting handling
Set explicit dial and read timeouts
2026-07-04 19:37:36 +07:00
vasyansk 8656f46674 Create 2026-07-03-scheduled-task-runs.md 2026-07-03 13:59:12 +07:00
vasyanskandClaude Opus 4.8 e104a77de1 Merge feat/scheduled-task-runs: recurring task execution
A 30s in-process scheduler auto-runs tasks on a recurring interval
(1/3/6/12/24h). Interval is measured from the last run's completion; a task
already running is never re-triggered. Enabling a schedule requires all
accounts tested OK. A breaker disables the schedule and flags the task broken
(red badge in list + detail) when a scheduled run errors or panics. Next-run
time (browser-local) and a run-log modal (per-run status/totals) are shown.
Migration 0004 adds tasks.schedule_interval_seconds/schedule_anchor/broken and
runs.trigger; task delete cascades runs as before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:40:39 +07:00
vasyansk 64bcb1d60b fix: anchor floor on re-enable; breaker on scheduled panic; propagate next-run err; reject negative interval; close phantom runs on startup 2026-07-03 13:31:58 +07:00
vasyansk e8acab6920 feat(web): schedule control, next-run, broken badge, run-log modal 2026-07-03 13:19:14 +07:00
vasyanskandClaude Opus 4.8 d2c69c6a5e feat(api): schedule + runs endpoints; next_run_at on task detail
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:13:10 +07:00
vasyansk be777dc908 feat(orchestrator): thread run trigger; breaker on scheduled runs with errors 2026-07-03 13:08:21 +07:00
vasyansk b9421d388c feat(scheduler): 30s polling scheduler with pure NextRun/dueTaskIDs 2026-07-03 13:07:05 +07:00
vasyansk 8f93dcd97b feat(store): task schedule columns, run trigger, scheduling queries 2026-07-03 13:01:20 +07:00
vasyanskandClaude Opus 4.8 e8f29064fb docs: spec for scheduled (recurring) task runs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 12:40:26 +07:00
63 changed files with 5610 additions and 197 deletions
+5
View File
@@ -6,3 +6,8 @@
!/internal/httpapi/webdist/index.html !/internal/httpapi/webdist/index.html
.DS_Store .DS_Store
# local cache of the impeccable design hook
.impeccable/
**/.impeccable/
*.csv
+10
View File
@@ -4,6 +4,16 @@ Single-binary Go server (embeds the React SPA) that copies IMAP mailboxes
between a source and a destination account. Non-destructive (copy only, between a source and a destination account. Non-destructive (copy only,
never deletes), deduplicated by Message-ID, resumable/idempotent re-runs. never deletes), deduplicated by Message-ID, resumable/idempotent re-runs.
## Screenshots
Task list — create migration jobs and track their status:
![Migration tasks list](docs/SCR-20260708-mbxn.png)
Task detail — run control, per-account credentials, live event log and progress:
![Task detail](docs/SCR-20260708-mcal.png)
## Quick start (plain HTTP on :80) ## Quick start (plain HTTP on :80)
```bash ```bash
+16
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"log/slog" "log/slog"
"net/http" "net/http"
_ "net/http/pprof" // registers /debug/pprof on http.DefaultServeMux (served only on PprofAddr)
"os" "os"
"github.com/golang-migrate/migrate/v4" "github.com/golang-migrate/migrate/v4"
@@ -13,6 +14,7 @@ import (
"github.com/vasyansk/imap-copier/internal/config" "github.com/vasyansk/imap-copier/internal/config"
"github.com/vasyansk/imap-copier/internal/httpapi" "github.com/vasyansk/imap-copier/internal/httpapi"
"github.com/vasyansk/imap-copier/internal/orchestrator" "github.com/vasyansk/imap-copier/internal/orchestrator"
"github.com/vasyansk/imap-copier/internal/scheduler"
"github.com/vasyansk/imap-copier/internal/store" "github.com/vasyansk/imap-copier/internal/store"
"github.com/vasyansk/imap-copier/internal/wshub" "github.com/vasyansk/imap-copier/internal/wshub"
) )
@@ -45,6 +47,20 @@ func main() {
orch := orchestrator.New(st, hub, cfg.EncKey, cfg.WorkerConcurrency) orch := orchestrator.New(st, hub, cfg.EncKey, cfg.WorkerConcurrency)
srv := httpapi.NewServer(cfg, st, orch, hub) srv := httpapi.NewServer(cfg, st, orch, hub)
go scheduler.New(st, orch).Start(context.Background())
// Optional pprof endpoint on a separate, non-published listener so we can
// pull live goroutine dumps (docker compose exec app wget -qO-
// localhost:6060/debug/pprof/goroutine?debug=2) without killing the process.
if cfg.PprofAddr != "" {
go func() {
slog.Info("pprof listening", "addr", cfg.PprofAddr)
if err := http.ListenAndServe(cfg.PprofAddr, nil); err != nil {
slog.Error("pprof serve", "err", err)
}
}()
}
slog.Info("listening", "addr", cfg.HTTPAddr) slog.Info("listening", "addr", cfg.HTTPAddr)
if err := http.ListenAndServe(cfg.HTTPAddr, srv.Router()); err != nil { if err := http.ListenAndServe(cfg.HTTPAddr, srv.Router()); err != nil {
slog.Error("serve", "err", err) slog.Error("serve", "err", err)
+4 -3
View File
@@ -8,7 +8,7 @@ services:
volumes: volumes:
- pgdata:/var/lib/postgresql - pgdata:/var/lib/postgresql
healthcheck: healthcheck:
test: ["CMD-SHELL", "pg_isready -U imap"] test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
interval: 5s interval: 5s
timeout: 3s timeout: 3s
retries: 5 retries: 5
@@ -23,7 +23,8 @@ services:
AUTH_PASS: ${AUTH_PASS} AUTH_PASS: ${AUTH_PASS}
ENC_KEY: ${ENC_KEY} ENC_KEY: ${ENC_KEY}
SESSION_SECRET: ${SESSION_SECRET} SESSION_SECRET: ${SESSION_SECRET}
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-4} WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
PPROF_ADDR: ${PPROF_ADDR:-:6060}
depends_on: depends_on:
postgres: postgres:
condition: service_healthy condition: service_healthy
@@ -34,7 +35,7 @@ services:
image: caddy:2-alpine image: caddy:2-alpine
labels: labels:
- "traefik.enable=true" - "traefik.enable=true"
- "traefik.http.routers.imap-copier.rule=Host(`imap-copier.amegahost.kz`)" - "traefik.http.routers.imap-copier.rule=Host(`${DOMAIN}`)"
- "traefik.http.routers.imap-copier.entrypoints=https" - "traefik.http.routers.imap-copier.entrypoints=https"
- "traefik.http.routers.imap-copier.tls=true" - "traefik.http.routers.imap-copier.tls=true"
- "traefik.http.routers.imap-copier.tls.certresolver=letsEncrypt" - "traefik.http.routers.imap-copier.tls.certresolver=letsEncrypt"
+3 -2
View File
@@ -8,7 +8,7 @@ services:
volumes: volumes:
- pgdata:/var/lib/postgresql - pgdata:/var/lib/postgresql
healthcheck: healthcheck:
test: ["CMD-SHELL", "pg_isready -U imap"] test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
interval: 5s interval: 5s
timeout: 3s timeout: 3s
retries: 5 retries: 5
@@ -21,7 +21,8 @@ services:
AUTH_PASS: ${AUTH_PASS} AUTH_PASS: ${AUTH_PASS}
ENC_KEY: ${ENC_KEY} ENC_KEY: ${ENC_KEY}
SESSION_SECRET: ${SESSION_SECRET} SESSION_SECRET: ${SESSION_SECRET}
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-4} WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
PPROF_ADDR: ${PPROF_ADDR:-:6060}
depends_on: depends_on:
postgres: postgres:
condition: service_healthy condition: service_healthy
Binary file not shown.

After

Width:  |  Height:  |  Size: 221 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 326 KiB

File diff suppressed because it is too large Load Diff
@@ -0,0 +1,546 @@
# Selective Account Run Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let the user run a migration for only the checkbox-selected accounts; with no selection, run all (unchanged).
**Architecture:** A pure `selectAccounts` filter narrows the orchestrator's account slice before the test gate; the run HTTP handler parses an optional `account_ids` body and passes it through. The accounts table gains row checkboxes and a "select all" header, and the single "Run migration" button becomes "Run selected (N)" when a selection exists. Copy is already idempotent (dedup by message key), so a re-run of selected accounts skips copied messages and delivers the missing tail.
**Tech Stack:** Go 1.x (net/http, encoding/json, standard testing), React + TypeScript + Vite, existing REST/WS client.
## Global Constraints
- Backend module path: `github.com/vasyansk/imap-copier`.
- `Run` with an empty/nil account-ID set MUST behave exactly as today (all accounts) — the scheduler depends on it.
- Preserve existing `handleRun` error mapping: `ErrNotTested` → 409 "accounts must pass connection tests first"; `ErrAlreadyRunning` → 409 "task is already running".
- Frontend has no unit-test harness; frontend tasks verify via `npm run build` + `tsc --noEmit` + manual app check.
- Follow existing code style: no new dependencies.
---
### Task 1: Orchestrator account filter + `Run` signature
**Files:**
- Modify: `internal/orchestrator/orchestrator.go`
- Test: `internal/orchestrator/orchestrator_test.go`
**Interfaces:**
- Produces:
- `func selectAccounts(accs []store.Account, ids []int64) []store.Account` — returns `accs` unchanged when `ids` is empty/nil; otherwise returns only the accounts whose `ID` is in `ids`, preserving input order.
- `var ErrNoAccountsSelected = errors.New("no matching accounts selected")`
- `func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)` — new trailing `accountIDs` param.
- [ ] **Step 1: Write the failing test**
Add to `internal/orchestrator/orchestrator_test.go`:
```go
func TestSelectAccounts(t *testing.T) {
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
if got := selectAccounts(accs, nil); len(got) != 3 {
t.Fatalf("nil ids must return all, got %d", len(got))
}
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
t.Fatalf("empty ids must return all, got %d", len(got))
}
got := selectAccounts(accs, []int64{3, 1})
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
t.Fatalf("must keep matching ids in input order, got %+v", got)
}
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
t.Fatalf("unknown ids must yield empty, got %d", len(got))
}
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
Expected: FAIL — `undefined: selectAccounts`.
- [ ] **Step 3: Add the filter helper and sentinel error**
In `internal/orchestrator/orchestrator.go`, add the sentinel next to the existing `var ErrNotTested`/`ErrAlreadyRunning`:
```go
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
```
Add the helper (place it near `gateOK`):
```go
// selectAccounts narrows accs to those whose ID is in ids, preserving input
// order. An empty or nil ids means "all accounts" — the scheduler and the
// unfiltered manual run rely on this.
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
if len(ids) == 0 {
return accs
}
want := make(map[int64]struct{}, len(ids))
for _, id := range ids {
want[id] = struct{}{}
}
out := make([]store.Account, 0, len(ids))
for _, a := range accs {
if _, ok := want[a.ID]; ok {
out = append(out, a)
}
}
return out
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
Expected: PASS.
- [ ] **Step 5: Thread `accountIDs` through `Run`**
In `internal/orchestrator/orchestrator.go`, change the `Run` signature and apply the filter after loading accounts. Replace:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
task, err := o.store.GetTask(ctx, taskID)
if err != nil {
return 0, err
}
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
if !gateOK(accs) {
return 0, ErrNotTested
}
```
with:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
task, err := o.store.GetTask(ctx, taskID)
if err != nil {
return 0, err
}
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
accs = selectAccounts(accs, accountIDs)
// A non-empty request that matched nothing is a client error, distinct
// from "not tested".
if len(accountIDs) > 0 && len(accs) == 0 {
return 0, ErrNoAccountsSelected
}
if !gateOK(accs) {
return 0, ErrNotTested
}
```
The rest of `Run` (endpoints, `TryMarkTaskRunning`, `CreateRun`, `go o.runAll(...)`) is unchanged — it already operates over the `accs` slice.
- [ ] **Step 6: Update the two `Run` call sites so the package compiles**
In `internal/scheduler/scheduler.go`, change:
```go
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
```
to:
```go
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
```
In `internal/httpapi/run.go`, inside `handleRun`, change:
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual")
```
to (IDs wired in Task 2; for now pass `nil` to keep it compiling):
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
```
- [ ] **Step 7: Verify the whole backend builds and tests pass**
Run: `go build ./... && go test ./internal/orchestrator/ ./internal/scheduler/ -v`
Expected: build succeeds; all tests PASS.
- [ ] **Step 8: Commit**
```bash
git add internal/orchestrator/orchestrator.go internal/orchestrator/orchestrator_test.go internal/scheduler/scheduler.go internal/httpapi/run.go
git commit -m "Add selectAccounts filter and accountIDs param to orchestrator.Run"
```
---
### Task 2: `handleRun` parses optional `account_ids`
**Files:**
- Modify: `internal/httpapi/run.go`
- Test: `internal/httpapi/run_test.go`
**Interfaces:**
- Consumes: `orchestrator.Run(ctx, taskID, "manual", accountIDs)`, `orchestrator.ErrNoAccountsSelected` (Task 1).
- Produces:
- `func parseRunAccountIDs(r *http.Request) ([]int64, error)` — returns `nil` for an empty body; otherwise decodes `{"account_ids":[...]}` and returns the slice (possibly empty). Returns an error on malformed JSON.
- [ ] **Step 1: Write the failing test**
Add to `internal/httpapi/run_test.go` (the `strings` import is already present):
```go
func TestParseRunAccountIDs(t *testing.T) {
// empty body => nil (run all)
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
ids, err := parseRunAccountIDs(req)
if err != nil || ids != nil {
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
}
// explicit selection
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
ids, err = parseRunAccountIDs(req)
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
}
// malformed JSON => error
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
if _, err := parseRunAccountIDs(req); err == nil {
t.Fatal("malformed body must error")
}
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
Expected: FAIL — `undefined: parseRunAccountIDs`.
- [ ] **Step 3: Implement `parseRunAccountIDs` and wire it into `handleRun`**
In `internal/httpapi/run.go`, add the imports `encoding/json` and `io` to the existing import block, then add the helper:
```go
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
raw, err := io.ReadAll(r.Body)
if err != nil {
return nil, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return nil, nil
}
var body struct {
AccountIDs []int64 `json:"account_ids"`
}
if err := json.Unmarshal(raw, &body); err != nil {
return nil, err
}
return body.AccountIDs, nil
}
```
Add `"bytes"` to the import block as well (used by `bytes.TrimSpace`).
Then replace the body of `handleRun`'s run call. Change:
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
}
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
http.Error(w, "task is already running", http.StatusConflict)
return
}
```
to:
```go
accountIDs, err := parseRunAccountIDs(r)
if err != nil {
http.Error(w, "bad request body", http.StatusBadRequest)
return
}
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
return
}
if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
}
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
http.Error(w, "task is already running", http.StatusConflict)
return
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
Expected: PASS.
- [ ] **Step 5: Verify the whole backend builds and the httpapi suite passes**
Run: `go build ./... && go test ./internal/httpapi/ -v`
Expected: build succeeds; all tests PASS.
- [ ] **Step 6: Commit**
```bash
git add internal/httpapi/run.go internal/httpapi/run_test.go
git commit -m "Parse optional account_ids in run handler"
```
---
### Task 3: Frontend `runTask` accepts account IDs
**Files:**
- Modify: `web/src/api.ts:141`
**Interfaces:**
- Produces: `runTask(id: number, accountIds?: number[]) => Promise<...>` — sends `{account_ids}` JSON only when a non-empty list is given; otherwise a bare POST (all accounts).
- [ ] **Step 1: Update `runTask`**
In `web/src/api.ts`, replace:
```ts
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
```
with:
```ts
export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
```
- [ ] **Step 2: Verify typecheck**
Run: `cd web && npx tsc --noEmit`
Expected: no errors from `api.ts` (pre-existing warnings elsewhere, if any, are unrelated).
- [ ] **Step 3: Commit**
```bash
git add web/src/api.ts
git commit -m "runTask accepts optional accountIds"
```
---
### Task 4: Accounts table checkboxes + selective Run button
**Files:**
- Modify: `web/src/pages/TaskDetail.tsx`
- Modify: `web/src/app.css`
**Interfaces:**
- Consumes: `runTask(id, accountIds?)` (Task 3).
- Produces: no exported interface; internal UI state only.
- [ ] **Step 1: Add selection state and helpers**
In `web/src/pages/TaskDetail.tsx`, add a state hook next to the other `useState` declarations (near line 90):
```tsx
const [selected, setSelected] = useState<Set<number>>(new Set())
```
- [ ] **Step 2: Compute selectable rows and the effective run set**
After the existing `const { task, accounts } = data` / `allTested` block (near line 369), add:
```tsx
const isRunning = task.status === 'running'
// A row is selectable only when both connection tests pass and no run is live.
const selectableIds = accounts
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
.map((a) => a.id)
const selectableSet = new Set(selectableIds)
// Effective set: the checked accounts, or all accounts when nothing is checked.
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
const runReady =
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
const allSelectableChecked =
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
function toggleOne(accId: number, checked: boolean) {
setSelected((prev) => {
const next = new Set(prev)
if (checked) next.add(accId)
else next.delete(accId)
return next
})
}
function toggleAll(checked: boolean) {
setSelected(checked ? new Set(selectableIds) : new Set())
}
```
- [ ] **Step 3: Make `onRun` send the selection**
Replace the existing `onRun` function (near line 331):
```tsx
async function onRun() {
setBusy('run')
setError(null)
try {
await runTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run')
} finally {
setBusy(null)
}
}
```
with:
```tsx
async function onRun() {
setBusy('run')
setError(null)
try {
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
await runTask(id, ids.length ? ids : undefined)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run')
} finally {
setBusy(null)
}
}
```
- [ ] **Step 4: Update the Run button label and gate**
Replace the run button (near line 421):
```tsx
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
{busy === 'run' ? 'Starting…' : 'Run migration'}
</button>
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
```
with:
```tsx
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !runReady || isRunning}>
{busy === 'run'
? 'Starting…'
: effectiveSelected.length > 0
? `Run selected (${effectiveSelected.length})`
: 'Run migration'}
</button>
{!runReady && accounts.length > 0 && (
<span className="hint">
{effectiveSelected.length > 0
? 'selected accounts must pass both connection tests'
: 'run unlocks once every account tests OK on both sides'}
</span>
)}
```
- [ ] **Step 5: Add the checkbox header column**
In the accounts table `<thead><tr>` (near line 542), add a leading `<th>` before `<th>Account</th>`:
```tsx
<th className="chk-col">
<input
type="checkbox"
aria-label="Select all accounts"
checked={allSelectableChecked}
ref={(el) => {
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
}}
disabled={isRunning || selectableIds.length === 0}
onChange={(e) => toggleAll(e.target.checked)}
/>
</th>
```
- [ ] **Step 6: Add the per-row checkbox cell and fix the empty-row colspan**
In the empty-row branch (near line 557), change `colSpan={9}` to `colSpan={10}`.
In the account `<tr>` map (near line 561), add a leading `<td>` as the first child, before the `<td>` with `acct-ident`:
```tsx
<td className="chk-col">
<input
type="checkbox"
aria-label={`Select ${a.src_login}`}
checked={selected.has(a.id)}
disabled={isRunning || !selectableSet.has(a.id)}
onChange={(e) => toggleOne(a.id, e.target.checked)}
/>
</td>
```
- [ ] **Step 7: Add checkbox column CSS**
In `web/src/app.css`, append:
```css
.tbl .chk-col {
width: 32px;
text-align: center;
padding-right: 0;
}
.tbl .chk-col input[type='checkbox'] {
cursor: pointer;
}
.tbl .chk-col input[type='checkbox']:disabled {
cursor: not-allowed;
opacity: 0.4;
}
```
- [ ] **Step 8: Build and typecheck**
Run: `cd web && npx tsc --noEmit && npm run build`
Expected: typecheck clean (any pre-existing oxlint/fast-refresh warnings are unrelated); Vite build succeeds.
- [ ] **Step 9: Manual verification in the running app**
Start the app, open a task with ≥2 accounts that have passed both tests:
1. With nothing checked, the button reads "Run migration" and runs all accounts (confirm `account_started` WS events for every account).
2. Check a subset; the button reads "Run selected (N)". Run it; confirm only the checked accounts start, and the unchecked accounts' Copied/Skipped/Status are untouched.
3. Confirm checkboxes for untested accounts are disabled, and the header checkbox toggles only the selectable rows.
- [ ] **Step 10: Commit**
```bash
git add web/src/pages/TaskDetail.tsx web/src/app.css
git commit -m "Add account selection checkboxes and selective run button"
```
---
## Notes for the executor
- Tasks 1 and 2 are backend and must land in order (Task 2 depends on the new `Run` signature and `ErrNoAccountsSelected`). Task 3 and 4 are frontend; Task 4 depends on Task 3.
- The orchestrator's `Run` is integration-heavy (spawns goroutines, needs a real store/hub), so it is not unit-tested directly; the pure `selectAccounts` filter and the `parseRunAccountIDs` helper carry the backend test coverage, and Task 4 Step 9 covers the end-to-end behavior manually.
@@ -0,0 +1,207 @@
# Scheduled (recurring) task runs — design
**Date:** 2026-07-03
**Status:** awaiting user review
## Context
Migrations are run manually today (the operator clicks "Run migration"). For
ongoing source→destination sync, the operator wants a task to run itself on a
recurring interval (e.g. every 1h or 6h) without babysitting it. Requirements
from the user:
- Per-task recurrence interval; the task runs automatically.
- Never start a run while the previous one is still running; measure the interval
from the **completion** of the last run (not its start).
- Show when the next run is due, in the **browser's local time**.
- A separate modal showing a log of runs with their status and totals.
- Deleting a task must clean up all this data along with the task.
Decisions confirmed with the user:
- **Run-log detail:** per-run totals only (reuse `runs`); no per-account snapshot.
- **Enable gate:** a schedule can be enabled only when **all accounts test OK**.
- **Breaker:** if a *scheduled* run finishes with errors, disable the schedule and
mark the task **broken** (red icon in the tasks list and the task detail).
Manual runs never trip the breaker.
- **First run:** one interval after enabling (`schedule_anchor + interval`).
- **Intervals:** presets 1 / 3 / 6 / 12 / 24 h, plus Off.
### Current architecture (as-is)
- `runs` table already has `started_at`, `finished_at`, `status`, and totals
(`migrations/0001_init.up.sql:35-44`). `CreateRun`/`FinishRun` in
`internal/store/runs.go`. No list-by-task query yet.
- `Orchestrator.Run(ctx, taskID)` gates on `gateOK` (all accounts test ok) and
`TryMarkTaskRunning` (atomic single-run), creates a run, and launches `runAll`
asynchronously; `runAll` sets the task status and calls `FinishRun`
(`internal/orchestrator/orchestrator.go`).
- `ON DELETE CASCADE` from `tasks` already removes `runs`, `accounts`, and
`migrated_messages` when a task is deleted.
- `main.go` wires store → orchestrator → hub → server and calls
`ResetRunningOnStartup` to clear phantom "running" after a restart. No scheduler.
- Task DTO / TS `Task` interface: `internal/httpapi/tasks.go`, `web/src/api.ts`.
## Decision: in-process polling scheduler
A single background goroutine (a `Scheduler`) started from `main.go`, ticking
every **30 s**. Each tick queries the DB for schedulable tasks and triggers the
due ones through the existing `Orchestrator.Run`. The DB is the source of truth,
so the scheduler is stateless and restart-safe (paired with the existing
`ResetRunningOnStartup`). Chosen over per-task timers (restart-fragile, in-memory
state) and external cron (breaks the single-binary model, needs auth wiring).
30 s poll precision is ample for hour-scale intervals.
## Data model
**Migration `0004_task_scheduling`:**
```sql
-- up
ALTER TABLE tasks ADD COLUMN schedule_interval_seconds INT NOT NULL DEFAULT 0;
ALTER TABLE tasks ADD COLUMN schedule_anchor TIMESTAMPTZ;
ALTER TABLE tasks ADD COLUMN broken BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE runs ADD COLUMN trigger TEXT NOT NULL DEFAULT 'manual';
-- down
ALTER TABLE runs DROP COLUMN trigger;
ALTER TABLE tasks DROP COLUMN broken;
ALTER TABLE tasks DROP COLUMN schedule_anchor;
ALTER TABLE tasks DROP COLUMN schedule_interval_seconds;
```
- `schedule_interval_seconds` — 0 = Off. Presets map to seconds (3600, 10800,
21600, 43200, 86400).
- `schedule_anchor` — set to `now()` when a schedule is enabled; the baseline for
the first run when no finished run exists yet.
- `broken` — the schedule breaker tripped; task needs operator attention.
- `runs.trigger``'manual' | 'scheduled'`; drives the breaker and enriches the
run log.
`store.Task` gains `ScheduleIntervalSeconds int64`, `ScheduleAnchor *time.Time`,
`Broken bool`. `store.Run` gains `Trigger string`, plus `StartedAt time.Time` and
`FinishedAt *time.Time` (needed for the log + due computation; not currently
scanned).
### Cascade / deletion
No new cleanup code. `runs` cascade-delete on task delete already; the schedule
columns live on `tasks` and vanish with the row. The in-memory scheduler polls
the DB, so a deleted task simply stops appearing. (Confirmed against
`migrations/0001_init.up.sql` FKs.)
## Store additions
- `SetTaskSchedule(ctx, id, intervalSeconds int64) error` — sets interval, sets
`schedule_anchor = now()` when enabling (interval>0) / leaves anchor when
disabling, and clears `broken` on any call (any explicit schedule change
un-breaks the task; enabling is additionally gated on all-accounts-OK in the
HTTP layer, disabling is always allowed).
- `SetTaskBroken(ctx, id) error``broken = true, schedule_interval_seconds = 0`
in one UPDATE (the breaker).
- `CreateRun(ctx, taskID, trigger string)` — extend the existing signature to
record the trigger.
- `ListRunsByTask(ctx, taskID) ([]Run, error)` — newest first, for the modal;
scans `id, started_at, finished_at, status, totals, trigger`.
- `LastFinishedRunAt(ctx, taskID) (*time.Time, error)` — most recent run with a
non-null `finished_at`, for due computation. (Or fold into `ListSchedulable`.)
- `ListSchedulableTasks(ctx) ([]ScheduledTask, error)` — tasks with
`schedule_interval_seconds > 0 AND NOT broken AND status <> 'running'`, joined
with their last finished run's `finished_at`, so the scheduler decides in one
query per tick. Returns the fields `dueAt` needs.
`GetTask`/`ListTasks` SELECT + Scan extend for the three new task columns.
## Scheduler (`internal/scheduler/`)
New package, one focused responsibility.
- `type Scheduler struct { store *store.Store; orch *orchestrator.Orchestrator }`
- `func (s *Scheduler) Start(ctx context.Context)``time.NewTicker(30s)` loop;
on each tick calls `s.tick(ctx)`; stops on `ctx.Done()`.
- `func (s *Scheduler) tick(ctx)``ListSchedulableTasks`, then for each due task
calls `s.orch.Run(ctx, taskID)` with `trigger = "scheduled"`. `Run` errors
(`ErrAlreadyRunning`, `ErrNotTested`) are logged, not fatal.
- **Pure, unit-tested decision function:**
`func dueAt(interval time.Duration, anchor time.Time, lastFinished *time.Time) time.Time`
`lastFinished + interval` if a finished run exists, else `anchor + interval`.
A task is due when `now >= dueAt(...)`. Kept pure so the tick logic is testable
without a DB or clock injection at the call site.
`main.go` starts it: `go scheduler.New(st, orch).Start(context.Background())`.
### Trigger threading + breaker
`Orchestrator.Run` gains a `trigger string` parameter (manual callers pass
`"manual"`; the scheduler passes `"scheduled"`). `Run` forwards it to
`CreateRun` and into `runAll`. In `runAll`, after the final status is computed:
```
if trigger == "scheduled" && totErr > 0 {
_ = o.store.SetTaskBroken(ctx, task.ID)
o.hub.Publish(Event{Type: "task_broken", TaskID: task.ID, ...})
}
```
The HTTP `handleRun` passes `"manual"`. This keeps the breaker precise: only
scheduled runs trip it.
## HTTP API
- **`PUT /api/tasks/{id}/schedule`** — body `{interval_seconds: int}`. Enabling
(interval>0) first checks `gateOK` (all accounts test OK) via
`ListAccountsByTask`; returns **409** with a clear message if not. Then
`SetTaskSchedule`. Disabling (0) always allowed.
- **`GET /api/tasks/{id}/runs`** — `ListRunsByTask` → run-log rows.
- **Task DTO** (`tasks.go`) gains `schedule_interval_seconds`, `broken`, and a
computed `next_run_at *string` (RFC3339, UTC; null when off / running / broken).
`next_run_at` is computed server-side from the same `dueAt` logic so the client
only formats it. `ListTasks` DTO also includes `broken` (for the red icon in the
list).
## Frontend
- **`api.ts`**: `Task` gains `schedule_interval_seconds?`, `broken?`,
`next_run_at?`. New `setTaskSchedule(taskId, intervalSeconds)` and
`listRuns(taskId)` (+ a `Run` interface).
- **`TaskDetail.tsx`**: in the run-control panel, an interval `<select>`
(Off/1h/3h/6h/12h/24h) bound to `setTaskSchedule`; a "Next run: <local time>"
line formatting `next_run_at` with `toLocaleString()` (shows "running…" when a
run is active, nothing when off); a red **broken** badge when `broken`. A
**Runs** button opens a new modal.
- **`RunLogModal.tsx`** (new): lists `listRuns(taskId)` rows — started/finished in
browser-local time, `trigger`, a `StatusBadge`, and copied/skipped/errors.
Reuses the existing `Modal` + `StatusBadge` + `.tbl` patterns.
- **`Tasks.tsx`**: red icon/indicator on rows where `broken`.
- WS: on `task_broken`/`run_done`, `TaskDetail` reloads (existing reload wiring);
`Tasks` list refreshes.
## Error handling
- Enable-while-not-tested → 409, surfaced in the existing `role="alert"` banner.
- Scheduler `Run` errors are logged and skipped; a broken task is skipped every
tick until the operator re-enables (which clears `broken`).
- Restart: `ResetRunningOnStartup` clears phantom running; the scheduler recomputes
due times from persisted `finished_at`/`anchor`.
## Testing
- **store**: `SetTaskSchedule` (sets interval+anchor, clears broken),
`SetTaskBroken` (interval→0, broken→true), `CreateRun` records trigger,
`ListRunsByTask` ordering + fields, `GetTask` returns new columns.
- **scheduler (pure)**: `dueAt` / due decision — not-yet-due, due-from-finished,
first-run-from-anchor, running skipped, broken skipped.
- **orchestrator**: scheduled run with errors trips `SetTaskBroken`; manual run
with errors does not.
- **cascade**: deleting a task removes its runs (extend existing cascade test).
- **E2E** (prod, per project rules): enable a schedule on a tested task → "Next
run" shows correct local time; (with a short test interval) the run auto-starts;
the Runs modal shows the entry with `scheduled` trigger; break an account → the
next scheduled run disables the schedule and marks the task broken (red icon in
list + detail).
## Out of scope (YAGNI)
- Arbitrary cron expressions; catch-up/backfill of missed runs (run once when due);
notifications; per-account breakdown in the run log (totals chosen); configurable
poll interval.
@@ -0,0 +1,100 @@
# Account Errors Modal — Design
Date: 2026-07-05
Status: approved
## Problem
When an account finishes `done_with_errors`, the UI shows only an error **count**
(e.g. `2`) and a single persisted `last_error`. There is no way to see the
individual errors — which folder, which message, and the actual error text. The
event log holds them live but is lost on reload, and message-level errors
(`res.Errors`) currently record only a counter, discarding their text entirely.
## Goal
Click the ERRORS count of an account (when `> 0`) to open a modal listing the
concrete errors of that account's **most recent run**: folder, kind, message
reference, error text, and timestamp.
## Scope decisions
- **Coverage:** errors of the latest run only. The list is cleared at the start
of each run (like the per-account counters).
- **Granularity:** every error is its own record — folder-level, message-level,
and account-level (connect/login) are distinct rows.
- **Cap:** at most 500 error rows per account per run (guard against a corrupt
mailbox producing thousands). On overflow a final synthetic row records
"… N more errors suppressed".
## Data model
New table (chosen over a JSONB column on `accounts`): a table gives atomic
`INSERT` per error with no read-modify-write races, trivial per-run clearing,
`ON DELETE CASCADE` with the account, and matches the existing `runs` /
`migrated_messages` shape.
Migration `0005_account_errors`:
```
account_errors(
id BIGSERIAL PRIMARY KEY,
account_id BIGINT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
run_id BIGINT, -- run the error belongs to (context)
kind TEXT NOT NULL, -- 'folder' | 'message' | 'account'
folder TEXT NOT NULL DEFAULT '',
message_ref TEXT NOT NULL DEFAULT '', -- e.g. "UID 42: <subject>"
error TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
CREATE INDEX ON account_errors(account_id);
```
Down migration: `DROP TABLE account_errors`.
## Backend
**Store** (`internal/store/account_errors.go`):
- `AddAccountError(ctx, accountID, runID int64, kind, folder, ref, msg string) error`
- `ClearAccountErrors(ctx, accountID int64) error`
- `ListAccountErrors(ctx, accountID int64) ([]AccountError, error)` — ordered by id.
- `AccountError` struct mirrors the row.
**Orchestrator** (`internal/orchestrator/orchestrator.go`):
- At account start (next to `ResetAccountCounters`): `ClearAccountErrors`.
- New `CopyDeps.OnError(kind, folder, ref, msg string)` callback. `CopyFolder`
invokes it on every message-level error (the `res.Errors++` sites in
`copy.go`) and folder-level error, passing UID/subject where available. The
orchestrator's `OnError` impl persists via `AddAccountError`, enforcing the
500-row cap with an in-worker counter.
- Folder-level copy error (already persisted to `last_error`) also emits
`OnError(kind="folder", ...)`.
- `accountFailed` (connect/login/decrypt) emits `OnError(kind="account", ...)`.
- `last_error` behavior is unchanged (still the latest single error for the
inline row hint).
**HTTP** (`internal/httpapi`):
- `GET /api/tasks/{id}/accounts/{accountId}/errors` → JSON `[]AccountError`.
## Frontend
- `web/src/api.ts`: `listAccountErrors(taskId, accountId)`.
- ERRORS cell becomes a button when the count `> 0`; opens `AccountErrorsModal`.
- `web/src/components/AccountErrorsModal.tsx` (built on existing `Modal`,
styled like `RunLogModal`): fetches on open, renders rows
`time · folder · kind · message_ref · error`, with loading / empty /
fetch-error states.
## Testing
- Store: `AddAccountError` / `ClearAccountErrors` / `ListAccountErrors` against
Postgres (existing store test harness).
- `CopyFolder`: `OnError` is invoked when a per-message step fails (e.g.
`IsMigrated` returns an error) — unit test with fakes, no server needed.
- HTTP: handler returns the account's errors as JSON.
## Out of scope
- Cross-run error history.
- Retrying individual failed messages from the modal.
- Exporting errors.
@@ -0,0 +1,111 @@
# Selective Account Run — Design
**Date:** 2026-07-08
**Status:** Approved (pending spec review)
## Problem
When a migration run fails for some accounts (e.g. transient `use of closed
network connection`), the user wants to re-run **only the affected accounts**,
not the whole task.
An earlier idea — per-error "retry" buttons inside the errors modal — was
rejected: on a `closed network connection` the copy aborts the whole folder
(`copy.go`, the `net.ErrClosed` branch), so the undelivered tail of the folder
never appears in the error list. A per-UID retry would replay only the few
visible errors and leave the invisible tail uncopied, creating a false sense of
completion.
The correct primitive is a **re-run of the selected accounts**. Copy is
idempotent — `IsMigrated`/`MarkMigrated` dedup by message key — so a re-run
skips everything already copied and delivers only the missing tail, covering the
network-drop case fully.
## Solution
Add checkboxes to the accounts table. The main "Run migration" button runs the
selected accounts; with no selection it runs all accounts (current behavior,
preserved).
## Backend
### `orchestrator.Run`
Change signature to:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)
```
- `accountIDs == nil || len == 0` → all accounts (unchanged behavior).
- Otherwise, filter the `ListAccountsByTask` result down to the requested IDs
**before** `gateOK`, so only the selected accounts are gate-checked and run.
- IDs not belonging to the task are silently dropped by the filter (defense in
depth; the handler also validates).
- If the filter yields zero accounts, return `ErrNotTested` (nothing runnable) —
reuses the existing 409 path; the handler maps empty selection separately (see
below).
Call sites updated:
- `scheduler.go``Run(ctx, id, "scheduled", nil)` (always all accounts).
- `run.go` (`handleRun`) → passes the parsed IDs.
`runAll`, `runAccount`, counters, `FinishRun`, and task-status reconciliation are
unchanged: they already operate over whatever `accs` slice they are handed.
### `handleRun`
- Parse an **optional** JSON body `{"account_ids": [1,2,3]}`. Empty/absent body
`nil` → all accounts.
- Validate parsed IDs against the task's accounts; if a non-empty selection
resolves to zero valid IDs → `400 Bad Request`.
- Existing error mapping preserved: `ErrNotTested` → 409 "accounts must pass
connection tests first"; `ErrAlreadyRunning` → 409.
## Frontend
### Accounts table (`TaskDetail.tsx`)
- New leading column with a per-row checkbox; a "select all" checkbox in the
header (selects only selectable rows).
- State: `selected: Set<number>` of account IDs.
- A row checkbox is disabled when the account has not passed **both** tests
(`test_src_status`/`test_dst_status` !== 'ok') or the task is `running`.
- "select all" reflects indeterminate/checked state over the selectable rows.
### Run button
- With a non-empty selection: label "Run selected (N)"; calls
`runTask(id, [...selected])`.
- With no selection: label "Run migration"; calls `runTask(id)` (all).
- Enable gate computed over the **effective set** (selected, or all when none
selected): every account in that set must be tested OK, and the task must not
be `running`.
- Side benefit: one untested account no longer blocks running the ready ones —
select only the ready accounts and run them.
### `api.ts`
```ts
export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length
? jsonBody({ account_ids: accountIds })
: { method: 'POST' })
```
## Out of scope (YAGNI)
- The errors modal (`AccountErrorsModal`) is untouched; no retry buttons there.
- No new "partial" task status. After a partial run the task status reflects the
last run's outcome over the accounts that ran — same as today.
- No selection persistence across reloads.
## Testing
- **Backend unit:** `orchestrator.Run` with a subset filters correctly; empty
filter runs all; unknown IDs dropped; empty-after-filter → `ErrNotTested`.
- **Backend handler:** `handleRun` parses body, validates IDs, maps empty
selection → 400, absent body → all.
- **Frontend:** manual verification in the running app — select a subset, run,
confirm only those accounts start (WS `account_started` events), and the
unselected accounts' counters/status are untouched.
+3 -1
View File
@@ -15,6 +15,7 @@ type Config struct {
EncKey []byte EncKey []byte
SessionSecret []byte SessionSecret []byte
WorkerConcurrency int WorkerConcurrency int
PprofAddr string // if non-empty, serve net/http/pprof here (e.g. ":6060")
} }
func Load() (Config, error) { func Load() (Config, error) {
@@ -24,7 +25,8 @@ func Load() (Config, error) {
AuthUser: os.Getenv("AUTH_USER"), AuthUser: os.Getenv("AUTH_USER"),
AuthPass: os.Getenv("AUTH_PASS"), AuthPass: os.Getenv("AUTH_PASS"),
SessionSecret: []byte(os.Getenv("SESSION_SECRET")), SessionSecret: []byte(os.Getenv("SESSION_SECRET")),
WorkerConcurrency: 4, WorkerConcurrency: 2,
PprofAddr: os.Getenv("PPROF_ADDR"),
} }
if v := os.Getenv("WORKER_CONCURRENCY"); v != "" { if v := os.Getenv("WORKER_CONCURRENCY"); v != "" {
n, err := strconv.Atoi(v) n, err := strconv.Atoi(v)
+2 -2
View File
@@ -31,7 +31,7 @@ func TestLoadDefaults(t *testing.T) {
if cfg.HTTPAddr != ":8080" { if cfg.HTTPAddr != ":8080" {
t.Errorf("HTTPAddr = %q, want :8080", cfg.HTTPAddr) t.Errorf("HTTPAddr = %q, want :8080", cfg.HTTPAddr)
} }
if cfg.WorkerConcurrency != 4 { if cfg.WorkerConcurrency != 2 {
t.Errorf("WorkerConcurrency = %d, want 4", cfg.WorkerConcurrency) t.Errorf("WorkerConcurrency = %d, want 2", cfg.WorkerConcurrency)
} }
} }
+109
View File
@@ -1,9 +1,11 @@
package csvimport package csvimport
import ( import (
"bufio"
"encoding/csv" "encoding/csv"
"fmt" "fmt"
"io" "io"
"regexp"
"strings" "strings"
) )
@@ -52,3 +54,110 @@ func Parse(r io.Reader) ([]Row, error) {
} }
return rows, nil return rows, nil
} }
// Kerio Connect exports users as a semicolon-separated file whose first line is
// a header. Only these columns matter; the rest (quotas, last login, …) is
// ignored. The file carries no domain — the caller supplies it.
const (
kerioColName = 0 // login without the domain part
kerioColDescription = 2 // Kerio keeps the plaintext password here
kerioColEnable = 3 // "Yes" / "No"
kerioMinColumns = 4
)
// domainRe accepts a bare DNS domain: labels of alphanumerics/hyphens with at
// least one dot and no scheme, user part, or whitespace.
var domainRe = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$`)
func normalizeDomain(domain string) (string, error) {
d := strings.ToLower(strings.TrimSpace(domain))
if d == "" {
return "", fmt.Errorf("domain is required")
}
if !domainRe.MatchString(d) {
return "", fmt.Errorf("invalid domain %q", domain)
}
return d, nil
}
// skipBOM consumes a leading UTF-8 byte-order mark, which Kerio writes into its
// exports and encoding/csv would otherwise glue onto the first header field.
func skipBOM(br *bufio.Reader) error {
b, err := br.Peek(3)
if err != nil && err != io.EOF {
return err
}
if len(b) == 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
_, _ = br.Discard(3)
}
return nil
}
// ParseKerio reads a Kerio Connect user export and maps every enabled, non-admin
// account onto both sides of a migration: the login and password are identical
// on source and destination, only the server differs. Rows for disabled accounts
// and the built-in admin are skipped.
func ParseKerio(r io.Reader, domain string) ([]Row, error) {
d, err := normalizeDomain(domain)
if err != nil {
return nil, err
}
br := bufio.NewReader(r)
if err := skipBOM(br); err != nil {
return nil, err
}
cr := csv.NewReader(br)
cr.Comma = ';'
cr.FieldsPerRecord = -1 // проверяем сами
cr.LazyQuotes = true // FullName нередко содержит одиночную кавычку
header, err := cr.Read()
if err != nil {
return nil, fmt.Errorf("cannot read header: %w", err)
}
if len(header) < kerioMinColumns || !strings.EqualFold(strings.TrimSpace(header[kerioColName]), "Name") {
return nil, fmt.Errorf("not a Kerio export: expected a header starting with Name;FullName;Description;Enable")
}
var rows []Row
seen := map[string]bool{}
for {
rec, err := cr.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
line, _ := cr.FieldPos(0)
if len(rec) == 1 && strings.TrimSpace(rec[0]) == "" {
continue
}
if len(rec) < kerioMinColumns {
return nil, fmt.Errorf("line %d: expected at least %d columns, got %d", line, kerioMinColumns, len(rec))
}
name := strings.ToLower(strings.TrimSpace(rec[kerioColName]))
if name == "" {
return nil, fmt.Errorf("line %d: Name is empty", line)
}
if !strings.EqualFold(strings.TrimSpace(rec[kerioColEnable]), "Yes") || name == "admin" {
continue
}
pass := strings.TrimSpace(rec[kerioColDescription])
if pass == "" {
return nil, fmt.Errorf("line %d: no password in the Description column for %q", line, name)
}
if seen[name] {
return nil, fmt.Errorf("line %d: duplicate Name %q", line, name)
}
seen[name] = true
login := name + "@" + d
rows = append(rows, Row{SrcLogin: login, SrcPass: pass, DstLogin: login, DstPass: pass})
}
if len(rows) == 0 {
return nil, fmt.Errorf("no enabled accounts found in the export")
}
return rows, nil
}
+97
View File
@@ -49,3 +49,100 @@ func TestParseZeroRowsErrors(t *testing.T) {
t.Fatal("expected error when no rows parsed") t.Fatal("expected error when no rows parsed")
} }
} }
const kerioHeader = "Name;FullName;Description;Enable;DataSource;Authentication;Role;Groups;MailAddress\n"
func TestParseKerioOK(t *testing.T) {
in := kerioHeader +
"j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n" +
"k.smith;Kim Smith;SrcPass22;Yes;Internal;Internal;No rights;all;k.smith\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 2 {
t.Fatalf("want 2 rows, got %d: %+v", len(rows), rows)
}
want := Row{
SrcLogin: "j.doe@example.test", SrcPass: "SrcPass11",
DstLogin: "j.doe@example.test", DstPass: "SrcPass11",
}
if rows[0] != want {
t.Fatalf("row 0: got %+v, want %+v", rows[0], want)
}
}
func TestParseKerioSkipsDisabledAndAdmin(t *testing.T) {
in := kerioHeader +
"admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n" +
"o.disabled;;OffPass111;No;Internal;Internal;No rights;all;o.disabled\n" +
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" {
t.Fatalf("only the enabled non-admin row must survive, got %+v", rows)
}
}
func TestParseKerioStripsBOM(t *testing.T) {
in := "\ufeff" + kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 {
t.Fatalf("want 1 row, got %+v", rows)
}
}
func TestParseKerioRejectsMissingHeader(t *testing.T) {
in := "j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("a file without the Kerio header must error")
}
}
func TestParseKerioRejectsBadDomain(t *testing.T) {
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
for _, domain := range []string{"", " ", "@example.test", "exam ple.test", "example", "info@example.test"} {
if _, err := ParseKerio(strings.NewReader(in), domain); err == nil {
t.Fatalf("domain %q must be rejected", domain)
}
}
}
func TestParseKerioTrimsAndLowercasesDomain(t *testing.T) {
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), " Example.TEST ")
if err != nil {
t.Fatalf("parse: %v", err)
}
if rows[0].SrcLogin != "info@example.test" {
t.Fatalf("domain must be trimmed and lowercased, got %q", rows[0].SrcLogin)
}
}
func TestParseKerioRejectsEmptyPassword(t *testing.T) {
in := kerioHeader + "info;;;Yes;Internal;Internal;No rights;all;info\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("an enabled account without a password must error")
}
}
func TestParseKerioRejectsDuplicateName(t *testing.T) {
in := kerioHeader +
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n" +
"info;;aaaaaaAa1;Yes;Internal;Internal;No rights;all;info\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("duplicate Name must error")
}
}
func TestParseKerioZeroRowsErrors(t *testing.T) {
in := kerioHeader + "admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("expected error when every row is filtered out")
}
}
+101
View File
@@ -154,6 +154,82 @@ func (s *Server) handleCreateAccount(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusCreated, map[string]int64{"id": id}) writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
} }
// handleUpdateAccountCredentials fixes the logins/passwords of an existing
// account — typically after an import brought in a wrong password and the
// connection test failed. An empty password field keeps the stored one, so the
// operator can correct one side without retyping the other.
func (s *Server) handleUpdateAccountCredentials(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
accID, err := pathID(r, "accountId")
if err != nil {
http.Error(w, "bad account id", http.StatusBadRequest)
return
}
task, err := s.store.GetTask(r.Context(), taskID)
if err != nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
acc, ok := s.findAccount(r, taskID, accID)
if !ok {
http.Error(w, "account not found", http.StatusNotFound)
return
}
if task.Status == "running" || acc.Status == "running" {
http.Error(w, "cannot change credentials while the account is running", http.StatusConflict)
return
}
var body struct {
SrcLogin string `json:"src_login"`
SrcPass string `json:"src_pass"`
DstLogin string `json:"dst_login"`
DstPass string `json:"dst_pass"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
http.Error(w, "bad json", http.StatusBadRequest)
return
}
// Same trimming as account creation: pasted logins/passwords often carry a
// stray space or newline that the IMAP server rejects.
body.SrcLogin = strings.TrimSpace(body.SrcLogin)
body.DstLogin = strings.TrimSpace(body.DstLogin)
body.SrcPass = strings.TrimSpace(body.SrcPass)
body.DstPass = strings.TrimSpace(body.DstPass)
if body.SrcLogin == "" || body.DstLogin == "" {
http.Error(w, "src_login and dst_login are required", http.StatusBadRequest)
return
}
encrypt := func(pass string) (*string, error) {
if pass == "" {
return nil, nil // keep the stored password
}
enc, err := crypto.Encrypt(s.cfg.EncKey, []byte(pass))
if err != nil {
return nil, err
}
return &enc, nil
}
srcEnc, err := encrypt(body.SrcPass)
if err != nil {
http.Error(w, "encrypt", http.StatusInternalServerError)
return
}
dstEnc, err := encrypt(body.DstPass)
if err != nil {
http.Error(w, "encrypt", http.StatusInternalServerError)
return
}
if err := s.store.UpdateAccountCredentials(r.Context(), accID, body.SrcLogin, body.DstLogin, srcEnc, dstEnc); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// findAccount returns the account with accID under taskID, or ok=false. // findAccount returns the account with accID under taskID, or ok=false.
func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) { func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) {
accs, err := s.store.ListAccountsByTask(r.Context(), taskID) accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
@@ -226,6 +302,31 @@ func (s *Server) handleProbeAccountFolders(w http.ResponseWriter, r *http.Reques
}) })
} }
// handleListAccountErrors returns the individual errors recorded for an account
// during its most recent run, for the per-account error modal.
func (s *Server) handleListAccountErrors(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
accID, err := pathID(r, "accountId")
if err != nil {
http.Error(w, "bad account id", http.StatusBadRequest)
return
}
if _, ok := s.findAccount(r, taskID, accID); !ok {
http.Error(w, "account not found", http.StatusNotFound)
return
}
errs, err := s.store.ListAccountErrors(r.Context(), accID)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, errs)
}
// handleSetAccountFolderMapping persists one account's rename map + excluded set. // handleSetAccountFolderMapping persists one account's rename map + excluded set.
func (s *Server) handleSetAccountFolderMapping(w http.ResponseWriter, r *http.Request) { func (s *Server) handleSetAccountFolderMapping(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id") taskID, err := pathID(r, "id")
+33
View File
@@ -2,8 +2,11 @@ package httpapi
import ( import (
"encoding/json" "encoding/json"
"errors"
"fmt"
"net/http" "net/http"
"github.com/jackc/pgx/v5/pgconn"
"github.com/vasyansk/imap-copier/internal/store" "github.com/vasyansk/imap-copier/internal/store"
) )
@@ -54,6 +57,36 @@ func (s *Server) handleUpdateEndpoint(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
// handleDeleteEndpoint removes an endpoint that no task references. A referenced
// endpoint is refused with 409 rather than a foreign-key error, and the same
// status covers the race where a task is created between check and delete.
func (s *Server) handleDeleteEndpoint(w http.ResponseWriter, r *http.Request) {
id, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
used, err := s.store.CountTasksUsingEndpoint(r.Context(), id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if used > 0 {
http.Error(w, fmt.Sprintf("endpoint is used by %d task(s) — delete them first", used), http.StatusConflict)
return
}
if err := s.store.DeleteEndpoint(r.Context(), id); err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
http.Error(w, "endpoint is used by a task — delete it first", http.StatusConflict)
return
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) { func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) {
eps, err := s.store.ListEndpoints(r.Context()) eps, err := s.store.ListEndpoints(r.Context())
if err != nil { if err != nil {
+8
View File
@@ -15,6 +15,7 @@ func (s *Server) Router() http.Handler {
api.HandleFunc("GET /api/endpoints", s.handleListEndpoints) api.HandleFunc("GET /api/endpoints", s.handleListEndpoints)
api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint) api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint)
api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint) api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint)
api.HandleFunc("DELETE /api/endpoints/{id}", s.handleDeleteEndpoint)
api.HandleFunc("GET /api/tasks", s.handleListTasks) api.HandleFunc("GET /api/tasks", s.handleListTasks)
api.HandleFunc("POST /api/tasks", s.handleCreateTask) api.HandleFunc("POST /api/tasks", s.handleCreateTask)
api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask) api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask)
@@ -22,10 +23,17 @@ func (s *Server) Router() http.Handler {
api.HandleFunc("POST /api/tasks/{id}/accounts", s.handleCreateAccount) api.HandleFunc("POST /api/tasks/{id}/accounts", s.handleCreateAccount)
api.HandleFunc("POST /api/tasks/{id}/probe", s.handleProbeFolders) api.HandleFunc("POST /api/tasks/{id}/probe", s.handleProbeFolders)
api.HandleFunc("PUT /api/tasks/{id}/folder-mapping", s.handleSetFolderMapping) api.HandleFunc("PUT /api/tasks/{id}/folder-mapping", s.handleSetFolderMapping)
api.HandleFunc("PUT /api/tasks/{id}/schedule", s.handleSetSchedule)
api.HandleFunc("GET /api/tasks/{id}/runs", s.handleListRuns)
api.HandleFunc("GET /api/tasks/{id}/accounts/{accountId}/errors", s.handleListAccountErrors)
api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount) api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount)
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/credentials", s.handleUpdateAccountCredentials)
api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV) api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV)
api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts) api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts)
api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun) api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun)
api.HandleFunc("POST /api/tasks/{id}/pause", s.handlePauseRun)
api.HandleFunc("POST /api/tasks/{id}/cancel", s.handleCancelRun)
api.HandleFunc("POST /api/tasks/{id}/resume", s.handleResumeRun)
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount) api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount)
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders) api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders)
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping) api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping)
+94 -2
View File
@@ -1,8 +1,11 @@
package httpapi package httpapi
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"io"
"net/http" "net/http"
"github.com/vasyansk/imap-copier/internal/crypto" "github.com/vasyansk/imap-copier/internal/crypto"
@@ -11,6 +14,17 @@ import (
"github.com/vasyansk/imap-copier/internal/store" "github.com/vasyansk/imap-copier/internal/store"
) )
// parseImportRows picks the CSV dialect from the "format" form field. A Kerio
// Connect export holds one login/password pair and no domain, so the operator
// supplies the domain alongside the file; anything else is the plain 4-column
// src/dst format.
func parseImportRows(r *http.Request, file io.Reader) ([]csvimport.Row, error) {
if r.FormValue("format") == "kerio" {
return csvimport.ParseKerio(file, r.FormValue("domain"))
}
return csvimport.Parse(file)
}
func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) { func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id") taskID, err := pathID(r, "id")
if err != nil { if err != nil {
@@ -23,7 +37,7 @@ func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
return return
} }
defer file.Close() defer file.Close()
rows, err := csvimport.Parse(file) rows, err := parseImportRows(r, file)
if err != nil { if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
@@ -63,13 +77,41 @@ func (s *Server) handleTestAccounts(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusAccepted) w.WriteHeader(http.StatusAccepted)
} }
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
raw, err := io.ReadAll(r.Body)
if err != nil {
return nil, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return nil, nil
}
var body struct {
AccountIDs []int64 `json:"account_ids"`
}
if err := json.Unmarshal(raw, &body); err != nil {
return nil, err
}
return body.AccountIDs, nil
}
func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) { func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id") taskID, err := pathID(r, "id")
if err != nil { if err != nil {
http.Error(w, "bad id", http.StatusBadRequest) http.Error(w, "bad id", http.StatusBadRequest)
return return
} }
runID, err := s.orch.Run(r.Context(), taskID) accountIDs, err := parseRunAccountIDs(r)
if err != nil {
http.Error(w, "bad request body", http.StatusBadRequest)
return
}
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
return
}
if errors.Is(err, orchestrator.ErrNotTested) { if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict) http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return return
@@ -85,6 +127,56 @@ func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID}) writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
} }
// handlePauseRun stops the live run but keeps the unfinished accounts
// resumable; handleCancelRun stops it for good. Both are no-ops (409) when the
// task has no run in flight.
func (s *Server) handlePauseRun(w http.ResponseWriter, r *http.Request) {
s.stopRun(w, r, s.orch.PauseTask)
}
func (s *Server) handleCancelRun(w http.ResponseWriter, r *http.Request) {
s.stopRun(w, r, s.orch.CancelTask)
}
func (s *Server) stopRun(w http.ResponseWriter, r *http.Request, stop func(int64) bool) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
if !stop(taskID) {
http.Error(w, "task is not running", http.StatusConflict)
return
}
w.WriteHeader(http.StatusAccepted)
}
// handleResumeRun restarts a paused task with the accounts its pause left
// unfinished; already-copied messages are skipped by the migration journal.
func (s *Server) handleResumeRun(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
runID, err := s.orch.ResumeTask(r.Context(), taskID)
switch {
case errors.Is(err, orchestrator.ErrNothingToResume):
http.Error(w, "no paused accounts to resume", http.StatusConflict)
return
case errors.Is(err, orchestrator.ErrNotTested):
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
case errors.Is(err, orchestrator.ErrAlreadyRunning):
http.Error(w, "task is already running", http.StatusConflict)
return
case err != nil:
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
}
func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) { func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id") taskID, err := pathID(r, "id")
if err != nil { if err != nil {
+78
View File
@@ -1,7 +1,9 @@
package httpapi package httpapi
import ( import (
"io"
"mime/multipart" "mime/multipart"
"net/http"
"net/http/httptest" "net/http/httptest"
"strings" "strings"
"testing" "testing"
@@ -26,3 +28,79 @@ func TestImportCSVFailsOnBadEncKey(t *testing.T) {
t.Fatalf("import must fail on bad EncKey, got %d", rw.Code) t.Fatalf("import must fail on bad EncKey, got %d", rw.Code)
} }
} }
// importReq builds a multipart import request with the given CSV payload and
// extra form fields, then returns it with the uploaded file ready to read.
func importReq(t *testing.T, csv string, fields map[string]string) (*http.Request, io.Reader) {
t.Helper()
body := &strings.Builder{}
mw := multipart.NewWriter(body)
for k, v := range fields {
_ = mw.WriteField(k, v)
}
fw, _ := mw.CreateFormFile("file", "a.csv")
fw.Write([]byte(csv))
mw.Close()
req := httptest.NewRequest("POST", "/api/tasks/1/import", strings.NewReader(body.String()))
req.Header.Set("Content-Type", mw.FormDataContentType())
req.SetPathValue("id", "1")
file, _, err := req.FormFile("file")
if err != nil {
t.Fatalf("form file: %v", err)
}
return req, file
}
const kerioCSV = "Name;FullName;Description;Enable;DataSource\n" +
"info;;InfoPass11;Yes;Internal\n"
func TestParseImportRowsUsesKerioParserWithDomain(t *testing.T) {
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio", "domain": "example.test"})
rows, err := parseImportRows(req, file)
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" || rows[0].DstLogin != "info@example.test" {
t.Fatalf("kerio rows must carry the supplied domain on both sides, got %+v", rows)
}
}
func TestParseImportRowsKerioRequiresDomain(t *testing.T) {
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio"})
if _, err := parseImportRows(req, file); err == nil {
t.Fatal("kerio import without a domain must error")
}
}
func TestParseImportRowsDefaultsToPlainFormat(t *testing.T) {
req, file := importReq(t, "a@x,p1,a@y,p2\n", nil)
rows, err := parseImportRows(req, file)
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "a@x" || rows[0].DstPass != "p2" {
t.Fatalf("no format field must keep the 4-column parser, got %+v", rows)
}
}
func TestParseRunAccountIDs(t *testing.T) {
// empty body => nil (run all)
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
ids, err := parseRunAccountIDs(req)
if err != nil || ids != nil {
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
}
// explicit selection
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
ids, err = parseRunAccountIDs(req)
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
}
// malformed JSON => error
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
if _, err := parseRunAccountIDs(req); err == nil {
t.Fatal("malformed body must error")
}
}
+94 -1
View File
@@ -1,9 +1,12 @@
package httpapi package httpapi
import ( import (
"context"
"encoding/json" "encoding/json"
"net/http" "net/http"
"time"
"github.com/vasyansk/imap-copier/internal/scheduler"
"github.com/vasyansk/imap-copier/internal/store" "github.com/vasyansk/imap-copier/internal/store"
) )
@@ -50,5 +53,95 @@ func (s *Server) handleGetTask(w http.ResponseWriter, r *http.Request) {
for _, a := range accs { for _, a := range accs {
views = append(views, accountDTO(a)) views = append(views, accountDTO(a))
} }
writeJSON(w, http.StatusOK, map[string]any{"task": task, "accounts": views}) tv, err := s.taskViewFor(r.Context(), task)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, map[string]any{"task": tv, "accounts": views})
}
// taskView augments the stored task with the server-computed next scheduled run
// (RFC3339 UTC; null when the schedule is off, the task is running, or broken).
type taskView struct {
store.Task
NextRunAt *string `json:"next_run_at"`
}
func (s *Server) taskViewFor(ctx context.Context, t store.Task) (taskView, error) {
if t.ScheduleIntervalSeconds <= 0 || t.Broken || t.Status == "running" || t.ScheduleAnchor == nil {
return taskView{Task: t}, nil
}
last, err := s.store.LastFinishedRunAt(ctx, t.ID)
if err != nil {
return taskView{}, err
}
n := scheduler.NextRun(time.Duration(t.ScheduleIntervalSeconds)*time.Second, *t.ScheduleAnchor, last)
str := n.UTC().Format(time.RFC3339)
return taskView{Task: t, NextRunAt: &str}, nil
}
// handleSetSchedule enables/changes/disables a task's recurring schedule.
// Enabling (interval>0) requires every account to pass its connection tests.
func (s *Server) handleSetSchedule(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
var body struct {
IntervalSeconds int64 `json:"interval_seconds"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
http.Error(w, "bad json", http.StatusBadRequest)
return
}
if body.IntervalSeconds < 0 {
http.Error(w, "interval_seconds must be >= 0", http.StatusBadRequest)
return
}
if body.IntervalSeconds > 0 {
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if !allAccountsTested(accs) {
http.Error(w, "all accounts must pass connection tests before scheduling", http.StatusConflict)
return
}
}
if err := s.store.SetTaskSchedule(r.Context(), taskID, body.IntervalSeconds); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// allAccountsTested mirrors the orchestrator's run gate: non-empty and every
// account OK on both sides.
func allAccountsTested(accs []store.Account) bool {
if len(accs) == 0 {
return false
}
for _, a := range accs {
if a.TestSrcStatus != "ok" || a.TestDstStatus != "ok" {
return false
}
}
return true
}
func (s *Server) handleListRuns(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
runs, err := s.store.ListRunsByTask(r.Context(), taskID)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, runs)
} }
+2 -3
View File
@@ -4,13 +4,12 @@ import (
"context" "context"
"github.com/emersion/go-imap/v2" "github.com/emersion/go-imap/v2"
"github.com/emersion/go-imap/v2/imapclient"
) )
// FolderMessageCount opens a folder read-only (EXAMINE) and returns how many // FolderMessageCount opens a folder read-only (EXAMINE) and returns how many
// messages it holds — used to plan an accurate overall progress total before // messages it holds — used to plan an accurate overall progress total before
// copying begins. It does not fetch any message bodies. // copying begins. It does not fetch any message bodies.
func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) { func FolderMessageCount(c *Client, folder string) (int64, error) {
sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait() sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait()
if err != nil { if err != nil {
return 0, err return 0, err
@@ -19,7 +18,7 @@ func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
} }
// ListFolders returns the mailbox names visible on an already-connected, logged-in client. // ListFolders returns the mailbox names visible on an already-connected, logged-in client.
func ListFolders(c *imapclient.Client) ([]string, error) { func ListFolders(c *Client) ([]string, error) {
mboxes, err := c.List("", "*", nil).Collect() mboxes, err := c.List("", "*", nil).Collect()
if err != nil { if err != nil {
return nil, err return nil, err
+54
View File
@@ -0,0 +1,54 @@
package imapx
import (
"testing"
"github.com/emersion/go-imap/v2"
)
// metaBatches must tile the sequence 1..total into contiguous, non-overlapping
// windows of at most batchSize, covering every message exactly once. A single
// unbounded FETCH 1:* is what wedges large mailboxes; batching keeps each
// command short so the server stays responsive and ctx can be checked between
// windows.
func TestMetaBatches(t *testing.T) {
cases := []struct {
total, size uint32
want []imap.SeqRange
}{
{0, 1000, nil},
{1, 1000, []imap.SeqRange{{Start: 1, Stop: 1}}},
{1000, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}}},
{1001, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}, {Start: 1001, Stop: 1001}}},
{2500, 1000, []imap.SeqRange{{Start: 1, Stop: 1000}, {Start: 1001, Stop: 2000}, {Start: 2001, Stop: 2500}}},
}
for _, c := range cases {
got := metaBatches(c.total, c.size)
if len(got) != len(c.want) {
t.Fatalf("total=%d size=%d: got %d windows %v, want %d %v", c.total, c.size, len(got), got, len(c.want), c.want)
}
for i := range got {
if got[i] != c.want[i] {
t.Fatalf("total=%d size=%d window %d: got %+v want %+v", c.total, c.size, i, got[i], c.want[i])
}
}
}
}
// Every window must be within 1..total and the windows must be gap-free so no
// message is skipped or fetched twice.
func TestMetaBatchesCoverage(t *testing.T) {
const total, size = 4321, 1000
got := metaBatches(total, size)
if got[0].Start != 1 {
t.Fatalf("first window must start at 1, got %d", got[0].Start)
}
if last := got[len(got)-1]; last.Stop != total {
t.Fatalf("last window must stop at total=%d, got %d", total, last.Stop)
}
for i := 1; i < len(got); i++ {
if got[i].Start != got[i-1].Stop+1 {
t.Fatalf("gap/overlap between window %d (%+v) and %d (%+v)", i-1, got[i-1], i, got[i])
}
}
}
+249 -49
View File
@@ -3,8 +3,13 @@ package imapx
import ( import (
"bytes" "bytes"
"context" "context"
"errors"
"fmt" "fmt"
"io" "io"
"log/slog"
"net"
"os"
"strings"
"time" "time"
"github.com/emersion/go-imap/v2" "github.com/emersion/go-imap/v2"
@@ -27,8 +32,42 @@ type CopyDeps struct {
// folder's messages have been examined so far — so the UI shows movement // folder's messages have been examined so far — so the UI shows movement
// while dedup decisions are made, before bodies start copying. // while dedup decisions are made, before bodies start copying.
OnScan func(scanned, total int64) OnScan func(scanned, total int64)
// OnError is called for each message-level error, with a message reference
// ("UID N: subject", empty when the envelope is unavailable) and the error
// text — so the orchestrator can persist individual errors for the
// per-account error modal. Folder-level errors are reported by the caller.
OnError func(ref, msg string)
// OnActivity is called repeatedly WHILE a single message body is streamed
// (each FETCH read chunk and each APPEND write chunk). Copying one large
// message can take longer than the orchestrator's stall timeout; without an
// in-body signal the watchdog can't tell a slow-but-live transfer from a
// wedged connection and cancels a healthy copy. May be nil.
OnActivity func()
// ReconnectSrc dials and logs in a FRESH source client, returning it. It is
// called when a body read times out (server under-delivered a literal),
// which leaves the current src connection desynced and unusable. CopyFolder
// swaps to the returned client, re-EXAMINEs the folder, and resumes. The
// implementation is expected to also update any external reference to the
// live src client (e.g. so a cancel path closes the right connection). If
// nil, a body-read timeout aborts the folder instead of recovering.
ReconnectSrc func() (*Client, error)
} }
// ErrBodyTimeout means a message body did not finish transferring within the
// idle deadline — the server stopped sending mid-literal. It is almost always a
// server announcing a BODY[] literal larger than the bytes it actually sends,
// which makes go-imap wait forever for bytes that never come. Distinct from a
// closed connection so the caller can skip just this one message and resume.
var ErrBodyTimeout = errors.New("message body read timed out")
// bodyIdleTimeout bounds how long a body read may go with NO bytes arriving
// before it is abandoned. It is generous enough for legitimately slow servers
// (even ~15 KB/s links keep bytes flowing far more often than this) yet well
// under the orchestrator's multi-minute stall watchdog, so an under-delivered
// literal is caught quickly and locally instead of stalling the whole account.
// A var (not const) so tests can shorten it.
var bodyIdleTimeout = 30 * time.Second
// CopyResult summarizes the outcome of one CopyFolder run. // CopyResult summarizes the outcome of one CopyFolder run.
type CopyResult struct { type CopyResult struct {
Copied int Copied int
@@ -36,6 +75,43 @@ type CopyResult struct {
Errors int Errors int
} }
// metaScanBatch bounds how many messages one Pass-1 metadata FETCH covers. A
// single unbounded FETCH 1:* over a large mailbox keeps one command open for
// the entire scan; under parallel load the server can stop responding and,
// since go-imap has no per-command deadline, the worker wedges forever. Short
// windows keep each command brief so the server stays responsive and ctx is
// checked between windows.
const metaScanBatch = 1000
// msgRef builds a human-readable reference for a message error: its UID plus
// subject when known, e.g. "UID 42: Invoice". Falls back to just the UID.
func msgRef(uid imap.UID, subject string) string {
if subject == "" {
return fmt.Sprintf("UID %d", uid)
}
return fmt.Sprintf("UID %d: %s", uid, subject)
}
// metaBatches tiles 1..total into contiguous, non-overlapping windows of at
// most batchSize, covering every sequence number exactly once.
func metaBatches(total, batchSize uint32) []imap.SeqRange {
if total == 0 || batchSize == 0 {
return nil
}
var out []imap.SeqRange
for start := uint32(1); start <= total; start += batchSize {
stop := start + batchSize - 1
if stop > total {
stop = total
}
out = append(out, imap.SeqRange{Start: start, Stop: stop})
if stop == total {
break // guard against uint32 overflow when total is near max
}
}
return out
}
// CopyFolder streams messages from srcFolder on src to dstFolder on dst. // CopyFolder streams messages from srcFolder on src to dstFolder on dst.
// //
// The source folder is opened read-only (EXAMINE) and is never mutated: // The source folder is opened read-only (EXAMINE) and is never mutated:
@@ -43,7 +119,7 @@ type CopyResult struct {
// held in memory only for the duration of a single FETCH->APPEND and is // held in memory only for the duration of a single FETCH->APPEND and is
// never written to disk. Messages already migrated (per deps.IsMigrated) // never written to disk. Messages already migrated (per deps.IsMigrated)
// are skipped without re-fetching their bodies. // are skipped without re-fetching their bodies.
func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) { func CopyFolder(ctx context.Context, src, dst *Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
var res CopyResult var res CopyResult
sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait() sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait()
@@ -67,48 +143,67 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
type queued struct { type queued struct {
uid imap.UID uid imap.UID
key string key string
subject string
flags []imap.Flag flags []imap.Flag
internalDate time.Time internalDate time.Time
} }
var todo []queued var todo []queued
metaSet := imap.SeqSet{imap.SeqRange{Start: 1, Stop: sel.NumMessages}}
fc := src.Fetch(metaSet, &imap.FetchOptions{
UID: true, Envelope: true, RFC822Size: true, Flags: true, InternalDate: true,
})
var scanned int64 var scanned int64
for { reportErr := func(ref, msg string) {
if err := ctx.Err(); err != nil { if deps.OnError != nil {
_ = fc.Close() deps.OnError(ref, msg)
return res, err
}
msg := fc.Next()
if msg == nil {
break
}
buf, err := msg.Collect()
if err != nil {
res.Errors++
continue
}
scanned++
key := MessageKey(buf.Envelope, buf.RFC822Size)
already, err := deps.IsMigrated(key)
if err != nil {
res.Errors++
} else if already {
res.Skipped++
if deps.OnProgress != nil {
deps.OnProgress(res.Copied, res.Skipped)
}
} else {
todo = append(todo, queued{uid: buf.UID, key: key, flags: buf.Flags, internalDate: buf.InternalDate})
}
if deps.OnScan != nil {
deps.OnScan(scanned, total)
} }
} }
if err := fc.Close(); err != nil { // Scan metadata in bounded windows instead of one FETCH 1:*, so each
return res, fmt.Errorf("fetch meta %q: %w", srcFolder, err) // command is short (the server stays responsive) and ctx is checked on
// every window boundary — not just between messages of one giant command.
for _, win := range metaBatches(sel.NumMessages, metaScanBatch) {
if err := ctx.Err(); err != nil {
return res, err
}
fc := src.Fetch(imap.SeqSet{win}, &imap.FetchOptions{
UID: true, Envelope: true, RFC822Size: true, Flags: true, InternalDate: true,
})
for {
if err := ctx.Err(); err != nil {
_ = fc.Close()
return res, err
}
msg := fc.Next()
if msg == nil {
break
}
buf, err := msg.Collect()
if err != nil {
res.Errors++
reportErr("", "read message metadata: "+err.Error())
continue
}
scanned++
key := MessageKey(buf.Envelope, buf.RFC822Size)
subject := ""
if buf.Envelope != nil {
subject = buf.Envelope.Subject
}
already, err := deps.IsMigrated(key)
if err != nil {
res.Errors++
reportErr(msgRef(buf.UID, subject), "dedup lookup: "+err.Error())
} else if already {
res.Skipped++
if deps.OnProgress != nil {
deps.OnProgress(res.Copied, res.Skipped)
}
} else {
todo = append(todo, queued{uid: buf.UID, key: key, subject: subject, flags: buf.Flags, internalDate: buf.InternalDate})
}
if deps.OnScan != nil {
deps.OnScan(scanned, total)
}
}
if err := fc.Close(); err != nil {
return res, fmt.Errorf("fetch meta %q: %w", srcFolder, err)
}
} }
// Pass 2: fetch bodies for the queued (new) messages, one at a time. // Pass 2: fetch bodies for the queued (new) messages, one at a time.
@@ -116,12 +211,39 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return res, err return res, err
} }
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate); err != nil { if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate, deps.OnActivity); err != nil {
res.Errors++ res.Errors++
reportErr(msgRef(q.uid, q.subject), "copy message: "+err.Error())
// A body-read timeout means the server under-delivered this message's
// literal; the src connection is now desynced. Mark the message
// migrated so this and future runs skip it (it is un-fetchable via a
// conforming client), then reconnect src and resume the folder with
// the remaining queued messages.
if errors.Is(err, ErrBodyTimeout) && deps.ReconnectSrc != nil {
if merr := deps.MarkMigrated(dstFolder, q.key); merr != nil {
reportErr(msgRef(q.uid, q.subject), "mark skipped: "+merr.Error())
}
newSrc, rerr := deps.ReconnectSrc()
if rerr != nil {
return res, fmt.Errorf("reconnect src after body timeout in %q: %w", srcFolder, rerr)
}
src = newSrc
if _, serr := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait(); serr != nil {
return res, fmt.Errorf("re-examine %q after reconnect: %w", srcFolder, serr)
}
continue
}
// A closed/broken connection won't recover: every remaining APPEND
// would fail identically. Abort the folder instead of logging
// thousands of the same error; a re-run resumes via dedup.
if errors.Is(err, net.ErrClosed) || strings.Contains(err.Error(), "use of closed network connection") {
return res, fmt.Errorf("dst connection lost in %q: %w", dstFolder, err)
}
continue continue
} }
if err := deps.MarkMigrated(dstFolder, q.key); err != nil { if err := deps.MarkMigrated(dstFolder, q.key); err != nil {
res.Errors++ res.Errors++
reportErr(msgRef(q.uid, q.subject), "mark migrated: "+err.Error())
continue continue
} }
res.Copied++ res.Copied++
@@ -132,11 +254,63 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
return res, nil return res, nil
} }
// slowMessage marks how long one message's FETCH or APPEND phase may take
// before it is logged as anomalous. Well below the orchestrator's 3-minute
// stall timeout, so a message that trips the watchdog always leaves a log line
// naming the phase (FETCH vs APPEND) and size — turning a silent stall into
// evidence of which side and which message is the culprit.
const slowMessage = 20 * time.Second
// deadlineReader arms an idle read deadline on the source socket before every
// read of a message body, so a server that stops sending mid-literal — having
// announced a larger BODY[] size than it actually delivers — trips the deadline
// instead of blocking go-imap forever waiting for bytes that never arrive. It
// also pings onActivity as bytes arrive, feeding the orchestrator's stall
// watchdog. The deadline is refreshed on each read, so it bounds IDLE time
// (no bytes) rather than total transfer time — a legitimately slow but steady
// download never trips it.
type deadlineReader struct {
c *Client
r io.Reader
idle time.Duration
on func()
}
func (d deadlineReader) Read(p []byte) (int, error) {
_ = d.c.SetReadDeadline(time.Now().Add(d.idle))
n, err := d.r.Read(p)
if n > 0 && d.on != nil {
d.on()
}
return n, err
}
// touchWriter wraps the APPEND write stream and pings onActivity on every
// non-empty write, so a long upload keeps the stall watchdog fed byte-by-byte.
type touchWriter struct {
w io.Writer
on func()
}
func (t touchWriter) Write(p []byte) (int, error) {
n, err := t.w.Write(p)
if n > 0 && t.on != nil {
t.on()
}
return n, err
}
// streamOne FETCHes BODY[] for one message and APPENDs it into dst without // streamOne FETCHes BODY[] for one message and APPENDs it into dst without
// spooling to disk. The body is buffered in RAM only for the duration of // spooling to disk. The body is buffered in RAM only for the duration of
// this single FETCH->APPEND round trip. // this single FETCH->APPEND round trip. onActivity (may be nil) fires as bytes
func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time) error { // move in either direction, feeding the orchestrator's stall watchdog.
//
// The body read is guarded by an idle deadline on src: if the server goes
// silent mid-literal, streamOne returns ErrBodyTimeout rather than hanging, so
// CopyFolder can skip the message and reconnect.
func streamOne(src, dst *Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time, onActivity func()) error {
bodySection := &imap.FetchItemBodySection{} bodySection := &imap.FetchItemBodySection{}
fetchStart := time.Now()
fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{ fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{
BodySection: []*imap.FetchItemBodySection{bodySection}, BodySection: []*imap.FetchItemBodySection{bodySection},
}) })
@@ -147,35 +321,61 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
return fmt.Errorf("no message for uid %v", uid) return fmt.Errorf("no message for uid %v", uid)
} }
var body []byte var body []byte
var readErr error
for { for {
item := msg.Next() item := msg.Next()
if item == nil { if item == nil {
break break
} }
if d, ok := item.(imapclient.FetchItemDataBodySection); ok { if d, ok := item.(imapclient.FetchItemDataBodySection); ok {
b, err := io.ReadAll(d.Literal) body, readErr = io.ReadAll(deadlineReader{c: src, r: d.Literal, idle: bodyIdleTimeout, on: onActivity})
if err != nil {
return err
}
body = b
} }
} }
// Clear the deadline before any further I/O on src (fetchCmd.Close reads the
// command's completion off the same socket).
_ = src.SetReadDeadline(time.Time{})
if readErr != nil {
if errors.Is(readErr, os.ErrDeadlineExceeded) {
return fmt.Errorf("%w: uid %v (server sent fewer bytes than the announced literal)", ErrBodyTimeout, uid)
}
return readErr
}
if err := fetchCmd.Close(); err != nil { if err := fetchCmd.Close(); err != nil {
return err return err
} }
if body == nil { if body == nil {
return fmt.Errorf("empty body uid %v", uid) return fmt.Errorf("empty body uid %v", uid)
} }
fetchDur := time.Since(fetchStart)
appendStart := time.Now()
appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate}) appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate})
if _, err := io.Copy(appendCmd, bytes.NewReader(body)); err != nil { // Append acquires go-imap's per-client encoder mutex and holds it until
// Close() calls enc.end(). Close() MUST run on every path: if io.Copy
// fails mid-write (server stall, idle timeout), returning without Close()
// leaks the mutex and the NEXT Append on this client deadlocks forever on
// beginCommand. Close() is idempotent and always releases the lock.
_, copyErr := io.Copy(touchWriter{w: appendCmd, on: onActivity}, bytes.NewReader(body))
closeErr := appendCmd.Close()
if copyErr != nil {
return fmt.Errorf("append body uid %v: %w", uid, copyErr)
}
if closeErr != nil {
return closeErr
}
if _, err := appendCmd.Wait(); err != nil {
return err return err
} }
if err := appendCmd.Close(); err != nil { appendDur := time.Since(appendStart)
return err
// One message that individually eats a large slice of the stall budget is
// the prime suspect behind a "no progress" cancel; name it, its size, and
// which phase was slow so the culprit is visible in the logs.
if fetchDur > slowMessage || appendDur > slowMessage {
slog.Warn("slow message copy", "uid", uid, "bytes", len(body),
"fetch", fetchDur.Round(time.Millisecond), "append", appendDur.Round(time.Millisecond))
} }
_, err := appendCmd.Wait() return nil
return err
} }
// keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8 // keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8
+50
View File
@@ -169,6 +169,56 @@ func TestCopyFolderPreservesInternalDate(t *testing.T) {
} }
} }
// TestCopyFolderReportsActivityDuringBody proves CopyFolder invokes OnActivity
// while a message body is being transferred (FETCH/APPEND), not only between
// messages. This is what keeps the orchestrator's stall watchdog from killing a
// single large-but-live message whose transfer legitimately exceeds the stall
// timeout: without an in-body activity signal, one slow message looks identical
// to a wedged connection.
func TestCopyFolderReportsActivityDuringBody(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
seedInbox(t, ep, "actsrc@localhost", "p", 1)
src, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = src.Logout().Wait() }()
if err := src.Login("actsrc@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
dst, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login("actdst@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
var activity int
deps := CopyDeps{
IsMigrated: func(string) (bool, error) { return false, nil },
MarkMigrated: func(_, _ string) error { return nil },
OnProgress: func(_, _ int) {},
OnActivity: func() { activity++ },
}
r, err := CopyFolder(ctx, src, dst, "INBOX", "INBOX", deps)
if err != nil {
t.Fatalf("CopyFolder: %v", err)
}
if r.Copied != 1 {
t.Fatalf("copied=%d want 1", r.Copied)
}
if activity == 0 {
t.Fatal("OnActivity never called during body transfer")
}
}
// Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped). // Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped).
func TestCopyFolderIdempotent(t *testing.T) { func TestCopyFolderIdempotent(t *testing.T) {
ep := testEP(t) // plain greenmail ep := testEP(t) // plain greenmail
+79 -10
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"crypto/tls" "crypto/tls"
"fmt" "fmt"
"net"
"time" "time"
"github.com/emersion/go-imap/v2/imapclient" "github.com/emersion/go-imap/v2/imapclient"
@@ -17,31 +18,99 @@ type Endpoint struct {
func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) } func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) }
func dialOnce(ep Endpoint) (*imapclient.Client, error) { // Client wraps an imapclient.Client together with the raw network connection it
// runs over. The embedded *imapclient.Client provides the full IMAP API; the
// retained conn lets callers impose a read deadline on the socket for the
// duration of a body transfer.
//
// This defends against servers that announce a BODY[] literal LARGER than the
// bytes they actually send (a protocol violation observed on some webmail
// servers). go-imap reads a literal strictly by its announced size, so a short
// literal makes it block forever waiting for bytes that never arrive. A
// deadline around the body read turns that infinite hang into a timeout the
// copier can recover from.
type Client struct {
*imapclient.Client
conn net.Conn
}
// SetReadDeadline sets (or, with the zero time, clears) a deadline on the
// underlying socket. Used to bound a single body read; always cleared again
// once the read completes so it never affects idle periods.
func (c *Client) SetReadDeadline(t time.Time) error {
return c.conn.SetReadDeadline(t)
}
// dialTimeout bounds establishing the TCP connection.
const dialTimeout = 30 * time.Second
// dialOnce establishes one connection and returns a ready *Client. ctx bounds
// the TCP dial. We deliberately do NOT impose a socket-level read deadline:
// a blanket read deadline can't tell an idle connection (e.g. dst sitting idle
// during a long src scan) from one stuck mid-response, and would wrongly close
// idle connections. Stall detection is done at the orchestrator level via a
// progress watchdog; go-imap's own per-command timeouts bound active commands.
func dialOnce(ctx context.Context, ep Endpoint) (*Client, error) {
d := &net.Dialer{Timeout: dialTimeout}
raw, err := d.DialContext(ctx, "tcp", ep.addr())
if err != nil {
return nil, err
}
switch ep.TLSMode { switch ep.TLSMode {
case "ssl": case "ssl":
return imapclient.DialTLS(ep.addr(), &imapclient.Options{ // NextProtos mirrors imapclient.DialTLS's ALPN advertisement.
TLSConfig: &tls.Config{ServerName: ep.Host}, tlsConn := tls.Client(raw, &tls.Config{ServerName: ep.Host, NextProtos: []string{"imap"}})
}) if err := tlsConn.HandshakeContext(ctx); err != nil {
_ = raw.Close()
return nil, err
}
c, err := waitGreeting(imapclient.New(tlsConn, nil))
if err != nil {
return nil, err
}
return &Client{Client: c, conn: tlsConn}, nil
case "starttls": case "starttls":
return imapclient.DialStartTLS(ep.addr(), &imapclient.Options{ // Deadline goes on the raw TCP conn: it sits beneath the TLS layer that
TLSConfig: &tls.Config{ServerName: ep.Host}, // NewStartTLS negotiates, and a TCP read deadline still interrupts the
}) // TLS read above it.
opts := &imapclient.Options{TLSConfig: &tls.Config{ServerName: ep.Host}}
c, err := imapclient.NewStartTLS(raw, opts)
if err != nil {
return nil, err
}
return &Client{Client: c, conn: raw}, nil
case "plain": case "plain":
return imapclient.DialInsecure(ep.addr(), nil) c, err := waitGreeting(imapclient.New(raw, nil))
if err != nil {
return nil, err
}
return &Client{Client: c, conn: raw}, nil
default: default:
_ = raw.Close()
return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode) return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode)
} }
} }
func Connect(ctx context.Context, ep Endpoint) (*imapclient.Client, error) { // waitGreeting blocks for the server's initial greeting so a mute server is
// caught at connect time rather than at the first command. NewStartTLS already
// awaits the greeting during its STARTTLS upgrade.
func waitGreeting(c *imapclient.Client) (*imapclient.Client, error) {
if err := c.WaitGreeting(); err != nil {
_ = c.Close()
return nil, err
}
return c, nil
}
func Connect(ctx context.Context, ep Endpoint) (*Client, error) {
const attempts = 3 const attempts = 3
var lastErr error var lastErr error
for i := 0; i < attempts; i++ { for i := 0; i < attempts; i++ {
if err := ctx.Err(); err != nil { if err := ctx.Err(); err != nil {
return nil, err return nil, err
} }
c, err := dialOnce(ep) c, err := dialOnce(ctx, ep)
if err == nil { if err == nil {
return c, nil return c, nil
} }
+45
View File
@@ -0,0 +1,45 @@
package imapx
import (
"context"
"time"
)
// KeepaliveInterval is how often an otherwise-idle IMAP connection is pinged
// with NOOP so the server does not drop it.
//
// The destination connection sits completely idle for the entire duration of
// the source-side metadata scan (Pass 1 of CopyFolder), which on a large
// mailbox runs for many minutes across all folders. With no traffic, the
// server closes the idle connection; go-imap's reader then tears the client
// down, and every subsequent APPEND fails with "use of closed network
// connection" — aborting each folder and copying nothing. A periodic NOOP
// keeps the connection warm. 60s is well under the idle timeout of any common
// IMAP server.
const KeepaliveInterval = 60 * time.Second
// Keepalive pings c with a NOOP every interval until ctx is cancelled, keeping
// an idle connection from being dropped by the server. It is meant to run in
// its own goroutine.
//
// It is safe to run concurrently with other commands on c: go-imap serializes
// command submission and supports multiple in-flight commands over a single
// connection (RFC 9051 §5.5 pipelining). A NOOP issued while another command is
// in flight simply queues behind it and completes when the server responds.
//
// Keepalive returns when ctx is done or when a NOOP fails — a failed NOOP means
// the connection is already gone, so there is nothing left to keep alive.
func Keepalive(ctx context.Context, c *Client, interval time.Duration) {
t := time.NewTicker(interval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if err := c.Noop().Wait(); err != nil {
return
}
}
}
}
+89
View File
@@ -0,0 +1,89 @@
package imapx
import (
"context"
"testing"
"time"
)
// TestKeepaliveReturnsOnContextCancel proves Keepalive is a well-behaved
// goroutine: it exits promptly when its context is cancelled instead of
// leaking.
func TestKeepaliveReturnsOnContextCancel(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
c, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = c.Logout().Wait() }()
if err := c.Login("ka1@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
kctx, cancel := context.WithCancel(ctx)
done := make(chan struct{})
go func() { Keepalive(kctx, c, 10*time.Millisecond); close(done) }()
// Let a few NOOPs fire, then cancel and require a prompt return.
time.Sleep(50 * time.Millisecond)
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Keepalive did not return within 2s of context cancel")
}
}
// TestKeepaliveDoesNotDisruptCopy runs Keepalive on the destination connection
// at an aggressive interval while CopyFolder is APPENDing to it, proving the
// concurrent NOOPs do not corrupt in-flight commands (the real risk of pinging
// a connection that is also being used for real work).
func TestKeepaliveDoesNotDisruptCopy(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
const n = 8
seedInbox(t, ep, "kasrc@localhost", "p", n)
src, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = src.Logout().Wait() }()
if err := src.Login("kasrc@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
dst, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login("kadst@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
kctx, cancel := context.WithCancel(ctx)
defer cancel()
go Keepalive(kctx, dst, 1*time.Millisecond)
seen := map[string]bool{}
deps := CopyDeps{
IsMigrated: func(k string) (bool, error) { return seen[k], nil },
MarkMigrated: func(_, k string) error { seen[k] = true; return nil },
OnProgress: func(_, _ int) {},
}
r, err := CopyFolder(kctx, src, dst, "INBOX", "INBOX", deps)
if err != nil {
t.Fatalf("CopyFolder with concurrent keepalive: %v", err)
}
if r.Copied != n {
t.Fatalf("copied=%d want %d", r.Copied, n)
}
if r.Errors != 0 {
t.Fatalf("errors=%d want 0", r.Errors)
}
}
+218
View File
@@ -0,0 +1,218 @@
package imapx
import (
"bufio"
"context"
"errors"
"fmt"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/emersion/go-imap/v2"
)
// underflowServer is a minimal IMAP server that reproduces the amega.kz bug: it
// answers a BODY[] FETCH by announcing a literal LARGER than the bytes it then
// sends, and afterwards goes silent — exactly what makes go-imap's strict
// literal reader block forever. LOGIN/EXAMINE and the Pass-1 metadata FETCH are
// answered normally so a full CopyFolder can reach the poisoned message.
//
// It serves one connection per accept and keeps accepting, so a reconnect gets
// a fresh, well-behaved session (its second EXAMINE reports zero messages, so
// the resumed folder simply finishes).
type underflowServer struct {
ln net.Listener
mu sync.Mutex
accepts int // how many connections have been accepted
stop chan struct{}
}
func newUnderflowServer(t *testing.T) *underflowServer {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
s := &underflowServer{ln: ln, stop: make(chan struct{})}
go s.serve()
return s
}
func (s *underflowServer) addr() Endpoint {
a := s.ln.Addr().(*net.TCPAddr)
return Endpoint{Host: "127.0.0.1", Port: a.Port, TLSMode: "plain"}
}
func (s *underflowServer) close() { close(s.stop); _ = s.ln.Close() }
func (s *underflowServer) serve() {
for {
conn, err := s.ln.Accept()
if err != nil {
return
}
s.mu.Lock()
s.accepts++
first := s.accepts == 1
s.mu.Unlock()
go s.handle(conn, first)
}
}
// handle drives one connection. On the FIRST connection the mailbox reports one
// message and its BODY[] fetch under-delivers; on any later connection (i.e.
// after a reconnect) the mailbox is empty so the resumed folder completes.
func (s *underflowServer) handle(conn net.Conn, first bool) {
defer func() { _ = conn.Close() }()
br := bufio.NewReader(conn)
fmt.Fprint(conn, "* OK IMAP4rev1 ready\r\n")
for {
line, err := br.ReadString('\n')
if err != nil {
return
}
fields := strings.Fields(line)
if len(fields) == 0 {
continue
}
tag := fields[0]
up := strings.ToUpper(line)
switch {
case strings.Contains(up, "LOGIN"):
fmt.Fprintf(conn, "%s OK LOGIN completed\r\n", tag)
case strings.Contains(up, "EXAMINE"), strings.Contains(up, "SELECT"):
n := 0
if first {
n = 1
}
fmt.Fprintf(conn, "* %d EXISTS\r\n", n)
fmt.Fprint(conn, "* OK [UIDVALIDITY 1] ok\r\n")
fmt.Fprintf(conn, "%s OK [READ-ONLY] EXAMINE completed\r\n", tag)
case strings.Contains(up, "BODY["), strings.Contains(up, "BODY.PEEK"):
// Poison: announce 100000 bytes, send 10, then stall until shutdown.
fmt.Fprint(conn, "* 1 FETCH (UID 1 BODY[] {100000}\r\n")
fmt.Fprint(conn, "0123456789")
<-s.stop
return
case strings.Contains(up, "FETCH"):
// Pass-1 metadata fetch for the single message.
fmt.Fprint(conn, "* 1 FETCH (UID 1 RFC822.SIZE 100 FLAGS () "+
"INTERNALDATE \"01-Jan-2020 00:00:00 +0000\" "+
"ENVELOPE (\"Wed, 01 Jan 2020 00:00:00 +0000\" \"poison\" NIL NIL NIL NIL NIL NIL NIL \"<poison@x>\"))\r\n")
fmt.Fprintf(conn, "%s OK FETCH completed\r\n", tag)
case strings.Contains(up, "LOGOUT"):
fmt.Fprintf(conn, "* BYE\r\n%s OK LOGOUT completed\r\n", tag)
return
case strings.Contains(up, "CREATE"), strings.Contains(up, "NOOP"):
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
default:
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
}
}
}
// TestStreamOneTimesOutOnLiteralUnderflow proves a server that announces a
// larger BODY[] literal than it sends makes streamOne return ErrBodyTimeout
// (bounded by bodyIdleTimeout) instead of hanging forever.
func TestStreamOneTimesOutOnLiteralUnderflow(t *testing.T) {
restore := shortenBodyIdle(200 * time.Millisecond)
defer restore()
srv := newUnderflowServer(t)
defer srv.close()
ctx := context.Background()
src, err := Connect(ctx, srv.addr())
if err != nil {
t.Fatalf("connect: %v", err)
}
defer func() { _ = src.Close() }()
if err := src.Login("u", "p").Wait(); err != nil {
t.Fatalf("login: %v", err)
}
done := make(chan error, 1)
go func() {
done <- streamOne(src, src, "INBOX", imap.UID(1), nil, time.Time{}, nil)
}()
select {
case err := <-done:
if !errors.Is(err, ErrBodyTimeout) {
t.Fatalf("want ErrBodyTimeout, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("streamOne hung on literal underflow instead of timing out")
}
}
// TestCopyFolderSkipsAndReconnectsOnUnderflow proves CopyFolder does not hang on
// a poisoned message: it marks the message migrated (so future runs skip it),
// invokes ReconnectSrc, and returns with the error counted — the folder is not
// wedged.
func TestCopyFolderSkipsAndReconnectsOnUnderflow(t *testing.T) {
restore := shortenBodyIdle(200 * time.Millisecond)
defer restore()
srv := newUnderflowServer(t)
defer srv.close()
ctx := context.Background()
src, err := Connect(ctx, srv.addr())
if err != nil {
t.Fatalf("connect: %v", err)
}
defer func() { _ = src.Close() }()
if err := src.Login("u", "p").Wait(); err != nil {
t.Fatalf("login: %v", err)
}
var marked []string
var reconnected bool
deps := CopyDeps{
IsMigrated: func(string) (bool, error) { return false, nil },
MarkMigrated: func(_, k string) error { marked = append(marked, k); return nil },
OnProgress: func(_, _ int) {},
ReconnectSrc: func() (*Client, error) {
reconnected = true
nc, derr := Connect(ctx, srv.addr())
if derr != nil {
return nil, derr
}
if lerr := nc.Login("u", "p").Wait(); lerr != nil {
return nil, lerr
}
return nc, nil
},
}
done := make(chan CopyResult, 1)
go func() {
r, _ := CopyFolder(ctx, src, src, "INBOX", "INBOX", deps)
done <- r
}()
select {
case r := <-done:
if r.Errors == 0 {
t.Fatalf("expected the poisoned message counted as an error, got %+v", r)
}
if !reconnected {
t.Fatal("ReconnectSrc was never called")
}
if len(marked) != 1 {
t.Fatalf("poisoned message should be marked migrated once, got %v", marked)
}
case <-time.After(8 * time.Second):
t.Fatal("CopyFolder hung on a poisoned message instead of skipping it")
}
}
func shortenBodyIdle(d time.Duration) func() {
old := bodyIdleTimeout
bodyIdleTimeout = d
return func() { bodyIdleTimeout = old }
}
+21
View File
@@ -0,0 +1,21 @@
package orchestrator
import "testing"
func TestShouldBreak(t *testing.T) {
cases := []struct {
trigger string
errs int64
want bool
}{
{"scheduled", 2, true}, // scheduled run with errors trips the breaker
{"scheduled", 0, false}, // scheduled run, clean — no break
{"manual", 5, false}, // manual run never trips the breaker
{"manual", 0, false},
}
for _, c := range cases {
if got := shouldBreak(c.trigger, c.errs); got != c.want {
t.Fatalf("shouldBreak(%q,%d)=%v want %v", c.trigger, c.errs, got, c.want)
}
}
}
+323 -34
View File
@@ -5,6 +5,7 @@ import (
"errors" "errors"
"log/slog" "log/slog"
"sync" "sync"
"sync/atomic"
"time" "time"
"github.com/vasyansk/imap-copier/internal/crypto" "github.com/vasyansk/imap-copier/internal/crypto"
@@ -15,6 +16,24 @@ import (
var ErrNotTested = errors.New("accounts not fully tested") var ErrNotTested = errors.New("accounts not fully tested")
var ErrAlreadyRunning = errors.New("task already running") var ErrAlreadyRunning = errors.New("task already running")
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
var ErrNothingToResume = errors.New("no paused accounts to resume")
// maxAccountErrors caps how many individual error rows one account records per
// run, so a corrupt mailbox producing thousands of failures can't bloat the
// account_errors table. The cap'th row is a synthetic "further errors
// suppressed" note.
const maxAccountErrors = 500
// A running account that emits no scan/copy progress for stallTimeout is wedged
// (silent server mid-FETCH, stalled APPEND). The watchdog cancels it so the
// connections close and the worker unwinds instead of hanging forever. This
// replaces socket-level read deadlines, which can't tell an idle connection
// from a stuck one. The threshold is generous so slow-but-live runs aren't cut.
const (
stallTimeout = 3 * time.Minute
stallCheckInterval = 30 * time.Second
)
// folderPlan is one source folder scheduled for copy and its destination name. // folderPlan is one source folder scheduled for copy and its destination name.
type folderPlan struct { type folderPlan struct {
@@ -44,6 +63,38 @@ func planFolders(folders []string, mapping map[string]string, excluded []string)
return plan return plan
} }
// A run stops either on its own or because the operator intervened. Pausing and
// cancelling take the same path — stop the in-flight work — and differ only in
// the status left behind: paused accounts are what Resume picks up again.
type stopReason int32
const (
stopNone stopReason = iota
stopPaused
stopCancelled
)
func (r stopReason) accountStatus() string {
if r == stopPaused {
return "paused"
}
return "cancelled"
}
// runHandle is the live state of one task's run: the cancel that stops every
// account under it, plus why it was stopped.
type runHandle struct {
cancel context.CancelFunc
reason atomic.Int32
}
func (h *runHandle) stopWith(r stopReason) {
h.reason.CompareAndSwap(int32(stopNone), int32(r))
h.cancel()
}
func (h *runHandle) stopReason() stopReason { return stopReason(h.reason.Load()) }
type Orchestrator struct { type Orchestrator struct {
store *store.Store store *store.Store
hub *wshub.Hub hub *wshub.Hub
@@ -52,10 +103,66 @@ type Orchestrator struct {
mu sync.Mutex mu sync.Mutex
cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy
runs map[int64]*runHandle // task_id -> live run
} }
func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator { func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator {
return &Orchestrator{store: s, hub: hub, encKey: encKey, concurrency: concurrency, cancels: map[int64]context.CancelFunc{}} return &Orchestrator{
store: s, hub: hub, encKey: encKey, concurrency: concurrency,
cancels: map[int64]context.CancelFunc{},
runs: map[int64]*runHandle{},
}
}
// PauseTask stops the task's live run, leaving every unfinished account
// "paused" so ResumeTask can pick them up. Returns false if nothing is running.
func (o *Orchestrator) PauseTask(taskID int64) bool { return o.stopRun(taskID, stopPaused) }
// CancelTask stops the task's live run and marks every unfinished account
// "cancelled". Returns false if nothing is running.
func (o *Orchestrator) CancelTask(taskID int64) bool { return o.stopRun(taskID, stopCancelled) }
func (o *Orchestrator) stopRun(taskID int64, reason stopReason) bool {
o.mu.Lock()
h, ok := o.runs[taskID]
o.mu.Unlock()
if !ok {
return false
}
h.stopWith(reason)
return true
}
func (o *Orchestrator) registerRun(taskID int64, h *runHandle) {
o.mu.Lock()
o.runs[taskID] = h
o.mu.Unlock()
}
func (o *Orchestrator) unregisterRun(taskID int64) {
o.mu.Lock()
delete(o.runs, taskID)
o.mu.Unlock()
}
// ResumeTask restarts a paused task with exactly the accounts the pause left
// unfinished. Everything already copied is skipped by the migration journal, so
// each account continues where it stopped.
func (o *Orchestrator) ResumeTask(ctx context.Context, taskID int64) (int64, error) {
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
ids := make([]int64, 0, len(accs))
for _, a := range accs {
if a.Status == "paused" {
ids = append(ids, a.ID)
}
}
if len(ids) == 0 {
return 0, ErrNothingToResume
}
return o.Run(ctx, taskID, "manual", ids)
} }
// CancelAccount aborts the in-flight copy for one account, if it is running. // CancelAccount aborts the in-flight copy for one account, if it is running.
@@ -82,6 +189,26 @@ func (o *Orchestrator) unregisterCancel(accountID int64) {
o.mu.Unlock() o.mu.Unlock()
} }
// selectAccounts narrows accs to those whose ID is in ids, preserving input
// order. An empty or nil ids means "all accounts" — the scheduler and the
// unfiltered manual run rely on this.
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
if len(ids) == 0 {
return accs
}
want := make(map[int64]struct{}, len(ids))
for _, id := range ids {
want[id] = struct{}{}
}
out := make([]store.Account, 0, len(ids))
for _, a := range accs {
if _, ok := want[a.ID]; ok {
out = append(out, a)
}
}
return out
}
func gateOK(accs []store.Account) bool { func gateOK(accs []store.Account) bool {
if len(accs) == 0 { if len(accs) == 0 {
return false return false
@@ -150,7 +277,13 @@ func (o *Orchestrator) testSide(ctx context.Context, ep imapx.Endpoint, accID in
}}) }})
} }
func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) { // shouldBreak reports whether a completed run should trip the schedule breaker:
// only scheduled runs that ended with errors.
func shouldBreak(trigger string, totErr int64) bool {
return trigger == "scheduled" && totErr > 0
}
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
task, err := o.store.GetTask(ctx, taskID) task, err := o.store.GetTask(ctx, taskID)
if err != nil { if err != nil {
return 0, err return 0, err
@@ -159,6 +292,12 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) {
if err != nil { if err != nil {
return 0, err return 0, err
} }
accs = selectAccounts(accs, accountIDs)
// A non-empty request that matched nothing is a client error, distinct
// from "not tested".
if len(accountIDs) > 0 && len(accs) == 0 {
return 0, ErrNoAccountsSelected
}
if !gateOK(accs) { if !gateOK(accs) {
return 0, ErrNotTested return 0, ErrNotTested
} }
@@ -174,23 +313,38 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64) (int64, error) {
_ = o.store.SetTaskStatus(ctx, taskID, "error") _ = o.store.SetTaskStatus(ctx, taskID, "error")
return 0, err return 0, err
} }
runID, err := o.store.CreateRun(ctx, taskID) runID, err := o.store.CreateRun(ctx, taskID, trigger)
if err != nil { if err != nil {
_ = o.store.SetTaskStatus(ctx, taskID, "error") _ = o.store.SetTaskStatus(ctx, taskID, "error")
return 0, err return 0, err
} }
o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}}) o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}})
go o.runAll(context.WithoutCancel(ctx), task, runID, accs, srcEP, dstEP) // dbCtx outlives the request so status/counter writes still land after a
// pause or cancel; runCtx is what Pause/Cancel actually stop, and every
// account's IMAP work hangs off it.
dbCtx := context.WithoutCancel(ctx)
runCtx, runCancel := context.WithCancel(dbCtx)
h := &runHandle{cancel: runCancel}
o.registerRun(taskID, h)
go o.runAll(dbCtx, runCtx, h, task, runID, accs, srcEP, dstEP, trigger)
return runID, nil return runID, nil
} }
func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint) { func (o *Orchestrator) runAll(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
defer o.unregisterRun(task.ID)
defer h.cancel()
defer func() { defer func() {
if r := recover(); r != nil { if r := recover(); r != nil {
slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r) slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r)
_ = o.store.FinishRun(ctx, runID, "error", 0, 0, 0) _ = o.store.FinishRun(ctx, runID, "error", 0, 0, 0)
_ = o.store.SetTaskStatus(ctx, task.ID, "error") _ = o.store.SetTaskStatus(ctx, task.ID, "error")
if trigger == "scheduled" {
_ = o.store.SetTaskBroken(ctx, task.ID)
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
Data: map[string]any{"task_id": task.ID, "errors": int64(0)}})
}
} }
}() }()
@@ -201,7 +355,17 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
sem := make(chan struct{}, o.concurrency) sem := make(chan struct{}, o.concurrency)
var wg sync.WaitGroup var wg sync.WaitGroup
for _, a := range accs { for i, a := range accs {
// Stopped mid-queue: the accounts that never started are marked with the
// same status as the ones that were interrupted, so a pause leaves the
// whole remainder resumable and a cancel leaves it cancelled.
if runCtx.Err() != nil {
st := h.stopReason().accountStatus()
for _, rest := range accs[i:] {
_ = o.store.SetAccountStatus(ctx, rest.ID, st)
}
break
}
wg.Add(1) wg.Add(1)
sem <- struct{}{} sem <- struct{}{}
go func(a store.Account) { go func(a store.Account) {
@@ -218,7 +382,7 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
mu.Unlock() mu.Unlock()
} }
}() }()
c, s, e := o.runAccount(ctx, task, runID, a, srcEP, dstEP) c, s, e := o.runAccount(ctx, runCtx, h, task, runID, a, srcEP, dstEP)
mu.Lock() mu.Lock()
totCopied += c totCopied += c
totSkipped += s totSkipped += s
@@ -228,30 +392,47 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
} }
wg.Wait() wg.Wait()
reason := h.stopReason()
status := "done" status := "done"
if totErr > 0 { switch {
case reason == stopPaused:
status = "paused"
case reason == stopCancelled:
status = "cancelled"
case totErr > 0:
status = "done_with_errors" status = "done_with_errors"
} }
_ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr) _ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr)
_ = o.store.SetTaskStatus(ctx, task.ID, status) _ = o.store.SetTaskStatus(ctx, task.ID, status)
o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID, o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID,
Data: map[string]any{"run_id": runID, "copied": totCopied, "skipped": totSkipped, "errors": totErr}}) Data: map[string]any{"run_id": runID, "status": status,
"copied": totCopied, "skipped": totSkipped, "errors": totErr}})
// An operator stopping the run is not a schedule failure, so leave the
// breaker alone even when the accounts that did run reported errors.
if reason == stopNone && shouldBreak(trigger, totErr) {
_ = o.store.SetTaskBroken(ctx, task.ID)
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
Data: map[string]any{"task_id": task.ID, "errors": totErr}})
}
} }
func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) { func (o *Orchestrator) runAccount(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{ o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "account_id": a.ID,
"src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port, "src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port,
"dst_login": a.DstLogin, "dst_host": dstEP.Host, "dst_port": dstEP.Port, "dst_login": a.DstLogin, "dst_host": dstEP.Host, "dst_port": dstEP.Port,
}}) }})
_ = o.store.SetAccountStatus(ctx, a.ID, "running") _ = o.store.SetAccountStatus(ctx, a.ID, "running")
_ = o.store.SetAccountError(ctx, a.ID, "") // clear any error from a previous run _ = o.store.SetAccountError(ctx, a.ID, "") // clear any error from a previous run
_ = o.store.ResetAccountCounters(ctx, a.ID) // start from zero; IncAccountCounters is additive _ = o.store.ResetAccountCounters(ctx, a.ID) // start from zero; IncAccountCounters is additive
_ = o.store.ClearAccountErrors(ctx, a.ID) // drop last run's per-error rows
// Per-account cancellable context: IMAP work uses actx (so CancelAccount // Per-account cancellable context: IMAP work uses actx, so both CancelAccount
// stops it); DB writes keep the parent ctx so status/counters persist even // and a task-wide pause/cancel (which cancels runCtx) stop it. DB writes keep
// after cancellation. ctx is context.WithoutCancel from runAll. // ctx — the uncancellable one from runAll — so status/counters persist even
actx, cancel := context.WithCancel(ctx) // after cancellation.
actx, cancel := context.WithCancel(runCtx)
o.registerCancel(a.ID, cancel) o.registerCancel(a.ID, cancel)
defer func() { defer func() {
o.unregisterCancel(a.ID) o.unregisterCancel(a.ID)
@@ -260,41 +441,107 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc) srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc)
if err != nil { if err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
} }
dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc) dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc)
if err != nil { if err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
} }
src, err := imapx.Connect(actx, srcEP) src, err := imapx.Connect(actx, srcEP)
if err != nil { if err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
} }
defer func() { _ = src.Logout().Wait() }()
if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil { if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err) _ = src.Logout().Wait()
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
} }
// srcClient holds the LIVE source connection. A body-read timeout (server
// under-delivering a literal) forces a mid-run reconnect via reconnectSrc,
// which swaps this pointer. The cancel goroutine and the deferred logout
// below both read through it, so they always act on the current connection
// rather than a stale one that was already replaced and logged out.
var srcClient atomic.Pointer[imapx.Client]
srcClient.Store(src)
defer func() { _ = srcClient.Load().Logout().Wait() }()
dst, err := imapx.Connect(actx, dstEP) dst, err := imapx.Connect(actx, dstEP)
if err != nil { if err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
} }
defer func() { _ = dst.Logout().Wait() }() defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil { if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "dst", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
}
// reconnectSrc dials and logs in a fresh source client, swaps it in as the
// live connection, and logs the old (desynced) one out. CopyFolder calls it
// to recover after a message body times out: the server left the connection
// mid-literal, so it can't be reused. Bound to actx, so a cancelled account
// fails the dial instead of reconnecting.
reconnectSrc := func() (*imapx.Client, error) {
nc, err := imapx.Connect(actx, srcEP)
if err != nil {
return nil, err
}
if err := nc.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
_ = nc.Logout().Wait()
return nil, err
}
if old := srcClient.Swap(nc); old != nil {
_ = old.Logout().Wait()
}
slog.Info("reconnected src after message body timeout", "account", a.ID, "src_login", a.SrcLogin)
return nc, nil
} }
// On cancel, close the connections so any in-flight network read (a slow // On cancel, close the connections so any in-flight network read (a slow
// FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately. // FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately.
go func() { go func() {
<-actx.Done() <-actx.Done()
_ = src.Close() _ = srcClient.Load().Close()
_ = dst.Close() _ = dst.Close()
}() }()
// Keep the destination connection warm. It sits idle for the whole
// source-side metadata scan (Pass 1 of CopyFolder), which on a large
// mailbox runs for minutes; without traffic the server drops it and every
// subsequent APPEND fails with "use of closed network connection", copying
// nothing. Periodic NOOPs prevent that. Only dst needs it — src is
// continuously busy scanning/fetching. Bound to actx so it stops with the
// account.
go imapx.Keepalive(actx, dst, imapx.KeepaliveInterval)
// Progress watchdog: track the last time we saw scan/copy activity; if it
// goes quiet for stallTimeout, cancel the account so the connections close
// and this worker unwinds (it would otherwise block forever on a silent
// server). touch() is called on every progress signal below.
var lastActivity atomic.Int64
lastActivity.Store(time.Now().UnixNano())
touch := func() { lastActivity.Store(time.Now().UnixNano()) }
go func() {
t := time.NewTicker(stallCheckInterval)
defer t.Stop()
for {
select {
case <-actx.Done():
return
case <-t.C:
if time.Since(time.Unix(0, lastActivity.Load())) > stallTimeout {
slog.Warn("account stalled with no progress; cancelling",
"account", a.ID, "src_login", a.SrcLogin, "stall", stallTimeout)
_ = o.store.SetAccountError(ctx, a.ID, "stalled: no progress for "+stallTimeout.String()+", cancelled")
cancel()
return
}
}
}
}()
folders, err := imapx.ListFolders(src) folders, err := imapx.ListFolders(src)
touch()
if err != nil { if err != nil {
return o.accountFailed(ctx, task.ID, a, srcEP, dstEP, "src", err) return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
} }
// Planning pass: decide folders from the account's own config, then EXAMINE // Planning pass: decide folders from the account's own config, then EXAMINE
@@ -311,6 +558,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
} }
plan[i].total = n plan[i].total = n
grandTotal += n grandTotal += n
touch()
} }
o.hub.Publish(wshub.Event{Type: "plan", TaskID: task.ID, Data: map[string]any{ o.hub.Publish(wshub.Event{Type: "plan", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "src_login": a.SrcLogin, "folders": len(plan), "total": grandTotal, "account_id": a.ID, "src_login": a.SrcLogin, "folders": len(plan), "total": grandTotal,
@@ -324,10 +572,34 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
var curFolder string var curFolder string
var curTotal int64 var curTotal int64
var lastEmit, lastScanEmit time.Time var lastEmit, lastScanEmit time.Time
// Persist individual errors for the per-account error modal, capped so a
// corrupt mailbox can't write unbounded rows. Runs on this goroutine, so
// the counter is race-free (same reasoning as the progress vars above).
var persistedErrs int
addErr := func(kind, folder, ref, msg string) {
persistedErrs++
switch {
case persistedErrs < maxAccountErrors:
_ = o.store.AddAccountError(ctx, a.ID, runID, kind, folder, ref, msg)
case persistedErrs == maxAccountErrors:
_ = o.store.AddAccountError(ctx, a.ID, runID, "account", "", "",
"too many errors — further errors suppressed")
}
}
deps := imapx.CopyDeps{ deps := imapx.CopyDeps{
IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) }, IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) },
MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) }, MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) },
OnError: func(ref, msg string) { addErr("message", curFolder, ref, msg) },
// Fires as bytes move within a single message's FETCH/APPEND, so the
// stall watchdog sees a large-but-live transfer as progress instead of
// cancelling it as a wedged connection.
OnActivity: touch,
// Recovers from a message body timeout (server under-delivering a
// literal) by swapping in a fresh source connection so the folder can
// resume with the remaining messages.
ReconnectSrc: reconnectSrc,
OnProgress: func(c, s int) { OnProgress: func(c, s int) {
touch()
now := time.Now() now := time.Now()
done := c + s done := c + s
// throttle to ~3/sec per account, but always emit folder completion // throttle to ~3/sec per account, but always emit folder completion
@@ -347,6 +619,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
}, },
// Fires after EXAMINE (before the long fetch) with the folder's message count. // Fires after EXAMINE (before the long fetch) with the folder's message count.
OnFolder: func(srcFolder, dstFolder string, total int64) { OnFolder: func(srcFolder, dstFolder string, total int64) {
touch()
curFolder, curTotal = srcFolder, total curFolder, curTotal = srcFolder, total
o.hub.Publish(wshub.Event{Type: "folder", TaskID: task.ID, Data: map[string]any{ o.hub.Publish(wshub.Event{Type: "folder", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "src_login": a.SrcLogin, "account_id": a.ID, "src_login": a.SrcLogin,
@@ -356,6 +629,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
// Fires while streaming metadata (dedup scan) so the UI shows movement // Fires while streaming metadata (dedup scan) so the UI shows movement
// before bodies start copying. Throttled to ~4/sec, always emit the last. // before bodies start copying. Throttled to ~4/sec, always emit the last.
OnScan: func(scanned, total int64) { OnScan: func(scanned, total int64) {
touch()
now := time.Now() now := time.Now()
if now.Sub(lastScanEmit) < 250*time.Millisecond && scanned < total { if now.Sub(lastScanEmit) < 250*time.Millisecond && scanned < total {
return return
@@ -370,12 +644,13 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
if actx.Err() != nil { if actx.Err() != nil {
break // cancelled — stop scheduling more folders break // cancelled — stop scheduling more folders
} }
res, err := imapx.CopyFolder(actx, src, dst, fp.src, fp.dst, deps) res, err := imapx.CopyFolder(actx, srcClient.Load(), dst, fp.src, fp.dst, deps)
folderErr := int64(0) folderErr := int64(0)
if err != nil && actx.Err() == nil { if err != nil && actx.Err() == nil {
slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err) slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err)
folderErr = 1 folderErr = 1
_ = o.store.SetAccountError(ctx, a.ID, "folder \""+fp.src+"\": "+err.Error()) _ = o.store.SetAccountError(ctx, a.ID, "folder \""+fp.src+"\": "+err.Error())
addErr("folder", fp.src, "", err.Error())
o.hub.Publish(wshub.Event{Type: "error", TaskID: task.ID, Data: map[string]any{ o.hub.Publish(wshub.Event{Type: "error", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "src_login": a.SrcLogin, "folder": fp.src, "error": err.Error(), "account_id": a.ID, "src_login": a.SrcLogin, "folder": fp.src, "error": err.Error(),
}}) }})
@@ -391,11 +666,18 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
} }
if actx.Err() != nil { if actx.Err() != nil {
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled") // A task-wide pause leaves the account resumable; anything else (per-account
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: task.ID, // cancel, stall watchdog, task-wide cancel) leaves it cancelled.
st := "cancelled"
if runCtx.Err() != nil {
st = h.stopReason().accountStatus()
}
_ = o.store.SetAccountStatus(ctx, a.ID, st)
o.hub.Publish(wshub.Event{Type: st, TaskID: task.ID,
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin, Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin,
"copied": copied, "skipped": skipped, "errors": errs}}) "copied": copied, "skipped": skipped, "errors": errs}})
slog.Info("account cancelled", "account", a.ID, "src_login", a.SrcLogin, "copied", copied, "skipped", skipped) slog.Info("account stopped", "account", a.ID, "src_login", a.SrcLogin,
"status", st, "copied", copied, "skipped", skipped)
return copied, skipped, errs return copied, skipped, errs
} }
@@ -411,11 +693,16 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
return copied, skipped, errs return copied, skipped, errs
} }
func (o *Orchestrator) accountFailed(ctx context.Context, taskID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) { func (o *Orchestrator) accountFailed(ctx, runCtx context.Context, h *runHandle, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
// A cancellation surfacing as an error is a cancel, not a failure. // A cancellation surfacing as an error is a stop, not a failure — and a
// task-wide pause must still leave the account resumable.
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled") st := "cancelled"
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: taskID, if runCtx.Err() != nil {
st = h.stopReason().accountStatus()
}
_ = o.store.SetAccountStatus(ctx, a.ID, st)
o.hub.Publish(wshub.Event{Type: st, TaskID: taskID,
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}}) Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}})
return 0, 0, 0 return 0, 0, 0
} }
@@ -425,7 +712,9 @@ func (o *Orchestrator) accountFailed(ctx context.Context, taskID int64, a store.
} }
slog.Error("account failed", "account", a.ID, "side", side, "login", login, "host", host, "port", port, "err", err) slog.Error("account failed", "account", a.ID, "side", side, "login", login, "host", host, "port", port, "err", err)
_ = o.store.SetAccountStatus(ctx, a.ID, "error") _ = o.store.SetAccountStatus(ctx, a.ID, "error")
_ = o.store.SetAccountError(ctx, a.ID, side+" "+login+"@"+host+": "+err.Error()) failMsg := side + " " + login + "@" + host + ": " + err.Error()
_ = o.store.SetAccountError(ctx, a.ID, failMsg)
_ = o.store.AddAccountError(ctx, a.ID, runID, "account", "", "", failMsg)
o.hub.Publish(wshub.Event{Type: "error", TaskID: taskID, o.hub.Publish(wshub.Event{Type: "error", TaskID: taskID,
Data: map[string]any{"account_id": a.ID, "side": side, "login": login, "host": host, "port": port, "error": err.Error()}}) Data: map[string]any{"account_id": a.ID, "side": side, "login": login, "host": host, "port": port, "error": err.Error()}})
return 0, 0, 1 return 0, 0, 1
@@ -22,3 +22,23 @@ func TestGateOK(t *testing.T) {
t.Fatal("empty accounts must fail gate") t.Fatal("empty accounts must fail gate")
} }
} }
func TestSelectAccounts(t *testing.T) {
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
if got := selectAccounts(accs, nil); len(got) != 3 {
t.Fatalf("nil ids must return all, got %d", len(got))
}
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
t.Fatalf("empty ids must return all, got %d", len(got))
}
got := selectAccounts(accs, []int64{3, 1})
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
t.Fatalf("must keep matching ids in input order, got %+v", got)
}
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
t.Fatalf("unknown ids must yield empty, got %d", len(got))
}
}
+83
View File
@@ -0,0 +1,83 @@
package orchestrator
import (
"context"
"testing"
)
func TestStopReasonAccountStatus(t *testing.T) {
if got := stopPaused.accountStatus(); got != "paused" {
t.Fatalf("stopPaused = %q want paused", got)
}
if got := stopCancelled.accountStatus(); got != "cancelled" {
t.Fatalf("stopCancelled = %q want cancelled", got)
}
// A run stopped without an operator reason (per-account cancel, stall
// watchdog) must not look like a pause, or Resume would pick it up.
if got := stopNone.accountStatus(); got != "cancelled" {
t.Fatalf("stopNone = %q want cancelled", got)
}
}
// The first stop wins: a cancel arriving after a pause must not downgrade the
// accounts a pause already promised to keep resumable, and vice versa.
func TestRunHandleFirstStopWins(t *testing.T) {
for _, tc := range []struct {
name string
first, later stopReason
}{
{"pause then cancel", stopPaused, stopCancelled},
{"cancel then pause", stopCancelled, stopPaused},
} {
t.Run(tc.name, func(t *testing.T) {
_, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
h.stopWith(tc.first)
h.stopWith(tc.later)
if got := h.stopReason(); got != tc.first {
t.Fatalf("reason = %v want %v", got, tc.first)
}
})
}
}
func TestRunHandleStopCancelsContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
if ctx.Err() != nil {
t.Fatal("context cancelled before stop")
}
h.stopWith(stopPaused)
if ctx.Err() == nil {
t.Fatal("stop must cancel the run context")
}
}
// Pause/Cancel report false for a task with no live run, which the HTTP layer
// turns into 409 instead of pretending it stopped something.
func TestStopRunWithoutLiveRun(t *testing.T) {
o := &Orchestrator{runs: map[int64]*runHandle{}}
if o.PauseTask(1) {
t.Fatal("PauseTask must report false with no live run")
}
if o.CancelTask(1) {
t.Fatal("CancelTask must report false with no live run")
}
_, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
o.registerRun(1, h)
if !o.PauseTask(1) {
t.Fatal("PauseTask must report true for a live run")
}
if got := h.stopReason(); got != stopPaused {
t.Fatalf("reason = %v want stopPaused", got)
}
o.unregisterRun(1)
if o.CancelTask(1) {
t.Fatal("unregistered run must not be stoppable")
}
}
+75
View File
@@ -0,0 +1,75 @@
// Package scheduler runs tasks on their recurring interval. A single goroutine
// polls the DB every pollInterval and triggers due tasks through the
// orchestrator. The DB is the source of truth, so the scheduler holds no state
// and is safe across restarts.
package scheduler
import (
"context"
"log/slog"
"time"
"github.com/vasyansk/imap-copier/internal/orchestrator"
"github.com/vasyansk/imap-copier/internal/store"
)
const pollInterval = 30 * time.Second
// NextRun is when a task should next run: one interval after its last completed
// run, or one interval after the schedule anchor if it has never completed one.
func NextRun(interval time.Duration, anchor time.Time, lastFinished *time.Time) time.Time {
base := anchor
if lastFinished != nil && lastFinished.After(anchor) {
base = *lastFinished
}
return base.Add(interval)
}
// dueTaskIDs returns the IDs of tasks whose next run is at or before now.
func dueTaskIDs(tasks []store.SchedulableTask, now time.Time) []int64 {
var ids []int64
for _, t := range tasks {
next := NextRun(time.Duration(t.IntervalSeconds)*time.Second, t.Anchor, t.LastFinished)
if !now.Before(next) {
ids = append(ids, t.ID)
}
}
return ids
}
type Scheduler struct {
store *store.Store
orch *orchestrator.Orchestrator
}
func New(st *store.Store, orch *orchestrator.Orchestrator) *Scheduler {
return &Scheduler{store: st, orch: orch}
}
// Start blocks, ticking until ctx is cancelled. Run it in a goroutine.
func (s *Scheduler) Start(ctx context.Context) {
ticker := time.NewTicker(pollInterval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
s.tick(ctx)
}
}
}
func (s *Scheduler) tick(ctx context.Context) {
tasks, err := s.store.ListSchedulableTasks(ctx)
if err != nil {
slog.Error("scheduler: list schedulable", "err", err)
return
}
for _, id := range dueTaskIDs(tasks, time.Now()) {
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
// ErrAlreadyRunning / ErrNotTested are expected races/edge cases, not fatal.
slog.Info("scheduler: run skipped", "task", id, "err", err)
}
}
}
+44
View File
@@ -0,0 +1,44 @@
package scheduler
import (
"testing"
"time"
"github.com/vasyansk/imap-copier/internal/store"
)
func TestNextRun(t *testing.T) {
anchor := time.Date(2026, 7, 3, 10, 0, 0, 0, time.UTC)
fin := time.Date(2026, 7, 3, 11, 0, 0, 0, time.UTC)
// No finished run yet → anchor + interval.
if got := NextRun(time.Hour, anchor, nil); !got.Equal(anchor.Add(time.Hour)) {
t.Fatalf("first run: got %v", got)
}
// Finished run exists → last finished + interval (from completion, not start).
if got := NextRun(time.Hour, anchor, &fin); !got.Equal(fin.Add(time.Hour)) {
t.Fatalf("recurring: got %v", got)
}
// Re-enable: anchor is newer than a stale lastFinished from before re-enable →
// use anchor, not the stale lastFinished (which would fire immediately/in the past).
staleFin := anchor.Add(-2 * time.Hour)
if got := NextRun(time.Hour, anchor, &staleFin); !got.Equal(anchor.Add(time.Hour)) {
t.Fatalf("re-enable: got %v, want %v", got, anchor.Add(time.Hour))
}
}
func TestDueTaskIDs(t *testing.T) {
now := time.Date(2026, 7, 3, 12, 0, 0, 0, time.UTC)
anchorOld := now.Add(-2 * time.Hour) // enabled 2h ago
finRecent := now.Add(-30 * time.Minute)
tasks := []store.SchedulableTask{
{ID: 1, IntervalSeconds: 3600, Anchor: anchorOld, LastFinished: nil}, // due: anchor+1h < now
{ID: 2, IntervalSeconds: 3600, Anchor: anchorOld, LastFinished: &finRecent}, // not due: fin+1h > now
{ID: 3, IntervalSeconds: 21600, Anchor: anchorOld, LastFinished: nil}, // not due: anchor+6h > now
}
ids := dueTaskIDs(tasks, now)
if len(ids) != 1 || ids[0] != 1 {
t.Fatalf("due IDs = %v, want [1]", ids)
}
}
+61
View File
@@ -0,0 +1,61 @@
package store
import (
"context"
"time"
)
// AccountError is one concrete error recorded during an account's run —
// folder-level, message-level, or account-level (connect/login).
type AccountError struct {
ID int64 `json:"id"`
AccountID int64 `json:"account_id"`
RunID int64 `json:"run_id"`
Kind string `json:"kind"` // folder | message | account
Folder string `json:"folder"`
MessageRef string `json:"message_ref"`
Error string `json:"error"`
CreatedAt time.Time `json:"created_at"`
}
// AddAccountError appends one error row for an account's current run.
func (s *Store) AddAccountError(ctx context.Context, accountID, runID int64, kind, folder, ref, msg string) error {
_, err := s.Pool.Exec(ctx,
`INSERT INTO account_errors (account_id, run_id, kind, folder, message_ref, error)
VALUES ($1,$2,$3,$4,$5,$6)`,
accountID, runID, kind, folder, ref, msg)
return err
}
// ClearAccountErrors removes an account's errors at the start of a run, so the
// list reflects only the current run (mirrors ResetAccountCounters).
func (s *Store) ClearAccountErrors(ctx context.Context, accountID int64) error {
_, err := s.Pool.Exec(ctx, `DELETE FROM account_errors WHERE account_id=$1`, accountID)
return err
}
// ListAccountErrors returns an account's errors in insertion order, for the
// per-account error modal.
func (s *Store) ListAccountErrors(ctx context.Context, accountID int64) ([]AccountError, error) {
rows, err := s.Pool.Query(ctx,
`SELECT id, account_id, run_id, kind, folder, message_ref, error, created_at
FROM account_errors WHERE account_id=$1 ORDER BY id`, accountID)
if err != nil {
return nil, err
}
defer rows.Close()
out := []AccountError{}
for rows.Next() {
var e AccountError
var runID *int64
if err := rows.Scan(&e.ID, &e.AccountID, &runID, &e.Kind, &e.Folder,
&e.MessageRef, &e.Error, &e.CreatedAt); err != nil {
return nil, err
}
if runID != nil {
e.RunID = *runID
}
out = append(out, e)
}
return out, rows.Err()
}
+75
View File
@@ -0,0 +1,75 @@
package store
import (
"context"
"testing"
)
// AddAccountError persists individual errors; ListAccountErrors returns them in
// insertion order; ClearAccountErrors wipes them for the next run.
func TestAccountErrorsAddListClear(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
runID, _ := s.CreateRun(ctx, taskID, "manual")
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
if err := s.AddAccountError(ctx, accID, runID, "folder", "INBOX", "", "examine failed"); err != nil {
t.Fatalf("add folder: %v", err)
}
if err := s.AddAccountError(ctx, accID, runID, "message", "INBOX", "UID 42: hi", "append rejected"); err != nil {
t.Fatalf("add message: %v", err)
}
errs, err := s.ListAccountErrors(ctx, accID)
if err != nil {
t.Fatalf("list: %v", err)
}
if len(errs) != 2 {
t.Fatalf("len=%d want 2", len(errs))
}
if errs[0].Kind != "folder" || errs[0].Folder != "INBOX" || errs[0].Error != "examine failed" {
t.Fatalf("errs[0]=%+v", errs[0])
}
if errs[1].Kind != "message" || errs[1].MessageRef != "UID 42: hi" || errs[1].Error != "append rejected" {
t.Fatalf("errs[1]=%+v", errs[1])
}
if errs[0].ID >= errs[1].ID {
t.Fatalf("expected insertion order by id: %d then %d", errs[0].ID, errs[1].ID)
}
if err := s.ClearAccountErrors(ctx, accID); err != nil {
t.Fatalf("clear: %v", err)
}
errs, err = s.ListAccountErrors(ctx, accID)
if err != nil {
t.Fatalf("list after clear: %v", err)
}
if len(errs) != 0 {
t.Fatalf("after clear len=%d want 0", len(errs))
}
}
// Deleting an account cascades its errors (ON DELETE CASCADE).
func TestAccountErrorsCascadeOnAccountDelete(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "u2", DstPassEnc: "y"})
_ = s.AddAccountError(ctx, accID, 0, "account", "", "", "login failed")
if err := s.DeleteAccount(ctx, accID); err != nil {
t.Fatalf("delete: %v", err)
}
var n int
if err := s.Pool.QueryRow(ctx, `SELECT count(*) FROM account_errors WHERE account_id=$1`, accID).Scan(&n); err != nil {
t.Fatalf("count: %v", err)
}
if n != 0 {
t.Fatalf("account_errors not cascaded: %d rows", n)
}
}
+15
View File
@@ -32,6 +32,21 @@ func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
return id, err return id, err
} }
// UpdateAccountCredentials replaces an account's logins and, when a new
// ciphertext is supplied, its passwords; a nil password keeps the stored one so
// the operator can fix only the side that failed. Both connection tests are
// reset to "unknown" because the previous verdicts no longer describe these
// credentials, which also forces a re-test before the account can run.
func (s *Store) UpdateAccountCredentials(ctx context.Context, id int64, srcLogin, dstLogin string, srcPassEnc, dstPassEnc *string) error {
_, err := s.Pool.Exec(ctx,
`UPDATE accounts SET src_login=$2, dst_login=$3,
src_pass_enc=COALESCE($4, src_pass_enc), dst_pass_enc=COALESCE($5, dst_pass_enc),
test_src_status='unknown', test_dst_status='unknown', last_error=''
WHERE id=$1`,
id, srcLogin, dstLogin, srcPassEnc, dstPassEnc)
return err
}
// DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE). // DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE).
func (s *Store) DeleteAccount(ctx context.Context, id int64) error { func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
_, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id) _, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id)
+41
View File
@@ -65,6 +65,47 @@ func TestResetAccountCounters(t *testing.T) {
} }
} }
// Fixing an imported account's credentials must replace only what the operator
// supplied: a nil password keeps the stored ciphertext, and both test verdicts
// go back to unknown because they described the old credentials.
func TestUpdateAccountCredentials(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "oldsrc", DstLogin: "u2", DstPassEnc: "olddst"})
_ = s.SetAccountTestStatus(ctx, accID, "src", "fail")
_ = s.SetAccountTestStatus(ctx, accID, "dst", "ok")
_ = s.SetAccountError(ctx, accID, "authentication failed")
newSrc := "newsrc"
if err := s.UpdateAccountCredentials(ctx, accID, "u@src.example", "u@dst.example", &newSrc, nil); err != nil {
t.Fatalf("update: %v", err)
}
accs, _ := s.ListAccountsByTask(ctx, taskID)
if len(accs) != 1 {
t.Fatalf("len=%d want 1", len(accs))
}
a := accs[0]
if a.SrcLogin != "u@src.example" || a.DstLogin != "u@dst.example" {
t.Fatalf("logins not updated: %q / %q", a.SrcLogin, a.DstLogin)
}
if a.SrcPassEnc != "newsrc" {
t.Fatalf("src password not updated: %q", a.SrcPassEnc)
}
if a.DstPassEnc != "olddst" {
t.Fatalf("nil password must keep the stored one, got %q", a.DstPassEnc)
}
if a.TestSrcStatus != "unknown" || a.TestDstStatus != "unknown" {
t.Fatalf("test statuses not reset: %q / %q", a.TestSrcStatus, a.TestDstStatus)
}
if a.LastError != "" {
t.Fatalf("last_error not cleared: %q", a.LastError)
}
}
func TestSetAccountFolderMapping(t *testing.T) { func TestSetAccountFolderMapping(t *testing.T) {
s := testStore(t) s := testStore(t)
ctx := context.Background() ctx := context.Background()
+36 -1
View File
@@ -18,6 +18,41 @@ func TestUpdateEndpoint(t *testing.T) {
} }
} }
func TestDeleteEndpoint(t *testing.T) {
s := testStore(t)
ctx := context.Background()
id, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a.com", Port: 993, TLSMode: "ssl"})
if err := s.DeleteEndpoint(ctx, id); err != nil {
t.Fatalf("delete: %v", err)
}
eps, _ := s.ListEndpoints(ctx)
if len(eps) != 0 {
t.Fatalf("endpoint not deleted: %d remain", len(eps))
}
}
func TestDeleteEndpointUsedByTaskRefused(t *testing.T) {
s := testStore(t)
ctx := context.Background()
ep1, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "s", Host: "a", Port: 993, TLSMode: "ssl"})
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
if _, err := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2}); err != nil {
t.Fatalf("create task: %v", err)
}
for _, id := range []int64{ep1, ep2} {
n, err := s.CountTasksUsingEndpoint(ctx, id)
if err != nil {
t.Fatalf("count: %v", err)
}
if n != 1 {
t.Fatalf("count for ep %d = %d, want 1", id, n)
}
if err := s.DeleteEndpoint(ctx, id); err == nil {
t.Fatalf("delete of referenced endpoint %d succeeded, want FK violation", id)
}
}
}
func TestDeleteAccountCascadesJournal(t *testing.T) { func TestDeleteAccountCascadesJournal(t *testing.T) {
s := testStore(t) s := testStore(t)
ctx := context.Background() ctx := context.Background()
@@ -49,7 +84,7 @@ func TestDeleteTaskCascades(t *testing.T) {
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"}) ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2}) taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2})
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "v", DstPassEnc: "y"}) accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "x", DstLogin: "v", DstPassEnc: "y"})
_, _ = s.CreateRun(ctx, taskID) _, _ = s.CreateRun(ctx, taskID, "manual")
if err := s.DeleteTask(ctx, taskID); err != nil { if err := s.DeleteTask(ctx, taskID); err != nil {
t.Fatalf("delete task: %v", err) t.Fatalf("delete task: %v", err)
+17
View File
@@ -26,6 +26,23 @@ func (s *Store) UpdateEndpoint(ctx context.Context, e Endpoint) error {
return err return err
} }
// CountTasksUsingEndpoint reports how many tasks reference the endpoint on
// either side, so a delete can be refused with a meaningful message instead of
// surfacing a raw foreign-key violation.
func (s *Store) CountTasksUsingEndpoint(ctx context.Context, id int64) (int, error) {
var n int
err := s.Pool.QueryRow(ctx,
`SELECT count(*) FROM tasks WHERE src_endpoint_id=$1 OR dst_endpoint_id=$1`, id).Scan(&n)
return n, err
}
// DeleteEndpoint removes an endpoint. Tasks reference endpoints without ON
// DELETE CASCADE, so Postgres rejects the delete while any task still uses it.
func (s *Store) DeleteEndpoint(ctx context.Context, id int64) error {
_, err := s.Pool.Exec(ctx, `DELETE FROM endpoints WHERE id=$1`, id)
return err
}
func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) { func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) {
var e Endpoint var e Endpoint
err := s.Pool.QueryRow(ctx, err := s.Pool.QueryRow(ctx,
+7
View File
@@ -6,6 +6,8 @@ import "context"
// process died mid-run (crash, container restart). A fresh process has no // process died mid-run (crash, container restart). A fresh process has no
// in-flight goroutines, so any persisted "running" is stale and would otherwise // in-flight goroutines, so any persisted "running" is stale and would otherwise
// wedge the task (the run-guard refuses to start, and accounts can't be edited). // wedge the task (the run-guard refuses to start, and accounts can't be edited).
// It also closes any run rows left stuck in "running" (finished_at NULL), which
// would otherwise surface as perpetually "running" in the run-log modal.
// Returns how many task and account rows were reset. // Returns how many task and account rows were reset.
func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, accounts int64, err error) { func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, accounts int64, err error) {
ct, err := s.Pool.Exec(ctx, `UPDATE accounts SET status='idle' WHERE status='running'`) ct, err := s.Pool.Exec(ctx, `UPDATE accounts SET status='idle' WHERE status='running'`)
@@ -18,6 +20,11 @@ func (s *Store) ResetRunningOnStartup(ctx context.Context) (tasks int64, account
return 0, accounts, err return 0, accounts, err
} }
tasks = ct.RowsAffected() tasks = ct.RowsAffected()
_, err = s.Pool.Exec(ctx,
`UPDATE runs SET status='error', finished_at=now() WHERE finished_at IS NULL`)
if err != nil {
return tasks, accounts, err
}
return tasks, accounts, nil return tasks, accounts, nil
} }
+14
View File
@@ -15,6 +15,7 @@ func TestResetRunningOnStartup(t *testing.T) {
// simulate a crash mid-run // simulate a crash mid-run
_ = s.SetTaskStatus(ctx, taskID, "running") _ = s.SetTaskStatus(ctx, taskID, "running")
_ = s.SetAccountStatus(ctx, accID, "running") _ = s.SetAccountStatus(ctx, accID, "running")
runID, _ := s.CreateRun(ctx, taskID, "manual") // phantom run: never finished
tn, an, err := s.ResetRunningOnStartup(ctx) tn, an, err := s.ResetRunningOnStartup(ctx)
if err != nil { if err != nil {
@@ -31,6 +32,19 @@ func TestResetRunningOnStartup(t *testing.T) {
if accs[0].Status == "running" { if accs[0].Status == "running" {
t.Fatal("account still running after reset") t.Fatal("account still running after reset")
} }
runs, _ := s.ListRunsByTask(ctx, taskID)
var found bool
for _, r := range runs {
if r.ID == runID {
found = true
if r.Status == "running" || r.FinishedAt == nil {
t.Fatalf("phantom run %d still running: status=%s finished_at=%v", runID, r.Status, r.FinishedAt)
}
}
}
if !found {
t.Fatalf("run %d not found", runID)
}
} }
func TestClearStuckAccountAndReconcile(t *testing.T) { func TestClearStuckAccountAndReconcile(t *testing.T) {
+48 -9
View File
@@ -1,20 +1,26 @@
package store package store
import "context" import (
"context"
"time"
)
type Run struct { type Run struct {
ID int64 ID int64 `json:"id"`
TaskID int64 TaskID int64 `json:"task_id"`
Status string Status string `json:"status"`
TotalCopied int64 StartedAt time.Time `json:"started_at"`
TotalSkipped int64 FinishedAt *time.Time `json:"finished_at"`
TotalErrors int64 TotalCopied int64 `json:"total_copied"`
TotalSkipped int64 `json:"total_skipped"`
TotalErrors int64 `json:"total_errors"`
Trigger string `json:"trigger"`
} }
func (s *Store) CreateRun(ctx context.Context, taskID int64) (int64, error) { func (s *Store) CreateRun(ctx context.Context, taskID int64, trigger string) (int64, error) {
var id int64 var id int64
err := s.Pool.QueryRow(ctx, err := s.Pool.QueryRow(ctx,
`INSERT INTO runs (task_id) VALUES ($1) RETURNING id`, taskID).Scan(&id) `INSERT INTO runs (task_id, trigger) VALUES ($1,$2) RETURNING id`, taskID, trigger).Scan(&id)
return id, err return id, err
} }
@@ -25,3 +31,36 @@ func (s *Store) FinishRun(ctx context.Context, id int64, status string, copied,
id, status, copied, skipped, errs) id, status, copied, skipped, errs)
return err return err
} }
// ListRunsByTask returns a task's runs, newest first, for the run-log modal.
func (s *Store) ListRunsByTask(ctx context.Context, taskID int64) ([]Run, error) {
rows, err := s.Pool.Query(ctx,
`SELECT id, task_id, started_at, finished_at, status,
total_copied, total_skipped, total_errors, trigger
FROM runs WHERE task_id=$1 ORDER BY id DESC`, taskID)
if err != nil {
return nil, err
}
defer rows.Close()
out := []Run{}
for rows.Next() {
var r Run
if err := rows.Scan(&r.ID, &r.TaskID, &r.StartedAt, &r.FinishedAt, &r.Status,
&r.TotalCopied, &r.TotalSkipped, &r.TotalErrors, &r.Trigger); err != nil {
return nil, err
}
out = append(out, r)
}
return out, rows.Err()
}
// LastFinishedRunAt returns the most recent finished run's timestamp, or nil if
// the task has never completed a run — the baseline for the next scheduled run.
// The same max(finished_at) rule is also inlined in ListSchedulableTasks's
// subquery — keep the two in sync if this changes.
func (s *Store) LastFinishedRunAt(ctx context.Context, taskID int64) (*time.Time, error) {
var t *time.Time
err := s.Pool.QueryRow(ctx,
`SELECT max(finished_at) FROM runs WHERE task_id=$1 AND finished_at IS NOT NULL`, taskID).Scan(&t)
return t, err
}
+81
View File
@@ -0,0 +1,81 @@
package store
import (
"context"
"testing"
)
func TestTaskScheduleAndBroken(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
// Defaults: no schedule, not broken.
tk, _ := s.GetTask(ctx, taskID)
if tk.ScheduleIntervalSeconds != 0 || tk.Broken || tk.ScheduleAnchor != nil {
t.Fatalf("defaults: interval=%d broken=%v anchor=%v", tk.ScheduleIntervalSeconds, tk.Broken, tk.ScheduleAnchor)
}
// Enable → interval set, anchor stamped, appears in schedulable list.
if err := s.SetTaskSchedule(ctx, taskID, 3600); err != nil {
t.Fatalf("set schedule: %v", err)
}
tk, _ = s.GetTask(ctx, taskID)
if tk.ScheduleIntervalSeconds != 3600 || tk.ScheduleAnchor == nil {
t.Fatalf("after enable: interval=%d anchor=%v", tk.ScheduleIntervalSeconds, tk.ScheduleAnchor)
}
sch, _ := s.ListSchedulableTasks(ctx)
if len(sch) != 1 || sch[0].ID != taskID || sch[0].IntervalSeconds != 3600 || sch[0].LastFinished != nil {
t.Fatalf("schedulable: %+v", sch)
}
// Breaker → broken true, interval 0, drops out of schedulable list.
if err := s.SetTaskBroken(ctx, taskID); err != nil {
t.Fatalf("set broken: %v", err)
}
tk, _ = s.GetTask(ctx, taskID)
if !tk.Broken || tk.ScheduleIntervalSeconds != 0 {
t.Fatalf("after break: broken=%v interval=%d", tk.Broken, tk.ScheduleIntervalSeconds)
}
if sch, _ := s.ListSchedulableTasks(ctx); len(sch) != 0 {
t.Fatalf("broken task still schedulable: %+v", sch)
}
// Re-enable clears broken.
_ = s.SetTaskSchedule(ctx, taskID, 21600)
if tk, _ = s.GetTask(ctx, taskID); tk.Broken {
t.Fatalf("re-enable did not clear broken")
}
}
func TestListRunsByTaskWithTrigger(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
r1, _ := s.CreateRun(ctx, taskID, "manual")
_ = s.FinishRun(ctx, r1, "done", 5, 1, 0)
_, _ = s.CreateRun(ctx, taskID, "scheduled") // still running (no finish)
runs, err := s.ListRunsByTask(ctx, taskID)
if err != nil || len(runs) != 2 {
t.Fatalf("list runs: %v len=%d", err, len(runs))
}
// newest first
if runs[0].Trigger != "scheduled" || runs[0].FinishedAt != nil {
t.Fatalf("run[0]: %+v", runs[0])
}
if runs[1].Trigger != "manual" || runs[1].FinishedAt == nil || runs[1].TotalCopied != 5 {
t.Fatalf("run[1]: %+v", runs[1])
}
// Last finished-at reflects the finished manual run.
lf, _ := s.LastFinishedRunAt(ctx, taskID)
if lf == nil {
t.Fatalf("LastFinishedRunAt nil, want the finished run's time")
}
}
+1 -1
View File
@@ -17,7 +17,7 @@ func testStore(t *testing.T) *Store {
} }
t.Cleanup(func() { t.Cleanup(func() {
s.Pool.Exec(context.Background(), s.Pool.Exec(context.Background(),
`TRUNCATE endpoints, tasks, accounts, runs, migrated_messages RESTART IDENTITY CASCADE`) `TRUNCATE endpoints, tasks, accounts, runs, migrated_messages, account_errors RESTART IDENTITY CASCADE`)
s.Pool.Close() s.Pool.Close()
}) })
return s return s
+75 -11
View File
@@ -1,14 +1,20 @@
package store package store
import "context" import (
"context"
"time"
)
type Task struct { type Task struct {
ID int64 `json:"id"` ID int64 `json:"id"`
Name string `json:"name"` Name string `json:"name"`
SrcEndpointID int64 `json:"src_endpoint_id"` SrcEndpointID int64 `json:"src_endpoint_id"`
DstEndpointID int64 `json:"dst_endpoint_id"` DstEndpointID int64 `json:"dst_endpoint_id"`
Status string `json:"status"` Status string `json:"status"`
FolderMapping map[string]string `json:"folder_mapping"` FolderMapping map[string]string `json:"folder_mapping"`
ScheduleIntervalSeconds int64 `json:"schedule_interval_seconds"`
ScheduleAnchor *time.Time `json:"-"`
Broken bool `json:"broken"`
} }
func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) { func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) {
@@ -26,9 +32,11 @@ func (s *Store) CreateTask(ctx context.Context, t Task) (int64, error) {
func (s *Store) GetTask(ctx context.Context, id int64) (Task, error) { func (s *Store) GetTask(ctx context.Context, id int64) (Task, error) {
var t Task var t Task
err := s.Pool.QueryRow(ctx, err := s.Pool.QueryRow(ctx,
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping `SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping,
schedule_interval_seconds, schedule_anchor, broken
FROM tasks WHERE id=$1`, id). FROM tasks WHERE id=$1`, id).
Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping) Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping,
&t.ScheduleIntervalSeconds, &t.ScheduleAnchor, &t.Broken)
return t, err return t, err
} }
@@ -40,7 +48,8 @@ func (s *Store) DeleteTask(ctx context.Context, id int64) error {
func (s *Store) ListTasks(ctx context.Context) ([]Task, error) { func (s *Store) ListTasks(ctx context.Context) ([]Task, error) {
rows, err := s.Pool.Query(ctx, rows, err := s.Pool.Query(ctx,
`SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping `SELECT id, name, src_endpoint_id, dst_endpoint_id, status, folder_mapping,
schedule_interval_seconds, schedule_anchor, broken
FROM tasks ORDER BY id DESC`) FROM tasks ORDER BY id DESC`)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -49,7 +58,8 @@ func (s *Store) ListTasks(ctx context.Context) ([]Task, error) {
out := []Task{} out := []Task{}
for rows.Next() { for rows.Next() {
var t Task var t Task
if err := rows.Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping); err != nil { if err := rows.Scan(&t.ID, &t.Name, &t.SrcEndpointID, &t.DstEndpointID, &t.Status, &t.FolderMapping,
&t.ScheduleIntervalSeconds, &t.ScheduleAnchor, &t.Broken); err != nil {
return nil, err return nil, err
} }
out = append(out, t) out = append(out, t)
@@ -80,3 +90,57 @@ func (s *Store) TryMarkTaskRunning(ctx context.Context, id int64) (bool, error)
} }
return ct.RowsAffected() == 1, nil return ct.RowsAffected() == 1, nil
} }
// SetTaskSchedule sets the recurrence interval (0 = off), stamps the anchor when
// enabling (the first run is one interval after this), and clears broken — any
// explicit schedule change un-breaks the task.
func (s *Store) SetTaskSchedule(ctx context.Context, id, intervalSeconds int64) error {
_, err := s.Pool.Exec(ctx,
`UPDATE tasks SET schedule_interval_seconds=$2,
schedule_anchor = CASE WHEN $2 > 0 THEN now() ELSE schedule_anchor END,
broken = false
WHERE id=$1`, id, intervalSeconds)
return err
}
// SetTaskBroken trips the schedule breaker: disables the schedule and flags the
// task for operator attention.
func (s *Store) SetTaskBroken(ctx context.Context, id int64) error {
_, err := s.Pool.Exec(ctx,
`UPDATE tasks SET broken=true, schedule_interval_seconds=0 WHERE id=$1`, id)
return err
}
// SchedulableTask is the per-tick decision input for the scheduler.
type SchedulableTask struct {
ID int64
IntervalSeconds int64
Anchor time.Time
LastFinished *time.Time
}
// ListSchedulableTasks returns tasks eligible to auto-run: schedule on, not
// broken, neither running nor paused — each joined with its last finished run
// time. A paused task waits for the operator to resume it; auto-starting a full
// run behind their back would defeat the pause.
func (s *Store) ListSchedulableTasks(ctx context.Context) ([]SchedulableTask, error) {
rows, err := s.Pool.Query(ctx,
`SELECT t.id, t.schedule_interval_seconds, t.schedule_anchor,
(SELECT max(finished_at) FROM runs r WHERE r.task_id=t.id AND r.finished_at IS NOT NULL)
FROM tasks t
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken
AND t.status <> 'running' AND t.status <> 'paused'`)
if err != nil {
return nil, err
}
defer rows.Close()
out := []SchedulableTask{}
for rows.Next() {
var st SchedulableTask
if err := rows.Scan(&st.ID, &st.IntervalSeconds, &st.Anchor, &st.LastFinished); err != nil {
return nil, err
}
out = append(out, st)
}
return out, rows.Err()
}
+4
View File
@@ -0,0 +1,4 @@
ALTER TABLE runs DROP COLUMN trigger;
ALTER TABLE tasks DROP COLUMN broken;
ALTER TABLE tasks DROP COLUMN schedule_anchor;
ALTER TABLE tasks DROP COLUMN schedule_interval_seconds;
+4
View File
@@ -0,0 +1,4 @@
ALTER TABLE tasks ADD COLUMN schedule_interval_seconds INT NOT NULL DEFAULT 0;
ALTER TABLE tasks ADD COLUMN schedule_anchor TIMESTAMPTZ;
ALTER TABLE tasks ADD COLUMN broken BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE runs ADD COLUMN trigger TEXT NOT NULL DEFAULT 'manual';
+1
View File
@@ -0,0 +1 @@
DROP TABLE account_errors;
+12
View File
@@ -0,0 +1,12 @@
CREATE TABLE account_errors (
id BIGSERIAL PRIMARY KEY,
account_id BIGINT NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
run_id BIGINT,
kind TEXT NOT NULL CHECK (kind IN ('folder','message','account')),
folder TEXT NOT NULL DEFAULT '',
message_ref TEXT NOT NULL DEFAULT '',
error TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX account_errors_account_id_idx ON account_errors (account_id);
+94 -9
View File
@@ -158,7 +158,8 @@ wait_test_ok() {
wait_test_ok wait_test_ok
wait_run_done() { wait_run_done() {
for ((i = 1; i <= 60; i++)); do # Generous: the resume scenario re-scans and copies thousands of messages.
for ((i = 1; i <= 600; i++)); do
local status local status
status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status') status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
if [[ "$status" == "done" ]]; then if [[ "$status" == "done" ]]; then
@@ -185,16 +186,100 @@ log "POST /run (second run, expect idempotency)"
api POST "/api/tasks/${TASK_ID}/run" >/dev/null api POST "/api/tasks/${TASK_ID}/run" >/dev/null
wait_run_done wait_run_done
# Counters are reset at the start of every run, so run 2's row shows run 2
# alone: it must copy nothing and skip what run 1 already migrated.
RES2=$(api GET "/api/tasks/${TASK_ID}") RES2=$(api GET "/api/tasks/${TASK_ID}")
RUN2_COPIED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].copied') RUN2_COPIED=$(echo "$RES2" | jq -r '.accounts[0].copied')
RUN2_SKIPPED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].skipped') RUN2_SKIPPED=$(echo "$RES2" | jq -r '.accounts[0].skipped')
RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors') RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors')
RUN2_COPIED_DELTA=$((RUN2_COPIED_TOTAL - RUN1_COPIED)) log "run 2: copied=$RUN2_COPIED skipped=$RUN2_SKIPPED errors=$RUN2_ERRORS"
RUN2_SKIPPED_DELTA=$((RUN2_SKIPPED_TOTAL - RUN1_SKIPPED))
log "run 2: copied_delta=$RUN2_COPIED_DELTA skipped_delta=$RUN2_SKIPPED_DELTA errors=$RUN2_ERRORS"
[[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors" [[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors"
[[ "$RUN2_COPIED_DELTA" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED_DELTA new messages (expected 0, not idempotent)" [[ "$RUN2_COPIED" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED new messages (expected 0, not idempotent)"
[[ "$RUN2_SKIPPED_DELTA" -gt 0 ]] || fail "run 2 skipped delta is $RUN2_SKIPPED_DELTA (expected >0)" [[ "$RUN2_SKIPPED" -eq "$RUN1_COPIED" ]] ||
fail "run 2 skipped $RUN2_SKIPPED of the $RUN1_COPIED messages run 1 copied"
log "PASS: run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED_DELTA skipped=$RUN2_SKIPPED_DELTA (idempotent)" log "run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED skipped=$RUN2_SKIPPED (idempotent)"
# ---------------------------------------------------------------------------
# Pause / resume: a second account with enough messages that the run is still
# in flight when the pause lands. Pausing must leave the account resumable, and
# resuming must finish it without re-copying what the first stretch already did.
# ---------------------------------------------------------------------------
SRC_USER2="src2@example.com"
DST_USER2="dst2@example.com"
# Large enough that the copy is still in flight when the pause lands — greenmail
# on a local socket copies well over a thousand small messages per second.
SEED_COUNT=3000
log "seeding ${SEED_COUNT} messages into ${SRC_USER2} INBOX (pause/resume scenario)"
python3 "$SEED_PY" 127.0.0.1 3143 "$SRC_USER2" "$MAIL_PASS" "$SEED_COUNT"
log "adding second account (src2 -> dst2)"
ACCOUNT2_ID=$(api POST "/api/tasks/${TASK_ID}/accounts" \
"{\"src_login\":\"${SRC_USER2}\",\"src_pass\":\"${MAIL_PASS}\",\"dst_login\":\"${DST_USER2}\",\"dst_pass\":\"${MAIL_PASS}\"}" | jq -r .id)
[[ "$ACCOUNT2_ID" =~ ^[0-9]+$ ]] || fail "bad second account id: $ACCOUNT2_ID"
log "account2_id=$ACCOUNT2_ID"
log "POST /test (both accounts)"
api POST "/api/tasks/${TASK_ID}/test" >/dev/null
for ((i = 1; i <= 30; i++)); do
BOTH_OK=$(api GET "/api/tasks/${TASK_ID}" |
jq -r '[.accounts[] | select(.test_src_status=="ok" and .test_dst_status=="ok")] | length')
[[ "$BOTH_OK" == "2" ]] && break
sleep 1
done
[[ "$BOTH_OK" == "2" ]] || fail "second account did not pass connection tests (ok count=$BOTH_OK)"
# Account view for account2, by id.
acct2() { api GET "/api/tasks/${TASK_ID}" | jq -r ".accounts[] | select(.id==${ACCOUNT2_ID}) | $1"; }
log "POST /run (account2 only)"
api POST "/api/tasks/${TASK_ID}/run" "{\"account_ids\":[${ACCOUNT2_ID}]}" >/dev/null
# Per-account counters are only written to the DB when a folder completes, so
# "copied so far" is invisible here — wait for the account to go running, give
# the copy a few seconds of real work, then pause mid-folder.
log "waiting for account2 to start running"
for ((i = 1; i <= 120; i++)); do
[[ "$(acct2 .status)" == "running" ]] && break
sleep 0.5
done
[[ "$(acct2 .status)" == "running" ]] || fail "account2 never reached running"
log "letting it copy for a few seconds, then pausing mid-folder"
sleep 5
curl -fsS -b "$COOKIE_JAR" -c "$COOKIE_JAR" -X POST "$BASE/api/tasks/${TASK_ID}/pause" >/dev/null ||
fail "pause rejected — the run finished before the pause landed, seed more messages"
log "waiting for the task to settle into paused"
for ((i = 1; i <= 60; i++)); do
TASK_STATUS=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
[[ "$TASK_STATUS" == "paused" ]] && break
sleep 1
done
[[ "$TASK_STATUS" == "paused" ]] || fail "task status=$TASK_STATUS after pause (expected paused)"
ACC2_STATUS=$(acct2 .status)
[[ "$ACC2_STATUS" == "paused" ]] || fail "account2 status=$ACC2_STATUS after pause (expected paused)"
log "paused (folder-level counters at copied=$(acct2 .copied) of $SEED_COUNT)"
log "POST /resume"
api POST "/api/tasks/${TASK_ID}/resume" >/dev/null
wait_run_done
RESUMED_COPIED=$(acct2 .copied)
RESUMED_SKIPPED=$(acct2 .skipped)
RESUMED_ERRORS=$(acct2 .errors)
RESUMED_TOTAL=$((RESUMED_COPIED + RESUMED_SKIPPED))
log "after resume: copied=$RESUMED_COPIED skipped=$RESUMED_SKIPPED errors=$RESUMED_ERRORS"
[[ "$RESUMED_ERRORS" == "0" ]] || fail "resumed run had errors"
# Counters reset per run, so the resumed run alone must account for every
# message: the ones it copied now plus the ones the paused stretch already did.
[[ "$RESUMED_TOTAL" -eq "$SEED_COUNT" ]] || fail "resumed run covered $RESUMED_TOTAL of $SEED_COUNT messages"
# Non-zero skipped is the proof that the paused stretch's work survived: those
# messages are in the migration journal, so the resume did not re-copy them.
[[ "$RESUMED_SKIPPED" -gt 0 ]] ||
fail "resumed run skipped nothing — the paused stretch's progress was lost"
log "PASS: idempotent re-run; pause was resumable, resume re-copied $RESUMED_COPIED and skipped $RESUMED_SKIPPED of $SEED_COUNT"
+64 -1
View File
@@ -18,6 +18,9 @@ export interface Task {
dst_endpoint_id: number dst_endpoint_id: number
status: string status: string
folder_mapping?: Record<string, string> folder_mapping?: Record<string, string>
schedule_interval_seconds?: number
broken?: boolean
next_run_at?: string | null
} }
export type TestStatus = 'pending' | 'ok' | 'fail' | string export type TestStatus = 'pending' | 'ok' | 'fail' | string
@@ -79,11 +82,21 @@ export const updateEndpoint = (
body: { role_label: string; host: string; port: number; tls_mode: TLSMode }, body: { role_label: string; host: string; port: number; tls_mode: TLSMode },
) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' }) ) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' })
export const deleteEndpoint = (id: number) => api(`/api/endpoints/${id}`, { method: 'DELETE' })
export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' }) export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' })
export const deleteAccount = (taskId: number, accountId: number) => export const deleteAccount = (taskId: number, accountId: number) =>
api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' }) api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' })
// Empty password fields keep the stored ones; both connection tests reset to
// unknown server-side, so the account must be re-tested afterwards.
export const updateAccountCredentials = (
taskId: number,
accountId: number,
body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
) => api(`/api/tasks/${taskId}/accounts/${accountId}/credentials`, { ...jsonBody(body), method: 'PUT' })
export const cancelAccount = (taskId: number, accountId: number) => export const cancelAccount = (taskId: number, accountId: number) =>
api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' }) api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' })
@@ -135,10 +148,60 @@ export const createAccount = (
export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' }) export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' })
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' }) export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
// Pause stops the run but leaves its unfinished accounts resumable; cancel ends
// it and marks them cancelled. Resume re-runs exactly the paused accounts.
export const pauseTask = (id: number) => api(`/api/tasks/${id}/pause`, { method: 'POST' })
export const cancelTask = (id: number) => api(`/api/tasks/${id}/cancel`, { method: 'POST' })
export const resumeTask = (id: number) => api<{ run_id: number }>(`/api/tasks/${id}/resume`, { method: 'POST' })
export interface Run {
id: number
task_id: number
status: string
started_at: string
finished_at: string | null
total_copied: number
total_skipped: number
total_errors: number
trigger: string
}
export const setTaskSchedule = (taskId: number, intervalSeconds: number) =>
api(`/api/tasks/${taskId}/schedule`, { ...jsonBody({ interval_seconds: intervalSeconds }), method: 'PUT' })
export const listRuns = (taskId: number) => api<Run[]>(`/api/tasks/${taskId}/runs`)
export interface AccountError {
id: number
account_id: number
run_id: number
kind: string // folder | message | account
folder: string
message_ref: string
error: string
created_at: string
}
export const listAccountErrors = (taskId: number, accountId: number) =>
api<AccountError[]>(`/api/tasks/${taskId}/accounts/${accountId}/errors`)
export const importCSV = (id: number, file: File) => { export const importCSV = (id: number, file: File) => {
const fd = new FormData() const fd = new FormData()
fd.append('file', file) fd.append('file', file)
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd }) return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
} }
// A Kerio Connect export carries no domain, so the operator supplies it here;
// the login and password apply to both sides of the migration.
export const importKerioCSV = (id: number, file: File, domain: string) => {
const fd = new FormData()
fd.append('file', file)
fd.append('format', 'kerio')
fd.append('domain', domain)
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
}
+200 -2
View File
@@ -255,11 +255,36 @@
} }
/* per-account live progress */ /* per-account live progress */
/* merged Source/Destination "Account" cell: single line when src == dst,
stacked (src over dst) when they differ. */
.acct-ident {
display: flex;
flex-direction: column;
gap: 2px;
}
.acct-dst {
color: var(--fg-dim);
font-size: 12px;
}
.acct-progress { .acct-progress {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
gap: 4px; gap: 4px;
min-width: 160px; min-width: 160px;
/* reserve room for the bar + two meta lines so toggling the scan line
during a run doesn't change the row height */
min-height: 40px;
}
/* Running rows get a fixed wide size up front so live progress updates
(which change text length and can wrap) never resize the row mid-scan.
The size reverts only when the run finishes and the class drops. */
.progress-cell--live {
min-width: 340px;
height: 48px;
vertical-align: top;
} }
.pbar { .pbar {
@@ -283,6 +308,8 @@
font-size: 10px; font-size: 10px;
color: var(--fg-dim); color: var(--fg-dim);
letter-spacing: 0.04em; letter-spacing: 0.04em;
/* keep progress text on one line so its length never reflows the row */
white-space: nowrap;
} }
.pscan { .pscan {
@@ -300,6 +327,77 @@
white-space: nowrap; white-space: nowrap;
} }
/* per-account errors modal: a paginated list so long messages wrap on the
full modal width instead of squeezing into narrow table columns. */
.err-empty {
padding: 24px 8px;
text-align: center;
color: var(--fg-dim);
}
.err-list {
list-style: none;
margin: 0;
padding: 0;
max-height: 60vh;
overflow-y: auto;
}
.err-item {
padding: 10px 4px;
border-bottom: 1px solid var(--border);
}
.err-item-head {
display: flex;
flex-wrap: wrap;
align-items: baseline;
gap: 8px;
font-size: 11px;
letter-spacing: 0.04em;
margin-bottom: 4px;
}
.err-kind {
color: var(--accent);
text-transform: uppercase;
}
.err-folder {
color: var(--fg);
}
.err-ref {
color: var(--fg-dim);
word-break: break-word;
}
.err-time {
margin-left: auto;
color: var(--fg-dim);
}
.err-text {
font-size: 12px;
line-height: 1.45;
color: var(--fail);
white-space: normal;
word-break: break-word;
}
.err-pager {
display: flex;
align-items: center;
justify-content: center;
gap: 14px;
padding-top: 12px;
}
.err-pageinfo {
font-size: 12px;
color: var(--fg-dim);
}
/* clear-log button: mirrors the .panel-label tab on the right edge */ /* clear-log button: mirrors the .panel-label tab on the right edge */
.log-clear { .log-clear {
position: absolute; position: absolute;
@@ -390,11 +488,23 @@
cursor: pointer; cursor: pointer;
} }
.map-toolbar {
display: flex;
align-items: center;
gap: 14px;
margin-bottom: 12px;
flex-wrap: wrap;
}
.map-toolbar .btn {
padding: 6px 12px;
font-size: 11px;
}
.map-all { .map-all {
display: flex; display: flex;
align-items: center; align-items: center;
gap: 8px; gap: 8px;
margin-bottom: 12px;
font-size: 11px; font-size: 11px;
text-transform: uppercase; text-transform: uppercase;
letter-spacing: 0.08em; letter-spacing: 0.08em;
@@ -557,6 +667,25 @@ table.tbl a.rowlink:focus-visible {
/* ---------- status badges ---------- */ /* ---------- status badges ---------- */
/* A badge that opens a dialog: the badge keeps its own look, the button only
contributes the affordance. */
.badge-btn {
padding: 0;
border: 0;
background: none;
font: inherit;
cursor: pointer;
}
.badge-btn:hover .badge {
filter: brightness(1.25);
}
.badge-btn:focus-visible {
outline: 1px solid var(--accent);
outline-offset: 2px;
}
.badge { .badge {
display: inline-flex; display: inline-flex;
align-items: center; align-items: center;
@@ -605,6 +734,39 @@ table.tbl a.rowlink:focus-visible {
} }
.badge-info .dot { background: var(--info); animation: pulse 1.4s ease-in-out infinite; } .badge-info .dot { background: var(--info); animation: pulse 1.4s ease-in-out infinite; }
/* ---------- schedule row ---------- */
.sched-row {
display: flex;
align-items: center;
gap: 12px;
margin-top: 14px;
flex-wrap: wrap;
}
.sched-row label {
font-size: 11px;
font-weight: 700;
letter-spacing: 0.1em;
text-transform: uppercase;
color: var(--fg-dim);
}
.sched-row select {
background: var(--bg-inset);
border: 1px solid var(--border);
color: var(--fg);
padding: 7px 10px;
font-size: 13px;
border-radius: 2px;
}
.sched-next {
font-size: 12px;
color: var(--fg-dim);
font-variant-numeric: tabular-nums;
}
/* ---------- tables ---------- */ /* ---------- tables ---------- */
.tbl-wrap { .tbl-wrap {
@@ -666,6 +828,16 @@ table.tbl a.rowlink:hover {
justify-content: flex-end; justify-content: flex-end;
} }
/* Marks accounts whose folders have already been mapped/excluded. */
.map-dot {
width: 8px;
height: 8px;
border-radius: 50%;
flex: none;
background: var(--ok);
box-shadow: 0 0 5px var(--ok);
}
.empty-row td { .empty-row td {
text-align: center; text-align: center;
color: var(--fg-faint); color: var(--fg-faint);
@@ -764,11 +936,24 @@ table.tbl a.rowlink:hover {
.upload-row { .upload-row {
display: flex; display: flex;
align-items: center; align-items: flex-start;
gap: 12px; gap: 12px;
flex-wrap: wrap; flex-wrap: wrap;
} }
/* One import route per column: the action on top, its sample file underneath. */
.upload-item {
display: flex;
flex-direction: column;
align-items: stretch;
gap: 6px;
}
.upload-item .link-btn {
align-self: center;
text-align: center;
}
.file-btn { .file-btn {
position: relative; position: relative;
overflow: hidden; overflow: hidden;
@@ -915,3 +1100,16 @@ table.tbl a.rowlink:hover {
padding: 12px 14px; padding: 12px 14px;
} }
} }
.tbl .chk-col {
width: 32px;
text-align: center;
padding-right: 0;
}
.tbl .chk-col input[type='checkbox'] {
cursor: pointer;
}
.tbl .chk-col input[type='checkbox']:disabled {
cursor: not-allowed;
opacity: 0.4;
}
@@ -0,0 +1,116 @@
import { useEffect, useState, type FormEvent } from 'react'
import { Modal } from './Modal'
import type { Account } from '../api'
type Props = {
open: boolean
busy: boolean
account: Account | null
onClose: () => void
onSubmit: (body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) => void
}
// Fixes the credentials of an account that failed its connection test — usually
// a wrong password that came in through a CSV import. Passwords are never sent
// back to the browser, so the fields start empty and an empty field means
// "keep the stored password".
export function AccountCredentialsModal({ open, busy, account, onClose, onSubmit }: Props) {
const [srcLogin, setSrcLogin] = useState('')
const [dstLogin, setDstLogin] = useState('')
const [srcPass, setSrcPass] = useState('')
const [dstPass, setDstPass] = useState('')
const [error, setError] = useState<string | null>(null)
useEffect(() => {
if (!open || !account) return
setSrcLogin(account.src_login)
setDstLogin(account.dst_login)
setSrcPass('')
setDstPass('')
setError(null)
}, [open, account])
function submit(e: FormEvent) {
e.preventDefault()
if (srcLogin.trim() === '' || dstLogin.trim() === '') {
setError('Both logins are required')
return
}
setError(null)
onSubmit({
src_login: srcLogin.trim(),
src_pass: srcPass,
dst_login: dstLogin.trim(),
dst_pass: dstPass,
})
}
return (
<Modal open={open} title={account ? `Edit credentials — ${account.src_login}` : 'Edit credentials'} onClose={onClose}>
<form onSubmit={submit}>
<p className="map-hint">
Leave a password field empty to keep the stored one. Saving resets both connection tests, so re-run{' '}
<strong>Test connections</strong> afterwards.
</p>
<div className="field-row">
<div className="field">
<label htmlFor="edit_src_login">Source login</label>
<input
id="edit_src_login"
data-modal-autofocus
value={srcLogin}
onChange={(e) => setSrcLogin(e.target.value)}
disabled={busy}
required
/>
</div>
<div className="field">
<label htmlFor="edit_src_pass">Source password</label>
<input
id="edit_src_pass"
type="password"
value={srcPass}
onChange={(e) => setSrcPass(e.target.value)}
placeholder="unchanged"
autoComplete="new-password"
disabled={busy}
/>
</div>
</div>
<div className="field-row">
<div className="field">
<label htmlFor="edit_dst_login">Destination login</label>
<input
id="edit_dst_login"
value={dstLogin}
onChange={(e) => setDstLogin(e.target.value)}
disabled={busy}
required
/>
</div>
<div className="field">
<label htmlFor="edit_dst_pass">Destination password</label>
<input
id="edit_dst_pass"
type="password"
value={dstPass}
onChange={(e) => setDstPass(e.target.value)}
placeholder="unchanged"
autoComplete="new-password"
disabled={busy}
/>
</div>
</div>
{error && <div className="error-banner">{error}</div>}
<div className="modal-actions">
<button type="button" className="btn" onClick={onClose} disabled={busy}>
Cancel
</button>
<button className="btn btn-primary" disabled={busy}>
{busy ? 'Saving…' : 'Save credentials'}
</button>
</div>
</form>
</Modal>
)
}
+91
View File
@@ -0,0 +1,91 @@
import { useEffect, useState } from 'react'
import { Modal } from './Modal'
import { listAccountErrors, type AccountError } from '../api'
const fmt = (iso: string) => (iso ? new Date(iso).toLocaleString() : '—')
const PAGE_SIZE = 50
export function AccountErrorsModal({
taskId,
account,
onClose,
}: {
taskId: number
account: { id: number; src_login: string } | null
onClose: () => void
}) {
const [errors, setErrors] = useState<AccountError[] | null>(null)
const [failed, setFailed] = useState(false)
const [page, setPage] = useState(0)
const open = account !== null
useEffect(() => {
if (!account) return
setErrors(null)
setFailed(false)
setPage(0)
listAccountErrors(taskId, account.id)
.then((e) => setErrors(e ?? []))
.catch(() => setFailed(true))
}, [taskId, account])
const total = errors?.length ?? 0
const pageCount = Math.max(1, Math.ceil(total / PAGE_SIZE))
const current = Math.min(page, pageCount - 1)
const slice = errors?.slice(current * PAGE_SIZE, current * PAGE_SIZE + PAGE_SIZE) ?? []
const title = account
? `Errors — ${account.src_login}${total ? ` (${total})` : ''}`
: 'Errors'
return (
<Modal open={open} title={title} onClose={onClose} size="lg">
{failed ? (
<div className="err-empty">failed to load errors</div>
) : errors === null ? (
<div className="err-empty">loading</div>
) : total === 0 ? (
<div className="err-empty">no errors recorded</div>
) : (
<>
<ul className="err-list">
{slice.map((e) => (
<li key={e.id} className="err-item">
<div className="err-item-head">
<span className="err-kind">{e.kind}</span>
{e.folder && <span className="err-folder">{e.folder}</span>}
{e.message_ref && <span className="err-ref">{e.message_ref}</span>}
<span className="err-time">{fmt(e.created_at)}</span>
</div>
<div className="err-text">{e.error}</div>
</li>
))}
</ul>
{pageCount > 1 && (
<div className="err-pager">
<button
type="button"
className="btn"
disabled={current === 0}
onClick={() => setPage(current - 1)}
>
prev
</button>
<span className="err-pageinfo">
page {current + 1} / {pageCount}
</span>
<button
type="button"
className="btn"
disabled={current >= pageCount - 1}
onClick={() => setPage(current + 1)}
>
next
</button>
</div>
)}
</>
)}
</Modal>
)
}
+73 -18
View File
@@ -7,6 +7,7 @@ type Props = {
dstFolders: string[] dstFolders: string[]
initialMapping: Record<string, string> initialMapping: Record<string, string>
initialExcluded: string[] initialExcluded: string[]
accountLabel?: string
onConfirm: (mapping: Record<string, string>, excluded: string[]) => void onConfirm: (mapping: Record<string, string>, excluded: string[]) => void
onCancel: () => void onCancel: () => void
} }
@@ -19,8 +20,23 @@ function defaultDst(src: string, dstFolders: string[], initial: Record<string, s
return src return src
} }
// Exchange/Kerio special folders and their mailcow counterparts. Keyed by the
// lowercased source name so casing differences between servers don't matter.
const DEFAULT_TARGETS: Record<string, string> = {
'deleted items': 'Trash',
'deleted messages': 'Trash',
'junk e-mail': 'Junk',
'junk email': 'Junk',
spam: 'Junk',
'sent items': 'Sent',
'sent messages': 'Sent',
}
// Source folders with no counterpart on mailcow — unchecked by the defaults.
const DEFAULT_EXCLUDED = new Set(['public folders'])
export function FolderMappingModal({ export function FolderMappingModal({
open, srcFolders, dstFolders, initialMapping, initialExcluded, onConfirm, onCancel, open, srcFolders, dstFolders, initialMapping, initialExcluded, accountLabel, onConfirm, onCancel,
}: Props) { }: Props) {
const [choice, setChoice] = useState<Record<string, string>>({}) const [choice, setChoice] = useState<Record<string, string>>({})
const [synced, setSynced] = useState<Record<string, boolean>>(() => { const [synced, setSynced] = useState<Record<string, boolean>>(() => {
@@ -30,16 +46,40 @@ export function FolderMappingModal({
// Options per select: all destination folders, plus the source name itself // Options per select: all destination folders, plus the source name itself
// (marked "create") when it does not already exist on the destination. // (marked "create") when it does not already exist on the destination.
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
// Options per select: all destination folders, plus any name not present there
// — the source folder itself and the current selection — marked "create".
const options = useMemo(() => { const options = useMemo(() => {
const set = new Set(dstFolders) const set = new Set(dstFolders)
return (src: string) => { return (src: string, current: string) => {
const opts = [...dstFolders] const opts = [...dstFolders]
if (!set.has(src)) opts.unshift(src) if (!set.has(current)) opts.unshift(current)
if (!set.has(src) && src !== current) opts.unshift(src)
return opts return opts
} }
}, [dstFolders]) }, [dstFolders])
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping) // Collapse the Exchange/Kerio folder layout onto mailcow's in one click: the
// per-account mapping is otherwise repetitive work when importing many users.
function applyDefaults() {
const nextChoice = { ...choice }
const nextSynced = { ...synced }
for (const src of srcFolders) {
const key = src.trim().toLowerCase()
if (DEFAULT_EXCLUDED.has(key)) {
nextSynced[src] = false
continue
}
const target = DEFAULT_TARGETS[key]
if (target) {
nextChoice[src] = target
nextSynced[src] = true
}
}
setChoice(nextChoice)
setSynced(nextSynced)
}
function confirm() { function confirm() {
const mapping: Record<string, string> = {} const mapping: Record<string, string> = {}
@@ -58,23 +98,38 @@ export function FolderMappingModal({
} }
return ( return (
<Modal open={open} title="Map folders (source → destination)" onClose={onCancel} size="lg"> <Modal
open={open}
title={accountLabel ? `Map folders — ${accountLabel}` : 'Map folders (source → destination)'}
onClose={onCancel}
size="lg"
>
<div className="map-body"> <div className="map-body">
<p className="map-hint"> <p className="map-hint">
Route each source folder to an existing destination folder. Leaving a folder mapped to its own name Route each source folder to an existing destination folder. Leaving a folder mapped to its own name
creates it on the destination if missing (e.g. map <code>Спам</code> <code>Spam</code> to avoid duplicates). creates it on the destination if missing (e.g. map <code>Спам</code> <code>Spam</code> to avoid duplicates).
</p> </p>
<label className="map-all"> <div className="map-toolbar">
<input <button
type="checkbox" type="button"
checked={srcFolders.every((f) => synced[f] !== false)} className="btn btn-ghost"
onChange={(e) => { onClick={applyDefaults}
const on = e.target.checked title="Map Deleted Items → Trash, Junk E-mail → Junk, Sent Items → Sent and skip Public Folders"
setSynced(Object.fromEntries(srcFolders.map((f) => [f, on]))) >
}} By default
/> </button>
sync all folders <label className="map-all">
</label> <input
type="checkbox"
checked={srcFolders.every((f) => synced[f] !== false)}
onChange={(e) => {
const on = e.target.checked
setSynced(Object.fromEntries(srcFolders.map((f) => [f, on])))
}}
/>
sync all folders
</label>
</div>
<div className="map-grid"> <div className="map-grid">
{srcFolders.map((src) => { {srcFolders.map((src) => {
const on = synced[src] !== false const on = synced[src] !== false
@@ -101,10 +156,10 @@ export function FolderMappingModal({
disabled={!on} disabled={!on}
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))} onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
> >
{options(src).map((f) => ( {options(src, val).map((f) => (
<option key={f} value={f}> <option key={f} value={f}>
{f} {f}
{f === src && !dstFolders.includes(src) ? ' (create)' : ''} {dstFolders.includes(f) ? '' : ' (create)'}
</option> </option>
))} ))}
</select> </select>
+82
View File
@@ -0,0 +1,82 @@
import { useEffect, useState, type FormEvent } from 'react'
import { Modal } from './Modal'
// A bare DNS domain: no scheme, no user part, no whitespace. Mirrors the
// server-side check in csvimport.normalizeDomain so bad input is caught here.
const domainRe = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/
type Props = {
open: boolean
busy: boolean
onClose: () => void
onSubmit: (file: File, domain: string) => void
}
export function KerioImportModal({ open, busy, onClose, onSubmit }: Props) {
const [domain, setDomain] = useState('')
const [file, setFile] = useState<File | null>(null)
const [error, setError] = useState<string | null>(null)
useEffect(() => {
if (!open) return
setDomain('')
setFile(null)
setError(null)
}, [open])
function submit(e: FormEvent) {
e.preventDefault()
const d = domain.trim().toLowerCase()
if (!domainRe.test(d)) {
setError('Enter a bare domain, e.g. galaxyhotel.kz')
return
}
if (!file) {
setError('Choose the Kerio export file')
return
}
setError(null)
onSubmit(file, d)
}
return (
<Modal open={open} title="Import from Kerio" onClose={onClose}>
<form onSubmit={submit}>
<p className="map-hint">
The Kerio user export lists a login and its password but no domain. The domain you enter is appended to every
login and used for both the source and the destination. Disabled accounts and <code>admin</code> are skipped.
</p>
<div className="field">
<label htmlFor="kerio_domain">Mail domain</label>
<input
id="kerio_domain"
data-modal-autofocus
value={domain}
onChange={(e) => setDomain(e.target.value)}
placeholder="galaxyhotel.kz"
disabled={busy}
/>
</div>
<div className="field">
<label htmlFor="kerio_file">Export file</label>
<input
id="kerio_file"
type="file"
accept=".csv,text/csv"
onChange={(e) => setFile(e.target.files?.[0] ?? null)}
disabled={busy}
/>
</div>
{error && <div className="error-banner">{error}</div>}
<div className="modal-actions">
<button type="button" className="btn" onClick={onClose} disabled={busy}>
Cancel
</button>
<button className="btn btn-primary" disabled={busy}>
{busy ? 'Importing…' : 'Import'}
</button>
</div>
</form>
</Modal>
)
}
+55
View File
@@ -0,0 +1,55 @@
import { useEffect, useState } from 'react'
import { Modal } from './Modal'
import { StatusBadge } from './StatusBadge'
import { listRuns, type Run } from '../api'
const fmt = (iso: string | null) => (iso ? new Date(iso).toLocaleString() : '—')
export function RunLogModal({ taskId, open, onClose }: { taskId: number; open: boolean; onClose: () => void }) {
const [runs, setRuns] = useState<Run[] | null>(null)
useEffect(() => {
if (!open) return
setRuns(null)
listRuns(taskId).then((r) => setRuns(r ?? [])).catch(() => setRuns([]))
}, [open, taskId])
return (
<Modal open={open} title="Run log" onClose={onClose} size="lg">
<div className="tbl-wrap">
<table className="tbl">
<thead>
<tr>
<th>Started</th>
<th>Finished</th>
<th>Trigger</th>
<th>Status</th>
<th>Copied</th>
<th>Skipped</th>
<th>Errors</th>
</tr>
</thead>
<tbody>
{runs === null ? (
<tr className="empty-row"><td colSpan={7}>loading</td></tr>
) : runs.length === 0 ? (
<tr className="empty-row"><td colSpan={7}>no runs yet</td></tr>
) : (
runs.map((r) => (
<tr key={r.id}>
<td>{fmt(r.started_at)}</td>
<td>{fmt(r.finished_at)}</td>
<td>{r.trigger}</td>
<td><StatusBadge status={r.status} /></td>
<td className="num-cell">{r.total_copied}</td>
<td className="num-cell">{r.total_skipped}</td>
<td className="num-cell">{r.total_errors}</td>
</tr>
))
)}
</tbody>
</table>
</div>
</Modal>
)
}
+25 -1
View File
@@ -1,5 +1,6 @@
import { useEffect, useState, type FormEvent } from 'react' import { useEffect, useState, type FormEvent } from 'react'
import { createEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api' import { createEndpoint, deleteEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
import { useConfirm } from '../components/ConfirmProvider'
const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode } const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode }
@@ -9,6 +10,7 @@ export function Endpoints() {
const [editingId, setEditingId] = useState<number | null>(null) const [editingId, setEditingId] = useState<number | null>(null)
const [error, setError] = useState<string | null>(null) const [error, setError] = useState<string | null>(null)
const [busy, setBusy] = useState(false) const [busy, setBusy] = useState(false)
const confirm = useConfirm()
function startEdit(ep: Endpoint) { function startEdit(ep: Endpoint) {
setEditingId(ep.id) setEditingId(ep.id)
@@ -29,6 +31,25 @@ export function Endpoints() {
useEffect(reload, []) useEffect(reload, [])
async function onDelete(ep: Endpoint) {
const ok = await confirm({
title: 'Delete endpoint',
message: `Delete endpoint "${ep.role_label}" (${ep.host}:${ep.port})?`,
confirmLabel: 'Delete',
danger: true,
})
if (!ok) return
setError(null)
try {
await deleteEndpoint(ep.id)
// The form still edits a row that no longer exists — drop back to create mode.
if (editingId === ep.id) cancelEdit()
reload()
} catch (e) {
setError(e instanceof Error ? e.message : 'Failed to delete endpoint')
}
}
async function submit(e: FormEvent) { async function submit(e: FormEvent) {
e.preventDefault() e.preventDefault()
setBusy(true) setBusy(true)
@@ -159,6 +180,9 @@ export function Endpoints() {
<td className="num-cell"> <td className="num-cell">
<button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}> <button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}>
edit edit
</button>{' '}
<button type="button" className="link-btn danger" onClick={() => onDelete(ep)} disabled={busy}>
delete
</button> </button>
</td> </td>
</tr> </tr>
+330 -37
View File
@@ -1,9 +1,13 @@
import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react' import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react'
import { cancelAccount, createAccount, deleteAccount, getTask, importCSV, probeAccountFolders, probeFolders, runTask, setAccountFolderMapping, testAccounts, type TaskDetail as TaskDetailData } from '../api' import { cancelAccount, cancelTask, createAccount, deleteAccount, getTask, importCSV, importKerioCSV, pauseTask, probeAccountFolders, probeFolders, resumeTask, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, updateAccountCredentials, type Account, type TaskDetail as TaskDetailData } from '../api'
import { connectTaskWS, type TaskEvent } from '../ws' import { connectTaskWS, type TaskEvent } from '../ws'
import { StatusBadge } from '../components/StatusBadge' import { StatusBadge } from '../components/StatusBadge'
import { useConfirm } from '../components/ConfirmProvider' import { useConfirm } from '../components/ConfirmProvider'
import { FolderMappingModal } from '../components/FolderMappingModal' import { FolderMappingModal } from '../components/FolderMappingModal'
import { RunLogModal } from '../components/RunLogModal'
import { AccountErrorsModal } from '../components/AccountErrorsModal'
import { KerioImportModal } from '../components/KerioImportModal'
import { AccountCredentialsModal } from '../components/AccountCredentialsModal'
const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' } const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' }
@@ -55,6 +59,8 @@ function describeEvent(ev: TaskEvent): string {
} }
case 'cancelled': case 'cancelled':
return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}` return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}`
case 'paused':
return `PAUSED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0} — resumable`
case 'error': { case 'error': {
const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : '' const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : ''
return `ERROR #${d.account_id}${where}: ${d.error}` return `ERROR #${d.account_id}${where}: ${d.error}`
@@ -62,7 +68,7 @@ function describeEvent(ev: TaskEvent): string {
case 'run_started': case 'run_started':
return `RUN started (run #${d.run_id})` return `RUN started (run #${d.run_id})`
case 'run_done': case 'run_done':
return `RUN finished: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}` return `RUN ${String(d.status ?? 'finished')}: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
default: default:
return JSON.stringify(ev.data) return JSON.stringify(ev.data)
} }
@@ -77,6 +83,7 @@ export function TaskDetail({ id }: { id: number }) {
const [mapState, setMapState] = useState<{ src: string[]; dst: string[]; creds: typeof emptyAccount } | null>(null) const [mapState, setMapState] = useState<{ src: string[]; dst: string[]; creds: typeof emptyAccount } | null>(null)
const [editMap, setEditMap] = useState<{ const [editMap, setEditMap] = useState<{
accId: number accId: number
label: string
src: string[] src: string[]
dst: string[] dst: string[]
mapping: Record<string, string> mapping: Record<string, string>
@@ -85,6 +92,11 @@ export function TaskDetail({ id }: { id: number }) {
const confirm = useConfirm() const confirm = useConfirm()
const [error, setError] = useState<string | null>(null) const [error, setError] = useState<string | null>(null)
const [live, setLive] = useState<Record<number, LiveProgress>>({}) const [live, setLive] = useState<Record<number, LiveProgress>>({})
const [showRuns, setShowRuns] = useState(false)
const [errorsFor, setErrorsFor] = useState<{ id: number; src_login: string } | null>(null)
const [kerioOpen, setKerioOpen] = useState(false)
const [credsFor, setCredsFor] = useState<Account | null>(null)
const [selected, setSelected] = useState<Set<number>>(new Set())
const fileInputRef = useRef<HTMLInputElement>(null) const fileInputRef = useRef<HTMLInputElement>(null)
function reload() { function reload() {
@@ -153,7 +165,7 @@ export function TaskDetail({ id }: { id: number }) {
}, },
} }
}) })
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || (ev.type === 'error' && d.folder == null))) { } else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || ev.type === 'paused' || (ev.type === 'error' && d.folder == null))) {
// terminal/reset for this account — drop live overlay, fall back to DB // terminal/reset for this account — drop live overlay, fall back to DB
setLive((prev) => { setLive((prev) => {
if (!(accId in prev)) return prev if (!(accId in prev)) return prev
@@ -164,7 +176,7 @@ export function TaskDetail({ id }: { id: number }) {
} }
// Structural events refresh the persisted view; `progress` is covered by live state. // Structural events refresh the persisted view; `progress` is covered by live state.
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'plan'].includes(ev.type)) { if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'paused', 'plan', 'task_broken'].includes(ev.type)) {
reload() reload()
} }
}), }),
@@ -210,7 +222,7 @@ export function TaskDetail({ id }: { id: number }) {
} }
} }
async function onEditFolders(a: { id: number; src_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) { async function onEditFolders(a: { id: number; src_login: string; dst_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) {
setBusy('probe') setBusy('probe')
setError(null) setError(null)
try { try {
@@ -224,6 +236,7 @@ export function TaskDetail({ id }: { id: number }) {
} }
setEditMap({ setEditMap({
accId: a.id, accId: a.id,
label: a.dst_login && a.dst_login !== a.src_login ? `${a.src_login}${a.dst_login}` : a.src_login,
src: res.src.folders ?? [], src: res.src.folders ?? [],
dst: res.dst.folders ?? [], dst: res.dst.folders ?? [],
mapping: a.folder_mapping ?? {}, mapping: a.folder_mapping ?? {},
@@ -251,20 +264,40 @@ export function TaskDetail({ id }: { id: number }) {
} }
} }
function downloadExampleCSV() { function downloadCSV(name: string, content: string) {
const sample = [ const url = URL.createObjectURL(new Blob([content], { type: 'text/csv' }))
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
].join('\n') + '\n'
const url = URL.createObjectURL(new Blob([sample], { type: 'text/csv' }))
const a = document.createElement('a') const a = document.createElement('a')
a.href = url a.href = url
a.download = 'imap-copier-accounts-example.csv' a.download = name
a.click() a.click()
URL.revokeObjectURL(url) URL.revokeObjectURL(url)
} }
// Plain import: comma-separated, no header, src_login,src_pass,dst_login,dst_pass.
function downloadExampleCSV() {
const sample =
[
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
].join('\n') + '\n'
downloadCSV('imap-copier-accounts-example.csv', sample)
}
// Kerio Connect export: semicolon-separated with a Name;FullName;Description;Enable
// header. The password lives in Description; disabled rows and admin are skipped
// on import, and the domain is supplied in the import dialog.
function downloadKerioExampleCSV() {
const sample =
[
'Name;FullName;Description;Enable',
'alice;Alice Smith;SrcPass1;Yes',
'bob;Bob Jones;SrcPass2;Yes',
'carol;Carol White (disabled, skipped);SrcPass3;No',
].join('\n') + '\n'
downloadCSV('kerio-users-example.csv', sample)
}
async function onFileChosen(e: ChangeEvent<HTMLInputElement>) { async function onFileChosen(e: ChangeEvent<HTMLInputElement>) {
const file = e.target.files?.[0] const file = e.target.files?.[0]
if (!file) return if (!file) return
@@ -281,6 +314,35 @@ export function TaskDetail({ id }: { id: number }) {
} }
} }
async function onKerioImport(file: File, domain: string) {
setBusy('import')
setError(null)
try {
await importKerioCSV(id, file, domain)
setKerioOpen(false)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Kerio import failed')
} finally {
setBusy(null)
}
}
async function saveCredentials(body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) {
if (!credsFor) return
setBusy('add')
setError(null)
try {
await updateAccountCredentials(id, credsFor.id, body)
setCredsFor(null)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to save credentials')
} finally {
setBusy(null)
}
}
async function onDeleteAccount(accId: number, login: string) { async function onDeleteAccount(accId: number, login: string) {
const ok = await confirm({ const ok = await confirm({
title: 'Remove account', title: 'Remove account',
@@ -326,7 +388,8 @@ export function TaskDetail({ id }: { id: number }) {
setBusy('run') setBusy('run')
setError(null) setError(null)
try { try {
await runTask(id) const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
await runTask(id, ids.length ? ids : undefined)
} catch (err) { } catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run') setError(err instanceof Error ? err.message : 'Failed to start run')
} finally { } finally {
@@ -334,6 +397,61 @@ export function TaskDetail({ id }: { id: number }) {
} }
} }
async function onPause() {
setBusy('run')
setError(null)
try {
await pauseTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to pause the run')
} finally {
setBusy(null)
}
}
async function onCancelRun() {
const ok = await confirm({
title: 'Cancel migration',
message: 'Stop the run and mark every unfinished account as cancelled? Copied messages are kept.',
confirmLabel: 'Cancel migration',
cancelLabel: 'Keep running',
danger: true,
})
if (!ok) return
setBusy('run')
setError(null)
try {
await cancelTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to cancel the run')
} finally {
setBusy(null)
}
}
async function onResume() {
setBusy('run')
setError(null)
try {
await resumeTask(id)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to resume the run')
} finally {
setBusy(null)
}
}
async function onSchedule(intervalSeconds: number) {
setError(null)
try {
await setTaskSchedule(id, intervalSeconds)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to set schedule')
}
}
if (notFound) { if (notFound) {
return ( return (
<div className="panel"> <div className="panel">
@@ -350,7 +468,51 @@ export function TaskDetail({ id }: { id: number }) {
} }
const { task, accounts } = data const { task, accounts } = data
const allTested = accounts.length > 0 && accounts.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok') const isRunning = task.status === 'running'
// Accounts a pause left unfinished — what Resume picks up.
const pausedCount = accounts.filter((a) => a.status === 'paused').length
// A failed connection test is the entry point for fixing the credentials that
// caused it — an imported account is otherwise only deletable.
const testCell = (a: Account, status: string) =>
status === 'fail' && !isRunning && a.status !== 'running' ? (
<button
type="button"
className="badge-btn"
title="Edit credentials for this account"
onClick={() => setCredsFor(a)}
>
<StatusBadge status={status} />
</button>
) : (
<StatusBadge status={status} />
)
// A row is selectable only when both connection tests pass and no run is live.
const selectableIds = accounts
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
.map((a) => a.id)
const selectableSet = new Set(selectableIds)
// Effective set: the checked accounts, or all accounts when nothing is checked.
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
const runReady =
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
const allSelectableChecked =
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
function toggleOne(accId: number, checked: boolean) {
setSelected((prev) => {
const next = new Set(prev)
if (checked) next.add(accId)
else next.delete(accId)
return next
})
}
function toggleAll(checked: boolean) {
setSelected(checked ? new Set(selectableIds) : new Set())
}
// Prefer live (WS) copied/skipped over the DB values, which only advance per // Prefer live (WS) copied/skipped over the DB values, which only advance per
// folder — so the summary moves in real time during a large folder. // folder — so the summary moves in real time during a large folder.
const totals = accounts.reduce( const totals = accounts.reduce(
@@ -374,6 +536,7 @@ export function TaskDetail({ id }: { id: number }) {
</h1> </h1>
</div> </div>
<StatusBadge status={task.status} /> <StatusBadge status={task.status} />
{task.broken && <span className="badge badge-fail" style={{ marginLeft: 8 }}><span className="dot" />broken</span>}
</div> </div>
<div className="panel"> <div className="panel">
@@ -401,10 +564,65 @@ export function TaskDetail({ id }: { id: number }) {
<button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}> <button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}>
{busy === 'test' ? 'Testing…' : 'Test connections'} {busy === 'test' ? 'Testing…' : 'Test connections'}
</button> </button>
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}> {isRunning ? (
{busy === 'run' ? 'Starting…' : 'Run migration'} <>
<button className="btn" onClick={onPause} disabled={busy !== null}>
{busy === 'run' ? 'Stopping…' : 'Pause'}
</button>
<button className="btn btn-danger" onClick={onCancelRun} disabled={busy !== null}>
Cancel
</button>
<span className="hint">pause keeps the unfinished accounts resumable</span>
</>
) : (
<>
{pausedCount > 0 && (
<button className="btn btn-primary" onClick={onResume} disabled={busy !== null}>
{busy === 'run' ? 'Resuming…' : `Resume (${pausedCount})`}
</button>
)}
<button
className={pausedCount > 0 ? 'btn' : 'btn btn-primary'}
onClick={onRun}
disabled={busy !== null || !runReady}
>
{busy === 'run'
? 'Starting…'
: effectiveSelected.length > 0
? `Run selected (${effectiveSelected.length})`
: 'Run migration'}
</button>
</>
)}
{!isRunning && !runReady && accounts.length > 0 && (
<span className="hint">
{effectiveSelected.length > 0
? 'selected accounts must pass both connection tests'
: 'run unlocks once every account tests OK on both sides'}
</span>
)}
</div>
<div className="sched-row">
<label htmlFor="sched">Schedule</label>
<select
id="sched"
value={task.schedule_interval_seconds ?? 0}
onChange={(e) => onSchedule(Number(e.target.value))}
>
<option value={0}>Off</option>
<option value={3600}>Every 1h</option>
<option value={10800}>Every 3h</option>
<option value={21600}>Every 6h</option>
<option value={43200}>Every 12h</option>
<option value={86400}>Every 24h</option>
</select>
{task.next_run_at && (
<span className="sched-next">Next run: {new Date(task.next_run_at).toLocaleString()}</span>
)}
{task.broken && <span className="badge badge-fail"><span className="dot" />broken</span>}
<button type="button" className="link-btn" onClick={() => setShowRuns(true)}>
runs
</button> </button>
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
</div> </div>
</div> </div>
@@ -463,13 +681,23 @@ export function TaskDetail({ id }: { id: number }) {
<div className="divider-label">or bulk import</div> <div className="divider-label">or bulk import</div>
<div className="upload-row"> <div className="upload-row">
<label className={`btn file-btn${busy !== null ? ' is-disabled' : ''}`}> <div className="upload-item">
{busy === 'import' ? 'Importing…' : 'Upload CSV'} <label className={`btn file-btn${busy !== null ? ' is-disabled' : ''}`}>
<input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} /> {busy === 'import' ? 'Importing…' : 'Upload CSV'}
</label> <input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} />
<button type="button" className="link-btn" onClick={downloadExampleCSV}> </label>
download example.csv <button type="button" className="link-btn" onClick={downloadExampleCSV}>
</button> download example.csv
</button>
</div>
<div className="upload-item">
<button type="button" className="btn" onClick={() => setKerioOpen(true)} disabled={busy !== null}>
Import from Kerio
</button>
<button type="button" className="link-btn" onClick={downloadKerioExampleCSV}>
download kerio example.csv
</button>
</div>
</div> </div>
</div> </div>
@@ -501,8 +729,19 @@ export function TaskDetail({ id }: { id: number }) {
<table className="tbl"> <table className="tbl">
<thead> <thead>
<tr> <tr>
<th>Source</th> <th className="chk-col">
<th>Destination</th> <input
type="checkbox"
aria-label="Select all accounts"
checked={allSelectableChecked}
ref={(el) => {
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
}}
disabled={isRunning || selectableIds.length === 0}
onChange={(e) => toggleAll(e.target.checked)}
/>
</th>
<th>Account</th>
<th>Src test</th> <th>Src test</th>
<th>Dst test</th> <th>Dst test</th>
<th>Status</th> <th>Status</th>
@@ -521,25 +760,34 @@ export function TaskDetail({ id }: { id: number }) {
) : ( ) : (
accounts.map((a) => ( accounts.map((a) => (
<tr key={a.id}> <tr key={a.id}>
<td className="chk-col">
<input
type="checkbox"
aria-label={`Select ${a.src_login}`}
checked={selected.has(a.id)}
disabled={isRunning || !selectableSet.has(a.id)}
onChange={(e) => toggleOne(a.id, e.target.checked)}
/>
</td>
<td> <td>
{a.src_login} <div className="acct-ident">
<span>{a.src_login}</span>
{a.dst_login !== a.src_login && (
<span className="acct-dst"> {a.dst_login}</span>
)}
</div>
{a.last_error && ( {a.last_error && (
<div className="acct-error" title={a.last_error}> <div className="acct-error" title={a.last_error}>
{a.last_error} {a.last_error}
</div> </div>
)} )}
</td> </td>
<td>{a.dst_login}</td> <td>{testCell(a, a.test_src_status)}</td>
<td> <td>{testCell(a, a.test_dst_status)}</td>
<StatusBadge status={a.test_src_status} />
</td>
<td>
<StatusBadge status={a.test_dst_status} />
</td>
<td> <td>
<StatusBadge status={a.status} /> <StatusBadge status={a.status} />
</td> </td>
<td className="progress-cell"> <td className={`progress-cell${a.status === 'running' ? ' progress-cell--live' : ''}`}>
{(() => { {(() => {
const lv = live[a.id] const lv = live[a.id]
if (!lv || !lv.total) return <span className="muted-note"></span> if (!lv || !lv.total) return <span className="muted-note"></span>
@@ -574,9 +822,33 @@ export function TaskDetail({ id }: { id: number }) {
</td> </td>
<td className="num-cell">{live[a.id]?.copied ?? a.copied}</td> <td className="num-cell">{live[a.id]?.copied ?? a.copied}</td>
<td className="num-cell">{live[a.id]?.skipped ?? a.skipped}</td> <td className="num-cell">{live[a.id]?.skipped ?? a.skipped}</td>
<td className="num-cell">{a.errors}</td> <td className="num-cell">
{a.errors > 0 ? (
<button
type="button"
className="link-btn danger"
onClick={() => setErrorsFor({ id: a.id, src_login: a.src_login })}
>
{a.errors}
</button>
) : (
a.errors
)}
</td>
<td className="num-cell"> <td className="num-cell">
<div className="row-actions"> <div className="row-actions">
{(() => {
const mapped =
Object.keys(a.folder_mapping ?? {}).length > 0 ||
(a.excluded_folders?.length ?? 0) > 0
return mapped ? (
<span
className="map-dot"
title="Folders already mapped for this account"
aria-label="Folders mapped"
/>
) : null
})()}
{a.status !== 'running' && data?.task.status !== 'running' && ( {a.status !== 'running' && data?.task.status !== 'running' && (
<button <button
type="button" type="button"
@@ -619,6 +891,11 @@ export function TaskDetail({ id }: { id: number }) {
dstFolders={mapState.dst} dstFolders={mapState.dst}
initialMapping={task.folder_mapping ?? {}} initialMapping={task.folder_mapping ?? {}}
initialExcluded={[]} initialExcluded={[]}
accountLabel={
mapState.creds.dst_login && mapState.creds.dst_login !== mapState.creds.src_login
? `${mapState.creds.src_login}${mapState.creds.dst_login}`
: mapState.creds.src_login
}
onCancel={() => setMapState(null)} onCancel={() => setMapState(null)}
onConfirm={confirmMapping} onConfirm={confirmMapping}
/> />
@@ -631,10 +908,26 @@ export function TaskDetail({ id }: { id: number }) {
dstFolders={editMap.dst} dstFolders={editMap.dst}
initialMapping={editMap.mapping} initialMapping={editMap.mapping}
initialExcluded={editMap.excluded} initialExcluded={editMap.excluded}
accountLabel={editMap.label}
onCancel={() => setEditMap(null)} onCancel={() => setEditMap(null)}
onConfirm={saveEditMapping} onConfirm={saveEditMapping}
/> />
)} )}
<RunLogModal taskId={id} open={showRuns} onClose={() => setShowRuns(false)} />
<AccountErrorsModal taskId={id} account={errorsFor} onClose={() => setErrorsFor(null)} />
<KerioImportModal
open={kerioOpen}
busy={busy === 'import'}
onClose={() => setKerioOpen(false)}
onSubmit={onKerioImport}
/>
<AccountCredentialsModal
open={credsFor !== null}
busy={busy === 'add'}
account={credsFor}
onClose={() => setCredsFor(null)}
onSubmit={saveCredentials}
/>
</> </>
) )
} }
+1
View File
@@ -163,6 +163,7 @@ export function Tasks() {
</td> </td>
<td> <td>
<StatusBadge status={t.status} /> <StatusBadge status={t.status} />
{t.broken && <span className="badge badge-fail" style={{ marginLeft: 8 }}><span className="dot" />broken</span>}
</td> </td>
<td className="num-cell"> <td className="num-cell">
<button <button