Compare commits

..
24 Commits
Author SHA1 Message Date
vasyanskandClaude Opus 5 0b12d2ed2d Cover pause and resume in the e2e run
The idempotency assertion compared counter deltas between the two runs,
but per-run counters are reset at the start of every run, so the second
run's row already showed that run alone — the delta was negative and the
script failed before reaching anything else.

With that fixed, a second account of 3000 messages exercises the new
stop path end to end: pause mid-folder, assert the task and the account
settle into paused, resume, and assert the resumed run covers every
message while skipping the ones the paused stretch had already copied.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:21:02 +07:00
vasyanskandClaude Opus 5 039ac2f1da Add pause and cancel to a running migration
A live run could only be stopped one account at a time, and stopping it
at all meant losing the queue: the accounts that had not started yet
stayed idle with no record that they were meant to run.

A run now carries a handle holding the context that stops every account
under it plus the reason it was stopped. Pause and cancel take the same
path and differ only in the status left behind — paused accounts are
what Resume re-runs, and the migration journal makes each one continue
where it stopped instead of re-copying. Accounts still queued when the
stop lands get the same status as the interrupted ones, so the whole
remainder is resumable after a pause and cancelled after a cancel.
Database writes keep using the uncancellable context, so statuses and
counters survive the stop.

The scheduler skips paused tasks: auto-starting a full run would defeat
the pause. An operator stopping a run no longer trips the schedule
breaker either — that is for failures, not for intent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:20:52 +07:00
vasyanskandClaude Opus 5 c077b368f3 Allow fixing an account's credentials from a failed test
An imported account with a wrong password could only be deleted and
re-added, which loses its folder mapping and its migration journal.
Clicking either FAIL badge now opens a dialog for both logins and both
passwords.

Passwords are never sent to the browser, so the password fields start
empty and an empty field keeps the stored ciphertext — one side can be
corrected without retyping the other. Saving resets both test verdicts
to unknown: they described the previous credentials, and the run gate
requires a passing test on both sides, so the account cannot start on an
unverified password.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 12:08:28 +07:00
vasyanskandClaude Opus 5 76ada57dd7 Add a defaults button to the folder mapping dialog
Mapping the Exchange/Kerio special folders onto mailcow's by hand is
repetitive work that scales with the number of accounts. "By default"
maps Deleted Items to Trash, Junk E-mail to Junk, Sent Items to Sent and
unchecks Public Folders, which mailcow has no counterpart for.

The destination select only offered the source folder as a name to
create, so a target missing on the destination could not be selected at
all. It now also offers the current selection, and marks any name absent
from the destination as "(create)".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:55 +07:00
vasyanskandClaude Opus 5 bb3635e517 Add a Kerio sample file to the bulk import
Only the plain four-column format had a downloadable example, leaving
the Kerio route undocumented in the UI. Each import button now carries
its own sample link underneath: the plain comma-separated layout and a
Kerio export with the Name;FullName;Description;Enable header, a
disabled row included to show what the import skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:45 +07:00
vasyanskandClaude Opus 5 8bc7ff026d Add endpoint deletion
The endpoints screen could only create and edit servers, so a mistyped
or retired endpoint stayed in the list forever.

Tasks reference endpoints without ON DELETE CASCADE, so a referenced
endpoint is refused with 409 and a count of the tasks using it rather
than cascading away migration history. The foreign-key violation is
mapped to the same status to cover a task created between check and
delete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:12 +07:00
vasyanskandClaude Opus 5 94fb410c59 Fix postgres healthcheck probing a nonexistent database
pg_isready without -d connects to a database named after the user, but
the database is imapcopier, so every probe logged a FATAL and the
healthcheck only passed because pg_isready treats "server rejects the
connection" as reachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:02 +07:00
vasyansk 6bf3a6c4ca Add body read timeout recovery to IMAP client
Introduce Client wrapper with socket deadline support
Add reconnection logic for body read timeouts
Implement test cases for underflow scenarios
Update orchestrator to handle reconnections
2026-07-21 05:29:35 +07:00
vasyansk c741cd19a0 Update docker-compose.yml 2026-07-21 04:41:25 +07:00
vasyansk b6e68bdd90 Add activity tracking to prevent stall timeouts during message transfers
Add OnActivity callback to CopyDeps to prevent stall timeouts during large message transfers
Implement touchReader and touchWriter wrappers to call OnActivity during FETCH and APPEND operations
Add slow message logging to identify performance bottlenecks
Add test case to verify activity reporting during message transfers
Clean up orchestrator account reset code formatting
2026-07-21 04:38:32 +07:00
vasyansk d125320667 fix dtt 2026-07-18 12:21:58 +07:00
vasyansk b352cda166 add kerio format support 2026-07-17 11:42:34 +07:00
vasyansk 5d296c39b1 Merge feat/selective-account-run: selective account run via checkboxes 2026-07-08 13:56:28 +07:00
vasyanskandClaude Opus 4.8 fc335dee3f Add selective account run implementation plan
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 13:55:02 +07:00
vasyansk b439eee164 Add account selection checkboxes and selective run button 2026-07-08 13:48:48 +07:00
vasyansk 2e5036aad5 runTask accepts optional accountIds 2026-07-08 13:44:53 +07:00
vasyansk 599c66189c Parse optional account_ids in run handler 2026-07-08 13:41:14 +07:00
vasyansk 315e56046b Add selectAccounts filter and accountIDs param to orchestrator.Run 2026-07-08 13:37:24 +07:00
vasyansk 692a83a468 fix domain 2026-07-08 13:34:53 +07:00
vasyansk 6d8471efdd add dick pics 2026-07-08 13:33:14 +07:00
vasyanskandClaude Opus 4.8 3eac1cea7d Add selective account run design spec
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 13:30:11 +07:00
vasyansk b076ac92d6 Add folder mapping status indicator to task list
Add account label to folder mapping modal

Update folder mapping logic to include destination login
2026-07-06 19:15:30 +07:00
vasyansk e84366eb0c Remove idle connection timeout handling
Remove error handling for closed connections

Add progress watchdog to detect stalled accounts

Improve error modal styling and pagination
2026-07-05 14:50:25 +07:00
vasyansk 95ddbf5619 Update copy.go 2026-07-05 13:59:56 +07:00
42 changed files with 3156 additions and 273 deletions
+1
View File
@@ -10,3 +10,4 @@
# local cache of the impeccable design hook
.impeccable/
**/.impeccable/
*.csv
+10
View File
@@ -4,6 +4,16 @@ Single-binary Go server (embeds the React SPA) that copies IMAP mailboxes
between a source and a destination account. Non-destructive (copy only,
never deletes), deduplicated by Message-ID, resumable/idempotent re-runs.
## Screenshots
Task list — create migration jobs and track their status:
![Migration tasks list](docs/SCR-20260708-mbxn.png)
Task detail — run control, per-account credentials, live event log and progress:
![Task detail](docs/SCR-20260708-mcal.png)
## Quick start (plain HTTP on :80)
```bash
+2 -2
View File
@@ -8,7 +8,7 @@ services:
volumes:
- pgdata:/var/lib/postgresql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U imap"]
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
interval: 5s
timeout: 3s
retries: 5
@@ -35,7 +35,7 @@ services:
image: caddy:2-alpine
labels:
- "traefik.enable=true"
- "traefik.http.routers.imap-copier.rule=Host(`imap-copier.amegahost.kz`)"
- "traefik.http.routers.imap-copier.rule=Host(`${DOMAIN}`)"
- "traefik.http.routers.imap-copier.entrypoints=https"
- "traefik.http.routers.imap-copier.tls=true"
- "traefik.http.routers.imap-copier.tls.certresolver=letsEncrypt"
+1 -1
View File
@@ -8,7 +8,7 @@ services:
volumes:
- pgdata:/var/lib/postgresql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U imap"]
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
interval: 5s
timeout: 3s
retries: 5
Binary file not shown.

After

Width:  |  Height:  |  Size: 221 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 326 KiB

@@ -0,0 +1,546 @@
# Selective Account Run Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Let the user run a migration for only the checkbox-selected accounts; with no selection, run all (unchanged).
**Architecture:** A pure `selectAccounts` filter narrows the orchestrator's account slice before the test gate; the run HTTP handler parses an optional `account_ids` body and passes it through. The accounts table gains row checkboxes and a "select all" header, and the single "Run migration" button becomes "Run selected (N)" when a selection exists. Copy is already idempotent (dedup by message key), so a re-run of selected accounts skips copied messages and delivers the missing tail.
**Tech Stack:** Go 1.x (net/http, encoding/json, standard testing), React + TypeScript + Vite, existing REST/WS client.
## Global Constraints
- Backend module path: `github.com/vasyansk/imap-copier`.
- `Run` with an empty/nil account-ID set MUST behave exactly as today (all accounts) — the scheduler depends on it.
- Preserve existing `handleRun` error mapping: `ErrNotTested` → 409 "accounts must pass connection tests first"; `ErrAlreadyRunning` → 409 "task is already running".
- Frontend has no unit-test harness; frontend tasks verify via `npm run build` + `tsc --noEmit` + manual app check.
- Follow existing code style: no new dependencies.
---
### Task 1: Orchestrator account filter + `Run` signature
**Files:**
- Modify: `internal/orchestrator/orchestrator.go`
- Test: `internal/orchestrator/orchestrator_test.go`
**Interfaces:**
- Produces:
- `func selectAccounts(accs []store.Account, ids []int64) []store.Account` — returns `accs` unchanged when `ids` is empty/nil; otherwise returns only the accounts whose `ID` is in `ids`, preserving input order.
- `var ErrNoAccountsSelected = errors.New("no matching accounts selected")`
- `func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)` — new trailing `accountIDs` param.
- [ ] **Step 1: Write the failing test**
Add to `internal/orchestrator/orchestrator_test.go`:
```go
func TestSelectAccounts(t *testing.T) {
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
if got := selectAccounts(accs, nil); len(got) != 3 {
t.Fatalf("nil ids must return all, got %d", len(got))
}
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
t.Fatalf("empty ids must return all, got %d", len(got))
}
got := selectAccounts(accs, []int64{3, 1})
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
t.Fatalf("must keep matching ids in input order, got %+v", got)
}
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
t.Fatalf("unknown ids must yield empty, got %d", len(got))
}
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
Expected: FAIL — `undefined: selectAccounts`.
- [ ] **Step 3: Add the filter helper and sentinel error**
In `internal/orchestrator/orchestrator.go`, add the sentinel next to the existing `var ErrNotTested`/`ErrAlreadyRunning`:
```go
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
```
Add the helper (place it near `gateOK`):
```go
// selectAccounts narrows accs to those whose ID is in ids, preserving input
// order. An empty or nil ids means "all accounts" — the scheduler and the
// unfiltered manual run rely on this.
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
if len(ids) == 0 {
return accs
}
want := make(map[int64]struct{}, len(ids))
for _, id := range ids {
want[id] = struct{}{}
}
out := make([]store.Account, 0, len(ids))
for _, a := range accs {
if _, ok := want[a.ID]; ok {
out = append(out, a)
}
}
return out
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
Expected: PASS.
- [ ] **Step 5: Thread `accountIDs` through `Run`**
In `internal/orchestrator/orchestrator.go`, change the `Run` signature and apply the filter after loading accounts. Replace:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
task, err := o.store.GetTask(ctx, taskID)
if err != nil {
return 0, err
}
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
if !gateOK(accs) {
return 0, ErrNotTested
}
```
with:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
task, err := o.store.GetTask(ctx, taskID)
if err != nil {
return 0, err
}
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
accs = selectAccounts(accs, accountIDs)
// A non-empty request that matched nothing is a client error, distinct
// from "not tested".
if len(accountIDs) > 0 && len(accs) == 0 {
return 0, ErrNoAccountsSelected
}
if !gateOK(accs) {
return 0, ErrNotTested
}
```
The rest of `Run` (endpoints, `TryMarkTaskRunning`, `CreateRun`, `go o.runAll(...)`) is unchanged — it already operates over the `accs` slice.
- [ ] **Step 6: Update the two `Run` call sites so the package compiles**
In `internal/scheduler/scheduler.go`, change:
```go
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
```
to:
```go
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
```
In `internal/httpapi/run.go`, inside `handleRun`, change:
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual")
```
to (IDs wired in Task 2; for now pass `nil` to keep it compiling):
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
```
- [ ] **Step 7: Verify the whole backend builds and tests pass**
Run: `go build ./... && go test ./internal/orchestrator/ ./internal/scheduler/ -v`
Expected: build succeeds; all tests PASS.
- [ ] **Step 8: Commit**
```bash
git add internal/orchestrator/orchestrator.go internal/orchestrator/orchestrator_test.go internal/scheduler/scheduler.go internal/httpapi/run.go
git commit -m "Add selectAccounts filter and accountIDs param to orchestrator.Run"
```
---
### Task 2: `handleRun` parses optional `account_ids`
**Files:**
- Modify: `internal/httpapi/run.go`
- Test: `internal/httpapi/run_test.go`
**Interfaces:**
- Consumes: `orchestrator.Run(ctx, taskID, "manual", accountIDs)`, `orchestrator.ErrNoAccountsSelected` (Task 1).
- Produces:
- `func parseRunAccountIDs(r *http.Request) ([]int64, error)` — returns `nil` for an empty body; otherwise decodes `{"account_ids":[...]}` and returns the slice (possibly empty). Returns an error on malformed JSON.
- [ ] **Step 1: Write the failing test**
Add to `internal/httpapi/run_test.go` (the `strings` import is already present):
```go
func TestParseRunAccountIDs(t *testing.T) {
// empty body => nil (run all)
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
ids, err := parseRunAccountIDs(req)
if err != nil || ids != nil {
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
}
// explicit selection
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
ids, err = parseRunAccountIDs(req)
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
}
// malformed JSON => error
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
if _, err := parseRunAccountIDs(req); err == nil {
t.Fatal("malformed body must error")
}
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
Expected: FAIL — `undefined: parseRunAccountIDs`.
- [ ] **Step 3: Implement `parseRunAccountIDs` and wire it into `handleRun`**
In `internal/httpapi/run.go`, add the imports `encoding/json` and `io` to the existing import block, then add the helper:
```go
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
raw, err := io.ReadAll(r.Body)
if err != nil {
return nil, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return nil, nil
}
var body struct {
AccountIDs []int64 `json:"account_ids"`
}
if err := json.Unmarshal(raw, &body); err != nil {
return nil, err
}
return body.AccountIDs, nil
}
```
Add `"bytes"` to the import block as well (used by `bytes.TrimSpace`).
Then replace the body of `handleRun`'s run call. Change:
```go
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
}
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
http.Error(w, "task is already running", http.StatusConflict)
return
}
```
to:
```go
accountIDs, err := parseRunAccountIDs(r)
if err != nil {
http.Error(w, "bad request body", http.StatusBadRequest)
return
}
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
return
}
if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
}
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
http.Error(w, "task is already running", http.StatusConflict)
return
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
Expected: PASS.
- [ ] **Step 5: Verify the whole backend builds and the httpapi suite passes**
Run: `go build ./... && go test ./internal/httpapi/ -v`
Expected: build succeeds; all tests PASS.
- [ ] **Step 6: Commit**
```bash
git add internal/httpapi/run.go internal/httpapi/run_test.go
git commit -m "Parse optional account_ids in run handler"
```
---
### Task 3: Frontend `runTask` accepts account IDs
**Files:**
- Modify: `web/src/api.ts:141`
**Interfaces:**
- Produces: `runTask(id: number, accountIds?: number[]) => Promise<...>` — sends `{account_ids}` JSON only when a non-empty list is given; otherwise a bare POST (all accounts).
- [ ] **Step 1: Update `runTask`**
In `web/src/api.ts`, replace:
```ts
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
```
with:
```ts
export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
```
- [ ] **Step 2: Verify typecheck**
Run: `cd web && npx tsc --noEmit`
Expected: no errors from `api.ts` (pre-existing warnings elsewhere, if any, are unrelated).
- [ ] **Step 3: Commit**
```bash
git add web/src/api.ts
git commit -m "runTask accepts optional accountIds"
```
---
### Task 4: Accounts table checkboxes + selective Run button
**Files:**
- Modify: `web/src/pages/TaskDetail.tsx`
- Modify: `web/src/app.css`
**Interfaces:**
- Consumes: `runTask(id, accountIds?)` (Task 3).
- Produces: no exported interface; internal UI state only.
- [ ] **Step 1: Add selection state and helpers**
In `web/src/pages/TaskDetail.tsx`, add a state hook next to the other `useState` declarations (near line 90):
```tsx
const [selected, setSelected] = useState<Set<number>>(new Set())
```
- [ ] **Step 2: Compute selectable rows and the effective run set**
After the existing `const { task, accounts } = data` / `allTested` block (near line 369), add:
```tsx
const isRunning = task.status === 'running'
// A row is selectable only when both connection tests pass and no run is live.
const selectableIds = accounts
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
.map((a) => a.id)
const selectableSet = new Set(selectableIds)
// Effective set: the checked accounts, or all accounts when nothing is checked.
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
const runReady =
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
const allSelectableChecked =
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
function toggleOne(accId: number, checked: boolean) {
setSelected((prev) => {
const next = new Set(prev)
if (checked) next.add(accId)
else next.delete(accId)
return next
})
}
function toggleAll(checked: boolean) {
setSelected(checked ? new Set(selectableIds) : new Set())
}
```
- [ ] **Step 3: Make `onRun` send the selection**
Replace the existing `onRun` function (near line 331):
```tsx
async function onRun() {
setBusy('run')
setError(null)
try {
await runTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run')
} finally {
setBusy(null)
}
}
```
with:
```tsx
async function onRun() {
setBusy('run')
setError(null)
try {
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
await runTask(id, ids.length ? ids : undefined)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run')
} finally {
setBusy(null)
}
}
```
- [ ] **Step 4: Update the Run button label and gate**
Replace the run button (near line 421):
```tsx
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
{busy === 'run' ? 'Starting…' : 'Run migration'}
</button>
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
```
with:
```tsx
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !runReady || isRunning}>
{busy === 'run'
? 'Starting…'
: effectiveSelected.length > 0
? `Run selected (${effectiveSelected.length})`
: 'Run migration'}
</button>
{!runReady && accounts.length > 0 && (
<span className="hint">
{effectiveSelected.length > 0
? 'selected accounts must pass both connection tests'
: 'run unlocks once every account tests OK on both sides'}
</span>
)}
```
- [ ] **Step 5: Add the checkbox header column**
In the accounts table `<thead><tr>` (near line 542), add a leading `<th>` before `<th>Account</th>`:
```tsx
<th className="chk-col">
<input
type="checkbox"
aria-label="Select all accounts"
checked={allSelectableChecked}
ref={(el) => {
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
}}
disabled={isRunning || selectableIds.length === 0}
onChange={(e) => toggleAll(e.target.checked)}
/>
</th>
```
- [ ] **Step 6: Add the per-row checkbox cell and fix the empty-row colspan**
In the empty-row branch (near line 557), change `colSpan={9}` to `colSpan={10}`.
In the account `<tr>` map (near line 561), add a leading `<td>` as the first child, before the `<td>` with `acct-ident`:
```tsx
<td className="chk-col">
<input
type="checkbox"
aria-label={`Select ${a.src_login}`}
checked={selected.has(a.id)}
disabled={isRunning || !selectableSet.has(a.id)}
onChange={(e) => toggleOne(a.id, e.target.checked)}
/>
</td>
```
- [ ] **Step 7: Add checkbox column CSS**
In `web/src/app.css`, append:
```css
.tbl .chk-col {
width: 32px;
text-align: center;
padding-right: 0;
}
.tbl .chk-col input[type='checkbox'] {
cursor: pointer;
}
.tbl .chk-col input[type='checkbox']:disabled {
cursor: not-allowed;
opacity: 0.4;
}
```
- [ ] **Step 8: Build and typecheck**
Run: `cd web && npx tsc --noEmit && npm run build`
Expected: typecheck clean (any pre-existing oxlint/fast-refresh warnings are unrelated); Vite build succeeds.
- [ ] **Step 9: Manual verification in the running app**
Start the app, open a task with ≥2 accounts that have passed both tests:
1. With nothing checked, the button reads "Run migration" and runs all accounts (confirm `account_started` WS events for every account).
2. Check a subset; the button reads "Run selected (N)". Run it; confirm only the checked accounts start, and the unchecked accounts' Copied/Skipped/Status are untouched.
3. Confirm checkboxes for untested accounts are disabled, and the header checkbox toggles only the selectable rows.
- [ ] **Step 10: Commit**
```bash
git add web/src/pages/TaskDetail.tsx web/src/app.css
git commit -m "Add account selection checkboxes and selective run button"
```
---
## Notes for the executor
- Tasks 1 and 2 are backend and must land in order (Task 2 depends on the new `Run` signature and `ErrNoAccountsSelected`). Task 3 and 4 are frontend; Task 4 depends on Task 3.
- The orchestrator's `Run` is integration-heavy (spawns goroutines, needs a real store/hub), so it is not unit-tested directly; the pure `selectAccounts` filter and the `parseRunAccountIDs` helper carry the backend test coverage, and Task 4 Step 9 covers the end-to-end behavior manually.
@@ -0,0 +1,111 @@
# Selective Account Run — Design
**Date:** 2026-07-08
**Status:** Approved (pending spec review)
## Problem
When a migration run fails for some accounts (e.g. transient `use of closed
network connection`), the user wants to re-run **only the affected accounts**,
not the whole task.
An earlier idea — per-error "retry" buttons inside the errors modal — was
rejected: on a `closed network connection` the copy aborts the whole folder
(`copy.go`, the `net.ErrClosed` branch), so the undelivered tail of the folder
never appears in the error list. A per-UID retry would replay only the few
visible errors and leave the invisible tail uncopied, creating a false sense of
completion.
The correct primitive is a **re-run of the selected accounts**. Copy is
idempotent — `IsMigrated`/`MarkMigrated` dedup by message key — so a re-run
skips everything already copied and delivers only the missing tail, covering the
network-drop case fully.
## Solution
Add checkboxes to the accounts table. The main "Run migration" button runs the
selected accounts; with no selection it runs all accounts (current behavior,
preserved).
## Backend
### `orchestrator.Run`
Change signature to:
```go
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)
```
- `accountIDs == nil || len == 0` → all accounts (unchanged behavior).
- Otherwise, filter the `ListAccountsByTask` result down to the requested IDs
**before** `gateOK`, so only the selected accounts are gate-checked and run.
- IDs not belonging to the task are silently dropped by the filter (defense in
depth; the handler also validates).
- If the filter yields zero accounts, return `ErrNotTested` (nothing runnable) —
reuses the existing 409 path; the handler maps empty selection separately (see
below).
Call sites updated:
- `scheduler.go``Run(ctx, id, "scheduled", nil)` (always all accounts).
- `run.go` (`handleRun`) → passes the parsed IDs.
`runAll`, `runAccount`, counters, `FinishRun`, and task-status reconciliation are
unchanged: they already operate over whatever `accs` slice they are handed.
### `handleRun`
- Parse an **optional** JSON body `{"account_ids": [1,2,3]}`. Empty/absent body
`nil` → all accounts.
- Validate parsed IDs against the task's accounts; if a non-empty selection
resolves to zero valid IDs → `400 Bad Request`.
- Existing error mapping preserved: `ErrNotTested` → 409 "accounts must pass
connection tests first"; `ErrAlreadyRunning` → 409.
## Frontend
### Accounts table (`TaskDetail.tsx`)
- New leading column with a per-row checkbox; a "select all" checkbox in the
header (selects only selectable rows).
- State: `selected: Set<number>` of account IDs.
- A row checkbox is disabled when the account has not passed **both** tests
(`test_src_status`/`test_dst_status` !== 'ok') or the task is `running`.
- "select all" reflects indeterminate/checked state over the selectable rows.
### Run button
- With a non-empty selection: label "Run selected (N)"; calls
`runTask(id, [...selected])`.
- With no selection: label "Run migration"; calls `runTask(id)` (all).
- Enable gate computed over the **effective set** (selected, or all when none
selected): every account in that set must be tested OK, and the task must not
be `running`.
- Side benefit: one untested account no longer blocks running the ready ones —
select only the ready accounts and run them.
### `api.ts`
```ts
export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length
? jsonBody({ account_ids: accountIds })
: { method: 'POST' })
```
## Out of scope (YAGNI)
- The errors modal (`AccountErrorsModal`) is untouched; no retry buttons there.
- No new "partial" task status. After a partial run the task status reflects the
last run's outcome over the accounts that ran — same as today.
- No selection persistence across reloads.
## Testing
- **Backend unit:** `orchestrator.Run` with a subset filters correctly; empty
filter runs all; unknown IDs dropped; empty-after-filter → `ErrNotTested`.
- **Backend handler:** `handleRun` parses body, validates IDs, maps empty
selection → 400, absent body → all.
- **Frontend:** manual verification in the running app — select a subset, run,
confirm only those accounts start (WS `account_started` events), and the
unselected accounts' counters/status are untouched.
+109
View File
@@ -1,9 +1,11 @@
package csvimport
import (
"bufio"
"encoding/csv"
"fmt"
"io"
"regexp"
"strings"
)
@@ -52,3 +54,110 @@ func Parse(r io.Reader) ([]Row, error) {
}
return rows, nil
}
// Kerio Connect exports users as a semicolon-separated file whose first line is
// a header. Only these columns matter; the rest (quotas, last login, …) is
// ignored. The file carries no domain — the caller supplies it.
const (
kerioColName = 0 // login without the domain part
kerioColDescription = 2 // Kerio keeps the plaintext password here
kerioColEnable = 3 // "Yes" / "No"
kerioMinColumns = 4
)
// domainRe accepts a bare DNS domain: labels of alphanumerics/hyphens with at
// least one dot and no scheme, user part, or whitespace.
var domainRe = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$`)
func normalizeDomain(domain string) (string, error) {
d := strings.ToLower(strings.TrimSpace(domain))
if d == "" {
return "", fmt.Errorf("domain is required")
}
if !domainRe.MatchString(d) {
return "", fmt.Errorf("invalid domain %q", domain)
}
return d, nil
}
// skipBOM consumes a leading UTF-8 byte-order mark, which Kerio writes into its
// exports and encoding/csv would otherwise glue onto the first header field.
func skipBOM(br *bufio.Reader) error {
b, err := br.Peek(3)
if err != nil && err != io.EOF {
return err
}
if len(b) == 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
_, _ = br.Discard(3)
}
return nil
}
// ParseKerio reads a Kerio Connect user export and maps every enabled, non-admin
// account onto both sides of a migration: the login and password are identical
// on source and destination, only the server differs. Rows for disabled accounts
// and the built-in admin are skipped.
func ParseKerio(r io.Reader, domain string) ([]Row, error) {
d, err := normalizeDomain(domain)
if err != nil {
return nil, err
}
br := bufio.NewReader(r)
if err := skipBOM(br); err != nil {
return nil, err
}
cr := csv.NewReader(br)
cr.Comma = ';'
cr.FieldsPerRecord = -1 // проверяем сами
cr.LazyQuotes = true // FullName нередко содержит одиночную кавычку
header, err := cr.Read()
if err != nil {
return nil, fmt.Errorf("cannot read header: %w", err)
}
if len(header) < kerioMinColumns || !strings.EqualFold(strings.TrimSpace(header[kerioColName]), "Name") {
return nil, fmt.Errorf("not a Kerio export: expected a header starting with Name;FullName;Description;Enable")
}
var rows []Row
seen := map[string]bool{}
for {
rec, err := cr.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
line, _ := cr.FieldPos(0)
if len(rec) == 1 && strings.TrimSpace(rec[0]) == "" {
continue
}
if len(rec) < kerioMinColumns {
return nil, fmt.Errorf("line %d: expected at least %d columns, got %d", line, kerioMinColumns, len(rec))
}
name := strings.ToLower(strings.TrimSpace(rec[kerioColName]))
if name == "" {
return nil, fmt.Errorf("line %d: Name is empty", line)
}
if !strings.EqualFold(strings.TrimSpace(rec[kerioColEnable]), "Yes") || name == "admin" {
continue
}
pass := strings.TrimSpace(rec[kerioColDescription])
if pass == "" {
return nil, fmt.Errorf("line %d: no password in the Description column for %q", line, name)
}
if seen[name] {
return nil, fmt.Errorf("line %d: duplicate Name %q", line, name)
}
seen[name] = true
login := name + "@" + d
rows = append(rows, Row{SrcLogin: login, SrcPass: pass, DstLogin: login, DstPass: pass})
}
if len(rows) == 0 {
return nil, fmt.Errorf("no enabled accounts found in the export")
}
return rows, nil
}
+97
View File
@@ -49,3 +49,100 @@ func TestParseZeroRowsErrors(t *testing.T) {
t.Fatal("expected error when no rows parsed")
}
}
const kerioHeader = "Name;FullName;Description;Enable;DataSource;Authentication;Role;Groups;MailAddress\n"
func TestParseKerioOK(t *testing.T) {
in := kerioHeader +
"j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n" +
"k.smith;Kim Smith;SrcPass22;Yes;Internal;Internal;No rights;all;k.smith\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 2 {
t.Fatalf("want 2 rows, got %d: %+v", len(rows), rows)
}
want := Row{
SrcLogin: "j.doe@example.test", SrcPass: "SrcPass11",
DstLogin: "j.doe@example.test", DstPass: "SrcPass11",
}
if rows[0] != want {
t.Fatalf("row 0: got %+v, want %+v", rows[0], want)
}
}
func TestParseKerioSkipsDisabledAndAdmin(t *testing.T) {
in := kerioHeader +
"admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n" +
"o.disabled;;OffPass111;No;Internal;Internal;No rights;all;o.disabled\n" +
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" {
t.Fatalf("only the enabled non-admin row must survive, got %+v", rows)
}
}
func TestParseKerioStripsBOM(t *testing.T) {
in := "\ufeff" + kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), "example.test")
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 {
t.Fatalf("want 1 row, got %+v", rows)
}
}
func TestParseKerioRejectsMissingHeader(t *testing.T) {
in := "j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("a file without the Kerio header must error")
}
}
func TestParseKerioRejectsBadDomain(t *testing.T) {
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
for _, domain := range []string{"", " ", "@example.test", "exam ple.test", "example", "info@example.test"} {
if _, err := ParseKerio(strings.NewReader(in), domain); err == nil {
t.Fatalf("domain %q must be rejected", domain)
}
}
}
func TestParseKerioTrimsAndLowercasesDomain(t *testing.T) {
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
rows, err := ParseKerio(strings.NewReader(in), " Example.TEST ")
if err != nil {
t.Fatalf("parse: %v", err)
}
if rows[0].SrcLogin != "info@example.test" {
t.Fatalf("domain must be trimmed and lowercased, got %q", rows[0].SrcLogin)
}
}
func TestParseKerioRejectsEmptyPassword(t *testing.T) {
in := kerioHeader + "info;;;Yes;Internal;Internal;No rights;all;info\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("an enabled account without a password must error")
}
}
func TestParseKerioRejectsDuplicateName(t *testing.T) {
in := kerioHeader +
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n" +
"info;;aaaaaaAa1;Yes;Internal;Internal;No rights;all;info\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("duplicate Name must error")
}
}
func TestParseKerioZeroRowsErrors(t *testing.T) {
in := kerioHeader + "admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n"
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
t.Fatal("expected error when every row is filtered out")
}
}
+76
View File
@@ -154,6 +154,82 @@ func (s *Server) handleCreateAccount(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
}
// handleUpdateAccountCredentials fixes the logins/passwords of an existing
// account — typically after an import brought in a wrong password and the
// connection test failed. An empty password field keeps the stored one, so the
// operator can correct one side without retyping the other.
func (s *Server) handleUpdateAccountCredentials(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
accID, err := pathID(r, "accountId")
if err != nil {
http.Error(w, "bad account id", http.StatusBadRequest)
return
}
task, err := s.store.GetTask(r.Context(), taskID)
if err != nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
acc, ok := s.findAccount(r, taskID, accID)
if !ok {
http.Error(w, "account not found", http.StatusNotFound)
return
}
if task.Status == "running" || acc.Status == "running" {
http.Error(w, "cannot change credentials while the account is running", http.StatusConflict)
return
}
var body struct {
SrcLogin string `json:"src_login"`
SrcPass string `json:"src_pass"`
DstLogin string `json:"dst_login"`
DstPass string `json:"dst_pass"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
http.Error(w, "bad json", http.StatusBadRequest)
return
}
// Same trimming as account creation: pasted logins/passwords often carry a
// stray space or newline that the IMAP server rejects.
body.SrcLogin = strings.TrimSpace(body.SrcLogin)
body.DstLogin = strings.TrimSpace(body.DstLogin)
body.SrcPass = strings.TrimSpace(body.SrcPass)
body.DstPass = strings.TrimSpace(body.DstPass)
if body.SrcLogin == "" || body.DstLogin == "" {
http.Error(w, "src_login and dst_login are required", http.StatusBadRequest)
return
}
encrypt := func(pass string) (*string, error) {
if pass == "" {
return nil, nil // keep the stored password
}
enc, err := crypto.Encrypt(s.cfg.EncKey, []byte(pass))
if err != nil {
return nil, err
}
return &enc, nil
}
srcEnc, err := encrypt(body.SrcPass)
if err != nil {
http.Error(w, "encrypt", http.StatusInternalServerError)
return
}
dstEnc, err := encrypt(body.DstPass)
if err != nil {
http.Error(w, "encrypt", http.StatusInternalServerError)
return
}
if err := s.store.UpdateAccountCredentials(r.Context(), accID, body.SrcLogin, body.DstLogin, srcEnc, dstEnc); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// findAccount returns the account with accID under taskID, or ok=false.
func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) {
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
+33
View File
@@ -2,8 +2,11 @@ package httpapi
import (
"encoding/json"
"errors"
"fmt"
"net/http"
"github.com/jackc/pgx/v5/pgconn"
"github.com/vasyansk/imap-copier/internal/store"
)
@@ -54,6 +57,36 @@ func (s *Server) handleUpdateEndpoint(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// handleDeleteEndpoint removes an endpoint that no task references. A referenced
// endpoint is refused with 409 rather than a foreign-key error, and the same
// status covers the race where a task is created between check and delete.
func (s *Server) handleDeleteEndpoint(w http.ResponseWriter, r *http.Request) {
id, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
used, err := s.store.CountTasksUsingEndpoint(r.Context(), id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if used > 0 {
http.Error(w, fmt.Sprintf("endpoint is used by %d task(s) — delete them first", used), http.StatusConflict)
return
}
if err := s.store.DeleteEndpoint(r.Context(), id); err != nil {
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
http.Error(w, "endpoint is used by a task — delete it first", http.StatusConflict)
return
}
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) {
eps, err := s.store.ListEndpoints(r.Context())
if err != nil {
+5
View File
@@ -15,6 +15,7 @@ func (s *Server) Router() http.Handler {
api.HandleFunc("GET /api/endpoints", s.handleListEndpoints)
api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint)
api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint)
api.HandleFunc("DELETE /api/endpoints/{id}", s.handleDeleteEndpoint)
api.HandleFunc("GET /api/tasks", s.handleListTasks)
api.HandleFunc("POST /api/tasks", s.handleCreateTask)
api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask)
@@ -26,9 +27,13 @@ func (s *Server) Router() http.Handler {
api.HandleFunc("GET /api/tasks/{id}/runs", s.handleListRuns)
api.HandleFunc("GET /api/tasks/{id}/accounts/{accountId}/errors", s.handleListAccountErrors)
api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount)
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/credentials", s.handleUpdateAccountCredentials)
api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV)
api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts)
api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun)
api.HandleFunc("POST /api/tasks/{id}/pause", s.handlePauseRun)
api.HandleFunc("POST /api/tasks/{id}/cancel", s.handleCancelRun)
api.HandleFunc("POST /api/tasks/{id}/resume", s.handleResumeRun)
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount)
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders)
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping)
+94 -2
View File
@@ -1,8 +1,11 @@
package httpapi
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"github.com/vasyansk/imap-copier/internal/crypto"
@@ -11,6 +14,17 @@ import (
"github.com/vasyansk/imap-copier/internal/store"
)
// parseImportRows picks the CSV dialect from the "format" form field. A Kerio
// Connect export holds one login/password pair and no domain, so the operator
// supplies the domain alongside the file; anything else is the plain 4-column
// src/dst format.
func parseImportRows(r *http.Request, file io.Reader) ([]csvimport.Row, error) {
if r.FormValue("format") == "kerio" {
return csvimport.ParseKerio(file, r.FormValue("domain"))
}
return csvimport.Parse(file)
}
func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
@@ -23,7 +37,7 @@ func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
return
}
defer file.Close()
rows, err := csvimport.Parse(file)
rows, err := parseImportRows(r, file)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
@@ -63,13 +77,41 @@ func (s *Server) handleTestAccounts(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusAccepted)
}
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
raw, err := io.ReadAll(r.Body)
if err != nil {
return nil, err
}
if len(bytes.TrimSpace(raw)) == 0 {
return nil, nil
}
var body struct {
AccountIDs []int64 `json:"account_ids"`
}
if err := json.Unmarshal(raw, &body); err != nil {
return nil, err
}
return body.AccountIDs, nil
}
func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
runID, err := s.orch.Run(r.Context(), taskID, "manual")
accountIDs, err := parseRunAccountIDs(r)
if err != nil {
http.Error(w, "bad request body", http.StatusBadRequest)
return
}
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
return
}
if errors.Is(err, orchestrator.ErrNotTested) {
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
@@ -85,6 +127,56 @@ func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
}
// handlePauseRun stops the live run but keeps the unfinished accounts
// resumable; handleCancelRun stops it for good. Both are no-ops (409) when the
// task has no run in flight.
func (s *Server) handlePauseRun(w http.ResponseWriter, r *http.Request) {
s.stopRun(w, r, s.orch.PauseTask)
}
func (s *Server) handleCancelRun(w http.ResponseWriter, r *http.Request) {
s.stopRun(w, r, s.orch.CancelTask)
}
func (s *Server) stopRun(w http.ResponseWriter, r *http.Request, stop func(int64) bool) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
if !stop(taskID) {
http.Error(w, "task is not running", http.StatusConflict)
return
}
w.WriteHeader(http.StatusAccepted)
}
// handleResumeRun restarts a paused task with the accounts its pause left
// unfinished; already-copied messages are skipped by the migration journal.
func (s *Server) handleResumeRun(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
http.Error(w, "bad id", http.StatusBadRequest)
return
}
runID, err := s.orch.ResumeTask(r.Context(), taskID)
switch {
case errors.Is(err, orchestrator.ErrNothingToResume):
http.Error(w, "no paused accounts to resume", http.StatusConflict)
return
case errors.Is(err, orchestrator.ErrNotTested):
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
return
case errors.Is(err, orchestrator.ErrAlreadyRunning):
http.Error(w, "task is already running", http.StatusConflict)
return
case err != nil:
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
}
func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) {
taskID, err := pathID(r, "id")
if err != nil {
+78
View File
@@ -1,7 +1,9 @@
package httpapi
import (
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"strings"
"testing"
@@ -26,3 +28,79 @@ func TestImportCSVFailsOnBadEncKey(t *testing.T) {
t.Fatalf("import must fail on bad EncKey, got %d", rw.Code)
}
}
// importReq builds a multipart import request with the given CSV payload and
// extra form fields, then returns it with the uploaded file ready to read.
func importReq(t *testing.T, csv string, fields map[string]string) (*http.Request, io.Reader) {
t.Helper()
body := &strings.Builder{}
mw := multipart.NewWriter(body)
for k, v := range fields {
_ = mw.WriteField(k, v)
}
fw, _ := mw.CreateFormFile("file", "a.csv")
fw.Write([]byte(csv))
mw.Close()
req := httptest.NewRequest("POST", "/api/tasks/1/import", strings.NewReader(body.String()))
req.Header.Set("Content-Type", mw.FormDataContentType())
req.SetPathValue("id", "1")
file, _, err := req.FormFile("file")
if err != nil {
t.Fatalf("form file: %v", err)
}
return req, file
}
const kerioCSV = "Name;FullName;Description;Enable;DataSource\n" +
"info;;InfoPass11;Yes;Internal\n"
func TestParseImportRowsUsesKerioParserWithDomain(t *testing.T) {
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio", "domain": "example.test"})
rows, err := parseImportRows(req, file)
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" || rows[0].DstLogin != "info@example.test" {
t.Fatalf("kerio rows must carry the supplied domain on both sides, got %+v", rows)
}
}
func TestParseImportRowsKerioRequiresDomain(t *testing.T) {
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio"})
if _, err := parseImportRows(req, file); err == nil {
t.Fatal("kerio import without a domain must error")
}
}
func TestParseImportRowsDefaultsToPlainFormat(t *testing.T) {
req, file := importReq(t, "a@x,p1,a@y,p2\n", nil)
rows, err := parseImportRows(req, file)
if err != nil {
t.Fatalf("parse: %v", err)
}
if len(rows) != 1 || rows[0].SrcLogin != "a@x" || rows[0].DstPass != "p2" {
t.Fatalf("no format field must keep the 4-column parser, got %+v", rows)
}
}
func TestParseRunAccountIDs(t *testing.T) {
// empty body => nil (run all)
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
ids, err := parseRunAccountIDs(req)
if err != nil || ids != nil {
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
}
// explicit selection
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
ids, err = parseRunAccountIDs(req)
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
}
// malformed JSON => error
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
if _, err := parseRunAccountIDs(req); err == nil {
t.Fatal("malformed body must error")
}
}
+2 -3
View File
@@ -4,13 +4,12 @@ import (
"context"
"github.com/emersion/go-imap/v2"
"github.com/emersion/go-imap/v2/imapclient"
)
// FolderMessageCount opens a folder read-only (EXAMINE) and returns how many
// messages it holds — used to plan an accurate overall progress total before
// copying begins. It does not fetch any message bodies.
func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
func FolderMessageCount(c *Client, folder string) (int64, error) {
sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait()
if err != nil {
return 0, err
@@ -19,7 +18,7 @@ func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
}
// ListFolders returns the mailbox names visible on an already-connected, logged-in client.
func ListFolders(c *imapclient.Client) ([]string, error) {
func ListFolders(c *Client) ([]string, error) {
mboxes, err := c.List("", "*", nil).Collect()
if err != nil {
return nil, err
-30
View File
@@ -1,30 +0,0 @@
package imapx
import (
"net"
"time"
)
// idleReadTimeout bounds how long a connection may go WITHOUT receiving any
// bytes from the server before the read is aborted. It is an *idle* timeout,
// not a total deadline: every successful read pushes it forward, so a slow but
// live transfer never trips it — only a genuinely dead/mute socket does. This
// is what stops an account from wedging in "running" forever when a server
// accepts a command (e.g. a large FETCH) and then goes silent.
var idleReadTimeout = 60 * time.Second
// idleConn wraps a net.Conn and arms a fresh read deadline before every Read.
// It lives beneath any TLS layer (the raw TCP conn), so the deadline governs
// the actual blocking network read regardless of encryption.
type idleConn struct {
net.Conn
timeout time.Duration
}
func (c *idleConn) Read(b []byte) (int, error) {
if c.timeout > 0 {
// Ignore the error: a closed conn will surface it from Read below.
_ = c.Conn.SetReadDeadline(time.Now().Add(c.timeout))
}
return c.Conn.Read(b)
}
-60
View File
@@ -1,60 +0,0 @@
package imapx
import (
"io"
"net"
"testing"
"time"
"context"
)
// A server that sends the IMAP greeting, accepts one command, then goes silent
// forever must NOT wedge the client: the idle read-timeout has to abort the
// blocked read so the command returns an error instead of hanging. This is the
// exact production failure (account stuck in "running" with zero progress).
func TestConnectIdleReadTimeoutUnwedgesSilentServer(t *testing.T) {
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
defer ln.Close()
go func() {
c, err := ln.Accept()
if err != nil {
return
}
defer c.Close()
_, _ = io.WriteString(c, "* OK IMAP4rev2 ready\r\n")
// Read the LOGIN command bytes, then never answer.
buf := make([]byte, 512)
_, _ = c.Read(buf)
// Block until the client gives up and closes the connection.
_, _ = c.Read(buf)
}()
old := idleReadTimeout
idleReadTimeout = 150 * time.Millisecond
defer func() { idleReadTimeout = old }()
port := ln.Addr().(*net.TCPAddr).Port
ep := Endpoint{Host: "127.0.0.1", Port: port, TLSMode: "plain"}
c, err := Connect(context.Background(), ep)
if err != nil {
t.Fatalf("Connect: %v", err)
}
done := make(chan error, 1)
go func() { done <- c.Login("user", "pass").Wait() }()
select {
case err := <-done:
if err == nil {
t.Fatal("expected an error from Login against a silent server, got nil")
}
case <-time.After(3 * time.Second):
t.Fatal("Login did not return: idle read-timeout was not enforced (connection wedged)")
}
}
+150 -13
View File
@@ -3,8 +3,13 @@ package imapx
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log/slog"
"net"
"os"
"strings"
"time"
"github.com/emersion/go-imap/v2"
@@ -32,8 +37,37 @@ type CopyDeps struct {
// text — so the orchestrator can persist individual errors for the
// per-account error modal. Folder-level errors are reported by the caller.
OnError func(ref, msg string)
// OnActivity is called repeatedly WHILE a single message body is streamed
// (each FETCH read chunk and each APPEND write chunk). Copying one large
// message can take longer than the orchestrator's stall timeout; without an
// in-body signal the watchdog can't tell a slow-but-live transfer from a
// wedged connection and cancels a healthy copy. May be nil.
OnActivity func()
// ReconnectSrc dials and logs in a FRESH source client, returning it. It is
// called when a body read times out (server under-delivered a literal),
// which leaves the current src connection desynced and unusable. CopyFolder
// swaps to the returned client, re-EXAMINEs the folder, and resumes. The
// implementation is expected to also update any external reference to the
// live src client (e.g. so a cancel path closes the right connection). If
// nil, a body-read timeout aborts the folder instead of recovering.
ReconnectSrc func() (*Client, error)
}
// ErrBodyTimeout means a message body did not finish transferring within the
// idle deadline — the server stopped sending mid-literal. It is almost always a
// server announcing a BODY[] literal larger than the bytes it actually sends,
// which makes go-imap wait forever for bytes that never come. Distinct from a
// closed connection so the caller can skip just this one message and resume.
var ErrBodyTimeout = errors.New("message body read timed out")
// bodyIdleTimeout bounds how long a body read may go with NO bytes arriving
// before it is abandoned. It is generous enough for legitimately slow servers
// (even ~15 KB/s links keep bytes flowing far more often than this) yet well
// under the orchestrator's multi-minute stall watchdog, so an under-delivered
// literal is caught quickly and locally instead of stalling the whole account.
// A var (not const) so tests can shorten it.
var bodyIdleTimeout = 30 * time.Second
// CopyResult summarizes the outcome of one CopyFolder run.
type CopyResult struct {
Copied int
@@ -85,7 +119,7 @@ func metaBatches(total, batchSize uint32) []imap.SeqRange {
// held in memory only for the duration of a single FETCH->APPEND and is
// never written to disk. Messages already migrated (per deps.IsMigrated)
// are skipped without re-fetching their bodies.
func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
func CopyFolder(ctx context.Context, src, dst *Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
var res CopyResult
sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait()
@@ -177,9 +211,34 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
if err := ctx.Err(); err != nil {
return res, err
}
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate); err != nil {
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate, deps.OnActivity); err != nil {
res.Errors++
reportErr(msgRef(q.uid, q.subject), "copy message: "+err.Error())
// A body-read timeout means the server under-delivered this message's
// literal; the src connection is now desynced. Mark the message
// migrated so this and future runs skip it (it is un-fetchable via a
// conforming client), then reconnect src and resume the folder with
// the remaining queued messages.
if errors.Is(err, ErrBodyTimeout) && deps.ReconnectSrc != nil {
if merr := deps.MarkMigrated(dstFolder, q.key); merr != nil {
reportErr(msgRef(q.uid, q.subject), "mark skipped: "+merr.Error())
}
newSrc, rerr := deps.ReconnectSrc()
if rerr != nil {
return res, fmt.Errorf("reconnect src after body timeout in %q: %w", srcFolder, rerr)
}
src = newSrc
if _, serr := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait(); serr != nil {
return res, fmt.Errorf("re-examine %q after reconnect: %w", srcFolder, serr)
}
continue
}
// A closed/broken connection won't recover: every remaining APPEND
// would fail identically. Abort the folder instead of logging
// thousands of the same error; a re-run resumes via dedup.
if errors.Is(err, net.ErrClosed) || strings.Contains(err.Error(), "use of closed network connection") {
return res, fmt.Errorf("dst connection lost in %q: %w", dstFolder, err)
}
continue
}
if err := deps.MarkMigrated(dstFolder, q.key); err != nil {
@@ -195,11 +254,63 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
return res, nil
}
// slowMessage marks how long one message's FETCH or APPEND phase may take
// before it is logged as anomalous. Well below the orchestrator's 3-minute
// stall timeout, so a message that trips the watchdog always leaves a log line
// naming the phase (FETCH vs APPEND) and size — turning a silent stall into
// evidence of which side and which message is the culprit.
const slowMessage = 20 * time.Second
// deadlineReader arms an idle read deadline on the source socket before every
// read of a message body, so a server that stops sending mid-literal — having
// announced a larger BODY[] size than it actually delivers — trips the deadline
// instead of blocking go-imap forever waiting for bytes that never arrive. It
// also pings onActivity as bytes arrive, feeding the orchestrator's stall
// watchdog. The deadline is refreshed on each read, so it bounds IDLE time
// (no bytes) rather than total transfer time — a legitimately slow but steady
// download never trips it.
type deadlineReader struct {
c *Client
r io.Reader
idle time.Duration
on func()
}
func (d deadlineReader) Read(p []byte) (int, error) {
_ = d.c.SetReadDeadline(time.Now().Add(d.idle))
n, err := d.r.Read(p)
if n > 0 && d.on != nil {
d.on()
}
return n, err
}
// touchWriter wraps the APPEND write stream and pings onActivity on every
// non-empty write, so a long upload keeps the stall watchdog fed byte-by-byte.
type touchWriter struct {
w io.Writer
on func()
}
func (t touchWriter) Write(p []byte) (int, error) {
n, err := t.w.Write(p)
if n > 0 && t.on != nil {
t.on()
}
return n, err
}
// streamOne FETCHes BODY[] for one message and APPENDs it into dst without
// spooling to disk. The body is buffered in RAM only for the duration of
// this single FETCH->APPEND round trip.
func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time) error {
// this single FETCH->APPEND round trip. onActivity (may be nil) fires as bytes
// move in either direction, feeding the orchestrator's stall watchdog.
//
// The body read is guarded by an idle deadline on src: if the server goes
// silent mid-literal, streamOne returns ErrBodyTimeout rather than hanging, so
// CopyFolder can skip the message and reconnect.
func streamOne(src, dst *Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time, onActivity func()) error {
bodySection := &imap.FetchItemBodySection{}
fetchStart := time.Now()
fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{
BodySection: []*imap.FetchItemBodySection{bodySection},
})
@@ -210,18 +321,24 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
return fmt.Errorf("no message for uid %v", uid)
}
var body []byte
var readErr error
for {
item := msg.Next()
if item == nil {
break
}
if d, ok := item.(imapclient.FetchItemDataBodySection); ok {
b, err := io.ReadAll(d.Literal)
if err != nil {
return err
body, readErr = io.ReadAll(deadlineReader{c: src, r: d.Literal, idle: bodyIdleTimeout, on: onActivity})
}
body = b
}
// Clear the deadline before any further I/O on src (fetchCmd.Close reads the
// command's completion off the same socket).
_ = src.SetReadDeadline(time.Time{})
if readErr != nil {
if errors.Is(readErr, os.ErrDeadlineExceeded) {
return fmt.Errorf("%w: uid %v (server sent fewer bytes than the announced literal)", ErrBodyTimeout, uid)
}
return readErr
}
if err := fetchCmd.Close(); err != nil {
return err
@@ -229,16 +346,36 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
if body == nil {
return fmt.Errorf("empty body uid %v", uid)
}
fetchDur := time.Since(fetchStart)
appendStart := time.Now()
appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate})
if _, err := io.Copy(appendCmd, bytes.NewReader(body)); err != nil {
// Append acquires go-imap's per-client encoder mutex and holds it until
// Close() calls enc.end(). Close() MUST run on every path: if io.Copy
// fails mid-write (server stall, idle timeout), returning without Close()
// leaks the mutex and the NEXT Append on this client deadlocks forever on
// beginCommand. Close() is idempotent and always releases the lock.
_, copyErr := io.Copy(touchWriter{w: appendCmd, on: onActivity}, bytes.NewReader(body))
closeErr := appendCmd.Close()
if copyErr != nil {
return fmt.Errorf("append body uid %v: %w", uid, copyErr)
}
if closeErr != nil {
return closeErr
}
if _, err := appendCmd.Wait(); err != nil {
return err
}
if err := appendCmd.Close(); err != nil {
return err
appendDur := time.Since(appendStart)
// One message that individually eats a large slice of the stall budget is
// the prime suspect behind a "no progress" cancel; name it, its size, and
// which phase was slow so the culprit is visible in the logs.
if fetchDur > slowMessage || appendDur > slowMessage {
slog.Warn("slow message copy", "uid", uid, "bytes", len(body),
"fetch", fetchDur.Round(time.Millisecond), "append", appendDur.Round(time.Millisecond))
}
_, err := appendCmd.Wait()
return err
return nil
}
// keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8
+50
View File
@@ -169,6 +169,56 @@ func TestCopyFolderPreservesInternalDate(t *testing.T) {
}
}
// TestCopyFolderReportsActivityDuringBody proves CopyFolder invokes OnActivity
// while a message body is being transferred (FETCH/APPEND), not only between
// messages. This is what keeps the orchestrator's stall watchdog from killing a
// single large-but-live message whose transfer legitimately exceeds the stall
// timeout: without an in-body activity signal, one slow message looks identical
// to a wedged connection.
func TestCopyFolderReportsActivityDuringBody(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
seedInbox(t, ep, "actsrc@localhost", "p", 1)
src, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = src.Logout().Wait() }()
if err := src.Login("actsrc@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
dst, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login("actdst@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
var activity int
deps := CopyDeps{
IsMigrated: func(string) (bool, error) { return false, nil },
MarkMigrated: func(_, _ string) error { return nil },
OnProgress: func(_, _ int) {},
OnActivity: func() { activity++ },
}
r, err := CopyFolder(ctx, src, dst, "INBOX", "INBOX", deps)
if err != nil {
t.Fatalf("CopyFolder: %v", err)
}
if r.Copied != 1 {
t.Fatalf("copied=%d want 1", r.Copied)
}
if activity == 0 {
t.Fatal("OnActivity never called during body transfer")
}
}
// Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped).
func TestCopyFolderIdempotent(t *testing.T) {
ep := testEP(t) // plain greenmail
+54 -22
View File
@@ -18,51 +18,83 @@ type Endpoint struct {
func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) }
// dialTimeout bounds establishing the TCP connection (matches go-imap's own
// default). The subsequent idleReadTimeout governs reads once connected.
// Client wraps an imapclient.Client together with the raw network connection it
// runs over. The embedded *imapclient.Client provides the full IMAP API; the
// retained conn lets callers impose a read deadline on the socket for the
// duration of a body transfer.
//
// This defends against servers that announce a BODY[] literal LARGER than the
// bytes they actually send (a protocol violation observed on some webmail
// servers). go-imap reads a literal strictly by its announced size, so a short
// literal makes it block forever waiting for bytes that never arrive. A
// deadline around the body read turns that infinite hang into a timeout the
// copier can recover from.
type Client struct {
*imapclient.Client
conn net.Conn
}
// SetReadDeadline sets (or, with the zero time, clears) a deadline on the
// underlying socket. Used to bound a single body read; always cleared again
// once the read completes so it never affects idle periods.
func (c *Client) SetReadDeadline(t time.Time) error {
return c.conn.SetReadDeadline(t)
}
// dialTimeout bounds establishing the TCP connection.
const dialTimeout = 30 * time.Second
// dialOnce establishes one connection and returns a ready *Client whose reads
// are guarded by idleReadTimeout. Unlike imapclient.Dial*, the underlying TCP
// conn is wrapped in idleConn so a server that stops responding mid-command
// unblocks the read instead of hanging forever. ctx bounds the TCP dial.
func dialOnce(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
// dialOnce establishes one connection and returns a ready *Client. ctx bounds
// the TCP dial. We deliberately do NOT impose a socket-level read deadline:
// a blanket read deadline can't tell an idle connection (e.g. dst sitting idle
// during a long src scan) from one stuck mid-response, and would wrongly close
// idle connections. Stall detection is done at the orchestrator level via a
// progress watchdog; go-imap's own per-command timeouts bound active commands.
func dialOnce(ctx context.Context, ep Endpoint) (*Client, error) {
d := &net.Dialer{Timeout: dialTimeout}
raw, err := d.DialContext(ctx, "tcp", ep.addr())
if err != nil {
return nil, err
}
conn := &idleConn{Conn: raw, timeout: idleReadTimeout}
switch ep.TLSMode {
case "ssl":
// NextProtos mirrors imapclient.DialTLS's ALPN advertisement.
tlsConn := tls.Client(conn, &tls.Config{ServerName: ep.Host, NextProtos: []string{"imap"}})
tlsConn := tls.Client(raw, &tls.Config{ServerName: ep.Host, NextProtos: []string{"imap"}})
if err := tlsConn.HandshakeContext(ctx); err != nil {
_ = conn.Close()
_ = raw.Close()
return nil, err
}
c := imapclient.New(tlsConn, nil)
return waitGreeting(c)
case "starttls":
opts := &imapclient.Options{TLSConfig: &tls.Config{ServerName: ep.Host}}
c, err := imapclient.NewStartTLS(conn, opts)
c, err := waitGreeting(imapclient.New(tlsConn, nil))
if err != nil {
return nil, err
}
return c, nil
return &Client{Client: c, conn: tlsConn}, nil
case "starttls":
// Deadline goes on the raw TCP conn: it sits beneath the TLS layer that
// NewStartTLS negotiates, and a TCP read deadline still interrupts the
// TLS read above it.
opts := &imapclient.Options{TLSConfig: &tls.Config{ServerName: ep.Host}}
c, err := imapclient.NewStartTLS(raw, opts)
if err != nil {
return nil, err
}
return &Client{Client: c, conn: raw}, nil
case "plain":
c := imapclient.New(conn, nil)
return waitGreeting(c)
c, err := waitGreeting(imapclient.New(raw, nil))
if err != nil {
return nil, err
}
return &Client{Client: c, conn: raw}, nil
default:
_ = conn.Close()
_ = raw.Close()
return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode)
}
}
// waitGreeting blocks for the server's initial greeting so a mute server is
// caught at connect time (bounded by idleReadTimeout) rather than at the first
// command. NewStartTLS already awaits the greeting during its STARTTLS upgrade.
// caught at connect time rather than at the first command. NewStartTLS already
// awaits the greeting during its STARTTLS upgrade.
func waitGreeting(c *imapclient.Client) (*imapclient.Client, error) {
if err := c.WaitGreeting(); err != nil {
_ = c.Close()
@@ -71,7 +103,7 @@ func waitGreeting(c *imapclient.Client) (*imapclient.Client, error) {
return c, nil
}
func Connect(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
func Connect(ctx context.Context, ep Endpoint) (*Client, error) {
const attempts = 3
var lastErr error
for i := 0; i < attempts; i++ {
+45
View File
@@ -0,0 +1,45 @@
package imapx
import (
"context"
"time"
)
// KeepaliveInterval is how often an otherwise-idle IMAP connection is pinged
// with NOOP so the server does not drop it.
//
// The destination connection sits completely idle for the entire duration of
// the source-side metadata scan (Pass 1 of CopyFolder), which on a large
// mailbox runs for many minutes across all folders. With no traffic, the
// server closes the idle connection; go-imap's reader then tears the client
// down, and every subsequent APPEND fails with "use of closed network
// connection" — aborting each folder and copying nothing. A periodic NOOP
// keeps the connection warm. 60s is well under the idle timeout of any common
// IMAP server.
const KeepaliveInterval = 60 * time.Second
// Keepalive pings c with a NOOP every interval until ctx is cancelled, keeping
// an idle connection from being dropped by the server. It is meant to run in
// its own goroutine.
//
// It is safe to run concurrently with other commands on c: go-imap serializes
// command submission and supports multiple in-flight commands over a single
// connection (RFC 9051 §5.5 pipelining). A NOOP issued while another command is
// in flight simply queues behind it and completes when the server responds.
//
// Keepalive returns when ctx is done or when a NOOP fails — a failed NOOP means
// the connection is already gone, so there is nothing left to keep alive.
func Keepalive(ctx context.Context, c *Client, interval time.Duration) {
t := time.NewTicker(interval)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if err := c.Noop().Wait(); err != nil {
return
}
}
}
}
+89
View File
@@ -0,0 +1,89 @@
package imapx
import (
"context"
"testing"
"time"
)
// TestKeepaliveReturnsOnContextCancel proves Keepalive is a well-behaved
// goroutine: it exits promptly when its context is cancelled instead of
// leaking.
func TestKeepaliveReturnsOnContextCancel(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
c, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = c.Logout().Wait() }()
if err := c.Login("ka1@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
kctx, cancel := context.WithCancel(ctx)
done := make(chan struct{})
go func() { Keepalive(kctx, c, 10*time.Millisecond); close(done) }()
// Let a few NOOPs fire, then cancel and require a prompt return.
time.Sleep(50 * time.Millisecond)
cancel()
select {
case <-done:
case <-time.After(2 * time.Second):
t.Fatal("Keepalive did not return within 2s of context cancel")
}
}
// TestKeepaliveDoesNotDisruptCopy runs Keepalive on the destination connection
// at an aggressive interval while CopyFolder is APPENDing to it, proving the
// concurrent NOOPs do not corrupt in-flight commands (the real risk of pinging
// a connection that is also being used for real work).
func TestKeepaliveDoesNotDisruptCopy(t *testing.T) {
ep := testEP(t)
ctx := context.Background()
const n = 8
seedInbox(t, ep, "kasrc@localhost", "p", n)
src, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = src.Logout().Wait() }()
if err := src.Login("kasrc@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
dst, err := Connect(ctx, ep)
if err != nil {
t.Fatal(err)
}
defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login("kadst@localhost", "p").Wait(); err != nil {
t.Fatal(err)
}
kctx, cancel := context.WithCancel(ctx)
defer cancel()
go Keepalive(kctx, dst, 1*time.Millisecond)
seen := map[string]bool{}
deps := CopyDeps{
IsMigrated: func(k string) (bool, error) { return seen[k], nil },
MarkMigrated: func(_, k string) error { seen[k] = true; return nil },
OnProgress: func(_, _ int) {},
}
r, err := CopyFolder(kctx, src, dst, "INBOX", "INBOX", deps)
if err != nil {
t.Fatalf("CopyFolder with concurrent keepalive: %v", err)
}
if r.Copied != n {
t.Fatalf("copied=%d want %d", r.Copied, n)
}
if r.Errors != 0 {
t.Fatalf("errors=%d want 0", r.Errors)
}
}
+218
View File
@@ -0,0 +1,218 @@
package imapx
import (
"bufio"
"context"
"errors"
"fmt"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/emersion/go-imap/v2"
)
// underflowServer is a minimal IMAP server that reproduces the amega.kz bug: it
// answers a BODY[] FETCH by announcing a literal LARGER than the bytes it then
// sends, and afterwards goes silent — exactly what makes go-imap's strict
// literal reader block forever. LOGIN/EXAMINE and the Pass-1 metadata FETCH are
// answered normally so a full CopyFolder can reach the poisoned message.
//
// It serves one connection per accept and keeps accepting, so a reconnect gets
// a fresh, well-behaved session (its second EXAMINE reports zero messages, so
// the resumed folder simply finishes).
type underflowServer struct {
ln net.Listener
mu sync.Mutex
accepts int // how many connections have been accepted
stop chan struct{}
}
func newUnderflowServer(t *testing.T) *underflowServer {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatalf("listen: %v", err)
}
s := &underflowServer{ln: ln, stop: make(chan struct{})}
go s.serve()
return s
}
func (s *underflowServer) addr() Endpoint {
a := s.ln.Addr().(*net.TCPAddr)
return Endpoint{Host: "127.0.0.1", Port: a.Port, TLSMode: "plain"}
}
func (s *underflowServer) close() { close(s.stop); _ = s.ln.Close() }
func (s *underflowServer) serve() {
for {
conn, err := s.ln.Accept()
if err != nil {
return
}
s.mu.Lock()
s.accepts++
first := s.accepts == 1
s.mu.Unlock()
go s.handle(conn, first)
}
}
// handle drives one connection. On the FIRST connection the mailbox reports one
// message and its BODY[] fetch under-delivers; on any later connection (i.e.
// after a reconnect) the mailbox is empty so the resumed folder completes.
func (s *underflowServer) handle(conn net.Conn, first bool) {
defer func() { _ = conn.Close() }()
br := bufio.NewReader(conn)
fmt.Fprint(conn, "* OK IMAP4rev1 ready\r\n")
for {
line, err := br.ReadString('\n')
if err != nil {
return
}
fields := strings.Fields(line)
if len(fields) == 0 {
continue
}
tag := fields[0]
up := strings.ToUpper(line)
switch {
case strings.Contains(up, "LOGIN"):
fmt.Fprintf(conn, "%s OK LOGIN completed\r\n", tag)
case strings.Contains(up, "EXAMINE"), strings.Contains(up, "SELECT"):
n := 0
if first {
n = 1
}
fmt.Fprintf(conn, "* %d EXISTS\r\n", n)
fmt.Fprint(conn, "* OK [UIDVALIDITY 1] ok\r\n")
fmt.Fprintf(conn, "%s OK [READ-ONLY] EXAMINE completed\r\n", tag)
case strings.Contains(up, "BODY["), strings.Contains(up, "BODY.PEEK"):
// Poison: announce 100000 bytes, send 10, then stall until shutdown.
fmt.Fprint(conn, "* 1 FETCH (UID 1 BODY[] {100000}\r\n")
fmt.Fprint(conn, "0123456789")
<-s.stop
return
case strings.Contains(up, "FETCH"):
// Pass-1 metadata fetch for the single message.
fmt.Fprint(conn, "* 1 FETCH (UID 1 RFC822.SIZE 100 FLAGS () "+
"INTERNALDATE \"01-Jan-2020 00:00:00 +0000\" "+
"ENVELOPE (\"Wed, 01 Jan 2020 00:00:00 +0000\" \"poison\" NIL NIL NIL NIL NIL NIL NIL \"<poison@x>\"))\r\n")
fmt.Fprintf(conn, "%s OK FETCH completed\r\n", tag)
case strings.Contains(up, "LOGOUT"):
fmt.Fprintf(conn, "* BYE\r\n%s OK LOGOUT completed\r\n", tag)
return
case strings.Contains(up, "CREATE"), strings.Contains(up, "NOOP"):
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
default:
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
}
}
}
// TestStreamOneTimesOutOnLiteralUnderflow proves a server that announces a
// larger BODY[] literal than it sends makes streamOne return ErrBodyTimeout
// (bounded by bodyIdleTimeout) instead of hanging forever.
func TestStreamOneTimesOutOnLiteralUnderflow(t *testing.T) {
restore := shortenBodyIdle(200 * time.Millisecond)
defer restore()
srv := newUnderflowServer(t)
defer srv.close()
ctx := context.Background()
src, err := Connect(ctx, srv.addr())
if err != nil {
t.Fatalf("connect: %v", err)
}
defer func() { _ = src.Close() }()
if err := src.Login("u", "p").Wait(); err != nil {
t.Fatalf("login: %v", err)
}
done := make(chan error, 1)
go func() {
done <- streamOne(src, src, "INBOX", imap.UID(1), nil, time.Time{}, nil)
}()
select {
case err := <-done:
if !errors.Is(err, ErrBodyTimeout) {
t.Fatalf("want ErrBodyTimeout, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("streamOne hung on literal underflow instead of timing out")
}
}
// TestCopyFolderSkipsAndReconnectsOnUnderflow proves CopyFolder does not hang on
// a poisoned message: it marks the message migrated (so future runs skip it),
// invokes ReconnectSrc, and returns with the error counted — the folder is not
// wedged.
func TestCopyFolderSkipsAndReconnectsOnUnderflow(t *testing.T) {
restore := shortenBodyIdle(200 * time.Millisecond)
defer restore()
srv := newUnderflowServer(t)
defer srv.close()
ctx := context.Background()
src, err := Connect(ctx, srv.addr())
if err != nil {
t.Fatalf("connect: %v", err)
}
defer func() { _ = src.Close() }()
if err := src.Login("u", "p").Wait(); err != nil {
t.Fatalf("login: %v", err)
}
var marked []string
var reconnected bool
deps := CopyDeps{
IsMigrated: func(string) (bool, error) { return false, nil },
MarkMigrated: func(_, k string) error { marked = append(marked, k); return nil },
OnProgress: func(_, _ int) {},
ReconnectSrc: func() (*Client, error) {
reconnected = true
nc, derr := Connect(ctx, srv.addr())
if derr != nil {
return nil, derr
}
if lerr := nc.Login("u", "p").Wait(); lerr != nil {
return nil, lerr
}
return nc, nil
},
}
done := make(chan CopyResult, 1)
go func() {
r, _ := CopyFolder(ctx, src, src, "INBOX", "INBOX", deps)
done <- r
}()
select {
case r := <-done:
if r.Errors == 0 {
t.Fatalf("expected the poisoned message counted as an error, got %+v", r)
}
if !reconnected {
t.Fatal("ReconnectSrc was never called")
}
if len(marked) != 1 {
t.Fatalf("poisoned message should be marked migrated once, got %v", marked)
}
case <-time.After(8 * time.Second):
t.Fatal("CopyFolder hung on a poisoned message instead of skipping it")
}
}
func shortenBodyIdle(d time.Duration) func() {
old := bodyIdleTimeout
bodyIdleTimeout = d
return func() { bodyIdleTimeout = old }
}
+278 -31
View File
@@ -5,6 +5,7 @@ import (
"errors"
"log/slog"
"sync"
"sync/atomic"
"time"
"github.com/vasyansk/imap-copier/internal/crypto"
@@ -15,6 +16,8 @@ import (
var ErrNotTested = errors.New("accounts not fully tested")
var ErrAlreadyRunning = errors.New("task already running")
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
var ErrNothingToResume = errors.New("no paused accounts to resume")
// maxAccountErrors caps how many individual error rows one account records per
// run, so a corrupt mailbox producing thousands of failures can't bloat the
@@ -22,6 +25,16 @@ var ErrAlreadyRunning = errors.New("task already running")
// suppressed" note.
const maxAccountErrors = 500
// A running account that emits no scan/copy progress for stallTimeout is wedged
// (silent server mid-FETCH, stalled APPEND). The watchdog cancels it so the
// connections close and the worker unwinds instead of hanging forever. This
// replaces socket-level read deadlines, which can't tell an idle connection
// from a stuck one. The threshold is generous so slow-but-live runs aren't cut.
const (
stallTimeout = 3 * time.Minute
stallCheckInterval = 30 * time.Second
)
// folderPlan is one source folder scheduled for copy and its destination name.
type folderPlan struct {
src, dst string
@@ -50,6 +63,38 @@ func planFolders(folders []string, mapping map[string]string, excluded []string)
return plan
}
// A run stops either on its own or because the operator intervened. Pausing and
// cancelling take the same path — stop the in-flight work — and differ only in
// the status left behind: paused accounts are what Resume picks up again.
type stopReason int32
const (
stopNone stopReason = iota
stopPaused
stopCancelled
)
func (r stopReason) accountStatus() string {
if r == stopPaused {
return "paused"
}
return "cancelled"
}
// runHandle is the live state of one task's run: the cancel that stops every
// account under it, plus why it was stopped.
type runHandle struct {
cancel context.CancelFunc
reason atomic.Int32
}
func (h *runHandle) stopWith(r stopReason) {
h.reason.CompareAndSwap(int32(stopNone), int32(r))
h.cancel()
}
func (h *runHandle) stopReason() stopReason { return stopReason(h.reason.Load()) }
type Orchestrator struct {
store *store.Store
hub *wshub.Hub
@@ -58,10 +103,66 @@ type Orchestrator struct {
mu sync.Mutex
cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy
runs map[int64]*runHandle // task_id -> live run
}
func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator {
return &Orchestrator{store: s, hub: hub, encKey: encKey, concurrency: concurrency, cancels: map[int64]context.CancelFunc{}}
return &Orchestrator{
store: s, hub: hub, encKey: encKey, concurrency: concurrency,
cancels: map[int64]context.CancelFunc{},
runs: map[int64]*runHandle{},
}
}
// PauseTask stops the task's live run, leaving every unfinished account
// "paused" so ResumeTask can pick them up. Returns false if nothing is running.
func (o *Orchestrator) PauseTask(taskID int64) bool { return o.stopRun(taskID, stopPaused) }
// CancelTask stops the task's live run and marks every unfinished account
// "cancelled". Returns false if nothing is running.
func (o *Orchestrator) CancelTask(taskID int64) bool { return o.stopRun(taskID, stopCancelled) }
func (o *Orchestrator) stopRun(taskID int64, reason stopReason) bool {
o.mu.Lock()
h, ok := o.runs[taskID]
o.mu.Unlock()
if !ok {
return false
}
h.stopWith(reason)
return true
}
func (o *Orchestrator) registerRun(taskID int64, h *runHandle) {
o.mu.Lock()
o.runs[taskID] = h
o.mu.Unlock()
}
func (o *Orchestrator) unregisterRun(taskID int64) {
o.mu.Lock()
delete(o.runs, taskID)
o.mu.Unlock()
}
// ResumeTask restarts a paused task with exactly the accounts the pause left
// unfinished. Everything already copied is skipped by the migration journal, so
// each account continues where it stopped.
func (o *Orchestrator) ResumeTask(ctx context.Context, taskID int64) (int64, error) {
accs, err := o.store.ListAccountsByTask(ctx, taskID)
if err != nil {
return 0, err
}
ids := make([]int64, 0, len(accs))
for _, a := range accs {
if a.Status == "paused" {
ids = append(ids, a.ID)
}
}
if len(ids) == 0 {
return 0, ErrNothingToResume
}
return o.Run(ctx, taskID, "manual", ids)
}
// CancelAccount aborts the in-flight copy for one account, if it is running.
@@ -88,6 +189,26 @@ func (o *Orchestrator) unregisterCancel(accountID int64) {
o.mu.Unlock()
}
// selectAccounts narrows accs to those whose ID is in ids, preserving input
// order. An empty or nil ids means "all accounts" — the scheduler and the
// unfiltered manual run rely on this.
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
if len(ids) == 0 {
return accs
}
want := make(map[int64]struct{}, len(ids))
for _, id := range ids {
want[id] = struct{}{}
}
out := make([]store.Account, 0, len(ids))
for _, a := range accs {
if _, ok := want[a.ID]; ok {
out = append(out, a)
}
}
return out
}
func gateOK(accs []store.Account) bool {
if len(accs) == 0 {
return false
@@ -162,7 +283,7 @@ func shouldBreak(trigger string, totErr int64) bool {
return trigger == "scheduled" && totErr > 0
}
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
task, err := o.store.GetTask(ctx, taskID)
if err != nil {
return 0, err
@@ -171,6 +292,12 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (i
if err != nil {
return 0, err
}
accs = selectAccounts(accs, accountIDs)
// A non-empty request that matched nothing is a client error, distinct
// from "not tested".
if len(accountIDs) > 0 && len(accs) == 0 {
return 0, ErrNoAccountsSelected
}
if !gateOK(accs) {
return 0, ErrNotTested
}
@@ -193,11 +320,21 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (i
}
o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}})
go o.runAll(context.WithoutCancel(ctx), task, runID, accs, srcEP, dstEP, trigger)
// dbCtx outlives the request so status/counter writes still land after a
// pause or cancel; runCtx is what Pause/Cancel actually stop, and every
// account's IMAP work hangs off it.
dbCtx := context.WithoutCancel(ctx)
runCtx, runCancel := context.WithCancel(dbCtx)
h := &runHandle{cancel: runCancel}
o.registerRun(taskID, h)
go o.runAll(dbCtx, runCtx, h, task, runID, accs, srcEP, dstEP, trigger)
return runID, nil
}
func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
func (o *Orchestrator) runAll(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
defer o.unregisterRun(task.ID)
defer h.cancel()
defer func() {
if r := recover(); r != nil {
slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r)
@@ -218,7 +355,17 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
sem := make(chan struct{}, o.concurrency)
var wg sync.WaitGroup
for _, a := range accs {
for i, a := range accs {
// Stopped mid-queue: the accounts that never started are marked with the
// same status as the ones that were interrupted, so a pause leaves the
// whole remainder resumable and a cancel leaves it cancelled.
if runCtx.Err() != nil {
st := h.stopReason().accountStatus()
for _, rest := range accs[i:] {
_ = o.store.SetAccountStatus(ctx, rest.ID, st)
}
break
}
wg.Add(1)
sem <- struct{}{}
go func(a store.Account) {
@@ -235,7 +382,7 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
mu.Unlock()
}
}()
c, s, e := o.runAccount(ctx, task, runID, a, srcEP, dstEP)
c, s, e := o.runAccount(ctx, runCtx, h, task, runID, a, srcEP, dstEP)
mu.Lock()
totCopied += c
totSkipped += s
@@ -245,23 +392,32 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
}
wg.Wait()
reason := h.stopReason()
status := "done"
if totErr > 0 {
switch {
case reason == stopPaused:
status = "paused"
case reason == stopCancelled:
status = "cancelled"
case totErr > 0:
status = "done_with_errors"
}
_ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr)
_ = o.store.SetTaskStatus(ctx, task.ID, status)
o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID,
Data: map[string]any{"run_id": runID, "copied": totCopied, "skipped": totSkipped, "errors": totErr}})
Data: map[string]any{"run_id": runID, "status": status,
"copied": totCopied, "skipped": totSkipped, "errors": totErr}})
if shouldBreak(trigger, totErr) {
// An operator stopping the run is not a schedule failure, so leave the
// breaker alone even when the accounts that did run reported errors.
if reason == stopNone && shouldBreak(trigger, totErr) {
_ = o.store.SetTaskBroken(ctx, task.ID)
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
Data: map[string]any{"task_id": task.ID, "errors": totErr}})
}
}
func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
func (o *Orchestrator) runAccount(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID,
"src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port,
@@ -272,10 +428,11 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
_ = o.store.ResetAccountCounters(ctx, a.ID) // start from zero; IncAccountCounters is additive
_ = o.store.ClearAccountErrors(ctx, a.ID) // drop last run's per-error rows
// Per-account cancellable context: IMAP work uses actx (so CancelAccount
// stops it); DB writes keep the parent ctx so status/counters persist even
// after cancellation. ctx is context.WithoutCancel from runAll.
actx, cancel := context.WithCancel(ctx)
// Per-account cancellable context: IMAP work uses actx, so both CancelAccount
// and a task-wide pause/cancel (which cancels runCtx) stop it. DB writes keep
// ctx — the uncancellable one from runAll — so status/counters persist even
// after cancellation.
actx, cancel := context.WithCancel(runCtx)
o.registerCancel(a.ID, cancel)
defer func() {
o.unregisterCancel(a.ID)
@@ -284,41 +441,107 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc)
if err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
}
dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc)
if err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
}
src, err := imapx.Connect(actx, srcEP)
if err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
}
defer func() { _ = src.Logout().Wait() }()
if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
_ = src.Logout().Wait()
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
}
// srcClient holds the LIVE source connection. A body-read timeout (server
// under-delivering a literal) forces a mid-run reconnect via reconnectSrc,
// which swaps this pointer. The cancel goroutine and the deferred logout
// below both read through it, so they always act on the current connection
// rather than a stale one that was already replaced and logged out.
var srcClient atomic.Pointer[imapx.Client]
srcClient.Store(src)
defer func() { _ = srcClient.Load().Logout().Wait() }()
dst, err := imapx.Connect(actx, dstEP)
if err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
}
defer func() { _ = dst.Logout().Wait() }()
if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
}
// reconnectSrc dials and logs in a fresh source client, swaps it in as the
// live connection, and logs the old (desynced) one out. CopyFolder calls it
// to recover after a message body times out: the server left the connection
// mid-literal, so it can't be reused. Bound to actx, so a cancelled account
// fails the dial instead of reconnecting.
reconnectSrc := func() (*imapx.Client, error) {
nc, err := imapx.Connect(actx, srcEP)
if err != nil {
return nil, err
}
if err := nc.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
_ = nc.Logout().Wait()
return nil, err
}
if old := srcClient.Swap(nc); old != nil {
_ = old.Logout().Wait()
}
slog.Info("reconnected src after message body timeout", "account", a.ID, "src_login", a.SrcLogin)
return nc, nil
}
// On cancel, close the connections so any in-flight network read (a slow
// FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately.
go func() {
<-actx.Done()
_ = src.Close()
_ = srcClient.Load().Close()
_ = dst.Close()
}()
// Keep the destination connection warm. It sits idle for the whole
// source-side metadata scan (Pass 1 of CopyFolder), which on a large
// mailbox runs for minutes; without traffic the server drops it and every
// subsequent APPEND fails with "use of closed network connection", copying
// nothing. Periodic NOOPs prevent that. Only dst needs it — src is
// continuously busy scanning/fetching. Bound to actx so it stops with the
// account.
go imapx.Keepalive(actx, dst, imapx.KeepaliveInterval)
// Progress watchdog: track the last time we saw scan/copy activity; if it
// goes quiet for stallTimeout, cancel the account so the connections close
// and this worker unwinds (it would otherwise block forever on a silent
// server). touch() is called on every progress signal below.
var lastActivity atomic.Int64
lastActivity.Store(time.Now().UnixNano())
touch := func() { lastActivity.Store(time.Now().UnixNano()) }
go func() {
t := time.NewTicker(stallCheckInterval)
defer t.Stop()
for {
select {
case <-actx.Done():
return
case <-t.C:
if time.Since(time.Unix(0, lastActivity.Load())) > stallTimeout {
slog.Warn("account stalled with no progress; cancelling",
"account", a.ID, "src_login", a.SrcLogin, "stall", stallTimeout)
_ = o.store.SetAccountError(ctx, a.ID, "stalled: no progress for "+stallTimeout.String()+", cancelled")
cancel()
return
}
}
}
}()
folders, err := imapx.ListFolders(src)
touch()
if err != nil {
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
}
// Planning pass: decide folders from the account's own config, then EXAMINE
@@ -335,6 +558,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
}
plan[i].total = n
grandTotal += n
touch()
}
o.hub.Publish(wshub.Event{Type: "plan", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "src_login": a.SrcLogin, "folders": len(plan), "total": grandTotal,
@@ -366,7 +590,16 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) },
MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) },
OnError: func(ref, msg string) { addErr("message", curFolder, ref, msg) },
// Fires as bytes move within a single message's FETCH/APPEND, so the
// stall watchdog sees a large-but-live transfer as progress instead of
// cancelling it as a wedged connection.
OnActivity: touch,
// Recovers from a message body timeout (server under-delivering a
// literal) by swapping in a fresh source connection so the folder can
// resume with the remaining messages.
ReconnectSrc: reconnectSrc,
OnProgress: func(c, s int) {
touch()
now := time.Now()
done := c + s
// throttle to ~3/sec per account, but always emit folder completion
@@ -386,6 +619,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
},
// Fires after EXAMINE (before the long fetch) with the folder's message count.
OnFolder: func(srcFolder, dstFolder string, total int64) {
touch()
curFolder, curTotal = srcFolder, total
o.hub.Publish(wshub.Event{Type: "folder", TaskID: task.ID, Data: map[string]any{
"account_id": a.ID, "src_login": a.SrcLogin,
@@ -395,6 +629,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
// Fires while streaming metadata (dedup scan) so the UI shows movement
// before bodies start copying. Throttled to ~4/sec, always emit the last.
OnScan: func(scanned, total int64) {
touch()
now := time.Now()
if now.Sub(lastScanEmit) < 250*time.Millisecond && scanned < total {
return
@@ -409,7 +644,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
if actx.Err() != nil {
break // cancelled — stop scheduling more folders
}
res, err := imapx.CopyFolder(actx, src, dst, fp.src, fp.dst, deps)
res, err := imapx.CopyFolder(actx, srcClient.Load(), dst, fp.src, fp.dst, deps)
folderErr := int64(0)
if err != nil && actx.Err() == nil {
slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err)
@@ -431,11 +666,18 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
}
if actx.Err() != nil {
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: task.ID,
// A task-wide pause leaves the account resumable; anything else (per-account
// cancel, stall watchdog, task-wide cancel) leaves it cancelled.
st := "cancelled"
if runCtx.Err() != nil {
st = h.stopReason().accountStatus()
}
_ = o.store.SetAccountStatus(ctx, a.ID, st)
o.hub.Publish(wshub.Event{Type: st, TaskID: task.ID,
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin,
"copied": copied, "skipped": skipped, "errors": errs}})
slog.Info("account cancelled", "account", a.ID, "src_login", a.SrcLogin, "copied", copied, "skipped", skipped)
slog.Info("account stopped", "account", a.ID, "src_login", a.SrcLogin,
"status", st, "copied", copied, "skipped", skipped)
return copied, skipped, errs
}
@@ -451,11 +693,16 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
return copied, skipped, errs
}
func (o *Orchestrator) accountFailed(ctx context.Context, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
// A cancellation surfacing as an error is a cancel, not a failure.
func (o *Orchestrator) accountFailed(ctx, runCtx context.Context, h *runHandle, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
// A cancellation surfacing as an error is a stop, not a failure — and a
// task-wide pause must still leave the account resumable.
if errors.Is(err, context.Canceled) {
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: taskID,
st := "cancelled"
if runCtx.Err() != nil {
st = h.stopReason().accountStatus()
}
_ = o.store.SetAccountStatus(ctx, a.ID, st)
o.hub.Publish(wshub.Event{Type: st, TaskID: taskID,
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}})
return 0, 0, 0
}
@@ -22,3 +22,23 @@ func TestGateOK(t *testing.T) {
t.Fatal("empty accounts must fail gate")
}
}
func TestSelectAccounts(t *testing.T) {
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
if got := selectAccounts(accs, nil); len(got) != 3 {
t.Fatalf("nil ids must return all, got %d", len(got))
}
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
t.Fatalf("empty ids must return all, got %d", len(got))
}
got := selectAccounts(accs, []int64{3, 1})
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
t.Fatalf("must keep matching ids in input order, got %+v", got)
}
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
t.Fatalf("unknown ids must yield empty, got %d", len(got))
}
}
+83
View File
@@ -0,0 +1,83 @@
package orchestrator
import (
"context"
"testing"
)
func TestStopReasonAccountStatus(t *testing.T) {
if got := stopPaused.accountStatus(); got != "paused" {
t.Fatalf("stopPaused = %q want paused", got)
}
if got := stopCancelled.accountStatus(); got != "cancelled" {
t.Fatalf("stopCancelled = %q want cancelled", got)
}
// A run stopped without an operator reason (per-account cancel, stall
// watchdog) must not look like a pause, or Resume would pick it up.
if got := stopNone.accountStatus(); got != "cancelled" {
t.Fatalf("stopNone = %q want cancelled", got)
}
}
// The first stop wins: a cancel arriving after a pause must not downgrade the
// accounts a pause already promised to keep resumable, and vice versa.
func TestRunHandleFirstStopWins(t *testing.T) {
for _, tc := range []struct {
name string
first, later stopReason
}{
{"pause then cancel", stopPaused, stopCancelled},
{"cancel then pause", stopCancelled, stopPaused},
} {
t.Run(tc.name, func(t *testing.T) {
_, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
h.stopWith(tc.first)
h.stopWith(tc.later)
if got := h.stopReason(); got != tc.first {
t.Fatalf("reason = %v want %v", got, tc.first)
}
})
}
}
func TestRunHandleStopCancelsContext(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
if ctx.Err() != nil {
t.Fatal("context cancelled before stop")
}
h.stopWith(stopPaused)
if ctx.Err() == nil {
t.Fatal("stop must cancel the run context")
}
}
// Pause/Cancel report false for a task with no live run, which the HTTP layer
// turns into 409 instead of pretending it stopped something.
func TestStopRunWithoutLiveRun(t *testing.T) {
o := &Orchestrator{runs: map[int64]*runHandle{}}
if o.PauseTask(1) {
t.Fatal("PauseTask must report false with no live run")
}
if o.CancelTask(1) {
t.Fatal("CancelTask must report false with no live run")
}
_, cancel := context.WithCancel(context.Background())
defer cancel()
h := &runHandle{cancel: cancel}
o.registerRun(1, h)
if !o.PauseTask(1) {
t.Fatal("PauseTask must report true for a live run")
}
if got := h.stopReason(); got != stopPaused {
t.Fatalf("reason = %v want stopPaused", got)
}
o.unregisterRun(1)
if o.CancelTask(1) {
t.Fatal("unregistered run must not be stoppable")
}
}
+1 -1
View File
@@ -67,7 +67,7 @@ func (s *Scheduler) tick(ctx context.Context) {
return
}
for _, id := range dueTaskIDs(tasks, time.Now()) {
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
// ErrAlreadyRunning / ErrNotTested are expected races/edge cases, not fatal.
slog.Info("scheduler: run skipped", "task", id, "err", err)
}
+15
View File
@@ -32,6 +32,21 @@ func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
return id, err
}
// UpdateAccountCredentials replaces an account's logins and, when a new
// ciphertext is supplied, its passwords; a nil password keeps the stored one so
// the operator can fix only the side that failed. Both connection tests are
// reset to "unknown" because the previous verdicts no longer describe these
// credentials, which also forces a re-test before the account can run.
func (s *Store) UpdateAccountCredentials(ctx context.Context, id int64, srcLogin, dstLogin string, srcPassEnc, dstPassEnc *string) error {
_, err := s.Pool.Exec(ctx,
`UPDATE accounts SET src_login=$2, dst_login=$3,
src_pass_enc=COALESCE($4, src_pass_enc), dst_pass_enc=COALESCE($5, dst_pass_enc),
test_src_status='unknown', test_dst_status='unknown', last_error=''
WHERE id=$1`,
id, srcLogin, dstLogin, srcPassEnc, dstPassEnc)
return err
}
// DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE).
func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
_, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id)
+41
View File
@@ -65,6 +65,47 @@ func TestResetAccountCounters(t *testing.T) {
}
}
// Fixing an imported account's credentials must replace only what the operator
// supplied: a nil password keeps the stored ciphertext, and both test verdicts
// go back to unknown because they described the old credentials.
func TestUpdateAccountCredentials(t *testing.T) {
s := testStore(t)
ctx := context.Background()
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "oldsrc", DstLogin: "u2", DstPassEnc: "olddst"})
_ = s.SetAccountTestStatus(ctx, accID, "src", "fail")
_ = s.SetAccountTestStatus(ctx, accID, "dst", "ok")
_ = s.SetAccountError(ctx, accID, "authentication failed")
newSrc := "newsrc"
if err := s.UpdateAccountCredentials(ctx, accID, "u@src.example", "u@dst.example", &newSrc, nil); err != nil {
t.Fatalf("update: %v", err)
}
accs, _ := s.ListAccountsByTask(ctx, taskID)
if len(accs) != 1 {
t.Fatalf("len=%d want 1", len(accs))
}
a := accs[0]
if a.SrcLogin != "u@src.example" || a.DstLogin != "u@dst.example" {
t.Fatalf("logins not updated: %q / %q", a.SrcLogin, a.DstLogin)
}
if a.SrcPassEnc != "newsrc" {
t.Fatalf("src password not updated: %q", a.SrcPassEnc)
}
if a.DstPassEnc != "olddst" {
t.Fatalf("nil password must keep the stored one, got %q", a.DstPassEnc)
}
if a.TestSrcStatus != "unknown" || a.TestDstStatus != "unknown" {
t.Fatalf("test statuses not reset: %q / %q", a.TestSrcStatus, a.TestDstStatus)
}
if a.LastError != "" {
t.Fatalf("last_error not cleared: %q", a.LastError)
}
}
func TestSetAccountFolderMapping(t *testing.T) {
s := testStore(t)
ctx := context.Background()
+35
View File
@@ -18,6 +18,41 @@ func TestUpdateEndpoint(t *testing.T) {
}
}
func TestDeleteEndpoint(t *testing.T) {
s := testStore(t)
ctx := context.Background()
id, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a.com", Port: 993, TLSMode: "ssl"})
if err := s.DeleteEndpoint(ctx, id); err != nil {
t.Fatalf("delete: %v", err)
}
eps, _ := s.ListEndpoints(ctx)
if len(eps) != 0 {
t.Fatalf("endpoint not deleted: %d remain", len(eps))
}
}
func TestDeleteEndpointUsedByTaskRefused(t *testing.T) {
s := testStore(t)
ctx := context.Background()
ep1, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "s", Host: "a", Port: 993, TLSMode: "ssl"})
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
if _, err := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2}); err != nil {
t.Fatalf("create task: %v", err)
}
for _, id := range []int64{ep1, ep2} {
n, err := s.CountTasksUsingEndpoint(ctx, id)
if err != nil {
t.Fatalf("count: %v", err)
}
if n != 1 {
t.Fatalf("count for ep %d = %d, want 1", id, n)
}
if err := s.DeleteEndpoint(ctx, id); err == nil {
t.Fatalf("delete of referenced endpoint %d succeeded, want FK violation", id)
}
}
}
func TestDeleteAccountCascadesJournal(t *testing.T) {
s := testStore(t)
ctx := context.Background()
+17
View File
@@ -26,6 +26,23 @@ func (s *Store) UpdateEndpoint(ctx context.Context, e Endpoint) error {
return err
}
// CountTasksUsingEndpoint reports how many tasks reference the endpoint on
// either side, so a delete can be refused with a meaningful message instead of
// surfacing a raw foreign-key violation.
func (s *Store) CountTasksUsingEndpoint(ctx context.Context, id int64) (int, error) {
var n int
err := s.Pool.QueryRow(ctx,
`SELECT count(*) FROM tasks WHERE src_endpoint_id=$1 OR dst_endpoint_id=$1`, id).Scan(&n)
return n, err
}
// DeleteEndpoint removes an endpoint. Tasks reference endpoints without ON
// DELETE CASCADE, so Postgres rejects the delete while any task still uses it.
func (s *Store) DeleteEndpoint(ctx context.Context, id int64) error {
_, err := s.Pool.Exec(ctx, `DELETE FROM endpoints WHERE id=$1`, id)
return err
}
func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) {
var e Endpoint
err := s.Pool.QueryRow(ctx,
+5 -2
View File
@@ -120,13 +120,16 @@ type SchedulableTask struct {
}
// ListSchedulableTasks returns tasks eligible to auto-run: schedule on, not
// broken, not currently running — each joined with its last finished run time.
// broken, neither running nor paused — each joined with its last finished run
// time. A paused task waits for the operator to resume it; auto-starting a full
// run behind their back would defeat the pause.
func (s *Store) ListSchedulableTasks(ctx context.Context) ([]SchedulableTask, error) {
rows, err := s.Pool.Query(ctx,
`SELECT t.id, t.schedule_interval_seconds, t.schedule_anchor,
(SELECT max(finished_at) FROM runs r WHERE r.task_id=t.id AND r.finished_at IS NOT NULL)
FROM tasks t
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken AND t.status <> 'running'`)
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken
AND t.status <> 'running' AND t.status <> 'paused'`)
if err != nil {
return nil, err
}
+94 -9
View File
@@ -158,7 +158,8 @@ wait_test_ok() {
wait_test_ok
wait_run_done() {
for ((i = 1; i <= 60; i++)); do
# Generous: the resume scenario re-scans and copies thousands of messages.
for ((i = 1; i <= 600; i++)); do
local status
status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
if [[ "$status" == "done" ]]; then
@@ -185,16 +186,100 @@ log "POST /run (second run, expect idempotency)"
api POST "/api/tasks/${TASK_ID}/run" >/dev/null
wait_run_done
# Counters are reset at the start of every run, so run 2's row shows run 2
# alone: it must copy nothing and skip what run 1 already migrated.
RES2=$(api GET "/api/tasks/${TASK_ID}")
RUN2_COPIED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].copied')
RUN2_SKIPPED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].skipped')
RUN2_COPIED=$(echo "$RES2" | jq -r '.accounts[0].copied')
RUN2_SKIPPED=$(echo "$RES2" | jq -r '.accounts[0].skipped')
RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors')
RUN2_COPIED_DELTA=$((RUN2_COPIED_TOTAL - RUN1_COPIED))
RUN2_SKIPPED_DELTA=$((RUN2_SKIPPED_TOTAL - RUN1_SKIPPED))
log "run 2: copied_delta=$RUN2_COPIED_DELTA skipped_delta=$RUN2_SKIPPED_DELTA errors=$RUN2_ERRORS"
log "run 2: copied=$RUN2_COPIED skipped=$RUN2_SKIPPED errors=$RUN2_ERRORS"
[[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors"
[[ "$RUN2_COPIED_DELTA" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED_DELTA new messages (expected 0, not idempotent)"
[[ "$RUN2_SKIPPED_DELTA" -gt 0 ]] || fail "run 2 skipped delta is $RUN2_SKIPPED_DELTA (expected >0)"
[[ "$RUN2_COPIED" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED new messages (expected 0, not idempotent)"
[[ "$RUN2_SKIPPED" -eq "$RUN1_COPIED" ]] ||
fail "run 2 skipped $RUN2_SKIPPED of the $RUN1_COPIED messages run 1 copied"
log "PASS: run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED_DELTA skipped=$RUN2_SKIPPED_DELTA (idempotent)"
log "run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED skipped=$RUN2_SKIPPED (idempotent)"
# ---------------------------------------------------------------------------
# Pause / resume: a second account with enough messages that the run is still
# in flight when the pause lands. Pausing must leave the account resumable, and
# resuming must finish it without re-copying what the first stretch already did.
# ---------------------------------------------------------------------------
SRC_USER2="src2@example.com"
DST_USER2="dst2@example.com"
# Large enough that the copy is still in flight when the pause lands — greenmail
# on a local socket copies well over a thousand small messages per second.
SEED_COUNT=3000
log "seeding ${SEED_COUNT} messages into ${SRC_USER2} INBOX (pause/resume scenario)"
python3 "$SEED_PY" 127.0.0.1 3143 "$SRC_USER2" "$MAIL_PASS" "$SEED_COUNT"
log "adding second account (src2 -> dst2)"
ACCOUNT2_ID=$(api POST "/api/tasks/${TASK_ID}/accounts" \
"{\"src_login\":\"${SRC_USER2}\",\"src_pass\":\"${MAIL_PASS}\",\"dst_login\":\"${DST_USER2}\",\"dst_pass\":\"${MAIL_PASS}\"}" | jq -r .id)
[[ "$ACCOUNT2_ID" =~ ^[0-9]+$ ]] || fail "bad second account id: $ACCOUNT2_ID"
log "account2_id=$ACCOUNT2_ID"
log "POST /test (both accounts)"
api POST "/api/tasks/${TASK_ID}/test" >/dev/null
for ((i = 1; i <= 30; i++)); do
BOTH_OK=$(api GET "/api/tasks/${TASK_ID}" |
jq -r '[.accounts[] | select(.test_src_status=="ok" and .test_dst_status=="ok")] | length')
[[ "$BOTH_OK" == "2" ]] && break
sleep 1
done
[[ "$BOTH_OK" == "2" ]] || fail "second account did not pass connection tests (ok count=$BOTH_OK)"
# Account view for account2, by id.
acct2() { api GET "/api/tasks/${TASK_ID}" | jq -r ".accounts[] | select(.id==${ACCOUNT2_ID}) | $1"; }
log "POST /run (account2 only)"
api POST "/api/tasks/${TASK_ID}/run" "{\"account_ids\":[${ACCOUNT2_ID}]}" >/dev/null
# Per-account counters are only written to the DB when a folder completes, so
# "copied so far" is invisible here — wait for the account to go running, give
# the copy a few seconds of real work, then pause mid-folder.
log "waiting for account2 to start running"
for ((i = 1; i <= 120; i++)); do
[[ "$(acct2 .status)" == "running" ]] && break
sleep 0.5
done
[[ "$(acct2 .status)" == "running" ]] || fail "account2 never reached running"
log "letting it copy for a few seconds, then pausing mid-folder"
sleep 5
curl -fsS -b "$COOKIE_JAR" -c "$COOKIE_JAR" -X POST "$BASE/api/tasks/${TASK_ID}/pause" >/dev/null ||
fail "pause rejected — the run finished before the pause landed, seed more messages"
log "waiting for the task to settle into paused"
for ((i = 1; i <= 60; i++)); do
TASK_STATUS=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
[[ "$TASK_STATUS" == "paused" ]] && break
sleep 1
done
[[ "$TASK_STATUS" == "paused" ]] || fail "task status=$TASK_STATUS after pause (expected paused)"
ACC2_STATUS=$(acct2 .status)
[[ "$ACC2_STATUS" == "paused" ]] || fail "account2 status=$ACC2_STATUS after pause (expected paused)"
log "paused (folder-level counters at copied=$(acct2 .copied) of $SEED_COUNT)"
log "POST /resume"
api POST "/api/tasks/${TASK_ID}/resume" >/dev/null
wait_run_done
RESUMED_COPIED=$(acct2 .copied)
RESUMED_SKIPPED=$(acct2 .skipped)
RESUMED_ERRORS=$(acct2 .errors)
RESUMED_TOTAL=$((RESUMED_COPIED + RESUMED_SKIPPED))
log "after resume: copied=$RESUMED_COPIED skipped=$RESUMED_SKIPPED errors=$RESUMED_ERRORS"
[[ "$RESUMED_ERRORS" == "0" ]] || fail "resumed run had errors"
# Counters reset per run, so the resumed run alone must account for every
# message: the ones it copied now plus the ones the paused stretch already did.
[[ "$RESUMED_TOTAL" -eq "$SEED_COUNT" ]] || fail "resumed run covered $RESUMED_TOTAL of $SEED_COUNT messages"
# Non-zero skipped is the proof that the paused stretch's work survived: those
# messages are in the migration journal, so the resume did not re-copy them.
[[ "$RESUMED_SKIPPED" -gt 0 ]] ||
fail "resumed run skipped nothing — the paused stretch's progress was lost"
log "PASS: idempotent re-run; pause was resumable, resume re-copied $RESUMED_COPIED and skipped $RESUMED_SKIPPED of $SEED_COUNT"
+30 -1
View File
@@ -82,11 +82,21 @@ export const updateEndpoint = (
body: { role_label: string; host: string; port: number; tls_mode: TLSMode },
) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' })
export const deleteEndpoint = (id: number) => api(`/api/endpoints/${id}`, { method: 'DELETE' })
export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' })
export const deleteAccount = (taskId: number, accountId: number) =>
api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' })
// Empty password fields keep the stored ones; both connection tests reset to
// unknown server-side, so the account must be re-tested afterwards.
export const updateAccountCredentials = (
taskId: number,
accountId: number,
body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
) => api(`/api/tasks/${taskId}/accounts/${accountId}/credentials`, { ...jsonBody(body), method: 'PUT' })
export const cancelAccount = (taskId: number, accountId: number) =>
api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' })
@@ -138,7 +148,16 @@ export const createAccount = (
export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' })
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
export const runTask = (id: number, accountIds?: number[]) =>
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
// Pause stops the run but leaves its unfinished accounts resumable; cancel ends
// it and marks them cancelled. Resume re-runs exactly the paused accounts.
export const pauseTask = (id: number) => api(`/api/tasks/${id}/pause`, { method: 'POST' })
export const cancelTask = (id: number) => api(`/api/tasks/${id}/cancel`, { method: 'POST' })
export const resumeTask = (id: number) => api<{ run_id: number }>(`/api/tasks/${id}/resume`, { method: 'POST' })
export interface Run {
id: number
@@ -176,3 +195,13 @@ export const importCSV = (id: number, file: File) => {
fd.append('file', file)
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
}
// A Kerio Connect export carries no domain, so the operator supplies it here;
// the login and password apply to both sides of the migration.
export const importKerioCSV = (id: number, file: File, domain: string) => {
const fd = new FormData()
fd.append('file', file)
fd.append('format', 'kerio')
fd.append('domain', domain)
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
}
+135 -7
View File
@@ -327,14 +327,75 @@
white-space: nowrap;
}
/* error text inside the per-account errors modal: wrap long messages */
.err-cell {
max-width: 420px;
/* per-account errors modal: a paginated list so long messages wrap on the
full modal width instead of squeezing into narrow table columns. */
.err-empty {
padding: 24px 8px;
text-align: center;
color: var(--fg-dim);
}
.err-list {
list-style: none;
margin: 0;
padding: 0;
max-height: 60vh;
overflow-y: auto;
}
.err-item {
padding: 10px 4px;
border-bottom: 1px solid var(--border);
}
.err-item-head {
display: flex;
flex-wrap: wrap;
align-items: baseline;
gap: 8px;
font-size: 11px;
letter-spacing: 0.04em;
margin-bottom: 4px;
}
.err-kind {
color: var(--accent);
text-transform: uppercase;
}
.err-folder {
color: var(--fg);
}
.err-ref {
color: var(--fg-dim);
word-break: break-word;
}
.err-time {
margin-left: auto;
color: var(--fg-dim);
}
.err-text {
font-size: 12px;
line-height: 1.45;
color: var(--fail);
white-space: normal;
word-break: break-word;
color: var(--fail);
}
.err-pager {
display: flex;
align-items: center;
justify-content: center;
gap: 14px;
padding-top: 12px;
}
.err-pageinfo {
font-size: 12px;
line-height: 1.4;
color: var(--fg-dim);
}
/* clear-log button: mirrors the .panel-label tab on the right edge */
@@ -427,11 +488,23 @@
cursor: pointer;
}
.map-toolbar {
display: flex;
align-items: center;
gap: 14px;
margin-bottom: 12px;
flex-wrap: wrap;
}
.map-toolbar .btn {
padding: 6px 12px;
font-size: 11px;
}
.map-all {
display: flex;
align-items: center;
gap: 8px;
margin-bottom: 12px;
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.08em;
@@ -594,6 +667,25 @@ table.tbl a.rowlink:focus-visible {
/* ---------- status badges ---------- */
/* A badge that opens a dialog: the badge keeps its own look, the button only
contributes the affordance. */
.badge-btn {
padding: 0;
border: 0;
background: none;
font: inherit;
cursor: pointer;
}
.badge-btn:hover .badge {
filter: brightness(1.25);
}
.badge-btn:focus-visible {
outline: 1px solid var(--accent);
outline-offset: 2px;
}
.badge {
display: inline-flex;
align-items: center;
@@ -736,6 +828,16 @@ table.tbl a.rowlink:hover {
justify-content: flex-end;
}
/* Marks accounts whose folders have already been mapped/excluded. */
.map-dot {
width: 8px;
height: 8px;
border-radius: 50%;
flex: none;
background: var(--ok);
box-shadow: 0 0 5px var(--ok);
}
.empty-row td {
text-align: center;
color: var(--fg-faint);
@@ -834,11 +936,24 @@ table.tbl a.rowlink:hover {
.upload-row {
display: flex;
align-items: center;
align-items: flex-start;
gap: 12px;
flex-wrap: wrap;
}
/* One import route per column: the action on top, its sample file underneath. */
.upload-item {
display: flex;
flex-direction: column;
align-items: stretch;
gap: 6px;
}
.upload-item .link-btn {
align-self: center;
text-align: center;
}
.file-btn {
position: relative;
overflow: hidden;
@@ -985,3 +1100,16 @@ table.tbl a.rowlink:hover {
padding: 12px 14px;
}
}
.tbl .chk-col {
width: 32px;
text-align: center;
padding-right: 0;
}
.tbl .chk-col input[type='checkbox'] {
cursor: pointer;
}
.tbl .chk-col input[type='checkbox']:disabled {
cursor: not-allowed;
opacity: 0.4;
}
@@ -0,0 +1,116 @@
import { useEffect, useState, type FormEvent } from 'react'
import { Modal } from './Modal'
import type { Account } from '../api'
type Props = {
open: boolean
busy: boolean
account: Account | null
onClose: () => void
onSubmit: (body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) => void
}
// Fixes the credentials of an account that failed its connection test — usually
// a wrong password that came in through a CSV import. Passwords are never sent
// back to the browser, so the fields start empty and an empty field means
// "keep the stored password".
export function AccountCredentialsModal({ open, busy, account, onClose, onSubmit }: Props) {
const [srcLogin, setSrcLogin] = useState('')
const [dstLogin, setDstLogin] = useState('')
const [srcPass, setSrcPass] = useState('')
const [dstPass, setDstPass] = useState('')
const [error, setError] = useState<string | null>(null)
useEffect(() => {
if (!open || !account) return
setSrcLogin(account.src_login)
setDstLogin(account.dst_login)
setSrcPass('')
setDstPass('')
setError(null)
}, [open, account])
function submit(e: FormEvent) {
e.preventDefault()
if (srcLogin.trim() === '' || dstLogin.trim() === '') {
setError('Both logins are required')
return
}
setError(null)
onSubmit({
src_login: srcLogin.trim(),
src_pass: srcPass,
dst_login: dstLogin.trim(),
dst_pass: dstPass,
})
}
return (
<Modal open={open} title={account ? `Edit credentials — ${account.src_login}` : 'Edit credentials'} onClose={onClose}>
<form onSubmit={submit}>
<p className="map-hint">
Leave a password field empty to keep the stored one. Saving resets both connection tests, so re-run{' '}
<strong>Test connections</strong> afterwards.
</p>
<div className="field-row">
<div className="field">
<label htmlFor="edit_src_login">Source login</label>
<input
id="edit_src_login"
data-modal-autofocus
value={srcLogin}
onChange={(e) => setSrcLogin(e.target.value)}
disabled={busy}
required
/>
</div>
<div className="field">
<label htmlFor="edit_src_pass">Source password</label>
<input
id="edit_src_pass"
type="password"
value={srcPass}
onChange={(e) => setSrcPass(e.target.value)}
placeholder="unchanged"
autoComplete="new-password"
disabled={busy}
/>
</div>
</div>
<div className="field-row">
<div className="field">
<label htmlFor="edit_dst_login">Destination login</label>
<input
id="edit_dst_login"
value={dstLogin}
onChange={(e) => setDstLogin(e.target.value)}
disabled={busy}
required
/>
</div>
<div className="field">
<label htmlFor="edit_dst_pass">Destination password</label>
<input
id="edit_dst_pass"
type="password"
value={dstPass}
onChange={(e) => setDstPass(e.target.value)}
placeholder="unchanged"
autoComplete="new-password"
disabled={busy}
/>
</div>
</div>
{error && <div className="error-banner">{error}</div>}
<div className="modal-actions">
<button type="button" className="btn" onClick={onClose} disabled={busy}>
Cancel
</button>
<button className="btn btn-primary" disabled={busy}>
{busy ? 'Saving…' : 'Save credentials'}
</button>
</div>
</form>
</Modal>
)
}
+55 -29
View File
@@ -3,6 +3,7 @@ import { Modal } from './Modal'
import { listAccountErrors, type AccountError } from '../api'
const fmt = (iso: string) => (iso ? new Date(iso).toLocaleString() : '—')
const PAGE_SIZE = 50
export function AccountErrorsModal({
taskId,
@@ -15,51 +16,76 @@ export function AccountErrorsModal({
}) {
const [errors, setErrors] = useState<AccountError[] | null>(null)
const [failed, setFailed] = useState(false)
const [page, setPage] = useState(0)
const open = account !== null
useEffect(() => {
if (!account) return
setErrors(null)
setFailed(false)
setPage(0)
listAccountErrors(taskId, account.id)
.then((e) => setErrors(e ?? []))
.catch(() => setFailed(true))
}, [taskId, account])
const total = errors?.length ?? 0
const pageCount = Math.max(1, Math.ceil(total / PAGE_SIZE))
const current = Math.min(page, pageCount - 1)
const slice = errors?.slice(current * PAGE_SIZE, current * PAGE_SIZE + PAGE_SIZE) ?? []
const title = account
? `Errors — ${account.src_login}${total ? ` (${total})` : ''}`
: 'Errors'
return (
<Modal open={open} title={account ? `Errors — ${account.src_login}` : 'Errors'} onClose={onClose} size="lg">
<div className="tbl-wrap">
<table className="tbl">
<thead>
<tr>
<th>Time</th>
<th>Kind</th>
<th>Folder</th>
<th>Message</th>
<th>Error</th>
</tr>
</thead>
<tbody>
<Modal open={open} title={title} onClose={onClose} size="lg">
{failed ? (
<tr className="empty-row"><td colSpan={5}>failed to load errors</td></tr>
<div className="err-empty">failed to load errors</div>
) : errors === null ? (
<tr className="empty-row"><td colSpan={5}>loading</td></tr>
) : errors.length === 0 ? (
<tr className="empty-row"><td colSpan={5}>no errors recorded</td></tr>
<div className="err-empty">loading</div>
) : total === 0 ? (
<div className="err-empty">no errors recorded</div>
) : (
errors.map((e) => (
<tr key={e.id}>
<td>{fmt(e.created_at)}</td>
<td>{e.kind}</td>
<td>{e.folder || '—'}</td>
<td>{e.message_ref || '—'}</td>
<td className="err-cell">{e.error}</td>
</tr>
))
)}
</tbody>
</table>
<>
<ul className="err-list">
{slice.map((e) => (
<li key={e.id} className="err-item">
<div className="err-item-head">
<span className="err-kind">{e.kind}</span>
{e.folder && <span className="err-folder">{e.folder}</span>}
{e.message_ref && <span className="err-ref">{e.message_ref}</span>}
<span className="err-time">{fmt(e.created_at)}</span>
</div>
<div className="err-text">{e.error}</div>
</li>
))}
</ul>
{pageCount > 1 && (
<div className="err-pager">
<button
type="button"
className="btn"
disabled={current === 0}
onClick={() => setPage(current - 1)}
>
prev
</button>
<span className="err-pageinfo">
page {current + 1} / {pageCount}
</span>
<button
type="button"
className="btn"
disabled={current >= pageCount - 1}
onClick={() => setPage(current + 1)}
>
next
</button>
</div>
)}
</>
)}
</Modal>
)
}
+62 -7
View File
@@ -7,6 +7,7 @@ type Props = {
dstFolders: string[]
initialMapping: Record<string, string>
initialExcluded: string[]
accountLabel?: string
onConfirm: (mapping: Record<string, string>, excluded: string[]) => void
onCancel: () => void
}
@@ -19,8 +20,23 @@ function defaultDst(src: string, dstFolders: string[], initial: Record<string, s
return src
}
// Exchange/Kerio special folders and their mailcow counterparts. Keyed by the
// lowercased source name so casing differences between servers don't matter.
const DEFAULT_TARGETS: Record<string, string> = {
'deleted items': 'Trash',
'deleted messages': 'Trash',
'junk e-mail': 'Junk',
'junk email': 'Junk',
spam: 'Junk',
'sent items': 'Sent',
'sent messages': 'Sent',
}
// Source folders with no counterpart on mailcow — unchecked by the defaults.
const DEFAULT_EXCLUDED = new Set(['public folders'])
export function FolderMappingModal({
open, srcFolders, dstFolders, initialMapping, initialExcluded, onConfirm, onCancel,
open, srcFolders, dstFolders, initialMapping, initialExcluded, accountLabel, onConfirm, onCancel,
}: Props) {
const [choice, setChoice] = useState<Record<string, string>>({})
const [synced, setSynced] = useState<Record<string, boolean>>(() => {
@@ -30,16 +46,40 @@ export function FolderMappingModal({
// Options per select: all destination folders, plus the source name itself
// (marked "create") when it does not already exist on the destination.
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
// Options per select: all destination folders, plus any name not present there
// — the source folder itself and the current selection — marked "create".
const options = useMemo(() => {
const set = new Set(dstFolders)
return (src: string) => {
return (src: string, current: string) => {
const opts = [...dstFolders]
if (!set.has(src)) opts.unshift(src)
if (!set.has(current)) opts.unshift(current)
if (!set.has(src) && src !== current) opts.unshift(src)
return opts
}
}, [dstFolders])
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
// Collapse the Exchange/Kerio folder layout onto mailcow's in one click: the
// per-account mapping is otherwise repetitive work when importing many users.
function applyDefaults() {
const nextChoice = { ...choice }
const nextSynced = { ...synced }
for (const src of srcFolders) {
const key = src.trim().toLowerCase()
if (DEFAULT_EXCLUDED.has(key)) {
nextSynced[src] = false
continue
}
const target = DEFAULT_TARGETS[key]
if (target) {
nextChoice[src] = target
nextSynced[src] = true
}
}
setChoice(nextChoice)
setSynced(nextSynced)
}
function confirm() {
const mapping: Record<string, string> = {}
@@ -58,12 +98,26 @@ export function FolderMappingModal({
}
return (
<Modal open={open} title="Map folders (source → destination)" onClose={onCancel} size="lg">
<Modal
open={open}
title={accountLabel ? `Map folders — ${accountLabel}` : 'Map folders (source → destination)'}
onClose={onCancel}
size="lg"
>
<div className="map-body">
<p className="map-hint">
Route each source folder to an existing destination folder. Leaving a folder mapped to its own name
creates it on the destination if missing (e.g. map <code>Спам</code> <code>Spam</code> to avoid duplicates).
</p>
<div className="map-toolbar">
<button
type="button"
className="btn btn-ghost"
onClick={applyDefaults}
title="Map Deleted Items → Trash, Junk E-mail → Junk, Sent Items → Sent and skip Public Folders"
>
By default
</button>
<label className="map-all">
<input
type="checkbox"
@@ -75,6 +129,7 @@ export function FolderMappingModal({
/>
sync all folders
</label>
</div>
<div className="map-grid">
{srcFolders.map((src) => {
const on = synced[src] !== false
@@ -101,10 +156,10 @@ export function FolderMappingModal({
disabled={!on}
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
>
{options(src).map((f) => (
{options(src, val).map((f) => (
<option key={f} value={f}>
{f}
{f === src && !dstFolders.includes(src) ? ' (create)' : ''}
{dstFolders.includes(f) ? '' : ' (create)'}
</option>
))}
</select>
+82
View File
@@ -0,0 +1,82 @@
import { useEffect, useState, type FormEvent } from 'react'
import { Modal } from './Modal'
// A bare DNS domain: no scheme, no user part, no whitespace. Mirrors the
// server-side check in csvimport.normalizeDomain so bad input is caught here.
const domainRe = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/
type Props = {
open: boolean
busy: boolean
onClose: () => void
onSubmit: (file: File, domain: string) => void
}
export function KerioImportModal({ open, busy, onClose, onSubmit }: Props) {
const [domain, setDomain] = useState('')
const [file, setFile] = useState<File | null>(null)
const [error, setError] = useState<string | null>(null)
useEffect(() => {
if (!open) return
setDomain('')
setFile(null)
setError(null)
}, [open])
function submit(e: FormEvent) {
e.preventDefault()
const d = domain.trim().toLowerCase()
if (!domainRe.test(d)) {
setError('Enter a bare domain, e.g. galaxyhotel.kz')
return
}
if (!file) {
setError('Choose the Kerio export file')
return
}
setError(null)
onSubmit(file, d)
}
return (
<Modal open={open} title="Import from Kerio" onClose={onClose}>
<form onSubmit={submit}>
<p className="map-hint">
The Kerio user export lists a login and its password but no domain. The domain you enter is appended to every
login and used for both the source and the destination. Disabled accounts and <code>admin</code> are skipped.
</p>
<div className="field">
<label htmlFor="kerio_domain">Mail domain</label>
<input
id="kerio_domain"
data-modal-autofocus
value={domain}
onChange={(e) => setDomain(e.target.value)}
placeholder="galaxyhotel.kz"
disabled={busy}
/>
</div>
<div className="field">
<label htmlFor="kerio_file">Export file</label>
<input
id="kerio_file"
type="file"
accept=".csv,text/csv"
onChange={(e) => setFile(e.target.files?.[0] ?? null)}
disabled={busy}
/>
</div>
{error && <div className="error-banner">{error}</div>}
<div className="modal-actions">
<button type="button" className="btn" onClick={onClose} disabled={busy}>
Cancel
</button>
<button className="btn btn-primary" disabled={busy}>
{busy ? 'Importing…' : 'Import'}
</button>
</div>
</form>
</Modal>
)
}
+25 -1
View File
@@ -1,5 +1,6 @@
import { useEffect, useState, type FormEvent } from 'react'
import { createEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
import { createEndpoint, deleteEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
import { useConfirm } from '../components/ConfirmProvider'
const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode }
@@ -9,6 +10,7 @@ export function Endpoints() {
const [editingId, setEditingId] = useState<number | null>(null)
const [error, setError] = useState<string | null>(null)
const [busy, setBusy] = useState(false)
const confirm = useConfirm()
function startEdit(ep: Endpoint) {
setEditingId(ep.id)
@@ -29,6 +31,25 @@ export function Endpoints() {
useEffect(reload, [])
async function onDelete(ep: Endpoint) {
const ok = await confirm({
title: 'Delete endpoint',
message: `Delete endpoint "${ep.role_label}" (${ep.host}:${ep.port})?`,
confirmLabel: 'Delete',
danger: true,
})
if (!ok) return
setError(null)
try {
await deleteEndpoint(ep.id)
// The form still edits a row that no longer exists — drop back to create mode.
if (editingId === ep.id) cancelEdit()
reload()
} catch (e) {
setError(e instanceof Error ? e.message : 'Failed to delete endpoint')
}
}
async function submit(e: FormEvent) {
e.preventDefault()
setBusy(true)
@@ -159,6 +180,9 @@ export function Endpoints() {
<td className="num-cell">
<button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}>
edit
</button>{' '}
<button type="button" className="link-btn danger" onClick={() => onDelete(ep)} disabled={busy}>
delete
</button>
</td>
</tr>
+263 -24
View File
@@ -1,11 +1,13 @@
import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react'
import { cancelAccount, createAccount, deleteAccount, getTask, importCSV, probeAccountFolders, probeFolders, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, type TaskDetail as TaskDetailData } from '../api'
import { cancelAccount, cancelTask, createAccount, deleteAccount, getTask, importCSV, importKerioCSV, pauseTask, probeAccountFolders, probeFolders, resumeTask, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, updateAccountCredentials, type Account, type TaskDetail as TaskDetailData } from '../api'
import { connectTaskWS, type TaskEvent } from '../ws'
import { StatusBadge } from '../components/StatusBadge'
import { useConfirm } from '../components/ConfirmProvider'
import { FolderMappingModal } from '../components/FolderMappingModal'
import { RunLogModal } from '../components/RunLogModal'
import { AccountErrorsModal } from '../components/AccountErrorsModal'
import { KerioImportModal } from '../components/KerioImportModal'
import { AccountCredentialsModal } from '../components/AccountCredentialsModal'
const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' }
@@ -57,6 +59,8 @@ function describeEvent(ev: TaskEvent): string {
}
case 'cancelled':
return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}`
case 'paused':
return `PAUSED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0} — resumable`
case 'error': {
const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : ''
return `ERROR #${d.account_id}${where}: ${d.error}`
@@ -64,7 +68,7 @@ function describeEvent(ev: TaskEvent): string {
case 'run_started':
return `RUN started (run #${d.run_id})`
case 'run_done':
return `RUN finished: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
return `RUN ${String(d.status ?? 'finished')}: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
default:
return JSON.stringify(ev.data)
}
@@ -79,6 +83,7 @@ export function TaskDetail({ id }: { id: number }) {
const [mapState, setMapState] = useState<{ src: string[]; dst: string[]; creds: typeof emptyAccount } | null>(null)
const [editMap, setEditMap] = useState<{
accId: number
label: string
src: string[]
dst: string[]
mapping: Record<string, string>
@@ -89,6 +94,9 @@ export function TaskDetail({ id }: { id: number }) {
const [live, setLive] = useState<Record<number, LiveProgress>>({})
const [showRuns, setShowRuns] = useState(false)
const [errorsFor, setErrorsFor] = useState<{ id: number; src_login: string } | null>(null)
const [kerioOpen, setKerioOpen] = useState(false)
const [credsFor, setCredsFor] = useState<Account | null>(null)
const [selected, setSelected] = useState<Set<number>>(new Set())
const fileInputRef = useRef<HTMLInputElement>(null)
function reload() {
@@ -157,7 +165,7 @@ export function TaskDetail({ id }: { id: number }) {
},
}
})
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || (ev.type === 'error' && d.folder == null))) {
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || ev.type === 'paused' || (ev.type === 'error' && d.folder == null))) {
// terminal/reset for this account — drop live overlay, fall back to DB
setLive((prev) => {
if (!(accId in prev)) return prev
@@ -168,7 +176,7 @@ export function TaskDetail({ id }: { id: number }) {
}
// Structural events refresh the persisted view; `progress` is covered by live state.
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'plan', 'task_broken'].includes(ev.type)) {
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'paused', 'plan', 'task_broken'].includes(ev.type)) {
reload()
}
}),
@@ -214,7 +222,7 @@ export function TaskDetail({ id }: { id: number }) {
}
}
async function onEditFolders(a: { id: number; src_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) {
async function onEditFolders(a: { id: number; src_login: string; dst_login: string; folder_mapping?: Record<string, string>; excluded_folders?: string[] }) {
setBusy('probe')
setError(null)
try {
@@ -228,6 +236,7 @@ export function TaskDetail({ id }: { id: number }) {
}
setEditMap({
accId: a.id,
label: a.dst_login && a.dst_login !== a.src_login ? `${a.src_login}${a.dst_login}` : a.src_login,
src: res.src.folders ?? [],
dst: res.dst.folders ?? [],
mapping: a.folder_mapping ?? {},
@@ -255,18 +264,38 @@ export function TaskDetail({ id }: { id: number }) {
}
}
function downloadCSV(name: string, content: string) {
const url = URL.createObjectURL(new Blob([content], { type: 'text/csv' }))
const a = document.createElement('a')
a.href = url
a.download = name
a.click()
URL.revokeObjectURL(url)
}
// Plain import: comma-separated, no header, src_login,src_pass,dst_login,dst_pass.
function downloadExampleCSV() {
const sample = [
const sample =
[
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
].join('\n') + '\n'
const url = URL.createObjectURL(new Blob([sample], { type: 'text/csv' }))
const a = document.createElement('a')
a.href = url
a.download = 'imap-copier-accounts-example.csv'
a.click()
URL.revokeObjectURL(url)
downloadCSV('imap-copier-accounts-example.csv', sample)
}
// Kerio Connect export: semicolon-separated with a Name;FullName;Description;Enable
// header. The password lives in Description; disabled rows and admin are skipped
// on import, and the domain is supplied in the import dialog.
function downloadKerioExampleCSV() {
const sample =
[
'Name;FullName;Description;Enable',
'alice;Alice Smith;SrcPass1;Yes',
'bob;Bob Jones;SrcPass2;Yes',
'carol;Carol White (disabled, skipped);SrcPass3;No',
].join('\n') + '\n'
downloadCSV('kerio-users-example.csv', sample)
}
async function onFileChosen(e: ChangeEvent<HTMLInputElement>) {
@@ -285,6 +314,35 @@ export function TaskDetail({ id }: { id: number }) {
}
}
async function onKerioImport(file: File, domain: string) {
setBusy('import')
setError(null)
try {
await importKerioCSV(id, file, domain)
setKerioOpen(false)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Kerio import failed')
} finally {
setBusy(null)
}
}
async function saveCredentials(body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) {
if (!credsFor) return
setBusy('add')
setError(null)
try {
await updateAccountCredentials(id, credsFor.id, body)
setCredsFor(null)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to save credentials')
} finally {
setBusy(null)
}
}
async function onDeleteAccount(accId: number, login: string) {
const ok = await confirm({
title: 'Remove account',
@@ -330,7 +388,8 @@ export function TaskDetail({ id }: { id: number }) {
setBusy('run')
setError(null)
try {
await runTask(id)
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
await runTask(id, ids.length ? ids : undefined)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to start run')
} finally {
@@ -338,6 +397,51 @@ export function TaskDetail({ id }: { id: number }) {
}
}
async function onPause() {
setBusy('run')
setError(null)
try {
await pauseTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to pause the run')
} finally {
setBusy(null)
}
}
async function onCancelRun() {
const ok = await confirm({
title: 'Cancel migration',
message: 'Stop the run and mark every unfinished account as cancelled? Copied messages are kept.',
confirmLabel: 'Cancel migration',
cancelLabel: 'Keep running',
danger: true,
})
if (!ok) return
setBusy('run')
setError(null)
try {
await cancelTask(id)
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to cancel the run')
} finally {
setBusy(null)
}
}
async function onResume() {
setBusy('run')
setError(null)
try {
await resumeTask(id)
reload()
} catch (err) {
setError(err instanceof Error ? err.message : 'Failed to resume the run')
} finally {
setBusy(null)
}
}
async function onSchedule(intervalSeconds: number) {
setError(null)
try {
@@ -364,7 +468,51 @@ export function TaskDetail({ id }: { id: number }) {
}
const { task, accounts } = data
const allTested = accounts.length > 0 && accounts.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
const isRunning = task.status === 'running'
// Accounts a pause left unfinished — what Resume picks up.
const pausedCount = accounts.filter((a) => a.status === 'paused').length
// A failed connection test is the entry point for fixing the credentials that
// caused it — an imported account is otherwise only deletable.
const testCell = (a: Account, status: string) =>
status === 'fail' && !isRunning && a.status !== 'running' ? (
<button
type="button"
className="badge-btn"
title="Edit credentials for this account"
onClick={() => setCredsFor(a)}
>
<StatusBadge status={status} />
</button>
) : (
<StatusBadge status={status} />
)
// A row is selectable only when both connection tests pass and no run is live.
const selectableIds = accounts
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
.map((a) => a.id)
const selectableSet = new Set(selectableIds)
// Effective set: the checked accounts, or all accounts when nothing is checked.
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
const runReady =
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
const allSelectableChecked =
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
function toggleOne(accId: number, checked: boolean) {
setSelected((prev) => {
const next = new Set(prev)
if (checked) next.add(accId)
else next.delete(accId)
return next
})
}
function toggleAll(checked: boolean) {
setSelected(checked ? new Set(selectableIds) : new Set())
}
// Prefer live (WS) copied/skipped over the DB values, which only advance per
// folder — so the summary moves in real time during a large folder.
const totals = accounts.reduce(
@@ -416,10 +564,43 @@ export function TaskDetail({ id }: { id: number }) {
<button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}>
{busy === 'test' ? 'Testing…' : 'Test connections'}
</button>
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
{busy === 'run' ? 'Starting…' : 'Run migration'}
{isRunning ? (
<>
<button className="btn" onClick={onPause} disabled={busy !== null}>
{busy === 'run' ? 'Stopping…' : 'Pause'}
</button>
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
<button className="btn btn-danger" onClick={onCancelRun} disabled={busy !== null}>
Cancel
</button>
<span className="hint">pause keeps the unfinished accounts resumable</span>
</>
) : (
<>
{pausedCount > 0 && (
<button className="btn btn-primary" onClick={onResume} disabled={busy !== null}>
{busy === 'run' ? 'Resuming…' : `Resume (${pausedCount})`}
</button>
)}
<button
className={pausedCount > 0 ? 'btn' : 'btn btn-primary'}
onClick={onRun}
disabled={busy !== null || !runReady}
>
{busy === 'run'
? 'Starting…'
: effectiveSelected.length > 0
? `Run selected (${effectiveSelected.length})`
: 'Run migration'}
</button>
</>
)}
{!isRunning && !runReady && accounts.length > 0 && (
<span className="hint">
{effectiveSelected.length > 0
? 'selected accounts must pass both connection tests'
: 'run unlocks once every account tests OK on both sides'}
</span>
)}
</div>
<div className="sched-row">
<label htmlFor="sched">Schedule</label>
@@ -500,6 +681,7 @@ export function TaskDetail({ id }: { id: number }) {
<div className="divider-label">or bulk import</div>
<div className="upload-row">
<div className="upload-item">
<label className={`btn file-btn${busy !== null ? ' is-disabled' : ''}`}>
{busy === 'import' ? 'Importing…' : 'Upload CSV'}
<input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} />
@@ -508,6 +690,15 @@ export function TaskDetail({ id }: { id: number }) {
download example.csv
</button>
</div>
<div className="upload-item">
<button type="button" className="btn" onClick={() => setKerioOpen(true)} disabled={busy !== null}>
Import from Kerio
</button>
<button type="button" className="link-btn" onClick={downloadKerioExampleCSV}>
download kerio example.csv
</button>
</div>
</div>
</div>
<div className="panel">
@@ -538,6 +729,18 @@ export function TaskDetail({ id }: { id: number }) {
<table className="tbl">
<thead>
<tr>
<th className="chk-col">
<input
type="checkbox"
aria-label="Select all accounts"
checked={allSelectableChecked}
ref={(el) => {
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
}}
disabled={isRunning || selectableIds.length === 0}
onChange={(e) => toggleAll(e.target.checked)}
/>
</th>
<th>Account</th>
<th>Src test</th>
<th>Dst test</th>
@@ -552,11 +755,20 @@ export function TaskDetail({ id }: { id: number }) {
<tbody>
{accounts.length === 0 ? (
<tr className="empty-row">
<td colSpan={9}>no accounts yet add one or import a CSV above</td>
<td colSpan={10}>no accounts yet add one or import a CSV above</td>
</tr>
) : (
accounts.map((a) => (
<tr key={a.id}>
<td className="chk-col">
<input
type="checkbox"
aria-label={`Select ${a.src_login}`}
checked={selected.has(a.id)}
disabled={isRunning || !selectableSet.has(a.id)}
onChange={(e) => toggleOne(a.id, e.target.checked)}
/>
</td>
<td>
<div className="acct-ident">
<span>{a.src_login}</span>
@@ -570,12 +782,8 @@ export function TaskDetail({ id }: { id: number }) {
</div>
)}
</td>
<td>
<StatusBadge status={a.test_src_status} />
</td>
<td>
<StatusBadge status={a.test_dst_status} />
</td>
<td>{testCell(a, a.test_src_status)}</td>
<td>{testCell(a, a.test_dst_status)}</td>
<td>
<StatusBadge status={a.status} />
</td>
@@ -629,6 +837,18 @@ export function TaskDetail({ id }: { id: number }) {
</td>
<td className="num-cell">
<div className="row-actions">
{(() => {
const mapped =
Object.keys(a.folder_mapping ?? {}).length > 0 ||
(a.excluded_folders?.length ?? 0) > 0
return mapped ? (
<span
className="map-dot"
title="Folders already mapped for this account"
aria-label="Folders mapped"
/>
) : null
})()}
{a.status !== 'running' && data?.task.status !== 'running' && (
<button
type="button"
@@ -671,6 +891,11 @@ export function TaskDetail({ id }: { id: number }) {
dstFolders={mapState.dst}
initialMapping={task.folder_mapping ?? {}}
initialExcluded={[]}
accountLabel={
mapState.creds.dst_login && mapState.creds.dst_login !== mapState.creds.src_login
? `${mapState.creds.src_login}${mapState.creds.dst_login}`
: mapState.creds.src_login
}
onCancel={() => setMapState(null)}
onConfirm={confirmMapping}
/>
@@ -683,12 +908,26 @@ export function TaskDetail({ id }: { id: number }) {
dstFolders={editMap.dst}
initialMapping={editMap.mapping}
initialExcluded={editMap.excluded}
accountLabel={editMap.label}
onCancel={() => setEditMap(null)}
onConfirm={saveEditMapping}
/>
)}
<RunLogModal taskId={id} open={showRuns} onClose={() => setShowRuns(false)} />
<AccountErrorsModal taskId={id} account={errorsFor} onClose={() => setErrorsFor(null)} />
<KerioImportModal
open={kerioOpen}
busy={busy === 'import'}
onClose={() => setKerioOpen(false)}
onSubmit={onKerioImport}
/>
<AccountCredentialsModal
open={credsFor !== null}
busy={busy === 'add'}
account={credsFor}
onClose={() => setCredsFor(null)}
onSubmit={saveCredentials}
/>
</>
)
}