Compare commits
21
Commits
b076ac92d6
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0b12d2ed2d
|
||
|
|
039ac2f1da
|
||
|
|
c077b368f3
|
||
|
|
76ada57dd7
|
||
|
|
bb3635e517
|
||
|
|
8bc7ff026d
|
||
|
|
94fb410c59
|
||
|
|
6bf3a6c4ca
|
||
|
|
c741cd19a0
|
||
|
|
b6e68bdd90
|
||
|
|
d125320667
|
||
|
|
b352cda166
|
||
|
|
5d296c39b1
|
||
|
|
fc335dee3f
|
||
|
|
b439eee164
|
||
|
|
2e5036aad5
|
||
|
|
599c66189c
|
||
|
|
315e56046b
|
||
|
|
692a83a468
|
||
|
|
6d8471efdd
|
||
|
|
3eac1cea7d
|
@@ -10,3 +10,4 @@
|
||||
# local cache of the impeccable design hook
|
||||
.impeccable/
|
||||
**/.impeccable/
|
||||
*.csv
|
||||
@@ -4,6 +4,16 @@ Single-binary Go server (embeds the React SPA) that copies IMAP mailboxes
|
||||
between a source and a destination account. Non-destructive (copy only,
|
||||
never deletes), deduplicated by Message-ID, resumable/idempotent re-runs.
|
||||
|
||||
## Screenshots
|
||||
|
||||
Task list — create migration jobs and track their status:
|
||||
|
||||

|
||||
|
||||
Task detail — run control, per-account credentials, live event log and progress:
|
||||
|
||||

|
||||
|
||||
## Quick start (plain HTTP on :80)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -8,7 +8,7 @@ services:
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U imap"]
|
||||
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
@@ -35,7 +35,7 @@ services:
|
||||
image: caddy:2-alpine
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.imap-copier.rule=Host(`imap-copier.amegahost.kz`)"
|
||||
- "traefik.http.routers.imap-copier.rule=Host(`${DOMAIN}`)"
|
||||
- "traefik.http.routers.imap-copier.entrypoints=https"
|
||||
- "traefik.http.routers.imap-copier.tls=true"
|
||||
- "traefik.http.routers.imap-copier.tls.certresolver=letsEncrypt"
|
||||
|
||||
+1
-1
@@ -8,7 +8,7 @@ services:
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U imap"]
|
||||
test: ["CMD-SHELL", "pg_isready -U imap -d imapcopier"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 326 KiB |
@@ -0,0 +1,546 @@
|
||||
# Selective Account Run Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Let the user run a migration for only the checkbox-selected accounts; with no selection, run all (unchanged).
|
||||
|
||||
**Architecture:** A pure `selectAccounts` filter narrows the orchestrator's account slice before the test gate; the run HTTP handler parses an optional `account_ids` body and passes it through. The accounts table gains row checkboxes and a "select all" header, and the single "Run migration" button becomes "Run selected (N)" when a selection exists. Copy is already idempotent (dedup by message key), so a re-run of selected accounts skips copied messages and delivers the missing tail.
|
||||
|
||||
**Tech Stack:** Go 1.x (net/http, encoding/json, standard testing), React + TypeScript + Vite, existing REST/WS client.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Backend module path: `github.com/vasyansk/imap-copier`.
|
||||
- `Run` with an empty/nil account-ID set MUST behave exactly as today (all accounts) — the scheduler depends on it.
|
||||
- Preserve existing `handleRun` error mapping: `ErrNotTested` → 409 "accounts must pass connection tests first"; `ErrAlreadyRunning` → 409 "task is already running".
|
||||
- Frontend has no unit-test harness; frontend tasks verify via `npm run build` + `tsc --noEmit` + manual app check.
|
||||
- Follow existing code style: no new dependencies.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Orchestrator account filter + `Run` signature
|
||||
|
||||
**Files:**
|
||||
- Modify: `internal/orchestrator/orchestrator.go`
|
||||
- Test: `internal/orchestrator/orchestrator_test.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `func selectAccounts(accs []store.Account, ids []int64) []store.Account` — returns `accs` unchanged when `ids` is empty/nil; otherwise returns only the accounts whose `ID` is in `ids`, preserving input order.
|
||||
- `var ErrNoAccountsSelected = errors.New("no matching accounts selected")`
|
||||
- `func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)` — new trailing `accountIDs` param.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `internal/orchestrator/orchestrator_test.go`:
|
||||
|
||||
```go
|
||||
func TestSelectAccounts(t *testing.T) {
|
||||
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
|
||||
|
||||
if got := selectAccounts(accs, nil); len(got) != 3 {
|
||||
t.Fatalf("nil ids must return all, got %d", len(got))
|
||||
}
|
||||
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
|
||||
t.Fatalf("empty ids must return all, got %d", len(got))
|
||||
}
|
||||
|
||||
got := selectAccounts(accs, []int64{3, 1})
|
||||
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
|
||||
t.Fatalf("must keep matching ids in input order, got %+v", got)
|
||||
}
|
||||
|
||||
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
|
||||
t.Fatalf("unknown ids must yield empty, got %d", len(got))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
|
||||
Expected: FAIL — `undefined: selectAccounts`.
|
||||
|
||||
- [ ] **Step 3: Add the filter helper and sentinel error**
|
||||
|
||||
In `internal/orchestrator/orchestrator.go`, add the sentinel next to the existing `var ErrNotTested`/`ErrAlreadyRunning`:
|
||||
|
||||
```go
|
||||
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
|
||||
```
|
||||
|
||||
Add the helper (place it near `gateOK`):
|
||||
|
||||
```go
|
||||
// selectAccounts narrows accs to those whose ID is in ids, preserving input
|
||||
// order. An empty or nil ids means "all accounts" — the scheduler and the
|
||||
// unfiltered manual run rely on this.
|
||||
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
|
||||
if len(ids) == 0 {
|
||||
return accs
|
||||
}
|
||||
want := make(map[int64]struct{}, len(ids))
|
||||
for _, id := range ids {
|
||||
want[id] = struct{}{}
|
||||
}
|
||||
out := make([]store.Account, 0, len(ids))
|
||||
for _, a := range accs {
|
||||
if _, ok := want[a.ID]; ok {
|
||||
out = append(out, a)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `go test ./internal/orchestrator/ -run TestSelectAccounts -v`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Thread `accountIDs` through `Run`**
|
||||
|
||||
In `internal/orchestrator/orchestrator.go`, change the `Run` signature and apply the filter after loading accounts. Replace:
|
||||
|
||||
```go
|
||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
|
||||
task, err := o.store.GetTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if !gateOK(accs) {
|
||||
return 0, ErrNotTested
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```go
|
||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
|
||||
task, err := o.store.GetTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
accs = selectAccounts(accs, accountIDs)
|
||||
// A non-empty request that matched nothing is a client error, distinct
|
||||
// from "not tested".
|
||||
if len(accountIDs) > 0 && len(accs) == 0 {
|
||||
return 0, ErrNoAccountsSelected
|
||||
}
|
||||
if !gateOK(accs) {
|
||||
return 0, ErrNotTested
|
||||
}
|
||||
```
|
||||
|
||||
The rest of `Run` (endpoints, `TryMarkTaskRunning`, `CreateRun`, `go o.runAll(...)`) is unchanged — it already operates over the `accs` slice.
|
||||
|
||||
- [ ] **Step 6: Update the two `Run` call sites so the package compiles**
|
||||
|
||||
In `internal/scheduler/scheduler.go`, change:
|
||||
|
||||
```go
|
||||
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```go
|
||||
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
|
||||
```
|
||||
|
||||
In `internal/httpapi/run.go`, inside `handleRun`, change:
|
||||
|
||||
```go
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual")
|
||||
```
|
||||
|
||||
to (IDs wired in Task 2; for now pass `nil` to keep it compiling):
|
||||
|
||||
```go
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Verify the whole backend builds and tests pass**
|
||||
|
||||
Run: `go build ./... && go test ./internal/orchestrator/ ./internal/scheduler/ -v`
|
||||
Expected: build succeeds; all tests PASS.
|
||||
|
||||
- [ ] **Step 8: Commit**
|
||||
|
||||
```bash
|
||||
git add internal/orchestrator/orchestrator.go internal/orchestrator/orchestrator_test.go internal/scheduler/scheduler.go internal/httpapi/run.go
|
||||
git commit -m "Add selectAccounts filter and accountIDs param to orchestrator.Run"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: `handleRun` parses optional `account_ids`
|
||||
|
||||
**Files:**
|
||||
- Modify: `internal/httpapi/run.go`
|
||||
- Test: `internal/httpapi/run_test.go`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `orchestrator.Run(ctx, taskID, "manual", accountIDs)`, `orchestrator.ErrNoAccountsSelected` (Task 1).
|
||||
- Produces:
|
||||
- `func parseRunAccountIDs(r *http.Request) ([]int64, error)` — returns `nil` for an empty body; otherwise decodes `{"account_ids":[...]}` and returns the slice (possibly empty). Returns an error on malformed JSON.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Add to `internal/httpapi/run_test.go` (the `strings` import is already present):
|
||||
|
||||
```go
|
||||
func TestParseRunAccountIDs(t *testing.T) {
|
||||
// empty body => nil (run all)
|
||||
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
|
||||
ids, err := parseRunAccountIDs(req)
|
||||
if err != nil || ids != nil {
|
||||
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
|
||||
}
|
||||
|
||||
// explicit selection
|
||||
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
|
||||
ids, err = parseRunAccountIDs(req)
|
||||
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
|
||||
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
|
||||
}
|
||||
|
||||
// malformed JSON => error
|
||||
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
|
||||
if _, err := parseRunAccountIDs(req); err == nil {
|
||||
t.Fatal("malformed body must error")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
|
||||
Expected: FAIL — `undefined: parseRunAccountIDs`.
|
||||
|
||||
- [ ] **Step 3: Implement `parseRunAccountIDs` and wire it into `handleRun`**
|
||||
|
||||
In `internal/httpapi/run.go`, add the imports `encoding/json` and `io` to the existing import block, then add the helper:
|
||||
|
||||
```go
|
||||
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
|
||||
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
|
||||
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
|
||||
raw, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(bytes.TrimSpace(raw)) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
var body struct {
|
||||
AccountIDs []int64 `json:"account_ids"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return body.AccountIDs, nil
|
||||
}
|
||||
```
|
||||
|
||||
Add `"bytes"` to the import block as well (used by `bytes.TrimSpace`).
|
||||
|
||||
Then replace the body of `handleRun`'s run call. Change:
|
||||
|
||||
```go
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual", nil)
|
||||
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
|
||||
http.Error(w, "task is already running", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
to:
|
||||
|
||||
```go
|
||||
accountIDs, err := parseRunAccountIDs(r)
|
||||
if err != nil {
|
||||
http.Error(w, "bad request body", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
|
||||
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
|
||||
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
if errors.Is(err, orchestrator.ErrAlreadyRunning) {
|
||||
http.Error(w, "task is already running", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `go test ./internal/httpapi/ -run TestParseRunAccountIDs -v`
|
||||
Expected: PASS.
|
||||
|
||||
- [ ] **Step 5: Verify the whole backend builds and the httpapi suite passes**
|
||||
|
||||
Run: `go build ./... && go test ./internal/httpapi/ -v`
|
||||
Expected: build succeeds; all tests PASS.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add internal/httpapi/run.go internal/httpapi/run_test.go
|
||||
git commit -m "Parse optional account_ids in run handler"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Frontend `runTask` accepts account IDs
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/src/api.ts:141`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `runTask(id: number, accountIds?: number[]) => Promise<...>` — sends `{account_ids}` JSON only when a non-empty list is given; otherwise a bare POST (all accounts).
|
||||
|
||||
- [ ] **Step 1: Update `runTask`**
|
||||
|
||||
In `web/src/api.ts`, replace:
|
||||
|
||||
```ts
|
||||
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```ts
|
||||
export const runTask = (id: number, accountIds?: number[]) =>
|
||||
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify typecheck**
|
||||
|
||||
Run: `cd web && npx tsc --noEmit`
|
||||
Expected: no errors from `api.ts` (pre-existing warnings elsewhere, if any, are unrelated).
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add web/src/api.ts
|
||||
git commit -m "runTask accepts optional accountIds"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Accounts table checkboxes + selective Run button
|
||||
|
||||
**Files:**
|
||||
- Modify: `web/src/pages/TaskDetail.tsx`
|
||||
- Modify: `web/src/app.css`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `runTask(id, accountIds?)` (Task 3).
|
||||
- Produces: no exported interface; internal UI state only.
|
||||
|
||||
- [ ] **Step 1: Add selection state and helpers**
|
||||
|
||||
In `web/src/pages/TaskDetail.tsx`, add a state hook next to the other `useState` declarations (near line 90):
|
||||
|
||||
```tsx
|
||||
const [selected, setSelected] = useState<Set<number>>(new Set())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Compute selectable rows and the effective run set**
|
||||
|
||||
After the existing `const { task, accounts } = data` / `allTested` block (near line 369), add:
|
||||
|
||||
```tsx
|
||||
const isRunning = task.status === 'running'
|
||||
// A row is selectable only when both connection tests pass and no run is live.
|
||||
const selectableIds = accounts
|
||||
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||
.map((a) => a.id)
|
||||
const selectableSet = new Set(selectableIds)
|
||||
// Effective set: the checked accounts, or all accounts when nothing is checked.
|
||||
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
|
||||
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
|
||||
const runReady =
|
||||
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||
const allSelectableChecked =
|
||||
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
|
||||
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
|
||||
|
||||
function toggleOne(accId: number, checked: boolean) {
|
||||
setSelected((prev) => {
|
||||
const next = new Set(prev)
|
||||
if (checked) next.add(accId)
|
||||
else next.delete(accId)
|
||||
return next
|
||||
})
|
||||
}
|
||||
|
||||
function toggleAll(checked: boolean) {
|
||||
setSelected(checked ? new Set(selectableIds) : new Set())
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Make `onRun` send the selection**
|
||||
|
||||
Replace the existing `onRun` function (near line 331):
|
||||
|
||||
```tsx
|
||||
async function onRun() {
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
await runTask(id)
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```tsx
|
||||
async function onRun() {
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
|
||||
await runTask(id, ids.length ? ids : undefined)
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update the Run button label and gate**
|
||||
|
||||
Replace the run button (near line 421):
|
||||
|
||||
```tsx
|
||||
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
|
||||
{busy === 'run' ? 'Starting…' : 'Run migration'}
|
||||
</button>
|
||||
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
|
||||
```
|
||||
|
||||
with:
|
||||
|
||||
```tsx
|
||||
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !runReady || isRunning}>
|
||||
{busy === 'run'
|
||||
? 'Starting…'
|
||||
: effectiveSelected.length > 0
|
||||
? `Run selected (${effectiveSelected.length})`
|
||||
: 'Run migration'}
|
||||
</button>
|
||||
{!runReady && accounts.length > 0 && (
|
||||
<span className="hint">
|
||||
{effectiveSelected.length > 0
|
||||
? 'selected accounts must pass both connection tests'
|
||||
: 'run unlocks once every account tests OK on both sides'}
|
||||
</span>
|
||||
)}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add the checkbox header column**
|
||||
|
||||
In the accounts table `<thead><tr>` (near line 542), add a leading `<th>` before `<th>Account</th>`:
|
||||
|
||||
```tsx
|
||||
<th className="chk-col">
|
||||
<input
|
||||
type="checkbox"
|
||||
aria-label="Select all accounts"
|
||||
checked={allSelectableChecked}
|
||||
ref={(el) => {
|
||||
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
|
||||
}}
|
||||
disabled={isRunning || selectableIds.length === 0}
|
||||
onChange={(e) => toggleAll(e.target.checked)}
|
||||
/>
|
||||
</th>
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Add the per-row checkbox cell and fix the empty-row colspan**
|
||||
|
||||
In the empty-row branch (near line 557), change `colSpan={9}` to `colSpan={10}`.
|
||||
|
||||
In the account `<tr>` map (near line 561), add a leading `<td>` as the first child, before the `<td>` with `acct-ident`:
|
||||
|
||||
```tsx
|
||||
<td className="chk-col">
|
||||
<input
|
||||
type="checkbox"
|
||||
aria-label={`Select ${a.src_login}`}
|
||||
checked={selected.has(a.id)}
|
||||
disabled={isRunning || !selectableSet.has(a.id)}
|
||||
onChange={(e) => toggleOne(a.id, e.target.checked)}
|
||||
/>
|
||||
</td>
|
||||
```
|
||||
|
||||
- [ ] **Step 7: Add checkbox column CSS**
|
||||
|
||||
In `web/src/app.css`, append:
|
||||
|
||||
```css
|
||||
.tbl .chk-col {
|
||||
width: 32px;
|
||||
text-align: center;
|
||||
padding-right: 0;
|
||||
}
|
||||
.tbl .chk-col input[type='checkbox'] {
|
||||
cursor: pointer;
|
||||
}
|
||||
.tbl .chk-col input[type='checkbox']:disabled {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.4;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 8: Build and typecheck**
|
||||
|
||||
Run: `cd web && npx tsc --noEmit && npm run build`
|
||||
Expected: typecheck clean (any pre-existing oxlint/fast-refresh warnings are unrelated); Vite build succeeds.
|
||||
|
||||
- [ ] **Step 9: Manual verification in the running app**
|
||||
|
||||
Start the app, open a task with ≥2 accounts that have passed both tests:
|
||||
1. With nothing checked, the button reads "Run migration" and runs all accounts (confirm `account_started` WS events for every account).
|
||||
2. Check a subset; the button reads "Run selected (N)". Run it; confirm only the checked accounts start, and the unchecked accounts' Copied/Skipped/Status are untouched.
|
||||
3. Confirm checkboxes for untested accounts are disabled, and the header checkbox toggles only the selectable rows.
|
||||
|
||||
- [ ] **Step 10: Commit**
|
||||
|
||||
```bash
|
||||
git add web/src/pages/TaskDetail.tsx web/src/app.css
|
||||
git commit -m "Add account selection checkboxes and selective run button"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notes for the executor
|
||||
|
||||
- Tasks 1 and 2 are backend and must land in order (Task 2 depends on the new `Run` signature and `ErrNoAccountsSelected`). Task 3 and 4 are frontend; Task 4 depends on Task 3.
|
||||
- The orchestrator's `Run` is integration-heavy (spawns goroutines, needs a real store/hub), so it is not unit-tested directly; the pure `selectAccounts` filter and the `parseRunAccountIDs` helper carry the backend test coverage, and Task 4 Step 9 covers the end-to-end behavior manually.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Selective Account Run — Design
|
||||
|
||||
**Date:** 2026-07-08
|
||||
**Status:** Approved (pending spec review)
|
||||
|
||||
## Problem
|
||||
|
||||
When a migration run fails for some accounts (e.g. transient `use of closed
|
||||
network connection`), the user wants to re-run **only the affected accounts**,
|
||||
not the whole task.
|
||||
|
||||
An earlier idea — per-error "retry" buttons inside the errors modal — was
|
||||
rejected: on a `closed network connection` the copy aborts the whole folder
|
||||
(`copy.go`, the `net.ErrClosed` branch), so the undelivered tail of the folder
|
||||
never appears in the error list. A per-UID retry would replay only the few
|
||||
visible errors and leave the invisible tail uncopied, creating a false sense of
|
||||
completion.
|
||||
|
||||
The correct primitive is a **re-run of the selected accounts**. Copy is
|
||||
idempotent — `IsMigrated`/`MarkMigrated` dedup by message key — so a re-run
|
||||
skips everything already copied and delivers only the missing tail, covering the
|
||||
network-drop case fully.
|
||||
|
||||
## Solution
|
||||
|
||||
Add checkboxes to the accounts table. The main "Run migration" button runs the
|
||||
selected accounts; with no selection it runs all accounts (current behavior,
|
||||
preserved).
|
||||
|
||||
## Backend
|
||||
|
||||
### `orchestrator.Run`
|
||||
|
||||
Change signature to:
|
||||
|
||||
```go
|
||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error)
|
||||
```
|
||||
|
||||
- `accountIDs == nil || len == 0` → all accounts (unchanged behavior).
|
||||
- Otherwise, filter the `ListAccountsByTask` result down to the requested IDs
|
||||
**before** `gateOK`, so only the selected accounts are gate-checked and run.
|
||||
- IDs not belonging to the task are silently dropped by the filter (defense in
|
||||
depth; the handler also validates).
|
||||
- If the filter yields zero accounts, return `ErrNotTested` (nothing runnable) —
|
||||
reuses the existing 409 path; the handler maps empty selection separately (see
|
||||
below).
|
||||
|
||||
Call sites updated:
|
||||
- `scheduler.go` → `Run(ctx, id, "scheduled", nil)` (always all accounts).
|
||||
- `run.go` (`handleRun`) → passes the parsed IDs.
|
||||
|
||||
`runAll`, `runAccount`, counters, `FinishRun`, and task-status reconciliation are
|
||||
unchanged: they already operate over whatever `accs` slice they are handed.
|
||||
|
||||
### `handleRun`
|
||||
|
||||
- Parse an **optional** JSON body `{"account_ids": [1,2,3]}`. Empty/absent body
|
||||
→ `nil` → all accounts.
|
||||
- Validate parsed IDs against the task's accounts; if a non-empty selection
|
||||
resolves to zero valid IDs → `400 Bad Request`.
|
||||
- Existing error mapping preserved: `ErrNotTested` → 409 "accounts must pass
|
||||
connection tests first"; `ErrAlreadyRunning` → 409.
|
||||
|
||||
## Frontend
|
||||
|
||||
### Accounts table (`TaskDetail.tsx`)
|
||||
|
||||
- New leading column with a per-row checkbox; a "select all" checkbox in the
|
||||
header (selects only selectable rows).
|
||||
- State: `selected: Set<number>` of account IDs.
|
||||
- A row checkbox is disabled when the account has not passed **both** tests
|
||||
(`test_src_status`/`test_dst_status` !== 'ok') or the task is `running`.
|
||||
- "select all" reflects indeterminate/checked state over the selectable rows.
|
||||
|
||||
### Run button
|
||||
|
||||
- With a non-empty selection: label "Run selected (N)"; calls
|
||||
`runTask(id, [...selected])`.
|
||||
- With no selection: label "Run migration"; calls `runTask(id)` (all).
|
||||
- Enable gate computed over the **effective set** (selected, or all when none
|
||||
selected): every account in that set must be tested OK, and the task must not
|
||||
be `running`.
|
||||
- Side benefit: one untested account no longer blocks running the ready ones —
|
||||
select only the ready accounts and run them.
|
||||
|
||||
### `api.ts`
|
||||
|
||||
```ts
|
||||
export const runTask = (id: number, accountIds?: number[]) =>
|
||||
api(`/api/tasks/${id}/run`, accountIds?.length
|
||||
? jsonBody({ account_ids: accountIds })
|
||||
: { method: 'POST' })
|
||||
```
|
||||
|
||||
## Out of scope (YAGNI)
|
||||
|
||||
- The errors modal (`AccountErrorsModal`) is untouched; no retry buttons there.
|
||||
- No new "partial" task status. After a partial run the task status reflects the
|
||||
last run's outcome over the accounts that ran — same as today.
|
||||
- No selection persistence across reloads.
|
||||
|
||||
## Testing
|
||||
|
||||
- **Backend unit:** `orchestrator.Run` with a subset filters correctly; empty
|
||||
filter runs all; unknown IDs dropped; empty-after-filter → `ErrNotTested`.
|
||||
- **Backend handler:** `handleRun` parses body, validates IDs, maps empty
|
||||
selection → 400, absent body → all.
|
||||
- **Frontend:** manual verification in the running app — select a subset, run,
|
||||
confirm only those accounts start (WS `account_started` events), and the
|
||||
unselected accounts' counters/status are untouched.
|
||||
@@ -1,9 +1,11 @@
|
||||
package csvimport
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/csv"
|
||||
"fmt"
|
||||
"io"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -52,3 +54,110 @@ func Parse(r io.Reader) ([]Row, error) {
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
// Kerio Connect exports users as a semicolon-separated file whose first line is
|
||||
// a header. Only these columns matter; the rest (quotas, last login, …) is
|
||||
// ignored. The file carries no domain — the caller supplies it.
|
||||
const (
|
||||
kerioColName = 0 // login without the domain part
|
||||
kerioColDescription = 2 // Kerio keeps the plaintext password here
|
||||
kerioColEnable = 3 // "Yes" / "No"
|
||||
kerioMinColumns = 4
|
||||
)
|
||||
|
||||
// domainRe accepts a bare DNS domain: labels of alphanumerics/hyphens with at
|
||||
// least one dot and no scheme, user part, or whitespace.
|
||||
var domainRe = regexp.MustCompile(`^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$`)
|
||||
|
||||
func normalizeDomain(domain string) (string, error) {
|
||||
d := strings.ToLower(strings.TrimSpace(domain))
|
||||
if d == "" {
|
||||
return "", fmt.Errorf("domain is required")
|
||||
}
|
||||
if !domainRe.MatchString(d) {
|
||||
return "", fmt.Errorf("invalid domain %q", domain)
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// skipBOM consumes a leading UTF-8 byte-order mark, which Kerio writes into its
|
||||
// exports and encoding/csv would otherwise glue onto the first header field.
|
||||
func skipBOM(br *bufio.Reader) error {
|
||||
b, err := br.Peek(3)
|
||||
if err != nil && err != io.EOF {
|
||||
return err
|
||||
}
|
||||
if len(b) == 3 && b[0] == 0xEF && b[1] == 0xBB && b[2] == 0xBF {
|
||||
_, _ = br.Discard(3)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParseKerio reads a Kerio Connect user export and maps every enabled, non-admin
|
||||
// account onto both sides of a migration: the login and password are identical
|
||||
// on source and destination, only the server differs. Rows for disabled accounts
|
||||
// and the built-in admin are skipped.
|
||||
func ParseKerio(r io.Reader, domain string) ([]Row, error) {
|
||||
d, err := normalizeDomain(domain)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
br := bufio.NewReader(r)
|
||||
if err := skipBOM(br); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
cr := csv.NewReader(br)
|
||||
cr.Comma = ';'
|
||||
cr.FieldsPerRecord = -1 // проверяем сами
|
||||
cr.LazyQuotes = true // FullName нередко содержит одиночную кавычку
|
||||
|
||||
header, err := cr.Read()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot read header: %w", err)
|
||||
}
|
||||
if len(header) < kerioMinColumns || !strings.EqualFold(strings.TrimSpace(header[kerioColName]), "Name") {
|
||||
return nil, fmt.Errorf("not a Kerio export: expected a header starting with Name;FullName;Description;Enable")
|
||||
}
|
||||
|
||||
var rows []Row
|
||||
seen := map[string]bool{}
|
||||
for {
|
||||
rec, err := cr.Read()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
line, _ := cr.FieldPos(0)
|
||||
if len(rec) == 1 && strings.TrimSpace(rec[0]) == "" {
|
||||
continue
|
||||
}
|
||||
if len(rec) < kerioMinColumns {
|
||||
return nil, fmt.Errorf("line %d: expected at least %d columns, got %d", line, kerioMinColumns, len(rec))
|
||||
}
|
||||
name := strings.ToLower(strings.TrimSpace(rec[kerioColName]))
|
||||
if name == "" {
|
||||
return nil, fmt.Errorf("line %d: Name is empty", line)
|
||||
}
|
||||
if !strings.EqualFold(strings.TrimSpace(rec[kerioColEnable]), "Yes") || name == "admin" {
|
||||
continue
|
||||
}
|
||||
pass := strings.TrimSpace(rec[kerioColDescription])
|
||||
if pass == "" {
|
||||
return nil, fmt.Errorf("line %d: no password in the Description column for %q", line, name)
|
||||
}
|
||||
if seen[name] {
|
||||
return nil, fmt.Errorf("line %d: duplicate Name %q", line, name)
|
||||
}
|
||||
seen[name] = true
|
||||
login := name + "@" + d
|
||||
rows = append(rows, Row{SrcLogin: login, SrcPass: pass, DstLogin: login, DstPass: pass})
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil, fmt.Errorf("no enabled accounts found in the export")
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
@@ -49,3 +49,100 @@ func TestParseZeroRowsErrors(t *testing.T) {
|
||||
t.Fatal("expected error when no rows parsed")
|
||||
}
|
||||
}
|
||||
|
||||
const kerioHeader = "Name;FullName;Description;Enable;DataSource;Authentication;Role;Groups;MailAddress\n"
|
||||
|
||||
func TestParseKerioOK(t *testing.T) {
|
||||
in := kerioHeader +
|
||||
"j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n" +
|
||||
"k.smith;Kim Smith;SrcPass22;Yes;Internal;Internal;No rights;all;k.smith\n"
|
||||
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if len(rows) != 2 {
|
||||
t.Fatalf("want 2 rows, got %d: %+v", len(rows), rows)
|
||||
}
|
||||
want := Row{
|
||||
SrcLogin: "j.doe@example.test", SrcPass: "SrcPass11",
|
||||
DstLogin: "j.doe@example.test", DstPass: "SrcPass11",
|
||||
}
|
||||
if rows[0] != want {
|
||||
t.Fatalf("row 0: got %+v, want %+v", rows[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioSkipsDisabledAndAdmin(t *testing.T) {
|
||||
in := kerioHeader +
|
||||
"admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n" +
|
||||
"o.disabled;;OffPass111;No;Internal;Internal;No rights;all;o.disabled\n" +
|
||||
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" {
|
||||
t.Fatalf("only the enabled non-admin row must survive, got %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioStripsBOM(t *testing.T) {
|
||||
in := "\ufeff" + kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||
rows, err := ParseKerio(strings.NewReader(in), "example.test")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if len(rows) != 1 {
|
||||
t.Fatalf("want 1 row, got %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioRejectsMissingHeader(t *testing.T) {
|
||||
in := "j.doe;Jane Doe;SrcPass11;Yes;Internal;Internal;No rights;all;j.doe\n"
|
||||
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||
t.Fatal("a file without the Kerio header must error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioRejectsBadDomain(t *testing.T) {
|
||||
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||
for _, domain := range []string{"", " ", "@example.test", "exam ple.test", "example", "info@example.test"} {
|
||||
if _, err := ParseKerio(strings.NewReader(in), domain); err == nil {
|
||||
t.Fatalf("domain %q must be rejected", domain)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioTrimsAndLowercasesDomain(t *testing.T) {
|
||||
in := kerioHeader + "info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n"
|
||||
rows, err := ParseKerio(strings.NewReader(in), " Example.TEST ")
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if rows[0].SrcLogin != "info@example.test" {
|
||||
t.Fatalf("domain must be trimmed and lowercased, got %q", rows[0].SrcLogin)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioRejectsEmptyPassword(t *testing.T) {
|
||||
in := kerioHeader + "info;;;Yes;Internal;Internal;No rights;all;info\n"
|
||||
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||
t.Fatal("an enabled account without a password must error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioRejectsDuplicateName(t *testing.T) {
|
||||
in := kerioHeader +
|
||||
"info;;InfoPass11;Yes;Internal;Internal;No rights;all;info\n" +
|
||||
"info;;aaaaaaAa1;Yes;Internal;Internal;No rights;all;info\n"
|
||||
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||
t.Fatal("duplicate Name must error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseKerioZeroRowsErrors(t *testing.T) {
|
||||
in := kerioHeader + "admin;;AdminPass1;Yes;Internal;Internal;Account admin;;admin\n"
|
||||
if _, err := ParseKerio(strings.NewReader(in), "example.test"); err == nil {
|
||||
t.Fatal("expected error when every row is filtered out")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -154,6 +154,82 @@ func (s *Server) handleCreateAccount(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusCreated, map[string]int64{"id": id})
|
||||
}
|
||||
|
||||
// handleUpdateAccountCredentials fixes the logins/passwords of an existing
|
||||
// account — typically after an import brought in a wrong password and the
|
||||
// connection test failed. An empty password field keeps the stored one, so the
|
||||
// operator can correct one side without retyping the other.
|
||||
func (s *Server) handleUpdateAccountCredentials(w http.ResponseWriter, r *http.Request) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
http.Error(w, "bad id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
accID, err := pathID(r, "accountId")
|
||||
if err != nil {
|
||||
http.Error(w, "bad account id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
task, err := s.store.GetTask(r.Context(), taskID)
|
||||
if err != nil {
|
||||
http.Error(w, "not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
acc, ok := s.findAccount(r, taskID, accID)
|
||||
if !ok {
|
||||
http.Error(w, "account not found", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
if task.Status == "running" || acc.Status == "running" {
|
||||
http.Error(w, "cannot change credentials while the account is running", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
SrcLogin string `json:"src_login"`
|
||||
SrcPass string `json:"src_pass"`
|
||||
DstLogin string `json:"dst_login"`
|
||||
DstPass string `json:"dst_pass"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
http.Error(w, "bad json", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Same trimming as account creation: pasted logins/passwords often carry a
|
||||
// stray space or newline that the IMAP server rejects.
|
||||
body.SrcLogin = strings.TrimSpace(body.SrcLogin)
|
||||
body.DstLogin = strings.TrimSpace(body.DstLogin)
|
||||
body.SrcPass = strings.TrimSpace(body.SrcPass)
|
||||
body.DstPass = strings.TrimSpace(body.DstPass)
|
||||
if body.SrcLogin == "" || body.DstLogin == "" {
|
||||
http.Error(w, "src_login and dst_login are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
encrypt := func(pass string) (*string, error) {
|
||||
if pass == "" {
|
||||
return nil, nil // keep the stored password
|
||||
}
|
||||
enc, err := crypto.Encrypt(s.cfg.EncKey, []byte(pass))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &enc, nil
|
||||
}
|
||||
srcEnc, err := encrypt(body.SrcPass)
|
||||
if err != nil {
|
||||
http.Error(w, "encrypt", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
dstEnc, err := encrypt(body.DstPass)
|
||||
if err != nil {
|
||||
http.Error(w, "encrypt", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if err := s.store.UpdateAccountCredentials(r.Context(), accID, body.SrcLogin, body.DstLogin, srcEnc, dstEnc); err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// findAccount returns the account with accID under taskID, or ok=false.
|
||||
func (s *Server) findAccount(r *http.Request, taskID, accID int64) (store.Account, bool) {
|
||||
accs, err := s.store.ListAccountsByTask(r.Context(), taskID)
|
||||
|
||||
@@ -2,8 +2,11 @@ package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
"github.com/vasyansk/imap-copier/internal/store"
|
||||
)
|
||||
|
||||
@@ -54,6 +57,36 @@ func (s *Server) handleUpdateEndpoint(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleDeleteEndpoint removes an endpoint that no task references. A referenced
|
||||
// endpoint is refused with 409 rather than a foreign-key error, and the same
|
||||
// status covers the race where a task is created between check and delete.
|
||||
func (s *Server) handleDeleteEndpoint(w http.ResponseWriter, r *http.Request) {
|
||||
id, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
http.Error(w, "bad id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
used, err := s.store.CountTasksUsingEndpoint(r.Context(), id)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
if used > 0 {
|
||||
http.Error(w, fmt.Sprintf("endpoint is used by %d task(s) — delete them first", used), http.StatusConflict)
|
||||
return
|
||||
}
|
||||
if err := s.store.DeleteEndpoint(r.Context(), id); err != nil {
|
||||
var pgErr *pgconn.PgError
|
||||
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
|
||||
http.Error(w, "endpoint is used by a task — delete it first", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (s *Server) handleListEndpoints(w http.ResponseWriter, r *http.Request) {
|
||||
eps, err := s.store.ListEndpoints(r.Context())
|
||||
if err != nil {
|
||||
|
||||
@@ -15,6 +15,7 @@ func (s *Server) Router() http.Handler {
|
||||
api.HandleFunc("GET /api/endpoints", s.handleListEndpoints)
|
||||
api.HandleFunc("POST /api/endpoints", s.handleCreateEndpoint)
|
||||
api.HandleFunc("PUT /api/endpoints/{id}", s.handleUpdateEndpoint)
|
||||
api.HandleFunc("DELETE /api/endpoints/{id}", s.handleDeleteEndpoint)
|
||||
api.HandleFunc("GET /api/tasks", s.handleListTasks)
|
||||
api.HandleFunc("POST /api/tasks", s.handleCreateTask)
|
||||
api.HandleFunc("GET /api/tasks/{id}", s.handleGetTask)
|
||||
@@ -26,9 +27,13 @@ func (s *Server) Router() http.Handler {
|
||||
api.HandleFunc("GET /api/tasks/{id}/runs", s.handleListRuns)
|
||||
api.HandleFunc("GET /api/tasks/{id}/accounts/{accountId}/errors", s.handleListAccountErrors)
|
||||
api.HandleFunc("DELETE /api/tasks/{id}/accounts/{accountId}", s.handleDeleteAccount)
|
||||
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/credentials", s.handleUpdateAccountCredentials)
|
||||
api.HandleFunc("POST /api/tasks/{id}/import", s.handleImportCSV)
|
||||
api.HandleFunc("POST /api/tasks/{id}/test", s.handleTestAccounts)
|
||||
api.HandleFunc("POST /api/tasks/{id}/run", s.handleRun)
|
||||
api.HandleFunc("POST /api/tasks/{id}/pause", s.handlePauseRun)
|
||||
api.HandleFunc("POST /api/tasks/{id}/cancel", s.handleCancelRun)
|
||||
api.HandleFunc("POST /api/tasks/{id}/resume", s.handleResumeRun)
|
||||
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/cancel", s.handleCancelAccount)
|
||||
api.HandleFunc("POST /api/tasks/{id}/accounts/{accountId}/probe", s.handleProbeAccountFolders)
|
||||
api.HandleFunc("PUT /api/tasks/{id}/accounts/{accountId}/folder-mapping", s.handleSetAccountFolderMapping)
|
||||
|
||||
+94
-2
@@ -1,8 +1,11 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/vasyansk/imap-copier/internal/crypto"
|
||||
@@ -11,6 +14,17 @@ import (
|
||||
"github.com/vasyansk/imap-copier/internal/store"
|
||||
)
|
||||
|
||||
// parseImportRows picks the CSV dialect from the "format" form field. A Kerio
|
||||
// Connect export holds one login/password pair and no domain, so the operator
|
||||
// supplies the domain alongside the file; anything else is the plain 4-column
|
||||
// src/dst format.
|
||||
func parseImportRows(r *http.Request, file io.Reader) ([]csvimport.Row, error) {
|
||||
if r.FormValue("format") == "kerio" {
|
||||
return csvimport.ParseKerio(file, r.FormValue("domain"))
|
||||
}
|
||||
return csvimport.Parse(file)
|
||||
}
|
||||
|
||||
func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
@@ -23,7 +37,7 @@ func (s *Server) handleImportCSV(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
defer file.Close()
|
||||
rows, err := csvimport.Parse(file)
|
||||
rows, err := parseImportRows(r, file)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
@@ -63,13 +77,41 @@ func (s *Server) handleTestAccounts(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
}
|
||||
|
||||
// parseRunAccountIDs reads an optional {"account_ids":[...]} run body. An empty
|
||||
// body means "all accounts" and yields a nil slice. Malformed JSON is an error.
|
||||
func parseRunAccountIDs(r *http.Request) ([]int64, error) {
|
||||
raw, err := io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(bytes.TrimSpace(raw)) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
var body struct {
|
||||
AccountIDs []int64 `json:"account_ids"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return body.AccountIDs, nil
|
||||
}
|
||||
|
||||
func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
http.Error(w, "bad id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual")
|
||||
accountIDs, err := parseRunAccountIDs(r)
|
||||
if err != nil {
|
||||
http.Error(w, "bad request body", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
runID, err := s.orch.Run(r.Context(), taskID, "manual", accountIDs)
|
||||
if errors.Is(err, orchestrator.ErrNoAccountsSelected) {
|
||||
http.Error(w, "no matching accounts selected", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if errors.Is(err, orchestrator.ErrNotTested) {
|
||||
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||
return
|
||||
@@ -85,6 +127,56 @@ func (s *Server) handleRun(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
|
||||
}
|
||||
|
||||
// handlePauseRun stops the live run but keeps the unfinished accounts
|
||||
// resumable; handleCancelRun stops it for good. Both are no-ops (409) when the
|
||||
// task has no run in flight.
|
||||
func (s *Server) handlePauseRun(w http.ResponseWriter, r *http.Request) {
|
||||
s.stopRun(w, r, s.orch.PauseTask)
|
||||
}
|
||||
|
||||
func (s *Server) handleCancelRun(w http.ResponseWriter, r *http.Request) {
|
||||
s.stopRun(w, r, s.orch.CancelTask)
|
||||
}
|
||||
|
||||
func (s *Server) stopRun(w http.ResponseWriter, r *http.Request, stop func(int64) bool) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
http.Error(w, "bad id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
if !stop(taskID) {
|
||||
http.Error(w, "task is not running", http.StatusConflict)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
}
|
||||
|
||||
// handleResumeRun restarts a paused task with the accounts its pause left
|
||||
// unfinished; already-copied messages are skipped by the migration journal.
|
||||
func (s *Server) handleResumeRun(w http.ResponseWriter, r *http.Request) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
http.Error(w, "bad id", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
runID, err := s.orch.ResumeTask(r.Context(), taskID)
|
||||
switch {
|
||||
case errors.Is(err, orchestrator.ErrNothingToResume):
|
||||
http.Error(w, "no paused accounts to resume", http.StatusConflict)
|
||||
return
|
||||
case errors.Is(err, orchestrator.ErrNotTested):
|
||||
http.Error(w, "accounts must pass connection tests first", http.StatusConflict)
|
||||
return
|
||||
case errors.Is(err, orchestrator.ErrAlreadyRunning):
|
||||
http.Error(w, "task is already running", http.StatusConflict)
|
||||
return
|
||||
case err != nil:
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, map[string]int64{"run_id": runID})
|
||||
}
|
||||
|
||||
func (s *Server) handleCancelAccount(w http.ResponseWriter, r *http.Request) {
|
||||
taskID, err := pathID(r, "id")
|
||||
if err != nil {
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -26,3 +28,79 @@ func TestImportCSVFailsOnBadEncKey(t *testing.T) {
|
||||
t.Fatalf("import must fail on bad EncKey, got %d", rw.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// importReq builds a multipart import request with the given CSV payload and
|
||||
// extra form fields, then returns it with the uploaded file ready to read.
|
||||
func importReq(t *testing.T, csv string, fields map[string]string) (*http.Request, io.Reader) {
|
||||
t.Helper()
|
||||
body := &strings.Builder{}
|
||||
mw := multipart.NewWriter(body)
|
||||
for k, v := range fields {
|
||||
_ = mw.WriteField(k, v)
|
||||
}
|
||||
fw, _ := mw.CreateFormFile("file", "a.csv")
|
||||
fw.Write([]byte(csv))
|
||||
mw.Close()
|
||||
req := httptest.NewRequest("POST", "/api/tasks/1/import", strings.NewReader(body.String()))
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
req.SetPathValue("id", "1")
|
||||
file, _, err := req.FormFile("file")
|
||||
if err != nil {
|
||||
t.Fatalf("form file: %v", err)
|
||||
}
|
||||
return req, file
|
||||
}
|
||||
|
||||
const kerioCSV = "Name;FullName;Description;Enable;DataSource\n" +
|
||||
"info;;InfoPass11;Yes;Internal\n"
|
||||
|
||||
func TestParseImportRowsUsesKerioParserWithDomain(t *testing.T) {
|
||||
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio", "domain": "example.test"})
|
||||
rows, err := parseImportRows(req, file)
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if len(rows) != 1 || rows[0].SrcLogin != "info@example.test" || rows[0].DstLogin != "info@example.test" {
|
||||
t.Fatalf("kerio rows must carry the supplied domain on both sides, got %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImportRowsKerioRequiresDomain(t *testing.T) {
|
||||
req, file := importReq(t, kerioCSV, map[string]string{"format": "kerio"})
|
||||
if _, err := parseImportRows(req, file); err == nil {
|
||||
t.Fatal("kerio import without a domain must error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseImportRowsDefaultsToPlainFormat(t *testing.T) {
|
||||
req, file := importReq(t, "a@x,p1,a@y,p2\n", nil)
|
||||
rows, err := parseImportRows(req, file)
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if len(rows) != 1 || rows[0].SrcLogin != "a@x" || rows[0].DstPass != "p2" {
|
||||
t.Fatalf("no format field must keep the 4-column parser, got %+v", rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseRunAccountIDs(t *testing.T) {
|
||||
// empty body => nil (run all)
|
||||
req := httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(""))
|
||||
ids, err := parseRunAccountIDs(req)
|
||||
if err != nil || ids != nil {
|
||||
t.Fatalf("empty body must yield nil ids, got %v err=%v", ids, err)
|
||||
}
|
||||
|
||||
// explicit selection
|
||||
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{"account_ids":[3,7]}`))
|
||||
ids, err = parseRunAccountIDs(req)
|
||||
if err != nil || len(ids) != 2 || ids[0] != 3 || ids[1] != 7 {
|
||||
t.Fatalf("must parse account_ids, got %v err=%v", ids, err)
|
||||
}
|
||||
|
||||
// malformed JSON => error
|
||||
req = httptest.NewRequest("POST", "/api/tasks/1/run", strings.NewReader(`{bad`))
|
||||
if _, err := parseRunAccountIDs(req); err == nil {
|
||||
t.Fatal("malformed body must error")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,13 +4,12 @@ import (
|
||||
"context"
|
||||
|
||||
"github.com/emersion/go-imap/v2"
|
||||
"github.com/emersion/go-imap/v2/imapclient"
|
||||
)
|
||||
|
||||
// FolderMessageCount opens a folder read-only (EXAMINE) and returns how many
|
||||
// messages it holds — used to plan an accurate overall progress total before
|
||||
// copying begins. It does not fetch any message bodies.
|
||||
func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
|
||||
func FolderMessageCount(c *Client, folder string) (int64, error) {
|
||||
sel, err := c.Select(folder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
@@ -19,7 +18,7 @@ func FolderMessageCount(c *imapclient.Client, folder string) (int64, error) {
|
||||
}
|
||||
|
||||
// ListFolders returns the mailbox names visible on an already-connected, logged-in client.
|
||||
func ListFolders(c *imapclient.Client) ([]string, error) {
|
||||
func ListFolders(c *Client) ([]string, error) {
|
||||
mboxes, err := c.List("", "*", nil).Collect()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
+131
-10
@@ -6,7 +6,9 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -35,8 +37,37 @@ type CopyDeps struct {
|
||||
// text — so the orchestrator can persist individual errors for the
|
||||
// per-account error modal. Folder-level errors are reported by the caller.
|
||||
OnError func(ref, msg string)
|
||||
// OnActivity is called repeatedly WHILE a single message body is streamed
|
||||
// (each FETCH read chunk and each APPEND write chunk). Copying one large
|
||||
// message can take longer than the orchestrator's stall timeout; without an
|
||||
// in-body signal the watchdog can't tell a slow-but-live transfer from a
|
||||
// wedged connection and cancels a healthy copy. May be nil.
|
||||
OnActivity func()
|
||||
// ReconnectSrc dials and logs in a FRESH source client, returning it. It is
|
||||
// called when a body read times out (server under-delivered a literal),
|
||||
// which leaves the current src connection desynced and unusable. CopyFolder
|
||||
// swaps to the returned client, re-EXAMINEs the folder, and resumes. The
|
||||
// implementation is expected to also update any external reference to the
|
||||
// live src client (e.g. so a cancel path closes the right connection). If
|
||||
// nil, a body-read timeout aborts the folder instead of recovering.
|
||||
ReconnectSrc func() (*Client, error)
|
||||
}
|
||||
|
||||
// ErrBodyTimeout means a message body did not finish transferring within the
|
||||
// idle deadline — the server stopped sending mid-literal. It is almost always a
|
||||
// server announcing a BODY[] literal larger than the bytes it actually sends,
|
||||
// which makes go-imap wait forever for bytes that never come. Distinct from a
|
||||
// closed connection so the caller can skip just this one message and resume.
|
||||
var ErrBodyTimeout = errors.New("message body read timed out")
|
||||
|
||||
// bodyIdleTimeout bounds how long a body read may go with NO bytes arriving
|
||||
// before it is abandoned. It is generous enough for legitimately slow servers
|
||||
// (even ~15 KB/s links keep bytes flowing far more often than this) yet well
|
||||
// under the orchestrator's multi-minute stall watchdog, so an under-delivered
|
||||
// literal is caught quickly and locally instead of stalling the whole account.
|
||||
// A var (not const) so tests can shorten it.
|
||||
var bodyIdleTimeout = 30 * time.Second
|
||||
|
||||
// CopyResult summarizes the outcome of one CopyFolder run.
|
||||
type CopyResult struct {
|
||||
Copied int
|
||||
@@ -88,7 +119,7 @@ func metaBatches(total, batchSize uint32) []imap.SeqRange {
|
||||
// held in memory only for the duration of a single FETCH->APPEND and is
|
||||
// never written to disk. Messages already migrated (per deps.IsMigrated)
|
||||
// are skipped without re-fetching their bodies.
|
||||
func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
|
||||
func CopyFolder(ctx context.Context, src, dst *Client, srcFolder, dstFolder string, deps CopyDeps) (CopyResult, error) {
|
||||
var res CopyResult
|
||||
|
||||
sel, err := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait()
|
||||
@@ -180,9 +211,28 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
|
||||
if err := ctx.Err(); err != nil {
|
||||
return res, err
|
||||
}
|
||||
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate); err != nil {
|
||||
if err := streamOne(src, dst, dstFolder, q.uid, q.flags, q.internalDate, deps.OnActivity); err != nil {
|
||||
res.Errors++
|
||||
reportErr(msgRef(q.uid, q.subject), "copy message: "+err.Error())
|
||||
// A body-read timeout means the server under-delivered this message's
|
||||
// literal; the src connection is now desynced. Mark the message
|
||||
// migrated so this and future runs skip it (it is un-fetchable via a
|
||||
// conforming client), then reconnect src and resume the folder with
|
||||
// the remaining queued messages.
|
||||
if errors.Is(err, ErrBodyTimeout) && deps.ReconnectSrc != nil {
|
||||
if merr := deps.MarkMigrated(dstFolder, q.key); merr != nil {
|
||||
reportErr(msgRef(q.uid, q.subject), "mark skipped: "+merr.Error())
|
||||
}
|
||||
newSrc, rerr := deps.ReconnectSrc()
|
||||
if rerr != nil {
|
||||
return res, fmt.Errorf("reconnect src after body timeout in %q: %w", srcFolder, rerr)
|
||||
}
|
||||
src = newSrc
|
||||
if _, serr := src.Select(srcFolder, &imap.SelectOptions{ReadOnly: true}).Wait(); serr != nil {
|
||||
return res, fmt.Errorf("re-examine %q after reconnect: %w", srcFolder, serr)
|
||||
}
|
||||
continue
|
||||
}
|
||||
// A closed/broken connection won't recover: every remaining APPEND
|
||||
// would fail identically. Abort the folder instead of logging
|
||||
// thousands of the same error; a re-run resumes via dedup.
|
||||
@@ -204,11 +254,63 @@ func CopyFolder(ctx context.Context, src, dst *imapclient.Client, srcFolder, dst
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// slowMessage marks how long one message's FETCH or APPEND phase may take
|
||||
// before it is logged as anomalous. Well below the orchestrator's 3-minute
|
||||
// stall timeout, so a message that trips the watchdog always leaves a log line
|
||||
// naming the phase (FETCH vs APPEND) and size — turning a silent stall into
|
||||
// evidence of which side and which message is the culprit.
|
||||
const slowMessage = 20 * time.Second
|
||||
|
||||
// deadlineReader arms an idle read deadline on the source socket before every
|
||||
// read of a message body, so a server that stops sending mid-literal — having
|
||||
// announced a larger BODY[] size than it actually delivers — trips the deadline
|
||||
// instead of blocking go-imap forever waiting for bytes that never arrive. It
|
||||
// also pings onActivity as bytes arrive, feeding the orchestrator's stall
|
||||
// watchdog. The deadline is refreshed on each read, so it bounds IDLE time
|
||||
// (no bytes) rather than total transfer time — a legitimately slow but steady
|
||||
// download never trips it.
|
||||
type deadlineReader struct {
|
||||
c *Client
|
||||
r io.Reader
|
||||
idle time.Duration
|
||||
on func()
|
||||
}
|
||||
|
||||
func (d deadlineReader) Read(p []byte) (int, error) {
|
||||
_ = d.c.SetReadDeadline(time.Now().Add(d.idle))
|
||||
n, err := d.r.Read(p)
|
||||
if n > 0 && d.on != nil {
|
||||
d.on()
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// touchWriter wraps the APPEND write stream and pings onActivity on every
|
||||
// non-empty write, so a long upload keeps the stall watchdog fed byte-by-byte.
|
||||
type touchWriter struct {
|
||||
w io.Writer
|
||||
on func()
|
||||
}
|
||||
|
||||
func (t touchWriter) Write(p []byte) (int, error) {
|
||||
n, err := t.w.Write(p)
|
||||
if n > 0 && t.on != nil {
|
||||
t.on()
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// streamOne FETCHes BODY[] for one message and APPENDs it into dst without
|
||||
// spooling to disk. The body is buffered in RAM only for the duration of
|
||||
// this single FETCH->APPEND round trip.
|
||||
func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time) error {
|
||||
// this single FETCH->APPEND round trip. onActivity (may be nil) fires as bytes
|
||||
// move in either direction, feeding the orchestrator's stall watchdog.
|
||||
//
|
||||
// The body read is guarded by an idle deadline on src: if the server goes
|
||||
// silent mid-literal, streamOne returns ErrBodyTimeout rather than hanging, so
|
||||
// CopyFolder can skip the message and reconnect.
|
||||
func streamOne(src, dst *Client, dstFolder string, uid imap.UID, flags []imap.Flag, internalDate time.Time, onActivity func()) error {
|
||||
bodySection := &imap.FetchItemBodySection{}
|
||||
fetchStart := time.Now()
|
||||
fetchCmd := src.Fetch(imap.UIDSetNum(uid), &imap.FetchOptions{
|
||||
BodySection: []*imap.FetchItemBodySection{bodySection},
|
||||
})
|
||||
@@ -219,18 +321,24 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
|
||||
return fmt.Errorf("no message for uid %v", uid)
|
||||
}
|
||||
var body []byte
|
||||
var readErr error
|
||||
for {
|
||||
item := msg.Next()
|
||||
if item == nil {
|
||||
break
|
||||
}
|
||||
if d, ok := item.(imapclient.FetchItemDataBodySection); ok {
|
||||
b, err := io.ReadAll(d.Literal)
|
||||
if err != nil {
|
||||
return err
|
||||
body, readErr = io.ReadAll(deadlineReader{c: src, r: d.Literal, idle: bodyIdleTimeout, on: onActivity})
|
||||
}
|
||||
body = b
|
||||
}
|
||||
// Clear the deadline before any further I/O on src (fetchCmd.Close reads the
|
||||
// command's completion off the same socket).
|
||||
_ = src.SetReadDeadline(time.Time{})
|
||||
if readErr != nil {
|
||||
if errors.Is(readErr, os.ErrDeadlineExceeded) {
|
||||
return fmt.Errorf("%w: uid %v (server sent fewer bytes than the announced literal)", ErrBodyTimeout, uid)
|
||||
}
|
||||
return readErr
|
||||
}
|
||||
if err := fetchCmd.Close(); err != nil {
|
||||
return err
|
||||
@@ -238,14 +346,16 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
|
||||
if body == nil {
|
||||
return fmt.Errorf("empty body uid %v", uid)
|
||||
}
|
||||
fetchDur := time.Since(fetchStart)
|
||||
|
||||
appendStart := time.Now()
|
||||
appendCmd := dst.Append(dstFolder, int64(len(body)), &imap.AppendOptions{Flags: keepFlags(flags), Time: internalDate})
|
||||
// Append acquires go-imap's per-client encoder mutex and holds it until
|
||||
// Close() calls enc.end(). Close() MUST run on every path: if io.Copy
|
||||
// fails mid-write (server stall, idle timeout), returning without Close()
|
||||
// leaks the mutex and the NEXT Append on this client deadlocks forever on
|
||||
// beginCommand. Close() is idempotent and always releases the lock.
|
||||
_, copyErr := io.Copy(appendCmd, bytes.NewReader(body))
|
||||
_, copyErr := io.Copy(touchWriter{w: appendCmd, on: onActivity}, bytes.NewReader(body))
|
||||
closeErr := appendCmd.Close()
|
||||
if copyErr != nil {
|
||||
return fmt.Errorf("append body uid %v: %w", uid, copyErr)
|
||||
@@ -253,9 +363,20 @@ func streamOne(src, dst *imapclient.Client, dstFolder string, uid imap.UID, flag
|
||||
if closeErr != nil {
|
||||
return closeErr
|
||||
}
|
||||
_, err := appendCmd.Wait()
|
||||
if _, err := appendCmd.Wait(); err != nil {
|
||||
return err
|
||||
}
|
||||
appendDur := time.Since(appendStart)
|
||||
|
||||
// One message that individually eats a large slice of the stall budget is
|
||||
// the prime suspect behind a "no progress" cancel; name it, its size, and
|
||||
// which phase was slow so the culprit is visible in the logs.
|
||||
if fetchDur > slowMessage || appendDur > slowMessage {
|
||||
slog.Warn("slow message copy", "uid", uid, "bytes", len(body),
|
||||
"fetch", fetchDur.Round(time.Millisecond), "append", appendDur.Round(time.Millisecond))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// keepFlags drops \Recent: it cannot be set via APPEND. go-imap v2 beta.8
|
||||
// no longer defines an imap.FlagRecent constant (RFC 9051 dropped \Recent
|
||||
|
||||
@@ -169,6 +169,56 @@ func TestCopyFolderPreservesInternalDate(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCopyFolderReportsActivityDuringBody proves CopyFolder invokes OnActivity
|
||||
// while a message body is being transferred (FETCH/APPEND), not only between
|
||||
// messages. This is what keeps the orchestrator's stall watchdog from killing a
|
||||
// single large-but-live message whose transfer legitimately exceeds the stall
|
||||
// timeout: without an in-body activity signal, one slow message looks identical
|
||||
// to a wedged connection.
|
||||
func TestCopyFolderReportsActivityDuringBody(t *testing.T) {
|
||||
ep := testEP(t)
|
||||
ctx := context.Background()
|
||||
|
||||
seedInbox(t, ep, "actsrc@localhost", "p", 1)
|
||||
|
||||
src, err := Connect(ctx, ep)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = src.Logout().Wait() }()
|
||||
if err := src.Login("actsrc@localhost", "p").Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dst, err := Connect(ctx, ep)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = dst.Logout().Wait() }()
|
||||
if err := dst.Login("actdst@localhost", "p").Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var activity int
|
||||
deps := CopyDeps{
|
||||
IsMigrated: func(string) (bool, error) { return false, nil },
|
||||
MarkMigrated: func(_, _ string) error { return nil },
|
||||
OnProgress: func(_, _ int) {},
|
||||
OnActivity: func() { activity++ },
|
||||
}
|
||||
|
||||
r, err := CopyFolder(ctx, src, dst, "INBOX", "INBOX", deps)
|
||||
if err != nil {
|
||||
t.Fatalf("CopyFolder: %v", err)
|
||||
}
|
||||
if r.Copied != 1 {
|
||||
t.Fatalf("copied=%d want 1", r.Copied)
|
||||
}
|
||||
if activity == 0 {
|
||||
t.Fatal("OnActivity never called during body transfer")
|
||||
}
|
||||
}
|
||||
|
||||
// Требует два ящика на greenmail. Первый запуск копирует N, второй — 0 (все skipped).
|
||||
func TestCopyFolderIdempotent(t *testing.T) {
|
||||
ep := testEP(t) // plain greenmail
|
||||
|
||||
+39
-5
@@ -18,6 +18,29 @@ type Endpoint struct {
|
||||
|
||||
func (e Endpoint) addr() string { return fmt.Sprintf("%s:%d", e.Host, e.Port) }
|
||||
|
||||
// Client wraps an imapclient.Client together with the raw network connection it
|
||||
// runs over. The embedded *imapclient.Client provides the full IMAP API; the
|
||||
// retained conn lets callers impose a read deadline on the socket for the
|
||||
// duration of a body transfer.
|
||||
//
|
||||
// This defends against servers that announce a BODY[] literal LARGER than the
|
||||
// bytes they actually send (a protocol violation observed on some webmail
|
||||
// servers). go-imap reads a literal strictly by its announced size, so a short
|
||||
// literal makes it block forever waiting for bytes that never arrive. A
|
||||
// deadline around the body read turns that infinite hang into a timeout the
|
||||
// copier can recover from.
|
||||
type Client struct {
|
||||
*imapclient.Client
|
||||
conn net.Conn
|
||||
}
|
||||
|
||||
// SetReadDeadline sets (or, with the zero time, clears) a deadline on the
|
||||
// underlying socket. Used to bound a single body read; always cleared again
|
||||
// once the read completes so it never affects idle periods.
|
||||
func (c *Client) SetReadDeadline(t time.Time) error {
|
||||
return c.conn.SetReadDeadline(t)
|
||||
}
|
||||
|
||||
// dialTimeout bounds establishing the TCP connection.
|
||||
const dialTimeout = 30 * time.Second
|
||||
|
||||
@@ -27,7 +50,7 @@ const dialTimeout = 30 * time.Second
|
||||
// during a long src scan) from one stuck mid-response, and would wrongly close
|
||||
// idle connections. Stall detection is done at the orchestrator level via a
|
||||
// progress watchdog; go-imap's own per-command timeouts bound active commands.
|
||||
func dialOnce(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
|
||||
func dialOnce(ctx context.Context, ep Endpoint) (*Client, error) {
|
||||
d := &net.Dialer{Timeout: dialTimeout}
|
||||
raw, err := d.DialContext(ctx, "tcp", ep.addr())
|
||||
if err != nil {
|
||||
@@ -42,16 +65,27 @@ func dialOnce(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
|
||||
_ = raw.Close()
|
||||
return nil, err
|
||||
}
|
||||
return waitGreeting(imapclient.New(tlsConn, nil))
|
||||
c, err := waitGreeting(imapclient.New(tlsConn, nil))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Client{Client: c, conn: tlsConn}, nil
|
||||
case "starttls":
|
||||
// Deadline goes on the raw TCP conn: it sits beneath the TLS layer that
|
||||
// NewStartTLS negotiates, and a TCP read deadline still interrupts the
|
||||
// TLS read above it.
|
||||
opts := &imapclient.Options{TLSConfig: &tls.Config{ServerName: ep.Host}}
|
||||
c, err := imapclient.NewStartTLS(raw, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c, nil
|
||||
return &Client{Client: c, conn: raw}, nil
|
||||
case "plain":
|
||||
return waitGreeting(imapclient.New(raw, nil))
|
||||
c, err := waitGreeting(imapclient.New(raw, nil))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Client{Client: c, conn: raw}, nil
|
||||
default:
|
||||
_ = raw.Close()
|
||||
return nil, fmt.Errorf("unknown tls_mode %q", ep.TLSMode)
|
||||
@@ -69,7 +103,7 @@ func waitGreeting(c *imapclient.Client) (*imapclient.Client, error) {
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func Connect(ctx context.Context, ep Endpoint) (*imapclient.Client, error) {
|
||||
func Connect(ctx context.Context, ep Endpoint) (*Client, error) {
|
||||
const attempts = 3
|
||||
var lastErr error
|
||||
for i := 0; i < attempts; i++ {
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
package imapx
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// KeepaliveInterval is how often an otherwise-idle IMAP connection is pinged
|
||||
// with NOOP so the server does not drop it.
|
||||
//
|
||||
// The destination connection sits completely idle for the entire duration of
|
||||
// the source-side metadata scan (Pass 1 of CopyFolder), which on a large
|
||||
// mailbox runs for many minutes across all folders. With no traffic, the
|
||||
// server closes the idle connection; go-imap's reader then tears the client
|
||||
// down, and every subsequent APPEND fails with "use of closed network
|
||||
// connection" — aborting each folder and copying nothing. A periodic NOOP
|
||||
// keeps the connection warm. 60s is well under the idle timeout of any common
|
||||
// IMAP server.
|
||||
const KeepaliveInterval = 60 * time.Second
|
||||
|
||||
// Keepalive pings c with a NOOP every interval until ctx is cancelled, keeping
|
||||
// an idle connection from being dropped by the server. It is meant to run in
|
||||
// its own goroutine.
|
||||
//
|
||||
// It is safe to run concurrently with other commands on c: go-imap serializes
|
||||
// command submission and supports multiple in-flight commands over a single
|
||||
// connection (RFC 9051 §5.5 pipelining). A NOOP issued while another command is
|
||||
// in flight simply queues behind it and completes when the server responds.
|
||||
//
|
||||
// Keepalive returns when ctx is done or when a NOOP fails — a failed NOOP means
|
||||
// the connection is already gone, so there is nothing left to keep alive.
|
||||
func Keepalive(ctx context.Context, c *Client, interval time.Duration) {
|
||||
t := time.NewTicker(interval)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if err := c.Noop().Wait(); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
package imapx
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestKeepaliveReturnsOnContextCancel proves Keepalive is a well-behaved
|
||||
// goroutine: it exits promptly when its context is cancelled instead of
|
||||
// leaking.
|
||||
func TestKeepaliveReturnsOnContextCancel(t *testing.T) {
|
||||
ep := testEP(t)
|
||||
ctx := context.Background()
|
||||
|
||||
c, err := Connect(ctx, ep)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = c.Logout().Wait() }()
|
||||
if err := c.Login("ka1@localhost", "p").Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
kctx, cancel := context.WithCancel(ctx)
|
||||
done := make(chan struct{})
|
||||
go func() { Keepalive(kctx, c, 10*time.Millisecond); close(done) }()
|
||||
|
||||
// Let a few NOOPs fire, then cancel and require a prompt return.
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
cancel()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("Keepalive did not return within 2s of context cancel")
|
||||
}
|
||||
}
|
||||
|
||||
// TestKeepaliveDoesNotDisruptCopy runs Keepalive on the destination connection
|
||||
// at an aggressive interval while CopyFolder is APPENDing to it, proving the
|
||||
// concurrent NOOPs do not corrupt in-flight commands (the real risk of pinging
|
||||
// a connection that is also being used for real work).
|
||||
func TestKeepaliveDoesNotDisruptCopy(t *testing.T) {
|
||||
ep := testEP(t)
|
||||
ctx := context.Background()
|
||||
|
||||
const n = 8
|
||||
seedInbox(t, ep, "kasrc@localhost", "p", n)
|
||||
|
||||
src, err := Connect(ctx, ep)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = src.Logout().Wait() }()
|
||||
if err := src.Login("kasrc@localhost", "p").Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dst, err := Connect(ctx, ep)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = dst.Logout().Wait() }()
|
||||
if err := dst.Login("kadst@localhost", "p").Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
kctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
go Keepalive(kctx, dst, 1*time.Millisecond)
|
||||
|
||||
seen := map[string]bool{}
|
||||
deps := CopyDeps{
|
||||
IsMigrated: func(k string) (bool, error) { return seen[k], nil },
|
||||
MarkMigrated: func(_, k string) error { seen[k] = true; return nil },
|
||||
OnProgress: func(_, _ int) {},
|
||||
}
|
||||
|
||||
r, err := CopyFolder(kctx, src, dst, "INBOX", "INBOX", deps)
|
||||
if err != nil {
|
||||
t.Fatalf("CopyFolder with concurrent keepalive: %v", err)
|
||||
}
|
||||
if r.Copied != n {
|
||||
t.Fatalf("copied=%d want %d", r.Copied, n)
|
||||
}
|
||||
if r.Errors != 0 {
|
||||
t.Fatalf("errors=%d want 0", r.Errors)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
package imapx
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/emersion/go-imap/v2"
|
||||
)
|
||||
|
||||
// underflowServer is a minimal IMAP server that reproduces the amega.kz bug: it
|
||||
// answers a BODY[] FETCH by announcing a literal LARGER than the bytes it then
|
||||
// sends, and afterwards goes silent — exactly what makes go-imap's strict
|
||||
// literal reader block forever. LOGIN/EXAMINE and the Pass-1 metadata FETCH are
|
||||
// answered normally so a full CopyFolder can reach the poisoned message.
|
||||
//
|
||||
// It serves one connection per accept and keeps accepting, so a reconnect gets
|
||||
// a fresh, well-behaved session (its second EXAMINE reports zero messages, so
|
||||
// the resumed folder simply finishes).
|
||||
type underflowServer struct {
|
||||
ln net.Listener
|
||||
mu sync.Mutex
|
||||
accepts int // how many connections have been accepted
|
||||
stop chan struct{}
|
||||
}
|
||||
|
||||
func newUnderflowServer(t *testing.T) *underflowServer {
|
||||
t.Helper()
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatalf("listen: %v", err)
|
||||
}
|
||||
s := &underflowServer{ln: ln, stop: make(chan struct{})}
|
||||
go s.serve()
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *underflowServer) addr() Endpoint {
|
||||
a := s.ln.Addr().(*net.TCPAddr)
|
||||
return Endpoint{Host: "127.0.0.1", Port: a.Port, TLSMode: "plain"}
|
||||
}
|
||||
|
||||
func (s *underflowServer) close() { close(s.stop); _ = s.ln.Close() }
|
||||
|
||||
func (s *underflowServer) serve() {
|
||||
for {
|
||||
conn, err := s.ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.accepts++
|
||||
first := s.accepts == 1
|
||||
s.mu.Unlock()
|
||||
go s.handle(conn, first)
|
||||
}
|
||||
}
|
||||
|
||||
// handle drives one connection. On the FIRST connection the mailbox reports one
|
||||
// message and its BODY[] fetch under-delivers; on any later connection (i.e.
|
||||
// after a reconnect) the mailbox is empty so the resumed folder completes.
|
||||
func (s *underflowServer) handle(conn net.Conn, first bool) {
|
||||
defer func() { _ = conn.Close() }()
|
||||
br := bufio.NewReader(conn)
|
||||
fmt.Fprint(conn, "* OK IMAP4rev1 ready\r\n")
|
||||
for {
|
||||
line, err := br.ReadString('\n')
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) == 0 {
|
||||
continue
|
||||
}
|
||||
tag := fields[0]
|
||||
up := strings.ToUpper(line)
|
||||
switch {
|
||||
case strings.Contains(up, "LOGIN"):
|
||||
fmt.Fprintf(conn, "%s OK LOGIN completed\r\n", tag)
|
||||
case strings.Contains(up, "EXAMINE"), strings.Contains(up, "SELECT"):
|
||||
n := 0
|
||||
if first {
|
||||
n = 1
|
||||
}
|
||||
fmt.Fprintf(conn, "* %d EXISTS\r\n", n)
|
||||
fmt.Fprint(conn, "* OK [UIDVALIDITY 1] ok\r\n")
|
||||
fmt.Fprintf(conn, "%s OK [READ-ONLY] EXAMINE completed\r\n", tag)
|
||||
case strings.Contains(up, "BODY["), strings.Contains(up, "BODY.PEEK"):
|
||||
// Poison: announce 100000 bytes, send 10, then stall until shutdown.
|
||||
fmt.Fprint(conn, "* 1 FETCH (UID 1 BODY[] {100000}\r\n")
|
||||
fmt.Fprint(conn, "0123456789")
|
||||
<-s.stop
|
||||
return
|
||||
case strings.Contains(up, "FETCH"):
|
||||
// Pass-1 metadata fetch for the single message.
|
||||
fmt.Fprint(conn, "* 1 FETCH (UID 1 RFC822.SIZE 100 FLAGS () "+
|
||||
"INTERNALDATE \"01-Jan-2020 00:00:00 +0000\" "+
|
||||
"ENVELOPE (\"Wed, 01 Jan 2020 00:00:00 +0000\" \"poison\" NIL NIL NIL NIL NIL NIL NIL \"<poison@x>\"))\r\n")
|
||||
fmt.Fprintf(conn, "%s OK FETCH completed\r\n", tag)
|
||||
case strings.Contains(up, "LOGOUT"):
|
||||
fmt.Fprintf(conn, "* BYE\r\n%s OK LOGOUT completed\r\n", tag)
|
||||
return
|
||||
case strings.Contains(up, "CREATE"), strings.Contains(up, "NOOP"):
|
||||
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
|
||||
default:
|
||||
fmt.Fprintf(conn, "%s OK completed\r\n", tag)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestStreamOneTimesOutOnLiteralUnderflow proves a server that announces a
|
||||
// larger BODY[] literal than it sends makes streamOne return ErrBodyTimeout
|
||||
// (bounded by bodyIdleTimeout) instead of hanging forever.
|
||||
func TestStreamOneTimesOutOnLiteralUnderflow(t *testing.T) {
|
||||
restore := shortenBodyIdle(200 * time.Millisecond)
|
||||
defer restore()
|
||||
|
||||
srv := newUnderflowServer(t)
|
||||
defer srv.close()
|
||||
ctx := context.Background()
|
||||
|
||||
src, err := Connect(ctx, srv.addr())
|
||||
if err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
defer func() { _ = src.Close() }()
|
||||
if err := src.Login("u", "p").Wait(); err != nil {
|
||||
t.Fatalf("login: %v", err)
|
||||
}
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- streamOne(src, src, "INBOX", imap.UID(1), nil, time.Time{}, nil)
|
||||
}()
|
||||
|
||||
select {
|
||||
case err := <-done:
|
||||
if !errors.Is(err, ErrBodyTimeout) {
|
||||
t.Fatalf("want ErrBodyTimeout, got %v", err)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("streamOne hung on literal underflow instead of timing out")
|
||||
}
|
||||
}
|
||||
|
||||
// TestCopyFolderSkipsAndReconnectsOnUnderflow proves CopyFolder does not hang on
|
||||
// a poisoned message: it marks the message migrated (so future runs skip it),
|
||||
// invokes ReconnectSrc, and returns with the error counted — the folder is not
|
||||
// wedged.
|
||||
func TestCopyFolderSkipsAndReconnectsOnUnderflow(t *testing.T) {
|
||||
restore := shortenBodyIdle(200 * time.Millisecond)
|
||||
defer restore()
|
||||
|
||||
srv := newUnderflowServer(t)
|
||||
defer srv.close()
|
||||
ctx := context.Background()
|
||||
|
||||
src, err := Connect(ctx, srv.addr())
|
||||
if err != nil {
|
||||
t.Fatalf("connect: %v", err)
|
||||
}
|
||||
defer func() { _ = src.Close() }()
|
||||
if err := src.Login("u", "p").Wait(); err != nil {
|
||||
t.Fatalf("login: %v", err)
|
||||
}
|
||||
|
||||
var marked []string
|
||||
var reconnected bool
|
||||
deps := CopyDeps{
|
||||
IsMigrated: func(string) (bool, error) { return false, nil },
|
||||
MarkMigrated: func(_, k string) error { marked = append(marked, k); return nil },
|
||||
OnProgress: func(_, _ int) {},
|
||||
ReconnectSrc: func() (*Client, error) {
|
||||
reconnected = true
|
||||
nc, derr := Connect(ctx, srv.addr())
|
||||
if derr != nil {
|
||||
return nil, derr
|
||||
}
|
||||
if lerr := nc.Login("u", "p").Wait(); lerr != nil {
|
||||
return nil, lerr
|
||||
}
|
||||
return nc, nil
|
||||
},
|
||||
}
|
||||
|
||||
done := make(chan CopyResult, 1)
|
||||
go func() {
|
||||
r, _ := CopyFolder(ctx, src, src, "INBOX", "INBOX", deps)
|
||||
done <- r
|
||||
}()
|
||||
|
||||
select {
|
||||
case r := <-done:
|
||||
if r.Errors == 0 {
|
||||
t.Fatalf("expected the poisoned message counted as an error, got %+v", r)
|
||||
}
|
||||
if !reconnected {
|
||||
t.Fatal("ReconnectSrc was never called")
|
||||
}
|
||||
if len(marked) != 1 {
|
||||
t.Fatalf("poisoned message should be marked migrated once, got %v", marked)
|
||||
}
|
||||
case <-time.After(8 * time.Second):
|
||||
t.Fatal("CopyFolder hung on a poisoned message instead of skipping it")
|
||||
}
|
||||
}
|
||||
|
||||
func shortenBodyIdle(d time.Duration) func() {
|
||||
old := bodyIdleTimeout
|
||||
bodyIdleTimeout = d
|
||||
return func() { bodyIdleTimeout = old }
|
||||
}
|
||||
@@ -16,6 +16,8 @@ import (
|
||||
|
||||
var ErrNotTested = errors.New("accounts not fully tested")
|
||||
var ErrAlreadyRunning = errors.New("task already running")
|
||||
var ErrNoAccountsSelected = errors.New("no matching accounts selected")
|
||||
var ErrNothingToResume = errors.New("no paused accounts to resume")
|
||||
|
||||
// maxAccountErrors caps how many individual error rows one account records per
|
||||
// run, so a corrupt mailbox producing thousands of failures can't bloat the
|
||||
@@ -61,6 +63,38 @@ func planFolders(folders []string, mapping map[string]string, excluded []string)
|
||||
return plan
|
||||
}
|
||||
|
||||
// A run stops either on its own or because the operator intervened. Pausing and
|
||||
// cancelling take the same path — stop the in-flight work — and differ only in
|
||||
// the status left behind: paused accounts are what Resume picks up again.
|
||||
type stopReason int32
|
||||
|
||||
const (
|
||||
stopNone stopReason = iota
|
||||
stopPaused
|
||||
stopCancelled
|
||||
)
|
||||
|
||||
func (r stopReason) accountStatus() string {
|
||||
if r == stopPaused {
|
||||
return "paused"
|
||||
}
|
||||
return "cancelled"
|
||||
}
|
||||
|
||||
// runHandle is the live state of one task's run: the cancel that stops every
|
||||
// account under it, plus why it was stopped.
|
||||
type runHandle struct {
|
||||
cancel context.CancelFunc
|
||||
reason atomic.Int32
|
||||
}
|
||||
|
||||
func (h *runHandle) stopWith(r stopReason) {
|
||||
h.reason.CompareAndSwap(int32(stopNone), int32(r))
|
||||
h.cancel()
|
||||
}
|
||||
|
||||
func (h *runHandle) stopReason() stopReason { return stopReason(h.reason.Load()) }
|
||||
|
||||
type Orchestrator struct {
|
||||
store *store.Store
|
||||
hub *wshub.Hub
|
||||
@@ -69,10 +103,66 @@ type Orchestrator struct {
|
||||
|
||||
mu sync.Mutex
|
||||
cancels map[int64]context.CancelFunc // account_id -> cancel of its in-flight copy
|
||||
runs map[int64]*runHandle // task_id -> live run
|
||||
}
|
||||
|
||||
func New(s *store.Store, hub *wshub.Hub, encKey []byte, concurrency int) *Orchestrator {
|
||||
return &Orchestrator{store: s, hub: hub, encKey: encKey, concurrency: concurrency, cancels: map[int64]context.CancelFunc{}}
|
||||
return &Orchestrator{
|
||||
store: s, hub: hub, encKey: encKey, concurrency: concurrency,
|
||||
cancels: map[int64]context.CancelFunc{},
|
||||
runs: map[int64]*runHandle{},
|
||||
}
|
||||
}
|
||||
|
||||
// PauseTask stops the task's live run, leaving every unfinished account
|
||||
// "paused" so ResumeTask can pick them up. Returns false if nothing is running.
|
||||
func (o *Orchestrator) PauseTask(taskID int64) bool { return o.stopRun(taskID, stopPaused) }
|
||||
|
||||
// CancelTask stops the task's live run and marks every unfinished account
|
||||
// "cancelled". Returns false if nothing is running.
|
||||
func (o *Orchestrator) CancelTask(taskID int64) bool { return o.stopRun(taskID, stopCancelled) }
|
||||
|
||||
func (o *Orchestrator) stopRun(taskID int64, reason stopReason) bool {
|
||||
o.mu.Lock()
|
||||
h, ok := o.runs[taskID]
|
||||
o.mu.Unlock()
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
h.stopWith(reason)
|
||||
return true
|
||||
}
|
||||
|
||||
func (o *Orchestrator) registerRun(taskID int64, h *runHandle) {
|
||||
o.mu.Lock()
|
||||
o.runs[taskID] = h
|
||||
o.mu.Unlock()
|
||||
}
|
||||
|
||||
func (o *Orchestrator) unregisterRun(taskID int64) {
|
||||
o.mu.Lock()
|
||||
delete(o.runs, taskID)
|
||||
o.mu.Unlock()
|
||||
}
|
||||
|
||||
// ResumeTask restarts a paused task with exactly the accounts the pause left
|
||||
// unfinished. Everything already copied is skipped by the migration journal, so
|
||||
// each account continues where it stopped.
|
||||
func (o *Orchestrator) ResumeTask(ctx context.Context, taskID int64) (int64, error) {
|
||||
accs, err := o.store.ListAccountsByTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
ids := make([]int64, 0, len(accs))
|
||||
for _, a := range accs {
|
||||
if a.Status == "paused" {
|
||||
ids = append(ids, a.ID)
|
||||
}
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return 0, ErrNothingToResume
|
||||
}
|
||||
return o.Run(ctx, taskID, "manual", ids)
|
||||
}
|
||||
|
||||
// CancelAccount aborts the in-flight copy for one account, if it is running.
|
||||
@@ -99,6 +189,26 @@ func (o *Orchestrator) unregisterCancel(accountID int64) {
|
||||
o.mu.Unlock()
|
||||
}
|
||||
|
||||
// selectAccounts narrows accs to those whose ID is in ids, preserving input
|
||||
// order. An empty or nil ids means "all accounts" — the scheduler and the
|
||||
// unfiltered manual run rely on this.
|
||||
func selectAccounts(accs []store.Account, ids []int64) []store.Account {
|
||||
if len(ids) == 0 {
|
||||
return accs
|
||||
}
|
||||
want := make(map[int64]struct{}, len(ids))
|
||||
for _, id := range ids {
|
||||
want[id] = struct{}{}
|
||||
}
|
||||
out := make([]store.Account, 0, len(ids))
|
||||
for _, a := range accs {
|
||||
if _, ok := want[a.ID]; ok {
|
||||
out = append(out, a)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func gateOK(accs []store.Account) bool {
|
||||
if len(accs) == 0 {
|
||||
return false
|
||||
@@ -173,7 +283,7 @@ func shouldBreak(trigger string, totErr int64) bool {
|
||||
return trigger == "scheduled" && totErr > 0
|
||||
}
|
||||
|
||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (int64, error) {
|
||||
func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string, accountIDs []int64) (int64, error) {
|
||||
task, err := o.store.GetTask(ctx, taskID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
@@ -182,6 +292,12 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (i
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
accs = selectAccounts(accs, accountIDs)
|
||||
// A non-empty request that matched nothing is a client error, distinct
|
||||
// from "not tested".
|
||||
if len(accountIDs) > 0 && len(accs) == 0 {
|
||||
return 0, ErrNoAccountsSelected
|
||||
}
|
||||
if !gateOK(accs) {
|
||||
return 0, ErrNotTested
|
||||
}
|
||||
@@ -204,11 +320,21 @@ func (o *Orchestrator) Run(ctx context.Context, taskID int64, trigger string) (i
|
||||
}
|
||||
o.hub.Publish(wshub.Event{Type: "run_started", TaskID: taskID, Data: map[string]any{"run_id": runID}})
|
||||
|
||||
go o.runAll(context.WithoutCancel(ctx), task, runID, accs, srcEP, dstEP, trigger)
|
||||
// dbCtx outlives the request so status/counter writes still land after a
|
||||
// pause or cancel; runCtx is what Pause/Cancel actually stop, and every
|
||||
// account's IMAP work hangs off it.
|
||||
dbCtx := context.WithoutCancel(ctx)
|
||||
runCtx, runCancel := context.WithCancel(dbCtx)
|
||||
h := &runHandle{cancel: runCancel}
|
||||
o.registerRun(taskID, h)
|
||||
|
||||
go o.runAll(dbCtx, runCtx, h, task, runID, accs, srcEP, dstEP, trigger)
|
||||
return runID, nil
|
||||
}
|
||||
|
||||
func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
|
||||
func (o *Orchestrator) runAll(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, accs []store.Account, srcEP, dstEP imapx.Endpoint, trigger string) {
|
||||
defer o.unregisterRun(task.ID)
|
||||
defer h.cancel()
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
slog.Error("run coordinator panicked", "task", task.ID, "run", runID, "panic", r)
|
||||
@@ -229,7 +355,17 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
||||
sem := make(chan struct{}, o.concurrency)
|
||||
var wg sync.WaitGroup
|
||||
|
||||
for _, a := range accs {
|
||||
for i, a := range accs {
|
||||
// Stopped mid-queue: the accounts that never started are marked with the
|
||||
// same status as the ones that were interrupted, so a pause leaves the
|
||||
// whole remainder resumable and a cancel leaves it cancelled.
|
||||
if runCtx.Err() != nil {
|
||||
st := h.stopReason().accountStatus()
|
||||
for _, rest := range accs[i:] {
|
||||
_ = o.store.SetAccountStatus(ctx, rest.ID, st)
|
||||
}
|
||||
break
|
||||
}
|
||||
wg.Add(1)
|
||||
sem <- struct{}{}
|
||||
go func(a store.Account) {
|
||||
@@ -246,7 +382,7 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
||||
mu.Unlock()
|
||||
}
|
||||
}()
|
||||
c, s, e := o.runAccount(ctx, task, runID, a, srcEP, dstEP)
|
||||
c, s, e := o.runAccount(ctx, runCtx, h, task, runID, a, srcEP, dstEP)
|
||||
mu.Lock()
|
||||
totCopied += c
|
||||
totSkipped += s
|
||||
@@ -256,23 +392,32 @@ func (o *Orchestrator) runAll(ctx context.Context, task store.Task, runID int64,
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
reason := h.stopReason()
|
||||
status := "done"
|
||||
if totErr > 0 {
|
||||
switch {
|
||||
case reason == stopPaused:
|
||||
status = "paused"
|
||||
case reason == stopCancelled:
|
||||
status = "cancelled"
|
||||
case totErr > 0:
|
||||
status = "done_with_errors"
|
||||
}
|
||||
_ = o.store.FinishRun(ctx, runID, status, totCopied, totSkipped, totErr)
|
||||
_ = o.store.SetTaskStatus(ctx, task.ID, status)
|
||||
o.hub.Publish(wshub.Event{Type: "run_done", TaskID: task.ID,
|
||||
Data: map[string]any{"run_id": runID, "copied": totCopied, "skipped": totSkipped, "errors": totErr}})
|
||||
Data: map[string]any{"run_id": runID, "status": status,
|
||||
"copied": totCopied, "skipped": totSkipped, "errors": totErr}})
|
||||
|
||||
if shouldBreak(trigger, totErr) {
|
||||
// An operator stopping the run is not a schedule failure, so leave the
|
||||
// breaker alone even when the accounts that did run reported errors.
|
||||
if reason == stopNone && shouldBreak(trigger, totErr) {
|
||||
_ = o.store.SetTaskBroken(ctx, task.ID)
|
||||
o.hub.Publish(wshub.Event{Type: "task_broken", TaskID: task.ID,
|
||||
Data: map[string]any{"task_id": task.ID, "errors": totErr}})
|
||||
}
|
||||
}
|
||||
|
||||
func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
|
||||
func (o *Orchestrator) runAccount(ctx, runCtx context.Context, h *runHandle, task store.Task, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint) (int64, int64, int64) {
|
||||
o.hub.Publish(wshub.Event{Type: "account_started", TaskID: task.ID, Data: map[string]any{
|
||||
"account_id": a.ID,
|
||||
"src_login": a.SrcLogin, "src_host": srcEP.Host, "src_port": srcEP.Port,
|
||||
@@ -283,10 +428,11 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
_ = o.store.ResetAccountCounters(ctx, a.ID) // start from zero; IncAccountCounters is additive
|
||||
_ = o.store.ClearAccountErrors(ctx, a.ID) // drop last run's per-error rows
|
||||
|
||||
// Per-account cancellable context: IMAP work uses actx (so CancelAccount
|
||||
// stops it); DB writes keep the parent ctx so status/counters persist even
|
||||
// after cancellation. ctx is context.WithoutCancel from runAll.
|
||||
actx, cancel := context.WithCancel(ctx)
|
||||
// Per-account cancellable context: IMAP work uses actx, so both CancelAccount
|
||||
// and a task-wide pause/cancel (which cancels runCtx) stop it. DB writes keep
|
||||
// ctx — the uncancellable one from runAll — so status/counters persist even
|
||||
// after cancellation.
|
||||
actx, cancel := context.WithCancel(runCtx)
|
||||
o.registerCancel(a.ID, cancel)
|
||||
defer func() {
|
||||
o.unregisterCancel(a.ID)
|
||||
@@ -295,38 +441,77 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
|
||||
srcPass, err := crypto.Decrypt(o.encKey, a.SrcPassEnc)
|
||||
if err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
}
|
||||
dstPass, err := crypto.Decrypt(o.encKey, a.DstPassEnc)
|
||||
if err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
}
|
||||
|
||||
src, err := imapx.Connect(actx, srcEP)
|
||||
if err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
}
|
||||
defer func() { _ = src.Logout().Wait() }()
|
||||
if err := src.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
_ = src.Logout().Wait()
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
}
|
||||
// srcClient holds the LIVE source connection. A body-read timeout (server
|
||||
// under-delivering a literal) forces a mid-run reconnect via reconnectSrc,
|
||||
// which swaps this pointer. The cancel goroutine and the deferred logout
|
||||
// below both read through it, so they always act on the current connection
|
||||
// rather than a stale one that was already replaced and logged out.
|
||||
var srcClient atomic.Pointer[imapx.Client]
|
||||
srcClient.Store(src)
|
||||
defer func() { _ = srcClient.Load().Logout().Wait() }()
|
||||
|
||||
dst, err := imapx.Connect(actx, dstEP)
|
||||
if err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
}
|
||||
defer func() { _ = dst.Logout().Wait() }()
|
||||
if err := dst.Login(a.DstLogin, string(dstPass)).Wait(); err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "dst", err)
|
||||
}
|
||||
|
||||
// reconnectSrc dials and logs in a fresh source client, swaps it in as the
|
||||
// live connection, and logs the old (desynced) one out. CopyFolder calls it
|
||||
// to recover after a message body times out: the server left the connection
|
||||
// mid-literal, so it can't be reused. Bound to actx, so a cancelled account
|
||||
// fails the dial instead of reconnecting.
|
||||
reconnectSrc := func() (*imapx.Client, error) {
|
||||
nc, err := imapx.Connect(actx, srcEP)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := nc.Login(a.SrcLogin, string(srcPass)).Wait(); err != nil {
|
||||
_ = nc.Logout().Wait()
|
||||
return nil, err
|
||||
}
|
||||
if old := srcClient.Swap(nc); old != nil {
|
||||
_ = old.Logout().Wait()
|
||||
}
|
||||
slog.Info("reconnected src after message body timeout", "account", a.ID, "src_login", a.SrcLogin)
|
||||
return nc, nil
|
||||
}
|
||||
|
||||
// On cancel, close the connections so any in-flight network read (a slow
|
||||
// FETCH/Collect that ctx.Err() checks can't interrupt) unblocks immediately.
|
||||
go func() {
|
||||
<-actx.Done()
|
||||
_ = src.Close()
|
||||
_ = srcClient.Load().Close()
|
||||
_ = dst.Close()
|
||||
}()
|
||||
|
||||
// Keep the destination connection warm. It sits idle for the whole
|
||||
// source-side metadata scan (Pass 1 of CopyFolder), which on a large
|
||||
// mailbox runs for minutes; without traffic the server drops it and every
|
||||
// subsequent APPEND fails with "use of closed network connection", copying
|
||||
// nothing. Periodic NOOPs prevent that. Only dst needs it — src is
|
||||
// continuously busy scanning/fetching. Bound to actx so it stops with the
|
||||
// account.
|
||||
go imapx.Keepalive(actx, dst, imapx.KeepaliveInterval)
|
||||
|
||||
// Progress watchdog: track the last time we saw scan/copy activity; if it
|
||||
// goes quiet for stallTimeout, cancel the account so the connections close
|
||||
// and this worker unwinds (it would otherwise block forever on a silent
|
||||
@@ -356,7 +541,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
folders, err := imapx.ListFolders(src)
|
||||
touch()
|
||||
if err != nil {
|
||||
return o.accountFailed(ctx, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
return o.accountFailed(ctx, runCtx, h, task.ID, runID, a, srcEP, dstEP, "src", err)
|
||||
}
|
||||
|
||||
// Planning pass: decide folders from the account's own config, then EXAMINE
|
||||
@@ -405,6 +590,14 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
IsMigrated: func(k string) (bool, error) { return o.store.IsMigrated(ctx, a.ID, k) },
|
||||
MarkMigrated: func(folder, k string) error { return o.store.MarkMigrated(ctx, a.ID, folder, k) },
|
||||
OnError: func(ref, msg string) { addErr("message", curFolder, ref, msg) },
|
||||
// Fires as bytes move within a single message's FETCH/APPEND, so the
|
||||
// stall watchdog sees a large-but-live transfer as progress instead of
|
||||
// cancelling it as a wedged connection.
|
||||
OnActivity: touch,
|
||||
// Recovers from a message body timeout (server under-delivering a
|
||||
// literal) by swapping in a fresh source connection so the folder can
|
||||
// resume with the remaining messages.
|
||||
ReconnectSrc: reconnectSrc,
|
||||
OnProgress: func(c, s int) {
|
||||
touch()
|
||||
now := time.Now()
|
||||
@@ -451,7 +644,7 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
if actx.Err() != nil {
|
||||
break // cancelled — stop scheduling more folders
|
||||
}
|
||||
res, err := imapx.CopyFolder(actx, src, dst, fp.src, fp.dst, deps)
|
||||
res, err := imapx.CopyFolder(actx, srcClient.Load(), dst, fp.src, fp.dst, deps)
|
||||
folderErr := int64(0)
|
||||
if err != nil && actx.Err() == nil {
|
||||
slog.Warn("folder copy error", "account", a.ID, "src_login", a.SrcLogin, "folder", fp.src, "err", err)
|
||||
@@ -473,11 +666,18 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
}
|
||||
|
||||
if actx.Err() != nil {
|
||||
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
|
||||
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: task.ID,
|
||||
// A task-wide pause leaves the account resumable; anything else (per-account
|
||||
// cancel, stall watchdog, task-wide cancel) leaves it cancelled.
|
||||
st := "cancelled"
|
||||
if runCtx.Err() != nil {
|
||||
st = h.stopReason().accountStatus()
|
||||
}
|
||||
_ = o.store.SetAccountStatus(ctx, a.ID, st)
|
||||
o.hub.Publish(wshub.Event{Type: st, TaskID: task.ID,
|
||||
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin,
|
||||
"copied": copied, "skipped": skipped, "errors": errs}})
|
||||
slog.Info("account cancelled", "account", a.ID, "src_login", a.SrcLogin, "copied", copied, "skipped", skipped)
|
||||
slog.Info("account stopped", "account", a.ID, "src_login", a.SrcLogin,
|
||||
"status", st, "copied", copied, "skipped", skipped)
|
||||
return copied, skipped, errs
|
||||
}
|
||||
|
||||
@@ -493,11 +693,16 @@ func (o *Orchestrator) runAccount(ctx context.Context, task store.Task, runID in
|
||||
return copied, skipped, errs
|
||||
}
|
||||
|
||||
func (o *Orchestrator) accountFailed(ctx context.Context, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
|
||||
// A cancellation surfacing as an error is a cancel, not a failure.
|
||||
func (o *Orchestrator) accountFailed(ctx, runCtx context.Context, h *runHandle, taskID, runID int64, a store.Account, srcEP, dstEP imapx.Endpoint, side string, err error) (int64, int64, int64) {
|
||||
// A cancellation surfacing as an error is a stop, not a failure — and a
|
||||
// task-wide pause must still leave the account resumable.
|
||||
if errors.Is(err, context.Canceled) {
|
||||
_ = o.store.SetAccountStatus(ctx, a.ID, "cancelled")
|
||||
o.hub.Publish(wshub.Event{Type: "cancelled", TaskID: taskID,
|
||||
st := "cancelled"
|
||||
if runCtx.Err() != nil {
|
||||
st = h.stopReason().accountStatus()
|
||||
}
|
||||
_ = o.store.SetAccountStatus(ctx, a.ID, st)
|
||||
o.hub.Publish(wshub.Event{Type: st, TaskID: taskID,
|
||||
Data: map[string]any{"account_id": a.ID, "src_login": a.SrcLogin}})
|
||||
return 0, 0, 0
|
||||
}
|
||||
|
||||
@@ -22,3 +22,23 @@ func TestGateOK(t *testing.T) {
|
||||
t.Fatal("empty accounts must fail gate")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectAccounts(t *testing.T) {
|
||||
accs := []store.Account{{ID: 1}, {ID: 2}, {ID: 3}}
|
||||
|
||||
if got := selectAccounts(accs, nil); len(got) != 3 {
|
||||
t.Fatalf("nil ids must return all, got %d", len(got))
|
||||
}
|
||||
if got := selectAccounts(accs, []int64{}); len(got) != 3 {
|
||||
t.Fatalf("empty ids must return all, got %d", len(got))
|
||||
}
|
||||
|
||||
got := selectAccounts(accs, []int64{3, 1})
|
||||
if len(got) != 2 || got[0].ID != 1 || got[1].ID != 3 {
|
||||
t.Fatalf("must keep matching ids in input order, got %+v", got)
|
||||
}
|
||||
|
||||
if got := selectAccounts(accs, []int64{99}); len(got) != 0 {
|
||||
t.Fatalf("unknown ids must yield empty, got %d", len(got))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package orchestrator
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestStopReasonAccountStatus(t *testing.T) {
|
||||
if got := stopPaused.accountStatus(); got != "paused" {
|
||||
t.Fatalf("stopPaused = %q want paused", got)
|
||||
}
|
||||
if got := stopCancelled.accountStatus(); got != "cancelled" {
|
||||
t.Fatalf("stopCancelled = %q want cancelled", got)
|
||||
}
|
||||
// A run stopped without an operator reason (per-account cancel, stall
|
||||
// watchdog) must not look like a pause, or Resume would pick it up.
|
||||
if got := stopNone.accountStatus(); got != "cancelled" {
|
||||
t.Fatalf("stopNone = %q want cancelled", got)
|
||||
}
|
||||
}
|
||||
|
||||
// The first stop wins: a cancel arriving after a pause must not downgrade the
|
||||
// accounts a pause already promised to keep resumable, and vice versa.
|
||||
func TestRunHandleFirstStopWins(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
first, later stopReason
|
||||
}{
|
||||
{"pause then cancel", stopPaused, stopCancelled},
|
||||
{"cancel then pause", stopCancelled, stopPaused},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
h := &runHandle{cancel: cancel}
|
||||
h.stopWith(tc.first)
|
||||
h.stopWith(tc.later)
|
||||
if got := h.stopReason(); got != tc.first {
|
||||
t.Fatalf("reason = %v want %v", got, tc.first)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHandleStopCancelsContext(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
h := &runHandle{cancel: cancel}
|
||||
if ctx.Err() != nil {
|
||||
t.Fatal("context cancelled before stop")
|
||||
}
|
||||
h.stopWith(stopPaused)
|
||||
if ctx.Err() == nil {
|
||||
t.Fatal("stop must cancel the run context")
|
||||
}
|
||||
}
|
||||
|
||||
// Pause/Cancel report false for a task with no live run, which the HTTP layer
|
||||
// turns into 409 instead of pretending it stopped something.
|
||||
func TestStopRunWithoutLiveRun(t *testing.T) {
|
||||
o := &Orchestrator{runs: map[int64]*runHandle{}}
|
||||
if o.PauseTask(1) {
|
||||
t.Fatal("PauseTask must report false with no live run")
|
||||
}
|
||||
if o.CancelTask(1) {
|
||||
t.Fatal("CancelTask must report false with no live run")
|
||||
}
|
||||
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
h := &runHandle{cancel: cancel}
|
||||
o.registerRun(1, h)
|
||||
if !o.PauseTask(1) {
|
||||
t.Fatal("PauseTask must report true for a live run")
|
||||
}
|
||||
if got := h.stopReason(); got != stopPaused {
|
||||
t.Fatalf("reason = %v want stopPaused", got)
|
||||
}
|
||||
o.unregisterRun(1)
|
||||
if o.CancelTask(1) {
|
||||
t.Fatal("unregistered run must not be stoppable")
|
||||
}
|
||||
}
|
||||
@@ -67,7 +67,7 @@ func (s *Scheduler) tick(ctx context.Context) {
|
||||
return
|
||||
}
|
||||
for _, id := range dueTaskIDs(tasks, time.Now()) {
|
||||
if _, err := s.orch.Run(ctx, id, "scheduled"); err != nil {
|
||||
if _, err := s.orch.Run(ctx, id, "scheduled", nil); err != nil {
|
||||
// ErrAlreadyRunning / ErrNotTested are expected races/edge cases, not fatal.
|
||||
slog.Info("scheduler: run skipped", "task", id, "err", err)
|
||||
}
|
||||
|
||||
@@ -32,6 +32,21 @@ func (s *Store) CreateAccount(ctx context.Context, a Account) (int64, error) {
|
||||
return id, err
|
||||
}
|
||||
|
||||
// UpdateAccountCredentials replaces an account's logins and, when a new
|
||||
// ciphertext is supplied, its passwords; a nil password keeps the stored one so
|
||||
// the operator can fix only the side that failed. Both connection tests are
|
||||
// reset to "unknown" because the previous verdicts no longer describe these
|
||||
// credentials, which also forces a re-test before the account can run.
|
||||
func (s *Store) UpdateAccountCredentials(ctx context.Context, id int64, srcLogin, dstLogin string, srcPassEnc, dstPassEnc *string) error {
|
||||
_, err := s.Pool.Exec(ctx,
|
||||
`UPDATE accounts SET src_login=$2, dst_login=$3,
|
||||
src_pass_enc=COALESCE($4, src_pass_enc), dst_pass_enc=COALESCE($5, dst_pass_enc),
|
||||
test_src_status='unknown', test_dst_status='unknown', last_error=''
|
||||
WHERE id=$1`,
|
||||
id, srcLogin, dstLogin, srcPassEnc, dstPassEnc)
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteAccount removes one account (and its migrated_messages via ON DELETE CASCADE).
|
||||
func (s *Store) DeleteAccount(ctx context.Context, id int64) error {
|
||||
_, err := s.Pool.Exec(ctx, `DELETE FROM accounts WHERE id=$1`, id)
|
||||
|
||||
@@ -65,6 +65,47 @@ func TestResetAccountCounters(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Fixing an imported account's credentials must replace only what the operator
|
||||
// supplied: a nil password keeps the stored ciphertext, and both test verdicts
|
||||
// go back to unknown because they described the old credentials.
|
||||
func TestUpdateAccountCredentials(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ctx := context.Background()
|
||||
epSrc, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||
epDst, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "dst", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||
taskID, _ := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: epSrc, DstEndpointID: epDst})
|
||||
accID, _ := s.CreateAccount(ctx, Account{TaskID: taskID, SrcLogin: "u", SrcPassEnc: "oldsrc", DstLogin: "u2", DstPassEnc: "olddst"})
|
||||
_ = s.SetAccountTestStatus(ctx, accID, "src", "fail")
|
||||
_ = s.SetAccountTestStatus(ctx, accID, "dst", "ok")
|
||||
_ = s.SetAccountError(ctx, accID, "authentication failed")
|
||||
|
||||
newSrc := "newsrc"
|
||||
if err := s.UpdateAccountCredentials(ctx, accID, "u@src.example", "u@dst.example", &newSrc, nil); err != nil {
|
||||
t.Fatalf("update: %v", err)
|
||||
}
|
||||
|
||||
accs, _ := s.ListAccountsByTask(ctx, taskID)
|
||||
if len(accs) != 1 {
|
||||
t.Fatalf("len=%d want 1", len(accs))
|
||||
}
|
||||
a := accs[0]
|
||||
if a.SrcLogin != "u@src.example" || a.DstLogin != "u@dst.example" {
|
||||
t.Fatalf("logins not updated: %q / %q", a.SrcLogin, a.DstLogin)
|
||||
}
|
||||
if a.SrcPassEnc != "newsrc" {
|
||||
t.Fatalf("src password not updated: %q", a.SrcPassEnc)
|
||||
}
|
||||
if a.DstPassEnc != "olddst" {
|
||||
t.Fatalf("nil password must keep the stored one, got %q", a.DstPassEnc)
|
||||
}
|
||||
if a.TestSrcStatus != "unknown" || a.TestDstStatus != "unknown" {
|
||||
t.Fatalf("test statuses not reset: %q / %q", a.TestSrcStatus, a.TestDstStatus)
|
||||
}
|
||||
if a.LastError != "" {
|
||||
t.Fatalf("last_error not cleared: %q", a.LastError)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSetAccountFolderMapping(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
@@ -18,6 +18,41 @@ func TestUpdateEndpoint(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteEndpoint(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ctx := context.Background()
|
||||
id, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "src", Host: "a.com", Port: 993, TLSMode: "ssl"})
|
||||
if err := s.DeleteEndpoint(ctx, id); err != nil {
|
||||
t.Fatalf("delete: %v", err)
|
||||
}
|
||||
eps, _ := s.ListEndpoints(ctx)
|
||||
if len(eps) != 0 {
|
||||
t.Fatalf("endpoint not deleted: %d remain", len(eps))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteEndpointUsedByTaskRefused(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ctx := context.Background()
|
||||
ep1, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "s", Host: "a", Port: 993, TLSMode: "ssl"})
|
||||
ep2, _ := s.CreateEndpoint(ctx, Endpoint{RoleLabel: "d", Host: "b", Port: 993, TLSMode: "ssl"})
|
||||
if _, err := s.CreateTask(ctx, Task{Name: "t", SrcEndpointID: ep1, DstEndpointID: ep2}); err != nil {
|
||||
t.Fatalf("create task: %v", err)
|
||||
}
|
||||
for _, id := range []int64{ep1, ep2} {
|
||||
n, err := s.CountTasksUsingEndpoint(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("count: %v", err)
|
||||
}
|
||||
if n != 1 {
|
||||
t.Fatalf("count for ep %d = %d, want 1", id, n)
|
||||
}
|
||||
if err := s.DeleteEndpoint(ctx, id); err == nil {
|
||||
t.Fatalf("delete of referenced endpoint %d succeeded, want FK violation", id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeleteAccountCascadesJournal(t *testing.T) {
|
||||
s := testStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
@@ -26,6 +26,23 @@ func (s *Store) UpdateEndpoint(ctx context.Context, e Endpoint) error {
|
||||
return err
|
||||
}
|
||||
|
||||
// CountTasksUsingEndpoint reports how many tasks reference the endpoint on
|
||||
// either side, so a delete can be refused with a meaningful message instead of
|
||||
// surfacing a raw foreign-key violation.
|
||||
func (s *Store) CountTasksUsingEndpoint(ctx context.Context, id int64) (int, error) {
|
||||
var n int
|
||||
err := s.Pool.QueryRow(ctx,
|
||||
`SELECT count(*) FROM tasks WHERE src_endpoint_id=$1 OR dst_endpoint_id=$1`, id).Scan(&n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// DeleteEndpoint removes an endpoint. Tasks reference endpoints without ON
|
||||
// DELETE CASCADE, so Postgres rejects the delete while any task still uses it.
|
||||
func (s *Store) DeleteEndpoint(ctx context.Context, id int64) error {
|
||||
_, err := s.Pool.Exec(ctx, `DELETE FROM endpoints WHERE id=$1`, id)
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) GetEndpoint(ctx context.Context, id int64) (Endpoint, error) {
|
||||
var e Endpoint
|
||||
err := s.Pool.QueryRow(ctx,
|
||||
|
||||
@@ -120,13 +120,16 @@ type SchedulableTask struct {
|
||||
}
|
||||
|
||||
// ListSchedulableTasks returns tasks eligible to auto-run: schedule on, not
|
||||
// broken, not currently running — each joined with its last finished run time.
|
||||
// broken, neither running nor paused — each joined with its last finished run
|
||||
// time. A paused task waits for the operator to resume it; auto-starting a full
|
||||
// run behind their back would defeat the pause.
|
||||
func (s *Store) ListSchedulableTasks(ctx context.Context) ([]SchedulableTask, error) {
|
||||
rows, err := s.Pool.Query(ctx,
|
||||
`SELECT t.id, t.schedule_interval_seconds, t.schedule_anchor,
|
||||
(SELECT max(finished_at) FROM runs r WHERE r.task_id=t.id AND r.finished_at IS NOT NULL)
|
||||
FROM tasks t
|
||||
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken AND t.status <> 'running'`)
|
||||
WHERE t.schedule_interval_seconds > 0 AND NOT t.broken
|
||||
AND t.status <> 'running' AND t.status <> 'paused'`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
+94
-9
@@ -158,7 +158,8 @@ wait_test_ok() {
|
||||
wait_test_ok
|
||||
|
||||
wait_run_done() {
|
||||
for ((i = 1; i <= 60; i++)); do
|
||||
# Generous: the resume scenario re-scans and copies thousands of messages.
|
||||
for ((i = 1; i <= 600; i++)); do
|
||||
local status
|
||||
status=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
|
||||
if [[ "$status" == "done" ]]; then
|
||||
@@ -185,16 +186,100 @@ log "POST /run (second run, expect idempotency)"
|
||||
api POST "/api/tasks/${TASK_ID}/run" >/dev/null
|
||||
wait_run_done
|
||||
|
||||
# Counters are reset at the start of every run, so run 2's row shows run 2
|
||||
# alone: it must copy nothing and skip what run 1 already migrated.
|
||||
RES2=$(api GET "/api/tasks/${TASK_ID}")
|
||||
RUN2_COPIED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].copied')
|
||||
RUN2_SKIPPED_TOTAL=$(echo "$RES2" | jq -r '.accounts[0].skipped')
|
||||
RUN2_COPIED=$(echo "$RES2" | jq -r '.accounts[0].copied')
|
||||
RUN2_SKIPPED=$(echo "$RES2" | jq -r '.accounts[0].skipped')
|
||||
RUN2_ERRORS=$(echo "$RES2" | jq -r '.accounts[0].errors')
|
||||
RUN2_COPIED_DELTA=$((RUN2_COPIED_TOTAL - RUN1_COPIED))
|
||||
RUN2_SKIPPED_DELTA=$((RUN2_SKIPPED_TOTAL - RUN1_SKIPPED))
|
||||
log "run 2: copied_delta=$RUN2_COPIED_DELTA skipped_delta=$RUN2_SKIPPED_DELTA errors=$RUN2_ERRORS"
|
||||
log "run 2: copied=$RUN2_COPIED skipped=$RUN2_SKIPPED errors=$RUN2_ERRORS"
|
||||
|
||||
[[ "$RUN2_ERRORS" == "0" ]] || fail "run 2 had errors"
|
||||
[[ "$RUN2_COPIED_DELTA" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED_DELTA new messages (expected 0, not idempotent)"
|
||||
[[ "$RUN2_SKIPPED_DELTA" -gt 0 ]] || fail "run 2 skipped delta is $RUN2_SKIPPED_DELTA (expected >0)"
|
||||
[[ "$RUN2_COPIED" -eq 0 ]] || fail "run 2 copied $RUN2_COPIED new messages (expected 0, not idempotent)"
|
||||
[[ "$RUN2_SKIPPED" -eq "$RUN1_COPIED" ]] ||
|
||||
fail "run 2 skipped $RUN2_SKIPPED of the $RUN1_COPIED messages run 1 copied"
|
||||
|
||||
log "PASS: run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED_DELTA skipped=$RUN2_SKIPPED_DELTA (idempotent)"
|
||||
log "run1 copied=$RUN1_COPIED skipped=$RUN1_SKIPPED; run2 copied=$RUN2_COPIED skipped=$RUN2_SKIPPED (idempotent)"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pause / resume: a second account with enough messages that the run is still
|
||||
# in flight when the pause lands. Pausing must leave the account resumable, and
|
||||
# resuming must finish it without re-copying what the first stretch already did.
|
||||
# ---------------------------------------------------------------------------
|
||||
SRC_USER2="src2@example.com"
|
||||
DST_USER2="dst2@example.com"
|
||||
# Large enough that the copy is still in flight when the pause lands — greenmail
|
||||
# on a local socket copies well over a thousand small messages per second.
|
||||
SEED_COUNT=3000
|
||||
|
||||
log "seeding ${SEED_COUNT} messages into ${SRC_USER2} INBOX (pause/resume scenario)"
|
||||
python3 "$SEED_PY" 127.0.0.1 3143 "$SRC_USER2" "$MAIL_PASS" "$SEED_COUNT"
|
||||
|
||||
log "adding second account (src2 -> dst2)"
|
||||
ACCOUNT2_ID=$(api POST "/api/tasks/${TASK_ID}/accounts" \
|
||||
"{\"src_login\":\"${SRC_USER2}\",\"src_pass\":\"${MAIL_PASS}\",\"dst_login\":\"${DST_USER2}\",\"dst_pass\":\"${MAIL_PASS}\"}" | jq -r .id)
|
||||
[[ "$ACCOUNT2_ID" =~ ^[0-9]+$ ]] || fail "bad second account id: $ACCOUNT2_ID"
|
||||
log "account2_id=$ACCOUNT2_ID"
|
||||
|
||||
log "POST /test (both accounts)"
|
||||
api POST "/api/tasks/${TASK_ID}/test" >/dev/null
|
||||
for ((i = 1; i <= 30; i++)); do
|
||||
BOTH_OK=$(api GET "/api/tasks/${TASK_ID}" |
|
||||
jq -r '[.accounts[] | select(.test_src_status=="ok" and .test_dst_status=="ok")] | length')
|
||||
[[ "$BOTH_OK" == "2" ]] && break
|
||||
sleep 1
|
||||
done
|
||||
[[ "$BOTH_OK" == "2" ]] || fail "second account did not pass connection tests (ok count=$BOTH_OK)"
|
||||
|
||||
# Account view for account2, by id.
|
||||
acct2() { api GET "/api/tasks/${TASK_ID}" | jq -r ".accounts[] | select(.id==${ACCOUNT2_ID}) | $1"; }
|
||||
|
||||
log "POST /run (account2 only)"
|
||||
api POST "/api/tasks/${TASK_ID}/run" "{\"account_ids\":[${ACCOUNT2_ID}]}" >/dev/null
|
||||
|
||||
# Per-account counters are only written to the DB when a folder completes, so
|
||||
# "copied so far" is invisible here — wait for the account to go running, give
|
||||
# the copy a few seconds of real work, then pause mid-folder.
|
||||
log "waiting for account2 to start running"
|
||||
for ((i = 1; i <= 120; i++)); do
|
||||
[[ "$(acct2 .status)" == "running" ]] && break
|
||||
sleep 0.5
|
||||
done
|
||||
[[ "$(acct2 .status)" == "running" ]] || fail "account2 never reached running"
|
||||
|
||||
log "letting it copy for a few seconds, then pausing mid-folder"
|
||||
sleep 5
|
||||
curl -fsS -b "$COOKIE_JAR" -c "$COOKIE_JAR" -X POST "$BASE/api/tasks/${TASK_ID}/pause" >/dev/null ||
|
||||
fail "pause rejected — the run finished before the pause landed, seed more messages"
|
||||
|
||||
log "waiting for the task to settle into paused"
|
||||
for ((i = 1; i <= 60; i++)); do
|
||||
TASK_STATUS=$(api GET "/api/tasks/${TASK_ID}" | jq -r '.task.status')
|
||||
[[ "$TASK_STATUS" == "paused" ]] && break
|
||||
sleep 1
|
||||
done
|
||||
[[ "$TASK_STATUS" == "paused" ]] || fail "task status=$TASK_STATUS after pause (expected paused)"
|
||||
|
||||
ACC2_STATUS=$(acct2 .status)
|
||||
[[ "$ACC2_STATUS" == "paused" ]] || fail "account2 status=$ACC2_STATUS after pause (expected paused)"
|
||||
log "paused (folder-level counters at copied=$(acct2 .copied) of $SEED_COUNT)"
|
||||
|
||||
log "POST /resume"
|
||||
api POST "/api/tasks/${TASK_ID}/resume" >/dev/null
|
||||
wait_run_done
|
||||
|
||||
RESUMED_COPIED=$(acct2 .copied)
|
||||
RESUMED_SKIPPED=$(acct2 .skipped)
|
||||
RESUMED_ERRORS=$(acct2 .errors)
|
||||
RESUMED_TOTAL=$((RESUMED_COPIED + RESUMED_SKIPPED))
|
||||
log "after resume: copied=$RESUMED_COPIED skipped=$RESUMED_SKIPPED errors=$RESUMED_ERRORS"
|
||||
[[ "$RESUMED_ERRORS" == "0" ]] || fail "resumed run had errors"
|
||||
# Counters reset per run, so the resumed run alone must account for every
|
||||
# message: the ones it copied now plus the ones the paused stretch already did.
|
||||
[[ "$RESUMED_TOTAL" -eq "$SEED_COUNT" ]] || fail "resumed run covered $RESUMED_TOTAL of $SEED_COUNT messages"
|
||||
# Non-zero skipped is the proof that the paused stretch's work survived: those
|
||||
# messages are in the migration journal, so the resume did not re-copy them.
|
||||
[[ "$RESUMED_SKIPPED" -gt 0 ]] ||
|
||||
fail "resumed run skipped nothing — the paused stretch's progress was lost"
|
||||
|
||||
log "PASS: idempotent re-run; pause was resumable, resume re-copied $RESUMED_COPIED and skipped $RESUMED_SKIPPED of $SEED_COUNT"
|
||||
|
||||
+30
-1
@@ -82,11 +82,21 @@ export const updateEndpoint = (
|
||||
body: { role_label: string; host: string; port: number; tls_mode: TLSMode },
|
||||
) => api(`/api/endpoints/${id}`, { ...jsonBody(body), method: 'PUT' })
|
||||
|
||||
export const deleteEndpoint = (id: number) => api(`/api/endpoints/${id}`, { method: 'DELETE' })
|
||||
|
||||
export const deleteTask = (id: number) => api(`/api/tasks/${id}`, { method: 'DELETE' })
|
||||
|
||||
export const deleteAccount = (taskId: number, accountId: number) =>
|
||||
api(`/api/tasks/${taskId}/accounts/${accountId}`, { method: 'DELETE' })
|
||||
|
||||
// Empty password fields keep the stored ones; both connection tests reset to
|
||||
// unknown server-side, so the account must be re-tested afterwards.
|
||||
export const updateAccountCredentials = (
|
||||
taskId: number,
|
||||
accountId: number,
|
||||
body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string },
|
||||
) => api(`/api/tasks/${taskId}/accounts/${accountId}/credentials`, { ...jsonBody(body), method: 'PUT' })
|
||||
|
||||
export const cancelAccount = (taskId: number, accountId: number) =>
|
||||
api(`/api/tasks/${taskId}/accounts/${accountId}/cancel`, { method: 'POST' })
|
||||
|
||||
@@ -138,7 +148,16 @@ export const createAccount = (
|
||||
|
||||
export const testAccounts = (id: number) => api(`/api/tasks/${id}/test`, { method: 'POST' })
|
||||
|
||||
export const runTask = (id: number) => api(`/api/tasks/${id}/run`, { method: 'POST' })
|
||||
export const runTask = (id: number, accountIds?: number[]) =>
|
||||
api(`/api/tasks/${id}/run`, accountIds?.length ? jsonBody({ account_ids: accountIds }) : { method: 'POST' })
|
||||
|
||||
// Pause stops the run but leaves its unfinished accounts resumable; cancel ends
|
||||
// it and marks them cancelled. Resume re-runs exactly the paused accounts.
|
||||
export const pauseTask = (id: number) => api(`/api/tasks/${id}/pause`, { method: 'POST' })
|
||||
|
||||
export const cancelTask = (id: number) => api(`/api/tasks/${id}/cancel`, { method: 'POST' })
|
||||
|
||||
export const resumeTask = (id: number) => api<{ run_id: number }>(`/api/tasks/${id}/resume`, { method: 'POST' })
|
||||
|
||||
export interface Run {
|
||||
id: number
|
||||
@@ -176,3 +195,13 @@ export const importCSV = (id: number, file: File) => {
|
||||
fd.append('file', file)
|
||||
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
|
||||
}
|
||||
|
||||
// A Kerio Connect export carries no domain, so the operator supplies it here;
|
||||
// the login and password apply to both sides of the migration.
|
||||
export const importKerioCSV = (id: number, file: File, domain: string) => {
|
||||
const fd = new FormData()
|
||||
fd.append('file', file)
|
||||
fd.append('format', 'kerio')
|
||||
fd.append('domain', domain)
|
||||
return api<{ imported: number }>(`/api/tasks/${id}/import`, { method: 'POST', body: fd })
|
||||
}
|
||||
|
||||
+59
-2
@@ -488,11 +488,23 @@
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.map-toolbar {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 14px;
|
||||
margin-bottom: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.map-toolbar .btn {
|
||||
padding: 6px 12px;
|
||||
font-size: 11px;
|
||||
}
|
||||
|
||||
.map-all {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 12px;
|
||||
font-size: 11px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.08em;
|
||||
@@ -655,6 +667,25 @@ table.tbl a.rowlink:focus-visible {
|
||||
|
||||
/* ---------- status badges ---------- */
|
||||
|
||||
/* A badge that opens a dialog: the badge keeps its own look, the button only
|
||||
contributes the affordance. */
|
||||
.badge-btn {
|
||||
padding: 0;
|
||||
border: 0;
|
||||
background: none;
|
||||
font: inherit;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.badge-btn:hover .badge {
|
||||
filter: brightness(1.25);
|
||||
}
|
||||
|
||||
.badge-btn:focus-visible {
|
||||
outline: 1px solid var(--accent);
|
||||
outline-offset: 2px;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
@@ -905,11 +936,24 @@ table.tbl a.rowlink:hover {
|
||||
|
||||
.upload-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
align-items: flex-start;
|
||||
gap: 12px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
/* One import route per column: the action on top, its sample file underneath. */
|
||||
.upload-item {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.upload-item .link-btn {
|
||||
align-self: center;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.file-btn {
|
||||
position: relative;
|
||||
overflow: hidden;
|
||||
@@ -1056,3 +1100,16 @@ table.tbl a.rowlink:hover {
|
||||
padding: 12px 14px;
|
||||
}
|
||||
}
|
||||
|
||||
.tbl .chk-col {
|
||||
width: 32px;
|
||||
text-align: center;
|
||||
padding-right: 0;
|
||||
}
|
||||
.tbl .chk-col input[type='checkbox'] {
|
||||
cursor: pointer;
|
||||
}
|
||||
.tbl .chk-col input[type='checkbox']:disabled {
|
||||
cursor: not-allowed;
|
||||
opacity: 0.4;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
import { useEffect, useState, type FormEvent } from 'react'
|
||||
import { Modal } from './Modal'
|
||||
import type { Account } from '../api'
|
||||
|
||||
type Props = {
|
||||
open: boolean
|
||||
busy: boolean
|
||||
account: Account | null
|
||||
onClose: () => void
|
||||
onSubmit: (body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) => void
|
||||
}
|
||||
|
||||
// Fixes the credentials of an account that failed its connection test — usually
|
||||
// a wrong password that came in through a CSV import. Passwords are never sent
|
||||
// back to the browser, so the fields start empty and an empty field means
|
||||
// "keep the stored password".
|
||||
export function AccountCredentialsModal({ open, busy, account, onClose, onSubmit }: Props) {
|
||||
const [srcLogin, setSrcLogin] = useState('')
|
||||
const [dstLogin, setDstLogin] = useState('')
|
||||
const [srcPass, setSrcPass] = useState('')
|
||||
const [dstPass, setDstPass] = useState('')
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!open || !account) return
|
||||
setSrcLogin(account.src_login)
|
||||
setDstLogin(account.dst_login)
|
||||
setSrcPass('')
|
||||
setDstPass('')
|
||||
setError(null)
|
||||
}, [open, account])
|
||||
|
||||
function submit(e: FormEvent) {
|
||||
e.preventDefault()
|
||||
if (srcLogin.trim() === '' || dstLogin.trim() === '') {
|
||||
setError('Both logins are required')
|
||||
return
|
||||
}
|
||||
setError(null)
|
||||
onSubmit({
|
||||
src_login: srcLogin.trim(),
|
||||
src_pass: srcPass,
|
||||
dst_login: dstLogin.trim(),
|
||||
dst_pass: dstPass,
|
||||
})
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal open={open} title={account ? `Edit credentials — ${account.src_login}` : 'Edit credentials'} onClose={onClose}>
|
||||
<form onSubmit={submit}>
|
||||
<p className="map-hint">
|
||||
Leave a password field empty to keep the stored one. Saving resets both connection tests, so re-run{' '}
|
||||
<strong>Test connections</strong> afterwards.
|
||||
</p>
|
||||
<div className="field-row">
|
||||
<div className="field">
|
||||
<label htmlFor="edit_src_login">Source login</label>
|
||||
<input
|
||||
id="edit_src_login"
|
||||
data-modal-autofocus
|
||||
value={srcLogin}
|
||||
onChange={(e) => setSrcLogin(e.target.value)}
|
||||
disabled={busy}
|
||||
required
|
||||
/>
|
||||
</div>
|
||||
<div className="field">
|
||||
<label htmlFor="edit_src_pass">Source password</label>
|
||||
<input
|
||||
id="edit_src_pass"
|
||||
type="password"
|
||||
value={srcPass}
|
||||
onChange={(e) => setSrcPass(e.target.value)}
|
||||
placeholder="unchanged"
|
||||
autoComplete="new-password"
|
||||
disabled={busy}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<div className="field-row">
|
||||
<div className="field">
|
||||
<label htmlFor="edit_dst_login">Destination login</label>
|
||||
<input
|
||||
id="edit_dst_login"
|
||||
value={dstLogin}
|
||||
onChange={(e) => setDstLogin(e.target.value)}
|
||||
disabled={busy}
|
||||
required
|
||||
/>
|
||||
</div>
|
||||
<div className="field">
|
||||
<label htmlFor="edit_dst_pass">Destination password</label>
|
||||
<input
|
||||
id="edit_dst_pass"
|
||||
type="password"
|
||||
value={dstPass}
|
||||
onChange={(e) => setDstPass(e.target.value)}
|
||||
placeholder="unchanged"
|
||||
autoComplete="new-password"
|
||||
disabled={busy}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
{error && <div className="error-banner">{error}</div>}
|
||||
<div className="modal-actions">
|
||||
<button type="button" className="btn" onClick={onClose} disabled={busy}>
|
||||
Cancel
|
||||
</button>
|
||||
<button className="btn btn-primary" disabled={busy}>
|
||||
{busy ? 'Saving…' : 'Save credentials'}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
</Modal>
|
||||
)
|
||||
}
|
||||
@@ -20,6 +20,21 @@ function defaultDst(src: string, dstFolders: string[], initial: Record<string, s
|
||||
return src
|
||||
}
|
||||
|
||||
// Exchange/Kerio special folders and their mailcow counterparts. Keyed by the
|
||||
// lowercased source name so casing differences between servers don't matter.
|
||||
const DEFAULT_TARGETS: Record<string, string> = {
|
||||
'deleted items': 'Trash',
|
||||
'deleted messages': 'Trash',
|
||||
'junk e-mail': 'Junk',
|
||||
'junk email': 'Junk',
|
||||
spam: 'Junk',
|
||||
'sent items': 'Sent',
|
||||
'sent messages': 'Sent',
|
||||
}
|
||||
|
||||
// Source folders with no counterpart on mailcow — unchecked by the defaults.
|
||||
const DEFAULT_EXCLUDED = new Set(['public folders'])
|
||||
|
||||
export function FolderMappingModal({
|
||||
open, srcFolders, dstFolders, initialMapping, initialExcluded, accountLabel, onConfirm, onCancel,
|
||||
}: Props) {
|
||||
@@ -31,16 +46,40 @@ export function FolderMappingModal({
|
||||
|
||||
// Options per select: all destination folders, plus the source name itself
|
||||
// (marked "create") when it does not already exist on the destination.
|
||||
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
|
||||
|
||||
// Options per select: all destination folders, plus any name not present there
|
||||
// — the source folder itself and the current selection — marked "create".
|
||||
const options = useMemo(() => {
|
||||
const set = new Set(dstFolders)
|
||||
return (src: string) => {
|
||||
return (src: string, current: string) => {
|
||||
const opts = [...dstFolders]
|
||||
if (!set.has(src)) opts.unshift(src)
|
||||
if (!set.has(current)) opts.unshift(current)
|
||||
if (!set.has(src) && src !== current) opts.unshift(src)
|
||||
return opts
|
||||
}
|
||||
}, [dstFolders])
|
||||
|
||||
const valueFor = (src: string) => choice[src] ?? defaultDst(src, dstFolders, initialMapping)
|
||||
// Collapse the Exchange/Kerio folder layout onto mailcow's in one click: the
|
||||
// per-account mapping is otherwise repetitive work when importing many users.
|
||||
function applyDefaults() {
|
||||
const nextChoice = { ...choice }
|
||||
const nextSynced = { ...synced }
|
||||
for (const src of srcFolders) {
|
||||
const key = src.trim().toLowerCase()
|
||||
if (DEFAULT_EXCLUDED.has(key)) {
|
||||
nextSynced[src] = false
|
||||
continue
|
||||
}
|
||||
const target = DEFAULT_TARGETS[key]
|
||||
if (target) {
|
||||
nextChoice[src] = target
|
||||
nextSynced[src] = true
|
||||
}
|
||||
}
|
||||
setChoice(nextChoice)
|
||||
setSynced(nextSynced)
|
||||
}
|
||||
|
||||
function confirm() {
|
||||
const mapping: Record<string, string> = {}
|
||||
@@ -70,6 +109,15 @@ export function FolderMappingModal({
|
||||
Route each source folder to an existing destination folder. Leaving a folder mapped to its own name
|
||||
creates it on the destination if missing (e.g. map <code>Спам</code> → <code>Spam</code> to avoid duplicates).
|
||||
</p>
|
||||
<div className="map-toolbar">
|
||||
<button
|
||||
type="button"
|
||||
className="btn btn-ghost"
|
||||
onClick={applyDefaults}
|
||||
title="Map Deleted Items → Trash, Junk E-mail → Junk, Sent Items → Sent and skip Public Folders"
|
||||
>
|
||||
By default
|
||||
</button>
|
||||
<label className="map-all">
|
||||
<input
|
||||
type="checkbox"
|
||||
@@ -81,6 +129,7 @@ export function FolderMappingModal({
|
||||
/>
|
||||
sync all folders
|
||||
</label>
|
||||
</div>
|
||||
<div className="map-grid">
|
||||
{srcFolders.map((src) => {
|
||||
const on = synced[src] !== false
|
||||
@@ -107,10 +156,10 @@ export function FolderMappingModal({
|
||||
disabled={!on}
|
||||
onChange={(e) => setChoice((c) => ({ ...c, [src]: e.target.value }))}
|
||||
>
|
||||
{options(src).map((f) => (
|
||||
{options(src, val).map((f) => (
|
||||
<option key={f} value={f}>
|
||||
{f}
|
||||
{f === src && !dstFolders.includes(src) ? ' (create)' : ''}
|
||||
{dstFolders.includes(f) ? '' : ' (create)'}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
import { useEffect, useState, type FormEvent } from 'react'
|
||||
import { Modal } from './Modal'
|
||||
|
||||
// A bare DNS domain: no scheme, no user part, no whitespace. Mirrors the
|
||||
// server-side check in csvimport.normalizeDomain so bad input is caught here.
|
||||
const domainRe = /^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/
|
||||
|
||||
type Props = {
|
||||
open: boolean
|
||||
busy: boolean
|
||||
onClose: () => void
|
||||
onSubmit: (file: File, domain: string) => void
|
||||
}
|
||||
|
||||
export function KerioImportModal({ open, busy, onClose, onSubmit }: Props) {
|
||||
const [domain, setDomain] = useState('')
|
||||
const [file, setFile] = useState<File | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
if (!open) return
|
||||
setDomain('')
|
||||
setFile(null)
|
||||
setError(null)
|
||||
}, [open])
|
||||
|
||||
function submit(e: FormEvent) {
|
||||
e.preventDefault()
|
||||
const d = domain.trim().toLowerCase()
|
||||
if (!domainRe.test(d)) {
|
||||
setError('Enter a bare domain, e.g. galaxyhotel.kz')
|
||||
return
|
||||
}
|
||||
if (!file) {
|
||||
setError('Choose the Kerio export file')
|
||||
return
|
||||
}
|
||||
setError(null)
|
||||
onSubmit(file, d)
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal open={open} title="Import from Kerio" onClose={onClose}>
|
||||
<form onSubmit={submit}>
|
||||
<p className="map-hint">
|
||||
The Kerio user export lists a login and its password but no domain. The domain you enter is appended to every
|
||||
login and used for both the source and the destination. Disabled accounts and <code>admin</code> are skipped.
|
||||
</p>
|
||||
<div className="field">
|
||||
<label htmlFor="kerio_domain">Mail domain</label>
|
||||
<input
|
||||
id="kerio_domain"
|
||||
data-modal-autofocus
|
||||
value={domain}
|
||||
onChange={(e) => setDomain(e.target.value)}
|
||||
placeholder="galaxyhotel.kz"
|
||||
disabled={busy}
|
||||
/>
|
||||
</div>
|
||||
<div className="field">
|
||||
<label htmlFor="kerio_file">Export file</label>
|
||||
<input
|
||||
id="kerio_file"
|
||||
type="file"
|
||||
accept=".csv,text/csv"
|
||||
onChange={(e) => setFile(e.target.files?.[0] ?? null)}
|
||||
disabled={busy}
|
||||
/>
|
||||
</div>
|
||||
{error && <div className="error-banner">{error}</div>}
|
||||
<div className="modal-actions">
|
||||
<button type="button" className="btn" onClick={onClose} disabled={busy}>
|
||||
Cancel
|
||||
</button>
|
||||
<button className="btn btn-primary" disabled={busy}>
|
||||
{busy ? 'Importing…' : 'Import'}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
</Modal>
|
||||
)
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import { useEffect, useState, type FormEvent } from 'react'
|
||||
import { createEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
|
||||
import { createEndpoint, deleteEndpoint, listEndpoints, updateEndpoint, type Endpoint, type TLSMode } from '../api'
|
||||
import { useConfirm } from '../components/ConfirmProvider'
|
||||
|
||||
const emptyForm = { role_label: '', host: '', port: '993', tls_mode: 'ssl' as TLSMode }
|
||||
|
||||
@@ -9,6 +10,7 @@ export function Endpoints() {
|
||||
const [editingId, setEditingId] = useState<number | null>(null)
|
||||
const [error, setError] = useState<string | null>(null)
|
||||
const [busy, setBusy] = useState(false)
|
||||
const confirm = useConfirm()
|
||||
|
||||
function startEdit(ep: Endpoint) {
|
||||
setEditingId(ep.id)
|
||||
@@ -29,6 +31,25 @@ export function Endpoints() {
|
||||
|
||||
useEffect(reload, [])
|
||||
|
||||
async function onDelete(ep: Endpoint) {
|
||||
const ok = await confirm({
|
||||
title: 'Delete endpoint',
|
||||
message: `Delete endpoint "${ep.role_label}" (${ep.host}:${ep.port})?`,
|
||||
confirmLabel: 'Delete',
|
||||
danger: true,
|
||||
})
|
||||
if (!ok) return
|
||||
setError(null)
|
||||
try {
|
||||
await deleteEndpoint(ep.id)
|
||||
// The form still edits a row that no longer exists — drop back to create mode.
|
||||
if (editingId === ep.id) cancelEdit()
|
||||
reload()
|
||||
} catch (e) {
|
||||
setError(e instanceof Error ? e.message : 'Failed to delete endpoint')
|
||||
}
|
||||
}
|
||||
|
||||
async function submit(e: FormEvent) {
|
||||
e.preventDefault()
|
||||
setBusy(true)
|
||||
@@ -159,6 +180,9 @@ export function Endpoints() {
|
||||
<td className="num-cell">
|
||||
<button type="button" className="link-btn" onClick={() => startEdit(ep)} disabled={busy}>
|
||||
edit
|
||||
</button>{' '}
|
||||
<button type="button" className="link-btn danger" onClick={() => onDelete(ep)} disabled={busy}>
|
||||
delete
|
||||
</button>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
+242
-23
@@ -1,11 +1,13 @@
|
||||
import { useEffect, useRef, useState, type ChangeEvent, type FormEvent } from 'react'
|
||||
import { cancelAccount, createAccount, deleteAccount, getTask, importCSV, probeAccountFolders, probeFolders, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, type TaskDetail as TaskDetailData } from '../api'
|
||||
import { cancelAccount, cancelTask, createAccount, deleteAccount, getTask, importCSV, importKerioCSV, pauseTask, probeAccountFolders, probeFolders, resumeTask, runTask, setAccountFolderMapping, setTaskSchedule, testAccounts, updateAccountCredentials, type Account, type TaskDetail as TaskDetailData } from '../api'
|
||||
import { connectTaskWS, type TaskEvent } from '../ws'
|
||||
import { StatusBadge } from '../components/StatusBadge'
|
||||
import { useConfirm } from '../components/ConfirmProvider'
|
||||
import { FolderMappingModal } from '../components/FolderMappingModal'
|
||||
import { RunLogModal } from '../components/RunLogModal'
|
||||
import { AccountErrorsModal } from '../components/AccountErrorsModal'
|
||||
import { KerioImportModal } from '../components/KerioImportModal'
|
||||
import { AccountCredentialsModal } from '../components/AccountCredentialsModal'
|
||||
|
||||
const emptyAccount = { src_login: '', src_pass: '', dst_login: '', dst_pass: '' }
|
||||
|
||||
@@ -57,6 +59,8 @@ function describeEvent(ev: TaskEvent): string {
|
||||
}
|
||||
case 'cancelled':
|
||||
return `CANCELLED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0}`
|
||||
case 'paused':
|
||||
return `PAUSED #${d.account_id} (${d.src_login}): copied ${d.copied ?? 0}, skipped ${d.skipped ?? 0} — resumable`
|
||||
case 'error': {
|
||||
const where = d.folder ? ` folder "${d.folder}"` : d.side ? ` (${d.side} ${at})` : ''
|
||||
return `ERROR #${d.account_id}${where}: ${d.error}`
|
||||
@@ -64,7 +68,7 @@ function describeEvent(ev: TaskEvent): string {
|
||||
case 'run_started':
|
||||
return `RUN started (run #${d.run_id})`
|
||||
case 'run_done':
|
||||
return `RUN finished: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
|
||||
return `RUN ${String(d.status ?? 'finished')}: copied ${d.copied}, skipped ${d.skipped}, errors ${d.errors}`
|
||||
default:
|
||||
return JSON.stringify(ev.data)
|
||||
}
|
||||
@@ -90,6 +94,9 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
const [live, setLive] = useState<Record<number, LiveProgress>>({})
|
||||
const [showRuns, setShowRuns] = useState(false)
|
||||
const [errorsFor, setErrorsFor] = useState<{ id: number; src_login: string } | null>(null)
|
||||
const [kerioOpen, setKerioOpen] = useState(false)
|
||||
const [credsFor, setCredsFor] = useState<Account | null>(null)
|
||||
const [selected, setSelected] = useState<Set<number>>(new Set())
|
||||
const fileInputRef = useRef<HTMLInputElement>(null)
|
||||
|
||||
function reload() {
|
||||
@@ -158,7 +165,7 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
},
|
||||
}
|
||||
})
|
||||
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || (ev.type === 'error' && d.folder == null))) {
|
||||
} else if (accId != null && (ev.type === 'account_started' || ev.type === 'account_done' || ev.type === 'cancelled' || ev.type === 'paused' || (ev.type === 'error' && d.folder == null))) {
|
||||
// terminal/reset for this account — drop live overlay, fall back to DB
|
||||
setLive((prev) => {
|
||||
if (!(accId in prev)) return prev
|
||||
@@ -169,7 +176,7 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
}
|
||||
|
||||
// Structural events refresh the persisted view; `progress` is covered by live state.
|
||||
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'plan', 'task_broken'].includes(ev.type)) {
|
||||
if (['account_started', 'account_test', 'account_done', 'run_started', 'run_done', 'error', 'folder', 'cancelled', 'paused', 'plan', 'task_broken'].includes(ev.type)) {
|
||||
reload()
|
||||
}
|
||||
}),
|
||||
@@ -257,18 +264,38 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
}
|
||||
}
|
||||
|
||||
function downloadCSV(name: string, content: string) {
|
||||
const url = URL.createObjectURL(new Blob([content], { type: 'text/csv' }))
|
||||
const a = document.createElement('a')
|
||||
a.href = url
|
||||
a.download = name
|
||||
a.click()
|
||||
URL.revokeObjectURL(url)
|
||||
}
|
||||
|
||||
// Plain import: comma-separated, no header, src_login,src_pass,dst_login,dst_pass.
|
||||
function downloadExampleCSV() {
|
||||
const sample = [
|
||||
const sample =
|
||||
[
|
||||
'alice@source.example,SrcPass1,alice@dest.example,DstPass1',
|
||||
'bob@source.example,SrcPass2,bob@dest.example,DstPass2',
|
||||
'carol@source.example,SrcPass3,carol@dest.example,DstPass3',
|
||||
].join('\n') + '\n'
|
||||
const url = URL.createObjectURL(new Blob([sample], { type: 'text/csv' }))
|
||||
const a = document.createElement('a')
|
||||
a.href = url
|
||||
a.download = 'imap-copier-accounts-example.csv'
|
||||
a.click()
|
||||
URL.revokeObjectURL(url)
|
||||
downloadCSV('imap-copier-accounts-example.csv', sample)
|
||||
}
|
||||
|
||||
// Kerio Connect export: semicolon-separated with a Name;FullName;Description;Enable
|
||||
// header. The password lives in Description; disabled rows and admin are skipped
|
||||
// on import, and the domain is supplied in the import dialog.
|
||||
function downloadKerioExampleCSV() {
|
||||
const sample =
|
||||
[
|
||||
'Name;FullName;Description;Enable',
|
||||
'alice;Alice Smith;SrcPass1;Yes',
|
||||
'bob;Bob Jones;SrcPass2;Yes',
|
||||
'carol;Carol White (disabled, skipped);SrcPass3;No',
|
||||
].join('\n') + '\n'
|
||||
downloadCSV('kerio-users-example.csv', sample)
|
||||
}
|
||||
|
||||
async function onFileChosen(e: ChangeEvent<HTMLInputElement>) {
|
||||
@@ -287,6 +314,35 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
}
|
||||
}
|
||||
|
||||
async function onKerioImport(file: File, domain: string) {
|
||||
setBusy('import')
|
||||
setError(null)
|
||||
try {
|
||||
await importKerioCSV(id, file, domain)
|
||||
setKerioOpen(false)
|
||||
reload()
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Kerio import failed')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
async function saveCredentials(body: { src_login: string; src_pass: string; dst_login: string; dst_pass: string }) {
|
||||
if (!credsFor) return
|
||||
setBusy('add')
|
||||
setError(null)
|
||||
try {
|
||||
await updateAccountCredentials(id, credsFor.id, body)
|
||||
setCredsFor(null)
|
||||
reload()
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to save credentials')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
async function onDeleteAccount(accId: number, login: string) {
|
||||
const ok = await confirm({
|
||||
title: 'Remove account',
|
||||
@@ -332,7 +388,8 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
await runTask(id)
|
||||
const ids = accounts.filter((a) => selected.has(a.id)).map((a) => a.id)
|
||||
await runTask(id, ids.length ? ids : undefined)
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to start run')
|
||||
} finally {
|
||||
@@ -340,6 +397,51 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
}
|
||||
}
|
||||
|
||||
async function onPause() {
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
await pauseTask(id)
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to pause the run')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
async function onCancelRun() {
|
||||
const ok = await confirm({
|
||||
title: 'Cancel migration',
|
||||
message: 'Stop the run and mark every unfinished account as cancelled? Copied messages are kept.',
|
||||
confirmLabel: 'Cancel migration',
|
||||
cancelLabel: 'Keep running',
|
||||
danger: true,
|
||||
})
|
||||
if (!ok) return
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
await cancelTask(id)
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to cancel the run')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
async function onResume() {
|
||||
setBusy('run')
|
||||
setError(null)
|
||||
try {
|
||||
await resumeTask(id)
|
||||
reload()
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'Failed to resume the run')
|
||||
} finally {
|
||||
setBusy(null)
|
||||
}
|
||||
}
|
||||
|
||||
async function onSchedule(intervalSeconds: number) {
|
||||
setError(null)
|
||||
try {
|
||||
@@ -366,7 +468,51 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
}
|
||||
|
||||
const { task, accounts } = data
|
||||
const allTested = accounts.length > 0 && accounts.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||
const isRunning = task.status === 'running'
|
||||
// Accounts a pause left unfinished — what Resume picks up.
|
||||
const pausedCount = accounts.filter((a) => a.status === 'paused').length
|
||||
|
||||
// A failed connection test is the entry point for fixing the credentials that
|
||||
// caused it — an imported account is otherwise only deletable.
|
||||
const testCell = (a: Account, status: string) =>
|
||||
status === 'fail' && !isRunning && a.status !== 'running' ? (
|
||||
<button
|
||||
type="button"
|
||||
className="badge-btn"
|
||||
title="Edit credentials for this account"
|
||||
onClick={() => setCredsFor(a)}
|
||||
>
|
||||
<StatusBadge status={status} />
|
||||
</button>
|
||||
) : (
|
||||
<StatusBadge status={status} />
|
||||
)
|
||||
// A row is selectable only when both connection tests pass and no run is live.
|
||||
const selectableIds = accounts
|
||||
.filter((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||
.map((a) => a.id)
|
||||
const selectableSet = new Set(selectableIds)
|
||||
// Effective set: the checked accounts, or all accounts when nothing is checked.
|
||||
const effectiveSelected = accounts.filter((a) => selected.has(a.id))
|
||||
const runSet = effectiveSelected.length > 0 ? effectiveSelected : accounts
|
||||
const runReady =
|
||||
runSet.length > 0 && runSet.every((a) => a.test_src_status === 'ok' && a.test_dst_status === 'ok')
|
||||
const allSelectableChecked =
|
||||
selectableIds.length > 0 && selectableIds.every((id) => selected.has(id))
|
||||
const someSelectableChecked = selectableIds.some((id) => selected.has(id))
|
||||
|
||||
function toggleOne(accId: number, checked: boolean) {
|
||||
setSelected((prev) => {
|
||||
const next = new Set(prev)
|
||||
if (checked) next.add(accId)
|
||||
else next.delete(accId)
|
||||
return next
|
||||
})
|
||||
}
|
||||
|
||||
function toggleAll(checked: boolean) {
|
||||
setSelected(checked ? new Set(selectableIds) : new Set())
|
||||
}
|
||||
// Prefer live (WS) copied/skipped over the DB values, which only advance per
|
||||
// folder — so the summary moves in real time during a large folder.
|
||||
const totals = accounts.reduce(
|
||||
@@ -418,10 +564,43 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
<button className="btn" onClick={onTest} disabled={busy !== null || accounts.length === 0}>
|
||||
{busy === 'test' ? 'Testing…' : 'Test connections'}
|
||||
</button>
|
||||
<button className="btn btn-primary" onClick={onRun} disabled={busy !== null || !allTested || task.status === 'running'}>
|
||||
{busy === 'run' ? 'Starting…' : 'Run migration'}
|
||||
{isRunning ? (
|
||||
<>
|
||||
<button className="btn" onClick={onPause} disabled={busy !== null}>
|
||||
{busy === 'run' ? 'Stopping…' : 'Pause'}
|
||||
</button>
|
||||
{!allTested && accounts.length > 0 && <span className="hint">run unlocks once every account tests OK on both sides</span>}
|
||||
<button className="btn btn-danger" onClick={onCancelRun} disabled={busy !== null}>
|
||||
Cancel
|
||||
</button>
|
||||
<span className="hint">pause keeps the unfinished accounts resumable</span>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
{pausedCount > 0 && (
|
||||
<button className="btn btn-primary" onClick={onResume} disabled={busy !== null}>
|
||||
{busy === 'run' ? 'Resuming…' : `Resume (${pausedCount})`}
|
||||
</button>
|
||||
)}
|
||||
<button
|
||||
className={pausedCount > 0 ? 'btn' : 'btn btn-primary'}
|
||||
onClick={onRun}
|
||||
disabled={busy !== null || !runReady}
|
||||
>
|
||||
{busy === 'run'
|
||||
? 'Starting…'
|
||||
: effectiveSelected.length > 0
|
||||
? `Run selected (${effectiveSelected.length})`
|
||||
: 'Run migration'}
|
||||
</button>
|
||||
</>
|
||||
)}
|
||||
{!isRunning && !runReady && accounts.length > 0 && (
|
||||
<span className="hint">
|
||||
{effectiveSelected.length > 0
|
||||
? 'selected accounts must pass both connection tests'
|
||||
: 'run unlocks once every account tests OK on both sides'}
|
||||
</span>
|
||||
)}
|
||||
</div>
|
||||
<div className="sched-row">
|
||||
<label htmlFor="sched">Schedule</label>
|
||||
@@ -502,6 +681,7 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
|
||||
<div className="divider-label">or bulk import</div>
|
||||
<div className="upload-row">
|
||||
<div className="upload-item">
|
||||
<label className={`btn file-btn${busy !== null ? ' is-disabled' : ''}`}>
|
||||
{busy === 'import' ? 'Importing…' : 'Upload CSV'}
|
||||
<input ref={fileInputRef} type="file" accept=".csv,text/csv" onChange={onFileChosen} disabled={busy !== null} />
|
||||
@@ -510,6 +690,15 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
download example.csv
|
||||
</button>
|
||||
</div>
|
||||
<div className="upload-item">
|
||||
<button type="button" className="btn" onClick={() => setKerioOpen(true)} disabled={busy !== null}>
|
||||
Import from Kerio
|
||||
</button>
|
||||
<button type="button" className="link-btn" onClick={downloadKerioExampleCSV}>
|
||||
download kerio example.csv
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="panel">
|
||||
@@ -540,6 +729,18 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
<table className="tbl">
|
||||
<thead>
|
||||
<tr>
|
||||
<th className="chk-col">
|
||||
<input
|
||||
type="checkbox"
|
||||
aria-label="Select all accounts"
|
||||
checked={allSelectableChecked}
|
||||
ref={(el) => {
|
||||
if (el) el.indeterminate = !allSelectableChecked && someSelectableChecked
|
||||
}}
|
||||
disabled={isRunning || selectableIds.length === 0}
|
||||
onChange={(e) => toggleAll(e.target.checked)}
|
||||
/>
|
||||
</th>
|
||||
<th>Account</th>
|
||||
<th>Src test</th>
|
||||
<th>Dst test</th>
|
||||
@@ -554,11 +755,20 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
<tbody>
|
||||
{accounts.length === 0 ? (
|
||||
<tr className="empty-row">
|
||||
<td colSpan={9}>no accounts yet — add one or import a CSV above</td>
|
||||
<td colSpan={10}>no accounts yet — add one or import a CSV above</td>
|
||||
</tr>
|
||||
) : (
|
||||
accounts.map((a) => (
|
||||
<tr key={a.id}>
|
||||
<td className="chk-col">
|
||||
<input
|
||||
type="checkbox"
|
||||
aria-label={`Select ${a.src_login}`}
|
||||
checked={selected.has(a.id)}
|
||||
disabled={isRunning || !selectableSet.has(a.id)}
|
||||
onChange={(e) => toggleOne(a.id, e.target.checked)}
|
||||
/>
|
||||
</td>
|
||||
<td>
|
||||
<div className="acct-ident">
|
||||
<span>{a.src_login}</span>
|
||||
@@ -572,12 +782,8 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
</div>
|
||||
)}
|
||||
</td>
|
||||
<td>
|
||||
<StatusBadge status={a.test_src_status} />
|
||||
</td>
|
||||
<td>
|
||||
<StatusBadge status={a.test_dst_status} />
|
||||
</td>
|
||||
<td>{testCell(a, a.test_src_status)}</td>
|
||||
<td>{testCell(a, a.test_dst_status)}</td>
|
||||
<td>
|
||||
<StatusBadge status={a.status} />
|
||||
</td>
|
||||
@@ -709,6 +915,19 @@ export function TaskDetail({ id }: { id: number }) {
|
||||
)}
|
||||
<RunLogModal taskId={id} open={showRuns} onClose={() => setShowRuns(false)} />
|
||||
<AccountErrorsModal taskId={id} account={errorsFor} onClose={() => setErrorsFor(null)} />
|
||||
<KerioImportModal
|
||||
open={kerioOpen}
|
||||
busy={busy === 'import'}
|
||||
onClose={() => setKerioOpen(false)}
|
||||
onSubmit={onKerioImport}
|
||||
/>
|
||||
<AccountCredentialsModal
|
||||
open={credsFor !== null}
|
||||
busy={busy === 'add'}
|
||||
account={credsFor}
|
||||
onClose={() => setCredsFor(null)}
|
||||
onSubmit={saveCredentials}
|
||||
/>
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user