Commit Graph
60 Commits
Author SHA1 Message Date
vasyanskandClaude Opus 5 c077b368f3 Allow fixing an account's credentials from a failed test
An imported account with a wrong password could only be deleted and
re-added, which loses its folder mapping and its migration journal.
Clicking either FAIL badge now opens a dialog for both logins and both
passwords.

Passwords are never sent to the browser, so the password fields start
empty and an empty field keeps the stored ciphertext — one side can be
corrected without retyping the other. Saving resets both test verdicts
to unknown: they described the previous credentials, and the run gate
requires a passing test on both sides, so the account cannot start on an
unverified password.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 12:08:28 +07:00
vasyanskandClaude Opus 5 8bc7ff026d Add endpoint deletion
The endpoints screen could only create and edit servers, so a mistyped
or retired endpoint stayed in the list forever.

Tasks reference endpoints without ON DELETE CASCADE, so a referenced
endpoint is refused with 409 and a count of the tasks using it rather
than cascading away migration history. The foreign-key violation is
mapped to the same status to cover a task created between check and
delete.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 11:50:12 +07:00
vasyansk 6bf3a6c4ca Add body read timeout recovery to IMAP client
Introduce Client wrapper with socket deadline support
Add reconnection logic for body read timeouts
Implement test cases for underflow scenarios
Update orchestrator to handle reconnections
2026-07-21 05:29:35 +07:00
vasyansk b6e68bdd90 Add activity tracking to prevent stall timeouts during message transfers
Add OnActivity callback to CopyDeps to prevent stall timeouts during large message transfers
Implement touchReader and touchWriter wrappers to call OnActivity during FETCH and APPEND operations
Add slow message logging to identify performance bottlenecks
Add test case to verify activity reporting during message transfers
Clean up orchestrator account reset code formatting
2026-07-21 04:38:32 +07:00
vasyansk d125320667 fix dtt 2026-07-18 12:21:58 +07:00
vasyansk b352cda166 add kerio format support 2026-07-17 11:42:34 +07:00
vasyansk 599c66189c Parse optional account_ids in run handler 2026-07-08 13:41:14 +07:00
vasyansk 315e56046b Add selectAccounts filter and accountIDs param to orchestrator.Run 2026-07-08 13:37:24 +07:00
vasyansk e84366eb0c Remove idle connection timeout handling
Remove error handling for closed connections

Add progress watchdog to detect stalled accounts

Improve error modal styling and pagination
2026-07-05 14:50:25 +07:00
vasyansk 95ddbf5619 Update copy.go 2026-07-05 13:59:56 +07:00
vasyanskandClaude Opus 4.8 45b0ff2358 feat(errors): per-account error modal with persisted error list
Accounts finishing done_with_errors showed only a count and a single
last_error. This adds a modal listing every concrete error of the
account's most recent run.

- migration 0005: account_errors table (kind folder|message|account,
  folder, message_ref, error, created_at; ON DELETE CASCADE; indexed)
- store: AddAccountError / ClearAccountErrors / ListAccountErrors
- copy: OnError callback captures per-message error text (previously
  only counted), with a "UID N: subject" reference
- orchestrator: clear errors at run start; persist folder/message/
  account errors; cap 500 rows/account/run with a suppressed-note row
- api: GET /api/tasks/{id}/accounts/{accountId}/errors
- web: AccountErrorsModal, clickable ERRORS count, api + styles

Verified: migration applies on Postgres 18; store add/list/clear and
cascade tests pass against real pg; backend build/vet/test green; web
tsc+vite build and oxlint clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9Eq4JtWjyNTv5qat3B3mM
2026-07-05 12:16:12 +07:00
vasyansk 2623bc8815 Reduce worker concurrency from 4 to 2
Reduce idle read timeout from 120s to 60s

Add batching for IMAP metadata fetches
2026-07-05 11:40:04 +07:00
vasyansk d5192e1cf7 Add optional pprof endpoint for debugging 2026-07-04 21:31:53 +07:00
vasyansk dca1363ec9 Add IMAP connection timeout handling
Introduce idleConn wrapper to prevent wedged connections
Add test for silent server timeout behavior
Implement proper TLS handshake and greeting handling
Set explicit dial and read timeouts
2026-07-04 19:37:36 +07:00
vasyansk 64bcb1d60b fix: anchor floor on re-enable; breaker on scheduled panic; propagate next-run err; reject negative interval; close phantom runs on startup 2026-07-03 13:31:58 +07:00
vasyanskandClaude Opus 4.8 d2c69c6a5e feat(api): schedule + runs endpoints; next_run_at on task detail
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 13:13:10 +07:00
vasyansk be777dc908 feat(orchestrator): thread run trigger; breaker on scheduled runs with errors 2026-07-03 13:08:21 +07:00
vasyansk b9421d388c feat(scheduler): 30s polling scheduler with pure NextRun/dueTaskIDs 2026-07-03 13:07:05 +07:00
vasyansk 8f93dcd97b feat(store): task schedule columns, run trigger, scheduling queries 2026-07-03 13:01:20 +07:00
vasyansk b59ce67497 fix: validate account on mapping PUT; gate folders btn during run; prune stale mapping keys; drop dead export 2026-07-03 12:08:22 +07:00
vasyansk bdb3b72f38 feat(api): probe existing account + persist per-account folder mapping 2026-07-03 11:50:32 +07:00
vasyansk bc72f8172f feat(orchestrator): build copy plan from per-account mapping + exclusions 2026-07-03 11:46:31 +07:00
vasyansk d6d17ee544 feat(store): per-account folder_mapping + excluded_folders columns 2026-07-03 11:39:21 +07:00
vasyansk 79fd200e57 many fixes 2026-07-03 11:18:40 +07:00
vasyanskandClaude Opus 4.8 477077d9e3 fix: consistent error reporting + reset mapping modal per account
Error consistency: folder-level failures were counted only toward the task's
done_with_errors status, not the account's error_count, so a task showed
DONE_WITH_ERRORS while its only account showed 0 errors / DONE. Now folder
errors increment the account counter and the account status becomes
done_with_errors when errs>0.

Visibility: persist accounts.last_error (migration 0002) so the failing folder
/ login error survives a page reload (shown red under the source login);
cleared at the start of each run.

Modal reset: the folder-mapping modal kept its selections across opens, so
adding a second account showed the first account's mapping. It now mounts
fresh per add (conditional render + key), reflecting the newly-probed folders.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 15:14:11 +07:00
vasyanskandClaude Opus 4.8 0bb584fe10 feat: folder mapping UI on account add (probe + map modal)
ADD now probes both connections, lists folders on each side, and opens a
mapping modal to route source->destination folders (e.g. Спам -> Spam) so we
append into the existing folder instead of creating a duplicate.

- store: SetTaskFolderMapping (+ round-trip test)
- httpapi: POST /tasks/{id}/probe (test both, return folder lists),
  PUT /tasks/{id}/folder-mapping
- web: FolderMappingModal (reuses Modal, size=lg), submitAccount probes then
  opens the modal; confirm creates the account and saves the task mapping

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 14:39:31 +07:00
vasyanskandClaude Opus 4.8 a2234077df feat: stream the metadata scan instead of Collect
CopyFolder now streams envelope metadata via Next() in a first pass (dedup +
queue new messages), then streams bodies for new ones in a second pass —
no more blocking Collect of the whole folder with zero feedback, and memory
stays flat (only new-message meta is held).

- imapx: two-pass streaming CopyFolder + CopyDeps.OnScan(scanned,total)
- orchestrator: throttled 'scan' events during the metadata pass
- web: per-account 'scanning folder: X/N' line under the progress bar;
  scan events kept out of the log to avoid flooding

Verified on greenmail: idempotency and internal-date preservation still hold.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 13:35:08 +07:00
vasyanskandClaude Opus 4.8 4959173f39 feat: planning pass counts all folders up front for account-wide progress
Before copying, EXAMINE every folder to sum the account's total message count
and emit a 'plan' event; progress events now carry account_total so the UI
shows a real overall bar, percent and ETA (not just per-folder).

- imapx.FolderMessageCount: read-only count of a folder
- orchestrator: plan pass + grandTotal, plan event, account_total in progress
- web: live progress keyed on account total; PLAN log line; overall bar/ETA

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 13:21:57 +07:00
vasyanskandClaude Opus 4.8 e1911ef13b feat: real-time progress with per-folder bar, speed and ETA
- orchestrator: progress events now carry account-level cumulative copied/
  skipped plus current folder done/total, throttled to ~3/sec per account
- web: RUN CONTROL counters and account copied/skipped read live WS values
  (DB only advances per folder, so the summary lagged); new Progress column
  shows a bar, percent, avg messages/sec and folder ETA while running

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 13:14:55 +07:00
vasyanskandClaude Opus 4.8 fa72f1b323 fix: recover from phantom 'running' state after crash/restart
The run-cancel registry is in-memory; a container restart mid-run leaves
accounts/tasks persisted as 'running' with no goroutine, wedging cancel
(not-in-map -> 409) and blocking remove/re-run.

- startup: ResetRunningOnStartup clears stale 'running' -> 'idle' on boot
- cancel handler: when no live goroutine, ClearStuckAccount + ReconcileTaskStatus
  reset the stuck account (and its task) instead of returning 409

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 12:57:39 +07:00
vasyanskandClaude Opus 4.8 f024f329fc feat: per-account cancel + folder message-count progress in log
- orchestrator: per-account cancellable context registry + CancelAccount;
  on cancel, close IMAP connections to unblock in-flight FETCH; account ends
  in 'cancelled' status with a cancelled event
- imapx: CopyDeps.OnFolder callback fires after EXAMINE with the folder's
  message count (before the long fetch) for visibility
- httpapi: POST /tasks/{id}/accounts/{accountId}/cancel
- web: per-row cancel button while running, folder event shows N messages,
  cancelled/done_with_errors status badges

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 12:47:07 +07:00
vasyanskandClaude Opus 4.8 79bd55aad8 fix(httpapi): trim whitespace on manually-added account login/password
Pasted app passwords (e.g. mail.ru) often carry a trailing space/newline that
the IMAP server rejects; the CSV path already trims, so make manual add match.
Source and destination passwords remain fully independent (src_pass_enc /
dst_pass_enc), verified end-to-end — this only strips surrounding whitespace.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 10:38:00 +07:00
vasyanskandClaude Opus 4.8 b88f88c1a3 feat: delete task/account, edit endpoint, richer event log
- store: DeleteAccount, DeleteTask, UpdateEndpoint (+ cascade/update tests)
- httpapi: DELETE /tasks/{id}, DELETE /tasks/{id}/accounts/{accountId},
  PUT /endpoints/{id}; delete guarded with 409 while task running
- orchestrator: enrich WS events with login/host/port/error (test + run + errors)
- web: delete buttons (task, account) with confirm, endpoint edit form,
  human-readable event log (source/dest, host:port, error text)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 09:23:14 +07:00
vasyanskandClaude Opus 4.8 7eff28053a fix(api): return empty arrays not null for empty lists (frontend null.length crash)
Root cause: store List* methods used 'var out []T' which stays nil on empty
result sets and serializes to JSON null; the SPA then crashed on .length/.map
(e.g. endpoints.length on the Tasks page right after deploy). Return []T{}
at the source; coerce null->[] on the frontend load sites as defense-in-depth.
Regression test asserts List* are non-nil when empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-02 09:04:33 +07:00
vasyanskandClaude Opus 4.8 8af7b11791 fix(orchestrator): recover from panics in run goroutines to avoid process crash and stuck 'running' task
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-01 19:36:40 +07:00
vasyansk 2429c786e4 fix(orchestrator): prevent concurrent double-run duplicating messages; reflect errors in status 2026-07-01 19:32:04 +07:00
vasyanskandClaude Opus 4.8 38005c0618 fix(api): add snake_case json tags to Endpoint/Task/request bodies for frontend contract
Go's encoding/json does not bridge snake_case <-> PascalCase field names,
so store.Endpoint, store.Task and the anonymous request bodies in
accounts.go/auth.go were silently decoding empty/zero values from the
frontend's snake_case JSON contract (tls_mode, role_label,
src_endpoint_id, dst_endpoint_id, src_login/pass, dst_login/pass).
Adds explicit json tags; DB layer is unaffected since pgx binds by
positional params, not struct-tag reflection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-01 19:04:46 +07:00
vasyanskandClaude Opus 4.8 4c57848c35 fix(httpapi): detect ws client disconnect via CloseRead to prevent subscriber leak
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-01 18:47:54 +07:00
vasyansk 9ec6acd414 feat(httpapi): websocket, router, embed static, main entrypoint 2026-07-01 18:37:48 +07:00
vasyansk 0bd4ba37e3 fix(httpapi): fail CSV import on encryption error instead of storing empty passwords 2026-07-01 18:33:09 +07:00
vasyansk bb83bbd989 feat(httpapi): REST resources for endpoints/tasks/accounts/csv/run 2026-07-01 18:27:11 +07:00
vasyanskandClaude Opus 4.8 839febb83a fix(httpapi): bind session token to current AuthUser; add negative auth tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MMHQTtnQtQqL8muAXHr9kd
2026-07-01 18:24:35 +07:00
vasyansk cae124931d feat(httpapi): env-based login and session auth middleware 2026-07-01 18:19:57 +07:00
vasyansk f9f01b981b fix(csvimport): report accurate physical line numbers via FieldPos; add blank-line + zero-row tests 2026-07-01 18:17:54 +07:00
vasyansk 7fe8896f4b feat(csvimport): validated CSV account parser 2026-07-01 18:12:57 +07:00
vasyansk 2def11a870 fix(orchestrator): reuse src connection to list folders instead of extra TestLogin 2026-07-01 18:10:52 +07:00
vasyansk 8c871d9d26 feat(orchestrator): worker pool run + account testing gate 2026-07-01 18:04:44 +07:00
vasyansk ec8835538b feat(wshub): per-task event hub with non-blocking publish 2026-07-01 17:59:43 +07:00
vasyansk a54ec1d148 fix(imapx): preserve source internal date on APPEND 2026-07-01 17:57:12 +07:00
vasyansk 06c0598b80 feat(imapx): streaming per-folder copy with dedup, idempotent 2026-07-01 17:39:43 +07:00