# FILING.md

## Purpose
Working notes for the Caya → Google Drive → Qonto filing and payment pipeline, so it can be resumed later by a cron job or future session without guessing.

## Pipeline Overview
1. Caya scans physical mail.
2. Caya drops resulting PDFs into Google Drive folder `01 Inbox`.
3. I read the PDFs locally with `pdftotext`.
4. For multi-page Caya PDFs, ignore the last two pages during extraction because they are usually scans of the envelope (front and back) and often contain stray numbers that can poison classification and amount extraction.
5. From each PDF, I extract the relevant fields:
   - recipient company
   - whether it is an invoice
   - payment reference
   - amount
   - IBAN
   - due date when available
6. Amount extraction should be label-based, not just "pick a plausible number".
   - Anchor amount extraction to nearby terms like `Gesamt`, `Total`, `Forderungen`, `Summe`, `offenstehend`, `zu zahlen`, `fällig`, `1. Summe aus diesem Beitragsbescheid`, or `Umlage`.
   - Ignore numeric values found near labels like `Hebesatz`, `%`, `bereits gezahlt`, `Freibetrag`, or `Bemessungsgrundlage`.
7. Invoice classification should be clue-based, not assumption-based.
   - Strong invoice / payment-request clues include `Rechnung`, `Mahnung`, or `Zahlungsaufforderung`.
   - `Mahnung` is a high-risk subtype: it may refer to an invoice already paid, or an original invoice that appears elsewhere in the same batch.
   - Municipal / public-body `Mahnung` or `Zahlungsaufforderung` letters are still processable when the payable amount, recipient company, payment reference, and payment target are clear enough.
   - If the letter says payment will happen by `Lastschrift` or references a `SEPA-Lastschrift-Mandat`, treat it as file-only unless something clearly contradicts that.
   - Some mail is informational only, such as `Kontoauszug`, `Abrechnung Verkauf von Wertpapieren`, or obvious flyers/marketing mail. These should be filed when relevant, but no payment action is needed.
8. If the document is not an invoice, or the recipient company is not one of the three known companies, I move it into `03 Manual` for manual review.
9. Some non-invoice documents still require human follow-up and therefore also belong in `03 Manual` rather than normal filing.
   - Examples include meter-reading requests and Transparenzregister / compliance notices that require portal updates.
10. If I am uncertain, or the document represents a use case we have not encountered yet, I move it into `04 Hold` instead of forcing a decision.
11. `04 Hold` is not terminal storage. On each filing run, I should also re-check existing documents in `04 Hold` to see whether newer rules or learned patterns now allow safe classification, filing, or routing to `03 Manual`.
12. If a re-check of an item in `04 Hold` now reaches a confident manual-only conclusion, I must move it out of `04 Hold` into `03 Manual` rather than leaving it in hold.
13. Before creating any Qonto payment request, I generate a dry validation summary containing:
   - company
   - creditor
   - IBAN
   - amount
   - due date
   - reference
   - confidence / extraction quality
   - current source-account balance
14. If the extraction confidence is low, fields conflict, or key values look suspicious, I move the document into `04 Hold` when the case is uncertain or novel; use `03 Manual` for known cases that simply require manual handling.
15. For invoices that need payment, I create a Qonto multi-transfer request via `qontoctl request create-multi-transfer` so it appears in Qonto's approval queue. This is safe as My User will approve this in Qonto later. 
16. Only after payment-request creation succeeds do I file the PDF into the matching company subfolder under `02 Filed`.
17. If payment-request creation fails for any reason (for example OAuth/auth/API failure), I must not leave the document in `02 Filed`; keep it in or move it back to `04 Hold` until the payment side is completed successfully.
18. Final approval and release happens on the Qonto side.

## Storage / Paths
- Google Drive account: `My User@gmail.com`
- Local Google Drive path: `/Users/remi/Library/CloudStorage/GoogleDrive/My Drive/`
- Inbox folder: `01 Inbox`
- Filed folder: `02 Filed`
- Manual-review folder: `03 Manual`
- Hold folder: `04 Hold`
- Filed subfolders:
  - `02-01 Company 1`
  - `02-02 Company 2`
  - `02-03 Company 3`
  - `02-04 Private` (private mail for Private and Spouse)

## Private Mail (Private / Spouse)
- The pipeline also covers the household's **private** mail (added 2026-07-12).
- Private recipients: **Private** and **Spouse**.
- Identifying address for private correspondence: **`My Private Address`** (the companies sit at My Companies' Address). Mail addressed to either name at that address → private.
- Known exception that overrides the address rule: Möwing & Gerdel invoices for Bauvorhaben Scabellstr. 8/9 use the private address by mistake but belong to Company 1 (see learned note).
- Private documents are filed into `02 Filed/02-04 Private`.
- All routing/extraction/Mahnung rules apply unchanged; only the banking side differs (bunq instead of Qonto, see Banking section).
- Default assumption until My User says otherwise: private payables are paid from the bunq **Bank Account** `DE88370190001111110111` (the Savings account is not used for payments).

## Extraction / Tooling
- PDF extraction tool: `pdftotext` (from poppler)
- `pdftotext` works well on the Caya scans tested so far.
- Manually scanned letters may contain no embedded text at all. If `pdftotext` returns empty or near-empty output, use OCR fallback before classification.
- OCR fallback is especially important for ad-hoc / manually scanned PDFs from ScanSnap or similar devices.
- Exact OCR fallback used on 2026-04-15 for a textless municipal Mahnung in `04 Hold`: render PDF pages to PNG with `pdftoppm`, then run OCR/extraction on those page images with the OpenClaw `image` tool.
  - Example render command: `pdftoppm -png "/path/to/input.pdf" "/tmp/ocr-pages/hold-page"`
  - Then pass the generated page images (for example `hold-page-1.png`, `hold-page-2.png`) to the `image` tool with a prompt asking for recipient company, creditor, document type, amount, due date, IBAN, BIC, and payment reference / `Kassenzeichen`.
- If OCR returns a weak, surprising, or workflow-breaking recipient/company match — especially one that differs by a small visual error from a known company or address — do not trust the first pass blindly. Re-run recipient extraction with a second pass that focuses specifically on the addressee block and cross-checks against nearby company/address clues before routing.
- Recipient-company OCR may contain small clerical or recognition errors. Use a bounded nearest-match heuristic before treating the company as unknown.
  - Normalize comparison strings first: lowercase, map `ß -> ss`, replace umlauts (`ä->ae`, `ö->oe`, `ü->ue`), strip punctuation, collapse whitespace.
  - Compare the OCR addressee against the three known company names and accept the nearest known company only when the difference is minimal and plausibly clerical.
  - Minimal acceptance rule: allow auto-match only when the normalized OCR company name is within an edit distance of **2** from a known company name **or** when the only meaningful difference is a single likely OCR/clerk fragment such as `6b` vs `66`, `1` vs `l`, or a missing/extra character.
  - Extra safety rule: require at least one supporting clue besides the near-name match, such as a matching street/address fragment, company-specific IBAN/account context, or prior known correspondence pattern.
  - If multiple known companies are similarly close, or the nearest match exceeds the threshold above, do **not** auto-route; keep it in `04 Hold` or move it to `03 Manual`.
- If the second pass still cannot identify the recipient confidently, keep the document in `04 Hold` or move it to `03 Manual` based on whether the uncertainty is real versus clearly manual.
- For Caya mail PDFs, default extraction should ignore the final two pages unless there is clear evidence they are not envelope scans.

## Banking / Payment Tooling
- Company banking CLI: `qontoctl` (three Qonto profiles, see Known Qonto Setup).
- Private banking CLI: `bunqctl` — installed at `/Users/remi/bunqctl` (run via the `bunqctl` wrapper there; add the dir to PATH or call it with the full path). Read `/Users/remi/bunqctl/README.md` for details.
- Both tools share the same safety model: nothing I create moves money by itself. Qonto requests wait in the approval queue; bunqctl only ever creates **draft** payments that My User approves in the bunq app. 

### bunqctl usage (private side)
- Balance / accounts: `bunqctl balance` (handshake already done; config in `bunq-production.conf`, never ask for or print the API key).
- Duplicate checks: `bunqctl transactions --account DE88370190001111110111 --since <date>` (or `--json` for parsing). The bunq account is brand new (opened ~2026-07-12, confirmed by My User): its history is authoritative only from then on. Older private invoices can never show up there — they may have been paid from other private accounts invisible to the pipeline; per the Mahnung directive, pay anyway when due and let bunq-app approval filter.
- The bunqctl README advises keeping `pay-batch` out of automated runs. **My User overrode this for the pipeline (2026-07-12): cron runs DO include the private payment step**, because drafts move no money and he keeps the power to not approve in the bunq app. Always use `--dry-run` first as validation, then `pay-batch ... --yes`.
- Payments: `bunqctl pay-batch` with one `--pay 'iban,name,amount,currency,description'` per item (or a headerless CSV `iban,name,amount,currency,description`). Currency empty = EUR.
- In non-interactive runs append `--yes` (skips the terminal prompt; the result is still only a draft in the bunq app).
- `--dry-run` previews without touching the API — use it as the dry-validation step before the real call.
- Batch per run like on the Qonto side: collect all private payables, create one draft batch at the end, file PDFs into `02-04` only after the draft is created successfully.
- There is no list/cancel for drafts in bunqctl: review and reject/approve happens in the bunq app only.
- Default debit account: Bank Account `DE88370190001111110111`.

## Known Qonto Setup
- Profile: `profile3`
  - Account holder: Company 3 GmbH
  - IBAN: `DE09 1001 0123 1111 3333 33`
  - OAuth status: verified active on 2026-04-13
  - Write-capability check: `transfer verify-payee` reached Qonto successfully on 2026-04-13
- Profile: `profile2`
  - Account holder: Company 2
  - IBAN: `DE09 1001 0123 1111 2222 22`
  - OAuth status: verified active on 2026-04-13
  - Write-capability check: `transfer verify-payee` succeeded on 2026-04-13
- Profile: `profile1`
  - Account holder: Company 1 GmbH
  - IBAN: `DE09 1001 0123 1111 1111 11`
  - OAuth status: verified active on 2026-04-13
  - Write-capability check: `transfer verify-payee` reached Qonto successfully on 2026-04-13

## Operating Rules
- Always validate extracted payment details before creating a payment request.
- Ignore the last two pages of typical Caya PDFs during classification and extraction, because they are usually envelope scans with irrelevant numbers.
- Use label-based extraction for payable amounts; do not rely on generic number picking when labels are available.
- Prefer amounts near `Gesamt` or `Total` when those labels are present and consistent with the rest of the document.
- Treat `Rechnung`, `Mahnung`, and `Zahlungsaufforderung` as strong invoice/payment clues, but do not auto-pay solely on keyword presence.
- Do not treat municipal or public-body reminders as manual-only by default. If extraction is clear and duplicate risk is acceptably low, process them.
- Treat `Kontoauszug`, `Abrechnung Verkauf von Wertpapieren`, and obvious flyers/marketing mail as non-payable documents: file them when relevant, but do not create a Qonto payment request.
- Do not file non-invoice documents that still require human follow-up. If the letter contains wording like `Aufforderung`, an explicit response deadline / due date, or asks the recipient to submit/update something online, default to `03 Manual`.
- Generate a dry validation summary before each Qonto request, including company, creditor, IBAN, amount, due date, reference, confidence, and current source-account balance.
- If confidence is low, fields conflict, or the extracted values look suspicious, do not force automation.
- Treat recipient/company extraction as a critical field. If OCR is the only source and the detected recipient would change routing materially, require a retry focused on the address block before deciding unknown-company handling.
- For OCR-only recipient matches, use the bounded nearest-match heuristic: normalized edit distance <= 2, or a single obvious clerical/OCR fragment difference such as `6b` vs `66`, plus at least one supporting clue before auto-routing to a known company.
- Use `04 Hold` for uncertain or previously unseen cases that should later improve the workflow.
- Treat `04 Hold` as a reprocessing queue: review its contents during later filing runs and promote documents out of hold once the workflow has enough confidence to decide.
- If a reprocessed hold item is now clearly understood but still needs human handling, move it from `04 Hold` to `03 Manual` immediately.
- Use `03 Manual` for cases that are understood but still require manual handling.
- For payable invoices, `02 Filed` means the payment-request step succeeded. If request creation fails, the document belongs in `04 Hold` until retried successfully.
- Do not block request creation just because the source account balance is low; include the balance in the summary and let My User decide on approval timing in Qonto.
- For **Mahnung** / reminder / late-payment documents, perform an additional duplicate/payment-state check before creating a Qonto request.
- **Directive from My User (2026-07-03): do not ask for chat confirmation before creating payment requests.** Once the dry validation summary is clean, create the `request create-multi-transfer` directly and report what was created — Qonto's approval queue is the confirmation step. Only stop to ask when something is genuinely undecidable from the documents (e.g. conflicting amounts with no authoritative source), not for "shall I proceed".
- **Directive from My User (2026-07-03): due Mahnungen are paid immediately, not delayed.** When a Mahnung names a known company and the creditor, amount, IBAN, and reference are clear, create the Qonto request in the same run. Do not route to `03 Manual` just because the Qonto history is too short to prove non-payment, the deadline is unreadable, or a prior reminder is implied — the Qonto approval step is the safety net for residual double-payment risk.
- Route a Mahnung to `03 Manual` only for concrete other reasons, e.g.: **positive** evidence of settlement (a matching completed payment/request in Qonto), conflicting or unextractable amounts, an unknown recipient company, or the original invoice appearing unpaid in the same batch (then pay only one of the two).
- Before creating a request for a reminder-style document, still check Qonto for existing requests or payments with the same creditor and reference to avoid paying twice; a clean-but-short history does not block payment.
- If a document is a `Rechnung` but states that collection will happen via `Lastschrift` or an existing `SEPA-Lastschrift-Mandat`, file it normally but do **not** create a Qonto payment request.
- Use `qontoctl request create-multi-transfer` for invoice payments so they enter Qonto's normal approval queue.
- **Batch per company per run (My User, 2026-07-03):** collect all payable items during the run and create **one** multi-transfer request per company at the end (multiple transfer objects in one JSON array), so My User approves once per company instead of once per document. Constraints: requests cannot span companies/debit accounts, have a single scheduled date, and are **immutable once created** — no edit/append exists, only `approve`/`decline`. If one line item is wrong, decline the whole request and re-create it without that item. Consequence: file each batched PDF into `02 Filed` only after the end-of-run batch request succeeds; if it fails, all its documents follow the `04 Hold` rule. Consider keeping an unusually risky item (e.g. an ambiguous Mahnung) in its own separate request so it doesn't block the routine batch.
- Do **not** use `qontoctl transfer create` for the filing workflow. That path triggers immediate SCA approval and is not suitable for asynchronous batching.
- Do not bypass Qonto-side approval controls.
- If a document is not an invoice, or the recipient cannot be matched to `Company 1 GmbH`, `Company 2`, `Company 3 GmbH`, or the private household (`Private` / `Spouse`, My Private Address), move it to `03 Manual`.
- The bounded nearest-match heuristic for OCR'd recipient names applies to the two private names as well (e.g. `Hartdt` → `Hardt`, `Anais`/`Anaïs` → `Anaís`).
- Treat meter-reading requests and Transparenzregister / Bundesanzeiger notices with required follow-up as `03 Manual`, not ordinary filing.
- For municipalities and similar public bodies, terminology may differ from private invoices. In such letters, the payment reference may be a `Kassenzeichen` rather than a customer reference / invoice number.
- For municipality letters such as `Zahlungsaufforderung`, read the body of the letter carefully. The payable amount may appear inline near wording like `Gesamtbetrag in Höhe von ...`, not only in any tabular attachment.
- For municipality or public-body `Mahnungen`, process them when creditor, company, amount, reference / `Kassenzeichen`, and payment rails are clear enough to avoid missing hard deadlines.

## Suggested Cron-Resumption Logic
When a cron job picks this up later, it should follow the steps below. For documents classified as **private** (Private / Spouse, My Private Address), every step applies analogously with: folder `02-04 Private`, banking via `bunqctl pay-batch --yes` (draft, approved in the bunq app), duplicate checks via `bunqctl transactions`, and balance via `bunqctl balance`.
1. Check `01 Inbox` for new PDFs.
2. Also check `04 Hold` for previously uncertain PDFs that may now be processable under newer rules.
3. Extract text with `pdftotext`.
4. If `pdftotext` returns empty or near-empty output, run OCR fallback before classification.
5. If OCR drives recipient/company detection and the result is weak, surprising, or would materially change routing, run a second recipient-focused extraction pass on the addressee block before treating the company as unknown.
6. Before treating an OCR-derived recipient as unknown, apply the bounded nearest-match heuristic against the three known companies: normalize spelling (`ß->ss`, umlauts, punctuation, whitespace), then accept auto-routing only for very small differences (normalized edit distance <= 2, or one obvious fragment error like `6b` vs `66`) and only when at least one additional clue supports the match.
7. Ignore the final two pages of each typical Caya PDF during extraction/classification unless they clearly contain real document content rather than envelope scans.
8. Classify each document by company and whether it is an invoice, using clues such as `Rechnung`, `Mahnung`, and `Zahlungsaufforderung`.
7. If the document is informational only (for example `Kontoauszug`, `Abrechnung Verkauf von Wertpapieren`, or an obvious flyer), file it when relevant and skip payment creation.
8. If a document is not an invoice or the company is unknown, move it to `03 Manual`.
9. If a non-invoice document still requires human follow-up — especially when it uses wording like `Aufforderung`, includes an explicit response deadline / due date, or asks for an online update/submission — move it to `03 Manual` rather than normal filing.
10. If a reprocessed document from `04 Hold` is now confidently identified as manual-only, move it to `03 Manual` instead of leaving it in hold.
11. If the case is uncertain or represents a new use case we have not encountered yet, move it to `04 Hold`.
12. For invoices in known companies, extract payment details using label-based field extraction where possible, preferring amounts near `Gesamt` or `Total` when consistent.
13. For municipality / public-body letters, also check the narrative body text for payable amounts and references; the reference may be labeled `Kassenzeichen`, and the amount may be stated inline near wording such as `Gesamtbetrag in Höhe von ...`.
14. If the document indicates `Lastschrift` / `SEPA-Lastschrift-Mandat`, file it but skip Qonto payment creation.
15. Generate a dry validation summary with company, creditor, IBAN, amount, due date, reference, confidence, and current source-account balance.
16. If confidence is low or fields conflict, move the document to `04 Hold` when the uncertainty is real or the pattern is unfamiliar.
17. If recipient extraction is OCR-dependent and the detected company would route the document away from a known-company payment path, retry recipient extraction on the addressee block before final routing.
18. If it is a `Mahnung` or reminder-style document, perform duplicate/payment-state checks before any request creation; if the open amount and payment target are unambiguous, it may still be processed. If anything material is ambiguous, move it to `04 Hold` or `03 Manual`, depending on whether the case is unclear versus clearly manual.
19. Municipal / public-body reminders are not auto-excluded. If the creditor, recipient company, amount, reference / `Kassenzeichen`, and payment rails are clear enough, create the Qonto request so the deadline is not missed.
20. For payable invoices, create the payment request in Qonto first. Only after that succeeds should the document be moved/filed into the correct `02 Filed` subfolder.
21. If Qonto request creation fails, keep the document in `04 Hold` or move it back there rather than filing it as completed.
22. Report back with a concise summary, including what was reprocessed from `04 Hold`, which reprocessed items were promoted to `03 Manual`, what was filed normally, what was submitted to Qonto for approval, what failed payment creation and therefore remained in `04 Hold`, and what remains in `04 Hold` for later workflow refinement.

## Qonto Multi-Transfer Request Format
Use `qontoctl request create-multi-transfer` with a JSON file containing a **plain array** of transfer objects. Do not wrap it in `{ "transfers": [...] }`.

Correct example payload:

 ⁠json
[
  {
    "credit_iban": "DE03160400000100300300",
    "credit_account_name": "IHK Potsdam",
    "credit_account_currency": "EUR",
    "amount": "56.00",
    "currency": "EUR",
    "reference": "3028009500268 01585800"
  }
]


⁠ Example command:

 ⁠bash
qontoctl -p hardtind request create-multi-transfer \
  --file /tmp/ihk-multi-transfer.json \
  --debit-iban DE74100101231840459267 \
  --scheduled-date 2026-04-14 \
  --note "IHK Potsdam test via filing workflow"