ingest: switch wealthfolio to pg-sync mirror reads
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful

The previous SQLite-direct reader queried `holdings_snapshot` (singular)
and `accounts.type` — both wrong against the live wealthfolio schema
(plural `holdings_snapshots`, column `account_type`). It silently
returned [] via the OperationalError fallback, leaving fire-planner with
stale account snapshots.

Switch to reading from the wealthfolio_sync PG mirror. The pg-sync
sidecar (defined in infra/stacks/wealthfolio) hourly mirrors SQLite to
Postgres with a clean schema. We read from `daily_account_valuation`
which already has total_value, cost_basis, and explicit fx_rate_to_base
per row — no JSON-decoding of position blobs.

CLI ingest no longer takes --db-path (no kubectl-exec gymnastics);
reads WEALTHFOLIO_SYNC_DB_CONNECTION_STRING from env. Falls back to
DB_CONNECTION_STRING for single-DB local dev.

13 new tests covering: latest-per-account, multi-currency FX, explicit
as-of, empty mirror, null cost_basis, full pipeline through upsert.
140 tests pass; mypy strict + ruff clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Viktor Barzin 2026-05-09 21:33:48 +00:00
parent 8880bd377f
commit 23d11bdf6d
5 changed files with 432 additions and 203 deletions

View file

@ -2,7 +2,7 @@
Sub-commands:
- migrate alembic upgrade head
- ingest [wealthfolio] load wealthfolio sqlite into account_snapshot
- ingest [wealthfolio] pull from wealthfolio_sync PG mirror into account_snapshot
- simulate run a single scenario, pretty-print
- recompute-all run the 120-scenario Cartesian, persist all
- serve run the FastAPI on-demand /recompute server
@ -21,10 +21,15 @@ from pathlib import Path
import click
import numpy as np
from sqlalchemy.ext.asyncio import async_sessionmaker
from fire_planner.db import create_engine_from_env, make_session_factory
from fire_planner.glide_path import get as get_glide
from fire_planner.ingest import wealthfolio as wf_ingest
from fire_planner.ingest.wealthfolio import upsert_snapshots
from fire_planner.ingest.wealthfolio_pg import (
create_wf_sync_engine_from_env,
read_account_snapshots_from_pg,
)
from fire_planner.reporters.cli import format_scenario
from fire_planner.reporters.pg import write_run
from fire_planner.returns.bootstrap import block_bootstrap
@ -57,32 +62,38 @@ def migrate() -> None:
type=click.Choice(["wealthfolio"]),
default="wealthfolio",
help="Data source — currently only wealthfolio is wired.")
@click.option("--db-path",
type=click.Path(exists=True, dir_okay=False, path_type=Path),
required=False,
help="Local sqlite path (after kubectl exec). Required for --source=wealthfolio.")
@click.option("--as-of",
type=click.DateTime(formats=["%Y-%m-%d"]),
default=None,
help="Snapshot date to read; defaults to MAX(snapshot_date) in the sqlite.")
def ingest(source: str, db_path: Path | None, as_of: date | None) -> None:
help="Valuation date to read; defaults to MAX(valuation_date) in the mirror.")
def ingest(source: str, as_of: date | None) -> None:
"""Pull external state into fire_planner.account_snapshot."""
if source == "wealthfolio":
if db_path is None:
raise click.UsageError("--db-path is required for --source=wealthfolio")
asyncio.run(_ingest_wealthfolio(db_path, as_of))
asyncio.run(_ingest_wealthfolio(as_of))
async def _ingest_wealthfolio(db_path: Path, as_of: date | None) -> None:
rows = wf_ingest.read_account_snapshots(db_path, as_of=as_of)
async def _ingest_wealthfolio(as_of: date | None) -> None:
"""Read account snapshots from wealthfolio_sync PG mirror, upsert."""
wf_engine = create_wf_sync_engine_from_env()
try:
wf_factory = async_sessionmaker(wf_engine, expire_on_commit=False)
async with wf_factory() as wf_sess:
rows = await read_account_snapshots_from_pg(wf_sess, as_of=as_of)
finally:
await wf_engine.dispose()
if not rows:
click.echo("warning: no rows read — wealthfolio sqlite empty or schema unrecognised",
err=True)
click.echo(
"warning: no rows read — wealthfolio_sync mirror is empty or "
"no rows on the requested date",
err=True,
)
engine = create_engine_from_env()
factory = make_session_factory(engine)
try:
async with factory() as sess:
n = await wf_ingest.upsert_snapshots(sess, rows)
n = await upsert_snapshots(sess, rows)
await sess.commit()
click.echo(f"wealthfolio ingest: {n} rows upserted")
finally: