ingest: switch wealthfolio to pg-sync mirror reads
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
The previous SQLite-direct reader queried `holdings_snapshot` (singular) and `accounts.type` — both wrong against the live wealthfolio schema (plural `holdings_snapshots`, column `account_type`). It silently returned [] via the OperationalError fallback, leaving fire-planner with stale account snapshots. Switch to reading from the wealthfolio_sync PG mirror. The pg-sync sidecar (defined in infra/stacks/wealthfolio) hourly mirrors SQLite to Postgres with a clean schema. We read from `daily_account_valuation` which already has total_value, cost_basis, and explicit fx_rate_to_base per row — no JSON-decoding of position blobs. CLI ingest no longer takes --db-path (no kubectl-exec gymnastics); reads WEALTHFOLIO_SYNC_DB_CONNECTION_STRING from env. Falls back to DB_CONNECTION_STRING for single-DB local dev. 13 new tests covering: latest-per-account, multi-currency FX, explicit as-of, empty mirror, null cost_basis, full pipeline through upsert. 140 tests pass; mypy strict + ruff clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
parent
8880bd377f
commit
23d11bdf6d
5 changed files with 432 additions and 203 deletions
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Sub-commands:
|
||||
- migrate — alembic upgrade head
|
||||
- ingest [wealthfolio] — load wealthfolio sqlite into account_snapshot
|
||||
- ingest [wealthfolio] — pull from wealthfolio_sync PG mirror into account_snapshot
|
||||
- simulate — run a single scenario, pretty-print
|
||||
- recompute-all — run the 120-scenario Cartesian, persist all
|
||||
- serve — run the FastAPI on-demand /recompute server
|
||||
|
|
@ -21,10 +21,15 @@ from pathlib import Path
|
|||
|
||||
import click
|
||||
import numpy as np
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
from fire_planner.db import create_engine_from_env, make_session_factory
|
||||
from fire_planner.glide_path import get as get_glide
|
||||
from fire_planner.ingest import wealthfolio as wf_ingest
|
||||
from fire_planner.ingest.wealthfolio import upsert_snapshots
|
||||
from fire_planner.ingest.wealthfolio_pg import (
|
||||
create_wf_sync_engine_from_env,
|
||||
read_account_snapshots_from_pg,
|
||||
)
|
||||
from fire_planner.reporters.cli import format_scenario
|
||||
from fire_planner.reporters.pg import write_run
|
||||
from fire_planner.returns.bootstrap import block_bootstrap
|
||||
|
|
@ -57,32 +62,38 @@ def migrate() -> None:
|
|||
type=click.Choice(["wealthfolio"]),
|
||||
default="wealthfolio",
|
||||
help="Data source — currently only wealthfolio is wired.")
|
||||
@click.option("--db-path",
|
||||
type=click.Path(exists=True, dir_okay=False, path_type=Path),
|
||||
required=False,
|
||||
help="Local sqlite path (after kubectl exec). Required for --source=wealthfolio.")
|
||||
@click.option("--as-of",
|
||||
type=click.DateTime(formats=["%Y-%m-%d"]),
|
||||
default=None,
|
||||
help="Snapshot date to read; defaults to MAX(snapshot_date) in the sqlite.")
|
||||
def ingest(source: str, db_path: Path | None, as_of: date | None) -> None:
|
||||
help="Valuation date to read; defaults to MAX(valuation_date) in the mirror.")
|
||||
def ingest(source: str, as_of: date | None) -> None:
|
||||
"""Pull external state into fire_planner.account_snapshot."""
|
||||
if source == "wealthfolio":
|
||||
if db_path is None:
|
||||
raise click.UsageError("--db-path is required for --source=wealthfolio")
|
||||
asyncio.run(_ingest_wealthfolio(db_path, as_of))
|
||||
asyncio.run(_ingest_wealthfolio(as_of))
|
||||
|
||||
|
||||
async def _ingest_wealthfolio(db_path: Path, as_of: date | None) -> None:
|
||||
rows = wf_ingest.read_account_snapshots(db_path, as_of=as_of)
|
||||
async def _ingest_wealthfolio(as_of: date | None) -> None:
|
||||
"""Read account snapshots from wealthfolio_sync PG mirror, upsert."""
|
||||
wf_engine = create_wf_sync_engine_from_env()
|
||||
try:
|
||||
wf_factory = async_sessionmaker(wf_engine, expire_on_commit=False)
|
||||
async with wf_factory() as wf_sess:
|
||||
rows = await read_account_snapshots_from_pg(wf_sess, as_of=as_of)
|
||||
finally:
|
||||
await wf_engine.dispose()
|
||||
|
||||
if not rows:
|
||||
click.echo("warning: no rows read — wealthfolio sqlite empty or schema unrecognised",
|
||||
err=True)
|
||||
click.echo(
|
||||
"warning: no rows read — wealthfolio_sync mirror is empty or "
|
||||
"no rows on the requested date",
|
||||
err=True,
|
||||
)
|
||||
|
||||
engine = create_engine_from_env()
|
||||
factory = make_session_factory(engine)
|
||||
try:
|
||||
async with factory() as sess:
|
||||
n = await wf_ingest.upsert_snapshots(sess, rows)
|
||||
n = await upsert_snapshots(sess, rows)
|
||||
await sess.commit()
|
||||
click.echo(f"wealthfolio ingest: {n} rows upserted")
|
||||
finally:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue