fth-catalog and loading data

Import CSV files into the Parquet catalog and verify it.

Commands

fth-catalog <root> instrument add <id> <price_precision> <size_precision> <tick>
fth-catalog <root> import quotes <id> <file.csv>
fth-catalog <root> import trades <id> <file.csv>
fth-catalog <root> list
fth-catalog <root> verify

Quote CSV

ts_event,bid,ask,bid_size,ask_size, with ts_event in Unix nanoseconds. A value with more decimals than the instrument allows is an error, never rounded.

Trade CSV

ts_event,price,size,aggressor,trade_id, where aggressor is B, S or N.

Layout

instruments.tsv
MANIFEST.tsv
quotes/<instrument>/<YYYY-MM-DD>.parquet
trades/<instrument>/<YYYY-MM-DD>.parquet
bars/<instrument>/<YYYY-MM-DD>.parquet

Integrity

• Partitions are by UTC day and are written once.
• Every file is listed in the manifest with its rows, time range and SHA-256, and the manifest carries a checksum over its own entries.
• A read refuses a file whose hash does not match.
• verify lists mismatched, missing and unlisted files and exits 2 if it finds any.
• A write lock stops two writers losing each other's updates.
• Crossed quotes and off-tick prices are refused at import.

Known limits

The hash is checked on the bytes read, then the file is reopened for parsing; a change between the two reads is not detected. Only CSV import exists.