Conversational ETL

A batch ETL pipeline that extracts structured conversational data from PDF chat exports, validates it, and loads it into SQLite.

Released

Nov 3, 2025

Status

Done

Source

GitHub

Key features

• The etl-pipeline command and etl_pipeline.pipeline.run_pipeline both call execute_full_pipeline.
• Extracts and cleans PDFs, parses them into rows, writes a structured CSV and diagnostics, and loads SQLite. Charts are optional.
• Fails closed when a required directory is missing, no PDFs are found, a subprocess step fails, a required CSV is missing or empty, or chat_history has no rows.
• A zero exit code means those required files and the database load passed. It does not mean every row is usable or that the PDF count matched.
• validate_csv.py can check headers, blank fields, and duplicate sequences. The orchestrator does not call it.

How to use

Python 3.9 or newer. From the repository root. Install, then run. The command below uses the sample PDFs in examples/sample_input/. For your own export, point --input at that directory. PyMuPDF is optional, and only a fallback when pdfplumber returns no text: pip install -e ".[pymupdf]".

pip install -e .
python -m etl_pipeline.cli run --input examples/sample_input --output results --no-with-visuals

That writes results/run_YYYYMMDD_HHMMSS/. Using the repository results/ folder may archive older run folders there first. Pass a different --output directory to skip that. Omit --no-with-visuals to draw charts, which is the default. The run folder holds the CSVs, chat_history.db, and diagnostics/.

Related work