ETLLIB
A command-line toolkit and Python library
for JSON, TSV, and Solr ETL
A command-line toolkit and Python library
for JSON, TSV, and Solr ETL
DARPA XDATA corpora such as Computrabajo employment TSVs and Kiva JSON dumps arrive as huge aggregates with mixed encodings, optional columns, and paging wrappers. They are not Solr documents yet.
You often need a small, installable set of commands — TSV to JSON, split an aggregate, post to Solr — that a person can run from the shell or that a larger workflow can invoke. You do not need another engine.
Mnemosyne-style pipelines such as BigTranslate call these tools as PGE steps. The same functions should also import cleanly from Python.
A command-line toolkit and Python library for munging JSON, TSV, and related data. ETLLib uses Apache TikaTM where field cleanup helps, and posts the result to Apache SolrTM. It is not Mnemosyne. Workflow engines call it; they do not replace it.
tsvtojson, repackage, poster, repackageandpost, translatejson, and similarity install on your PATH and do one job each.
The same steps are functions in etl.etllib. If you would rather not shell out, import them. The CLIs are thin wrappers around that API.
BigTranslate and similar pipelines invoke ETLLib as PGE commands for TSV conversion, JSON splitting, and Solr ingest. Bring your own workflow; ETLLib stays small.
A typical employment-TSV or Kiva-JSON path is three commands. Encoding recovery, optional columns, and Tika field cleanup happen along the way.
tsvtojson reads a column-header file and an encoding list, then emits an aggregate JSON document keyed by object type — for example employmentjobs.
repackage pulls each record out of the wrapper (and can run Tika over messy fields). One country-day file becomes thousands of per-job JSON documents.
poster (or repackageandpost, skipping the intermediate files) sends each document to Solr so the corpus can be queried and faceted.
A walkthrough from CSV/TSV through tsvtojson and repackage is on the project wiki.