Building a Literature Review as a Reproducible Data Pipeline

There are two ways to read twenty papers for a review. The first is the default: open PDF one, highlight, move on. By paper fourteen you cannot remember why paper three got included. By paper twenty your "method" is a feeling. The second way is to treat the review as a data engineering problem, where the papers are rows, the decisions are logged, and the final table is generated by a script instead of assembled by hand at 2 a.m.

My Big Data Analytics course review ("Big Data with Machine Learning: A Review", twenty papers from four journals, 2025 onward) is being built the second way. Here is the pipeline, stage by stage.

Stage 1: the pool is a query, not a memory

Everything starts with the database, not with a PDF. Scripts (build_pool.py, pool_openalex.py) pull candidates from OpenAlex by query, normalize the metadata into one JSON pool file, and record the query strings alongside. The selection criteria (four journals, publication year, relevance screen) are code, so running the pipeline again next month yields the same pool or a diff that says exactly what changed.

This matters because a review's coverage claim is only as good as its query. "We searched the literature" is not a method. A named query against a named database on a named date is.

Stage 2: screening is a decision log

screen_pool.py walks the pool and logs every include or exclude decision with its reason. Twenty papers in, the rest excluded, each with a one-line justification tied to a criterion written before screening started. The flow diagram in the paper (PRISMA-style: identified, screened, selected) comes out of that log, which means the numbers add up because they are computed, not estimated.

The test I apply: a reviewer asks "why isn't paper X in your review?" If the answer is "I hadn't seen it", the review has a hole. If the answer is a logged decision with a criterion, the review has a boundary. Reviews are allowed boundaries. They are not allowed holes.

Stage 3: extraction notes, one file per paper

Each included paper gets a fixed set of fields extracted into its own note: problem setting, ML paradigms used, data modality, platform, domain, reported results, stated limitations. The format is identical across all twenty, which feels rigid and is exactly the point. Fixed fields are what let you compare papers without rereading them, and what make the notes loadable by a script.

Stage 4: the table regenerates itself

The main comparison table (table1.csv) is generated from the extraction notes by script. Change a note, rerun, the table updates, including the citation numbers. The taxonomy that organizes the review (ML paradigm × data modality × platform × domain) is built the same way. Nothing in the final paper's quantitative scaffolding is hand-maintained, because hand-maintained tables are where stale numbers come from.

When this is overkill

For a course essay responding to three assigned readings, this system is pure ceremony. Set it up only when at least two things are true: the paper set is large enough that memory fails (somewhere around fifteen papers, in my experience), and someone will audit your selection, which includes every reviewer who reads the words "systematic review".

The setup cost is real, roughly a day of scripting on top of the reading. It pays itself back the first time the paper set changes, which always happens eventually: swap two candidates and the pool, the log, and the table absorb it in minutes instead of an afternoon of archaeology.

The takeaway

A literature review is a dataset about papers, so give it the properties you would want in any dataset: a recorded collection procedure, decision logs, a schema, and outputs regenerated from source. The reading itself stays manual, and should. What the pipeline removes is the part where twenty papers of careful reading collapse into one table you typed from memory.