Data Source Verification

A systematic workflow for verifying that every data point in a research dataset can be traced back to its original source paper, figure, table, or text passage.

When to Use

• - Building datasets from literature (CSV, JSON, tables)
• Populating tables or plots with values from multiple papers
• Reviewing existing datasets for data integrity
• Before submitting any paper that includes compiled data

Core Rule

Every numerical value must be traceable to a specific location in the original paper. If you cannot find the value in the cited source, it is unverified and must be flagged — never included as confirmed data.

Data Provenance Chain

CODEBLOCK0

Every link in this chain must be auditable. If someone asks "where did this number come from?", the answer should be: paper X, Table Y, column Z — and we have the PDF to prove it.

Citation Source Management

Project Setup (init)

Create a Citation_Sources/ directory for the project:

CODEBLOCK1

CITATION.md Template

Every cited paper gets a CITATION.md file:

CODEBLOCK2

Adding a Source (add)

When adding a new citation:

1. 1. Create the folder: INLINECODE3
2. Download the original PDF — always try to get the actual paper, not just the abstract
3. Download supplementary information if it contains data
4. Create CITATION.md from the template
5. Extract the specific values you need, recording exact table/figure/page locations
6. Mark the PDF status and verification status

Verification Workflow

Step 1: Collect with Provenance

When extracting data from a paper, record ALL of the following for each value:

CODEBLOCK3

Never record a value without filling in the Location, Data type, and Verified fields.

Step 2: Verify Against Original

For each data point:

1. 1. Always download the original PDF — don't trust web scraping, abstracts, or secondary sources
2. Find the exact value in a table, figure, or text passage
3. Record where you found it — table number, figure number, page, equation
4. Note the measurement method — experimental technique, simulation, estimate
5. Check units — convert if needed, note the original units
6. Track the data type: DFT-calculated, experimentally measured, or derived (note assumptions)

If the paper is behind a paywall and you cannot verify:

• - Mark as INLINECODE4
• Note this limitation in CITATION.md

Step 3: Cross-Check the Full Chain

Verify consistency at every step:

CODEBLOCK4

Any mismatch at any step is a flag.

Step 4: Flag Problems

Mark any value with one of these status levels:

Step 5: Flag Discrepancies

When multiple sources report different values for the same quantity:

• - Record both values with their sources
• Note the discrepancy explicitly (e.g., "B = 45 GPa (Author A, Table 2) vs B = 86 GPa (Author B, Fig. 3)")
• Check if the difference is due to measurement method, sample preparation, or temperature
• Let the user decide which value to use — do not silently pick one

Dataset Format

When building compiled datasets, always include provenance columns:

CSV format:
CODEBLOCK5

JSON format:

{
  "material": "Li6PS5Cl",
  "property": "thermal_conductivity",
  "value": 0.69,
  "unit": "W/m·K",
  "source": {
    "paper": "Cheng et al. 2021",
    "doi": "10.1002/smll.202101693",
    "location": "Table 2, row 5",
    "method": "TDTR",
    "dataType": "experimental",
    "verified": true
  }
}

Audit Workflow (audit)

Scan all CITATION.md files and generate a report:

1. 1. List all unique sources in Citation_Sources/
2. For each source, check:

1. 3. Generate audit summary:

CODEBLOCK7

1. 4. Report findings — list verified, flagged, and misattributed values
2. Recommend action for each flagged value

Export (export)

Generate a summary table of all data values and their provenance:

CODEBLOCK8

Red Flags

Watch for these indicators of unreliable data:

• - Value attributed to a paper but no specific table/figure cited
• "Estimated from family properties" without a clear methodology
• Values that appear in reviews but cannot be traced to original measurements
• Round numbers that suggest estimation rather than measurement (e.g., 2800 m/s vs 2837 m/s)
• Same value appearing in multiple papers without independent measurement
• DFT values presented as experimental without noting the distinction
• Discrepancies between different sources for the same quantity left unaddressed

Rules

1. 1. Never assume a citation is correct — always verify against the original paper
2. Always download the PDF — don't trust abstracts, web scraping, or secondary sources
3. Secondary sources are not verification — a review paper citing a value does not confirm it
4. Flag immediately when a value cannot be found in its cited source
5. Track data type — distinguish DFT-calculated, experimentally measured, and derived values
6. Flag discrepancies — when two sources disagree, note both values and let the user decide
7. Prefer measured over estimated — clearly label the difference
8. Document everything — future researchers need the audit trail
9. When in doubt, exclude — a smaller verified dataset beats a larger unverified one