NASA / Data integrity & investigative analytics
Follow a finding back to the data.
USAspending / IDAS checks how spending records survive ingestion and normalization, then packages the evidence an analyst needs to investigate a discrepancy.
A suspicious pattern is only useful if its records can be trusted.
A missing award, a repeated identifier or a sudden change in row counts can originate in the spending data—or in the pipeline that moves it. The module checks those transformations before asking an analyst to interpret the pattern.
The design keeps the inspected data unchanged. Reports, samples and configuration snapshots are written to a separate evidence location; optional schema proposals enter a governed review process.
01 / System anatomy
Read the data. Preserve the evidence.
CSV → Parquet
API extract and converted files
- Compare
- Counts for the same slice
- Exact path
- Normalized row differences
- Retain
- Manifest expectations
Bronze
Staged awards with lineage
- Compare
- Bronze ↔ Silver slice
- Distinguish
- Normalization / deduplication
- Retain
- Source count and slice context
Silver
Normalized, deduplicated awards
- Assess
- Candidate key quality
- Compare
- Schema ↔ approved registry
- Inspect
- Parquet reader / file health
CLI · Notebook · Databricks Job
The same specified checks, with different execution entry points.
Requested mode → effective mode → fallback reason
Configuration · UTC timestamps · run URL · lock ownership
Investigation bundle
Report + findings + bounded samples + manifests + configuration + logs
Inspect the bundle ↓Schema proposal
Pending → approved → effective
Or rejected, with the review context retained
02 / What is actually checked
Six signals, with different meanings.
| Signal | Inputs and calculation | Default decision / retained evidence |
|---|---|---|
| CSV ↔ Parquet parity | Absolute count difference divided by max(1, CSV count), for the same slice. | PASS at ≤1%; otherwise FAIL. Exact mode compares normalized rows in both directions. |
| Bronze ↔ Silver parity | Absolute count difference divided by max(1, Bronze count). | PASS at ≤5%; otherwise FAIL. Keep slice context: intentional deduplication also changes row counts. |
| Natural-key quality | Sample candidate identifiers; assess missing values and a duplicate-rate proxy. Refine the null rate near a threshold. | Null-rate thresholds: INFO >1%, WARN >5%, FAIL >10%. Refinement band ε = 0.005. Keep the selected key and estimates. |
| Schema drift | Compare normalized column names, types and nullability with the approved registry. | Match, additive change, widened nullability or breaking change; retain the fingerprint and structural diff. |
| Parquet reader errors | Attempt a Spark read; collect offending paths from the exception. | FAIL when offending paths are identified; WARN if the reader fails without a located path. This bounded check is not an exhaustive file scan. |
| Small-file pressure | Inspect Parquet file sizes and count. | WARN if any file is ≤4 MiB, or the scan set contains >500 files. Retain counts and a sample of paths. |
Equal row counts do not establish equal contents. A schema fingerprint checks a column definition, not the values in every row. Each signal answers a narrower question than “is this dataset correct?”
03 / Work through the decision
What does an 80% row-count gap mean?
The specification’s sample finding contains 31,625 Bronze rows and 6,325 Silver rows. Applying the Phase 3 rule gives a difference of 25,300 rows, or 80% of the Bronze count. That exceeds the default 5% threshold.
The check identifies a discrepancy to explain. It does not establish whether the cause is an intentional reduction, a slice mismatch, missing input, or faulty processing. Exact row comparison and the transformation record are the next evidence to inspect.
When “exact” falls back to a count comparison
The supplied exact-difference function uses a default ceiling of 1,000,000 rows on each side. If either side exceeds that ceiling, it returns approximate mode and the reason exact_max_rows_exceeded. A request for exact mode must therefore be distinguished from the mode actually executed.
Keep mode_requested, mode_effective and fallback_reason in the finding and resolved configuration. Mode handling ↗ · Exact-difference pseudocode ↗
04 / The deliverable
A finding travels with its context.
The analyst receives more than a score. The evidence root keeps the explanation, row samples and execution context together under an investigation identifier.
INV-YYYYMMDDThhmmssZ-<uuid>/
UTC timestamp + unique run identifier
report.md
Read the human summary
Check outcomes, explanations and the link back to the execution run.
findings.jsonl
Inspect individual findings
Finding code, severity, details and execution-mode provenance.
config.json · metrics.json
Recover the conditions
Resolved parameters, counters, sample allocation and truncation.
samples/
Inspect the underlying rows
Bounded Parquet samples allocated across findings, rather than an unlimited data dump.
artifacts/
Recover the data definitions
Manifest snapshots, schema snapshots, registry match and structural differences.
logs.ndjson
Trace the execution
Structured events with investigation context and UTC timestamps.
PER FINDING
500 rows
Default sample ceiling.
PER INVESTIGATION
2,000 rows
Total default ceiling, distributed round-robin.
WHEN A CAP IS REACHED
Record truncation
A bounded sample is a view into the finding, not a complete export.
Inspect the source example’s finding fields
This is the small illustrative payload on page 7, with the execution identifier and timestamp omitted here. The calculation above applies the later Phase 3 rule to these counts.
{
"code": "ROW_PARITY",
"severity": "FAIL",
"message": "Silver row count deviates",
"details": {
"bronze_rows": 31625,
"silver_rows": 6325
}
}
The example uses the generic code ROW_PARITY; §5.8 later distinguishes ROW_PARITY_P1 and ROW_PARITY_P3. Specification · p. 7 ↗ · Canonical codes ↗
05 / Questions for an investigator
From an unusual pattern to a reviewable question.
| Screen | Pattern to inspect | Question for review |
|---|---|---|
| Split-threshold patterns | Clusters of awards just below a configured limit within a vendor/time window. | What procurement context explains the grouping? |
| Duplicate awards | Repeated keys or similar vendor/date/amount combinations. | Separate awards, amendments, or duplicated records? |
| Round-dollar concentrations | Unusual frequency of exact multiples or endings. | Does the award type or reporting convention explain it? |
| End-of-period spikes | Concentrated obligations near a fiscal boundary. | Expected seasonality or a change needing explanation? |
| Vendor identity inconsistency | One identifier associated with differing names, or the reverse. | Alias, organizational change, or incorrect linkage? |
| Geography / type mismatch | A record disagrees with configured geography/type expectations. | Is the rule applicable, and is the record mapped correctly? |
The third column supplies reading questions, not additional detection rules. The detailed implementation material in this document focuses on data-integrity checks and evidence handling.
06 / Change with an audit trail
A different schema needs an explanation.
A schema fingerprint hashes normalized column names, types and nullability. A changed fingerprint leads to a structural comparison: which columns appeared, disappeared, changed type or began allowing missing values?
The registry keeps approved schema versions separate from proposed changes. The proposed version, structural diff and evidence location enter review together. A detection does not approve its own schema change.
| Change | Specified interpretation | Review consequence |
|---|---|---|
| Fingerprint matches | Approved structure | PASS. |
| New columns / widened nullability | Candidate minor evolution | WARN; review the proposed change. |
| Existing type changes / unresolved drops | Breaking change | FAIL; candidate major version. Apply approved rename mappings before classification. |
| Cosmetic changes | Patch-level guidance | Retain a note; the outline uses INFO for an otherwise unexplained fingerprint mismatch. |
- Submit Attach a diff, version proposal and evidence.
- Review Steward reviews; major changes have board oversight.
- Activate Platform engineering applies an approved version.
07 / Running the specification
Operational details that keep an investigation traceable.
One writer per scope
Locks carry an owner, scope, heartbeat and expiry. The specified default lifetime is 45 minutes; a stale lock has a separate reclamation path.
Locking · pp. 12–16 ↗Explicit outcomes
Exit codes distinguish success, failed acceptance, runtime error, pre-flight failure, lock conflict and invalid arguments. An execution failure should not look like a clean dataset.
Exit status ↗Recoverable context
CLI flags override notebook parameters, then environment values, then defaults. Saving the resolved configuration makes the conditions inspectable later.
Inputs and precedence ↗Read the complete specification.
Bee R. Davis. USAspending → IDAS Investigative Module: Technical Specification (v1.2.1). Title-page date: 8 October 2026. Developed for NASA.
The original 29-page PDF is unchanged. This companion explains the specified architecture, reconstructs its checks and calculates one illustrative count-parity example. No production investigation results are presented.