Project brief
This concept would create a local analysis tool for checking sensor and simulation datasets before they enter an engineering workflow. A small schema would define channel names, units, expected types, timestamp rules, and optional plausibility bounds. The validator would distinguish a structural error from a flagged value that still deserves engineering review. A review screen and machine-readable issue file would show where a rule was applied, while preserving the original data. Example fixtures would include deliberately malformed files so the behavior can be demonstrated without fabricated operational datasets. Proposed deliverables are a schema format, a command-line interface, and a concise review interface. It would remain a data-quality aid whose checks depend on the assumptions supplied by the user.
Design concept developed for this portfolio. The diagrams describe the proposed system; implementation and testing are part of the validation plan.
The engineering challenge
Catch preventable data-handling errors while preserving source values and making every rule traceable.
Engineering approach
- Define channel, unit, and timestamp schemas.
- Separate parsing errors from engineering review flags.
- Produce row-level issue references and summary counts.
- Exercise the workflow with intentionally malformed examples.
Validation plan
- Use fixtures containing known unit and timestamp errors.
- Verify that source data remains unchanged.
- Check that each reported issue resolves to a source row and rule.
Concept scope
- Unit-aware channel schema
- Timestamp consistency checks
- Missing-channel detection
- Original-data preservation
- Machine-readable issue export
Software & engineering tools
Python, pandas, Pint, JSON Schema, Streamlit