Building FeatureSmith: Bringing Code-Review Discipline to ML Datasets

### The Data Quality Gap in Modern Machine Learning

In software engineering, code quality is enforced by automated pipelines: static analysis, linters, type checkers, and unit test suites block broken code from reaching production.

However, in machine learning engineering, datasets are frequently fed into training pipelines with minimal automated validation. Silent dataset bugs—such as target leakage, hidden missingness, schema drift, and invalid identifier correlation—go undetected until a model fails in validation or production.

What is Target Leakage?

Target leakage occurs when a feature contains information that will not be available at prediction time. For example: - A timestamp feature recorded after the prediction event. - An ID or status code that indirectly encodes the target label. - High correlation between a preprocessed vector and the target variable.

When target leakage is present, a model achieves artificially high training accuracy (e.g. 99.9%) but fails completely in real-world deployment.

How FeatureSmith Solves It

FeatureSmith introduces 8 automated reviewers powered by Polars that run pre-training dataset reviews in milliseconds: 1. **Schema Consistency**: Detects unexpected column drops or type mutations. 2. **Missingness Ratios**: Highlights critical missingness thresholds. 3. **Cardinality & Uniqueness**: Identifies identifier columns and low-variance features. 4. **Target Leakage Detectors**: 6 specialized heuristics catching temporal leaks and correlations. 5. **ML Readiness Scorecard**: Computes an explainable 0–100 quality score.

import featuresmith as fs

# Run pre-training dataset review report = fs.review("data/train.csv", target="survived")

print(f"Readiness Score: {report.score}/100") if report.has_leakage: print("CRITICAL: Target leakage detected!") ```

Deterministic CI/CD Exit Gates

FeatureSmith provides a CLI that exits with `0` when quality checks pass and `1` when critical findings exist. This allows data teams to run dataset reviews automatically inside GitHub Actions before expensive training jobs start.