Skip to content

DataMeshOps-Guard

Category Prize Target — Challenge Category: Production ML Agents

DataMeshOps-Guard is an autonomous production ML protection agent that utilizes DataHub's metadata graph to shield ML models from target leakage, upstream schema drift, and feature store corruption.

What It Does

  1. ML Context Extraction — Inspects production ML model URNs, maps relationships to feature stores and raw datasets via DataHub MCP lineage traversal
  2. Leakage & Drift Auditing — Analyzes feature table schema definitions to identify:
  3. Target leakage: future outcome variables used as training features
  4. Schema drift: type conversions in upstream ingestion (e.g., FLOAT → STRING)
  5. Model Quarantine — Isolates the compromised model entity via graph mutations:
  6. add_tags: #ml-quarantined, #target-leakage-detected
  7. add_structured_properties: model_health_score, quarantine_reason, schema_drift_detail
  8. update_description: Full diagnostic Markdown report
  9. raise_incident: CRITICAL incident blocking downstream pipelines
  10. Downstream Blocking — Any CI/CD pipeline or agent reading DataHub will observe the quarantine state and abort

Running Live

export DATAHUB_GMS_URL="http://localhost:8080/api/gms"
export TOOLS_IS_MUTATION_ENABLED="true"
export OPENAI_API_KEY="your-key-here"

python3 -m datameshops_guard.cli audit --model customer_churn_v4

Offline Playback (Zero Setup)

python3 -m datameshops_guard.cli audit --model customer_churn_v4
# No API key needed — deterministic mock fallback runs automatically
# or view the pre-recorded cast:
# asciinema play examples/guard_demo_playback.cast

Example Output

See examples/guard_audit_report.json for full MCP tool call log with leakage detection.

See examples/guard_mutation_before_after.json for before/after DataHub aspect diff on the ML model entity.

Detected Issues in Demo

Issue Field Detail
Target Leakage future_churn_status_30d Encodes future outcome — cannot be a training feature
Schema Drift patient_age Type changed FLOAT → STRING in upstream patient_vitals ETL
Health Score — 0.12 / 1.00 — model unsafe for inference