DataMeshOps
DataMeshOps is an autonomous metadata-aware data remediation and ML governance engine built for the Build with DataHub: The Agent Hackathon. It features a high-performance shared core library (datameshops-core) powering two autonomous agents that deeply integrate with DataHub via the MCP Server.
Agents
🛠️ DataMeshOps-Healer — Grand Prize Target
Challenge: Agents That Do Real Work
Autonomously triages broken data pipelines, traces 3-hop upstream lineage via DataHub MCP, synthesizes non-destructive SQL fixes, and writes remediation knowledge back to the DataHub metadata graph.
🔒 DataMeshOps-Guard — Category Prize Target
Challenge: Production ML Agents
Protects production ML models by tracing feature store lineage, detecting target leakage and upstream schema drift, quarantining compromised models, and raising blocking governance incidents via DataHub MCP.
Quickstart
git clone https://github.com/rohitdas-ai/datameshops.git
cd datameshops
python3 -m venv venv && source venv/bin/activate
pip install -e .
# Run DataMeshOps-Healer (remediation agent)
python3 -m datameshops_healer.cli run --dataset nyc-taxi
# Run DataMeshOps-Guard (ML protection agent)
python3 -m datameshops_guard.cli audit --model customer_churn_v4
No API key needed — both agents run with deterministic mock fallback. Supply OPENAI_API_KEY or GEMINI_API_KEY for live LLM-powered execution.
Key DataHub Integration Points
| MCP Tool | Used By | Purpose |
|---|---|---|
search |
Both | Find entities in DataHub catalog |
get_lineage |
Both | 3-hop upstream/downstream traversal |
list_schema_fields |
Both | Schema field inspection |
get_dataset_queries |
Healer | Historic SQL query fetch |
add_tags |
Both | Write #auto-remediated, #ml-quarantined |
add_structured_properties |
Both | Write audit and health properties |
update_description |
Both | Append Markdown post-mortems |
raise_incident |
Guard | Create blocking governance incidents |
update_incident_status |
Healer | Resolve open incidents |