This blog is based on my work at CloudRaft!
AI jobs often run for long periods on expensive hardware like GPUs. When a job fails halfway, you don’t just lose progress—you waste valuable time and costly resources. Workflow orchestration solves this by providing fault tolerance, letting you break complex tasks into manageable steps, set dependencies, and recover from failures. This is especially critical in machine learning, where robust, efficient execution is paramount.
This blog is based on my work at CloudRaft!
Observability is the backbone of any modern infrastructure, enabling organizations to monitor system health, optimize performance, and ensure seamless operations. However, when legacy observability systems reach their limits—whether due to scalability challenges, high costs, or lack of vendor support—businesses must pivot to more future-proof solutions.
The migration presented a unique set of challenges: handling seven years of monitoring data (roughly 100TB of uncompressed data), complex migration as there is no direct support to migrate from InfluxDB to Grafana Mimir, Grafana dashboards rewrite from InfluxQL to PromQL format for 100s of dashboards. In this blog, we’ll walk through the entire migration process, the challenges faced, and the architectural choices that enabled a seamless transition to Grafana Mimir.