Ritesh Sonawane

Types of LLM Explained: Dense, MoE, and Everything In Between

Sat, 18 Jul 2026 00:00:00 +0000

Ask someone what model they’re running and they’ll usually give you a name, Llama, GPT-4, DeepSeek. Ask them what architecture that model actually uses and why it was picked, and the conversation usually stalls.

That gap matters. The architecture behind a model decides how it processes tokens, how much GPU memory it eats, how fast it responds, and what kind of hardware it wants. Two models with similar benchmark scores can have completely different cost and latency profiles once you look under the hood. If you work in AI infrastructure, this is the layer where your capacity planning, your hardware choices, and your cost per token all actually come from.

LLM Inference KPIs Every SRE Should Know

Mon, 06 Jul 2026 00:00:00 +0000

Understanding LLM Inference KPIs: From TTFT to KV Cache

Monitoring LLM inference is not like monitoring a web service. The metrics are different, the failure modes are different, and the relationship between what you measure and what the user experiences is more indirect. This post covers the core KPIs, what they actually measure, and how to reason about them operationally.

First, Understand the Two Phases

Every LLM inference request goes through two distinct phases. If you do not understand the difference, the metrics will not make sense.

Kubernetes Gateway API Inference Extension

Fri, 26 Jun 2026 00:00:00 +0000

Kubernetes Gateway API Inference Extension: What It Is and Why It Matters

If you are running LLM workloads on Kubernetes today, you have almost certainly hit the same wall. Your standard Ingress or Gateway routes traffic to pods based on HTTP path or headers. Round-robin distributes load across replicas. Everything looks fine from the control plane perspective. But your GPU utilization is uneven, some replicas are getting hammered while others sit idle, latency is unpredictable, and you have no way to differentiate a low-latency interactive chat request from a background batch job that can afford to wait.

NVIDIA DSX

Mon, 08 Jun 2026 00:00:00 +0000

Introduction

For years, building an AI data center meant assembling a puzzle from dozens of vendors, each speaking a different language. Chips from one company, software from another, cooling from a third, and a facility design that nobody had fully validated end to end. The result was expensive guesswork at massive scale.

NVIDIA’s answer to this is the DSX platform, announced at GTC Taipei on May 31, 2026. It is a full-stack, codesigned framework that gives infrastructure builders a single validated playbook to design, deploy, and operate AI factories at scale. The goal is simple: turn every megawatt into more intelligence, at the lowest possible token cost.

ndots in Kubernetes

Tue, 07 Apr 2026 00:00:00 +0000

Let’s see how the ndots option works in Kubernetes.

In Kubernetes, we connect to running pods either directly or via a Kubernetes Service. This post focuses on how service DNS resolution actually works under the hood.

The resolv.conf file

In Linux, /etc/resolv.conf is used by the system resolver for domain name resolution. Kubernetes automatically injects this file into every pod with the configuration needed to resolve services inside the cluster.

Here is an example from a CNPG cluster pod running in the database namespace:

Celery to Argo Workflows

Tue, 09 Dec 2025 00:00:00 +0000

This blog is based on my work at CloudRaft!

AI jobs often run for long periods on expensive hardware like GPUs. When a job fails halfway, you don’t just lose progress—you waste valuable time and costly resources. Workflow orchestration solves this by providing fault tolerance, letting you break complex tasks into manageable steps, set dependencies, and recover from failures. This is especially critical in machine learning, where robust, efficient execution is paramount.

Scaling Prometheus with Thanos

Tue, 22 Jul 2025 00:00:00 +0000

This blog is based on my work at CloudRaft!

Observability is a crucial pillar of any application, and monitoring is an essential component of it. Having a well-suited, robust monitoring system is crucial. It can help you detect issues in your application and provide insights once it is deployed. It aids in performance, resource management, and observability. Most importantly, it can help you save costs by identifying issues in your infrastructure. One of the most popular tools in monitoring is Prometheus.

100 TB from InfluxDB to Grafana Mimir

Mon, 07 Apr 2025 00:00:00 +0000

This blog is based on my work at CloudRaft!

Introduction

Observability is the backbone of any modern infrastructure, enabling organizations to monitor system health, optimize performance, and ensure seamless operations. However, when legacy observability systems reach their limits—whether due to scalability challenges, high costs, or lack of vendor support—businesses must pivot to more future-proof solutions.

The migration presented a unique set of challenges: handling seven years of monitoring data (roughly 100TB of uncompressed data), complex migration as there is no direct support to migrate from InfluxDB to Grafana Mimir, Grafana dashboards rewrite from InfluxQL to PromQL format for 100s of dashboards. In this blog, we’ll walk through the entire migration process, the challenges faced, and the architectural choices that enabled a seamless transition to Grafana Mimir.

Overview of gRPC

Sun, 09 Feb 2025 00:00:00 +0000

Taking HTTP/2 to next Level

Before understanding gRPC lets see what is RPC

What is RPC(remote procedure call)?

RPC is a communication protocol which allows a program to execute functions on a remote server as if it was a local function. RPC simplifies distributed computing by hiding network complexity.

Consider you have a program calling a function, but here this function execution is on a remote server. The program which is calling this function does needs to know anything about the network details.

Monitoring with Prometheus

Thu, 04 Jul 2024 00:00:00 +0000

This blog is based on my work at CloudRaft!

Monitoring is the lifeline that ensures the seamless operation of systems, applications, and networks. Whether managing a small startup or a large enterprise, the ability to monitor infrastructure in real time is crucial for maintaining performance, ensuring reliability, and quickly resolving issues before they escalate. The average cost of downtime in IT industry as per Pingdom is between $5600 to $9600 per minute and as more and more digitization happens, it is going to increase further.

Streamlining Model Lifecycle with KubeRay

Fri, 24 May 2024 00:00:00 +0000

This blog is based on my work at CloudRaft!

Introduction

As the demand for ML and LLM models continues to grow, so does the need for reliability and scalability. Integrating Kubernetes into model development emerges as a powerful solution. By leveraging Kubernetes, we can streamline the process of model development, decrease costs, and enhance model reliability. This can be achieved using Ray on Kubernetes. But before that, let’s take a look at the lifecycle of an ML model.

Making Kubernetes Simple with Talos

Tue, 07 May 2024 00:00:00 +0000

This blog is based on my work at CloudRaft! (My first blog at CloudRaft)

Kubernetes marks its 10th anniversary this year with the release of version v1.30.0, solidifying its status as the cloud platform of choice. Managed Kubernetes clusters like EKS, GKS and AKS represent 73% of the total cluster, remaining 27% are self-managed as per Dynatrace. The last decade has been an era of public cloud but due to increasing costs, some businesses are trying to find a balance with the hybrid cloud. Approximately 76% of organizations now leverage multiple clouds which is a combination of public and private clouds as per VMware. Kubernetes allows us to build a multi-cloud and private cloud layer on the hardware of choice and in a cost-effective way without committing to one specific cloud.

About

Mon, 01 Jan 0001 00:00:00 +0000

Experience

Site Reliability Engineer — CloudRaft

April 2024 – March 2026 • Remote

Owned infrastructure architecture and platform engineering across 10+ client environments spanning telecom, AI/ML, fintech, and enterprise SaaS.

EnkryptAI — AI/ML Infrastructure & Production Operations

Owned complete AWS EKS infrastructure and VPC-level architecture, serving as DevOps Lead for enterprise client deployments
Architected and developed 3 cloud-agnostic Helm Charts (EnkryptAI Stack, Platform Stack, Platform Core) to orchestrate 70+ microservices in fully air-gapped multi-cloud deployments (AWS, GCP, Azure) with GPU node support and VRAM-based scheduling
Onboarded 2 enterprise clients on AKS and EKS environments
Re-architected Redteaming job orchestration by migrating from Celery-based execution to a Kubernetes-native stack using Argo Workflows, Argo Events, and NATS
Created a Golang-based container entrypoint script, reducing Kubernetes pod startup time from 5 minutes to 30 seconds for a NextJS application
Managed on-call rotation using incident.io for guardrails and red-teaming services, ensuring production uptime and incident response
Deployed production guardrails application on NVIDIA A30/H100 GPUs using NVIDIA GPU Operator
Resolved critical compatibility and deployment issues with NVIDIA GPU Operator on Azure Kubernetes Service (AKS)
Deployed on-prem OpenFGA integrated with CloudNativePG (CNPG) as the backing PostgreSQL database to implement fine-grained, scalable RBAC across the platform
Migrated Elasticsearch to OpenSearch using the OpenSearch Kubernetes Operator and transitioned Kibana dashboards to OpenSearch Dashboards, ensuring continuity of audit logging
Customized and deployed Supabase Helm Chart with CloudNativePG (CNPG) to migrate Supabase from managed cloud to on-prem VPC deployment within air-gapped enterprise environments
Implemented DevSecOps practices by integrating SBOM generation and Grype-based vulnerability scanning into CI/CD pipelines, with automated Slack alerts for high-severity findings
Configured Devspace for developers enabling rapid deployment to AWS GPU nodes
Supported SOC 2 and ISO 27001 compliance initiatives

Composio — Enterprise SaaS Platform & Customer Onboarding

Implemented Replicated as enterprise portal for customer onboarding, reducing onboarding time from days to 1 hour
Migrated existing clients to Replicated platform, enabling centralized release management and automated updates
Wrote preflight and postflight validation checks for installation reliability
Built multi-channel CI/CD pipeline with GitHub Actions supporting Unstable, Stable, and Nightly releases
Led client onboarding for 6+ production environments across AWS EKS and GCP GKE
Customized Temporal Helm charts for client-specific requirements
Automated deployment of unstable Replicated releases to GKE for continuous testing

SkySwitch — 100TB Monitoring Migration (Telecom)

Led migration of 100TB+ InfluxDB data to Grafana Mimir with zero downtime
Built custom Golang conversion tool to transform InfluxDB data to OpenMetrics format for Grafana Mimir
Migrated 1000+ Grafana dashboards from InfluxQL to PromQL
Re-architected single-tenant Mimir to multi-tenant model, reducing compactor startup time from hours to 5 minutes
Reduced infrastructure compute and memory usage by 30%

NeevCloud — GPU Cloud Platform

Built production-grade GPU cloud platform on bare-metal Kubernetes (kubeadm) with KubeVirt and NVIDIA GPUs
Developed Golang-based VM provisioning API with GPU-aware scheduling and inventory validation logic
Implemented custom x-api-key authentication middleware for VLLM model endpoints using Traefik
Configured Cilium CNI to assign public IPs to KubeVirt VMs
Resolved GPU passthrough issues and fixed VM network persistence using KubeMacPool
Enforced AWS-style resource quotas to prevent over-provisioning
Achieved 99.9%+ uptime for commercial GPU workloads with CI/CD via GitHub Actions, ArgoCD, and Harbor

Rezolve.ai — Enterprise Observability & Cost Optimization

Designed and deployed an observability stack across 4 production AKS clusters using Prometheus, Grafana, OpenTelemetry Operator, Loki, and Tempo
Enabled RED metrics and service graphs for production services
Identified infrastructure inefficiencies using Steampipe/Powerpipe, delivering $36,000 annual cost savings
Led Kubernetes version upgrades to v1.31 across all clusters
Planned and executed Jenkins server upgrade

Ditto — Multi-Cloud BYOC Platform

Using Cluster API, provisioned kubeadm clusters on client environments across AWS (CAPA), GCP (CAPG), and Azure (CAPZ)
Developed custom Golang migration script for transition from MachinePool to MachineDeployment for AWS and GCP based Kubernetes clusters
Automated deployment of Velero, Cluster Autoscaler, and Node Problem Detector using ArgoCD ApplicationSets across multi-cloud environments

Clika — AI Job Orchestration Platform

Built Golang-based Kubernetes job scheduler with credit-based billing and a ledger system
Implemented S3-compatible storage integration for ML model outputs with access controls
Integrated VictoriaMetrics and VictoriaLogs for real-time job observability with custom API endpoints
Provisioned GKE infrastructure via Terraform with CI/CD automation using ArgoCD and GitHub Actions

IFF — Enterprise Authentication Integration (Fortune 500)

Implemented organization-wide SAML-based authentication for ArgoCD, enhancing secure access management across enterprise deployments

DrDroid — CI/CD Modernization

Modernized bash-based deployments to production-grade CI/CD pipeline using GitHub Actions and ArgoCD with Image Updater
Set up Metabase for database analytics and insights

Kubernetes Homelab

I run a multi-node k3s cluster at pidoku.co.in to test cloud-native infrastructure patterns before using them in production.

Work with me

Mon, 01 Jan 0001 00:00:00 +0000

I am available for remote SRE, platform engineering, DevOps, and Kubernetes infrastructure work.

What I do

Kubernetes platforms. Production clusters on managed Kubernetes (EKS, GKE, AKS) and on bare metal with kubeadm, Talos, and k3s. Cloud-agnostic Helm charts, including fully air-gapped multi-cloud installs.

Observability. Prometheus, Grafana, Thanos, Mimir, Loki, Tempo, OpenTelemetry, VictoriaMetrics. RED metrics, service graphs, and alerting that people actually act on.

Migrations. Moving large systems without downtime — time-series databases, cloud-to-cloud, orchestration engines, logging stacks.