Short lessons and coding problems in Python, with interactive demos of gradient descent, BPE tokenization, attention and the KV cache. ML foundations are open now; LLM internals and writing an LLM from scratch are next.
ML Inference Systems Engineer · Istanbul, Turkey · open to collaborations
Building production ML inference systems. Google Developer Expert, Cloud AI · Google Dev Group organizer · Speaker · OSS contributor
Forward diffusion. What a diffusion model is trained to undo: the photo x₀ mixed with Gaussian noise ε (seed 42), with ᾱt from diffusers' cosine schedule (squaredcos_cap_v2, 1000 timesteps). On load, the photo goes through the 50 timesteps a 50-step DDIM sampler visits, t = 980 down to 0; after that, the brush raises t wherever it passes, and t falls back to 0. No model runs here: the load-in is the noising played backward, not a model denoising, and the brush gives each pixel its own t, where a model sees one t for the whole image.
Continuous batching. Requests join and leave at every step, so the queue drains instead of waiting on the slowest reply.
Continuous batching. A request-level server reserves memory for the longest possible reply and runs each batch until its slowest request finishes, so under this traffic its queue fills up and requests get turned away. vLLM-style continuous batching lets requests join and leave at every step. Flip it and watch p99 time to first token.
KV cache. When the KV cache fills, the scheduler stops admitting and preempts requests: memory, not compute, caps how many run at once.
KV cache. Every token a request holds keeps its keys and values in GPU memory. When that pool is full, the scheduler stops admitting and preempts the newest request: its KV is dropped and it redoes its prefill later. Give the server less memory and fewer requests fit at once, so preemptions start sooner and throughput drops: memory, not compute, limits how many requests run at once. Synthetic traffic: long prompts and long replies.
MIG. Three models on one H100 split by MIG meet the same latency targets as three whole GPUs, with a third of the hardware.
MIG. Three models, each with its own traffic and latency target (p99 time to first token: 8B 2 s, 3B 500 ms, 1B 300 ms). Kubernetes hands each pod a whole GPU, so they take three underused H100s. Time-slicing shares one H100: the models take turns and every switch wastes GPU time (assumed here to be about 30%), so the 8B misses its target. MIG splits one H100 into isolated slices and meets every target with a third of the GPUs, though each slice has less headroom than a whole GPU. At Trendyol this gave about 1.75× more serving capacity.
Model routing. A small System1 model sends easy requests to a 1B model and hard ones to the 8B, cutting cost until misroutes hurt answers.
Model routing. A small System1 model scores every request, and only the hard ones go to the large model; the rest go to a 1B model that costs a fraction as much. Raising the threshold cuts cost — on-prem by letting one H100 serve all the traffic, with the API by shrinking the bill — but push it too far and hard requests land on the small model and get worse answers. Difficulty and answer quality are modelled, not measured.
Real traffic. Real traffic peaks well above its average, so a GPU sized for the average falls behind at the peak, which random traffic hides.
Real traffic. An hour of Azure's production LLM traffic, replayed 12× faster with every 4th request so that one H100 gets its share; each request still runs at real speed. Its busiest five minutes carry about 1.4× the hour's average, and at the peak the GPU that served the average with room to spare falls behind: the queue grows, p99 goes red and requests get 503s. Random traffic at the same average rate stays green. Size for the peak, not the average. Traffic: Azure LLM inference trace 2023, conversations (Microsoft, CC-BY 4.0; Patel et al., Splitwise, ISCA 2024), every 4th request, 12× faster.
Failure. Kill one of two replicas and its requests retry on the survivor, which falls behind until a replacement comes up.
Failure. Two replicas of Llama 3 8B, each on its own H100, share the traffic. Kill one and its requests come back as client retries: their KV cache died with the GPU, so each redoes its prefill and starts its reply over, and p99 spikes. The survivor can't carry both GPUs' traffic alone, so its queue grows and requests get 503s. Turn the autoscaler on and a replacement H100 comes up after a cold start, and the queues drain. Cold start: 30 s, assumed (a warm spare node loading the weights); a fresh node takes minutes.
Sandbox. Build any deployment with the knobs or the terminal, watch its config follow, then take the SLO challenge.
Sandbox. Much of what the other tabs fix, you choose here: up to three H100s, each whole, split by MIG or time-sliced, a model per slice (the same on every GPU), batching, GPU memory, traffic and the router, and you can kill a GPU; tab 04's routing gateway stays in tab 04. The terminal under the footer does the same by typing (try help), the config pane shows what you'd deploy, and the footer's meter what this deployment has cost so far. The SLO challenge replays the peak of tab 05's Azure hour against your deployment: keep TTFT p99 within 2 s with no 503s, as cheaply as you can. Par: two whole H100s pass at $0.77 per 1M output tokens; one H100 falls behind at the peak.
a live LLM serving simulator: batching · KV cache · MIG · routing · real traffic · failure · sandbox
knobs
terminal · type help
config · what you'd deploy
A model of LLM serving, not a benchmark: step times come from the H100's specs and each model's real size. Pick a tab, send a burst, then pause and watch it drain.
I work on ML inference systems: deploying and optimizing the platforms that serve recommendation engines, search ranking, and increasingly AI agents in production. Most recently that meant a platform serving 40M+ across 35 different countries users at over 1M requests per minute.
My work sits at the intersection of ML, systems, and infrastructure — vLLM and Triton for serving, and Kubernetes across cloud and on-prem. I care about the parts of ML that don't make it into most conference talks: p99 latency, GPU utilization, what breaks at 3am.
I'm a Google Developer Expert in Cloud AI. I co-organize GDG Istanbul and speak regularly at DevFest, Build with AI, and GDG events across Turkey and abroad, in both Turkish and English.
These are the real attention weights of GPT-2 small reading the three paragraphs of this section, pooled from tokens to words. GPT-2 only looks backwards, so arcs always point to earlier words.
Each square is one head; rows are layers 0–11, and brighter squares look further back. Click one to switch.
computed by scripts/ml_figures.py · <|endoftext|> sink removed, rows renormalized

Mar 2024 – Sep 2026
Turkcell
2022 – 2024
A team of research, judge and builder agents talking over A2A, with a self-hosted Gemma model on Cloud Run GPUs.
A recorded run of my issue-to-PR agents, replayed with its real timestamps, tokens and cost. All recorded runs →
Short lessons and coding problems in Python, with interactive demos of gradient descent, BPE tokenization, attention and the KV cache. ML foundations are open now; LLM internals and writing an LLM from scratch are next.
Planner, coder and reviewer agents in an ADK graph, routed by real diffs and test results instead of model claims. Commands run in sandboxes with no credentials or network, every run is cost-capped, and a human approves before a pull request opens. Deployed on Agent Runtime with keyless CI/CD. Watch recorded runs · architecture
Four 4-bit models under AIPerf at 1 to 16 concurrent users. Bursty traffic made time-to-first-token about 5x worse than steady traffic at the same average rate, and the p99 knee came early. Scripts and raw per-request data included.
Serve a ResNet-18 with dynamic batching and two model instances, load test it with perf_analyzer, and deploy it to GKE. Runs on a laptop CPU. Companion code for my intro article.
Collects new AI papers and posts every day, ranks them by what you like (with or without an LLM), and lets you read full papers without leaving the terminal. Includes hybrid search and Q&A over everything you've read.
A Claude Code plugin (goygoy): hand Claude a task and it answers "Başımla beraber abi! 🫡" — out loud, or in its own reply.
I'm open to collaborations: talks and workshops, open-source work, and teams figuring out how to serve models or agents in production. Pick a slot, or just write to compare notes on inference at scale.