Dynamo AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

Dynamo AI AI

A Hackable ML Compiler Stack in 5,000 Lines of Python [P]

Hey r/MachineLearning, The modern ML (LLM) compiler stack is brutal. TVM is 500K+ lines of C++. PyTorch piles Dynamo, Inductor, and Triton on top of each other. Then there's XLA, MLIR, Halide, Mojo. There is no tutorial that covers the high-level design of an ML compiler without dropping you straight into the guts of one of these frameworks. I built a reference compiler from scratch in ~5K lines of pure Python that emits raw CUDA. It takes a small model (TinyLlama, Qwen2.5-7B) and lowers it to a sequence of CUDA kernels through six IRs. The goal isn't to beat Triton; it is to build a hackable, easy-to-follow compiler. Full article: A Principled ML Compiler Stack in 5,000 Lines of Python Repo: deplodock The pipeline consists of six IRs, each closer to the hardware than the last. Walking the following PyTorch code through every stage (real reference compiler output with names shortened for brevity and comments added): torch.relu(torch.matmul(x + bias, w)) # x: (16, 64), bias: (64,), w: (64, 16) Torch IR. Captured FX graph, 1:1 mirror of PyTorch ops: bias_bc = bias[j] -> (16, 64) float32 add = add(x, bias_bc) -> (16, 64) float32 matmul = matmul(add, w, has_bias=False) -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 Tensor IR. Every op is decomposed into Elementwise / Reduction / IndexMap. Minimal unified op surface, so future frontends (ONNX, JAX) plug in without touching downstream passes: bias_bc = bias[j] -> (16, 64) float32 w_bc = w[j, k] -> (16, 64, 16) float32 add = add(x, bias_bc) -> (16, 64) float32 add_bc = add[i, j] -> (16, 64, 16) float32 prod = multiply(add_bc, w_bc) -> (16, 64, 16) float32 red = sum(prod, axis=-2) -> (16, 1, 16) float32 matmul = red[i, na, j] -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 The (16, 64, 16) intermediate looks ruinous, but it's never materialized; the next stage fuses it out. Loop IR. Each kernel has a loop nest fused with adjacent kernels. Prologue, broadcasted multiply, reduction, output layout, and epilogue all collapse into a single loop nest with no intermediate buffers. === merged_relu -> relu === for a0 in 0..16: # free (M) for a1 in 0..16: # free (N) for a2 in 0..64: # reduce (K) in0 = load bias[a2] in1 = load x[a0, a2] in2 = load w[a2, a1] v0 = add(in1, in0) # prologue (inside reduce) v1 = multiply(v0, in2) acc0 <- add(acc0, v1) v2 = relu(acc0) # epilogue (outside reduce) merged_relu[a0, a1] = v2 Tile IR. The first GPU-aware IR. Loop axes get scheduled onto threads/blocks, Stage hoists shared inputs into shared memory, and a 2×2 register tile lets each thread accumulate four outputs at once. The K-axis is tiled into two outer iterations of 32-wide reduce. Three-stage annotations below carry the heaviest optimizations: buffers=2@a2 — double-buffer the smem allocation along the a2 K-tile loop, so loads for iteration a2+1 overlap compute for a2. async — emit cp.async.ca.shared.global so the warp doesn't block on global→smem transfers; pairs with commit_group/wait_group fences in Kernel IR. pad=(0, 1, 0) — add 1 element of padding to the middle smem dim so warp-wide loads don't all hit the same bank.kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile # meta: double-buffered, sync (small, no async needed) bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 ​ kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 x_smem = Stage(x, origin=(0, (a2 * 32)), slab=(a0:8@0, a3:32@1, cell:2@0)) pad=(0, 1, 0) buffers=2@a2 async w_smem = Stage(w, origin=((a2 * 32), 0), slab=(a3:32@0, a1:8@1, cell:2@1)) buffers=2@a2 async # reduce for a3 in 0..32: in0 = load bias_smem[a2, a3] in1 = load x_smem[a2, a0, a3, 0]; in2 = load x_smem[a2, a0, a3, 1] in3 = load w_smem[a2, a3, a1, 0]; in4 = load w_smem[a2, a3, a1, 1] # prologue, reused 2× across N v0 = add(in1, in0); v1 = add(in2, in0) # 2×2 register tile acc0 <- add(acc0, multiply(v0, in3)) acc1 <- add(acc1, multiply(v0, in4)) acc2 <- add(acc2, multiply(v1, in3)) acc3 <- add(acc3, multiply(v1, in4)) # epilogue relu[a0*2, a1*2 ] = relu(acc0) relu[a0*2, a1*2 + 1] = relu(acc1) relu[a0*2 + 1, a1*2 ] = relu(acc2) relu[a0*2 + 1, a1*2 + 1] = relu(acc3) Kernel IR. Schedule materialized into hardware primitives. THREAD/BLOCK become threadIdx/blockIdx, async Stage becomes Smem + cp.async fill with commit/wait fences, sync Stage becomes a strided fill loop. Framework-agnostic: same IR could lower to Metal or HIP: kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): Init(acc0..acc3, op=add) for a2 in 0..2: # K-tile Smem bias_smem[2, 32] (float) StridedLoop(flat = a0*8 + a1; < 32; += 64): bias_smem[a2, flat] = load bias[a2*32 + flat] Sync # pad row to 33 to kill bank conflicts Smem x_smem[2, 8, 33, 2] (float) StridedLoop(flat = a0*8 + a1; < 512; += 64): cp.async x_smem[a2, flat/64, (flat/2)%32, flat%2] <- x[flat/64*2 + flat%2, a2*3

My actual AWS bill running Claude in production for 5 months

So I've been running Claude Haiku 4.5 on AWS Bedrock for about 5 months now across a few different production apps. Thought I'd share what the bill actually looks like because there's a lot of vague "it's cheap" or "it costs a fortune" talk and not enough actual numbers. My setup: a Next.js app on AWS Amplify that uses Bedrock for two things. First, a customer facing AI chat widget (RAG with a knowledge base, about 16 docs). Second, an AI readiness assessment tool that generates personalized reports. Both use Haiku 4.5 because honestly Sonnet is overkill for what I need. The actual numbers (last 3 months average): Chat widget costs about $3.50/month. Most conversations are short. The RAG retrieval from S3 Vectors costs almost nothing, like $0.03/month for the vector store. The trick is keeping the system prompt tight and using the knowledge base to inject context only when needed instead of stuffing everything into the prompt. Assessment reports cost about $4.80/month. Each report is a 150 word personalized analysis. I cap the output at 400 tokens and set a daily cap at 100 reports. Worst case is maybe $8/month but it never hits that. Total Bedrock cost: roughly $8 to $12/month. I set a $20/month AWS budget alarm with alerts at 50%, 80%, and 100%. Haven't hit the 80% alert once. What actually saved me money: Haiku instead of Sonnet. For my use cases the quality difference is negligible but cost difference is like 10x. I tested both extensively before committing. Sonnet gave slightly more polished prose in the reports but nobody noticed or cared. Daily cost caps in DynamoDB. Not just rate limiting per IP (I do that too, 20 requests per 15 min for chat) but a hard atomic counter in DynamoDB that blocks all AI calls after hitting the daily limit. Survives Lambda cold starts unlike in memory counters. Keeping maxOutputTokens low. Assessment prompt uses 400 max. Chat uses 1024. You'd be surprised how much quality you can get in a tight token budget when your prompt is specific about format and length. Bedrock Guardrails for free safety. Content filtering, prompt attack detection, PII blocking. The guardrail evaluation calls are free, you only pay for the model invocation. So I get a full safety layer at $0 extra. The gotcha nobody warns you about: Lambda cold starts can make your in memory rate limiters useless. I had a bug where my daily cost cap was resetting every time a new Lambda instance spun up, so theoretically someone could have burned through way more than intended. Moving the counter to DynamoDB with atomic UpdateItem fixed it permanently. Cost of that DynamoDB table? Like $0.50/month with on demand pricing. What I'd do differently: I probably overengineered the safety stuff early on. The $20/month budget alarm alone would have caught any runaway costs. But the DynamoDB cap gives me peace of mind for the chat widget since it's public facing and I can't control how many people use it. If you're building something similar and debating Bedrock vs the API directly, Bedrock's advantage is the IAM integration. No API keys floating around in env vars, your Lambda just assumes a role and talks to the model. One less secret to manage. Anyone else running Haiku on Bedrock? Curious what your monthly spend looks like for similar workloads. submitted by /u/ecompanda [link] [comments]

I built a full-stack serverless AI agent platform on AWS in 29 hours using Claude Code — here's the entire journey as a tutorial

TL;DR: Built a complete AWS serverless platform that runs AI agents for ~$0.01/month — entirely through conversational prompts to Claude Code over 5 weeks. Documented every prompt, failure, and fix as a 7-chapter vibe coding tutorial. GitHub repo. What I built Serverless OpenClaw runs the OpenClaw AI agent on-demand on AWS — with a React web chat UI and Telegram bot. The entire infrastructure deploys with a single cdk deploy. The twist: every line of code was written through Claude Code conversations. No manual coding — just prompts, reviews, and course corrections. The numbers Metric Value Development time ~29 hours across 5 weeks Total AWS cost ~$0.25 during development Monthly running cost ~$0.01 (Lambda) Unit tests 233 E2E tests 35 CDK stacks 8 TypeScript packages 6 (monorepo) Cold start 1.35s (Lambda), 0.12s warm The cost journey This was the most fun part. Claude Code helped me eliminate every expensive AWS component one by one: What we eliminated Savings NAT Gateway -$32/month ALB (Application Load Balancer) -$18/month Fargate always-on -$15/month Interface VPC Endpoints -$7/month each Provisioned DynamoDB Variable Result: From a typical ~$70+/month serverless setup down to $0.01/month on Lambda with zero idle costs. Fargate Spot is available as a fallback for long-running tasks. How Claude Code was used This wasn't "generate a function" — it was full architecture sessions: Architecture design: "Design a serverless platform that costs under $1/month" → Claude Code produced the PRD, CDK stacks, network design TDD workflow: Claude Code wrote tests first, then implementation. 233 tests before a single deploy Debugging sessions: Docker build failures, cold start optimization (68s → 1.35s), WebSocket auth issues — all solved conversationally Phase 2 migration: Moved from Fargate to Lambda Container Image mid-project. Claude Code handled the entire migration including S3 session persistence and smart routing The prompts were originally in Korean, and Claude Code handled bilingual development seamlessly. Vibe Coding Tutorial (7 chapters) I reconstructed the entire journey from Claude Code conversation logs into a step-by-step tutorial: # Chapter Time Key Topics 1 The $1/Month Challenge ~2h PRD, architecture design, cost analysis 2 MVP in a Weekend ~8h 10-step Phase 1, CDK stacks, TDD 3 Deployment Reality Check ~4h Docker, secrets, auth, first real deploy 4 The Cold Start Battle ~6h Docker optimization, CPU tuning, pre-warming 5 Lambda Migration ~4h Phase 2, embedded agent, S3 sessions 6 Smart Routing ~3h Lambda/Fargate hybrid, cold start preview 7 Release Automation ~2h Skills, parallel review, GitHub releases Each chapter includes: the actual prompt given → what Claude Code did → what broke → how we fixed it → lessons learned → reproducible commands. Start the tutorial here → Tech stack TypeScript monorepo (6 packages) on AWS: CDK for IaC, API Gateway (WebSocket + REST), Lambda + Fargate Spot for compute, DynamoDB, S3, Cognito auth, CloudFront + React SPA, Telegram Bot API. Multi-LLM support via Anthropic API and Amazon Bedrock. Patterns you can steal API Gateway instead of ALB — Saves $18+/month. WebSocket + REST on API Gateway with Lambda handlers Public subnet Fargate (no NAT) — $0 networking cost. Security via 6-layer defense (SG + Bearer token + TLS + localhost + non-root + SSM) Lambda Container Image for agents — Zero idle cost, 1.35s cold start. S3 session persistence for context continuity Smart routing — Lambda for quick tasks, Fargate for heavy work, automatic fallback between them Cold start message queuing — Messages during container startup stored in DynamoDB, consumed when ready (5-min TTL) The repo is MIT licensed and PRs are welcome. Happy to answer questions about any of the architecture decisions, cost optimization tricks, or how to structure long Claude Code sessions for infrastructure projects. GitHub | Tutorial submitted by /u/Consistent-Milk-6643 [link] [comments]