Semantic Kernel

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Semantic Kernel AI

Fearless Concurrency on the GPU: Safe GPU inference in Rust, competitive with vLLM/SGLang [R]

I maintain cuTile Rust and just posted the paper "Fearless Concurrency on the GPU." As more GPU code gets AI-generated, the bottleneck moves from writing it to trusting it. cuTile Rust lets you write or generate GPU kernels whose memory safety and data-race freedom are verified by the compiler, through Rust's ownership and borrow checking. You get those guarantees by construction. It's a tile-based programming model that lowers to CUDA Tile IR, carrying Rust's ownership model across the launch boundary. You partition a mutable output into disjoint mutable sub-tensors, pass inputs as shared references, and write tile kernels with single-threaded semantics that the compiler maps to thread blocks. End to end, we built Grout, a Qwen3 inference engine, on cuTile Rust with Hugging Face. At batch-1 decode it reaches 171 tok/s for Qwen3-4B on an RTX 5090 and 82 tok/s for Qwen3-32B on a B200, competitive with vLLM and SGLang. Batch-1 decode is memory-bandwidth-bound, and Grout's throughput is consistent with our HBM roofline analysis. Many of Grout's kernels still use the unsafe path today, but they can be migrated to safe variants, providing a verifiable target for generated kernels. We've started a collection of such kernels in the cutile-kernels crate in the repo. If this is your thing, contributing safe variants helps grow a library of safe, high-performance kernels that future kernel synthesis can draw from. On the kernel side, the safety is effectively free. On a B200 the safe GEMM is within 0.3% of a hand-written low-level version (~92% of dense f16 peak), and element-wise hits ~7 TB/s, matching cuTile Python within measurement noise. Some additional caveats worth noting: Grout is batch-1 with a small set of supported models (a research case study, not a drop-in server), it's NVIDIA-only (lowers to Tile IR), and GEMM still slightly trails cuBLAS at some sizes. - Paper: https://arxiv.org/abs/2606.15991 - Code: https://github.com/nvlabs/cutile-rs - Grout: https://github.com/huggingface/grout Hope you enjoy the paper and learn something new! Happy to answer any questions :) submitted by /u/Exciting_Suspect9088 [link] [comments]

Claude Opus co-authored a JVMCI compiler that emits AArch64 machine code HotSpot accepts — 11.7x faster than C2 on a hot method

https://preview.redd.it/1hcf7ykh3g6h1.png?width=1200&format=png&auto=webp&s=3ed1125661e4b955565b81e8592c0275c9aaf3b7 Some context for people unfamiliar with the JVM layer: JVMCI (JEP 243) is a JDK interface that lets you replace HotSpot's C2 JIT compiler for specific methods — instead of C2 generating machine code, you emit it, and HotSpot installs and runs it as a native method. It's how GraalVM plugs in its compiler. Nobody does this by hand for a single Java method. I wanted to try. Why this method, and why I could even see the opportunity: I work on Hexana, a plugin for JetBrains IDEs and VS Code with a JIT viewer that shows the machine code C2 compiled a method into, side-by-side with the bytecode it came from. Staring at a hot bytecode-interpreter method in that view, the waste was impossible to unsee — ~1.5 KB of opcode dispatch, operand-stack bounds checks, and deopt stubs, sitting next to what is semantically sixteen rounds of straight-line long arithmetic. C2 emits that generic shape because it can't know the program is fixed. The gap was right there on screen, so I tried to close it. The task: a small bytecode interpreter running a 16-round mixing kernel, C2 = 385 ns/op baseline. The goal was to write a JVMCI compiler that reads the interpreter's fixed program at compile time and emits specialized, straight-line AArch64 — no dispatch loop, no operand stack, constants folded to immediates. The first Futamura projection, from scratch. I did this with Claude Opus 4.8 (1M context), mostly across one long session. Let me describe exactly what that looked like, because I think the sub will find the failure mode more interesting than the success. What Opus produced The assembler in the repo breaks into three layers: ~550 lines of buffer/relocation infrastructure — vendored from the JDK's own JVMCI test assembler (GPL), not generated ~130 lines of new AArch64 instruction encodings (bit-field arithmetic derived from the ARM spec) — Opus session ~330 lines of partial-evaluator logic (reads code[]/consts[] at compile time, emits straight-line instructions per opcode) — Opus session The encodings are not magic — they are integer arithmetic over ARM-spec fields, the same thing any assembler does. Opus derived them from the spec and got them right on the first JMH run for the arithmetic instructions. For the control-flow and linking instructions, it needed one correction pass. I drove architecture; Opus did the codegen. It runs. On all 4096 test inputs the specialized run equals an independent reference. 33 ns/op, ~11.7x vs C2's 385. The genuinely hard part: the nmethod entry barrier The first install attempt failed immediately: nmethod entry barrier is missing HotSpot (JDK 17+) rejects any JVMCI-installed nmethod that does not open with an exact entry-barrier protocol — and verifies the instruction encoding, not just its presence. The protocol is not in the JVMCI javadoc. It is in HotSpot's C++ verifier code. Here is what the working emitter looks like: public void emitNmethodEntryBarrier() { recordMark(config.MARKID_FRAME_COMPLETE); DataSectionReference guard = new DataSectionReference(); guard.setOffset(data.position()); data.emitInt(0); recordMark(config.MARKID_ENTRY_BARRIER_PATCH); recordDataPatchInCode(guard); emitLoadRegister(rscratch1, DWORD, 0xdead); // ldr w8, =guard (the 0x18.. the verifier checks) emitLoadRegister(rscratch2, DWORD, r28, disarmedOff); // ldr w9, [rthread, #disarmed_offset] emitCmpReg(rscratch1, rscratch2); int toSkip = emitCondBranch(COND_EQ); // b.eq skip emitLoadPointer48(rscratch1, nmethodEntryBarrier); emitBlr(rscratch1); // call the barrier stub patchBranchTo(toSkip, codePos(), COND_EQ); } The specific contract: a section_word relocation on a data-section guard word, a ldr w, =guard literal load (HotSpot's verifier literally checks for the 0x18 prefix encoding), a thread-register disarmed-field compare, and a conditional stub call. Get any of those wrong and the install fails or silently corrupts state. To reverse-engineer that contract, I fanned out three specialist subagent prompts in parallel — one focused on HotSpot C++ (the barrier infrastructure), one on AArch64 encoding (what instruction pattern satisfies the 0x18.. check), one on JVMCI relocation protocol (what MARKID_ENTRY_BARRIER_PATCH actually triggers). Each returned a partial picture; the synthesis was what produced the working emitter. This is the part I would not have gotten through alone in a week; the parallel context-load on three different internals domains is where the 1M context window actually mattered. The candid finding that surprised me While I was getting the JVMCI compiler working, I tried a simpler approach in parallel: a -javaagent that uses ASM bytecode rewriting to inject a specialized fast path into run at class-load time — no machine code, just Java the shape C2 likes, with a guard that falls back to the original interpreter for any other program. That route got 26 ns

RT @BradSmi: It was great to sit down with former UK Prime Minister @RishiSunak to talk about the Mythos Moment, named for @AnthropicAI's l…

RT @BradSmi: It was great to sit down with former UK Prime Minister @RishiSunak to talk about the Mythos Moment, named for @AnthropicAI's l…

New findings in Nature Methods highlight how Project Ex Vivo is helping researchers uncover patterns in cell behavior that may lead to more personalized therapies for patients dealing with cancer. M

New findings in Nature Methods highlight how Project Ex Vivo is helping researchers uncover patterns in cell behavior that may lead to more personalized therapies for patients dealing with cancer. Microsoft researcher Lorin Crawford explains more: https://t.co/gyUk7EO3mT https://t.co/nldvUB9TQM

The race is just the part you see. Behind it, 2,000 people at @MercedesAMGF1 turn thousands of simulations into one car across the line — powered by Microsoft. https://t.co/kEoSbmXy2t https://t.co/wUb

The race is just the part you see. Behind it, 2,000 people at @MercedesAMGF1 turn thousands of simulations into one car across the line — powered by Microsoft. https://t.co/kEoSbmXy2t https://t.co/wUbymQfvA5

RT @XBOX: WHAT A LINEUP | #XBOXShowcase https://t.co/mlIDJ11sX1

RT @XBOX: WHAT A LINEUP | #XBOXShowcase https://t.co/mlIDJ11sX1

No room, no margin, no second chances. @MercedesAMGF1 and Kimi take Monaco, and the work that won it started long before lights out. https://t.co/X5o0cHvYr3

No room, no margin, no second chances. @MercedesAMGF1 and Kimi take Monaco, and the work that won it started long before lights out. https://t.co/X5o0cHvYr3

Students from 40+ countries brought AI-powered ideas to life at @redbull Basement. 🌎 This year’s winner, Lifeline AI by USC graduate Darnell Adler, is a personal safety app that sends silent emergenc

Students from 40+ countries brought AI-powered ideas to life at @redbull Basement. 🌎 This year’s winner, Lifeline AI by USC graduate Darnell Adler, is a personal safety app that sends silent emergency alerts without unlocking a phone or saying a word. Congrats, Darnell — see https://t.co/3UFU05wF0h

RT @BradSmi: Our newest AI Diffusion Report is out, and I sat down with Juan M. Lavista Ferres on Tools and Weapons to dig into what the fi…

RT @BradSmi: Our newest AI Diffusion Report is out, and I sat down with Juan M. Lavista Ferres on Tools and Weapons to dig into what the fi…

A special #MSBuild crossover pod just dropped 👀 @SatyaNadella joined @swyx, @Saranormous, and @EladGil for a wide-ranging convo on AI, platforms, builders, and what comes next. Full episode: https:/

A special #MSBuild crossover pod just dropped 👀 @SatyaNadella joined @swyx, @Saranormous, and @EladGil for a wide-ranging convo on AI, platforms, builders, and what comes next. Full episode: https://t.co/jfMD3od4Yy https://t.co/VHS2Qz4yT2

RT @nvidia: The agentic AI era is here. From Taipei, Jensen Huang joined @satyanadella at #MSBuild to show how NVIDIA and @Microsoft are b…

RT @nvidia: The agentic AI era is here. From Taipei, Jensen Huang joined @satyanadella at #MSBuild to show how NVIDIA and @Microsoft are b…

RT @MayoClinic: Mayo Clinic and @Microsoft are collaborating to develop a frontier AI model designed specifically for healthcare. The model…

RT @MayoClinic: Mayo Clinic and @Microsoft are collaborating to develop a frontier AI model designed specifically for healthcare. The model…

RT @windowsdev: Powered by WinGet, Windows Developer Configurations enables you to quickly set up a distraction-free dev environment with V…

RT @windowsdev: Powered by WinGet, Windows Developer Configurations enables you to quickly set up a distraction-free dev environment with V…

RT @mustafasuleyman: So proud of the team today. Six months of super intense and outstanding work. I was honored to stand up and rep the wo…

RT @mustafasuleyman: So proud of the team today. Six months of super intense and outstanding work. I was honored to stand up and rep the wo…

RT @msdev: From yesterday’s #MicrosoftBuild keynote: a developer-optimized Windows 11 experience built to help you build and ship faster. h…

RT @msdev: From yesterday’s #MicrosoftBuild keynote: a developer-optimized Windows 11 experience built to help you build and ship faster. h…