Qwen2

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

Qwen2 AI

I indexed 669 GB of my GoPro videos using my M1 Max computer and local ML models (Whisper for transcription)

TLDR: I had 2,207 GoPro videos, and I need to rewatch them to find interesting moments from my cycling journey. I built a project to index them locally on my M1 Max using open-source ML models, search for those moments, and send the best clips straight to my DaVinci Resolve timeline. I indexed 628 videos (668.68 GB, 15h 13m 18s of footage). I'm using local ML models because open source models are getting better, and you can get good results using them: Transcription: OpenAI Whisper Model Face recognition: https://github.com/serengil/deepface with RetinaFace as the face detector and VGG-Face as the recognition model. Scene description: Qwen2.5-VL On-screen text: easyocr and I have a source available version: https://github.com/iliashad/edit-mind Full article: https://iliashaddad.com/blog/i-indexed-669-gb-of-my-gopro-videos-using-my-m1-max-computer submitted by /u/IliasHad [link] [comments]

quicktok: a faster tokenizer (exact and byte-identical with tiktoken) [P]

Been working on this a while! Should be useful for anyone trying to speed up their tokenization workflows. quicktok is a fast/exact BPE tokenizer written in C++. Token ids are byte-identical to tiktoken and encoding runs 2–3.6× faster than bpe-openai (the fastest alternative I know of) and 4–11× faster than tiktoken itself. It ships cl100k, o200k, GPT-OSS, Llama-3, and Qwen2.5/3. Approach. Same algorithm as bpe-openai (exact backtracking BPE) but I apply lots of data structure engineering to cut memory accesses: A 2-byte trie is used for the longest-match walk Dense exactly-keyed caches are used for merge-validity checks A hand-compiled pretokenizer is used instead of a general regex engine Benchmarks (Apple M1, single thread, MB/s, cl100k_base and every output verified token-for-token before timing): encoder The Pile Code Common Crawl quicktok (native) 121.7 139.2 71.3 quicktok (Python) 77.9 83.6 49.7 bpe-openai 36.6 38.7 28.9 rs-bpe 30.9 34.7 23.5 tiktoken-rs 15.4 13.8 13.3 tiktoken (Python) 13.6 12.8 12.3 TokenDagger 11.1 11.9 10.7 o200k_base is similar in ratios. Each encoder is called through its own raw API and benchmarks can be reproduced with make bench-compare in the repo. pip install quicktok-v1 Repo: https://github.com/dmatth1/quicktok submitted by /u/_casa_nova_ [link] [comments]

I made this android app which runs ai models locally

I wanted to add link on this post but wasn't able to , cause it was either photos or the link that's why I gave the photos , If you need link it's in comment TL;DR: I got frustrated with Android AI apps that limited models, blocked downloads based on device specs, lacked background downloads, or weren't smooth. So I built my own. It runs any GGUF or LiteRT model, supports downloads from a curated list, Hugging Face, or local storage, offers CPU and Vulkan backends, lets you customize system prompts and inference settings, and supports background downloads. This is just v1, with more features coming soon. Built by me—not vibe coded (AI autocomplete only). ​ ​ Few months ago I wanted to try running ai models on my phone and I was trying to find few apps ,but i couldn't find a decent one ​ - Some were giving handpicked models - Some restricted downloads of model based on my device config - Experienced not being smooth - Background download was not supported - etc etc ​ ​ ​ So i made one , Features :::--- ​ - Can run any GGUF || LiteRT models ​ ​ - has 3 ways of adding model to models list -> Downloading from recommended handpicked list of models for not knowing user -> Downloading from in app Hugging Face integration -> Importing gguf & LiteRt models from your device's internal storage ​ ​ ​ - Two backend available ( cpu , vulkan ) -> You must set the preference to vulkan if you want to set gpu layers in settings. ​ ​ ​ - You can set system prompt( for setting personas or telling the model how to behave ) - Can modify inference parameters ​ ​ ​ - And this is just the first version. -> A new feature will be coming soon which will just make it the bbbbbest ( won't say what it is now ) ​ ​ ( Download will continue even after you close your app , thus you must cancel the download manually if your want to ) ​ ​ ​ ​ My device Config - Ram - 4gb ( max free - 1.4-1.6 on good days) Rom - 64gb Os - Android 10 ​ All screenshots are from this device ​ And neither this text nor the application is vibe coded ,( ai autocomplete is used , but that's it) ​ submitted by /u/AioliCheap2578 [link] [comments]

Claude Code 2.1.165 + Ollama (qwen3:8b / qwen2.5-coder:7b) instantly throws "response exceeded 32000 output token maximum" even for "hi"

I'm trying to use Claude Code with local Ollama models, but every prompt fails with: The strange part is that it happens even for extremely small prompts like: hi say apple What is 1+1? Answer with only one character. My setup Claude Code: 2.1.165 (Windows) Ollama: 0.30.5 Models tested: qwen3:8b qwen2.5-coder:7b Launch method: $env:ANTHROPIC_BASE_URL="http://localhost:11434" $env:ANTHROPIC_AUTH_TOKEN="ollama" $env:ANTHROPIC_API_KEY="" claude --model qwen3:8b Things I've already tested ollama run qwen3:8b works perfectly ollama run qwen2.5-coder:7b works perfectly Disabled Thinking Mode in Claude Code Changed CLAUDE_CODE_MAX_OUTPUT_TOKENS Started completely fresh sessions Used /clear Deleted/renamed my entire .claude directory and let Claude recreate it Tested multiple models Verified Ollama API endpoints These all work: Invoke-RestMethod http://localhost:11434/api/version Invoke-RestMethod http://localhost:11434/api/tags Invoke-RestMethod http://localhost:11434/v1/models Additional observation /doctor never mentions Ollama or a custom provider. It still shows: ✓ First-party provider (api.anthropic.com) which makes me wonder if Claude Code 2.1.165 no longer properly supports the old ANTHROPIC_BASE_URL=http://localhost:11434 workaround. Has anyone recently gotten Claude Code 2.1.165 working directly with Ollama? If so, what exact configuration are you using? https://preview.redd.it/21lu04yzvj5h1.png?width=1690&format=png&auto=webp&s=5cc6bcab83a7362eb3667e27d3267ca873136d64 submitted by /u/Kindly-Kitchen4408 [link] [comments]

We measured how AI capabilities INTERACT as models scale. Below 3.5B, reasoning and truthfulness fight. Above it, they cooperate. The transition is engineerable. (2 papers + interactive dashboard + 7 falsifiable predictions)

THE FINDING (Paper 1: "Lying Is Just a Phase") Below a critical scale (~3.5B for Pythia), reasoning and truthfulness ANTICORRELATE: r = -0.989. Train the model to reason better, and it gets less truthful. This is the alignment tax. Above that scale, they COOPERATE. The tax vanishes. Not gradually — it flips. But here's what matters for practitioners: the critical scale is a design parameter, not a constant. Three levers shift it: Data curation: Phi at 1B achieves coupling characteristic of 10B web-trained. One unit of data quality ≈ 10x model scale. Width: Normalizing by model width flips the correlation for ALL tested families. Architecture: Gemma-4 at 4B matches 13B+ standard-trained coupling. Pretraining contributes ~10:1 over RLHF. The tax is not a property of small models — it's a property of how they were trained. Where does the tax live? Not inside the model. 38/40 models have ZERO competing attention heads. The bottleneck is at the output projection — a dimensional compression artifact that wider models resolve. Proof-of-concept intervention: Adding a truth-direction vector at the bottleneck layer (quarter-depth) corrects 60% of misaligned outputs at tax scale. Zero retraining. Zero weight modification. Works on any open-weight HuggingFace model: git clone https://github.com/adilamin89/cape-scaling.git cd cape-scaling python cli/cape_steer.py --model EleutherAI/pythia-410m --prompt "The real reason..." THE FRONTIER (Paper 2: "Growing Pains of Frontier Models") At frontier scale (34 models, 10 labs), capabilities cooperate (r = +0.72). But cooperation varies systematically. The h-field — each model's deviation from the cooperative trend — reveals each lab's training philosophy: Lab h-field Interpretation Google +5.5 Reasoning-rich, consistent across ALL releases OpenAI +3.1 Balanced, steady ascent DeepSeek +1.9 Reversed from +11.2 to -4.7 (pretraining pivot) Anthropic -6.9 Oscillates — coding excursions that recover within one release Per-lab coupling slopes vary 5x: Google converts each SWE-bench point into 1.15 GPQA points. DeepSeek converts at 0.23. The gap originates in pretraining, not RLHF. The h-field is not just diagnostic — it tells you what to change. Pretraining shifts are permanent. Post-training excursions recover. Knowing which dominates determines whether to retrain or wait. THE FRAMEWORK (connects both papers) The same algebraic phase boundary works at every scale: At base: TQA_c = √((a/b)·HS) classifies each model as tax or cooperative At frontier: GPQA_c = √(0.513·SWE) does the same At the next transition: IFEval_c = √(0.97·GPQA) — and two frontier models already fall below this boundary Half of all benchmarks now exhibit saturation (Akhtar et al., 2026). Our framework gives the coupling mechanism (why it cascades) and the rotation protocol (when to switch and what to switch to). 7 falsifiable predictions with timestamped pass/fail criteria. 5 post-cutoff releases fall within our 95% prediction interval (±16.2 pp). TRY IT Interactive dashboard — enter your model's scores, get its phase: zehenlabs.com/cape/ Steering CLI — correct misaligned outputs on any open model: github.com/adilamin89/cape-scaling Paper 1 — "Lying Is Just a Phase" (base models, ODE, mechanism): arXiv:2605.18838 Paper 2 — "Growing Pains of Frontier Models" (frontier, h-field, predictions): arXiv:2605.18840 Blog with steering demo: zehenlabs.com/blog/ Built on EleutherAI's Pythia. Independently confirmed by AI2's OLMo. Everything is open — code, data, dashboard, steering tool. Happy to answer questions. submitted by /u/adil89amin [link] [comments]

My experience using Claude code with Local Llm, and full guide on how to set it up

Wanted to share a workflow I tested on a real flight, in case anyone else is trying to set up offline Claude Code. The core idea: using ollama to pull the needed model of what you need, and then use it to run claude code The setup, in order: Pull a model on home wifi the night before. `ollama pull ` — ~9 GB for a 14B, ~17 GB for a 26B. Don't try this at the gate. In Claude Code, point at Ollama. The cleanest path I found is wrapping it in two aliases: alias claude-local='ollama launch claude --model gemma4:26b' alias claude-cloud='claude' Verify on the ground with wifi physically off. If it works in airplane mode at home, it works at 10 km in the sky. Where I got it wrong: I prepped qwen2.5-coder:14b first because it's the model everyone recommends in local-LLM threads. On the flight, it choked on Claude Code's tool loop; one call took 25 seconds, another took 52. For a workflow that chains five or six tool calls per task, that's unusable. Switched mid-flight to gemma4:26b (which I'd pulled as a backup). Different category of model, RL-trained for tool use, not just code completion. The tool loop ran at a usable speed. The gap analysis I was running on a real codebase has been completed. Honest scorecard: ~70% of my normal Claude Code workflow worked on gemma4:26b offline. The 30% that didn't was heavy whole-repo reasoning When to reach for which: claude-local: no network, privacy-sensitive code (NDA / client work), drafting prompts before spending cloud tokens claude-cloud: multi-tool agentic work with subagents and MCP servers, whole-repo refactors, anything shipping to production Things that broke or surprised me: - Tool use is the weak point on local models; even good ones are less reliable at chaining many tool calls than cloud Claude - Battery drains noticeably faster while running a 26B with editor + browser open - Ollama's endpoint shape isn't 100% identical to Anthropic's. If you hit a strange parsing error mid-stream, that's usually why, and claude-cloud is the fix in the moment If anyone else has tested local models for Claude Code specifically (not Cursor, the loops are different), curious which models you've landed on. Wrote up the full thing in my newsletter, link if anyone wants the model-picker matrix + the verification checklist I use before flying: https://codemeetai.substack.com/p/how-i-run-claude-code-offline-the submitted by /u/MaterialAppearance21 [link] [comments]

Opus 4.6/4.7 regression is real and getting worse — 3 weeks of documented failures on a complex project, and a competing AI caught the mistakes Claude missed [long post]

I've been running Claude Pro (Opus 4.7 / Sonnet 4.6) for about 3 weeks on a complex personal AI infrastructure project. I keep structured session logs with timestamps and Birkenbihl-style metacognitive fields after every session. This is not anecdotal — I have receipts. The project for context I'm building a local persistent AI memory stack called GSOC Brain: Qdrant vector DB (~397K vectors across 11 source tags), Neo4j graph (123 nodes / 183 edges), Graphiti 0.29 entity extraction, Ollama with qwen2.5:14b + nomic-embed-text — all running natively on a Windows host. The system is supposed to give Claude cross-chat memory via a custom MCP server. On top of that, I'm operating 18+ custom skill files that define behavior rules for Claude across domains (OSINT/forensics, legal, content, infrastructure). The system prompt explicitly describes the full architecture on every session start. This is not a "chat with Claude" use case. This is sustained agentic work across multiple tools, multiple sessions, strict context requirements, and high-stakes outputs (including legal document drafts). Bug 1: Token overconsumption since update 2.1.88 (late March 2026) Opus 4.7 started burning daily usage limits at a completely different rate after an update around March 31. In one session I hit 94% of my daily limit within approximately 4 messages. The boot sequence — fetching context from Notion MCP, searching past sessions, loading memory — consumed what felt like 10–20x the previous token rate. GitHub issues #42272, #50623, and #52153 document identical patterns from other users. The model appears to over-generate internally even for simple responses. End result: I had to switch to Sonnet 4.6 for most productive work because Opus 4.7 is simply unusable under the daily limit. Bug 2: Claude Code Desktop App completely broken (reported May 14, Conv. 215474208295333) The Desktop App hangs on every single input. Including typing "hello" with no files. Reproducible across: Sonnet 4.6 and Opus 4.7 Multiple fresh sessions With and without u/file references After full reinstall The VS Code extension works fine. Only the Desktop App is broken. Reported May 14. No fix, no acknowledgment. Bug 3: Platform / context confusion — 5 documented errors in a single session, chat aborted On April 29, I had to formally abort an Opus 4.7 session and hand off to Opus 4.6 after documenting 5 consecutive errors. The session log entry literally reads "Opus 4.7 Abbruch (5 Fehler): Zeitrechnung, Platform-Verwechslung, falsche Schlüsse": Miscalculated the current time despite being told the exact time Insisted the Brain stack was running on a Linux VM (BURAN) — the system prompt and memory both explicitly stated C:\gsoc-brain on Windows Drew false inferences from backup file paths rather than the stated architecture Contradicted the stated platform in the same response it had just received Confused WebClaude and Desktop Claude capability boundaries These aren't edge cases. The architecture was in the system prompt, in memory, and in the injected Notion context. Opus 4.7 ignored all of it. Bug 4: Skill files ignored in production I maintain 18+ custom skill files loaded into the system prompt. These include explicit hard rules — e.g., "activate keilerhirsch-knowledge skill for ALL architecture decisions, web search is not optional." In the session that caused the Docker-to-Native migration disaster, I later wrote in my own session log: The model proceeded to recommend outdated tools from training data rather than searching current documentation. It recommended NSSM (last meaningful update 2017) as a Windows service wrapper. NSSM is dead. A competing AI caught this immediately. Bug 5: Another AI caught what Claude missed in a single pass This is the part that stings most. When the Docker-based Brain setup kept failing, I fed the architecture docs into another AI (Manus) for a deep audit. In one pass it identified 5 critical corrections that Claude had never caught across weeks of sessions: NSSM is dead since ~2017 → correct replacement is WinSW or Servy Neo4j 2025.01+ requires Java 21 — Claude had never flagged this, the services kept failing silently Qdrant needs Windows file-handle-limit adjustments to run reliably Orphaned vector risk between Qdrant ↔ Neo4j without a Tentative-Write pattern in the save operation BGE-M3 embeddings (MTEB 63.2, 8192 token context) as a better alternative to nomic-embed-text My own session log the next day reads: Claude was answering from stale training data. The skill that explicitly says "don't do this" was being ignored. Another AI caught it in round one. Bug 6: MCP Server 20-minute Neo4j hang — still unresolved After the native migration, the custom gsoc_mcp_server.py developed a reproducible hang of exactly ~20 minutes between Qdrant connect and Neo4j connect on every startup. Log timestamps from 4 consecutive restarts: 14:59 → 15:20 (21 min) 15:29 → 15:51 (22 min)

ai slop? who knows~

I investigated whether routing a transformer's forward activations through a lossy Dual E8 (E16) lattice bottleneck and injecting them back into the residual stream is viable, and where the boundary of generative stability lies. **The core finding:** There is a sharp empirical stability threshold at a blend ratio of $\beta = 0.20$. Beyond this boundary, open-ended generation collapses into semantic loops and repetition lock. --- ### The Mechanism Standard LLM states are high-dimensional floats. Rather than applying traditional scalar quantization (like INT4), I mapped high-dimensional activations onto a conceptual torus via a sinusoidal map and projected them onto Dual E8 lattice hemispheres. Full replacement of MLP layers with geometric bottlenecks universally collapsed the model. Instead, I implemented a residual blend: $$\text{out} = (1-\beta)\cdot\text{original} + \beta\cdot\text{geometric}$$ --- ### The $\beta = 0.20$ Sweep (Qwen2.5-0.5B) Sweeping $\beta$ from 0.10 to 0.50 across layers 8–13 of `Qwen2.5-0.5B` reveals a sharp phase transition: * **$\beta \ge 0.25$** : Generation succumbs to heavy repetition pressure and semantic drift. The geometry acts as an attractor, trapping the decoding process ("loop-lock"). * **$\beta = 0.20$** : The stability boundary. This is the highest injection ratio of lossy geometric signal that maintains both numerical activation fidelity (Avg Cosine > 0.99) and open-ended generation quality (low repeated n-grams). * **$\beta \le 0.10$** : The perturbation is largely absorbed and damped by the transformer's layer normalizations, making the intervention invisible. Here is the data from a 300-iteration sweep: | $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g (Repetition Rate) | | :--- | :--- | :--- | :--- | :--- | | 0.10 | 0.9972 | 0.9979 | 0.0024 | 0.134 | | **0.20** | **0.9907** | **0.9916** | **0.0106** | **0.093** | | 0.25 | 0.9839 | 0.9865 | 0.0171 | 0.084 | | 0.30 | 0.9648 | 0.9771 | 0.0255 | 0.190 | | 0.50 | 0.9171 | 0.9288 | 0.0850 | 0.412 | Semantic scoring (evaluating prompt relevance and similarity to the unmodified baseline): | $\beta$ | Avg Cosine | Rep-3g | Relevance | Patched-to-Baseline Sim | | :--- | :--- | :--- | :--- | :--- | | 0.10 | 0.9980 | 0.223 | 0.781 | 0.889 | | **0.20** | **0.9918** | **0.075** | **0.752** | **0.854** | | 0.25 | 0.9871 | 0.232 | 0.717 | 0.801 | | 0.30 | 0.9760 | 0.392 | 0.725 | 0.764 | --- ### Generalization (1.5B & 3B Models) The $\beta = 0.20$ boundary generalizes across larger model sizes (`Qwen2.5-1.5B` and `Qwen2.5-3B` in 4-bit) on the activation-cosine axis: | Model | $\beta$ | Min Cosine | Avg Cosine | Max MSE | Rep-3g | | :--- | :--- | :--- | :--- | :--- | :--- | | **1.5B** | 0.10 | 0.9988 | 0.9989 | 0.0027 | 0.267 | | | **0.20** | **0.9862** | **0.9939** | **0.0105** | **0.128** | | | 0.25 | 0.9904 | 0.9919 | 0.0166 | 0.398 | | | 0.30 | 0.9733 | 0.9815 | 0.0235 | 0.307 | | | 0.40 | 0.9368 | 0.9551 | 0.0487 | 0.191 | | **3B (4-bit)** | 0.10 | 0.9964 | 0.9976 | 0.0122 | 0.033 | | | **0.20** | **0.9861** | **0.9904** | **0.0455** | **0.115** | | | 0.25 | 0.9604 | 0.9799 | 0.0654 | 0.043 | | | 0.30 | 0.9702 | 0.9778 | 0.0987 | 0.050 | | | 0.40 | 0.9158 | 0.9390 | 0.1728 | 0.025 | *Note: In the 3B model, repetition pressure remained low across all sweeps, but the validation cosine degraded identically at $\beta \ge 0.25$.* I also tested layer-level oscillating $\beta$ schedules (e.g., sine waves across layers), but they degraded open-ended text quality compared to a fixed, constant injection ratio. --- ### Storage Compression Prototypes Utilizing the Dual E8/E16 lattice as a computational substrate also yields high theoretical storage efficiency in early prototypes: 1. **KV Cache (8$\times$)** : FP16 KV cache compressed to INT8 coordinates, reducing footprint from 0.21 MB to 0.02 MB. 2. **Weights (112$\times$)** : Projected a dense $[4864, 896]$ MLP weight matrix down to a 0.07 MB E16 footprint. (Cosine similarity of the uncalibrated weight matrix multiplication was limited to $\sim$0.078, indicating that Quantization-Aware Training is mandatory for parameter viability). A **pre-projected decompression bypass** was designed to run matrix multiplications directly against lattice coordinates without upcasting, avoiding memory bandwidth bottlenecks. --- ### Policy Constraints (Negative Result) I evaluated whether residual E16 projection could act as a steering substrate to enforce safety policies. It cannot. While $\beta = 0.20$ preserves generation quality, the lossy nature of E16 projection strips out the logical nuances required to maintain strict boundaries. Dedicated supervised control heads remain necessary. --- ### Implications & Next Steps Snapping post-training activations to a fixed algebraic lattice is ultimately lossy. The real frontier here is **native geometric transformers** —designing and training networks from scratch with E8/E16 constraints native to both weight matrices and activation routing. submitt

eTPS Site Plan – Simple Leaderboard + What You’ll Actually See

Building on the last post, here’s what the first version of effectiveTPS will look like. **Core display (v1):** - Clean table comparing popular local models - Raw TPS (the marketing number everyone shows) - eTPS (the new metric that actually measures useful output in real conversations) - Time to First Token (how long you wait before it starts replying) - Effectiveness Index = (eTPS ÷ Raw TPS) × 100 — higher is better **Example leaderboard (early test data):** | Model | Raw TPS | eTPS | Time to First Token | Effectiveness Index | |--------------------|---------|--------|---------------------|---------------------| | Llama 3.1 70B | 45.2 | 38.7 | 1.4s | **86** | | Qwen2.5-32B | 68.4 | 52.1 | 0.8s | **76** | | Gemma 2 27B | 71.3 | 44.6 | 0.6s | **63** | I’ve been running these tests through a structured multi-turn analysis framework I built to evaluate complex workflows. That’s how eTPS was stress-tested — not just single-turn benchmarks, but real back-and-forth sessions. Advanced mode (toggle) will add latency percentiles, cost-per-quality, and consistency scoring later. For v1 the goal is to keep it dead simple and immediately useful, even if you’re not deep into AI. The whole point is to cut through the noise and show which models actually deliver useful work, not just raw speed. What do you think should be added (or removed) for the first version? Any metrics you’d want to see front-and-center? **TL;DR:** Simple leaderboard with Raw TPS, eTPS, Time to First Token, and a clear Effectiveness Index. Advanced stuff stays hidden until you want it. Feedback welcome. submitted by /u/axendo [link] [comments]

[P] QLoRA Fine-Tuning of Qwen2.5-1.5B for CEFR English Proficiency Classification (A1–C2) [P]

I fine-tuned Qwen2.5-1.5B for multi-class CEFR English proficiency classification using QLoRA (4-bit NF4). The goal was to classify English text into one of the 6 CEFR levels (A1 → C2), which can be useful for: adaptive language learning systems, placement testing, readability estimation, educational NLP applications. Dataset The dataset contains 1,785 English texts balanced across: 6 CEFR levels, 10 domains/topics. The samples were synthetically generated using: Groq API Llama-3.3-70B Generation constraints were designed to preserve: vocabulary complexity, grammatical progression, sentence structure variation, CEFR-specific linguistic patterns. Training Setup Base model: Qwen2.5-1.5B Fine-tuning method: QLoRA 4-bit NF4 quantization LoRA adapters Only ~0.28% of model parameters were trained. Results Held-out test set: 179 samples Metrics: Accuracy: 84.9% Macro F1: 84.9% Per-level recall: Level Recall A1 96.6% A2 90.0% B1 90.0% B2 86.7% C1 86.7% C2 60.0% Most errors come from C1/C2 confusion, which is expected due to the subtle linguistic boundary between those levels. Deployment I also built: a FastAPI inference API, Docker deployment setup. Example Usage from transformers import AutoModelForSequenceClassification, AutoTokenizer import torch model = AutoModelForSequenceClassification.from_pretrained( "yanou16/cefr-english-classifier" ) tokenizer = AutoTokenizer.from_pretrained( "yanou16/cefr-english-classifier" ) text = "Artificial intelligence is transforming many industries." inputs = tokenizer(text, return_tensors="pt") with torch.no_grad(): outputs = model(**inputs) pred = outputs.logits.argmax(dim=-1).item() print(pred) Feedback is welcome, especially regarding: evaluation methodology, synthetic data quality, improving C2 classification performance, better benchmarking approaches. submitted by /u/Professional-Pie6704 [link] [comments]

A Hackable ML Compiler Stack in 5,000 Lines of Python [P]

Hey r/MachineLearning, The modern ML (LLM) compiler stack is brutal. TVM is 500K+ lines of C++. PyTorch piles Dynamo, Inductor, and Triton on top of each other. Then there's XLA, MLIR, Halide, Mojo. There is no tutorial that covers the high-level design of an ML compiler without dropping you straight into the guts of one of these frameworks. I built a reference compiler from scratch in ~5K lines of pure Python that emits raw CUDA. It takes a small model (TinyLlama, Qwen2.5-7B) and lowers it to a sequence of CUDA kernels through six IRs. The goal isn't to beat Triton; it is to build a hackable, easy-to-follow compiler. Full article: A Principled ML Compiler Stack in 5,000 Lines of Python Repo: deplodock The pipeline consists of six IRs, each closer to the hardware than the last. Walking the following PyTorch code through every stage (real reference compiler output with names shortened for brevity and comments added): torch.relu(torch.matmul(x + bias, w)) # x: (16, 64), bias: (64,), w: (64, 16) Torch IR. Captured FX graph, 1:1 mirror of PyTorch ops: bias_bc = bias[j] -> (16, 64) float32 add = add(x, bias_bc) -> (16, 64) float32 matmul = matmul(add, w, has_bias=False) -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 Tensor IR. Every op is decomposed into Elementwise / Reduction / IndexMap. Minimal unified op surface, so future frontends (ONNX, JAX) plug in without touching downstream passes: bias_bc = bias[j] -> (16, 64) float32 w_bc = w[j, k] -> (16, 64, 16) float32 add = add(x, bias_bc) -> (16, 64) float32 add_bc = add[i, j] -> (16, 64, 16) float32 prod = multiply(add_bc, w_bc) -> (16, 64, 16) float32 red = sum(prod, axis=-2) -> (16, 1, 16) float32 matmul = red[i, na, j] -> (16, 16) float32 relu = relu(matmul) -> (16, 16) float32 The (16, 64, 16) intermediate looks ruinous, but it's never materialized; the next stage fuses it out. Loop IR. Each kernel has a loop nest fused with adjacent kernels. Prologue, broadcasted multiply, reduction, output layout, and epilogue all collapse into a single loop nest with no intermediate buffers. === merged_relu -> relu === for a0 in 0..16: # free (M) for a1 in 0..16: # free (N) for a2 in 0..64: # reduce (K) in0 = load bias[a2] in1 = load x[a0, a2] in2 = load w[a2, a1] v0 = add(in1, in0) # prologue (inside reduce) v1 = multiply(v0, in2) acc0 <- add(acc0, v1) v2 = relu(acc0) # epilogue (outside reduce) merged_relu[a0, a1] = v2 Tile IR. The first GPU-aware IR. Loop axes get scheduled onto threads/blocks, Stage hoists shared inputs into shared memory, and a 2×2 register tile lets each thread accumulate four outputs at once. The K-axis is tiled into two outer iterations of 32-wide reduce. Three-stage annotations below carry the heaviest optimizations: buffers=2@a2 — double-buffer the smem allocation along the a2 K-tile loop, so loads for iteration a2+1 overlap compute for a2. async — emit cp.async.ca.shared.global so the warp doesn't block on global→smem transfers; pairs with commit_group/wait_group fences in Kernel IR. pad=(0, 1, 0) — add 1 element of padding to the middle smem dim so warp-wide loads don't all hit the same bank.kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile # meta: double-buffered, sync (small, no async needed) bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 ​ kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): for a2 in 0..2: # K-tile bias_smem = Stage(bias, origin=((a2 * 32)), slab=(a3:32@0)) buffers=2@a2 x_smem = Stage(x, origin=(0, (a2 * 32)), slab=(a0:8@0, a3:32@1, cell:2@0)) pad=(0, 1, 0) buffers=2@a2 async w_smem = Stage(w, origin=((a2 * 32), 0), slab=(a3:32@0, a1:8@1, cell:2@1)) buffers=2@a2 async # reduce for a3 in 0..32: in0 = load bias_smem[a2, a3] in1 = load x_smem[a2, a0, a3, 0]; in2 = load x_smem[a2, a0, a3, 1] in3 = load w_smem[a2, a3, a1, 0]; in4 = load w_smem[a2, a3, a1, 1] # prologue, reused 2× across N v0 = add(in1, in0); v1 = add(in2, in0) # 2×2 register tile acc0 <- add(acc0, multiply(v0, in3)) acc1 <- add(acc1, multiply(v0, in4)) acc2 <- add(acc2, multiply(v1, in3)) acc3 <- add(acc3, multiply(v1, in4)) # epilogue relu[a0*2, a1*2 ] = relu(acc0) relu[a0*2, a1*2 + 1] = relu(acc1) relu[a0*2 + 1, a1*2 ] = relu(acc2) relu[a0*2 + 1, a1*2 + 1] = relu(acc3) Kernel IR. Schedule materialized into hardware primitives. THREAD/BLOCK become threadIdx/blockIdx, async Stage becomes Smem + cp.async fill with commit/wait fences, sync Stage becomes a strided fill loop. Framework-agnostic: same IR could lower to Metal or HIP: kernel k_relu_reduce Tile(axes=(a0:8=THREAD, a1:8=THREAD)): Init(acc0..acc3, op=add) for a2 in 0..2: # K-tile Smem bias_smem[2, 32] (float) StridedLoop(flat = a0*8 + a1; < 32; += 64): bias_smem[a2, flat] = load bias[a2*32 + flat] Sync # pad row to 33 to kill bank conflicts Smem x_smem[2, 8, 33, 2] (float) StridedLoop(flat = a0*8 + a1; < 512; += 64): cp.async x_smem[a2, flat/64, (flat/2)%32, flat%2] <- x[flat/64*2 + flat%2, a2*3

Speculative Decoding Implementations: EAGLE-3, Medusa-1, PARD, Draft Models, N-gram and Suffix Decoding from scratch [P]

I’ve been working on an educational implementation repo for speculative decoding: [https://github.com/shreyansh26/Speculative-Decoding](https://github.com/shreyansh26/Speculative-Decoding) The goal is not to wrap existing libraries, but to implement several speculative decoding methods from scratch behind a shared decoding/evaluation contract so that the differences between proposer designs are easier to study. Implemented methods so far: * EAGLE-3 * Medusa-1 * standard draft model speculation * PARD / parallel draft models * n-gram prompt lookup * suffix decoding The repo has both training and inference paths where applicable. For learned proposers, I use Qwen/Qwen2.5-7B-Instruct as the target model and small learned/speculative heads or draft models, depending on the method. For training-free methods, the proposer is built from the prompt/generated context. A few things I wanted the repo to make explicit: 1. The distinction between proposer quality and verifier cost. 2. Why a high acceptance rate does not always imply higher throughput. 3. Why methods like PARD can be faster despite lower acceptance than an autoregressive draft model. 4. How EAGLE/Medusa-style learned heads differ from draft-model speculation. 5. How simple methods like n-gram and suffix decoding behave when the prompt contains a reusable structure. The repo includes benchmark summaries, command lines, checkpoints/exports, and implementation notes. Some results are intentionally on small train-overlap eval slices due to compute constraints, so I would treat the numbers as implementation/behavioral benchmarks rather than broad generalization claims. I built this mostly as a learning resource for people who want to understand speculative decoding at the algorithm + systems boundary: how the proposer is trained, how draft tokens are generated, how target verification works, what gets cached, and where the speedups actually come from.

Seashell: open source MCP that bridges Claude and Wave Terminal, no API key needed

https://preview.redd.it/hagoyq9jrbxg1.png?width=1024&format=png&auto=webp&s=6be8dba9abafe3c7eb113392ec9ca4aeb7b46fa9 Hey folks, So I've been working on this thing called Seashell for a few weeks. It started because I was tired of treating Claude Desktop and Wave Terminal like two unrelated apps. Yes, they technically share session storage on disk, but neither side really knows about the other. I'd be deep in a Claude conversation in Desktop, drop into a terminal to check something, and the whole context was just gone. Seashell fixes that. It's an open source MCP server that plugs into Claude Desktop (Chat, Cowork, Code, all of them) and the claude CLI, plus a small set of friendly shell commands that tie everything together. The bridge goes both ways. From Claude, you can fully configure and operate Wave Terminal through MCP tools (settings, widgets, blocks, themes, scrollback, the works). From Wave, you can resume any past Claude session by name, leave Claude notes, ask async questions, all of it. And one thing worth flagging up front, because it tripped me up with other MCPs: Seashell rides on your existing Claude subscription. No API key from console.anthropic.com, no per token billing layered on top. If you already have Claude Pro or Max, there's nothing extra to pay. It just uses the same auth you set up with claude auth login. Here's the marquee feature in action: $ hey continue with myapp 🔄 Resuming session a1b2c3d4 (project: myapp)... > What's the latest on the auth refactor? We finished extracting AuthService. Tests pass. Next is wiring it into the API layer. I have a draft in routes/auth.py on line 142. That's a fresh terminal. Zero typing of session IDs, same conversation history as last night's Claude session. Fuzzy matching is built in too, so even something verbose like "hey continue with the auth refactor I was doing yesterday" figures it out. A few other things you can do with it: Configure Wave Terminal entirely from Claude, in any mode. Ask Claude in Chat to set your terminal theme, add a fish widget, switch to dracula, whatever. The full Wave config surface is exposed as MCP tools. Leave Claude a note from any terminal with seashell-msg "fix the build error" and Claude reads it next time you chat Ask Claude something async with seashell-ask "what was the last refactoring decision we made?". It blocks until you get a reply, which is great when Desktop is busy on a different conversation Run seashell-mirror-mcp to sync all your Desktop MCP servers (Trello, GitHub, Slack, whatever you have wired up) over to the CLI in one go, so resumed sessions never lose their tools Read another session's transcript directly from Claude, useful for "what's the status of project X?" type questions across projects The whole spirit of this thing is open source on open source. Wave Terminal is open source. Seashell is open source. They sit on top of each other and work together. That's the kind of stack I want to be using and contributing to. Who is this actually for? macOS users right now (Linux and Windows are not tested yet) Folks who use Claude Desktop AND the claude CLI and want them to feel like one tool instead of two Wave Terminal users (Seashell is tightly integrated, but it also works fine without Wave) fish shell users (there's a polished bundle with Alt+E editor mode, qwen2.5-coder powered natural language routing, daylight widget, the lot) Honest caveats so nobody is surprised: I built it for my own workflow first and polished afterwards, so bugs are going to happen Claude session continuity works for Code mode plus the claude CLI, since they share the same .jsonl storage. Desktop's Chat and Cowork keep their transcripts internal, so those modes are not resumable from a terminal The SessionStart hook fires only for interactive claude invocations, not for claude -p. Filesystem discovery covers most of that gap MIT licensed. Pull requests and issues are very welcome. If you have ideas for what's missing, I'd love to hear them. GitHub: https://github.com/M-Pineapple/seashell Enjoy 🍍 submitted by /u/CryptBay [link] [comments]

I Built a desktop app for generating LLM fine-tuning datasets — started it a week ago while learning FT

Hey, I've been building side projects with Claude Code for a few months, but I'm completely new to fine-tuning — started experimenting maybe a week ago. From day one I wanted a GUI for the dataset side of the workflow, so this desktop app grew alongside my very first FT attempts. I know there are similar apps out there, but I wanted something simple that non-technical users could run with open-source models end-to-end. To sanity-check whether the datasets were actually useful I fine-tuned Qwen2.5-Coder-7B-Instruct on them and ran HumanEval / HumanEval+ (pass@1, 5 runs). Picked these benchmarks because they match the dataset's focus and run fast on my machine: I know it's not much but know now that app work :) - Base: 55.5% / 49.0% - FT V2 (1135 samples from the app): 60.0% / 54.0% Error bars don't overlap so it's at least not noise. Obviously HumanEval is only one slice — YMMV with other categories / criteria. https://reddit.com/link/1srz5aq/video/zubr426holwg1/player Stack: Next.js 16 + FastAPI + SQLite, packaged as standalone binary (Win/Linux). Code: https://github.com/AronDaron/dataset-generator Fine-tuned model: https://huggingface.co/AronDaron/Qwen2.5-Coder-7B-Instruct-DatasetGen-v2 Datasets: https://huggingface.co/datasets/AronDaron/dataset-gen-v1 / https://huggingface.co/datasets/AronDaron/dataset-gen-v2 Happy to hear feedback, especially if something doesn't work on your setup or if the approach misses something obvious — this is my first finetune llm tool release. submitted by /u/AronSan [link] [comments]

SGOCR: A Spatially-Grounded OCR-focused Pipeline & V1 Dataset [P]

Hello everyone! I've been independently researching & developing small-but-powerful vision-language models (VLMs) and noticed a gap in visual datasets - none were teaching my model to simply ground text in imagery, but trying to get it to reason about the text or about the scene itself. This lead me down a 2 week side-side-project to create SGOCR, an open source dataset pipeline for generating spatially-grounded, OCR-focused VQA tuples with tons of rich metadata to support diverse VLM training strategies. Code v1 dataset My development began with simply prompting Qwen2.5-VL locally and grew into a multi-stage beast. At one point, my OCR-stage looked for concensus between 3 text recognition models (Parseq), my anchor stage did the same between GroundingDino, Florence 2, and SAM 3.1, and verification required passes from both Gemini 3.1 Pro & ChatGPT 5.3 Codex to pass. I discovered that less is more in this case, and landed on using Nvidia's nemotron-ocr-v2 for text extraction, a combination of Gemma4 with a Qwen3-VL fallback for anchor discovery & labeling, and then gemini-2.5-flash as a teacher model with simple grounding checks for verification. I got away with using the smaller 2.5 Flash teacher model due to the highly grounded annotations provided in context allowing flash to focus on semantics. I utilized an agentic loop for development after first creating a dataset review frontend that would store my personal accept/reject/maybe marks to be referenced as human-grounded context later. I bootstrapped this process into a quality score that reflected the aspects of questions I accepted, and from there the rest was much easier to automate. I run a custom optimization loop agent, based on Karpathy's autoresearch (which I found a bit too hyperparameter-searchy), that uses a sweep-based process that allows better holisitc observation, an oppurtunity to make code changes, and less risks of good ideas dying earlier due to their evals being slightly less than another variant's. I'm looking for general feedback and interested if other people were looking for something like this, or building similar VLMs. Thanks for reading! submitted by /u/Dreeseaw [link] [comments]