Checkly

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Checkly AI

Built an LLM training framework that actually runs on older GPUs without crashing [P]

Hey guys, I was playing around with Nanotron recently and got super frustrated by how many heavy, hardware-specific dependencies it imports at the module level ( flash-attn , triton, functorch , etc.). If you try to run it on older or budget GPUs like a T4 or V100, it just crashes on import. So I wrote Picotron (https://github.com/Syntropy-AI-Labs/picotron) to solve this. It's a clean-room rewrite that gets rid of all mandatory GPU-specific dependencies. It runs on pretty much any GPU that supports PyTorch (defaults to FP16 on older cards under compute capability 8.0, and BF16 on newer ones). It falls back to standard PyTorch SDPA by default, but still hooks into FlashAttention-2 at runtime if it detects you have it installed. I used an AI assistant to write a lot of the boilerplate/code modules, but I've got it working locally and just trained a tiny 2M model on FineWeb-Edu. Also added configs for: • GQA / MLA (Multi-head Latent Attention) • QK-Norm & logit soft-capping (Gemma 2 style) • Parallel FFN/Attn runs • ZeRO-1 wrapping on DDP Roadmap is pretty short right now: MoE prep (routing capacity factors and load balancing loss) Making dataset prep easier than streaming manually Check it out if you've been fighting with CUDA dependency hell: https://github.com/Syntropy-AI-Labs/picotron submitted by /u/Capital_Savings_9942 [link] [comments]

Hiding messages in the least significant mantissa bits of fine-tuned ONNX model weights [P]

Hey everyone, I'd like to share my project along with a short explanation of the process and why it came about in the first place. To start off, I'm not exactly the best at cryptography/steganography, in my case it's always been something that sat in the background, as one of the sub-fields needed for another (main) field I'm actually interested in. For this project I tried to look up as much information as possible about what's currently considered best practice (I mainly relied on NIST for this), what implications exist, and what potential "attacks" exist against this way of hiding information, but I honestly can't say whether I covered everything, which is why I wanted to share this project here, mainly for the sake of learning. I'd be grateful for any feedback on what I could have done better / what I might have missed, etc. Right now, I consider this project closed at this point and will most likely not update it further, although I'd like to apply all the feedback to my own knowledge going forward. For over a month I did a lot of research into using ML models as a carrier for hiding data. I needed this as one of the stages for my main project. That's how I ended up on the topic of hiding information in model weights. Initially I assumed a simple method of directly writing data into randomly selected weights. I quickly concluded, though, that this would be absurdly trivial to detect, and potentially also to read. Next came the idea of using something like a deterministic coordinate map describing where to read the data from (location-id + position-id). The program wouldn't modify all the bits needed to write the message instead, it would write separate bits representing already-existing values (pointing to specific locations in the model) from which the existing 0s and 1s would need to be read. In practice, only parties A and B would know how to derive these positions. This way, someone unaware of the algorithm would only see what looks like noise of varying values. However, after a theoretical analysis of a practical implementation, this idea had serious flaws. Even setting aside the fact that the main goal was steganography and not encryption, the mere presence of additional data could be relatively easily detected, for instance through delta analysis against a reference model, or through analysis of the statistical properties of the weights. On top of that, this method would really only allow transmitting a very small amount of data, because just indicating, say, the word "example" would look like this: "01100101011110000110000101101101011100000110110001100101", so it would be extremely impractical. In other words, even if the hidden message itself couldn't be read, one could still suspect that the model contains hidden information, which would defeat the whole point of steganography. While I found the previous option conceptually pretty interesting, I moved on, which led me to the question: "How do I hide data in the weights in a way that won't be visible?" That led me to the next idea: since every fine-tuning process naturally changes some of a model's weights anyway, why not hide information only in the weights that get modified during training regardless? In that case, the fine-tuning itself would provide a natural and logical explanation for the presence of those changes, including when compared against a reference model. It was only later that I found out that similar/identical concepts had already been described in the scientific literature, although they remain a fairly niche research direction. Skipping over the implementation details (since everything is described in the README and SECURITY files, and I don't want to dump even bigger wall of text here), this is how the first implementation of the solution (part of my main project) came about. After further research I noticed that most existing publications focus on the academic side, while the available GitHub repositories were often poorly documented, limited in functionality, good steganographically but weak cryptographically, or were just a small piece of larger projects. Personally, I couldn't find any project implementing a similar idea specifically using models saved in the ONNX format. So I decided to split this part off and refine it as a separate proof of concept, and that's how ONNXStego came about. If anyone's interested in the security, limitations, or implementation details, feel free to check out the repository. I personally learned a great deal from this project and tried to describe the final conclusions/information I gathered while learning as precisely as possible, so I'm hoping the project can also be useful to others for their own purposes or projects. (If this counts as self-promotion, I apologize in advance, and I can remove this post for that reason too if needed, I tried to describe the whole process behind it as accurately as I could, to make the post as educationally useful as possible). Link: https://

Anyone else feel like a ghost in the machine? The bizarre isolation of AI training.

I have been working in the AI training and data annotation space for a while now, and it is easily one of the strangest industries I have ever been a part of. On one hand, the perks are real. The flexibility is unmatched, you can work in your sweatpants, and sometimes you get genuinely fascinating prompts that actually challenge your brain, whether you are grading complex code, checking historical facts, or analyzing legal logic. But on the other hand, the complete and total isolation is starting to get pretty bizarre. We are helping build the future of technology, yet we do it in total silos. If you have ever been in an official platform Slack or forum, you know the vibe. You are constantly walking on eggshells. You cannot openly ask about sudden dry spells, you cannot critique confusing or contradictory guidelines without worrying about a random shadowban, and the second a project ends, you are instantly booted from the channel. Any temporary "coworkers" you had just vanish overnight. It feels like the platforms go out of their way to keep us from actually talking to one another without a moderator watching over our shoulders. It is a weird mix of having total freedom but zero community. I am curious what everyone else’s experience has been like lately. What are your personal pros and cons of the gig right now? How do you deal with the isolation, or do you actually prefer the ghost lifestyle? Also, out of pure curiosity, how do you even explain what you do for a living to your friends and family without their eyes glazing over? submitted by /u/Smooth_Sailing102 [link] [comments]

Showcase: Building ML models that "watch" MMA fights and label events and positional changes making these moments all searchable on a timeline [P]

Hey all, a bit of background - I'm an ex Amateur MMA fighter and BJJ brown belt and am also in the AI/ML space ... weird combo but wanted to know if anyone else was at the intersection of ML/AI and MMA/BJJ. In short, I'm building AI models that "watch" fights and are able to detect positions and moments throughout the fights - things like standing vs clinching vs ground (with intention of becoming more granular in time) along with detecting knockdowns, takedowns, etc. There's a timeline at the bottom of each fight with markers for different moments so you can jump straight to them. Anyway this is where my worlds collide and was curious for thoughts for anyone who wants to check it out. If you do, it's at https://cagesight.ai. All feedback welcome. Thanks all. submitted by /u/UnholyCathedral [link] [comments]

i analyzed 500+ companies job postings to see what new roles are emerging due to ai

i kept seeing doomer posts talking about how ai is going to take away all jobs. i believe in the opposite - ai is going to add more jobs in the long term than it cuts, and i kept seeing evidence of that now. there were job titles i'd come across that 2-3 years were much more niche. i got curious to map this out so i created a site to track this. i scraped various job boards and filtered out titles which have exploded since 2022 thanks to ai and if you go through the onboarding, it'll match you to roles you are eligible for. i don't want this post to just be a promo so here are the top 5 roles we are seeing break out. the number in brackets is the number of such jobs we are tracking ai trainer / data annotation (1,218) forward-deployed engineer (485) ai solutions eng / architect (316) agent (engineer/pm/research) (260) applied-ai engineer (169) if you're interested in checking out the site for more roles/jobs you can check it out here: https://alterwork.com/roles any feedback would be great, thanks submitted by /u/AppropriateHamster [link] [comments]

A debugger for RL reward functions that detects reward hacking during training [P]

While experimenting with GRPO training, I kept running this shit that when reward increases, it becomes difficult to tell whether the policy is genuinely improving or simply exploiting the reward function. So I built a small library called rewardspy that wraps an existing reward function and continuously monitors indicators that often precede reward hacking. It currently tracks things like rolling reward statistics, reward variance collapse, reward component imbalance, response length drift, reward slope changes, GRPO group collapse, anol. This is my first major RL project so I would absolutely love some technical advice Check it out here: https://github.com/AvAdiii/rewardspy (credits to u/Oranoleo12, posting on their behalf) submitted by /u/BaniyanChor [link] [comments]

When there is no answer key for scientific discovery how do we verify an ai hypothesis

I have been thinking a lot about the actual limits of AI-driven scientific discovery, specifically how we evaluate models when they are proposing genuinely new hypotheses where no "answer key" exists. When we test LLMs on standard benchmarks, we have a clean dataset with known solutions. But if we task a frontier model with proposing a novel chemical compound for carbon capture, or finding an undocumented biological pathway, there is literally no ground truth in the literature. The immediate response is usually "just run the physical experiment." But wet-labs are incredibly slow and expensive. You can't synthesize thousands of candidate compounds blindly. This means the bottleneck for AI in science isn't our ability to generate hypotheses, it's our ability to verify them under absolute uncertainty. The traditional way to check model outputs is self-reflection or self-grading. But this is a dead-end for discovery. If you ask a model to double-check its own chemical structure, it has the exact same theoretical blind spots that generated it in the first place. It just agrees with itself louder. I was reading about a new multi-agent research engine called Apodex that launched earlier this month, and they rely heavily on this split. Instead of a single model doing the work, they use independent verifier agents that are completely blind to the generator's internal prompts. The verifier's job is to take the proposed hypothesis, re-derive the underlying physical logic from first principles, and find contradictions. Those contradictions are then fed back to the generator as constraints for a revision pass. Instead of a self-check, making verification a completely distinct, adversarial step is the only way to squeeze out actual science from these models. If we can't verify, we can't truly discover. If the AI doesn't have an isolated checker, then we are just generating highly plausible guesses. How are your teams handling this transition? When a model proposes a candidate solution in your research, what is your standard of evidence before you spend actual physical or computational resources to test it? submitted by /u/Glad_Ad217 [link] [comments]

Traditional SDLC vs Agentic SDLC

Traditional Software Development Life Cycle vs Agentic Software Development Life Cycle in 2026. What do you think? submitted by /u/Illustrious-King8421 [link] [comments]

Free and Open Source remarkable and unusual Ambient Audio Visual experience

Hi everyone - For a year I tried to realize this idea I had to algorithmically draw a field of cubes based on my observing how I do it as a person. I tried this with Perplexity, gemini and OpenAI. ALL failed! Part of the problem is that to do what I am doing requires every point, every line and every face created by my algorithm to be accounted for so that the algorithmically (not generatively) created result NEVER crosses over itself but will fill up the entire screen over time before starting over. I tried Claude as a last resort, and I kid you not: in 90 minutes I had a working prototype. So, a couple of things: Go to my website https://alienConsumerSciences.com (linked in this post) TRY the 2D and 3D versions of what I call Mandra Corners READ the papers on the algorithm and the collaborative experience EXAMINE the completely interactive dev log on that page PLAY the game I created (based on code I wrote for a playdate game ) called CENTERING (the simplest most complicated game you will ever play at the intersection of meditation, gaming and flow states) Everything I have mentioned is available at https://alienConsumerSciences.com PLEASE check it out :) Always free and open source, NO dependencies, runs fully in a browser! submitted by /u/katastatik [link] [comments]

Dev Log on Steam Recommender[P]

Since the steam sale is live I wanted to post a Dev log on my personal project https://nextsteamgame.com/ sharing some outcomes from the web traffic and how I changed the project from the great feedback I got! I made a post about a month ago explaining how I made this opensource explainable search engine built around steam reviews to people find new video games, Not through Relevancy but through aspect based similarity. Check out the old post for a better explanation if you want! https://www.reddit.com/r/MachineLearning/comments/1tb8k3n/steam_recommender_using_similarity_undergraduate/ I wanted to say thank you to all the people of r/datascience and r/MachineLearning that gave me feedback and tried out my tool! I improved the UI/UX of the website to make the vectors more clear and controllable, I Implemented a thumbs up and down feature on recommendations to see if users even like the tool. I also wanted to share the after effects of promoting this tool on reddit! from the 2,652 searches I got in the website 913 of them resulted in steam clicks! the games that were discovered were all in a uniform distribution and did not share much of a pattern showing me that the engine did its job in helping people find niche games across all genres! (More images attached to post to see data viz) I wanted to disclose that I made this tool to not make any profit of some kind, but it does use posthog so I can collect diagnostics now. submitted by /u/Expensive-Ad8916 [link] [comments]

Built a loop engineering skill for PRs in Claude Code — branch, two independent reviews, CI handling, merge handoff. Here's what I learned building it.

Most agentic coding tools are really good at one thing: writing code fast. you describe a problem, they implement it, done. and honestly that part has gotten pretty good. The part nobody talks about is everything after the first commit. review. CI. merge discipline. That's where real codebases fall apart, and none of the tools I'd tried had a serious answer for it. So I built /pr-loop, a Claude Code skill that drives a work item through the full PR pipeline. I want to share how it works and what surprised me along the way. What it actually does You give it a GitHub issue number, or just describe the task. It does the rest in order: Reads your project's contribution rules before touching anything (CONTRIBUTING.md, CLAUDE.md, README, whatever exists). It's learning your commit format, your branch naming, your test commands, your merge rules. if none of that is documented, it asks you. It doesn't guess. Then it checks if the issue itself is actually actionable. If the description is vague or the acceptance criteria are missing, it stops and asks. a 3-word issue title is not enough to act on. this saved me from a surprising number of expensive wrong turns. Then it branches, implements, runs a conflict check, and runs your local gates (tests, linter, build). all green before it pushes anything. fFr large changes - touching a lot of files or requiring a design decision - it opens a draft PR early to get directional feedback before going all in on the implementation. This one sounds small but it's changed how i structure work. The part i'm actually proud of: three separate agent contexts This is the core design decision, and I think it's the right one. There are three completely separate contexts: the author (writes the code), two reviewers (run in parallel, can't see each other), and an optional merger. The context that wrote the code never reviews it. Never merges it. You might think that doesn't matter for an AI. It does. An agent reviewing its own code has the same blind spots the author had. It'll miss the same edge cases. It'll rationalize the same tradeoffs. Keeping the contexts separate isn't just a rule - it produces meaningfully different output. The two reviewers aren't doing the same thing either. One does structured analysis: security, correctness, performance, maintainability. The other actually runs the gates and probes whether the change does what the PR claims. different lenses, different failure modes caught. The review loop Every finding gets addressed. that includes nits. a reviewer flags a variable name, it gets renamed. Nothing gets silently dropped or marked "low priority" to die in a backlog. There's a 3-round cap. If findings are still surfacing after three rounds of review and fix, the loop pauses and asks you. Because if it can't resolve something in three passes, it's probably a judgment call that needs a human. Merge is opt-in The default is to stop at handoff. once both reviews are clean and CI is green, the skill reports: "PR open, CI green, both reviews clean. awaiting your merge." and stops. What's still missing To be honest with you: it doesn't handle multi-repo or monorepo cross-package changes well yet. If your PR touches two packages with independent CI, the current logic doesn't know how to wait on both properly. It also can't handle merge conflicts autonomously. If the branch gets conflicted mid-pipeline, it surfaces it to you and stops. That's the right call - auto-resolving non-trivial conflicts is how you get subtle bugs - but it does mean you have to intervene sometimes. And there's no escalation path yet for when a reviewer fires a Request Changes that I genuinely can't resolve without a product decision. Right now it pauses and asks. I want to eventually route those to a separate "product judgment" agent, but I haven't built that. the skill file itself is ~105 lines no framework. no orchestration layer. no dependencies beyond git and the gh CLI. just a markdown file with a structure that Claude Code reads and executes. Medium Article with Skill Details : https://medium.com/developersglobal/loop-engineering-in-practice-i-built-a-105-line-skill-that-runs-the-full-pr-pipeline-fc102b050127 submitted by /u/Naive_Maybe6984 [link] [comments]

Claude needs to learn to eat his vegetables... at the end of code runs before shipping code audits, complexity tests, dead and silent failing code, documentation gaps all need checking. This /eat-the-broccoli skill helps with that across languages (currently python and rust)

Claude doesn't always like to eat his vegetables, but he knows sometimes even broccoli is healthy. So we gave him a skill for it: /eat-the-broccoli. It's the quality pass he knows he should run after every sprint but often skips anyway. Two parts. First the boring deterministic stuff, your linter, type checker, tests, dependency CVEs, dead code, silent failures. Tools you already have, run in one shot. The second part is the one that earns the name. The pattern hunt: A checklist of the failure classes that pass every test and still ship broken. Races, stale caches, silent fallbacks, empty values hiding missing data, the deployed copy that's quietly drifted from the source. Linters miss these because they're structural, not syntactic. They need an eye, and one question: this looks intentional, is it actually? Every pattern carries that check, so you catch the real bugs... without the deterministic sweep complaining when things are deliberate. This helps most in complex codebases, and particularly in vibecoded and iterative workflows. The kind where you've been moving fast, the model has been moving faster, and nobody is quite sure what broke three commits back. An empty list that used to mean no results, now quietly meaning the database timed out and got swallowed. A flag you passed that fell back to its default without a word. Tests pass, it looks fine, and it isn't. That's what broccoli is for. You choose how deep, a quick pass before a commit, a standard sweep before a release, a deep one after a big refactor. And how wide, just what changed, one module, or the whole thing. When something turns out to be intentional, you record it once and it stops nagging. The checklist gets sharper every time someone adds the bug that just bit them. Works in any language, wire in your own tools, MIT licensed. Here: github.com/EmpiricaAI/broccoli submitted by /u/entheosoul [link] [comments]

Analyzed over 30 of my opus ultra code sessions and created a prompt template to improve dynamic workflows - orchestrate agent spawning, reduce token burn and enforce verifier sub-agents

I analyzed 30+ of my own Opus ultra code sessions with Claude to understand how the dynamic workflow executes and where tokens were getting spent and identify any scope for savings. In ultra code mode Claude runs a task by writing deterministic JavaScript, calling subagents for each step, and keeping intermediate results in script variables instead of the chat. Separate verifier agents check the output, so the same agent is not reviewing its own work. The token burns came from multiple items: Small tasks expanded into large workflows Even a small intervention triggered too many subagents The same files were being read by multiple agents Constant introspection of self work No judgement of delegating tasks to cheaper sonnet or haiku models where it could Verification became an open loop of checking, revising and checking again Changing my prompts to specifically orchestrate how opus should execute the workflow started making a ton of difference. The prompt structure defines the tasks, defines the outcome goals, the verification and approval process for a task output of each sub-agent, pre-plan delegation to cheaper models, reduce introspection since verifier can take care of the judgement, cap the number of sub-agents. The prompt structure also makes Claude breakdown the task on hand and propose a subagent count scaled to the independent pieces of work, which I approve or correct before the final run. I built it into a skill command that rewrites my simple prompt into a workflow appropriate prompt that dictates the orchestration and execution of dynamic workflow in ultra-code mode. I have shared the skill here if others want to us it - Click here Nevertheless, it is not fool proof but still better than letting Claude loose and decide on its own, because the orchestrating javascript itself is the first thing it creates in Ultra-code mode, the probability of it adhering to the prompt is very high, but it is still probabilistic. Hope you find it useful. submitted by /u/coolreddy [link] [comments]

I ran my Claude Code model router for 13 days. Here are the real numbers.

I built a small Claude Code routing layer called Gearbox. The goal is to stop sending everything to expensive models by default. Each subagent delegation gets routed to the cheapest model tier that should be able to handle the task. The intended ladder is: scout → Haiku, read-only exploration grunt → Haiku, simple mechanical edits, no logic changes builder → Sonnet, scoped implementation architect → Opus, hard reasoning and debugging verifier → checks diffs before results are accepted I have now run it on my own real projects for 13 days, from June 12 to June 25. Actual usage: 155 delegations 8 projects 23 sessions Model split: Haiku: 48 delegations, 31.0% Sonnet: 77 delegations, 49.7% Opus: 9 delegations, 5.8% No model recorded: 21 delegations, 13.5% So on the surface, the routing worked pretty well. About one-third of all work went to Haiku, and Opus stayed under 6%. But there is a pretty big flaw in the current version. Only 48.4% of delegations went to named Gearbox tier agents. The other 51.6% went to either the generic general-purpose agent or the built-in Explore proxy. That matters because the safety rules are inside the named agent files. If the work goes to a generic agent, it does not necessarily get the read-only scout rules, the grunt restrictions, or the expected verifier path. So the real conclusion is not: “Gearbox solved model routing.” It is more like: “Gearbox is already reducing expensive model usage, but half the traffic is still escaping the intended tier ladder.” The annoying part is that I cannot yet prove why. It might be: my fallback path firing when a named agent is unavailable Claude Code choosing a generic agent on its own missing metadata in how I log delegations some mix of all three The current log records the routing decision, but not enough about the outcome. So v0.2 needs to add: fallback reason selected agent intended agent escalation event verifier verdict whether the diff was accepted or rejected The analyzer is included in the repo at: bench/analyze-log.py It reads your own gearbox-log.jsonl files and does an independent recount with assertions before printing results. Repo: github.com/Adityaraj0421/gearbox Would be curious how others are handling Claude Code subagent routing. Are you letting the orchestrator pick agents freely, or forcing named agents more strictly? submitted by /u/Known-Delay-9689 [link] [comments]

Is it always this funny?

Been using Codex for bulk stuff before passing off to Claude and they randomly started being funny as hell lol. “I’m upgrading the validator too so that particular gremlin doesn’t get a second career” 🤣 submitted by /u/12thYearSenior [link] [comments]