MLflow

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

MLflow AI

Best examples of ML projects with good dataset/task code abstractions? [D]

I am working on a benchmark and need to manage several interlocking components: datasets and metadata, diverse ML tasks (varying inputs and outputs), and baseline experiments covering models, training, and evaluations. Any pointers to projects that handle these through clean/minimal data structures like Dataclasses or Pydantic. Specifically, I want to see how others manage: Dataset Information: Representing dataset cards, metadata, and split definitions as first-class objects. Task Schemas: Defining ML tasks with specific input and output types to ensure consistency across different models. Experiment Composition: Structures that link a model and training configuration to a specific evaluation and prediction set. If you have seen repositories that maintain these abstractions with minimal boilerplate and high type safety, please share them. I am interested in internal code organization rather than external tools like W&B or MLflow. Definitely aware of cookie-cutter data-science, looking for for datastructures. submitted by /u/LetsTacoooo [link] [comments]

Anyone running LLM evals through Claude Code MCP instead of the web dashboard

Saw an OrqAI webinar on wiring Claude Code into an observability platform through MCP so the whole eval loop runs from the terminal. Got me curious about the broader pattern because the specific backend matters less than what the workflow changes. The standard eval loop is a lot of clicking. Open dashboard, filter traces, spot failure patterns, write an evaluator, run it, compare, attach the good one. Moving that into Claude Code through MCP changes the shape of the work. The parts that actually seem useful. Reading 200 traces and grouping them into failure modes is tedious by hand, the agent does the taxonomy in one pass and you correct it in natural language. Generating synthetic edge cases for evaluator stress testing is the other one, describing the cases you want beats hand writing 30 borderline PASS/FAIL examples. This only works if the observability tool has a real MCP server, not just trace export. Langfuse, Braintrust, MLflow, Orq all ship something like this now. Anyone actually running this pattern in prod. Curious how the agent generated taxonomies hold up at scale and whether the synthetic datasets end up good enough for real stress testing. Can attach the video for reference in comments, let me know. submitted by /u/Skid_gates_99 [link] [comments]

I built a Claude Code toolkit for ML on Databricks, because all the tips out there are for software engineers, not ML engineers/ML data scientists.

Hey everyone, I've been using Claude Code for ML work on Databricks for a few months now and wanted to share something I put together that might help others in the same boat. What I kept running into If you've looked into Claude Code tips and best practices online, you'll notice almost all of them are geared toward software development: edit code, run tests, ship it. And that's great, but the ML workflow on Databricks is just... different. Your code doesn't run locally. Your laptop is CPU-only but your real training happens on a GPU cluster. You can't just run your script and see if it works, you have to get your code onto the cluster, submit a job, wait, then go fish out the metrics from MLflow. And if you've dealt with DBR 15+ quirks (Workspace path errors, wheel installation changes, stale pydantic caching), you know how much time you can lose on stuff that has nothing to do with your actual model. The thing that bugged me most was that Claude would help me write great training code, and then I'd spend the next 15 minutes manually uploading, submitting, checking results, and copying metrics back so Claude knew what happened. It felt like I was the middleware. What I ended up building Over time I built up a set of Claude Code skills and agents that automate this loop. I finally cleaned them up and put them in a repo in case they're useful to anyone else: github.com/duonginspace/claude-code-databricks-ml The highlights: /run-on-databricks: builds your project as a wheel, uploads to DBFS, submits the job, waits, and pulls MLflow metrics back. One slash command instead of 5 manual steps. /iterate: you say "try adding label smoothing" and Claude implements it, submits to Databricks, pulls results, compares with previous runs, and suggests what to try next. /compare-runs: ranks your experiments, shows what helped and what hurt. /init-databricks-ml: this is the one I wish I had when I started. It scaffolds a complete project with submit/pull scripts, Makefile, MCP config, and all the DBR 15+ workarounds already baked in. /explore-data, /research-papers, /train-local: for the rest of the workflow (EDA, literature search, quick local smoke tests before burning GPU time) There are also 3 agents that the skills delegate to (experiment runner, data analyst, research agent), a /commit command, and a status bar script that shows your context window usage, git branch, and rate limits. What it actually gives you Claude can finally close the loop. It doesn't just write your code and hand it back to you, it submits, tracks, and learns from results. You go from "copilot" to something closer to a junior researcher who can run experiments on their own. You skip the Databricks onboarding tax. The DBR 15+ gotchas alone (DBFS vs Workspace paths, runtime wheel installation, stale module caching, MLflow experiment naming) cost me days to figure out. /init-databricks-ml handles all of it from day one. Faster iteration cycles. Instead of context-switching between your editor, the Databricks UI, and MLflow every time you want to try something, you stay in the terminal. Say "try X" and come back to a comparison table. Your experiments stay organized. Every run gets logged to MLflow automatically, and /compare-runs gives you a ranked summary instead of you eyeballing dashboards. It's easier to spot what's actually working. Less wasted GPU time. /train-local lets you smoke-test on CPU before burning cluster hours, and the skills are structured to catch obvious issues early. It's modular. You don't have to use everything. Install just the one skill you need, or the whole toolkit. They work independently. Install git clone https://github.com/duonginspace/claude-code-databricks-ml.git cd claude-code-databricks-ml bash setup.sh Copies everything to ~/.claude/. MIT licensed. This is very much shaped by my own workflow, so it won't be perfect for everyone. But if you're doing ML on Databricks with Claude Code, or thinking about trying it, I hope it gives you a head start. Would love to hear how others are handling this, and happy to answer any questions. submitted by /u/duongnguyen0512 [link] [comments]

[D] Litellm supply chain attack and what it means for api key management

If you missed it, litellm versions 1.82.7 and 1.82.8 on pypi got compromised. malicious .pth file that runs on every python process start, no import needed. it scrapes ssh keys, aws/gcp creds, k8s secrets, crypto wallets, env vars (aka all your api keys). karpathy posted about it. the attacker got in through trivy (a vuln scanner ironically) and stole litellm's publish token. 2000+ packages depend on litellm downstream including dspy and mlflow. the only reason anyone caught it was because the malicious code had a fork bomb bug that crashed machines. This made me rethink how i manage model api keys. having keys for openai, anthropic, google, deepseek all sitting in .env files across projects is a massive attack surface. switched to running everything through zenmux a while back so theres only one api key to rotate if something goes wrong. not a perfect solution but at least i dont have 6 different provider keys scattered everywhere. Run pip show litellm right now. if youre on anything above 1.82.6 treat it as full compromise. submitted by /u/Zestyclose_Ring1123 [link] [comments]