Davis Blalock

If you liked this thread and/or think others might benefit from this work, consider retweeting it! https://t.co/yaoKztyB4h We really think this is something most practicing ML people should be using

The code is a single Python file with nice Triton kernels and an easy interface for applying the optimizations to new optimizers: https://t.co/WbZGdCCJjF You can also `pip install flashoptim` and the

Finally, for gradients, we just keep them in the same precision as the backward pass weights instead of upcasting them. We also optionally enable gradient release, which steps params and frees their

With these techniques, FlashOptim optimizers do the same updates as your current optimizers, but use less memory. This means you can use bigger batch sizes and/or train larger models if you’re too G

Just to emphasize: You can make B-bit quantization go from broken to great with one line of preprocessing. You don’t need an exotic new data format. Tuning the companding functions for what you’re qu

We use different companding functions for momentum-like optimizer states and variance-like optimizer states. These functions dramatically reduce the quantization error, effectively buying more bits of

That tool is companding. Companding is invertible preprocessing to make your data distribution better fit your quantization format. It’s a decades-old idea ubiquitous in compression and signal proces

To reduce optimizer state sizes, we start by linear quantizing each state variable to 8 bits + a 16-bit scale for every 32 elements. However, we found that this diverged out of the box. So we had to

Given this bound, you can just *integer quantize* where in the interval [val_min, val_max] the original x32 is. This N-bit integer is your N ECC bits. This scheme yields exact reconstruction for most

The core observations are this: a) When you downcast an f32 value x32 to a lower-precision float x’, there is a fixed interval of f32 values that map to x'. b) You can tightly bound this interval [val

Now what’s in these ECC bits and how do we use them? You might think the ECC bits are just the low bits of the float32, but this breaks for any dtype combination that isn’t (bf16, f32). Instead, we

This weight splitting also makes it easy to reduce the precision of the master weights. To get 24-bit master weights without horrible 3-byte datatypes, we just use 8-bit ECCs and B=16-bit downcast we

We eliminate the downcasting redundancy by splitting up the 32-bit master weight into the B bit low-precision version and a (32 - B) bit error correction code (ECC) that lets us recover the 32-bit val

There are two sources of potential savings in parameter memory: 1) Avoiding redundancy from downcasting. Many mixed-precision setups keep model.parameters() in f32 but use a separate low-precision co

🚀 Today we’re releasing FlashOptim: better implementations of Adam, SGD, etc, that compute the same updates but save tons of memory. You can use it right now via `pip install flashoptim`. 🚀 https:/

First, observe that the memory required by an optimization algorithm isn’t just the optimizer state: you also have to store a gradient and a high-precision “master weight” for each parameter. So if y

Legal limbo also means that both compliant and non-compliant actors look the same from the outside. The non-compliant say nothing about their data because they did violate rules, and the compliant sa

Anyway, I'm not a lawyer and I make no claims about what should happen with AI and public data, but I hope this thread helped you understand a bit about what is happening. Further disclaimer: all of

A consequence of this caring-about-rules spectrum is that there's a "compliance gap" in many evals. If you have an idea where a given company (or even academic lab) sits, you should be either more or

This legal ambiguity gives rise to hilarious situations like technical leaders saying they "can't recall" what went into their models.