Prodigy

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

Prodigy AI

[New Optimizer] 🌹 Rose: low VRAM, easy to use, great results, Apache 2.0 [P]

Hello, World! I recently released a new PyTorch optimizer I've been researching and developing on my own for the last couple of years. It's named "Rose" in memory of my mother, who loved to hear about my discoveries and progress with AI. Without going too much into the technical details (which you can read about in the GitHub repo), here are some of its benefits: It's stateless, which means it uses less memory than even 8-bit AdamW. If it weren't for temporary working memory, its memory use would be as low as plain vanilla SGD (without momentum). Fast convergence, low VRAM, and excellent generalization. Yeah, I know... sounds too good to be true. Try it for yourself and tell me what you think. I'd really love to hear everyone's experiences, good or bad. Apache 2.0 license You can find the code and more information at: https://github.com/MatthewK78/Rose Benchmarks can sometimes be misleading. For example, sometimes training loss is higher in Rose than in Adam, but validation loss is lower in Rose. The actual output of the trained model is what really matters in the end, and even that can be subjective. I invite you to try it out for yourself and come to your own conclusions. With that said, here are some quick benchmarks. MNIST training, same seed: [Rose] lr=3e-3, default hyperparameters text Epoch 1: avg loss 0.0516, acc 9827/10000 (98.27%) Epoch 2: avg loss 0.0372, acc 9874/10000 (98.74%) Epoch 3: avg loss 0.0415, acc 9870/10000 (98.70%) Epoch 4: avg loss 0.0433, acc 9876/10000 (98.76%) Epoch 5: avg loss 0.0475, acc 9884/10000 (98.84%) Epoch 6: avg loss 0.0449, acc 9892/10000 (98.92%) Epoch 7: avg loss 0.0481, acc 9907/10000 (99.07%) Epoch 8: avg loss 0.0544, acc 9918/10000 (99.18%) Epoch 9: avg loss 0.0605, acc 9901/10000 (99.01%) Epoch 10: avg loss 0.0668, acc 9904/10000 (99.04%) Epoch 11: avg loss 0.0566, acc 9934/10000 (99.34%) Epoch 12: avg loss 0.0581, acc 9929/10000 (99.29%) Epoch 13: avg loss 0.0723, acc 9919/10000 (99.19%) Epoch 14: avg loss 0.0845, acc 9925/10000 (99.25%) Epoch 15: avg loss 0.0690, acc 9931/10000 (99.31%) [AdamW] lr=2.5e-3, default hyperparameters text Epoch 1: avg loss 0.0480, acc 9851/10000 (98.51%) Epoch 2: avg loss 0.0395, acc 9871/10000 (98.71%) Epoch 3: avg loss 0.0338, acc 9887/10000 (98.87%) Epoch 4: avg loss 0.0408, acc 9884/10000 (98.84%) Epoch 5: avg loss 0.0369, acc 9896/10000 (98.96%) Epoch 6: avg loss 0.0332, acc 9897/10000 (98.97%) Epoch 7: avg loss 0.0344, acc 9897/10000 (98.97%) Epoch 8: avg loss 0.0296, acc 9910/10000 (99.10%) Epoch 9: avg loss 0.0356, acc 9892/10000 (98.92%) Epoch 10: avg loss 0.0324, acc 9911/10000 (99.11%) Epoch 11: avg loss 0.0334, acc 9910/10000 (99.10%) Epoch 12: avg loss 0.0323, acc 9916/10000 (99.16%) Epoch 13: avg loss 0.0310, acc 9918/10000 (99.18%) Epoch 14: avg loss 0.0292, acc 9930/10000 (99.30%) Epoch 15: avg loss 0.0295, acc 9925/10000 (99.25%) I used a slightly modified version of this: https://github.com/facebookresearch/schedule_free/tree/main/examples/mnist Highest accuracy scores from 20 MNIST training runs (20 epochs each) with different seeds: ```python from scipy.stats import mannwhitneyu rose = [99.34, 99.24, 99.28, 99.28, 99.24, 99.31, 99.24, 99.21, 99.25, 99.33, 99.29, 99.28, 99.27, 99.30, 99.33, 99.26, 99.29, 99.26, 99.32, 99.25] adamw = [99.3, 99.15, 99.27, 99.2, 99.22, 99.3, 99.22, 99.15, 99.25, 99.29, 99.2, 99.22, 99.3, 99.23, 99.2, 99.25, 99.22, 99.28, 99.32, 99.22] result = mannwhitneyu(rose, adamw, alternative="greater", method="auto") print (result.statistic, result.pvalue) ``` Mann-Whitney U result: 292.0 0.006515916656300127 Memory overhead (optimizer state relative to parameters): Rose: 0× SGD (no momentum): 0× Adafactor: ~0.5-1× (factorized) SGD (momentum): 1× AdaGrad: 1× Lion: 1× Adam/AdamW/RAdam/NAdam: 2× Sophia: ~2× Prodigy: ~2-3× OpenAI has a challenge in the GitHub repo openai/parameter-golf. Running a quick test without changing anything gives this result: [Adam] final_int8_zlib_roundtrip_exact val_loss:3.79053424 val_bpb:2.24496788 If I simply replace optimizer_tok and optimizer_scalar in the train_gpt.py file, I get this result: [Rose] final_int8_zlib_roundtrip_exact val_loss:3.74317755 val_bpb:2.21692059 I left optimizer_muon as-is. As a side note, I'm not trying to directly compete with Muon's performance. However, a big issue with Muon is that it only supports 2D parameters, and it relies on other optimizers such as Adam to fill in the rest. It also uses more memory. One of the biggest strengths of my Rose optimizer is the extremely low memory use. Here is a more detailed look if you're curious (warmup steps removed): [Adam] text world_size:2 grad_accum_steps:4 sdp_backends:cudnn=False flash=True mem_efficient=False math=False attention_mode:gqa num_heads:8 num_kv_heads:4 tie_embeddings:True embed_lr:0.05 head_lr:0.0 matrix_lr:0.04 scalar_lr:0.04 train_batch_tokens:16384 train_seq_len:1024 iterations:200 warmup_steps:20 max_wallclock_seconds:6

Honest question for y’all

I’m a linguist and somewhat of a language prodigy. I have my own language learning method and curriculum which I would like to develop into simulation game to help people acquire language more naturally and restore people’s curiosity about other cultures and languages. I have great hesitation working with open AI because I know that whatever work I do on there is not necessarily private, it can be stolen, or OpenAi itself may steal/sell the idea. I have also found ChatGPT to be severely lacking with its language capacities and workflow with the AI in general is extremely frustrating. It makes numerous mistakes, offer suggestions that were not warranted, and just does a poor job overall. And yes, I am paying for the version that I have. My idea is unique and nobody has done it yet … I am also not going to go into too much depth on here about the idea. Incorporating AI will make building out my idea much quicker, however I don’t feel like OpenAI is the right platform. Does anyone have any suggestions? Thanks in advance … submitted by /u/soulviche [link] [comments]