I saw QAT stuff going on for a week or two. I wonder if finetunes going forward will all use QAT as a base?
Or will they do Loras?
I'd wondered for a bit why i see some Lora's where that are almost as big as the base model, vs say SDXL where a Lora is 200Mb vs a 6Gb base.
Either way, I'll be keeping eye eyes peeled for interesting releases everywhere.
I mean, like, we are talking about a 50M model here.
On models smaller than say 256Mb there's no reason to use anything but full precision.
Though with Prism-ml/Bonsai showing you can get stuff done on 1 and 1.58bit, we may see some very small things in the near future... Just hope nothing that would bork at important times.
I've wondered, if a sufficiently reasoned model can get away with missing a lot of data because it can reason it out, and thus the smaller size may be worth it to be faster/leaner (even if it needs more tokens to reason the solution).
I'd love to see this treatment on some of the larger models. Been using G4 26B's and used to use 70B models, those squashed down would remove the need to quantize at all. It would even make the 100B+ models workable.
(note, 8Gb VRAM so memory is definitely the bottleneck)
