so, how did it do?
#2
by
robbiemu
- opened
Now that you published the full fat version, I have to ask -- did you measure perplexity, or have any measure or recommendation based on feel for which quantization is "still SOTA" but smaller?
No I haven’t. It’s very costly.
In the past I’ve tested my fine tune models with:
https://github.com/EleutherAI/lm-evaluation-harness
It has the same test set that the open LLM leaderboard uses so it gives you a good comparison.
So far I haven’t tested the quantized models
Yes that makes sense. I really wanted to use this model, i do a fair bit with language learning in ai, dating back to my masters capstone in 2019, but it turns out i would be stuck in the 2bit regime here :(
Its get if you too provide these :) thanks for answering my question
robbiemu
changed discussion status to
closed
