GODELEV

/

Ant-5M

like 2

Thanks !!
But tbh I dont think this model is cool , LIke i blundered the model architeture , in making of as deep model as possible.
Though I learnt a lot in the process.

Oh if anything I think the depth was good, maybe ideal would have been more around 9 layers or something, but I think the main issue is the tokenizer being too big, other than that I reckon it looks promising!

Yeah , You are right
I ll do 4096 next time.
I think that will be perfect .

Yeah reinvest the parameters into width, would be great!

Also mind if I add it to the leaderboard?

Well .....Yeah add it
Thanks !!