PIQA and HellaSwag problem
The formula seems pretty solid at first, but when you actually look at it, you realize that the only benchmarks that actually matter are only piqa and hellaswag.
A model that gets 100 percent on both ARCs, and arithmark and with hellaswag + piqa at the baseline would get 45.21 int index.
That seems fair until you realize that a model that simply scores 100 on piqa and just 75 on hellaswag with all the others at the absolute baseline gets a 45.66.
Someone could simply make a 5M parameter model trained on only problems similar to piqa and hellaswag, submit it, and have it be crowned "the smartest model on the leaderboard" even if it does not have any actual intelligence or any capability to be a genuienly useful assistant.
On top of that, piqa and hellaswag aren't even the hardest benchmarks, piqa is the easiest benchmark and it matters 2 times more than ARC challenge which is much, much harder and proves way more than piqa, and arithmark 3.0 is the only benchmark in the list that proves a model can do math yet it apparently should matter way less than just a 2 choice commonsense benchmark.
To further prove my point, look at the leaderboard categories, for every top 1 model in the 50M, all, and 100M categories there they only have the most hellaswag and piqa compared to other models and are not the top 1 on any of the other benchmarks.
I'm not saying piqa and hellaswag are useless, i'm saying they just shouldn't matter this much.