"Perhaps the eight smaller LLMs can even collaboratively learn from each other" That's not how MoE works.

Also, the LHC is an engineering breakthrough in search of a scientific breakthrough; actually pretty much like GPT-4 - if OpenAI really intended for it to become a scientific breakthrough, which they aren't. That is exactly what I'm criticizing. The overhype, the crazy speculation based on unknown assumptions, etc.