This would be a good way to verify emergent model capability to synthesize new knowledge.
You give an LLM all the information from right before a topic was discovered or invented, and then you see if it can independently generate the new knowledge or not.
It would be hard to know for sure if a discovery was genuine or accidentally included in the training data though.
I saw Musk repost a boast that Grok created a whole new ("superior") element design for a incandescent bulb using Edison's patent. The implication was that Grok was superior to Edison's team. I was just sat there thinking about the 100+ years of incandescent bulb research that Grok has sucked up from various science papers and random Internet archives. Surely none of that was any help at all /s.
You give an LLM all the information from right before a topic was discovered or invented, and then you see if it can independently generate the new knowledge or not.
It would be hard to know for sure if a discovery was genuine or accidentally included in the training data though.