I heard one of the guests on a recent Lex Fridman podcast remark that a professor friend of his was involved in some research where they proved that, for any given thing, if you optimize it then things improve up to a point, but if you continue to optimize for that one thing endlessly then it always gets worse in the end.
I don’t know the paper, but this sounds like Goodhart’s Law - when you try to optimize a system by optimizing for a proxy for the outcome you want, the proxy measurement will stop being a good proxy because the system will be optimized around the proxy instead of whatever you actually wanted.
Hmm I think it was a more fundamental result than just an eponymous law. It sounded like they proved it mathematically applied to anything. Was hoping one of the HN crowd might know the actual research that was mentioned.
Does anyone know what paper he's referring to??