> Technically, the models can already learn on the fly. Just that the knowledge it can learn is limited to the context length.

Isn't that just improving the prompt to the non-learning model?