Sure, excessive rate of change is bad. Even if it's all validated and well-proven.
> If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes
To measure P, I have to make the change.
There's plenty of situations where accumulating enough data to meet a p<0.05 threshold would take months. If my prior is that it's a reasonably good change and has a decent chance to be helpful, and then I take a measurement that makes that roughly 5x more likely... it's OK to push the button. There are many decisions in business that have to be made with much less information or statistical proof than this.
Sure, excessive rate of change is bad. Even if it's all validated and well-proven.
> If your P threshold isn't low enough then you can get stuck in an endless treadmill of making changes
To measure P, I have to make the change.
There's plenty of situations where accumulating enough data to meet a p<0.05 threshold would take months. If my prior is that it's a reasonably good change and has a decent chance to be helpful, and then I take a measurement that makes that roughly 5x more likely... it's OK to push the button. There are many decisions in business that have to be made with much less information or statistical proof than this.