You're right that noise comes from the other ear. But I am able to listen at a lower volume when ANC is activated. I often listen while walking near traffic noise, which the ANC is fairly effective at defeating. With ANC — even in just one ear — I can preserve my hearing a bit.
My answer is that ANC helps reducing playback volume to battle with noise so it still worth.