Not a biologist, but that kinda makes sense: you only need one single healthy sperm to reach the egg in order to fertilize it, so even men with pretty bad sperm counts are probably still able to have children. So the average sperm quality may drop for some time without a corresponding drop in fertility. But then the drop in fertility will be all the worse...