No, that's probably true. That's where the last part of my post applies: this is rarely a conscious application of the Protestant work ethic, and more just the way it has shaped our culture to say that rest for rest's sake, and especially taking time off from work just for that, is Wrong. Specifically, it's Lazy.
Vacation is a normal thing: culturally, we have a strong understanding of it, and an agreement that going to visit another country, or to Disneyworld, or for a camping trip, is what you're supposed to do with paid time off.
So any time you are spending resting is, effectively, sinful.
Most people don't put it that way these days, but the basic idea is still deeply embedded in so much of American culture.