Circadian rhythms are primary drivers for various hormones in your body (e.g. melatonin, body temps, cortisol levels). These rhythms are tightly tied to external/evolutionary triggers. Yours body resets its internal clock* based on external cues, the strongest of which are eating and exposure to daylight. Since we don't have little clocks saying it's 6:05 in our heads, the body uses blue light hitting the retinas as a proxy. It's the knock-on-effects of having misaligned cortisol levels (which further induce lack of sleep / stress) that are the problem, not the blue light.

*The natural rhythm of each individual differs, hence the cave/space experiments where some people naturally fall into a ~25h cycle [due to lack of external cues]).