Indeed, according to [1] a bit more than half of the Linux source is the drivers. But I don't know if those proportions still holds after compilation.

[1] https://unix.stackexchange.com/a/223753