I'm actually a little surprised that the answer wasn't something along the lines of "# is simple straight lines, while B requires a bunch of Bezier splines for TrueType to render." I guess the graphics cards must accelerate that or something. I'm surprised that line-breaking algorithms take that long.
# probably has a decent number of splines in it as well around the corners. It depends on the font, of course, but the Ubuntu font, for example, certainly does.
