"Ok, since we used it for internet scale, the standard memcached isn't good enough for us. So tell me, how would you improve memcached for internet scale?"
A: "Umm..."
Real story.
[The answer is to switch memcached from TCP to UDP implementation, but then you need to know first how memcached work internally and probably intimately]
The problem is that most of these companies ask specific questions regarding the weird libraries or esoteric programming languages that they specifically use to solve a very specific problems.
Might as well go and hire the contributors of those libraries/languages really.
When I first graduated from college, I took an R&D position doing financial modeling. Multiple times a day, I would walk into my boss's office to tell him I had no idea how to implement the next part. He would start to explain and then use a term like "Gaussian distribution" and I'd have to stop him. Down the rabbit hole we went, filling up the white board with graphs and formula. Once we reached the point where I was adept enough to proceed ("oh, ok, so you just add them all.") we would begin to pop things back off the stack until we were back at the original problem.
It was a painful process for me. I'm sure it was even more painful for my boss. After a couple of months, I reached a point where I didn't really need help deciphering statistics.
A few months later, I read through the code of a co-worker who had been there much longer. I noticed some deficiencies in a statistical search method. When I asked my boss why my co-worker implemented all his models using such a poor technique, he told me, "Because he doesn't come into my office and say he has no idea what he's doing."
As a manager, this is the best part of my job. Almost every other bit is drudgery, administration, and distasteful politics.[1]
I love it when an engineer, junior or senior, wanders over and asks a question about something deep or complicated. We can write math or pseudocode on the whiteboard. Even just drawing ``network'' diagrams of things talking to other things helps get me through the day.[2] It's especially rewarding when the lightbulb goes on.
I'm willing to be late to meetings with executives, skip lunch, or stay late at night for these sessions. Don't assume your boss found it painful at all. It's quite likely the best part of his day.
I had the advantage of the same sort of boss at my first real job. In retrospect, he was remarkably patient with me, showing me how to _learn_ the craft of developing software. I had previously been a hack, and he gave me enough knowledge to build my own skillset extremely quickly. (Secret weapons are always handy.)
[1] I went into management because I have an authority problem. Principally, a problem when I perceive ill-informed or untalented authority figures making poor choices that affect me. After a year or two in management, when I learned enough about operations, finance, marketing, etc., I came to the realization that I'm now just in the middle of a pyramid of the same problem. However, I stay because I have priced myself out of the programmer market. I now use the extra money to bootstrap a startup with a friend and save the rest away from the "emergency/retirement" fund so I can burn longer when I do jump ship.
[2] I am not unaware that this is vain and probably more of a status issue than altruism. ``I've still got it,'' and all that.
It wasn't quite that simple. Yes I'd heard of it, but I didn't know how to randomly sample from a Gaussian distribution, nor why it was necessary to use it in the algorithm I was working on ("why can't I just sample from a uniform distribution?").[1]
Also, I was in a special situation. I had a strong passion for the work I was doing, knew my boss before I got hired, and had done related research that showed I was able to push my way through challenging problems.
Sample from a univariate distribution by taking the inverse function of the cumulative distribution and plugging in a [0,1] uniform random number. So, if X is a real valued random variable, then its 'cumulative distribution' F_X(x) = P(X <= x) where P denotes probability. Then F_X is monotone increasing so has an inverse function, say, G(u) = F_X^(-1). Then if U is a random variable uniform on [0,1], G(U) has distribution F_X like X does. Note: Have to patch up a little in the case F_X is not strictly monotone increasing.
For the inverse of the mean zero, variance 1 Gaussian distribution, look in the old NBS AMS-55 math handbook. They have two nice, simple formulas that are plenty good as approximations; one formula is a little more accurate than the other.
For getting samles from a multivariate Gaussian, that is the second lecture!
I'm not in SF but from what I can see, the programming job market is only good these days for "rockstars" willing to work for entry-level salaries. There aren't very many good, high-paying programming jobs these days. Everything is entry level and low pay. This is why I think there are so many startups -- the qualified people are exiting the job market en masse.
The salary part might be true (I have no idea), but there are definitely not many entry level positions - most everyone as far as I've seen wants at least mid-level skillsets.
And if you're not talking about just the bay area specifically, you're definitely off base.
Have to agree here, I just got done hiring a bunch of student assistants, then got hired myself at a new job at a company that is hiring another dozen people over the next year. Every month at our SF Drupal User Groups, there are usually around half a dozen people hiring. No esoteric programming skills needed for any of this, just some work in FOSS, especially on Drupal, could easily land you a job. Not to mention I just had two students graduate who both got good jobs, one at Sybase.
