That's an interesting question. I also wonder how you would get designers to start open source projects without getting the opposite of a "developer-centric" project where developers are the second class citizens.
Can you (or others) elaborate a bit more on what a place where a designer could expect to create really great work would look like? Is it more of a personal aesthetic thing (something that is personally interesting)? Is it related to some of the other points in this thread (unfriendly tools, gruff face to the community, lack of control etc.)?
That a tough one - it's always easier to identify a problem than craft a solution. Here's a shot:
* Projects should include screenshots or other visuals (no matter what their state of completion) on project pages. Browsing the usual Github page shows no visuals at all. Not every project is going to be heavily visually oriented, but most of the project pages I see look like a giant "Designers not wanted" sign.
* An environment where design can happen collaboratively. The "design thinking" ethos requires understanding what the problem is before trying to solve it. To really involve designers, some sort of sandbox where ideas can be expressed as something other than code, feature requests, bug fixes or "64x64px.png goes here." Would be ideal - a way to say "I see this working like this..." in a way a visual designer can express. Lots of discussion happens via chat, etc, but pretty hard to access if you're already feeling excluded.
* Something recognizable as a team structure. If projects need a designer, a project page noting an unfilled slot on the team would be ideal, along with some details. Icon designer? UX specialist? Copywriter? What do you need?
* Integrating rapid prototyping tools for design exploration. One of the tough tasks I find in interaction design is to get a feel for how a complex system is going to work without building it. I've see everything from Flash to Filemaker Pro used to build working models, to then be built out in another tool. Photoshop comps are great, but not a great way to simulate a lot of complex interaction models. If designers and developers are going to speak the same language, it's probably not going to be code.
I teach at a university of the visual arts, and I've often wondered about finding a way to engage students in open source projects, but to be honest, I've always been to fearful of what the outcome might be to really push this. My worry is that for a lot of developers, a logo and a few icons is all they are looking for. Fair enough, but not terribly attractive to the types of designers that projects really need.
Finally, designers need to understand what a big deal software is. You almost never see application design (web, desktop or mobile) represented in the big design award annuals, and the types of things that do get in tend to be gimmicky, Flash-driven web sites for consumer products.
Yeah, I know what you mean about solutions. Actually solving the problem at hand is a whole different world than finding the problem.
I know that I've been guilty of not making/updating screenshots very frequently - they tend to be a pain. Something to keep in mind for the future, though.
A collaborative design environment is a hard one. I've found it really useful for code in the past but as you said, that doesn't help a whole lot on the design front. The only practical thing I can think of at the moment is sending links to static images back and forth over chat, but that doesn't get over the barrier to entry on chat.
The only other things I can think of there would be screencasting (which usually costs money and tends to be a little more one-way) and VNC but that requires a lot more trust than I have for strangers (or than I would expect them to have for me).
Team structure might be a hard one. I can't speak for all developers but I generally don't like having a formal team structure if it can be avoided. Do you think that the role part is important or just discrete, specific tasks? If you were thinking of more of a control thing, that might be even harder. Unless I'm the only developer, I generally don't expect to have much control over a specific area. Then again, I also wouldn't expect people to be jerks and completely change something I did without at least talking about it first.
As far as rapid prototypes for web applications go, I LOVE wicket (http://wicket.apache.org/) because your templates are pretty much CSS and HTML (none of that pseudo-code in templates crap) and you have pretty much complete control over your javascript. You can mock up an entire site in HTML and only need minimal changes to get it working with backend code.
I don't have as much experience with desktop applications but I wonder if rough HTML/CSS might also work for the interactive parts. I'd have to think about that one and it might be an excuse to try mocking up a desktop app I've been thinking about.
I don't know if you'd be interested or if it would work, but I'd be up for at least talking about less uncertain ways to get your students involved in open source.
I have a couple of ideas for projects that I'd love to get design help on as long as I'm not completely left out of the design loop. Getting more people (and especially non-coders) involved in open source and helping in education would be bonuses.
Can you (or others) elaborate a bit more on what a place where a designer could expect to create really great work would look like? Is it more of a personal aesthetic thing (something that is personally interesting)? Is it related to some of the other points in this thread (unfriendly tools, gruff face to the community, lack of control etc.)?