So, I have actually spent a lot of time thinking about these exact problems, but it would have been too overwhelming to try include that information in this initial deck.
I'm not even sure if I can adequately summarize it here :-)
(I'm planning on covering this stuff in a subsequent presentation.)
I played around with a few approaches to the things you're asking. Had good results with specialized interlocked container-type classes (e.g. xlist(); a simplified list-type object) that parallel threads could use to persist simple scalar objects (string, int, bytes).
I think there's definitely room for sharing techniques that exploit the fact that threads inherently share address space -- I don't want to say that shared-nothing is the only paradigm supported and all communication must be done by message passing (like Rust?), because that's not the best solution for all problems.
As for the main-thread/parallel-thread pause/run relationship... it won't be as black and white as I allude to in the deck -- the main thread will still be running, albeit with a limited memory view (i.e. you'll be restricted with what you can do in the main thread whilst parallel threads are running).
Ideally, the only time all the parallel threads get paused is when global state needs to be updated by the main thread. Constantly pausing the parallel threads just because the main thread needs to do some periodic work won't be ideal.
The application you're referring to... is it something that exists, or are you just using hypothetical examples? I'm always curious to hear of architectures where threads need to constantly talk to each other in order for work to get done -- does your app fit this bill?
I'm just talking theory, but there is a whole class of applications that are very difficult to express without any shared state, or state shared by relative expensive message passing.
Re: target problems... the catalyst behind PyParallel can ultimately be tied back to the discussions on python-ideas@ in Sept/Oct 2012 that led to Python 3.4's asyncio.
I wanted to show that, hey, there's a different way you can approach async I/O that, when paired with better kernel I/O primitives, actually allows you to exploit parallelism too (i.e. use all my cores).
I'm particularly interested in problems that are both I/O-bound (or driven) and compute heavy, which is common in the enterprise. The parallel aspect of PyParallel is an area I'm still flushing out (I wanted to get the async stuff working first, and I'm happy with the results). I definitely want to spend the next sprint focusing on using PyParallel for parallel computation problems, where you typically go from sequential execution, fan out to parallel compute, then fan back in to sequential execution. This is common with aggregation-oriented "parallel data" problems.
I'm definitely less familiar with problems that inherently require a lot of cross-talk between threads, like the agglomerative clustering referred to in that paper linked above.
Things like `async.signal_and_wait(object1, object2)` are actually pretty darn useful. Again, it's thanks to the vibrant set of synchronization primitives provided by Windows.
Rust doesn't require all communication to be done by message passing.
One of the big reasons it's effective in Servo is that it can prove that multiple threads working on a shared, mutable data structure are operating in a safe manner.
I'm not even sure if I can adequately summarize it here :-)
(I'm planning on covering this stuff in a subsequent presentation.)
I played around with a few approaches to the things you're asking. Had good results with specialized interlocked container-type classes (e.g. xlist(); a simplified list-type object) that parallel threads could use to persist simple scalar objects (string, int, bytes).
I think there's definitely room for sharing techniques that exploit the fact that threads inherently share address space -- I don't want to say that shared-nothing is the only paradigm supported and all communication must be done by message passing (like Rust?), because that's not the best solution for all problems.
As for the main-thread/parallel-thread pause/run relationship... it won't be as black and white as I allude to in the deck -- the main thread will still be running, albeit with a limited memory view (i.e. you'll be restricted with what you can do in the main thread whilst parallel threads are running).
Ideally, the only time all the parallel threads get paused is when global state needs to be updated by the main thread. Constantly pausing the parallel threads just because the main thread needs to do some periodic work won't be ideal.
The application you're referring to... is it something that exists, or are you just using hypothetical examples? I'm always curious to hear of architectures where threads need to constantly talk to each other in order for work to get done -- does your app fit this bill?