In our lab, we often need to run independent python functions on multiple processors. By independent, I mean the functions don’t need to communicate with each other (embarrassingly parallel). Our current scripts divide the individual “tasks” evenly in the beginning and then run each “chunk” (i.e., subset of all tasks) on a separate processor. But occasionally one of these chunks will take much longer to finish than the others, so the processors aren’t all used optimally.
- There needs to be periodic “load balancing.”
- We need a solution that works on windows if possible.
- We’re looking for something written in native python that can be easily incorporated into our projects, without requiring the installation of any third-party modules.