python – Multiprocessing a for loop?

python – Multiprocessing a for loop?

You can simply use multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open({}.fits.format(name))
    <process>

if __name__ == __main__:
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool

You can use multiprocessing.Pool:

from multiprocessing import Pool
class Engine(object):
    def __init__(self, parameters):
        self.parameters = parameters
    def __call__(self, filename):
        sci = fits.open(filename + .fits)
        manipulated = manipulate_image(sci, self.parameters)
        return manipulated

try:
    pool = Pool(8) # on 8 processors
    engine = Engine(my_parameters)
    data_outputs = pool.map(engine, data_inputs)
finally: # To make sure processes are closed in the end, even if errors happen
    pool.close()
    pool.join()

python – Multiprocessing a for loop?

Alternatively

with Pool() as pool: 
    pool.map(fits.open, [name + .fits for name in datainput])

Leave a Reply

Your email address will not be published. Required fields are marked *