Working with ipyparallel

In this example we want to use basico from ipython, in a ipyparallel, as the normal approach with using multiprocessing does not seem to work across operating systems within the jupyter notebooks (read: i could not make it work on Windows).

from basico import *
import ipyparallel as ipp

we could create a cluster manually, in my case i just created one in the cluster menu in the jupyter setup, so to access it we just need to run:

cluster = ipp.Cluster.from_file()
rc = cluster.connect_client_sync()
# wait to get the engines and print their id
rc.wait_for_engines(6); rc.ids

Using existing profile dir: 'C:\\Users\\fbergmann\\.ipython\\profile_default'
Using existing profile dir: 'C:\\Users\\fbergmann\\.ipython\\profile_default'

[0, 1, 2, 3, 4, 5]

Now we create a direct view with all of the workers:

dview = rc[:]

The model we use is the BioModel 68, since we will need the model many times, i download it once and save it to a local file:

m = load_biomodel(68)
save_model('bm68.cps', model=m)
remove_datamodel(m)

And define the worker method, the worker just loads a biomodel (in case it was not loaded into the worker before, otherwise it accesses the current model). It then chooses a random initial concentration for 2 species, and computes the steady state of the model. Finally the initial concentrations used and the fluxes computed are return. In case a seed is specified, it will be used to initialize the rng:

def worker_method(seed=None):
    import basico
    import random
    if seed is not None:
        random.seed(seed)
    if basico.get_num_loaded_models() == 0:
        m = basico.load_model('bm68.cps')
    else:
        m = basico.get_current_model()

    # we sample the model as described in Mendes (2009)
    cysteine = 0.3 * 10 ** random.uniform(0, 3)
    adomed = random.uniform(0, 100)

    # set the sampled initial concentration.
    basico.set_species('Cysteine', initial_concentration=cysteine, model=m)
    basico.set_species('S-adenosylmethionine', initial_concentration=adomed, model=m)

    # compute the steady state
    _ = basico.run_steadystate(model=m)

    # retrieve the current flux values
    fluxes = basico.get_reactions(model=m).flux

    # and return as tuple
    return (cysteine, adomed, fluxes[1], fluxes[2])

We can invoke this method synchronusly in the notebook to obtain the result:

worker_method()

(0.3570060030772281,
 95.66492382480568,
 0.012320973396985456,
 0.9876790266030145)

now we want to do that many times over (passing along the index to set the seed every time to get about the same result on my machine):

%%time
sync_results = []
for i in range(2000):
    sync_results.append(worker_method(i))

Wall time: 8.86 s

here a utility method plotting the result:

def plot_result(results):
    cys = [x[0] for x in results]
    ado = [x[1] for x in results]
    y1 = [x[2] for x in results]
    y2 = [x[3] for x in results]
    plt.plot(ado, y1, 'x')
    plt.plot(ado, y2, 'o')
    plt.show()

plot_result(sync_results)

So that for me took some 10 seconds (even though the model was loaded already), so the next thing to do is to run it in parallel. Here i start 10000 runs, passing along the seed for each one, to make it reproducible:

%%time
ar_map = dview.map_async(worker_method, [i for i in range(10000)])
ar_map.wait_for_output()

Wall time: 21.5 s

True

And now we can plot that as well:

plot_result(ar_map.get())