Working with ipyparallel

In this example we want to use basico from ipython, in a ipyparallel, as the normal approach with using multiprocessing does not seem to work across operating systems within the jupyter notebooks (read: i could not make it work on Windows).

[1]:
from basico import *
import ipyparallel as ipp

we could create a cluster manually, in my case i just created one in the cluster menu in the jupyter setup, so to access it we just need to run:

[2]:
cluster = ipp.Cluster.from_file()
rc = cluster.connect_client_sync()
# wait to get the engines and print their id
rc.wait_for_engines(6); rc.ids
Using existing profile dir: 'C:\\Users\\fbergmann\\.ipython\\profile_default'
Using existing profile dir: 'C:\\Users\\fbergmann\\.ipython\\profile_default'
[2]:
[0, 1, 2, 3, 4, 5]

Now we create a direct view with all of the workers:

[3]:
dview = rc[:]

The model we use is the BioModel 68, since we will need the model many times, i download it once and save it to a local file:

[4]:
m = load_biomodel(68)
save_model('bm69.cps', model=m)
remove_datamodel(m)

And define the worker method, the worker just loads a biomodel (in case it was not loaded into the worker before, otherwise it accesses the current model). It then chooses a random initial concentration for 2 species, and computes the steady state of the model. Finally the initial concentrations used and the fluxes computed are return. In case a seed is specified, it will be used to initialize the rng:

[5]:
def worker_method(seed=None):
    import basico
    import random
    if seed is not None:
        random.seed(seed)
    if basico.get_num_loaded_models() == 0:
        m = basico.load_model('bm68.cps')
    else:
        m = basico.get_current_model()

    # we sample the model as described in Mendes (2009)
    cysteine = 0.3 * 10 ** random.uniform(0, 3)
    adomed = random.uniform(0, 100)

    # set the sampled initial concentration.
    basico.set_species('Cysteine', initial_concentration=cysteine, model=m)
    basico.set_species('S-adenosylmethionine', initial_concentration=adomed, model=m)

    # compute the steady state
    _ = basico.run_steadystate(model=m)

    # retrieve the current flux values
    fluxes = basico.get_reactions(model=m).flux

    # and return as tuple
    return (cysteine, adomed, fluxes[1], fluxes[2])

We can invoke this method synchronusly in the notebook to obtain the result:

[6]:
worker_method()
[6]:
(0.3570060030772281,
 95.66492382480568,
 0.012320973396985456,
 0.9876790266030145)

now we want to do that many times over (passing along the index to set the seed every time to get about the same result on my machine):

[7]:
%%time
sync_results = []
for i in range(2000):
    sync_results.append(worker_method(i))
Wall time: 8.86 s

here a utility method plotting the result:

[8]:
def plot_result(results):
    cys = [x[0] for x in results]
    ado = [x[1] for x in results]
    y1 = [x[2] for x in results]
    y2 = [x[3] for x in results]
    plt.plot(ado, y1, 'x')
    plt.plot(ado, y2, 'o')
    plt.show()
[9]:
plot_result(sync_results)
../_images/notebooks_Working_with_ipyparallel_16_0.png

So that for me took some 10 seconds (even though the model was loaded already), so the next thing to do is to run it in parallel. Here i start 10000 runs, passing along the seed for each one, to make it reproducible:

[10]:
%%time
ar_map = dview.map_async(worker_method, [i for i in range(10000)])
ar_map.wait_for_output()
Wall time: 21.5 s
[10]:
True

And now we can plot that as well:

[11]:
plot_result(ar_map.get())
../_images/notebooks_Working_with_ipyparallel_20_0.png