id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	blockedby	blocking	notify_on_close	platform	project
803	"Multiprocessing ""spawn"" method does not work"	Tristan Croll	tic20@…	"(No idea who best to assign this to, so giving it to Tom for now).

I'm exploring the multiprocessing module as a way to get some parallelism into ISOLDE - specifically, it looks like it should be quite straightforward to get the OpenMM simulation running in a subprocess so I can keep it out of the graphics loop, only updating coordinates when they're ready. There are two general methods for generating the subprocess: 'fork' (just create an exact copy of the current Python process in memory) and 'spawn' (start a new Python instance to communicate with via pipes). The 'fork' method works straightforwardly - but is only available on Unix machines and each subprocess carries all the memory overhead of whatever's loaded in ChimeraX. Both good reasons to avoid it. 

The 'spawn' method is more universal and lightweight but is currently broken for ChimeraX - but can be made to work quite straightforwardly. The new processes it attempts to start crash with the unrecognised flag '-s' (and 'E' is also there and would probably cause trouble as well). These arise in `multiprocessing.spawn.get_command_line()` lines 88-89, where it calls `subprocess._args_from_interpreter_flags()` and adds the results to the executable call, thinking it's calling Python directly rather than through the ChimeraX wrapper. I guess it would be simple enough to have ChimeraX ignore those flags, or alternatively replace the offending lines in spawn.py:

{{{
        #opts = util._args_from_interpreter_flags()
        #return [_python_exe] + opts + ['-c', prog, '--multiprocessing-fork']
        return [_python_exe] + ['-c', prog, '--multiprocessing-fork']
}}}

Once that's done, the following toy example works as expected:
{{{
import multiprocessing as mp
from multiprocessing import sharedctypes
from math import ceil

# If using the spawn method, foo needs to be defined somewhere that 
# ChimeraX knows where to find and re-import it. 
def foo(bfactors,start,end,out):
    for i in range(1000):
        c = bfactors.copy()
        c.sort()
    out[start:end] = bfactors
    

def run_multiproc_test(atoms, nproc):
    
    # Have to explicitly re-import foo here
    from chimerax.isolde.multiproc_test import foo
    ctx = mp.get_context('spawn')
    import numpy
    natoms = len(atoms)
    arr = sharedctypes.Array('d', natoms, lock = False)
    ret = numpy.empty(natoms, numpy.float32)
    stride = int(ceil(natoms/nproc))
    proclist = []
    bfactors = atoms.bfactors
    for i in range(nproc):
        start = stride*i
        end = start+stride
        if end > natoms:
            end = natoms
        p = ctx.Process(target=foo, args=(bfactors[start:end],start,end,arr))
        proclist.append(p)
        p.start()
    
    for p in proclist:
        p.join()
    ret[:] = arr
    return ret

}}}

... except that I get the following spammed to the terminal window:

{{{
Tool ""File History"" failed to start
Traceback (most recent call last):
  File ""/home/tic20/apps/chimerax/lib/python3.6/site-packages/chimerax/core/tools.py"", line 422, in start_tools
    bi.start_tool(session, tool_name)
  File ""/home/tic20/apps/chimerax/lib/python3.6/site-packages/chimerax/core/toolshed/info.py"", line 487, in start_tool
    % tool_name)
chimerax.core.toolshed.ToolshedError: tool ""File History"" is not supported without a GUI
Tool ""Density Map Toolbar"" failed to start
Traceback (most recent call last):
  File ""/home/tic20/apps/chimerax/lib/python3.6/site-packages/chimerax/core/tools.py"", line 422, in start_tools
    bi.start_tool(session, tool_name)
  File ""/home/tic20/apps/chimerax/lib/python3.6/site-packages/chimerax/core/toolshed/info.py"", line 487, in start_tool
    % tool_name)
chimerax.core.toolshed.ToolshedError: tool ""Density Map Toolbar"" is not supported without a GUI
}}}

(which also appears when running any other non-gui module such as pip install).

Oh, and if running run_multiproc_test() from the console, you'll need to first do:

{{{
import sys
mod = sys.modules['__main__']
mod.__spec__ = None
}}}

... since for whatever reason IPython's interactive shell DummyMod doesn't have __spec__ defined."	defect	closed	major		Core		fixed		Greg Couch Tom Goddard				all	ChimeraX