User-installed bundles interfering with build

[Eric Pettersen]

When doing "make install" from top ChimeraX directory:

/Applications/Xcode.app/Contents/Developer/usr/bin/make build
if [ ! -d "/Users/pett/src/chimerax/build/lib/python3.9/site-packages/chimerax" ]; then mkdir -p /Users/pett/src/chimerax/build/lib/python3.9/site-packages/chimerax; fi
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C add_charge build
PYTHONNOUSERSITE=1 ../../../ChimeraX.app/Contents/bin/python3.9 -I -m chimerax.core --nogui --exit --safemode --cmd "devel build . exit true "
Unknown bundle name 'ChimeraX-MouseModes' listed in Initializations section for bundle SEQCROW
Traceback (most recent call last):

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py",
line 197, in _run_module_as_main

return _run_code(code, main_globals, None,

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py",
line 87, in _run_code

exec(code, run_globals)

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/main.py", line 1043, in <module>

exit_code = init(sys.argv)

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/main.py", line 642, in init

toolshed.init(sess.logger, debug=sess.debug,

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/init.py", line 1376, in init

_toolshed = Toolshed(*args, kw)

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/init.py", line 365, in init

self.reload(logger, check_remote=check_remote,

rebuild_cache=rebuild_cache, _session=session)

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/init.py", line 424, in reload

self._installed_bundle_info.load(logger, cache_file=cache_file,

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/installed.py", line 60, in load

self.extend(self._order_bundles(dist_bundle_map, logger))

File

"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/installed.py", line 248, in _order_bundles

explicit_reverse_order.setdefault(dist_key, set()).add(d.key)

UnboundLocalError: local variable 'dist_key' referenced before assignment

UnboundLocalError: local variable 'dist_key' referenced before assignment

File
"/Users/pett/src/chimerax/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-
packages/chimerax/core/toolshed/installed.py", line 248, in _order_bundles
explicit_reverse_order.setdefault(dist_key, set()).add(d.key)

_See log for complete Python traceback._

make[4]: * [wheel] Error 1
make[3]: * [add_charge.build] Error 2
make[2]: * [install] Error 2
make[1]: * [bundles.install] Error 2
make: * [install] Error 2

[Greg Couch]

This should be fixed by modifying the devel install command. We should be explicitly using "devel install user false" in our source tree.

[Greg Couch]

Zach, please take a look at this. Also check if PYTHONNOUSERSITE=1 is still needed.

[Zach Pearson]

Adding "user false" to the command does not help, but this error looks like it happens before we even get into bundle builder. Maybe when we're in safemode we shouldn't look at the user's installed bundles or try to reload the bundle cache. What do you think?

[Greg Couch]

Yes, that's a good idea. That could be part of the safemode behavior. As long as we can still install/update "user" bundles, we're good.

[Zach Pearson]

What's happening is that during main, we're initializing a Toolshed instance but not passing it safe_mode until a couple lines later in the call to ts.bootstrap_bundles()

I added safe_mode to Toolshed's initializer, and was able to build add_charge, but during our build process we have BUILD_AND_INSTALL defined, so when we try to install it we run into the next problem: if we use safe_mode as a flag for whether or not to build the bundle cache, then when we go to install a bundle it's not there for ts.find_bundle.

So we can return None if we're in safe_mode, but then there's nothing to set the install timestamp on. And if we guard that behind safemode, we're back to dist_key referenced before assignment when we go to install it except this time the traceback is in toolshed_utils:

Distribution is in ./dist/ChimeraX_AddCharge-1.5.10-py3-none-any.whl
Installing bundle
Executing: toolshed install ./dist/ChimeraX_AddCharge-1.5.10-py3-none-any.whl userOnly false noDeps true
Errors may have occurred when running pip:
pip standard error:
---

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: /Users/zjp/git/rbvi/ChimeraX/ChimeraX.app/Contents/bin/python3.9 -m pip install --upgrade pip
---
pip standard output:
---
Looking in indexes: https://pypi.org/simple, https://cxtoolshed.rbvi.ucsf.edu/pypi/
Processing ./dist/ChimeraX_AddCharge-1.5.10-py3-none-any.whl
ChimeraX-AddCharge is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.
---
No bundles were installed
Unknown bundle name 'ChimeraX-MouseModes' listed in Initializations section for bundle SEQCROW
Traceback (most recent call last):
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/commands/devel.py", line 174, in _run
    unbound_method(bb, *args, **kw)
  File "/Users/zjp/git/rbvi/ChimeraX/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/bundle_builder/bundle_builder.py", line 154, in make_install
    run(session, cmd)
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/commands/run.py", line 38, in run
    results = command.run(text, log=log, return_json=return_json)
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/commands/cli.py", line 2897, in run
    result = ci.function(session, **kw_args)
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/commands/toolshed.py", line 301, in toolshed_install
    ts.install_bundle(bundles, logger, **kw)
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/toolshed/__init__.py", line 919, in install_bundle
    _install_bundle(self, bundle, logger, per_user=per_user, reinstall=reinstall, session=session, no_deps=no_deps)
  File "/Users/zjp/git/rbvi/ChimeraX/ChimeraX.app/Contents/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/chimerax/toolshed_utils/__init__.py", line 256, in _install_bundle
    changes = toolshed.reload(logger, rebuild_cache=True, report=True)
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/toolshed/__init__.py", line 426, in reload
    self._installed_bundle_info.load(logger, cache_file=cache_file,
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/toolshed/installed.py", line 60, in load
    self.extend(self._order_bundles(dist_bundle_map, logger))
  File "/Users/zjp/git/rbvi/ChimeraX/src/bundles/core/src/toolshed/installed.py", line 248, in _order_bundles
    explicit_reverse_order.setdefault(dist_key, set()).add(d.key)
UnboundLocalError: local variable 'dist_key' referenced before assignment

It's honestly probably simpler to just temporarily move the user site directory before a build and restore it afterward.

[Zach Pearson]

During a programmer's meeting I said I would investigate whether this was still a problem. It is still a problem.