March 24, 2023, updated July 3, 2023
ChimeraX allows interacting with molecular and cellular 3D data with a virtual reality headset and hand controllers. This is enabled with the ChimeraX vr on command and uses SteamVR and requires Windows (10 or 11) and a high-end VR-capable graphics card.
ChimeraX can send scenes to a Quest 2 or Quest Pro VR headset using menu Tools / General / Send to Quest. This is in ChimeraX daily builds newer than March 22, 2023. Any kind of data shown in ChimeraX can be viewed, and it can be sent from any computer (Mac, Linux, Windows) that can run ChimeraX. It does not require a VR capable computer. It can allow "pass-through" video that shows the room in addition to your molecular and cellular data.
The scenes from ChimeraX are sent as GLTF format files (file suffix .glb). You need to install an app on the Quest to view these files. I did not find an app that could do this so I wrote one called LookSee. It is not available on the Oculus App store currently so it requires a somewhat arduous process to install it from a ".apk" file.
To install Quest apps that do not come from the Meta Oculus app store you use a computer application called SideQuest. Meta makes this especially painful -- you need to enable Oculus developer mode to "side load" applications provided as .apk files. Install the SideQuest "Advanced installer", not the "Easy Installer" because you need it to install applications from ".apk" files. Follow the Advanced Installer tutorial.
The video claims you have to provide a credit card number and phone number to verify the account. I found this not to be true. I did not provide any credit card number (and would not give that to Meta). It was sufficient to enable 2-factor authentication with a cell-phone to enable developer mode. Also there is no need to create a SideQuest account unless you also want to download apps from the SideQuest app store.
Once SideQuest is installed, you run it, connect your Quest headset to the PC by a USB cable and click Allow in the headset so the computer can access the headset. SideQuest will show in the upper left corner that it sees the Quest headset. Then pressing the folder icon labeled with a down arrow among the SideQuest upper right icons allows you to choose the looksee_v7.apk file and installs it on the Quest in a few seconds. Older versions are listed below.
|Quest applications from unknown sources|
Show apps on the Quest using the App Library button (far right of Quest icon bar) and from the menu in the upper right corner of the App Library panel choose "Unknown Sources". That will show the installed LookSee app that you can click on to launch.
To move a model you hold the grip button (on the side) of either hand controller. You need to point toward the center of the model (within at least 90 degrees). If there are multiple models open you can move whichever you point at, and each hand can move a separate model.
Pushing the hand controller thumbstick backward or forward scales a model you are pointing at bigger or smaller. If multiple models are shown they can be zoomed to different scales. Pressing both grip buttons with both controllers pointing at the same model and moving the controllers further or close to each other (pinch gesture) will also scale the model.
The A or X button on the controllers shows user interface panels positioned at the tip of the wand. The panels list the currently open models, show files you can open, allow you to start a multi-person meeting, or set options such as whether pass-through video is enabled. Touch a button with the blue or yellow wand until the button turns blue and than click the trigger button (index finger) on the hand controller. The "x" button in the lower right corner of the panels closes them or pressing the A or X button on the controller closes them.
Pressing the Models button shows currently opened models which can be hidden or reshown using the shown checkbutton or closed with the close button. Pressing the Files button shows a list of the GLTF files saved on the Quest headset that you can open.
The Meeting button shows Start and Join buttons that allow starting a new meeting or joining a meeting started by another person. See the meeting documentation for details.
The Options button shows several settings. You can turn off pass-through video with the Show Room checkbutton. The Show Table button shows a virtual table. The Smooth Edges setting reduces the jagged crawling edges using graphics multisampling. Disabling it can allow showing slightly larger models before the rendering becomes too slow causing flicker. To allow receiving files from ChimeraX enable the "Receive files from ChimeraX" button and use the "Send to Quest" ChimeraX tool as described in the next section. Enabling the "Open new files" checkbutton means that new files transfered to the Quest from ChimeraX or via other means such as using SideQuest will automatically be opened as soon as the files arrive on the headset.
|LookSee||ChimeraX Send to Quest|
To display a scene from ChimeraX on the Quest headset use the ChimeraX Send to Quest tool, menu Tools / General / Send to Quest in ChimeraX daily builds April 12, 2023 or newer (not in ChimeraX 1.6). Enable "Receive files from ChimeraX" in the LookSee user interface on the Quest by clicking the checkbutton. The "Receive files from ChimeraX" text will change to "Receive files at 192.168.0.157" indicating the internet address for the Quest headset. Enter that address (192.168.0.157 in this example) into the ChimeraX Send to Quest panel under Options. After entering the address, the Send to Quest button transfers the current scene shown in ChimeraX to the Quest headset and displays it. This works by ChimeraX creating a GLTF file and sending it to the LookSee application.
Sometimes ChimeraX cannot send a scene file to the Quest headset. This can happen because the Quest headset is on a WiFi network that is not visible to the computer that is running ChimeraX. One solution is to have the computer running ChimeraX use the same WiFi network as the Quest headset. Even if both the computer and the Quest headset are on the same WiFi network sending a file may not be possible. This is often the case on "Guest" WiFi networks that block any communication between devices as a security measure. For example, Send to Quest does not work on the University of California San Francisco Guest WiFi. But it does work on the UCSF Secure WiFi network. Secure WiFi networks are more likely to allow the connected devices to communicate with each other.
There are two other ways to send scenes from ChimeraX to the Quest headset for viewing that are more trouble to setup and use but may help if firewalls block your PC from sending files to the Quest using wifi.
|Windows File Explorer|
Another way to transfer a GLTF scene file to LookSee on Windows is to connect the Quest headset with a USB cable to the PC. In the Quest headset you will need to press the Allow button that appears to allow the PC to write files to the Quest. Then the Windows the File Explorer will show a "Quest 2" or "Quest Pro" entry in the left column. You can then drag and drop *.glb files written by ChimeraX (command "save myscene.glb") to the following Quest directory
/ThisPC/Quest 2/Internal shared storage/Android/data/com.UCSF.LookSee/files
On Mac or Linux there is no file browser access to a connected headset.
A third way to transfer a GLTF scene file from a computer to the Quest headset is to use the Android Debug Bridge command-line program adb. You connect the Quest to a computer (Mac, Linux or Windows) with a USB cable. Then the adb program can copy files from computer to Quest using a command like
adb push myscene.glb /sdcard/Android/data/com.UCSF.LookSee/files
The ChimeraX Send to Quest tool can use adb if desired using the "Use adb" setting under Options. Then pressing the "Send to Quest" button will write the current scene as a GLTF file and use adb to transfer the file. In the LookSee app the user interface has a "Open New Files" checkbutton (default off). If you enable this (touch and click with trigger) then when a new file is sent with Send to Quest it will automatically open. If not enabled, you can reshow the user interface and there will be an Open button that can be pressed to open the new model.
The "adb" program is part of Android Studio SDK Platform Tools. You get adb by downloading the SDK Platform-Tools zip file.
It is also possible for adb to send the file to the Quest by wifi without the usb cable being connected using the "adb tcpip 5555" and "adb connect [quest-ip-address]" commands from a terminal.
|ChimeraX controls to speed up scene rendering|
The Quest 2 and Quest Pro will show the molecular data without using the computer. You can disconnect the headset from the computer if you wish. The Quest headsets run on about 5 watts of power, far less than the 300 watts of a typical PC VR graphics card. So they cannot render complex scenes fast enough. Testing shows the LookSee app can show about 900,000 triangles without flickering if the room is not shown or about 700,00 triangles if the room is shown with pass-through video.
The Send to Quest panel in ChimeraX shows how many triangles the currently displayed scene is using. This will often be too large because ChimeraX uses a 10 million triangle target on laptop and desktop computers. You can reduce the number of triangles used to draw each atom, bond, and ribbon residues with the buttons in the panel. Ribbons have a number of sides (around the circumference) and a number of divisions (along the ribbon axis for one residue). Usually you can vastly reduce the number of triangles while maintaining acceptable appearance. For density maps using step 2 (every other grid point) or showing only a zone around an atomic model (Map toolbar, Zone icon, or "volume zone" command) can reduce the number of triangles.
The Send to Quest app limits the triangles of each scene (if the maximum checkbutton is enabled). But you can open as many scenes as you want and if the total number of triangles is too high, the Quest will start to flicker or lose tracking which can cause nausea and should be avoided. It may even be hard to quit the application (click the System hand controller button and click on the Quit button that appears).
Handle larger scenes. Currently LookSee uses GLTFast to read GLTF files and create Unity graphics. The speed of the graphics shader programs it uses determines how many triangles can be in a scene before flickering occurs. The Unity standard shaders handle about 1.7 times as many triangles as the GLTFast shaders. Bigger scenes could be shown with simpler shaders. The fancier material properties (e.g. metalicity and roughness) offered by slower shaders are less important for molecular data than handling larger scenes.
Multi-person sessions with pass-through. The pass-through video would be especially welcome for multi-person sessions in the same physical room where everyone sees the same models. There is relatively little data to synchronize between multiple Quest headsets, just model positions and scales and hand wand positions. There is an Oculus API called "Spatial Anchors" and a "Shared Spaces" capability that help align the coordinate systems so the models are rendered in the same physical room position for each viewer.
The implementation, debugging, optimization and design ideas are discussed in ChimeraX ticket #8357 and details of transfering files to the headset in ticket #8509.
The version number is reported on the splash screen when the LookSee application is started. Source code for LookSee is on Github.