How to make Augmented Reality Videos with ChimeraX

Tom Goddard
July 7, 2020
SBGrid webinar, hosted by Michelle Ottaviano, Pete Meyer and Jason Key

Molecule Show and Tell

A video recording of this webinar is on YouTube.

7-year old at school show and tell: "This is my pet snake!"
Augmented reality using virtual reality.	Augmented reality using laptop webcam. Video.

Why use augmented reality? It is a good way to tell a story.
I put a coronavirus cell entry AR video on YouTube in February to show what ChimeraX VR is about. Viewers ignored the VR aspect but loved the AR, at one point it had 300 likes and 0 dislikes, comments like "This is what 21st century science is supposed to look like", and 28000 views.
AR video can be a good way to tell a molecular story to researchers too.
I'll show two methods for making an AR video in ChimeraX: 1) using VR and depth camera, or 2) using a webcam.

Show remdesivir example live AR demo including drug, virus RNA, EM micrograph, EM map, atomic model, and surface depiction.
All kinds of data viewable in ChimeraX can be shown in AR.
How was this done?
Used VR headset tethered to Windows 10 PC. Any headset that works with SteamVR can be used, e.g. Oculus Rift S, Valve Index, Oculus Rift, Vive, Vive Pro, Windows Mixed Reality ($400 - $1400). I used Vive Pro.
Used Intel RealSense D435 depth camera ($180). Gives color image and distance from camera for every pixel. Distance is used so person appears correctly in front of or behind molecule.
Used Vive tracker ($100). This tells ChimeraX where the RealSense camera is in space and where it is pointed. Vive tracker is not necessary but avoids tedious position recalibration when camera is moved.
ChimeraX command vr on turns on virtual reality.
ChimeraX command realsense on turns on depth camera.

Use Windows for VR. Mac or Linux can be made to work (macOS, linux) with great difficulty and unreliability. Almost all VR is done on Windows so it is the only well-supported operating system.
For general VR use I like Oculus Rift S. Simple with no external base station or power connections. Good image quality. Good price ($400).
For AR if using a Vive tracker, it needs lighthouse base stations used by Valve Index, Vive, Vive Pro headsets. It won't work with Oculus headsets.
VR needs high-power graphics card, cost about $300 and up. I used GTX 1080 Ti.

AR using VR requires expensive and finicky equipment.
I just added to ChimeraX (July 8, 2020) the webcam command that needs only a laptop or desktop web camera to do simple augmented reality. Use ChimeraX daily build to try this.
Tested on MacBook Pro laptops (2012 and 2019) and a Windows laptop and desktop (Logitec C920 camera). May not work with all webcams.
No depth information for captured video, so video is place behind molecules.
A green object (e.g a pen) in the video will appear in front of models, useful as a pointer.
Pointer color can be changed using webcam command. Use command "usage webcam" to list options,

On macOS use QuickTime Player (in /Applications) menu File / New Screen Recording to capture ChimeraX window and audio.
On Windows I use Flashback Express or Flashback Pro 5 for screen capture of ChimeraX window.