Visualizing Tomograms

Tom Goddard
November 3, 2022
CZI Imaging Institute workshop

• Databases, file standards, community data deposition of segmentations needed to develop better visualization software.
• Promising visualization technologies.
  • Virtual reality for optimal stereoscopic perception of barely visible features.
  • Volumetric lighting to see barely visible features.
  • Interactive segmentation with machine learning.
  • Easier animating of large-scale molecular mechanisms, like Janet Iwasa.

Tomograms and segmentations

   Chikungunya virus assembly and budding visualized in situ using cryogenic electron tomography.
   Chmielewski D, Schmid MF, Simmons G, Jin J, Chiu W.
   Nat Microbiol. 2022 Aug;7(8):1270-1279.

Tomogram of budding Chikungunya virus particles and segmentation. Subtomogram averaging on partially budded viruses, 12 states. 185 tomograms, 1900 budding viruses, 545 nucleocapsid-like particles, 7700 trimer spikes.

Large EM data: Connectomics

   Dense connectomic reconstruction in layer 4 of the somatosensory cortex.
   Motta A, Berning M, Boergens KM, Staffler B, Beining M, Loomba S, Hennig P, Wissler H, Helmstaedter M.
   Science. 2019 Nov 29;366(6469)

• 0.1 millimeter cube of mouse cortex.
• Automated segmentation followed by 4000 hours of student cleanup in 2019.
• 15 million segments, 390,000 synapses, 96 neuron cell bodies (but no full neurons).
• 200 Gbytes (6690 x 9945 x 3420 grid), 11 x 11 x 28 nm voxel size.
• Segmentation available in custom hdf5 files.
• ChimeraX visualization.

Software for visualizing segmentations

• Amira - commercial Thermo Fisher (cost?)
• IMOD
• ArtiaX - subtomogram particle placements
• ChimeraX - quite limited, reads Amira and IMOD files.

Segmentations at the EMDB

• The EMDB has been working on how to handle segmentations, holding 3 workshops:
  • 2011 Data management challenges in three-dimensional EM. Patwardhan et al., 2012;
    
  • 2012 A 3D cellular context for the macromolecular world. Patwardhan et al., 2014;
    
  • 2015 Building bridges between cellular and molecular structural biology. Patwardhan et al., 2017

Prototype EMDB segmentation web viewer

Mock-up of what a volume browser for 3D cellular imaging data could look like, using HIV and simian immunodeficiency virus (SIV) as an example, by integrating data from different imaging modalities. Figure 2 from 2012 workshop "A 3D Cellular Context for the Macromolecular World".

Prototype EMDB segmentation annotation tool

• Assign names and colors to segmented regions.
• Use names from cell component ontologies.

Mock-up of a possible Segmentation-Annotation Tool (SAT). Figure 3 from 2015 workshop "Building bridges between cellular and molecular structural biology."

EMDB Segmentation File Format

• Paul Korir has been implementing EMDB-SFF file format, converters, and annotation editor for about 5 years.
• EMDB / EMPIAR may start accepting segmentation in this format soon.
• Conversions from several formats are implemented:
  • AmiraMesh (.am)
  • Amira HyperSurface (.surf)
  • Segger (.seg)
  • EMDB Map masks (.map)
  • Stereolithography (.stl)
  • IMOD (.mod)

Research community adoption

• Implementation of a standard segmentation file format is probably half (or less) of the battle.
• Getting researchers to routinely deposit segmentations could take another 10 years.
• How can we accelerate adoption?
• Better visualization and analysis software for segmentations is unlikely without standard file formats and research community buy-in to routinely archive segmentations.

How CZI Imaging Institute can help EMDB segmentation effort

• integration into Napari (another CZI product)
• support for more robust file readers
• improve usability
• extend the number of readers supported e.g. Tiff, Imaris, Dragonfly and many others
• deposition tools right from the app (CLI, GUI)"

Questions

• If EMDB accepts segmentations how can the community be encouraged to deposit them?
• If it is not practical to give the EMDB hundreds of segmented tomograms for one project, should just the ones shown in published figures be deposited?
• What are the visualization needs when studying hundreds of tomograms? High through-put visualization.
• Should visualization of large tiled datasets (e.g. connectomics) use its own special purpose software different from visualizing tomograms?

Seeing what is hard to see in 3D images

• Virtual reality - stereoscopic and immersive visualization.
• Lighting for better contrast and boundary perception.
• Machine learning segmentation methods.

Virtual Reality

• Mike Schmid will show examples where virtual reality has been useful for seeing novel structures in tomography in the next talk.
  
• Good stereoscopic perception. Objects in the background don't confound the foreground.
• Natural head movements to find best view points.

Lighting

• Ambient shadow lighting to more readily distinguish densely packed structures.
• Used for atomic models and single-particle EM in ChimeraX for several years.

Mitochondrial ATP synthase dimer atomic model and single-particle map
Shadows from all directions.	No shadows.

• Allen Cell Institute has developed AGAVE for 3D light microscopy.
• Can be done with raytracing to handle transparent 3D images.

	AGAVE 3D microscopy viewer with lighting controls.
3D light microscopy of cells rendered with and without lighting.

Machine learning segmentation methods

• The most progress on seeing objects in tomograms is likely to be from machine learning segmentation methods.
• EMAN2 can segment tomograms to mark ribosomes, filaments, vesicles or other objects.
• Interactively box of positive and negative examples, then train a neural net.
• Can work with just 20 positive examples, and 100 negative examples boxed by hand in 10 minutes.
• Boxes are in 2D (64x64 pixels) and trained neural net operates on 2D planes.
• For positive boxes you also interactively paintbrush the feature of interest.
• Training the network takes about 20 minutes, using TensorFlow.

• Convolutional Neural Networks for Automated Annotation of Cellular Cryo-Electron Tomograms.
  Muyuan Chen, Wei Dai, Stella Y Sun, Darius Jonasch, Cynthia Y He, Michael F Schmid, Wah Chiu & Steven J Ludtke
  Nature Methods volume 14, pages 983-985 (2017)

Animating cell processes

• Chikungunya virus article supplementary video gives a rotating view of segmentation.
• Animations that present hypotheses are harder to make.
  • Phage T4 non-uniform contraction video.
  • Kinesin rotational cargo transport animation made by Janet Iwasa for Erik Schäffer lab article
    Kinesin rotates unidirectionally and generates torque while walking on microtubules (PNAS 2017).
    Does large cargo really rotate?

Chikungunya segmentation.	Phage T4 sheath contraction.	Kinesin motor rotation.