The U.S. Department of Energy’s (DOE) Argonne National Laboratory will be home to one of the nation’s first exascale supercomputers when Aurora arrives in 2022. With access to pre-production hardware and software, these researchers are among the first in the world to utilise exascale technologies for science.
Using far more advanced imaging techniques than those of their earlier contemporaries, the researchers are working to develop a brain connectome, an accurate map that lays out every connection between every neuron and the precise location of the associated dendrites, axons and synapses that help form the communications or signalling pathways of a brain.
The project is a wide-ranging collaboration between computer scientists and neuroscientists, and academic and corporate research institutions. The research was impossible to be done until the advancement of ultra-high-resolution imaging techniques and more powerful supercomputing resources. These technologies allow for finer resolution of microscopic anatomy and the ability to wrangle the sheer size of the data.
Only the computing power of an Aurora, an exascale machine capable of performing a billion billion calculations per second, will meet the near-term challenges in brain mapping. Currently, without that power, the team is working on smaller brain samples, some of them only one cubic millimetre. Even this small mass of neurological matter can generate a petabyte of data. With the goal of one day mapping a whole mouse brain, about a centimetre cubed, the amount of data would increase by a thousandfold at a reasonable resolution.
If we do not improve today’s technology, the compute time for a whole mouse brain would be something like 1,000,000 days of work on current supercomputers. Using all of Aurora, if everything worked beautifully, it could still take 1,000 days. So, the problem of reconstructing a brain connectome requires exascale resources and beyond.
– Argonne Senior Computer Scientist
Working primarily with mouse brain samples, the team is developing a computational pipeline to analyse the data obtained from a complicated process of staining, slicing and imaging. The process begins with samples of brain tissue which are stained with heavy metals to provide visual contrast and then sliced extremely thin with a precision cutting tool called an ultramicrotome. These slices are mounted for imaging with Argonne’s massive-data-producing electron microscope, generating a collection of smaller images, or tiles.
The resulting tiles have to be digitally reassembled, or stitched together, to reconstruct the slice. Each of those slices has to be stacked and aligned properly to reproduce the 3D volume. At this point, neurons are traced through the 3D volume by a process known as segmentation to identify neuron shape and synaptic connectivity.
This segmentation step relies on an Artificial Intelligence technique called a convolutional neural network; in this case, a type of network developed by Google for the reconstruction of neural circuits from electron microscopy images of the brain. While it has demonstrated better performance than past approaches, the technique also comes with a high computational cost when applied to large volumes.
Using supercomputers for this work demands efficiency at every scale, from distributing large datasets across the compute nodes, to running algorithms on the individual nodes with high-bandwidth communication, to writing the final results to the parallel file system. Large-scale analysis of the results truly starts to probe questions about what emerges from the neurons and their connectivity.
The team’s preparations for exascale will serve as a benefit to other exascale system users. For example, the algorithms they are developing for their electron microscopy data will find applications with X-ray data. With the right tools in place and exascale computing at hand, the development and analysis of large-scale, precision connectomes will help researchers fill the gaps in some age-old questions.