If you have been following our series on AlphaFold, you are now familiar with the original method used by the DeepMind AI team on trying to resolve one of the most difficult bio-related problems: the protein-folding problem.
The original version of AlphaFold worked as a two-step system, as we discussed in previous articles. One of the major differences with the new version is that the new version uses an end-to-end model instead of gradient descent. The end-to-end approach is based on the attention-based neural network method.
The new AlphaFold2 system still uses multiple sequence alignment (MSA) of evolutionarily-related, similar sequences, and a representation of amino acid residue pairs. However, instead of using the MSA to predict inter-residue distances, DeepMind produced a deep learning architecture that takes the MSA as an input and produces the target protein sequence's final structure. According to the scheme DeepMind shared on their blog and presentation, AlphaFold2 keeps the raw sequences and iteratively “attends” to them.
This is what DeepMind wrote on their abstract for CASP14:
“We found that existing deep-learning architectures overly favor sequence-local interactions and do not sufficiently account for global structural constraints. To remedy this, we have developed a novel, attention-based deep learning architecture to achieve self-consistent structure prediction. We also allow the deep learning algorithm to attend arbitrarily over the full MSA instead of using pairwise co-evolution features like mutual information or pseudolikelihood, allowing the algorithm to ignore irrelevant sequences as well as to extract much richer information from the MSA.”
Figure 1: Overview of the main neural network architecture of AlphaFold2. Photo credit: DeepMind. Source: https://predictioncenter.org/casp14/doc/presentations/2020_12_01_TS_predictor_AlphaFold2.pdf
One of the big changes in AlphaFold2 is that transformers are introduced in this model. Transformers refer to a type of deep learning network architecture normally used in novel Natural Language processing, such as GPT and BERT. In the scheme below of AlphaFold2’s overall architecture, we can distinguish 3 parts: Embedding, Trunk, and Heads.
The first two parts create matrix descriptions about potential interactions and relations between amino acids. In the “Head”, the evolutionary information and predicted pairwise distances are passed onto a structure module that builds XYZ coordinates of the protein structure.
This module uses a 3D equivariant transformer architecture (a SE(3)–equivariant transformer) that updates the backbone coordinates and also builds the side chains.
In their blog, the DeepMind team explains that “by iterating this process, the network develops strong predictions of the underlying physical structure”. This is possible since in each iteration the structure module takes in a spatial configuration, which is currently the best of its kind, and predicts local shifts and rotations of individual parts of the chain.
Additionally, the structure module outputs a confidence score that can predict which parts of each predicted protein structure are reliable.
Figure 2: The task of the structure module is to predict new positions and orientations of protein backbone (red triangles) as well as the confidence score. Photo credit: DeepMind. Source: https://predictioncenter.org/casp14/doc/presentations/2020_12_01_TS_predictor_AlphaFold2.pdf
With the limited information available regarding AlphaFold2, we have drawn a rough comparison between this new version and the old AlphaFold. Now let’s sit tight and wait for DeepMind’s further updates on their new project. While we wait, you can look further into this topic by going back to our previous article on AlphaFold or checking out this article on AI in bio research.