This website serve as an interative tool to visualize transformer model by its latent space. The detail of the method is explained in this paper. Code is available at Here

## Introduction

Word embedding vectors can be factorized into a sparse linear combination of word factors

\begin{align*} % \vspace{-0.2in} \text{apple} =& 0.09\text{dessert}" + 0.11\text{organism}" + \\ & 0.16\text{fruit}" + 0.22\text{mobile&IT}" + 0.42\text{other}". \end{align*}

We view the latent representation of words in a transformer as contextualized word embedding. Similarly, our hypothesis is that a contextualized word embedding vector can also be factorized as a sparse linear superposition of a set of elementary elements, which we call transformer factors.

We feed many sentences into a transformer model. Then we collect the hidden state for each word from each layers. Let $X$ be set of hidden states from different layers, $\forall x \in X$, we assume $x$ is a sparse linear superposition of transformer factors: $$\label{sparse} x = \Phi \alpha + \epsilon, \ s.t. \ \alpha \succeq 0,$$

where $\Phi\in{\rm I\!R}^{d\times m}$ is a dictionary matrix with columns $\Phi_{:,c}\$, $\alpha \in{\rm I\!R}^m$ is a sparse vector of coefficients to be inferred and $\epsilon$ is a vector containing independent Gaussian noise samples, which are assumed to be small relative to $x$. Each column $\Phi_{:,c}\$ of $\Phi$ is a transformer factor and its corresponding sparse coefficient $\alpha_c$ is its activation level.

How the dictionary is learned is explained in the paper. Transformer factors $\Phi_{:,c}$ are divided into three different levels: low level, mid level and high level:

Low level transformer factors correspond to word-level disambiguation.

Mid level transformer factors correspond to sentence-level patterns.

High level transformer factors correspond to long range patterns.

The details on how to divide those transformer factors into three levels is also explained in the paper.

## Visualization

In the following box, input a number $c$ indicating the transformer factor $\Phi_{:,c}$ you want to visualize. Then click the button "" to visualize this transformer factor at a particular layer. For a transformer factor $\Phi_{:,c}$ and for a layer-$l$, the visualization is done by listing the 200 word and context with the largest sparse coefficients $\alpha^{(l)}_c$'s

### ← Enter an integer from 0 to 531, indicating the transformer factor you want to visualize.

Alternatively, you can click the three buttom below to get a random low level, mid level or high level transformer factor:

← (We generate saliency map for high level transformer factor using LIME)

*

This is the "importance score" for this transformer factor. The importance score reflects the average activation of this transformer factor in different layers.