Text Embeddings: The basis of modern NLP

Introduction

All graphics in this article are interactive due to the nature of trying to represent text graphically. To see which sentence each point represents, simply hover the mouse over it. The results obtained have been generated on the “STS benchmark” dataset.

In the world of machine learning it is common to have to work with unstructured data types, that is, data that does not follow any structure or schema. Of these, one of the most important is text, which is the focus of this article.

When we work with text, we have the difficulty of how we can treat it with typical machine learning algorithms and, while some simple algorithms can operate with unstructured data (such as a simple Markov chain), a large part of the machine learning arsenal is not accessible when the text is presented in its original state. 

For this reason, it is common to transform this text into another representation, one that computers can work with more easily. We call this representation an “embedding” and it consists simply of a vector, or matrix, of numerical values. In this way, it is possible to feed text to a machine learning system. As an example of a very famous system that makes use of embeddings to deal with text, we have ChatGPT.

For example, we want to represent the word “sun” as embedding and one way to do it would be to generate it manually: we could generate different features and attribute them a score as embedding, such as it has a 1 in brightness, a 0 in blue color and a 0.9 in size, thus creating the embedding [1,0,0.9]. 

But here we run into a disadvantage: the number of features that can be obtained are almost infinite and not all of them offer the same information. Ideally, we want every element of the embedding to represent as much as possible. In addition, there is the problem of the vocabulary of the text, if all my text talks about stars, this embedding would not serve to differentiate them from each other.

Designing Text Embeddings

With this in mind, it would be desirable to be able to automatically design an embedding that can represent a text and that this representation is adapted to the vocabulary used. 

In the following, we will analyze some of the existing methods to create embeddings and we will see what representations are obtained through them.

TF-IDF method

One of the first methods that appeared to make embeddings is the classic TF-IDF (term frequency – inverse document frequency), which consists of a simple calculation where the text is represented by each unique word, by multiplying the frequency of that word in the text (TF), multiplied by the logarithm of the division of the number of total documents with the number of documents that contain this word. 

This calculation is interesting, since it gives a higher value to words that appear in fewer documents and, the more the word is mentioned in the same document, the more important that word is for that document. 

The focus of this method is the information provided by the word. It is not so much the meaning of the words that matters, but the information they provide. This is due to Zipf’s law, which dictates that vocabulary frequency follows an inverse power-law distribution. 

In this line, the most common words are those that do not add much information, such as “the”, which is the most frequent, while a word such as “approval” has a frequency of 1. The interesting thing about this line is that it exists in every observed natural language, and allows us to estimate the frequency of words, and with that, a measure of “importance”.

This embedding of tf-idf has some interesting features: first, it has conceptually infinite dimensions, there are as many as the size of the observed vocabulary. Also, it is limited to represent text sequences, it would not make much sense if we wanted to apply it to singular words. Still, it is computationally easy and is often used as a base benchmark. If the algorithm cannot do better than tf-idf, it is considered poor.

To show the space that tf-idf originates, we apply on the tf-idf dataset and then do the same with PCA on the embeddings matrix, which generates the graphical representation of this space.

As we can see, tf-idf is limited, although the texts are close to its similar texts as can be seen in the Y-axis, which represents mainly texts about the stock market. These are distributed in an interesting way, forming straight lines. And that is because there is no closeness between words, either a word is the same or it is not (such that cat and dog are as close as airplane and dog); but, conceptually, we do not consider that to be the case.

Word Embedding Method

Due to these limitations with embeddings such as those obtained with tf-idf, more complex embeddings were developed to represent this semantic similarity between words, so that terms belonging to the same “topic” would be similar to each other.

Of these, one of the best known are the GLoVe embeddings, which are embeddings generated only on words. The mathematics of obtaining these embeddings is beyond the scope of this article, but we can take the opportunity to first see how words can be related in this space.

Note that similar words are close to each other, such as the fact that numbers are close to numbers. With this in mind, we could try to create a “semantic” embedding of the above phrases by making use of these word embeddings, for example, using the middle embedding between the word embeddings.

The results no longer have the previous form of two clear straight lines and appear to be poorer. The reason is that we have overlooked another important part of the relationship between words: the order. For example, the phrase “Green men watch dogs” would generate the same embedding as “Green dogs watch men”, but they are clearly not the same.

Sentence Embedding Method

These limitations on embeddings only at the word level led to the development of new types of text embeddings that use word embeddings as a base, but generate a new textual embedding taking into account the order, among other possible factors. 

To show an example we will make use of DistilRoBerTa, a fairly small transformer. The mathematics behind it is beyond the scope of this article, but the important thing is that it generates a bidirectional embedding, that is, the information obtained takes into account the position of each word, including those that precede or succeed it.

We can see that these results are considerably better than the original ones, when we used a simple embedding of tf-idf. And that is because it now encodes the semantic information of the texts, information that is vital to understand the language.

Because the space obtained as embeddings may not be linear, as is the case for the latter, we also added a representation using t-SNE to reduce the dimensionality, so that we can see these relationships between text more appropriately.

In this space we observe several small clusters of highly similar texts, which is close to the reality of the original text distribution.

Conclusion

We have seen what textual embeddings are and what they are for, and we have also analyzed the logic behind them and the space they generate, observing how different embeddings generated different spaces depending on the information they contained: starting with the simplest embedding, embeddings with some semantic logic and embeddings with modern techniques, the same principles that are used for more complex systems.

That’s all! If you found this post interesting, we encourage you to visit the Data Science category to see all related posts and to share it on networks. See you soon!