Skip to content
January 11, 2025

Vector database and conversion of text to vectors for similarity search.

· 5 min read · Containers, GenAI
  • Vector database is a specialized type of database which plays a key role in building Gen AI based applications
  • Traditional database help us to find exact matches, but vector database find items that are ‘like’ the one you’re searching for.
  • Vectors have dimensionality and a direction. Example velocity can be described by a vector because it has a magnitude, namely speed, as well as a direction. Joseph is heading north at 70 mph in his car.
  • One of the crucial steps in many machine learning and natural language processing (NLP) tasks is converting text into vectors. In this process, the text data, which is inherently non-numeric, is converted into numerical form so that computers can process and analyze it
  • Here’s a simple breakdown of how this transformation happens:
  • The first step is Tokenization, where the text is split into smaller units, typically words or subwords. In some cases, characters may also be treated as tokens.
  • For the given text: “Mardi gras is almost here”
  • After tokenization tokens: [“Mardi”, “gras”, “is”, “almost”, “here”]
  • The second step is Text Representation, where the text needs to be represented as vectors (numbers).
  • There are several techniques to convert tokens into numerical representations. Some common ones include:
  • Bag of Words (BoW): In this technique , the text is represented as a vector where each dimension corresponds to a word in a fixed vocabulary, and the value in that dimension represents the frequency of that word in the text.
  • Example: If the vocabulary is [“Mardi”, “Parade”, “Gras”], and the sentence is “Mardi gras is almost here,” the vector would be: Vector: [1, 0, 1]
  • 1 for “Mardi” (appears once), 0 for “Parade” (doesn’t appear) and 1 for “Gras” (appears once)
  • The main disadvantage of BoW is that it ignores the word order and context.
  • One more technique is TF-IDF (Term Frequency-Inverse Document Frequency)
  • TF-IDF improves on BoW by giving more importance to words that are unique to a document and less to words that appear frequently in many documents (common words like “the” or “is”).
  • TF: Measures how frequently a word appears in a document.
  • IDF: Measures how rare or common a word is across all documents.
  • For example, in a collection of documents, the word “Mardi” and “Gras” might have a higher score in one document if it appears often in that document, but not in many others.
  • There are other more advanced techniques that map words to vectors of continuous values in a high-dimensional space. The key idea is that words with similar meanings should be represented by vectors that are close to each other in this space.
  • Word2Vec and GloVe are popular methods for generating word embeddings. These models learn word representations based on the context in which words appear, meaning that words used in similar contexts end up having similar vector representations.
  • Example: The word “bank” would have a different vector representation in the sentences “I went to the bank to fish” and “I deposited money in the bank.”
  • The third step is Vector Representation, once the text is processed and converted into tokens, the method chosen (BoW, TF-IDF, Word2Vec, etc.) is used to generate vectors.
  • These vectors now serve as numerical representations of the text, allowing the computer to perform mathematical operations
  • For example: Word2Vec might turn the word “Mardi” into a vector: [0.3, -0.1, 0.8]
  • Sentence embeddings (e.g., from BERT) might turn the sentence “Mardi gras is almost here” into a vector like: [0.1, -0.5, 0.7, …]
  • The 4th step in the process is Usage, meaning using these numerical representations for various tasks:
  • Similarity: You can calculate how similar two pieces of text are by comparing their vectors (e.g., using cosine similarity).
  • Classification: You can feed the vectors into machine learning models for tasks like sentiment analysis, topic classification, etc.
  • Search and Retrieval: Search engines often use vector representations of text to return the most relevant results based on similarity.
  • Once the text is represented as vectors, If the use case is to find 2 similar vectors, several mathematical formulas can be used to determine if two vectors are similar.
  • One such formula or technique is cosine similarity.
  • Vectors are considered similar when they point in nearly the same direction, as shown in the first diagram. They are considered unrelated when they are perpendicular to each other, and opposite when they point in completely opposite directions
  • The formula for cosine similarity is:
  • where A is the vector for Sentence 1 , B is the vector for Sentence 2 , . is the dot product of the vectors and || A || and || B || are the magnitudes (or lengths) of the vectors
  • By calculating this similarity, you can quantify how close the two sentences are in meaning. A high cosine similarity (closer to 1) means the sentences are very similar in meaning, while a low cosine similarity (closer to 0) means they are quite different
  • Let’s say we have two sentences: Sentence 1 – Mardi gras is almost here.” and Sentence 2 – “Carnival celebrations are coming soon.”
  • After tokenization and conversion into vectors (using methods like Word2Vec or BERT), each sentence would be represented by a vector, such as
  • Sentence 1 Vector: [0.55, -0.12, 0.78, 0.34] and Sentence 2 Vector: [0.52, -0.14, 0.80, 0.30]
  • These vectors represent the numerical encoding of the sentences.
  • The values in the vectors correspond to the semantic meaning of each word, and the entire sentence is represented as a point in a multi-dimensional space
  • In this case, the two sentences are about similar events—Mardi Gras and Carnival celebrations—so the cosine similarity value will be relatively high (indicating the sentences are quite similar).
  • Even though the sentences use different words, they convey similar meanings, which will be captured by their vectors and reflected in the cosine similarity score.

A vector database stores high-dimensional vectors, which are numerical representations of data such as text, enabling efficient similarity search. By converting text into vectors (using methods like word embeddings), these databases allow for finding semantically similar items, even if they don’t share exact keywords. To dive deeper into advanced topics, you can explore techniques like fine-tuning embeddings, optimizing search algorithms, and integrating vector databases with machine learning models for more complex applications.



Tags: , , ,

Leave a Reply