The previous blog post looked at the basics of using cosine similarity to find matching vectors.
BERT (Bidirectional Encoder Representations from Transformers) was developed by Google and is based on transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection.
In a nutshell it has the knowledge of context and can be used to answer questions using pre-trained models in that language.
It quickly allows us to train sample data such that it can answer queries on that sample data. If the sample data is not massive, then it runs fast even on normal CPUs.
As a quick test, if we created a model from a single paragraph on the life of Einstein:
text = “Albert Einstein was born at Ulm, in Württemberg, Germany, on March 14, 1879. Six weeks later the family moved to Munich, where he later on began his schooling at the Luitpold Gymnasium. Later, they moved to Italy and Albert continued his education at Aarau, Switzerland and in 1896 he entered the Swiss Federal Polytechnic School in Zurich to be trained as a teacher in physics and mathematics. In 1901, the year he gained his diploma, he acquired Swiss citizenship and, as he was unable to find a teaching post, he accepted a position as technical assistant in the Swiss Patent Office. In 1905 he obtained his doctor’s degree.”
And then we load BERT with a pre-trained language model called Squad2. Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.
Then our query is encoded and converted into tokens and then it is queried against the model to look for the best matches (similar to what we did in the previous step of Cosine Similarity). And then instead of just giving the matching sentences, it crafts a relevant answer from the matches that will make sense. Some sample questions and responses:
Q. Where did Einstein study?
Luitpold Gymnasium
==========================
Q. What were his qualifications?
physics and mathematics
==========================
Q. Did he become a doctor?
In 1905 he obtained his doctor ’ s degree
==========================
Q. Did he drink tea?
[CLS]
This is the simplest and fastest way to get a contextual search model without having to use a heavy-duty LLM.
Leave a Reply