Semantic Search
A ready-to-use search engine that taps into your organizational knowledge base.
This model is no longer supported. For better results, see Jamba 1.5.
Semantic Search is an advanced querying mechanism that organizations can smoothly integrate into their digital platforms to enhance the relevance of search results. Instead of matching keywords, the outcomes are determined by interpreting the meaning and relationship within the content.
Semantic Search is the retrieval part of AI21's RAG Engine. The RAG Engine contains built-in mechanisms for every step of the way, including document extraction, chunking, embeddings, vector storage, and a retrieval mechanism that fetches the most relevant contexts from your organizational knowledge base.
Organizations can easily embed this model and get an enhanced search experience. They can also use it as part of a larger retrieval augmented generation (RAG) based system to retrieve relevant documents and use them as a prompt for a large language model (such as Jurassic-2). This is also integrated within the full solution of our Contextual Answers, which allows companies instantaneously answer user questions about any organizational data.
As an example, consider a help center with many short articles. Each of those articles was crafted to be precise and on point, and you probably want users to read them as a whole. However, when there are many of them, it's not that easy to find that one specific article to answer the user's question. To take your help center to the next level, let's build a top-notch search experience using these capabilities. Ready to get started?
Step 1: Upload your files
You can upload your files to the RAG Engine, where we offer free storage up to 1 GB (want more? Contact us: [email protected]). You can also integrate your organization’s data sources, such as Google Drive, Amazon S3, and others, to automatically sync documents with RAG Engine. To enable data source integration, contact us
In this example, we will upload the help center articles from Wordtune - our award winning writing assistant help center.
The purpose of this system is to provide a link to the (short) article that contains the answer. Therefore, we will make use of the publicUrl field. This is not mandatory, but for our is case we have to make sure that every document has an associated publicUrl. This can be done with a simple call using our Python SDK or an HTTP request:
from ai21 import AI21Client
client = AI21Client(
# This is the default and can be omitted
api_key=os.environ.get("AI21_API_KEY"),
)
file_id = client.library.files.create(
file_path=file_path,
public_url="<your_url_here>",
)
For more details about the Library operations, see here.
Step 2: Enter a query
To search, all your users need to do is to simply type in their query. It can be a question, a statement, or even just some keywords. For instance, see the following question:
response = client.library.search.create(query="Can I integrate this in my platform?")
A successful query returns the following JSON:
{
"id":"d570bceb-b997-3b0e-a70f-ede0890b89da",
"results":[
{
"text":"Wordtune is now offering its own application programming interface (API), with the same high-quality writing and reading capabilities that you know and love from Wordtune!\n\nWith Wordtune API, developers can integrate Wordtune’s cutting-edge natural language processing (NLP) technology into their own apps and platforms.\n\nCompanies can also use Wordtune API to build AI-powered products that optimize users’ reading and writing tasks.\n\nWordtune API is only accessible through the AI21 Studio website, our platform where businesses and developers can use our models to build NLP-powered apps and services.",
"fileId":"23d79939-0aef-45bb-8c02-484497b64cec",
"fileName":"desktop/folder/What is Wordtune API.txt",
"score":1.43962753,
"publicUrl":"https://support.wordtune.com/en/articles/7063855-what-is-wordtune-api"
},
{
"text":"The Wordtune Google Chrome Extension is compatible with a wide range of sites and platforms, such as Gmail, Google Docs, Slack Web, Facebook and many more. We are constantly working to adapt the Wordtune Extension to new sites and platforms. If there is a particular one you'd find it useful on, reach out to the team by clicking on the messenger in the bottom right hand corner of this page and we'll prioritize it.In the meantime, many of our users use the Wordtune Editor to generate suggestions, and copy and paste the results into their workspace. ",
"fileId":"db10e1c2-c2ba-4fd4-8478-bb76ccc8917b",
"fileName":"desktop/folder/The Wordtune extension doesnt work on a specific page or platform.txt",
"score":1.20192647,
"publicUrl":"https://support.wordtune.com/en/articles/5316564-the-wordtune-extension-doesn-t-work-on-a-specific-page-or-platform"
},
{
"text":"Wordtune is currently available as a Chrome Extension, an Edge Addon and on our online Wordtune Editor available on almost any browser. \n\nThe Wordtune Google Chrome Extension is compatible with most websites such as Gmail, Google Docs, Slack Web, Facebook, Outlook Web, Whatsapp Web, Twitter, TikTok, Reddit, Quora, Discord Web and many more.\nWe are constantly working to adapt the Wordtune Extension to new sites and platforms. If there is a particular one you'd find it useful on, let us know by clicking on the chat widget on this page and we'll prioritize it.In the meantime, many of our users use the Wordtune Editor to generate suggestions, and copy and paste the results into their workspace.",
"fileId":"7d73b0b9-2b05-4be8-bb95-74fc88dcf88e",
"fileName":"desktop/folder/What websites and platforms is Wordtune available on.txt",
"score":1.04121089,
"publicUrl":"https://support.wordtune.com/en/articles/5775553-what-websites-and-platforms-is-wordtune-available-on"
},
{
"text":"Watch the following video to learn how to install the Wordtune extension on Google Chrome: \n\nIn case you are having trouble viewing the video, here is a step-by-step guide:\n\nHow to Install Wordtune on Google Chrome:\n\n\nGo to the Google Chrome store and search for Wordtune. \n\n\nThen you can click “Add to Chrome” and download it.\n\n\nBe sure to pin the Wordtune icon to your browser so it'll be there as an extension wherever you go.\n\n\n\n\n\nTo use the Wordtune extension, start writing and highlight your text. A purple Wordtune icon will appear above your selection. Press the icon to view rewrite suggestions. \n\nIn Gmail, you can apply the Wordtune Extension as follows:\n\n\nWordtune’s widget was designed to help you review and apply our suggestions efficiently. You can use it on as many platforms as you wish.\n\nYou can also take advantage of selecting the tone of your writing. To find out more, visit this page. \n\nAny questions? \nReach out to us by clicking on the messenger icon in the lower-right corner of this screen.\n\n\n\n\n\n\n",
"fileId":"91f7a5c1-0c99-4dd2-96fa-e025b05f9aea",
"fileName":"desktop/folder/How to Install Wordtune Extension on Google Chrome.txt",
"score":1.03771663,
"publicUrl":"https://support.wordtune.com/en/articles/5898694-how-to-install-wordtune-extension-on-google-chrome"
}
]
}
Where the top rated article is indeed the one who contains the answer, about Wordtune API.
A few things here:
- Each document is given a score. This is a similarity score between the document and the query, where a higher score indicates a greater similarity. Despite there being no maximum score (scores can be very large in some cases), you can filter results by the relative differences between documents. In the response above, for instance, you can see that the first and second results differ greatly. The second and last two have a large difference as well.
- By default, the search mechanism returns up to 5 documents (you can adjust this if you want - see the full API reference. However, we only return documents with a score above 1. Hence, in the response above, only four documents were retrieved.
- In our case, we only return the
publicUrl(and can also display relevant text chunks). For a more sophisticated question-answering mechanism, see our Contextual Answers API (which makes use of Semantic Search).
Example - statement query
Consider the following statement as a query:
response = client.library.search.create(query="I want discount")
For our use case, we only want to return the URL of the top result. We can simply return:
print(response.results[0].public_url)
# Returned URL:
https://support.wordtune.com/en/articles/5316519-wordtune-s-30-discount-for-students-educators-and-ngos
The full response in this case will be:
{
"id":"1f9579b4-e4c6-b98c-f3f1-82cb12cc0712",
"results":[
{
"text":"If you are a student or educator with a valid academic email address, or if you work for a non-profit organization, you are eligible for a 30% discount on both our annual and monthly plans. This is for both Wordtune and Wordtune Read premium subscriptions. To apply, please submit your request here. It may take up to 3 working days to process each application. Please note: the discount is available for free users, and for future invoices only.\nWe cannot offer refunds for past discounts, or stack multiple discounts onto each other.If you have any other questions, reach out to us by clicking on the messenger icon in the lower-right corner of this screen.💌",
"fileId":"e0d2f400-5aca-4dfd-9563-beeda9f606d7",
"fileName":"desktop/folder/Wordtunes 30 discount for students educators and NGOs.txt",
"score":1.51325369,
"publicUrl":"https://support.wordtune.com/en/articles/5316519-wordtune-s-30-discount-for-students-educators-and-ngos"
},
{
"text":"\nDiscount for students, educators and Non-Profits\nIf you are a student or educator with a valid .edu email address & are currently enrolled at or working at an academic institution, you are eligible for a 30% discount on both our annual and monthly plans. To apply for the student discount, please visit this page here. It may take up to 3 business days to process this request. For more questions, please contact our support via the messenger in the bottom right hand corner of this screen.\n\nTeams account\nIf you are interested in Wordtune Plus or Unlimited Plans for your business, check out Wordtune for Teams here. \n\nPayment methods\nAt the moment, you can use credit cards, debit cards, and Google Pay to pay for your Wordtune Plus or Unlimited subscription.\n\nRefunds\nWe do not offer refunds but you can cancel your subscription before the next billing cycle by navigating to your account page & clicking on \"Manage Plan\".\n\nTerms of use\nFor our full terms of use, visit www.wordtune.com/terms-of-use\n\n",
"fileId":"115a0bfc-4481-45be-a7f9-067a41d279d5",
"fileName":"desktop/folder/Wordtune Billing policy.txt",
"score":1.38712943,
"publicUrl":"https://support.wordtune.com/en/articles/5310819-wordtune-billing-policy"
},
{
"text":"If you just want to cancel, or end your Premium subscription, you can do so by following the steps here in this article HERE.",
"fileId":"d1ffc472-fe7c-4d04-99e8-45b0bcd1fd18",
"fileName":"desktop/folder/How can I delete my account.txt",
"score":1.12262166,
"publicUrl":"https://support.wordtune.com/en/articles/5859886-how-can-i-delete-my-account"
},
{
"text":"Addressing your concern\nWhen we want to delete our accounts, we want to remove it in a certain way and different people mean different things when requesting to delete their account\n\nYou may be asking to:\n\n\nRemove the extension\n\n\nCancel your subscription\n\n\nDelete everything about your account\n\nWe'll go over each of these options for you, so you can get the answer you need.",
"fileId":"d1ffc472-fe7c-4d04-99e8-45b0bcd1fd18",
"fileName":"desktop/folder/How can I delete my account.txt",
"score":1.05305421,
"publicUrl":"https://support.wordtune.com/en/articles/5859886-how-can-i-delete-my-account"
}
]
}
Again, you can see that there is a large difference between the top results and all the rest.
Example - result not in the docs
If a relevant text does not exist in the documents, the search mechanism will return an empty response instead of irrelevant results. Consider the following query, for example:
response = client.library.search.create(query="are you hiring")
The response will be as follows:
{
"id":"54164cd8-a8a6-3026-9c71-8660bbb8faeb",
"results":[
]
}
Since our documents only comprise of the help center articles, there is no relevant text to retrieve. With regard to the last query, the answer is, of course, yes! Find out more about our career opportunities on our careers page.
Improving your results
-
If you find that Semantic Search is providing irrelevant extracts, use labels and paths to make the search more focused.
-
Modify the max segments parameter to refine the number of search results.
Links
- Playground: Try it in the browser, with all the API parameters.
- API documentation: Learn how to use it via REST or our SDK.
Updated about 2 months ago