PlaygroundAPI KeyAccount

Contextual Answers (In-prompt)

Pass in text and a question about the text; get an answer based entirely on the content passed in

The in-prompt contextual answers model answer questions based on content passed into the prompt. The response will be limited to the information provided in the prompt. Information gained during model training will not be used in the answer. If the prompt does not provide the correct information to answer the question, the model will not try to infer the answer.

Try it


Links

Example use cases

Example 1: Querying financial reports

Imagine you are performing research and rely on financial reports to base your findings. Let's take the following part from JPMorgan Chase & Co. 2021 annual report:

Document context
In 2020 and 2021, enormous QE — approximately $4.4 trillion, or 18%, of 2021 gross domestic product (GDP) — and enormous fiscal stimulus (which has been and always will be inflationary) — approximately $5 trillion, or 21%, of 2021 GDP — stabilized markets and allowed companies to raise enormous amounts of capital. In addition, this infusion of capital saved many small businesses and put more than $2.5 trillion in the hands of consumers and almost $1 trillion into state and local coffers. These actions led to a rapid decline in unemployment, dropping from 15% to under 4% in 20 months — the magnitude and speed of which were both unprecedented. Additionally, the economy grew 7% in 2021 despite the arrival of the Delta and Omicron variants and the global supply chain shortages, which were largely fueled by the dramatic upswing in consumer spending and the shift in that spend from services to goods. Fortunately, during these two years, vaccines for COVID-19 were also rapidly developed and distributed.
In today's economy, the consumer is in excellent financial shape (on average), with leverage among the lowest on record, excellent mortgage underwriting (even though we've had home price appreciation), plentiful jobs with wage increases and more than $2 trillion in excess savings, mostly due to government stimulus. Most consumers and companies (and states) are still flush with the money generated in 2020 and 2021, with consumer spending over the last several months 12% above pre-COVID-19 levels. (But we must recognize that the account balances in lower-income households, smaller to begin with, are going down faster and that income for those households is not keeping pace with rising inflation.)
Today's economic landscape is completely different from the 2008 financial crisis when the consumer was extraordinarily overleveraged, as was the financial system as a whole — from banks and investment banks to shadow banks, hedge funds, private equity, Fannie Mae and many other entities. In addition, home price appreciation, fed by bad underwriting and leverage in the mortgage system, led to excessive speculation, which was missed by virtually everyone — eventually leading to nearly $1 trillion in actual losses.

Rather than reading the entire report, just ask what you want to know:

  • Question: Did the economy shrink after the Omicron variant arrived?
  • Answer: No, the economy grew 7% in 2021, despite the arrival of the Delta and Omicron variants and the global supply chain shortages.

In addition, you can ask more complex questions, where the answer requires deductions rather than just extracting the correct sentence from the document context. This will result in abstractive, rather than extractive, answers that draw on several different parts of the document. For example, look at the following question:

  • Question: Did COVID-19 eventually help the economy?
  • Answer: The dramatic rise in consumer spending and the shift in that spend from services to goods, along with QE and fiscal stimulus, helped to stimulate the economy.

📘

Do not invent answers

One main concern with LLMs is that they might give an answer that is neither 100% accurate nor grounded in the provided text. The Contextual Answers model is fine-tuned not to return an answer if the answer isn't in the provided context. For instance, if you asked the model a question about the 1929 financial crisis, based on the prompt above, it would not return an answer:

  • Question: How did COVID-19 affect the financial crisis of 1929?
  • Answer: [empty]

Try it out.

Example 2: User help documentation

Pass in relevant content from a user guide and let a user ask questions in natural language, rather than doing keyword search and reading the full documentation.

The next example takes information about managing Wikipedia pages from the Wikipedia docs.

Content
Further information: Wikipedia:Policies and guidelines § Not part of the encyclopedia, and Wikipedia:Be bold § Wikipedia namespace
See also: Wikipedia:User pages § What may I not have in my user pages?
Pages within the "Project namespace" themselves are not part of the encyclopedia proper. These administrative pages are intended for use by editors or by automated tools for the organization and governance of the encyclopedia. Consequently, they do not generally need to conform to the same content protocols or style conventions as articles. Nevertheless, these pages, as with all pages, should be accessible and must comply with Wikipedia's conduct and legal policies.

The project namespace is not a free web host and should not be used as a long-term archive to host pages that look like articles. Articles in the project namespace under construction may be moved to the draft namespace or Userspace draft allowing time for their development and feedback before being moved to Wikipedia's mainspace. Stalled or abandoned drafts and pages in the project namespace that violate policies applicable to non-content pages may be subject to deletion. Editors may not violate copyrights or harass anywhere on Wikipedia. Under the criteria for speedy deletion, these types of pages are subject to expeditious deletion. See below for more information.

Pages within project namespace
The project namespace pages are organized according to their function within the overall project schema. This includes Wikipedia official policies and guidelines, process pages, discussion pages, optional essays, maintenance pages, informative pages, and historical pages.

For lists of pages in the project namespace, see:

Wikipedia:Directories and indexes – a handy list of Wikipedia's directories and indexes.
Wikipedia:Department directory – a list of the different administrative divisions of Wikipedia.
Wikipedia:Editor's index to Wikipedia – an enormous list of the Wikipedia community, intended to help find anything not in the encyclopedia itself.
A main gateway for the project namespace is Community portal. The Community Portal includes categorized links to the more commonly used pages related to the project. A link to the Community Portal page is available in the sidebar.

Policies and guidelines pages
Policies and guidelines pages describe Wikipedia's best practice and clarify principles that are widely accepted by the community and have been through the Wikipedia review process. These pages are marked with the {{policy}}, {{guideline}} or {{MoS guideline}} template. For summaries of key policies, see List of policies. For summaries of key guidelines, see List of guidelines. For summaries of guidelines about Wikipedia's house style, see the Manual of Style contents page. For information on editing policies or guidelines see Edits to policies and guidelines.

Category:Wikipedia policies – contains important rules that are widely accepted and procedures for important processes such as deletion; there are relatively few of those.
Category:Wikipedia guidelines – contains consensual rules-of-thumb that are not strict, but are considered by most editors to be useful most of the time.
Category:Wikipedia Manual of Style – contains style guidelines widely accepted among editors.
Process pages
Process pages help facilitate application of the policies and guidelines governing all Wikipedia pages (e.g., Wikipedia:Articles for deletion). See Wikipedia processes for details.

Category:Wikipedia processes
Information and discussions
Many pages in the Wikipedia namespace have nothing to do with rules or implementation of those rules, and thus do not belong in the above categories. See Wikipedia:The difference between policies, guidelines and essays for more information.

Discussion and noticeboard pages
Some pages are designed for discussions, such as the Village pumps. Others serve as noticeboards to draw attention to discussions taking place elsewhere, such as the centralized discussions page and various requests for comments pages. While many of these pages operate like talk pages, some of them have posting rules (at the top of the page) which may be administratively enforced, and they also have their own talk pages, usually about the management of the noticeboard or other process in question. To ask a question or make a request, see the request directory. For a listing of ongoing discussions and current requests, see the dashboard.

Category:Wikipedia discussion
Category:Wikipedia noticeboards
Category:Wikipedia requests
Maintenance pages
Maintenance pages are used to help facilitate the organization and clean-up of articles to bring them up to encyclopedic standards. These editor- and bot-generated pages typically contain articles and other pages requiring maintenance or attention. (e.g., Wikipedia:Dusty articles). The Special namespace also contains bot generated maintenance reports (e.g. Special:UncategorizedPages). See Wikipedia maintenance for details.

Category:Wikipedia maintenance
Special:SpecialPages
How-to and information pages
Informative and instructional pages are typically edited by the community; while not policies or guidelines themselves, they are intended to supplement or clarify Wikipedia guidelines, policies, or other Wikipedia processes and practices that are communal norms. Where essay pages offer advice or opinions through viewpoints, information pages should supplement or clarify technical or factual information about Wikipedia impartially. In comparison to policies and guidelines, information pages, like essay pages, have a limited status, and can reflect varying levels of consensus and vetting. These pages are typically marked with the {{Information page}}, {{Wikipedia how-to}}, or {{Supplement}}[a] template.

There is a large amount of overlap between the Help namespace (which provides mainly technical information) and the Wikipedia namespace (which provides mainly procedural information and interpretation). For this reason, redirects and hatnotes are often set up between these two namespaces. See Help:About help pages for more information. For a listing of how-to and information pages, see the help directory.

Category:Wikipedia information pages
Category:Wikipedia how-to
Category:Wikipedia supplemental pages
Essay pages
Essays about Wikipedia may be written by anyone; some represent widespread norms, others only represent minority viewpoints. Essays can be long monologues or short theses , serious or funny, informative or opinionated. Essays like information pages have no official status, and do not speak for the Wikipedia community as they may be created and edited without overall community oversight. Essays are typically marked with one of the various {{essay}} templates. See Wikipedia:Essays for more information. For a listing of essays, see the essay directory.

Category:Wikipedia essays
WikiProject pages
Pages of a WikiProject are the central place for editor collaboration and organization on a particular topic area. Many WikiProjects compose "advice essays" about how to apply Wikipedia's policies and guidelines to their specific subject area. Pages with Wikipedia:WikiProject prefix form a WikiProject pseudo-namespace. These pages are typically marked with the {{WikiProject status}} or {{WikiProject advice}} template. For a listing of projects, see the directory of WikiProjects. For additional resources, or if you have any questions, please visit the WikiProject Council.

Category:WikiProjects
Historical pages
See also: Wikipedia:Policies and guidelines § historical
A historical page or process is one which is no longer in use, or is no longer relevant or consensus has changed about its content. They are kept as records of past Wikipedia processes to give context to historical discussions and to inform future discussions on similar topics. These pages are typically marked with the {{historical}}, {{superseded}} or {{dormant}} templates. See HISTORICAL for more details.

Category:Wikipedia history
Wikipedia:Historical archive
Wikipedia:History of Wikipedian processes and people
Creating new project pages
Further information: Wikipedia:Avoid writing redundant essays and Wikipedia:Content forking/Internal § Policy forks
See also: Wikipedia:User pages § What may I have in my user pages?, and Wikipedia:Drafts § Creating and editing drafts
Essays and information pages may be established by writing them and adding {{Essay}}, {{Information page}}, {{Wikipedia how-to}}, or similar templates to the page. Essays and information pages in the "Wikipedia namespace" should not be used to create an alternative rule set. Creation of new guideline and policy pages require discussion and a high level of consensus from the entire community before publication (see WP:PROPOSAL for procedural details). For general recommendations about the creation and improvement of existing guideline and policy pages, see Wikipedia:Policy writing is hard.

The project namespace contains many pages and a lot of information. Try to avoid creating new pages unnecessarily. Before creating a new page, you should consider looking through the topical index, essay directory and help directory to see if the material belongs at a page that already exists. It is strongly recommended that you propose a WikiProject at the proposal page, if there are enough willing editors to participate in the project. Already existing WikiProjects can be found in the directory of WikiProjects.

Deletion of project pages
Main page: Wikipedia:Miscellany for deletion
See also: Wikipedia:User pages § Deletion of user pages, and Wikipedia:Drafts § Deleting a draft
Iif editing can improve the page, this should be done rather than deleting the page. Disputes over page content are usually not dealt with by deleting the page, except in severe cases. The content issues should be discussed at the relevant talk page, and other methods of dispute resolution should be used first, such as listing on "requests for comments" for further input.

The usual deletion process is to post a request at miscellany for deletion to have project pages deleted. Make sure that a notice of the request is posted to the talk page of the affected project. If the project is under a parent WikiProject, a notice should be posted there as well. Essays, information pages, and other informal pages that are only supported by a small minority of the community may be moved to the primary author's userspace over deletion. The Wikipedia community has historically tolerated a wide range of Wikipedia related subjects and viewpoints on user pages.

Policies, guidelines and process pages should not be nominated for deletion, as such nominations will probably be considered disruptive, and the ensuing discussions closed early. The {{Disputed tag}} template is typically used for claims that an essay, WikiProject advice or information page was recently assigned guideline or policy status without proper or sufficient consensus being established. See WP:Local consensus for more details.

It is generally preferable that inactive WikiProjects not be deleted, but instead be marked as {{WikiProject status|inactive}}, or redirected to a relevant WikiProject, or changed to a task force of a parent WikiProject, unless the WikiProject was incompletely created or is entirely undesirable. See WP:INACTIVEWP for more details.

If a project page clearly satisfies a "general" speedy deletion criterion, it should be tagged with the appropriate template. For issues only affecting specific revisions on a page (where other page versions are fine) revision delete is usually more appropriate. To request revision-deletion for copyright violations, see {{Copyvio-revdel}} for details.

For a complete listing of past and current project namespace deletion requests at MfD, see all MfD pages with Wikipedia prefix.

Let's ask a common user question in natural language:

  • Question: How do I delete my wiki page?
  • Answer: To delete a page, first post a request at miscellany for deletion to have project pages deleted. Make sure that a notice of the request is posted to the talk page of the affected project. If the project is under a parent WikiProject, a notice should be posted there as well.

Improving your results


Improving the Question and Answers

Both RAG Engine and Contextual Answers expect a single, concise question. These questions should focus on surfacing answers directly from the document, as opposed to reasoning based on clues in the document. Follow these guidelines:

  • Queries should be a single question only; avoid compound questions or giving further instructions.

  • Just ask a question, do not use prompt engineering techniques such as system prompts or explaining importance of the problem. This will not help Contextual Answers.

  • The question should be unambiguously answerable. Word questions to clearly specify what is desired. For example, if your document corpus covers quarterly reports for the past 10 years, then "What was the earnings in Q1 2023?" is much clearer than "What was the earnings last quarter?"

  • If you wish to have answers be longer or shorter, use the answerLength = “short” / “medium” / “long” parameter.

Benchmarking your answers

When evaluating Contextual Answers (as when evaluating any model) the evaluation process is crucial for assessing the performance. Follow these general steps to refine your evaluation methods:

Create a Test Set: Begin with 10 or more questions, each with their own contexts and "golden answers." This helps in establishing a baseline for measuring improvements. We recommend having a diverse set of question types.

Ensure Accuracy of your Test Set: Verify that the gold answers are correct and indeed contained within the given context. While you can use answers that are sourced from other large language models, it is essential to ensure that the responses are actually accurate and correct and are contained within the context.

Comprehensive Evaluation: Evaluate not only the True Positive instances (correct answers) but also consider True Negative instances (correctly identified as "Answer not in doc"). This ensures a balanced evaluation of the model.

Evaluation Correctness of the response: Evaluation can be done either manually or automatically (for example, by using an LLM). However, if an LLM is used, care should be taken to avoid biases in evaluation, since LLMs generally prefer responses an LLM of a similar type. It is recommended that human evaluation be used either entirely or at least to evaluate the LLM classification of correctness of the Contextual Answers response.

Updated 24 days ago