Playing nicely with GenAI#

Overview#

You click Search docs and you see an option to Ask Supabase AI:

You ask a question:

Supabase sends the question to the OpenAI Completions API and then displays the answer from OpenAI:

How it works#

The question that you asked was:

how to store images in supabase

But the question that was actually sent to OpenAI follows this pattern:

You are a very enthusiastic Supabase representative who loves to help
people! Given the following sections from the Supabase documentation,
answer the question using only that information, outputted in markdown
format. If you are unsure and the answer is not explicitly written in
the documentation, say "Sorry, I don't know how to help with that."

Context sections:
${contextText}

Question:
${sanitizedQuery}

Answer as markdown (including related code snippets if available):

(We know exactly what prompt Supabase sends to OpenAI because the code is open source. You can view it at …/clippy-search/index.ts.)

• The first paragraph provides instructions to the model about how it should respond.

• The next paragraph contains chunks of relevant content from the docs. ${contextText} is a placeholder that will be replaced with actual docs content before Supabase calls the OpenAI API. This is where generative AI might have a big impact on how we author docs.

• ${sanitizedQuery} is your original query (how to store images in supabase) that has been fed through OpenAI’s [Moderation] API to ensure that the model can consume the query safely.

• The purpose of the last paragraph is the same as the purpose for the first.

“Chunks of relevant content from the docs”#

${contextText} is “chunks of relevant content from the docs”. This is how you keep the model’s answer grounded in facts. How do you determine chunks of relevant content? I am pretty sure that you can use any approach. For example, maybe your docs site is already integrated with Algolia. You should be able to feed the query (how to store images in supabase) into Algolia and then use the content from the top Algolia results as your ${contextText}. However, a lot of people (including Supabase Docs) are using embeddings to find relevant content. It is basically AI-powered search. More on embeddings in the next section.

Here is a key constraint. You might be thinking, “let’s just prepend every question with our entire corpus of documentation.” The first problem is that these models currently can only handle around 16K characters (not words). The second problem is that you are billed based on the size of your inputs. Maxing out your input size on every API call will probably get expensive.

(Here is how to calculate the 16K character limit. Models describes the token limit for each model. gpt-3.5-turbo, the most production-ready model at the moment, has a token limit of 4K. What are tokens and how to count them? says that a token is roughly 4 English characters.)

Embeddings#

I have a hunch that embeddings will have many applications for docs so let’s dig into them a bit.

Embeddings are pretty much statistical representations of text. You don’t actually use embeddings when communicating with the OpenAI API. You only use them to figure out what docs content is related to a query.

In Python you generate an embedding for our favorite query like this:

openai.Embedding.create(
    input=['how to store images in supabase'],
    model='text-embedding-ada-002'
)

OpenAI returns an array of numbers like this:

[
    -0.006929283495992422,
    -0.005336422007530928,
    ...
    -4.547132266452536e-05,
    -0.024047505110502243
]

Here’s another clincher. In order for this to be useful you also must have embeddings for all of your docs content. In Sphinx this was actually pretty trivial to set up. This post is getting long and I am tired so I will save that discussion for another day. Long story short, when you build your docs site, you need to chunk each doc into small, logical sections and then generate an embedding for each section. Once you have that “database” of embeddings for every docs section, it’s pretty easy (for a proficient programmer) to compare the query embedding against each docs section embedding in order to find out what docs sections are most closely related to the query. You then map the docs section embeddings back to the actual docs content and replace ${contextText} with that docs content.

Check out Text Embeddings Visually Explained to build up your intuition about embeddings. It’s a pretty cool rabbithole. Also, it’s such a nice piece of conceptual technical writing!