February 11, 2023

In my last post, I shared an example of using LangChain, the OpenAI Embeddings API, FAISS vector search, and GPT-3 to create a question-answering AI agent, that answers questions based on the information it has learned from a collection of Cloudflare’s markdown documentation.

I was surprised both at how well it worked, and how little up-front effort was required to go from nothing to something. However, it felt a little too magical. I was left with the feeling that I didn’t really understand what was going on under the hood, and I wanted to dig a little deeper to better understand how it worked.

One way to approach this is to just start logging more aggressively from inside the LangChain classes, but spelunking through the code seemed hard. Coincidentally a few days ago @imjaredz shared his project PromptLayer in the LangChain discord, and it felt like the perfect tool for the job.

PromptLayer records all of the prompts that your code sends to the GPT-3 API, along with all of the responses. Then it presents them in a way that makes it super easy to debug your prompts and make incremental improvements.

So let’s get started!

Installing PromptLayer with LangChain

Installing PromptLayer with LangChain was super easy. The diff ended up being 3 lines long.

The first change was swapping out the from langchain import OpenAI import to from promptlayer.langchain.llms import OpenAI, which records your prompts/results, and sends them to PromptLayer .

Next, I added the the pl_tags argument to the OpenAI call, which tags all of the prompts/responses inside of PromptLayer, making it easy to search for things in the PromptLayer dashboard.

The promptlayer.langchain.llms import OpenAI class reads the PROMPTLAYER_API_KEY value from your exported environment variables, so no additional code was needed to pull that in.

Below I’ve included the pip command, and diff, so you can get a feel for what setup looks like. You can also see it here on github.

pip install promptlayer

diff --git a/qa.py b/qa.py
index 980a798..f4aee69 100644
--- a/qa.py
+++ b/qa.py
@@ -3,7 +3,7 @@ import pickle

import faiss

-from langchain import OpenAI
+from promptlayer.langchain.llms import OpenAI
from langchain.chains import VectorDBQAWithSourcesChain


@@ -26,6 +26,7 @@ chain = VectorDBQAWithSourcesChain.from_llm(
max_tokens=1500,
model_name="text-davinci-003",
max_retries=3,
+        pl_tags=["cloudflare-qa-agent"],
),
vectorstore=store,
)

Running a Prompt

Once I had PromptLayer installed, it was time to run a prompt and see what happened! I threw it an easy question that I know it could answer, based on the Cloudflare documentation.

python qa.py "Does cloudflare support web3?"

Answer:
Yes, Cloudflare supports web3.

Sources:
cloudflare-md-docs/web3/_index.md
cloudflare-md-docs/web3/ethereum-gateway/_index.md
cloudflare-md-docs/web3/reference/migration-guide.md
cloudflare-md-docs/web3/about.md

Inspecting the Prompt

Immediately after the answer was printed, I noticed that the PromptLayer Dashboard was populated, and displaying all of hte prompts that I threw at it.

The layout is super simple and easy to understand. The sidebar on the left contains a list of all of your prompts arranged by timestamp. You can search based on tags or the contents of the prompts. You can see the entire prompt text, the entire response text, and all of the raw JSON data that was sent to and from GPT-3.

Another neat feature is the share button in the top-right corner of the dashboard. It gives you a link that you can share with collaborators or friends; letting others see what you see.

The Prompts

I’ve included all of the raw text from the prompts and responses below in a collapsable format, but it is actually much easier to read them directly from the PromptLayer share links here:

1. Prompt + Response 1
2. Prompt + Response 2
3. Prompt + Response 3
4. Prompt + Response 4
5. Prompt + Response 5 Prompt 1

Prompt Text:

Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text verbatim.
---
pcx_content_type: concept
title: Ethereum Gateway
weight: 4
---

# Ethereum Gateway

Cloudflare's Ethereum Gateway gives you read and write access to the [Ethereum network](/web3/ethereum-gateway/concepts/ethereum/) without installing any software on your computer.

In particular, users can read all information that has been agreed upon by the consensus of existing nodes in the network. In addition, they can write their own transactions and smart contracts to be stored by these nodes in a distributed manner. Anyone else on the network will be able to view these transactions, and even run your smart contracts using their own supply of the Ethereum currency.

These interactions take place through the official [Ethereum JSON-RPC API](https://github.com/ethereum/execution-apis) and use [Cloudflare-supported API methods](/web3/ethereum-gateway/reference/supported-api-methods/).

## Availability

{{<feature-table id="web3.ethereum">}}
Question: Does cloudflare support web3?
Relevant text, if any:

Response Text:

Cloudflare's Ethereum Gateway gives you read and write access to the [Ethereum network](/web3/ethereum-gateway/concepts/ethereum/) without installing any software on your computer. These interactions take place through the official [Ethereum JSON-RPC API](https://github.com/ethereum/execution-apis) and use [Cloudflare-supported API methods](/web3/ethereum-gateway/reference/supported-api-methods/).

Prompt 2

Prompt Text:

Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text verbatim.
Gives you read and write access to the [Polygon network](/web3/polygon-gateway/) without installing any software on your computer.

{{</feature>}}

---

## Benefits

Cloudflare's Web3 gateways provide HTTP-accessible interfaces to Web3 networks, providing:

- **Ease of access**: Access content from Web3 networks without installing or running any special software.
- **Security**: Get the protection benefits of Cloudflare's global Anycast network for [enhanced security](https://blog.cloudflare.com/cloudflare-thwarts-17-2m-rps-ddos-attack-the-largest-ever-reported/).
- **Reduced maintenance**: Cloudflare — and not your developers — maintains and monitors security, reliability, and performance.
- **Reliability**: Cloudflare's global Anycast network provides a high level of [reliability and availability](https://www.cloudflare.com/network/).
- **Performance**: With Cloudflare's edge network of data centers in over 270 cities, content can be cached and served from data centers close to your end users.

---

## More resources

{{<resource-group>}}

{{<resource header="Plans" href="https://www.cloudflare.com/plans/#overview" icon="documentation-clipboard">}}Compare available Cloudflare plans{{</resource>}}

{{<resource header="Pricing" href="https://dash.cloudflare.com/?to=/:account/:zone/web3/" icon="price">}}Explore pricing options for Web3 Gateways in the dashboard{{</resource>}}

{{</resource-group>}}
Question: Does cloudflare support web3?
Relevant text, if any:

Response Text:

Cloudflare's Web3 gateways provide HTTP-accessible interfaces to Web3 networks, providing:

Prompt 3

Prompt Text:

Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text verbatim.
---
pcx_content_type: reference
title: Legacy gateway migration
weight: 4
---

# Legacy gateway migration

As announced in [our blog post](https://blog.cloudflare.com/ea-web3-gateways/), Cloudflare is deprecating legacy hostnames that point to our public gateway endpoints at `cloudflare-eth.com` and `cloudflare-ipfs.com`.

If you created a hostname pointing to these gateways during the [private beta](https://blog.cloudflare.com/announcing-web3-gateways/), you should migrate to use our new Web3 gateways to avoid a disruption in service.

---

## Migration guide

The migration is a simple process.

First, create a [Cloudflare account](/learning-paths/get-started/#account-setup).

Then create a new [Web3 custom gateway](/web3/how-to/manage-gateways/#create-a-gateway) with your existing hostname.

Alternatively, you could also create a [Web3 custom gateway](/web3/how-to/manage-gateways/#create-a-gateway) for a new hostname and then modify your application to use your newly created hostname ([IPFS](/web3/how-to/use-ipfs-gateway/) or [Ethereum](/web3/how-to/use-ethereum-gateway/)).
Question: Does cloudflare support web3?
Relevant text, if any:

Response Text:

"Cloudflare is deprecating legacy hostnames that point to our public gateway endpoints at `cloudflare-eth.com` and `cloudflare-ipfs.com`...Then create a new [Web3 custom gateway](/web3/how-to/manage-gateways/#create-a-gateway) with your existing hostname."

Prompt 4

Prompt Text:

Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text verbatim.
---
pcx_content_type: concept
title: About
weight: 2
layout: list
---

# How the gateways work

When you [create a gateway](/web3/how-to/manage-gateways/#create-a-gateway), Cloudflare automatically creates and adds specific [DNS records](/web3/reference/gateway-dns-records/) to your Cloudflare account.

When the hostname associated with your gateway receives requests, its DNS records route these requests to a Cloudflare Workers script.

![Cloudflare's Web3 gateways provide HTTP-accessible interfaces to the IPFS and Ethereum networks. For more details, continue reading.](/web3/static/web3-gateway-flow-diagram.png)

## Read operations

If the API call to the Worker is a read operation and the requested content is cached, the Workers script will respond with the requested information via HTTP to the client.

If the requested content is not cached, it will first be requested via API call to Cloudflare IPFS or Ethereum nodes, cached at the edge for future requests, and returned via HTTP response to the client.

## Write operations

*Only available for gateways to EVM-based chains, such as [Ethereum](/web3/how-to/use-ethereum-gateway) and [Polygon](/web3/how-to/use-polygon-gateway).*

If the API call to the gateway is a write operation, the gateway will make an API call to one of the Cloudflare nodes, and the transaction is placed in the local mempool and propagated to peers.
Question: Does cloudflare support web3?
Relevant text, if any:

Response Text:

"When you [create a gateway](/web3/how-to/manage-gateways/#create-a-gateway), Cloudflare automatically creates and adds specific [DNS records](/web3/reference/gateway-dns-records/) to your Cloudflare account."

Prompt 5

Prompt Text:

Given the following extracted parts of a long document and a question, create a final answer with references ("SOURCES").
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
ALWAYS return a "SOURCES" part in your answer.

QUESTION: Which state/country's law governs the interpretation of the contract?
=========
Content: This Agreement is governed by English law and the parties submit to the exclusive jurisdiction of the English courts in  relation to any dispute (contractual or non-contractual) concerning this Agreement save that either party may apply to any court for an  injunction or other relief to protect its Intellectual Property Rights.
Source: 28-pl
Content: No Waiver. Failure or delay in exercising any right or remedy under this Agreement shall not constitute a waiver of such (or any other)  right or remedy.

11.7 Severability. The invalidity, illegality or unenforceability of any term (or part of a term) of this Agreement shall not affect the continuation  in force of the remainder of the term (if any) and this Agreement.

11.8 No Agency. Except as expressly stated otherwise, nothing in this Agreement shall create an agency, partnership or joint venture of any  kind between the parties.

11.9 No Third-Party Beneficiaries.
Source: 30-pl
Content: (b) if Google believes, in good faith, that the Distributor has violated or caused Google to violate any Anti-Bribery Laws (as  defined in Clause 8.5) or that such a violation is reasonably likely to occur,
Source: 4-pl
=========
FINAL ANSWER: This Agreement is governed by English law.
SOURCES: 28-pl

QUESTION: What did the president say about Michael Jackson?
=========
Content: Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.

Last year COVID-19 kept us apart. This year we are finally together again.

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.

With a duty to one another to the American people to the Constitution.

And with an unwavering resolve that freedom will always triumph over tyranny.

Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated.

He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined.

He met the Ukrainian people.

From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.

Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland.
Source: 0-pl
Content: And we won’t stop.

We have lost so much to COVID-19. Time with one another. And worst of all, so much loss of life.

Let’s use this moment to reset. Let’s stop looking at COVID-19 as a partisan dividing line and see it for what it is: A God-awful disease.

Let’s stop seeing each other as enemies, and start seeing each other for who we really are: Fellow Americans.

We can’t change how divided we’ve been. But we can change how we move forward—on COVID-19 and other issues we must face together.

I recently visited the New York City Police Department days after the funerals of Officer Wilbert Mora and his partner, Officer Jason Rivera.

They were responding to a 9-1-1 call when a man shot and killed them with a stolen gun.

Officer Mora was 27 years old.

Officer Rivera was 22.

Both Dominican Americans who’d grown up on the same streets they later chose to patrol as police officers.

I spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves.
Source: 24-pl
Content: And a proud Ukrainian people, who have known 30 years  of independence, have repeatedly shown that they will not tolerate anyone who tries to take their country backwards.

To all Americans, I will be honest with you, as I’ve always promised. A Russian dictator, invading a foreign country, has costs around the world.

And I’m taking robust action to make sure the pain of our sanctions  is targeted at Russia’s economy. And I will use every tool at our disposal to protect American businesses and consumers.

Tonight, I can announce that the United States has worked with 30 other countries to release 60 Million barrels of oil from reserves around the world.

America will lead that effort, releasing 30 Million barrels from our own Strategic Petroleum Reserve. And we stand ready to do more if necessary, unified with our allies.

These steps will help blunt gas prices here at home. And I know the news about what’s happening can seem alarming.

But I want you to know that we are going to be okay.
Source: 5-pl
Content: More support for patients and families.

To get there, I call on Congress to fund ARPA-H, the Advanced Research Projects Agency for Health.

It’s based on DARPA—the Defense Department project that led to the Internet, GPS, and so much more.

ARPA-H will have a singular purpose—to drive breakthroughs in cancer, Alzheimer’s, diabetes, and more.

A unity agenda for the nation.

We can do this.

My fellow Americans—tonight , we have gathered in a sacred space—the citadel of our democracy.

In this Capitol, generation after generation, Americans have debated great questions amid great strife, and have done great things.

We have fought for freedom, expanded liberty, defeated totalitarianism and terror.

And built the strongest, freest, and most prosperous nation the world has ever known.

Now is the hour.

Our moment of responsibility.

Our test of resolve and conscience, of history itself.

It is in this moment that our character is formed. Our purpose is found. Our future is forged.

Well I know this nation.
Source: 34-pl
=========
FINAL ANSWER: The president did not mention Michael Jackson.
SOURCES:

QUESTION: Does cloudflare support web3?
=========
Content:  Cloudflare's Web3 gateways provide HTTP-accessible interfaces to Web3 networks, providing:
Source: cloudflare-md-docs/web3/_index.md

Content:  Cloudflare's Ethereum Gateway gives you read and write access to the [Ethereum network](/web3/ethereum-gateway/concepts/ethereum/) without installing any software on your computer. These interactions take place through the official [Ethereum JSON-RPC API](https://github.com/ethereum/execution-apis) and use [Cloudflare-supported API methods](/web3/ethereum-gateway/reference/supported-api-methods/).
Source: cloudflare-md-docs/web3/ethereum-gateway/_index.md

Content:  "Cloudflare is deprecating legacy hostnames that point to our public gateway endpoints at `cloudflare-eth.com` and `cloudflare-ipfs.com`...Then create a new [Web3 custom gateway](/web3/how-to/manage-gateways/#create-a-gateway) with your existing hostname."
Source: cloudflare-md-docs/web3/reference/migration-guide.md

Content:
"When you [create a gateway](/web3/how-to/manage-gateways/#create-a-gateway), Cloudflare automatically creates and adds specific [DNS records](/web3/reference/gateway-dns-records/) to your Cloudflare account."
Source: cloudflare-md-docs/web3/about.md
=========
FINAL ANSWER:

Response Text:

Yes, Cloudflare supports web3.
SOURCES: cloudflare-md-docs/web3/_index.md, cloudflare-md-docs/web3/ethereum-gateway/_index.md, cloudflare-md-docs/web3/reference/migration-guide.md, cloudflare-md-docs/web3/about.md

Analysis

While the findings weren’t mindblowing, I did learn some counter-intuitive tidbits that would change how I use LangChain if I were to use it for a production-grade thing.

I didn’t expect LangChain to perform more than 2 API requests to GPT-3, but it turns out it actually made 5 requests. If the vector search had returned more results, I believe it would have made more requests.

The final prompt was surprisingly large, and I wonder if it would be possible to use a custom prompt template that achieves the same result in half the tokens. The 2-shot prompt prefix context was 1,491 tokens long (6080 characters). This prompt template is the default for the VectorDBQAWithSourcesChain chain.

I wish PromptLayer had some additional information visualizing the cost of the prompts and responses. It would be pretty neat if it had a token counter/visualizer sort of like OpenAI’s tokenizer tool, as well as a "summary" that totaled everything up.

I manually counted all of the tokens, so I could estimate how much running this one query costs, and it’s higher than I expected 😅

• Prompt 1: 492 tokens (1811 characters)
• Prompt 2: 462 tokens (1556 characters)
• Prompt 3: 398 tokens (1554 characters)
• Prompt 4: 462 tokens (1720 characters)
• Prompt 5: 1975 tokens (7575 characters)

Total: 3,789 tokens (14,216 characters)

At the time of writing this post, GPT-3’s Davinci model costs $0.0200/ 1K tokens. If I’m doing my math correctly, that means that this single query cost $0.07578 to run.

It’s not going to break the bank for running toys, but if you had one of these LangChain bots go viral on Twitter, it would probably add up very quickly 😬

All-in-all, I’m pretty impressed with LangChain. It’s a very cool tool to have in your tool belt, and as time goes on, it’s only going to get better. I’ve already been impressed by how the community is being run, and how quickly things are being improved. 👏

Over time I’m excited to see how PromptLayer evolves. In my mind, it could be a complete end-to-end observability tool for large language models and potentially even image generation models. Think New Relic/Datadog/Honeycomb for GPT-3 and friends. 🤯