[5 Tipst]How to Send Longer Text Inputs to ChatGPT API?

The power of OpenAI’s ChatGPT lies in its ability to process and generate human-like text. But what if you have a large piece of text that exceeds the model’s token limit?

This article will guide you through several methods to successfully submit longer text inputs to the ChatGPT API.

Whether you are a seasoned developer or a curious beginner, this guide will provide you with the tools to utilize the ChatGPT API to its full potential.

Understanding the Token Limit

Before we dive into the methods, it’s crucial to understand the token limit of the ChatGPT API. Each model has a maximum number of tokens it can process in a single API call. For instance, as of my knowledge cutoff in September 2021:

GPT-3 model has a token limit of 4096 tokens.
GPT-4 model can handle up to 32768 tokens.

Remember, each token in English can be as short as one character or as long as one word.

For example, “ChatGPT” is one token, while “Chat GPT” is two tokens. Therefore, managing your token usage efficiently is key to processing longer text inputs.

Method 1: Using the File Parameter

One of the simplest ways to handle longer text inputs is to utilize the file the parameter in the API.

This parameter can accept a file of any length, allowing you to upload your entire text file to the ChatGPT API for processing.

This method is especially useful for large text files and makes the API’s token limit less of a constraint.

Method 2: Multiple API Calls with the Prompt Parameter

If you prefer not to upload your entire text file, another method is to use the prompt parameter multiple times.

A. Using Chat GPT Chat Response Prompt:

You can use the following prompt to input the longer text without ChatGPT.

ChatGPT Prompt:

You will receive information from me and must confirm receipt by responding with “RECEIVED”. DO NOT WRITE ANYTHING ELSE. NOT A SINGLE WORD MORE. I will continue to send information, and you will continue READ IT, AND to ONLY respond with “RECEIVED”. This process will continue until I send a message saying “Start Writing” to you. Once you receive the “Start Writing” message from ME, you should perform the as stated.

A. Python Code:

This involves splitting your text into sections, each within the token limit, and sending these as separate prompts in successive API calls.

To retain context, include responses from previous requests in the subsequent ones.

Here is a Python code example illustrating the process:

import openai

# Set up your OpenAI API credentials
openai.api_key = 'YOUR_API_KEY'

# Initialize the conversation with an empty string as context
context = ''

# Split your longer text into smaller chunks
text_chunks = ["This is the first chunk of text.", "This is the second chunk of text.", "And so on..."]

# Iterate through each chunk and send requests
for chunk in text_chunks:
    # Append the current chunk to the existing context
    input_text = context + chunk

    # Send the API request
    response = openai.Completion.create(
        engine='text-davinci-003',
        prompt=input_text,
        max_tokens=500,  # Adjust according to the maximum token limit for your plan
        temperature=0.7,
        n=1,
        stop=None,
        context=context
    )

    # Get the generated message from the response
    message = response.choices[0].text.strip()

    # Append the message to the context for the next iteration
    context += message

    # Process the generated message or store the results

    # Rest of your code...

Remember to replace ‘YOUR_API_KEY‘ with your actual API key

Misunderstandings about Overcoming the Limit

There have been several misconceptions floating around about how to deal with the token limit. Let’s clear them up:

File Parameter: Contrary to some claims, there’s no ‘file’ parameter available in the OpenAI API that can process long text inputs, at least as of my knowledge cutoff in September 2021.
Stream Parameter: OpenAI API doesn’t support streaming inputs. While the API does support streaming outputs, meaning the API can return parts of the response as they become available, you can’t send text to the API one token at a time.
Batch Parameter: There’s no ‘batch’ parameter that allows sending multiple text files to the API simultaneously.

Strategies for Processing Long Texts

Despite these limitations, there are still several strategies you can use to work with long texts:

Dividing Text into Smaller Fragments: This is your bread-and-butter strategy. Break your text into smaller parts and process each part separately. But remember, don’t lose the context of the conversation as you do this.
Use of Embeddings: If you’re tech-savvy, this is for you. Represent your text as embeddings, which you can then process. It might sound complicated, but there’s some handy code from the OpenAI Cookbook that explains how to do this.
ChatGPT Retrieval Plugin: Another one for tech enthusiasts. This plugin creates a vector database of your document’s text, which can then be processed by the language model.
Using the GPT-4 API: For those who like the latest and greatest, GPT-4 can handle over 25,000 words of text, making it perfect for long-form content creation and extended conversations.
Storing Data in Cloud Storage Service: And finally, for the cloud storage users, you can store your data in a service like Amazon S3 or Google Cloud Storage. You can then grant the OpenAI API access to the data using credentials or access keys.

Strategy	Description	Suitable For
Dividing Text	Break text into smaller parts	Everyone
Use of Embeddings	Represent text as embeddings	Tech-savvy users
ChatGPT Retrieval Plugin	Create a vector database of your text	Tech-savvy users
GPT-4 API	Use the latest model to handle large texts	Early adopters
Cloud Storage	Store data in cloud storage and grant OpenAI API access	Cloud storage users

Mastering the Token Methodology and Working with Long Texts:

1. Text Embeddings:

One way to work with long texts in GPT-4 is by using embeddings. This technique involves turning the text into a vector representation that can capture the semantic content of the text.

OpenAI provides resources like the OpenAI Cookbook, which includes code examples on how to use embeddings with web-crawled Q&A data.

2. Retrieval Plugins:

The ChatGPT retrieval plugin is another tool that can be used. This plugin allows for the creation of a vector database of your document’s text, which can then be processed by the language model.

3. Dividing Text into Smaller Fragments:

Long text can be divided into smaller fragments, with the relevant pieces retrieved according to your task.

This can be beneficial when using the API, as smaller pieces of text can be processed more easily than a single large text.

4. API’s Streaming Capabilities:

The OpenAI API supports streaming input, which allows you to send text to the API in smaller chunks over time, rather than sending all the text at once.

This can be useful when working with large amounts of text, as it can help you avoid hitting the input length limit.

5. Storing and Retrieving Data:

When working with very large amounts of data, one option is to store the data in a cloud storage service, such as Amazon S3 or Google Cloud Storage.

You can upload the data to the cloud storage service and then grant the OpenAI API access to the data using credentials or access keys.

When you make a query to the API, the API will retrieve the data from the cloud storage service and process it.

Read More: