Exporting Raw Conversations with the Botpress API

The Botpress API allows you to access your bot's conversations from an external service. This is very useful if you need to manage or analyze the conversations somehow. In this tutorial we give steps to use the API for this purpose and steps to format the response data. Let's get started!

Access the API documentation to verify the schema/structure of the Conversations (opens in a new tab) and the Messages (opens in a new tab) and see if the data is useful for your use case.

You can find your personal access token in your Admin Dashboard (opens in a new tab). You can find your bot ID in the URL of your bot's dashboard, it's the letters and numbers right after chatbots/

1. Getting the Conversations list from the API

Using Axios

Send a GET request to https://api.botpress.cloud/v1/chat/conversations adding the following headers: Authorization: Bearer <your-personal-access-token> and x-bot-id: <your-bot-id>.

This is how the request would look like using Axios in a Javascript application (e.g in a Execute Code in Botpress):

const requestConfig = {
  headers: {
    // Authorization header with a Bearer token for authentication.
    Authorization: `Bearer ${process.env.BOTPRESS_PERSONAL_ACCESS_TOKEN}`,
    // The unique identifier of the bot for which we want to retrieve conversations.
    'x-bot-id': process.env.BOTPRESS_BOT_ID,
  },
}
 
// An empty array to store all retrieved conversations.
const allConversations = []
 
// Send an HTTP GET request to the specified API endpoint to get conversations.
const getConversations = await axios.get(
  `https://api.botpress.cloud/v1/chat/conversations?nextToken=${paginationToken}`,
  requestConfig
)
 
// Add the retrieved conversations to the list.
allConversations.push(...getConversations.data.conversations)

Using the Botpress Client library

The same logic to get conversations but using the Botpress Client library (opens in a new tab) in a Node.js application:

import { Client } from '@botpress/client'
 
const client = new Client({
  token: process.env.BOTPRESS_ACCESS_TOKEN,
  botId: process.env.BOTPRESS_BOT_ID,
})
 
const allConversations = []
 
const getConversations = await client.listConversations({
  nextToken: paginationToken,
})
 
allConversations.push(getConversations.conversations)

The API returns a maximum of 20 conversations per request. You will need to use the nextToken property to fetch the next batch of conversations. The nextToken property is returned in the meta object of the response.

2. Getting all Conversations from the API

Now that you know the methods of getting the first conversations from the API, let's see how to get them. You can get them gradually or all at once, but you need to use the pagination token regardless.

Getting conversations gradually

Create a function that will fetch the next batch of conversations when called. You could use a button to trigger it or trigger it automatically with infinite scroll. Pass the nextToken property retrieved from the first request as a parameter to the function, then update it on subsequent calls. More about pagination here (opens in a new tab).

async function loadConversations(paginationToken) {
  const getConversations = await axios.get(
    `https://api.botpress.cloud/v1/chat/conversations?nextToken=${paginationToken}`,
    {
      ...requestConfig,
    }
  )
 
  return getConversations.data
  // The returned value will be an object: `{ conversations: [...], meta: { nextToken: "string" } }`
}

Getting all conversations

Use a loop to fetch all conversations at once using the above logic. This is not recommended for bots with a large number of conversations as it may take a long time to complete and may cause timeouts. The use of a library like p-retry (opens in a new tab) to retry requests automatically is recommended.

const allConversations = []
let paginationToken
 
do {
  // Send an HTTP GET request to the specified API endpoint to get conversations.
  const getConversations = await axios.get(
    `https://api.botpress.cloud/v1/chat/conversations?nextToken=${paginationToken}`,
    requestConfig
  )
 
  // Add the retrieved conversations to the list.
  allConversations.push(...getConversations.data.conversations)
 
  // Retrieve the token for the next set of conversations, if available.
  paginationToken = getConversations.data.meta.nextToken
 
  // As long as paginationToken has a value, this function will execute again
} while (paginationToken)

2. Getting the Messages list

Once you have gone through the list of conversations and found out the id of the desired conversation to export, you can then use the API to list the actual messages.

In the example below we are using the Botpress Client library (opens in a new tab) and assume the client is set up already:

const conversationId = 'the desired conversation id here'
 
const allMessages = []
 
const getMessages = await client.listMessages({
  conversationId,
  nextToken: paginationToken,
})
 
allMessages.push(getMessages.messages)

You might create a loader function or a finite loop as shown above to get all the messages. This is how the allMessages array is gonna look like:

[
  {
    "id": "503dda87-3e54-40b6-8167-55d685a85345",
    "createdAt": "2023-10-24T20:20:44.812Z",
    "conversationId": "ec669jj8-e465-482d-8fbe-cb348bf81212",
    "payload": {
      "text": "Pick one or many from the list",
      "options": [
        {
          "label": "First option",
          "value": "First option"
        },
        {
          "label": "Second option",
          "value": "Second option"
        }
      ]
    },
    "tags": {},
    "userId": "efc0f8e8-7e51-4458-8907-ea1e43eb8852",
    "type": "choice",
    "direction": "outgoing"
  },
  {
    "id": "41bb8564-e7ab-4113-c3d8-5fdhh27e4c34",
    "createdAt": "2023-10-24T20:20:49.201Z",
    "conversationId": "ec66ffb8-e465-482d-8fbe-csd348b81212",
    "payload": {
      "type": "quick_reply",
      "text": "First option",
      "payload": "First option"
    },
    "tags": {
      "webchat:id": "cc70c21b-4g24-4f5c-8f52-43a900b7b047"
    },
    "userId": "f96da97d-4215-447f-86e2-b9e33232e6d3",
    "type": "quick_reply",
    "direction": "incoming"
  },
  {
    "id": "c6ee8bd6-c862-54d2-97cb-998f555d4251",
    "createdAt": "2023-10-24T20:20:49.795Z",
    "conversationId": "ec669dd8-e465-482d-8fty-cb3348581212",
    "payload": {
      "imageUrl": "https://s3.us-east-1.amazonaws.com/cloud-studio-botsbca2d219-2316w6llinepa/f4b77ea0-d597-4d3f-988d-9c66f127d542/media/2360a7bb-162b-4ed1-8551-7000fdb0f7ef.png"
    },
    "tags": {},
    "userId": "efb454t5-7e51-4458-8007-ea1e43eb8852",
    "type": "image",
    "direction": "outgoing"
  }
]

In this example the bot sends a Multiple Choice card, then the user answers by clicking one of the options, and then the bot sends an Image card. Notice how the message payload changes according to the message type, so be mindful of that when reading the payload property.

3. Formatting the Messages list

Now that you have the list of messages of a conversation it's time to format it so it becomes readable. We are going to use a custom function for that:

function formatMessage(message) {
  let messageText = ''
 
  if (message.direction === 'incoming') {
    messageText += 'User: '
  } else {
    messageText += 'Bot: '
  }
 
  if (message.type === 'text' || message.type === 'quick_reply') {
    messageText += message.payload.text
  } else if (message.type === 'choice') {
    messageText += `${message.payload.text}: ${message.payload.options.map((option) => option.label).join(' / ')}`
  } else {
    messageText += `Media message of type ${message.type}`
  }
 
  messageText += `\n at ${new Date(message.createdAt).toLocaleString()}`
 
  return messageText
}
 
const history = allMessages.map((message) => formatMessage(message)).join('\n\n')

The history string will end up looking like this:

Bot: Pick one or many from the list: First Option / Second Option
at 24/10/2023, 5:20:44 PM
 
User: First Option
at 24/10/2023, 5:21:00 PM
 
Bot: Media message of type image
at 24/10/2023, 5:21:09 PM

You can customize the formatMessage function to have different formatting or to return more information of the messages.
Now use the history variable however you like - send it via email, add it to a Google Sheets document, send it to Botpress to do further processing, etc.

Start Here From Within your Bot