Autopilot

Fonoster’s Autopilot is a component within the platform that allows you to create powerful conversational experiences. It is built on top of Fonoster Programmable Voice and uses the latest advances in Large Language Models (LLMs) to provide a natural and engaging experience.

​

Overview

The following is an example of creating an Autopilot application using the SDK.

First, add the following content to a file named appConfig.yaml:

name: "Awesome Autopilot"
type: "AUTOPILOT"
speechToText:
  productRef: "stt.deepgram"
  config:
    model: "nova-3"
    languageCode: "en-US"
textToSpeech:
  productRef: "tts.deepgram"
  config:
    voice: "aura-asteria-en"
intelligence:
  productRef: "llm.groq"
  config:
    conversationSettings:
      firstMessage: "Hello, I'm your AI assitant."
      systemPrompt: |
        You are a Customer Service Representative. You are here to help the caller with their needs.
      goodbyeMessage: "Thank you so much, bye!"
      systemErrorMessage: "I'm sorry, I didn't understand that. Can you please repeat it?"
      idleOptions:
        message: "Are you still there?"
    languageModel:
      provider: "groq"
      model: "llama-3.3-70b-versatile"
      maxTokens: 240
      temperature: 0.4

Then, create the application as follows:

fonoster applications:create --from-file appConfig.yaml

Similarly, to update the application, you can use the applications:update command with the from-file flag.

​

General configuration

The Autopilot configuration is divided into a general section and three sub-sections: speechToText, textToSpeech, and intelligence.

The general section contains name, type, and endpoint properties.

The name property is the name of the Autopilot application. The type property is the type of the application, which should always be set to AUTOPILOT. The endpoint is an optional property allowing you to specify the endpoint for self-hosted Autopilots.

​

Speech settings

Autopilot applications support a variety of speech-to-text and text-to-speech vendors. The speechToText and textToSpeech objects allow you to define the speech-to-text and text-to-speech engines to use.

You can mix and match vendors to suit your needs. For example, you can use Deepgram for speech-to-text and Google for text-to-speech. Please check the Speech Vendors section for more information on configuring speech-to-text and text-to-speech.

​

Conversational settings

The conversationSettings object allows you to define the Autopilot’s conversational behavior. The conversation settings are independent of the language model used.

The following is a list of the supported settings:

A few noteworthy settings include the maxSpeechWaitTimeout, initialDtmf, idleOptions, and vad.

​

Max Speech Wait Timeout

The maxSpeechWaitTimeout property allows you to specify the maximum time in milliseconds to wait for the caller before returning the speech-to-text result. If the caller does not speak within the specified time, the speech-to-text engine will return the result.

A value that is too low may result in the speech-to-text engine returning the result before the caller finishes speaking. A value that is too high may result in the speech-to-text engine waiting too long for the caller to speak.

​

Initial DTMF

Sometimes, users will use call forwarding to reach the number in Fonoster. Some telephony service providers require a Dual-tone multi-frequency (DTMF) to be played before connecting the call. The initialDtmf property allows you to specify a DTMF to play when the session starts.

​

Voice Activity Detection (VAD)

The vad object allows you to configure the voice activity detection settings. Voice activity detection is used to detect when the caller is speaking and when they are not speaking.

The vad object has the activationThreshold, deactivationThreshold, debounceFrames properties. The actionThreshold property is the activation threshold for voice activity detection. The deactivationThreshold property is the deactivation threshold for voice activity detection. The debounceFrames property is the number of frames to debounce the voice activity detection.

A lower activation threshold will make the detection more sensitive to the caller’s speech. A higher activation threshold will make detecting voice activity less sensitive to the caller’s speech.

A lower deactivation threshold will result in more aggressive voice activity detection deactivation. A higher deactivation threshold will result in less aggressive voice activity detection deactivation.

The debounceFrames parameter introduces a delay mechanism that ensures that transitions between “speech” and “non-speech” states are stable and not too sensitive to small fluctuations in the input audio signal. Here’s how it works:

By requiring multiple consecutive frames (debounceFrames) to confirm speech or non-speech, the system filters out short bursts of noise or brief gaps in speech that might otherwise cause erratic state changes.

​

Langue model configuration

The languageModel object allows you to define the language model the Autopilot uses. The language model is responsible for generating responses to the user’s input.

The following is a list of the supported settings:

​

LLM providers and models

The Autopilot supports multiple language model providers. The following is a list of the supported providers:

​

Knowledge bases

Coming soon…

​

Tools

Fonoster’s Autopilot allows you to use tools to enhance the conversational experience. Tools are used to perform specific actions during the conversation.

​

Built-in tools

The following is a list of built-in tools available for an agent:

​

Custom tools

You can add custom tools under intelligence.config.languageModel.tools, which is an array where each tool is defined as an object. These tools enable your assistant to interact with external services, APIs, or execute specific actions.

Each tool must follow the tool schema, for consistency and compatibility.

The following example demonstrates how to add a custom tool that fetches available appointment times for a specific date:

name: getAvailableTimes
description: Get available appointment times for a specific date.
requestStartMessage: "I'm looking for available appointment times for the date you provided."
parameters:
  type: object
  properties:
    date:
      type: string
      format: date // only 'enum' and 'date-time' are supported
  required:
    - date
operation:
  method: get
  url: https://api.example.com/appointment-times
  headers:
    x-api-key: your-api-key

​

Key Components of a Tool Definition:

• name: A unique identifier for the tool
• description: A brief explanation of what the tool does
• requestStartMessage: The message sent when the tool is triggered
• parameters: Defines the expected input structure in accordance with the JSON Schema standard, which is also required for OpenAI compatible tool calling
  • type: Defines the structure of the input (typically object)
  • properties: Specifies the fields expected in the input
  • required: Lists the fields that must be provided
• operation:
  • method: The HTTP method (get and post are supported)
  • url: The endpoint to send the request to
  • headers: Any necessary headers, such as authentication keys

For additional details, refer to the tool schema documentation.

​

Autopilot’s Test Cases

The Autopilot supports automated testing through test cases defined in the configuration. Test cases allow you to verify the behavior of your Autopilot before deploying it to production.

The following is an example of creating a test case for Fonoster Autopilot:

testCases:
  evalsLanguageModel:
    provider: openai
    model: gpt-4o
    apiKey: sk-proj-REDACTED
  scenarios:
    - ref: test-case-1
      description: Test Case 1 Description
      telephonyContext:
        callDirection: FROM_PSTN
        ingressNumber: '+1234567890'
        callerNumber: '+1234567890'
      conversation:
        - userInput: 'Hi, can you tell me what''s in the menu?'
          expected:
            text:
              type: similar
              response: |
                We have a variety of sandwiches, salads, and drinks. Anything
                in particular you're looking for?
        - userInput: 'Nevermind, I want to speak to a human'
          expected:
            text:
              type: similar
              response: |
                I'll transfer you to a human representative. Please hold while I
                connect you.
            tools:
              - tool: transfer
                parameters: {}

​

Evaluation Language Model

The evalsLanguageModel section defines the model used to evaluate test cases:

​

Test Scenarios

Each test scenario represents a complete conversation flow. The scenario includes:

• ref: A unique identifier for the test case
• description: A brief description of what the test case verifies
• telephonyContext: Emulates the context of a real phone call with the following properties:
  • callDirection: The direction of the call (e.g., “FROM_PSTN”)
  • ingressNumber: The number being called
  • callerNumber: The number making the call

This information is available to the AI model to help it understand the context of the call and might be use in your prompts.

​

Conversation Turns

Each scenario contains a series of conversation turns. A turn represents a single interaction between the user and the Autopilot, consisting of:

The expected object can validate:

• Text responses via the text property:

  text:
    type: "similar"
    response: "Expected response..."
  

• Tool usage via the tools property:

  tools:
    - tool: "toolName"
      parameters:
        param1: "value1"
        param2: "valid-date" # Special keyword to test for a valid date