The Awaken CoPilot integration is designed to work directly and easily within Intelligent Agent, providing AI enrichment to audio interactions in real-time. It comprises a few components working together:

 

Once integrated, building Workflows enriched by CoPilot is as simple as adding an Awaken Audio - Start Control to initiate the audio streaming, and adding a CoPilot - Intent Actions Control to any Page that you want to define intents to be listened for and any actions to occur when the intent is matched. The audio stream will automatically end when the record is completed, so no further management is needed beyond initiating the stream; starting the audio stream has been left as a triggered action simply so that when handling a non-audio interaction (such as an email Workflow) that the silent interaction doesn't get streamed for processing.

 

 

 

 

Broadly, making use of Awaken CoPilot within Intelligent Agent has 6 steps:

 

 

Prior to using the CoPilot integration, you will need to request that the licence part be added to your System licence and API credentials be generated. Please contact your Intelligent Agent vendor, and provide the following information:

 

Once your request has been received and approved, you will be provided a link to download the Awaken audio capture service, and your Intelligent Agent licence will be updated to include the relevant licence parts (requiring the locally cached Intelligent Agent licence to be updated). If any licence parts are provided as the Named type, then ensure that they have been assigned to the appropriate Users.

 

You will also receive an automatic email at the specified email address, providing a username, temporary password, and initial setup URL for the Awaken Cloud API:

 

Follow the URL in the email, providing the username and temporary password, and specify a new password. Be sure to record these details, as they are required to configure and operate the local audio capture service on agent machines!

 

 

To feed the selected generative AI with content, the audio both received and generated by the agent needs to be captured, converted into text, and then made available to the generative AI. Awaken Intelligence provide a simple yet configurable solution to this transcription and distribution that can be deployed for all agents via standard enterprise tools, comprising a Windows service and a configuration file. The configuration file is the appsettings.json file that is in the Windows service's installation folder, and will need to be populated with four details:

 

For example:

 

Once these details are populated into the configuration file in the Windows service's installation folder, and the Windows service has been started, then it is possible to trigger the recording of the agent's microphone and speaker output by the addition of an Awaken Audio - Start Control to a Workflow. The service will then also manage the streaming of the captured audio to the Awaken Intelligence cloud service for audio-to-text transcription and identification of key data, and finally passing this transcribed text on to the Workflow.

 

 

 

 

In order for CoPilot to be able to provide the context-sensitive guidance and intent checks against your agent interactions, access to a generative AI model is needed. These services can be procured from various vendors with varying different capabilities and qualities, but currently Intelligent Agent only has an integration with Microsoft's Azure OpenAI Service. As such, you will need an account and access to an instance of this service, and to create an integration URL and API key to be used in the Intelligent Agent Connector.

 

 

For guidance on creating and configuring an AI deployment, please see the appropriate Microsoft documentation. CoPilot supports any of the currently-available AI models provide by Microsoft Azure, but currently only formally supports the 2023-08-01-preview API version for these models. When picking an AI model and service level, please ensure that the request and token limits will be appropriate for your expected usage as well as a margin of overhead.

 

Once the AI deployment has been generated, an endpoint URL and key will be needed for use by the AI - GPT - Azure Connector. The endpoint URL that can be found within the deployment's configuration will typically be of the format https://{your-resource-name}.openai.azure.com, but the deployment's ID and API version need to be appended for use by the Connector - please see an example below, and example Microsoft documentation can be found here.

 

 

 

Once the generative AI details have been obtained, a Connector must be configured with the details to power CoPilot. At this time Intelligent Agent only has an integration with Microsoft's Azure OpenAI Service, so you will need to configure an AI - GPT - Azure Connector.

 

The Temperature setting allows tuning of how deterministic the model will be in its responses, with lower numbers causing greater consistency, whereas higher numbers cause greater creativity. Note that you can configure multiple connectors using the same generative AI details but different temperatures, so you could even tune the temperature of requests across different phases of a Workflow or between different Workflows by using differently-configured Connectors.

 

Ensure that the Connector is active, so that it can be used within Workflows.

 

 

 

In order to confirm that CoPilot is operational, it is easiest to create a small test Workflow to verify the various bits of functionality are working normally.

 

Create a new basic Workflow, and ensure that the Start Page contains an Awaken Audio - Start Field. Add a CoPilot - Intent Actions Field, and configure it with a Connector and one or more intents.

 

Publish this test Workflow, create a test Campaign, and then proceed to confirming that the CoPilot setup is operational.

 

 

Once all of the above items have been configured, then operation of the CoPilot system can be verified as follows:

 

Once operation has been confirmed, then it is recommended that the CoPilot widget and Workflow Design for Awaken CoPilot articles are reviewed for more general guidance.

 

 

CoPilot only currently supports English audio, but a range of accents have been tested successfully. Other languages are expected to be supported in a future release.

 

CoPilot is designed to be used in a single Workflow at a time for each agent; swapping between different interactions isn't supported.

 

The CoPilot transcription and intent history isn't preserved between successive record runs if a record is resumed or rescheduled. The CoPilot history and action will only relate to the current record run.

 

Intents are checked on an utterance-by-utterance basis, so if the content relevant to an utterance is delivered over the course of successive events then it may cause an intent to not be triggered. Having the agent briefly summarise the relevant details to confirm with the caller may be a simple yet effective way to address this should it occur.

 

 

If the agent sees a "You are not licensed to use this feature" message within the CoPilot widget, then this indicates that they haven't been successfully assigned the CoPilot licence part.

 

If the agent doesn't see the Awaken Audio Settings button on the top-right toolbar next to the Activities list, or if they get a message about not being connected to the audio service, then this indicates that they haven't been successfully assigned the Integration - Audio Capture - Awaken licence part.

 

If the agent only sees output (e.g., headset) but no input (e.g., microphone) devices in the Awaken Audio Settings, then this may indicate the agent is accessing a remote PC (e.g., using Remote Desktop to access an office PC or virtual machine), or there are permissions issues within Windows that prevent the other audio devices from being detected properly.

 

If a message is seen to the left of the CoPilot widget's icon saying "Trying to reconnect the audio socket", then this indicates multiple Intelligent Agent windows are open on the Desktop or in a Workflow or preview simultaneously. This is a result of a restriction to only one audio connection being allowed between the agent's local audio service and the browser at a time, and can be remedied by simply ensuring the agent doesn't have multiple instances of the CoPilot widget loaded simultaneously.

 

If intents aren't being completed when expected, then it is most likely to be the case that the transcribed audio doesn't contained the expected phrases. If running the record as a User with the Designer licence part, then it is possible to see the transcribed text within the Conversation tab of the CoPilot widget. This should help indicate whether the issue is with the audio not being transcribed as expected, or if there are other issues such as pauses in the conversation causing the required intent to be split across multiple utterances.

 

If the transcription quality appears to be low, then a common cause for this is either poor audio quality, or too-short utterances. If the audio stream has background noise or the audio is overly compressed, then the speech may be difficult to distinguish and therefore not transcribed accurately. Similarly, if the utterances are very short, then there is less ability for the transcription platform to analyse the sentence and self-correct any words with more-likely alternatives. This is most noticeable when utterances are only a few words long, and some transcribed words may be substituted with similar-sounding words.

 

If utterances take a long time to appear in the Conversations tab of the CoPilot widget, or intents take a long time to be checked off, this may be a symptom of background noise in the recording causing the utterances to not naturally break at pauses in the conversation where they would normally be submitted. It is advised to check the background noise level on the agent's microphone and the incoming audio stream to see if this may be contributing to delayed transcription and intents.

 

Unusual words (most commonly proper nouns) may be misidentified by the transcription platform, as it makes use of a generic dictionary. This should be taken into account when designing any intents. Adding a custom dictionary is expected to be supported in a future release.

 

If an intent is checked off but the linked action isn't performed, then this may be because the agent has navigated away from the Page after the intent was originally added; intents will only trigger actions while on the Page that they are added, and won't work even if you reload the same Page or return from a different Page.

 

If transcriptions are being generated as expected, but intents aren't being checked off and the AI tab isn't responding to queries, then this may indicate that you have used more tokens than your current AI service's deployment allows. Operation should resume when the rate limit is next refresh (typically once per minute), but likely indicates that you need to increase the cap on your AI service's deployment.

 

If any of the CoPilot functionality doesn't appear to have loaded or be operational, then some components may have been blocked by false positives from advert blocking software within the browser. It is recommended to disable any adblock extensions, and check the F12 Developer Tools to see if there are any errors displayed in the Console tab.