Conversation Metrics for Digital Assistants

With digital assistant Insights reports, you can find out:

  • The number of conversations initiated from a digital assistant over a given time period and their rate of completion.

    Conversations are not the same as metered requests. To find out more about metering, refer to Oracle PaaS and IaaS Universal Credits Service Descriptions.
  • The popularity of the skills registered to a digital assistant as determined by the traffic to each skill.

These reports display data when Enable Insights, located in the General page of Settings This is an image of the Settings icon. is switched on. To access the reports, open a digital assistant and then select This is an image of the Insights icon. in its left navbar.

You can also view detailed reports on individual skills that show things such as how often each intent is called (and which percentage of those calls have completed) and the paths that users take through the skill. See Insights.

Report Types

  • Overview - Use this dashboard to quickly find out the total number of voice and text conversations by channel and by time period. The report's metrics break this total down by the number of complete, incomplete, and in-progress conversations. In addition, this report tells you how the skill completed, or failed to complete, conversations by ranking the usage of the skill's transactional and answer intents in bar charts and word clouds.
  • Conversations - Displays the transcripts for the conversations that occurred duing a session. You can read a plain text version of this conversation and also review it within the context of the digital assistant's skill routing and intent resolution.
  • Events - Displays metrics and graphs for the external events relayed to the skills within the digital assistant and the outbound events sent to external sournces.
  • Retrainer - This is the counterpart to the skill-level Retrainer, where you improve the intent resolution for the registered skills using the live data that flows through the digital assistant.
  • Export - Lets you download a CSV file of the Insights data collected by Oracle Digital Assistant. You can create a custom Insights report from the CSV.

Review the Overview Metrics and Graphs

Click This is an image of the Insights icon. in the left navbar to access the following reports.
  • Completed – The number of conversations that were routed through the digital assistant and were then completed by the individual skills. Conversations are counted as complete when the traversal through the dialog flow ends with a return transition or at a state with the insightsEndConversation property.
  • Incomplete – Conversations that users didn't complete, because they abandoned the skill, or couldn't complete because of system-level errors, timeouts, or flaws in the skill's design.
  • In Progress – The number of skill conversations (that were initiated by the digital assistant) that are still ongoing. Use this metric to track multi-turn conversations. An in-progress conversation becomes an incomplete conversation after a session expires.
  • No Match - The number of times that the digital assistant could not match any of its registered skills to a user message.
  • Canceled - The number of times that users exited a skill by entering "cancel".
  • Help - The number of times the Help system intent was invoked.

    When Insights has been disabled for a digital assistant, the Completed count will still continue to increase if Insights has been enabled for any of the member skills. Despite this, there will be no data logged for the digital-assistant specific Help and No Match metrics until you swich on Enable Insights.
    Description of da-insights-overview.png follows

  • Languages: Charts the number of conversations for each skill by supported language.
    Description of da_languages_graph.png follows

  • Performance: Shows the number of conversations by skill.

    The Trend view provides a graph of completed conversations over the selected time period.

    Description of da-insights-skills-trend.png follows

  • Skills - The Summary view shows the number of conversations handled by the skill for a given period. By hovering over the progress bar, you can find out the number of completed conversations out of the total. Description of da-insights-skills-summary.png follows
    If a skill has been versioned during the selected time period, you can find out the by-version distribution of conversations using the Traffic graph. Clicking a skill opens this graph, which illustrates the volume of conversations handled by each version of the skill in terms of the total conversations handled by the skill for the selected period. Each arc on the Traffic graph represents a version of the skill, with the length and accompanying percentage indicating the volume of conversations that it handled. The hover text for each of these arcs describes the percentage in terms of the conversation count. To break this number down, say, to find out what this count means in terms of the intents invoked for a particular version of the skill, click the arc to drill down to the skill-level Insights.

    Description of da_insights_skill_traffic.png follows

    If there are any incomplete conversations during the selected period, the total number is broken down by the following error categories:
    • Timeouts – Timeouts are triggered when an in-progress conversation is idle for more than an hour, causing the session to expire.
    • System-Handled Errors – System-handled errors are handled by the system, not the skill. These errors occur when the dialog flow definition is not equipped with error handling, either globally in the defaultTransitions node, or at the state level with error transitions.
    • Infinite Loop – Infinite loops can occur because of flaws in the dialog flow definition, such as incorrectly defined transitions.
    • Canceled - The number of times that users exited a skill by explicitly canceling the conversation.

    Description of da_incomplete_conversations_chart.png follows

View the Conversations Report

Instead of filtering an exported spreadsheet, you can use the Conversation report to track the digital assistant's routing of the chat to its various skills and also find out which intents were invoked along the way.

This report, which you can filter by both channel and skill, lists the conversations in chronological order by Session ID (the ID the user's session on a channel) and presents a transcript of the conversation in its User Messages and Skill Responses columns. The Session ID is created when a client connects to channel and contains all of the data from the skills that participated in a conversation. This ID expires if there's no activity for 24 hours, but may never expire if the client continues with its conversations.
Description of da_conversations_report.png follows

Clicking View Conversations in the Actions column enables you to view all of the conversations that occurred for the session in the context of a chat window. In this mode, you can see the digital assistant routing at work by seeing how it switched skills in response to the user input and also find out which of the skills' intents were invoked as a result of this context switching. When a session spans multiple days, the earliest conversations display first.

The View and Skills filters that you can apply to the Conversations report do not alter the contents of the conversation transcripts displayed in the User Messages and Skill Responses columns or in the View Conversations mode.
You can share the session with a colleague who has access to your instance by providing a link that can be pasted into a browser window. To get this link, click Options then Copy Conversation Link.

Apply the ODA Retrainer

You can improve the intent resolution for the registered skills by updating their training data with customer input. Some of this input may not have been resolved to any skill, or it may have been routed to the wrong skill. The Retrainer helps you evaluate this user input and add it to a skill if you consider it a useful addition to the training data.
Description of da_retrainer.png follows

By default, Retrainer applies the NoMatch filter so that it returns all of the user messages that could not be matched to any skill registered to the digital assistant. For each of these returned phrases, the report presents the top two highest-ranking skills, the Win Margin that separates them and, through a horizontal bar chart, their contrasting confidence scores.

Before you use the Retrainer, there are a couple of things to keep in mind:To update a skill's training corpus from the Retrainer:
  1. Filter the registered skills. For example, you can filter for all of the phrases that matched to particular skill within the digital assistant, or you can apply the NoMatch for the phrases that did not match up with any of the skills.
  2. Select the utterance.
  3. Select a draft version of the skill for the utterance. If no draft version exists, then create one. If you can't select a skill, it's because it uses native multi-language support. In this case, you can't update the skill from here. You'll have to use the skill-level retrainer instead.
  4. Select the intent. After you add the phrase as an utterance in the training corpus, you can no longer select it for retraining.
  5. If your skill supports more than one native language, then you can add the utterance to the language-appropriate training set by choosing from among the languages in the Language menu.

    This option is only available for natively supported skills.

    Description of select_native_language_for_skill_da_retrainer.png follows

  6. Retrain the skill.
  7. Republish the skill.
  8. Update the digital assistant with the latest version of the skill.

PII Anonymization

Like the skill-level insights, you can apply anonymization to Personally Identifiable Information (PII) values. At the digital assistant level, anonymization is applied to the conversations handled by the digital assistant. Anonymization enabled at the digital assistant level does not extend to the skills. If anonymization is enabled for a skill, but not for the digital assistant, then only the skill conversations will be anonymized and vice versa. For completely anonymized conversations, you need to apply anonymization to the digital assistant and its skills separately.

You can anonymize the PII values recognized for the following system entities:
  • URL
Enable PII Anonymization
  1. Click Settings > General.
  2. Switch on Enable PII Anonymization.
  3. Click Add Entity to select the entity values that you want to anonymize in the Insights reports and the logs.

    Anonymized values for the selected entities are persisted only after you enable anonymization. They are not applied to prior conversations. Depending on the date range selected for the Insights reports or export files, the entity values might appear in both their actual and anonymized forms. You can apply anonymization to any non-anonymized value when you create an export task. These anonyms apply only to the exported file and are not persisted in the database.
    If you want to discontinue the anonymization for an entity, or if you don't want an anonym to be used at all, the select the entity and then click Delete Entity. Once you delete an entity, the actual value appears in the Insights report and throughout the Insights reports for subsequent conversations, even if it previously appeared in its anonymized form.

    Anonymization is permanent (the export task-applied anonymization notwithstanding). You can't recover the actual values after you enable anonymization.

Description of pii_settings_configuration.png follows

Create an Export Task

If you want another perspective on Insights reporting, then you can create your own reports from exported Insights data. This data is exported in a CSV file. You can write a script to sort the contents of this file.

To create an export task:
  1. Open the Exports page and then click + Export.
  2. Enter a name for the report and then enter a date range.
  3. Click Enable PII anonymization for the exported file to replace any non-anonymized values with anonyms in the exported file. These anonyms exist only in the exported file if PII is not enabled in the digital assistant settings. They are not persisted to the database and they do not appear in the logs or in the Conversations report. This option is enabled by default whenever you set anonymization for the digital assistant.

    The PII anonymization that's enabled for the skill or digital assistant settings factors into how PII values that get anonymized in the export file and also contributes to the consistency of the anonymization in the export file.
  4. Click Export.
  5. When the task succeeds, click Completed to download a ZIP of the CSV.

    The CSV file for a digital assistant contains data for the skills that are directly called through the digital assistant. The data for skills called outside of the context of a digital assistant is not included in this file.
Description of insights_export_dialog.png follows

The data may be spread across a series of CSVs when the export task returns more than 1,048,000 rows. In such cases, the ZIP file will contain a series of ZIP files, each containing a CSV.