Oracle Cloud Infrastructure Documentation

Analytics

The Analytics reports enable you to gauge how well your skills and digital assistants resolve intents and complete chat sessions both on an individual basis and as a group.

You can access these instance-wide reports by choosing Analytics from the left menu.

Description of left_menu_analytics.png follows

By default, the report renders metrics and graphs for all the instance's skills and digital assistants. You can change the view to review the cross-version performance of a digital assistant or skill by querying the Digital Assistant/Skill menu.
This is an image of the Analytics report Digital Assistant/Skill menu.

Metrics

These metrics render for both the instance and for individual skills.
  • Total Utterances – A comparison of the voice and text traffic.
  • Total Resolved and Unresolved Intents – The comparison and trend of the voice and text utterances that got resolved to intents to those that did not.
  • Defection Rate – For the skills integrated with live agents through Oracle B2C Service Chat and Oracle Fusion Service Chat, you can use this report to track the number of conversations that were completed because skills, not human agents, answered customer questions or fulfilled customer requests.
  • Engagement Channels – A break down of each skill or digital assistant in the instance by channel usage. This chart also includes the Agent Channel (when it exists) for integrations with Oracle B2C Service.
  • Languages Used – A breakdown of each skill or digital assistant in the instance by language.

Skill- and Digital Assistant-Level Reports

The Analytics report for individual skills contains the same metrics that are rendered at the instance level, but here they're aggregated across all versions of a selected skill.
Description of total_user_utterances_sessions_versions.png follows

Traffic may have increased or decreased because of changes introduced by a new version of the skill. To access the Insights report for a particular version of the skill, click the version number to the left of the graphs.
This is an image of the version drill down links.

Skill Performance

The skill-level enables you assess the skill's completion rate, the number completed conversations taken against the total conversations a given period. The Skill Performance report includes other factors that influence the skill’s performance: the number of errors (system-handled errors, infinite loops, timeouts) and unresolved intents. For these, the report counts the number of intents that couldn’t resolve to the confidence threshold for all of incomplete conversations during the period. These metrics are aggregated across all versions of the skill. To compare the metrics for a specific version, click Compare Versions.

Compare Metrics Across Different Versions of Skills

Ideally, each new version of a skill should result in an increase in the number of completed conversations and simultaneous downward trend in the number of errors and incomplete conversations.
Description of compare_versions.png follows

To compare how different versions have impacted the success of a skill, click Compare Conversations in the Skills report to open the dialog.
This is an image of the Compare Versions link.

You can then filter the chart and KPIs by the following metrics:
  • Total Count – The number of conversations (completed, incomplete, in progress)
  • Completion Rate – The ratio of completed conversations to total conversations
  • Incomplete Rate – The ratio of incomplete conversation to total conversations
  • Unresolved – The number of unresolved conversations
You can narrow down the versions that you want to compare by selecting a version in the legend.
Description of apply_metrics.png follows

Clicking the summary tile opens the analytics for that version of the skill.