We recommend that you set aside 20% percent of your corpus for intent testing and use the remaining 80% to train your intents. Keep these two sets separate so that the test utterances, which you incorporate into test cases, remain "unknown" to your skill.

Apply the 80/20 split to the each intent's data set. Randomize your utterances before making this split to allow the training models to weigh the terms and patterns in the utterances equally.

The Utterance Tester is your window to your skill's cognition. By entering phrases that are not part of the training corpus, you can find out how well you've crafted your intents by reviewing the intent confidence ranking and the returned JSON. This ranking, which is the skill's estimate for the best candidate to resolve the user input, demonstrates its acuity at the current time.

Using the Utterance Tester, you can perform quick tests for one-off testing, or you can incorporate an utterance as a test case to gauge intent resolution across different versions of training models.

1. Click Test Utterances (located at the left side).
2. If your skill supports multiple native languages, choose the testing language. Choosing this option ensures that the utterance will be added to the corresponding language version of the corpus. The skill's primary language is selected by default.
3. Enter a string of text.
4. Click Test and then take a look at the ranking and the entities detected in the utterance (if any).
5. Review the Intent Confidence scores. (The progress bars for each intent listed are green if they meet or exceed the Confidence Level or red if they fall short).
   If your skill’s top-ranking candidate isn’t what you expect, you might need to retrain the intents after doing one or both of the following:

   • Update the better candidate’s corpus with the input text that you just entered—Select the appropriate intent and then click Add to Intent.

     Caution:

     Consider how adding new test phrase might affect the training data. Adding a test phrase can change how the utterances that are similar to it get classified after retraining. In addition, adding a test phrase invalidates the test, because the incorporation of a test phrase into the training set ensures that the test will be successful. Rather than adding a test phrase to the training data, you should instead save it as a test case.

   • In the Intents page, you can edit an utterance Edit () or remove it. A FAQ intent, for example, might receive a top rank because of the scope and phrasing of its constituent utterances. If you don’t want your users to get a FAQ whenever they ask typical questions, you’ll need to revise the corpus.

   You need to retrain an intent whenever you add, change, or delete an utterance. Training Needed displays whenever you make a change to the training data.

6. If your intents aren't resolving as intended, you can expand the JSON window to review the matched intents, scores, and detected entities in the returned JSON.
7. Click Reset.

Each test has an utterance and the intent that it's expected to resolve to, which is known as a label match. A test case can also include matching entity values and the expected language for the utterance. You can run test cases when you’re developing a skill and later on, when the skill is in production, you can use the test cases for regression testing. In the latter case, you can run test cases to find out if a new release of the training model has negatively affected intent resolution.

Like the test cases that you create with the Conversation Tester, utterance test cases are part of the skill and are carried along with each version. If you extend a skill, then the extension inherits the test cases. Whereas conversation test cases are intended to test a scenario, utterance test cases are intended to test fragments of a conversation independently, ensuring that each utterance resolves to the correct intent.

You can find out how similar your test phrase is to the utterances in the training corpus by clicking View Similar Utterances. This tool provides you with an added perspective on the skill's training data by showing you how similar its utterances are to the test phrase, and by extension, how similar the utterances are to one another across intents. Using this tool, you can find out if the similarity of the test phrase to utterances belonging to other intents is the reason why the test phrase is not resolving as expected. It might even point out where training data belongs to the wrong intent because if its similarity to the test phrase.

The list generated by this tool ranks 20 utterances (along with their associated intents) that are closest to the test phrase. Ideally, the top-ranking utterance on this list – the one most like the test phrase – belongs to the intent that's targeted for the test phrase. If the closest utterance that belongs to the expected intent is further down, then a review of the list might provide a few hints as to why. For example, if you're testing a Transactions intent utterance, how much money did I transfer yesterday?, you'd expect the top-ranking utterance to likewise belong to a Transactions intent. However, if this test utterance is resolving to the wrong intent, or resolving below the confidence level, the list might reveal that it has more in common with highly ranked utterances with similar wording that belong to other intents. The Balances intent's How much money do I have in all of my accounts?, for example, might be closer to the test utterance than the Transactions intent's lower-ranked How much did I deposit in April? utterance.

You can access the list, which is generated for skills trained on Trainer Tm, by clicking View Similar Utterances in the Utterance Tester or from the Test Cases report.

Note

You can only use this tool for skills trained on Trainer Tm (it's not available for skills trained with Ht).