1. Click + Add Entity.
2. Complete the Create Entity dialog. Keep in mind that the Name and Description appear in the crowd worker pages for Entity Annotation Jobs.
   • Enter a name that identifies the annotated content. A unique name helps crowd workers.
   • Enter a description. Although this is an optional property, crowd workers use it, along with the Name property, to differentiate entities.
   • Choose ML Entity from the list.
3. Switch on Exclude System Entity Matches when the training annotations contain names, locations, numbers, or other content that could potentially clash with system entity values. Setting this option prevents the model from extracting system entity values that are within the input that's resolved to this ML entity. It enforces a boundary around this input so that the model recognizes it only as an ML entity value and does not parse it further for system entity values. You can set this option for composite bag entities that reference ML entities.
4. Click Create.
5. Click +Value List Entities to associate this entity with up to five Value List Entities. This is optional, but associating an ML Entity with a Value List Entity combines the contextual extraction of the ML Entity and the context-agnostic extraction of the Value List Entity.
6. Click the DataSet tab. This page lists all the utterances for each ML Entity in your skill, which include the utterances that you've added yourself to bootstrap the entity, those submitted from crowd sourcing jobs, or have been imported as JSON objects. From this page, you can add utterances manually or in bulk by uploading a JSON file. You can also manage the utterances from this page by editing them (including annotating or re-annotating them), or by deleting, importing, and exporting them.
   • Add utterances manually:
     • Click Add Utterance. After you've added the utterance, click Edit Annotations to open the Entity List.
       Note
       
       You can only add one utterance at a time. If you want to add utterances in bulk, you can either add them through an Entity Annotation job, or you can upload a JSON file.
     • Highlight the text relevant to the ML Entity, then complete the labeling by selecting the ML Entity from the Entity List. You can remove an annotation by clicking x in the label.
       
       
       
       
   • Add utterances from a JSON file. This JSON file contains a list of utterance objects.

     [
       {
         "Utterance": {
           "utterance": "I expensed $35.64 for group lunch at Joe's on 4/7/21",
           "languageTag": "en",
           "entities": [
             {
               "entityValue": "Joe's"   
               "entityName": "VendorName",
               "beginOffset": 37,
               "endOffset": 42
             }
           ]
         }
       },
       {
         "Utterance": {
           "utterance": "Give me my $30 for Coffee Klatch on 7/20",
           "languageTag": "en",
           "entities": [
             {
               "entityName": "VendorName",
               "beginOffset": 19,
               "endOffset": 32
             }
           ]
         }
       }
     ]

     You can upload it by clicking More > Import to retrieve it from your local system.
     The entities object describes the ML entities that have been identified within the utterance. Although the preceding example illustrates a single entities object for each utterance, an utterance may contain multiple ML entities which means multiple entities objects:

     [
       {
         "Utterance": {
           "utterance": "I want this and that",
           "languageTag": "en",
           "entities": [
             {
               "entityName": "ML_This",
               "beginOffset": 7,
               "endOffset": 11
             },
             {
               "entityName": "ML_That",
               "beginOffset": 16,
               "endOffset": 20
             }
           ]
         }
       },
       {
         "Utterance": {
           "utterance": "I want less of this and none of that",
           "languageTag": "en",
           "entities": [
             {
               "entityName": "ML_This",
               "beginOffset": 15,
               "endOffset": 19
             },
             {
               "entityName": "ML_That",
               "beginOffset": 32,
               "endOffset": 36
             }
           ]
         }
       }
     ]

     entityName identifies the ML Entity itself and entityValue identifies the text labeled for the entity. entityValue is an optional key that you can use to validate the labeled text against changes made to the utterance. The label itself is identified by the beginOffset and endOffset properties, which represent the offset for the characters that begin and end the label. This offset is determined by character, not by word, and is calculated from the first character of the utterance (0-1).

     Note
     
     You can't create the ML Entities from this JSON. They must exist before you upload the file.
     If you don't want to determine the offsets, you can leave the entities object undefined and then apply the labels after you upload the JSON file.

     [
       {
         "Utterance": {
           "utterance": "I expensed $35.64 for group lunch at Joe's on 4/7/21",
           "languageTag": "en",
           "entities": []
             
           
         }
       },
       {
         "Utterance": {
           "utterance": "Give me my $30 for Coffee Klatch on 7/20",
           "languageTag": "en",
           "entities": []
           
         }
       }
     ]

     The system checks for duplicates to prevent redundant entries. Only changes made to the entities definition in the JSON file are applied. If an utterance has been changed in the JSON file, then it's considered a new utterance.
   • Edit an annotated utterance:
     • Click Edit to remove the annotation.
       Note
       
       A modified utterance is considered a new (unannotated) utterance.
     • Click Edit Annotations to open the Entity List.
     • Highlight the text, then select an ML Entity from the Entity List.
     • If you need to remove an annotation, click x in the label.
7. When you've completed annotating the utterances. Click Train to update both trainer Tm and the Entity model.
8. Test the recognition by entering a test phrase in the Utterance Tester, ideally one with a value not found in any training data. Check the results to find out if the model detected the correct ML Entity and if the text has been labeled correctly and completely.
9. Associate the ML Entity with an intent.

entity,value,synonyms
PizzaSize,Large,lrg:lrge:big
PizzaSize,Medium,med
PizzaSize,Small,little

entity,en:value,en:synonyms
PizzaSize,Large,lrg:lrge:big
PizzaSize,Medium,med
PizzaSize,Small,
PizzaSize,Extra Large,XL

entity,en:value,en:synonyms,fr:value,fr:synonyms
PizzaSize,Large,lrg:lrge:big,grande,grde:g
PizzaSize,Medium,med,moyenne,moy
PizzaSize,Small,,petite,p
PizzaSize,Extra Large,XL,pizza extra large,

• If you import a pre-20.12 CSV into a 20.12 skill (including those that support native languages or use translation services), the values and synonyms are imported as primary languages.
• All entity values for both the primary and secondary languages must be unique within an entity, so you can't import a CSV if the same value has been defined more than once for a single entity. Duplicate values may occur in pre-20.12 versions, where values can be considered unique because of variations in letter casing. This is not true for 20.12, where casing is more strictly enforced. For example, you can't import a CSV if it has both PizzaSize, Small and PizzaSize, SMALL. If you plan to upgrade Version 20.12, you must first resolve all entity values that are the same, but differentiated only by letter casing before performing the upgrade.
• Primary language support applies to skills created using Version 20.12 and higher, so you must first remove language tags and any secondary language entries before you can import a Version 20.12 CSV into a skill created with a prior version.

• You can import a multi-lingual CSV into skills that do not use native language support, including those that use translation services.
• If you import a multi-lingual CSV into a skill that supports native languages or uses translation services, then only rows that provide a valid value for the primary language are imported. The rest are ignored.

1. Click Entities () in the side navbar.

2. Click More, choose Import Value list entities, and then select the .csv file from your local system.
   
   
   

3. Add the entity or entities to an intent (or to an entity list and then to an intent).

• The CSVs exported from skills created with, or upgraded to, Version 20.12 are equipped for native language support though the primary (and sometimes secondary) language tags that are appended to the value and synonyms columns. For example, the CSV in the following snippet has a set of value and synonyms columns for the skill's primary language, English (en) and another set for its secondary language, French (fr):

  entity,en:value,en:synonyms,fr:value,fr:synonyms

  The primary language tags are included in all 20.12 CSVs regardless of native language support. They are present in skills that are not intended to perform any type of translation (native or through a translation service) and in skills that use translation services.
• The CSVs exported from skills running on versions prior to 20.12 have the entity, value, and synonyms columns, but no language tags.

1. Click Entities () in the side navbar.

2. Click More, choose Export Value list entities and then save the file.
   
   
   

   The exported .csv file is named for your skill. If you're going to use this file as an import, then you may need to perform some of the edits described in Import Intents from a CSV File if you're going to import it to, or export it from, Version 20.12 skills and prior versions.

1. Click + Entity.
2. Choose Dynamic Entities from the Type list.
3. If the backend service is unavailable or hasn't yet pushed any values, or if you do not maintain the service, click + Value to add mock values that you can use for testing purposes. Typically, you would add these static values before the dynamic entity infrastructure is in place. These values are lost when you clone, version, or export a skill. After you provision the entity values through the API, you can overwrite, or retain, these values (though in most cases you would overwrite them).
4. Click Create.

• You can query for the dynamic entities configured for a skill using the generated entityId with the botId. You include these values in the calls to create the push requests and objects that update the entity values.
• An entity cannot have more than 150,000 values. To reduce the likelihood of exceeding this limit when you're dealing with large amounts of data, send PATCH requests with your deletions before you send PATCH requests with your additions.

1. Create concise ML Entities. The ML Entity definition is at the base of a useful training set, so clarity is key in terms of its name and the description which help crowd workers annotate utterances.

   Because crowd workers rely on the ML Entity descriptions and names, you must ensure that your ML Entities are easily distinguishable from each other, especially when there's potential overlap. If the differences are not clear to you, it's likely that crowd workers will be confused. For example, the Merchant and Account Type entities may be difficult to differentiate in some cases. In "Transfer $100 from my savings account to Pacific Gas and Electric," you can clearly label "savings" as Account Type and Pacific Gas and Electric as Merchant. However, the boundary between the two can be blurred in sentences like "Need to send money to John, transfer $100 from my savings to his checking account." Is "checking account" an Account type, or a Merchant name? In this case, you may decide that any recipient should always be a merchant name rather than an account type.

2. In preparation of crowd sourcing the training utterances, consider the typical user input for different entity extraction contexts. For example, can the value be extracted in the user's initial message (initial utterance context), or is it extracted from responses to the skill's prompts (slot utterance context)?

   Context	Description	Example Utterances (detected ML Entity values in bold)
   Initial utterance context	A message that's usually well-structured and includes ML Entity values. For an expense reporting skill, for example, the utterance would include a value that the model can detect for an ML Entity called Merchant.	Create an expense for team dinner at John's Pasta Shop for $85 on May 3
   Slot utterance context	A user message that provides the ML Entity in response to a prompt, either because of conversation design (the skill prompts with "Who is the merchant?") or to slot a value because it hasn't been provided by a previously submitted response. In other circumstances, the ML Entity value may have already been provided, but may be included in other user messages in the same conversation. For example, the skill might prompt users to provide additional expense details or describe the image of an uploaded receipt.	Merchant is John's Pasta Shop. Team dinner. Amount $85. John's Pasta Shop. Description is TurboTaxi from home to CMH airport. Grandiose Shack Hotel receipt for cloud symposium

3. Gather your training and testing data.
   • If you already have a sufficient collection of utterances, you may want to assess them for entity distribution and entity value diversity before you launch an Entity Annotation job.
   • If you don't have enough training data, or if you're starting from scratch, launch an Intent Paraphrasing Job. To gather viable (and abundant) utterances for training and testing, integrate the entity context into the job by creating tasks for each intent. To gather diverse phrases, consider breaking down each intent by conversation context.
   • For the task's prompt, provide crowd workers context and ask them, "How would you respond?" or "What would you say?" Use the accompanying hints to provide examples and to illustrate different contexts. For example:

     Prompt	Hint
     You're talking to an expense reporting bot, and you want to create an expense. What would be the first thing you would say?	Ensure that the merchant name is in the utterance. You might say something like, "Create an expense for team dinner at John's Pasta Shop for $85 on May 3."

     This task asks for phrases that not only initiate the conversation, but also include a merchant name. You might also want utterances that reflect responses prompted by the skill when the user doesn't provide a value. For example, "Merchant is John's Pasta Shop" in response to the skill's "Who is the merchant?" prompt.

     Prompt	Hint
     You've submitted an expense to the an expense reporting bot, but didn't provide a merchant name. How would you respond?	Identify the merchant. For example, "Merchant is John's Pasta Shop."
     You've uploaded an image of a receipt to an expense reporting bot. It's now asking you to describe the receipt. How would you respond?	Identify the merchant's name on the receipt. For example: "Grandiose Shack Hotel receipt for cloud symposium."

     To test false positives for testing – words and phrases that the model should not identify as ML Entities – you may also want to collect "negative examples". These utterances do include an ML Entity value.

     Context	Example Utterances
     Initial utterance context	Pay me back for Tuesday's dinner
     Slot utterance context	Pos presentation dinner. Amount $50. 4 people. Description xerox lunch for 5 Hotel receipt for interview stay

   • Gather a large training set by setting an appropriate number of paraphrases per intent. For the model to generalize successfully, your data set must contain somewhere between 500 and 5000 occurrences for each ML entity. Ideally, you should avoid the low end of this range.
4. Once the crowd workers have completed the job (or have completed enough utterances that you can cancel the job), you can either add the utterances, or launch an Intent Validation job to verify them. You can also download the results to your local system for additional review.
5. Reserve about 20% of the utterances for testing. To create CSVs for the Utterance Tester from the downloaded CSVs for Intent Paraphrasing and Intent Validation jobs:
   • For Intent Paraphrasing jobs: transfer the contents in the result column (the utterances provided by crowd workers) to the utterance column in the Utterance Tester CSV. Transfer the contents of the intentName column to the expectedIntent column in the Utterance Tester CSV.
   • For Intent Validation jobs: transfer the contents in the prompt column (the utterances provided by crowd workers) to the utterance column in the Utterance Tester CSV. Transfer the contents of the intentName column to the expectedIntent column in the Utterance Tester CSV.
6. Add the remaining utterances to a CSV file with a single column, utterance. Create an Entity Annotation Job by uploading this CSV. Because workers are labeling the entity values, they will likely classify negative utterances as "I'm not sure" or "None of the entities apply."
7. After the Entity Annotation job is complete, you can add the results, or you can launch an Entity Validation job to verify the labeling. Only the utterances that workers deem correct in an Entity Validation job can be added to the corpus.

   Tip:

   You can add, remove, or adjust the annotation labels in the Dataset tab of the Entities page.
8. Train the entity by selecting Entity.
9. Run test cases to evaluate entity recognition using the utterances that you reserved from the Intent Paraphrasing job. You can divide up these utterances into different test suites to test different behaviors (unknown values, punctuation that may not be present in the training data, false positives, and so on). Because there may be a large number of these utterances, you can create test suites by uploading a CSV into the Utterance Tester.
   
   
   
   
   
   Note
   
   The Utterance Tester only displays entity labels for passing test cases. Use a Quick Test instead to view the labels for utterances that resolve below the confidence threshold.
10. Use the results to refine the data set. Iteratively add, remove, or edit the training utterances until test run results indicate the model is effectively identifying ML Entities.
    Note
    
    To prevent inadvertant entity matches that degrade the user experience, switch on Exclude System Entity Matches if the training data contains names, locations, numbers.