Disambiguate

What is disambiguation#

Disambiguation assists in cleaning your NLU models to reduce confusion between intents. It helps by making it easy to observe and correct two types of problems:

  1. Utterances from IntentA that instead match other intents:
    In this case there is a strong likely-hood of there being mislabeled utterances within IntentA. IntentA would have a high count of False Negatives .
  2. Utterances from other intents that instead match IntentA:
    This is what we call a greedy intent, and usually surfaces intents that try to cover too broad a topic. IntentA would have a high count of False Positives.

How to disambiguate#

Disambiguation always focuses on one intent at a time - what we'll call the disambiguated intent.
You want to start by finding intents that require your attention (disambiguation candidates).

Below are useful evaluation metrics to identify potential disambiguation candidates.

  • A low F1 score is a good proxy metric to determine if an intent has any labelling issues.
  • A high False negative score indicates that there are many utterances from the intent matching other intents.
  • A high False positive score indicates that there are many utterances from other intents matching the intent.

From an evaluation report#

Open your most recent evaluation report and sort your intents by F1 score in descending order. You're looking for intents with low F1 scores. Clicking on the "open" icon of the intent will take you to that intent and allow disambiguation.

From the intent list#

Once an evaluation has run, you'll be able to sort your intents by their evaluation metrics, such as F1, False positive and False negative. Pin the intent of interest and click the "disambiguate" button.

The disambiguate interface#

Important notions#

  • Disambiguated intent: the intent from which you initiated the disambiguation experience.
  • Conflicting intent(s): other intents that are conflicting with the disambiguated intent.
  • Conflicting utterances: utterances from both the disambiguated intent and the current conflicting intent that are likely causing the problems.

The disambiguation experience will surface intents that are conflicting with the intent from which you launched disambiguation. Once a conflicting intent is selected (defaults to the first conflicting intent), you will see a side by side comparison of the utterances causing the conflict between the two intents.
You can then move utterances from one intent to the other to resolve the issues.

Settings#

Evaluation report used#

This is the evaluation report that is driving the scores seen in the disambiguation interface.
By default, disambiguate will launch by using the most recent evaluation report. You can always change the report used. However, keep in mind that older evaluation reports are less accurate.

Minimum error threshold#

A control allowing you to set a minimum threshold for conflicts. Any issues that score below this threshold will be hidden from sight, including both conflicting intents & conflicting utterances.

Conflicting intents#

Intents that are conflicting with the disambiguated intent.

Conflicting utterances#

Utterances from the disambiguated intent and the conflicting intent that are conflicting with one another. Typically, you will notice a high similarity between these utterances. The purpose of disambiguation is to move the utterances to the proper intent.