Reducing confusion

60 minutes

Prior to this session if you are planning to use an external NLU engine (Rasa/DialogFlow typically) please let us know. This should be setup and run before the session, it's likely to take a while to run so won't be practicable to do during the workshop.

• Exploring the different NLU options available - KNN, HumanFirst NLU, and external NLUs

  • Understanding the key differences including speed, cost-effectiveness, and pre-optimization of HumanFirst NLU
    • Using HumanFirst NLU for day-to-day tasks
    • Conducting goldenization test or regular verification on your selected external NLU
    • Automating calls to trigger evaluation and integrate results into your CI/CD pipeline.

• Understanding evaluation concepts

  • Understanding how and when to use a K-Fold test and a blindset
  • Navigating to intents from the Evaluation report
  • Accessing F1 and evaluation test data via the Intents or Data pane
  • Sorting by F1 and identifying the starting point of the lowest performing intents
    • Reviewing instances with zero performance asnd typical causes such as parent intents lacking training data or child intents with insufficient training

• Diving into Disambiguation

  • Pinning intents and initiating disambiguation
  • Setting a minimum confusion threshold
  • Selecting an NLU engine
  • Shifting elements back and forth between intents
  • Availability of practice dataset Disambiguation Exercise Academy Ex03

• Reviewing the Evaluation reports

  • Understanding how the evaluation powers the intent-level information on the left panel, including F1 scores and phrase counts
    • Identifying intents with high numbers of false positives (greedy intents)
    • and those with high numbers of false negatives (weak intents)
  • Understanding how infer powers the utterance pane, evaluating data using your model

• Recap on searching by Uncertainty

• Understanding Margin Score

  • Identifying phrases situated between two intents

• Transitioning analysis to use data from the chosen NLU

• Solutions within Disambiguation

  • Splitting intents to reduce greediness and establish clearer sub-intents
  • Merging confused intents that the classifier can't distinguish, potentially utilizing entities for differentiating answer types
  • Clarifying the ground truth for confused intents by redistributing utterances between them
  • Migrating to a superior classifier (e.g., transitioning from Standard to Advanced, which is straightforward for platforms like DialogFlow)

This session will equip you with the necessary insights and tools to streamline your onboarding process effectively.