Chatbots work well when domain is well understood by the AI system. As the AI chatbot relies on NLP to understand the semantics of the input message, unless the NLP parser is trained on the domain, the accuracy of recognizing the intent and topics of interest would be very low or not as per acceptable criteria.
Take an example of a shopping chatbot which advises user what to buy based on the latest fashion trends.
Consider 3 queries below from a user –
Query 1 – Show me medium size trending black and white dresses for Christmas party
Query 2 – Show me white color, 3 inches platform heels
Query 3 – Find And black and white floral dress under 2000
Here the chatbot needs to understand the following
- Understand the shopping language.
- Understand the intent – It’s a shopping query
- Understand the domain – Its shopping query for apparel and shoes. (i.e. there can be multiple domains – grocery, electronics, books etc.)
- Understand clothing shopping category and terminology –
- Category – dresses, sandals etc.
- Variants – sizes (medium/large etc.), color (various colors and combinations like black and white), heel size (3 inches. etc.)
- Prices and ranges – 2000, etc.
- Brands like – AND, Nike etc.
Out of the box, any chatbot implementation wouldn’t understand the domain. You need to train the chatbot on the custom domain to recognize the context and the language.
For instance, out of the box NLP parsers would not recognize “AND” as a brand. Let’s inspect how well some of the leading Cloud AI NLP services recognizes the sentence – “Find And black and white floral dress under 2000”
Here is a snapshot from Watson NLP (out of the box) implementation.
Figure: Keywords from Watson NLP
Figure: Concepts from Watson NLP
Figure: Part of Speech from Watson NLP
As you see, the Watson NLP recognizes “white floral dress” as keywords and “Black” as concept. Ideally it should have recognized “black and white” as concept as we are looking for a combination of these colors.
The dress could also be a concept, as its quite generic. The floral can be keyword which has a dependency on dress. Identifying all the facts in the right way it’s important, as based on the facts you would convert this to a search query to get the required details from the data store (or from respective search indexes).
For instance, the above should result as –
Color = “black and white”
Category = “Dress”
Gender = “Female”
Price < 2000
Pattern = “floral” or Keyword within category = “floral”
(where color, category, gender, price, pattern are all the columns or indexes you are searching against.)
The Watson NLP parser doesn’t recognize “And” as brand and recognizes “And” as a conjunction (“CCONJ”) in part of speech, which is expected as its not trained on it.
Let’s check how Google NP classifies this sentence. Here is a snapshot from Google NLP.
Figure: Entity classification from Google NLP
Figure: Part of Speech from Google NLP
As you see from above figures, Google NLP identify the entity as “dress”, but doesn’t identify the colors “black and white”. With respect to part of speed tagging, its similar to Watson NLP, recognizing “And” and as conjunction (“CONJ”) and not as brand.
The above is true for any of the available NLP implementation (that is available today), where it fails to understand all the correct context of the sentence. The use case was pretty simple. Even if we train the NLP implementation on these examples, it would fall short as you need to plugin specific NLP rules for such conditions to get the desired results. As the complexity and context that needs to be inferred increases, training would also not help as you can never come up with a generalized model for such conditions. That is the single most limitation if we only rely on today’s generation of NLP implementation.
Based on my experience on building a sophisticated shopping personalized advisor, none of the out of the box AI NLP implementation fitted the requirements. A simple scenario is these 3 sets of sentences – “black and white dress”, “and black dress” and “blue jeans and white shirt”. In all these 3 examples, the use of word “and” means different meaning. In the first case, its represents a combined color ““black and white”, in second instance “and” represent a brand and in third instance two queries joined by a conjunction (i.e. and). Even with required training, a generalizing model was not possible with any of the available solutions. These are just one of the many examples I am highlighting. Imagine the complexity when dealing with medical literature. In our case, we ended up building our domain specific NLP implementation which worked for all such scenarios.
In general, while designing chatbot solutions, start with a closed domain and what kind of questions the chatbot needs to answer. Don’t start building a general purpose chatbot from start, as it would be difficult to get the required accuracy. Secondly, if you are using any cloud vendor or third party implementation, ensure your use cases can be simply solved by the default implementation or you need to build components to work around it.