Natural Language Processing Chatbots: The Beginner's Guide

What is NLP? How does it work? How is it used? What are the workings and advantages of a natural language processing chatbot and do you actually need one? We break down this AI sub-domain with focus on its application in chatbot technologies in plain terms, to make the concept comprehensible and accessible to anyone - including non-techies.‍

Chatbots, though they have been on the business technology scene for quite a few years, are still a hot topic. Generally speaking, you get two types of approaches to their creation:

• Companies determined to develop bots capable of natural conversations indistinguishable from human ones using NLP and machine learning;
• Companies focused on functional bots good at accomplishing specific tasks and do so quickly and efficiently, making the bots’ perceived humanity a secondary matter.

The issue is that with all the hype around artificial intelligence, it’s often difficult to maintain perspective; distinguish fact from fiction; manage user expectations and customer experience, or simply resist being seduced by technology for its own sake (rather than keeping your business/marketing goal in mind).

Frankly, a chatbot doesn’t necessarily need to fool you into thinking it's human to be successful in completing its raison d'être. At this stage of tech development, trying to do that would be a huge mistake rather than help.

In fact, while any talk of chatbots is usually accompanied by the mention of AI, machine learning and natural language processing (NLP), many highly efficient bots are pretty “dumb” and far from appearing human.

Rule-based bots with option buttons, yes-no questions, keyword recognition, and quick answer suggestions seem to generate great results. Sometimes even better than NLP!

Nevertheless, this doesn’t mean you should give up on it. Natural Language Processing does have an important role in the matrix of bot development and business operations alike. The key to successful application of NLP is understanding how and when to use it.

To make the sense of it all, we broke down the essence NLP, its inner workings and possible uses.

Why Machines Need NLP?

Everything we express in written or verbal form encompasses a huge amount of information that goes way beyond the meaning of individual words.

The combination of topic, tone, selection of words, sentence structure, punctuation/expressions allows humans to interpret that information, its value, and intent.

Theoretically, humans are programmed to understand and often even predict other people’s behavior using that complex set of information.

The problem?

One person can generate hundreds of words in a declaration, each sentence with its own complexity and contextual undertone.

To scale bot’s understanding means analyzing several hundred or thousands of people and their possible declarations which differ further from one geography to the next...This kind of conversational data is called unstructured data and can’t be fit into neatly stacked rows and columns.

It’s complicated, messy and hard to manipulate.

The only way to teach a machine about all that, is to let it learn from experience.

What is NLP?

Natural language is the language humans use to communicate with one another. On the other hand, programming language was developed so humans can tell machines what to do in a way machines can understand.

For example, English is a natural language while Java is a programming one.

Natural Language Processing facilitates human-to-machine communication without humans needing to “speak” Java or any other programming language as it allows machines to obtain and process information from written or verbal user inputs.

In essence, a chatbot developer creates NLP models that enable computers to decode and even mimic the way humans communicate.

Unlike common word processing operations, NLP doesn’t treat speech or text just as a sequence of symbols. It also takes into consideration the hierarchical structure of the natural language - words create phrases; phrases form sentences;  sentences turn into coherent ideas.

In other words, NLP software doesn’t just look for keywords. It uses pre-programmed or acquired knowledge to decode meaning and intent from factors such as sentence structure, context, idioms, etc.

For instance, good NLP software should be able to recognize whether the user’s “Why not?” indicates agreement or a question that requires an answer.

NLP is divided in two key categories:

1. Natural Language Understanding (NLU)
2. Natural Language Generation (NLG)

How Does NLP Fit into the AI World?

The words AI, NLP, and ML (machine learning) are sometimes used almost interchangeably. However, there is an order to the madness of their relationship.

Hierarchically, natural language processing is considered a subset of machine learning while NLP and ML both fall under the larger category of artificial intelligence.

• Artificial intelligence is one of the main subfields of computer science created in the 60s concerned with programming machines to solve tasks intrinsic to humans but hard for computers. A successful AI could do anything a human can (except the purely physical things) including moving around in the world, planning, recognizing objects and sounds, speaking, translating, performing social or business transactions, produce creative work such as making art or poetry, and so forth. Needless to say, we are still very far from creating anything close to that “ideal”.
• Natural language processing is the linguistically oriented discipline in computer science that is concerned with the capacity of software to understand natural human language - written as well as spoken.
• Machine learning is focused on creating a software system that can learn from their own observations and past experience.

Since, when it comes to our natural language, there is such an abundance of different types of inputs and scenarios, it’s impossible for any one developer to program for every case imaginable. Hence, for natural language processing in AI to truly work, it must be supported by machine learning.

Simply put, machine learning allows the NLP algorithm to learn from every new conversation and thus improve itself autonomously through practice.

NLP is not Just About Creating Intelligent Chatbots...

NLP is a tool for computers to analyze, comprehend, and derive meaning from natural language in an intelligent and useful way. This goes way beyond the most recently developed chatbots and smart virtual assistants. In fact, natural language processing algorithms are everywhere from search, online translation, spam filters and spell checking.

So, by using NLP, developers can organize and structure the mass of unstructured data to perform tasks such as intelligent:

• Automatic summarization (intelligently shortening long pieces of text)
• Automatic suggestions (used to speed up writing of emails, messages, and other texts)
• Translation (translating phrases and ideas instead of word for word)
• Named entity recognition (used to locate and classify named entities in unstructured natural languages into pre-defined categories such as the organizations; person names; locations; codes; quantities; price; time; percentages, etc.)
• Relationship extraction (extraction of semantic relationships among the identified entities in natural language text/speech such as “is located in”, “is married to”, “is employed by”, “lives in”, etc.)
• Sentiment analysis (helps identify, for instance, positive, negative and neutral opinion form text or speech widely used to gain insights from social media comments, forums or survey responses)
• Speech recognition (enables computers to recognize and transform spoken language into text - dictation - and, if programmed, act upon that recognition - e.g. in case of assistants like Google Assistant Cortana or Apple’s Siri)
• Topic segmentation (automatically divides written texts, speech or recordings into shorter, topically coherent segments and is used in improving information retrieval or speech recognition)

How Does Natural Language Processing Work?

One of the best things about NLP is that it’s probably the easiest part of AI to explain to non-technical people.

Take one of the most common natural language processing application examples — the prediction algorithm in your email. The software is not just guessing what you will want to say next but analyzes the likelihood of it based on tone and topic. Engineers are able to do this by giving the computer and “NLP training”.

‍In other words, they provide the software with a huge amount of data about language including sentences and phrases as well as transcripts from live conversations/emails. This way, over time, computer programs are able to learn how to pair words together; what it is we are trying to convey, and what we need to achieve with that communication.

Naturally, predicting what you will type in a business email is significantly simpler than understanding and responding to a conversation.

Still, the decoding/understanding of the text is, in both cases, largely based on the same principle of classification.

Generally, the “understanding” of the natural language (NLU) happens through the analysis of the text or speech input using a hierarchy of classification models.

Every model helps the next by narrowing down the scope until the computer gets to the final “understanding” stage.

• Domain Classifier segments natural input into one of a pre-set group of conversational domains. This is only necessary for solutions that have to handle conversations concerning varied topics, requiring specialized vocabulary each. For example, being able to classify a domain is essential for virtual assistants such as Siri. Assistant’s domain classifiers are likely to include domains such as weather, sports, navigation or music, among others.
• Intent Classifier does exactly what you think it does. It determines what the person is trying to accomplish by assigning every input and contributing it to one of the intents specified in your NLP algorithm. Example intents include “find the nearest store”, “find opening hours”, “find a product”, etc.
• Entity Recognizer extracts the words and phrases (entities) which are essential to fulfilling the user’s query/intent. E.g, if the user is trying to book a table at your restaurant the needed entities under this intent, would include time, date and number of guests.
• Role Classifiers are further differentiation labels you can apply to your entities if it's possible for that entity to have different meanings/consequences based on context. E.g., you can classify the time entity further by labelling it “closed” and “open”.

Following the logic of classification, whenever the NLP algorithm classifies the intent and entities needed to fulfil it, the system (or bot) is able to “understand” and so provide an action or a quick response.

Three Pillars of an NLP Based Chatbot

Now it's time to take a closer look at all the core elements that make NLP chatbot happen.

1) Dialog System

To communicate, people use mouths to speak, ears to hear, fingers to type, and eyes to read.

Chatbot, too, needs to have an interface compatible with the ways humans receive and share information with communication. That is what we call a dialog system, or else, a conversational agent.

There are no set dialog system components.

But for a dialog system to, indeed, be a dialog system, it has to be capable of producing output and accept input. Other than that, they can adopt a variety of forms. You can differentiate them based on:

• Modality (text-based, speech-based, graphical or mixed)
• Device
• Style (command-based, menu-driven and - of course - natural language)
• Initiative (system, user, or mixed)

2) Natural Language Understanding

So, you already know NLU is an essential sub-domain of NLP and have a general idea of how it works.

Still, it’s important to point out that the ability to process what the user is saying is probably the most obvious weakness in NLP based chatbots today. Human languages are just way too complex. Besides enormous vocabularies, they are filled with multiple meanings many of which are completely unrelated.

To nail the NLU is more important than making the bot sound 110% human with impeccable NLG.

Why?

If a bot understands the users and fulfils their intent, most won’t care if that response is a bit taciturn… It doesn't work the other way around. A bot that can’t derive meaning from the natural input efficiently can have the smoothest small talk skills and nobody will care. Not even a little!

3) Natural Language Generation

Given that the NLP chatbot successfully parsed and understood the user’s input, its programming will determine an appropriate response and “translate” it back to natural language.Needless to say, that response doesn’t appear out of thin air.

For the NLP to produce a human-friendly narrative, the format of the content must be outlined be it through rules-based workflows, templates, or intent-driven approaches. In other words, the bot must have something to work with in order to create that output.

‍Currently, every NLG system relies on narrative design - also called conversation design - to produce that output. This narrative design is guided by rules known as “conditional logic”.

These rules trigger different outputs based on which conditions are being met and which are not.

Do You Need an NLP Chatbot?

Let’s be clear.

Using NLP for simple and straightforward use cases is over the top and completely unnecessary.

In fact, if used in an inappropriate context, natural language processing chatbot can be an absolute buzzkill and hurt rather than help your business. If a task can be accomplished in just a couple of clicks, making the user type it all up is most certainly not making things easier.

On the other hand, if the alternative means presenting the user with an excessive number of options at once, NLP chatbot can be useful. It can save your clients from confusion/frustration by simply asking them to type or say what they want.

It’s not much different from coming up to the staff member at the counter in the real world. AI is cool but if it fails to be useful, no one will really care how “modern” your company is.

What Can NLP Chatbots Learn From Rule-Based Bots

There are many who will argue that a chatbot not using AI and natural language isn’t even a chatbot but just a mare auto-response sequence on a messaging-like interface.

You have two choices here.

You can decide to stay hung up on nomenclature or create a chatbot capable of completing tasks, achieving goals and delivering results.Being obsessed with the purity of AI bot experience is just not good for business.

In fact, when it comes down to it, your NLP bot can learn A LOT about efficiency and practicality from those rule-based “auto-response sequences” we dare to call chatbots.

1) Constrain the Input & Leverage Rich Controls

Why would you make anyone type out a message if a quick tap or click can do the trick?

At times, constraining user input can be a great way to focus and speed up query resolution.

So, when logical, falling back upon rich elements such as buttons, carousels or quick replies won’t make your bot seem any less intelligent.

To the contrary…Besides the speed, rich controls also help to reduce users’ cognitive load. Hence, they don’t need to wonder about what is the right thing to say or ask.When in doubt, always opt for simplicity.

2) Do the Dialog Flow Diagram

NLP bots generate responses based on user inputs.

So, technically, designing a conversation doesn’t require you to draw up a diagram of the conversation flow.However!Having a branching diagram of the possible conversation paths helps you think through what you are building.

Consequently, it's easier to design a natural-sounding, fluent narrative. You can draw up your map the old fashion way or use a digital tool. Both Landbot’s visual bot builder or any mind-mapping software will serve the purpose well.

3) Define End to the Conversation

Lack of a conversation ender can easily become an issue and you would be surprised how many NLB chatbots actually don’t have one.

If the user isn’t sure whether or not the conversation has ended your bot might end up looking stupid or it will force you to work on further intents that would have otherwise been unnecessary.

Rule and choice-based bots don’t have this issue. Once it’s over, it’s over.

No confusion there. Hence, make it clear the conversation has ended, verbally or visually.

You can even offer additional instructions to relaunch the conversation.

4) Don’t Get Caught Up in Handling Corner Cases

Bots that don’t use AI don’t care about corner cases. They take people down one of the outlined paths and it's over.

There is a lesson here… don’t hinder the bot creation process by handling corner cases. Especially so if you are still in your prototyping phase.

Focus on developing the core intents and developing them well. Don't waste your time focusing on use cases that are highly unlikely to occur any time soon. You can come back to those when your bot is popular and the probability of that corner case taking place is more significant.

If you really want to feel safe, if the user isn’t getting the answers he or she wants, you can set up a trigger for human agent takeover.

5) Design for an Easy Conversation Restart

Include a restart button and make it obvious.Just because it’s a supposedly intelligent natural language processing chatbot, it doesn’t mean users can’t get frustrated with or make the conversation “go wrong”.

Save your users/clients/visitors the frustration and allows to restart the conversation whenever they see fit.

Can you Build NLP Chatbot Without Coding?

Unfortunately, a no-code natural language processing chatbot is still a fantasy. You need an experienced developer/narrative designer to build the classification system and train the bot to understand and generate human-friendly responses.

However, there are tools that can help you significantly simplify the process.

For example, one of the most widely used NLP chatbot development platforms is Google’s Dialogflow which connects to the Google Cloud Platform.

By giving developers/narrative designers a clean and user-friendly interface and taking care of the natural langue processing and machine learning and other deeper concepts “behind the scene”, it allows you to focus on the conversation flow and build bots.

NLP is far from being simple even with the use of a tool such as DialogFlow. However, it does make the task at hand more comprehensible and manageable.

Another thing you can do to simplify your NLP chatbot building process is using a visual no-code bot builder - like Landbot - as your base in which you integrate the NLP element.

We might be a bit biased about Landbot. BUT, when it comes to streamlining the entire process of bot creation, it’s hard to argue against it. While the builder is usually used to create a choose-your-adventure type of conversational flows, it does allow for Dialogflow integration.

This means you can offer your users a smart NLP based assistant while taking advantage of no-code features in the creation and management process such as digital tool integration; human takeover; rich controls; front-end bot design; web/website/messaging app integration and so forth…

Even better?

The use of Dialogflow and a no-code chatbot building platform like Landbot allows you to combine the smart and natural aspects of NLP with the practical and functional aspects of choice-based bots.