What is natural language understanding?

What is Natural Language Understanding (NLU)?

Natural Language Understanding (NLU) is a field of computer science which analyses what human language means, rather than simply what individual words say.

Natural Language Understanding seeks to intuit many of the connotations and implications that are innate in human communication such as the emotion, effort, intent, or goal behind a speaker’s statement. It uses algorithms and artificial intelligence, backed by large libraries of information, to understand our language.

Rather than relying on computer language syntax, Natural Language Understanding enables computers to comprehend and respond accurately to the sentiments expressed in natural language text.

What is the difference between Natural Language Understanding (NLU) and Natural Language Processing (NLP)?

Natural Language Understanding is a subset area of research and development that relies on foundational elements from Natural Language Processing (NLP) systems, which map out linguistic elements and structures. Natural Language Processing focuses on the creation of systems to understand human language, whereas Natural Language Understanding seeks to establish comprehension.

What is Natural Language Generation?

Natural Language Generation is the production of human language content through software. It transforms data into a language translation that we can understand. It is often used in response to Natural Language Understanding processes.

An example might be using a voice assistant to answer a query. The voice assistant uses the framework of Natural Language Processing to understand what is being said, and it uses Natural Language Generation to respond in a human-like manner. There is Natural Language Understanding at work as well, helping the voice assistant to judge the intention of the question.

Image Source: IBM

How does Natural Language Understanding (NLU) work?

Breaking down human language into a system using computational linguistics

In the Natural Language Understanding process, first the software needs to extract information from the data source (verbal or textual). This unstructured data could look like the following example:

Typed request into a search engine:

“tickets New York to Miami 25 April 8pm”

Natural Language Understanding analysis:

Tickets [intent to buy flights]
New York [location]
Miami [location]
25 April [date]
8pm [time]

Natural Language Understanding deconstructs human speech using trained algorithms until it forms a structured ontology, or a set of concepts and categories that have established relationships with one another. This computational linguistics data model is then applied to text or speech as in the example above, first identifying key parts of the language.

Two key concepts in natural language processing are intent recognition and entity recognition. Using the example, we can understand each type.

Intent recognition identifies what the person speaking or writing intends to do. Identifying their objective helps the software to understand what the goal of the interaction is. In this example, the NLU technology is able to surmise that the person wants to purchase tickets, and the most likely mode of travel is by airplane. The search engine, using Natural Language Understanding, would likely respond by showing search results that offer flight ticket purchases.

Entity recognition identifies which distinct entities are present in the text or speech, helping the software to understand the key information. Named entities would be divided into categories, such as people’s names, business names and geographical locations. Numeric entities would be divided into number-based categories, such as quantities, dates, times, percentages and currencies. In our example, “25 April” and “8pm” are numeric entities.

Using data modelling to learn what we really mean

Of course, Natural Language Understanding can only function well if the algorithms and machine learning that form its backbone have been adequately trained, with a significant database of information provided for it to refer to. Without sophisticated software, understanding implicit factors is difficult.

NLU tools should be able to tag and categorise the text they encounter appropriately. In order to categorise or tag texts with humanistic dimensions such as emotion, effort, intent, motive, intensity, and more, Natural Language Understanding systems leverage both rules based and statistical machine learning approaches.

In our example, the Natural Language Understanding-enabled search engine is able to infer intent because past statistical machine learning data tells it that requests for tickets for these locations usually result in users clicking through to purchase flight tickets. Knowledge of that relationship and subsequent action helps to strengthen the model.

Taking action and forming a response

Without a strong relational model, the resulting response isn’t likely to be what the user intends to find. The key aim of any Natural Language Understanding-based tool is to respond appropriately to the input in a way that the user will understand.

Using our example, an unsophisticated software tool could respond by showing data for all types of transport, and display timetable information rather than links for purchasing tickets. Without being able to infer intent accurately, the user won’t get the response they’re looking for.

Why is Natural Language Understanding important?

Wondering how natural language understanding can help you? We’ve got you covered.

Here are a few reasons why Natural Language Understanding is increasingly vital for business:

Human language is complicated for computers to grasp

Human language is fluid, complex and full of subtleties. Two people may read or listen to the same passage and walk away with completely different interpretations. If humans struggle to develop perfectly aligned understanding of human language due to these congenital linguistic challenges, it stands to reason that machines will struggle when encountering this unstructured data.

NLU can help equip many types of technology with a similar level of understanding to humans, even down to parsing typing errors and incorrect naming. Natural Language Understanding can be used for:

• Internal and external email responses
• Social media comments
• Chat bots discussions
• Internet search queries
• Interactive Voice Response (IVR) systems for calls
• Voice assistants
• And more

There’s a growing need for understanding at scale

The NLU field is dedicated to developing strategies and techniques for understanding context in individual records and at scale. NLU systems empower analysts to distill large volumes of unstructured text into coherent groups without reading them one by one. This allows us to resolve tasks such as content analysis, topic modelling, machine translation, and question answering at volumes that would be impossible to achieve using human effort alone.

This is particularly important, given the scale of unstructured text that is generated on an everyday basis. NLU-enabled technology will be needed to get the most out of this information, and save you time, money and energy to respond in a way that consumers will appreciate.

It’s already being used by millions of businesses and consumers

The NLP market is predicted reach more than $43 billion in 2025, nearly 14 times more than it was in 2017. Millions of businesses already use NLU-based technology to analyse human input and gather actionable insights.

There are 4.95 billion internet users globally, 4.62 billion social media users, and over two thirds of the world using mobile, and all of them will likely encounter and expect NLU-based responses. Consumers are accustomed to getting a sophisticated reply to their individual, unique input – 20% of Google searches are now done by voice, for example. Without using NLU tools in your business, you’re limiting the customer experience you can provide.

Customers expect to be heard as individuals

In our research, we’ve found that more than 60% of consumers think that businesses need to care more about them, and would buy more if they felt the company cared. Part of this care is not only being able to adequately meet expectations for customer experience, but to provide a personalised experience. Accenture reports that 91% of consumers say they are more likely to shop with companies that provide offers and recommendations that are relevant to them specifically.

Trying to meet customers on an individual level is difficult when the scale is so vast. Rather than using human resource to provide a tailored experience, NLU software can capture, process and react to the large quantities of unstructured data that customers provide at scale.

You can reduce costs and internal effort

Let’s take an example of how you could lower call centre costs and improve customer satisfaction using NLU-based technology.

Your software can take a statistical sample of recorded calls and perform speech recognition after transcribing the calls to text using machine translation. The NLU-based text analysis can link specific speech patterns to negative emotions and high effort levels. Using predictive modelling algorithms, you can identify these speech patterns automatically in forthcoming calls and recommend a response from your customer service representatives as they are on the call to the customer. This reduces the cost to serve with shorter calls, and improves customer feedback.

This is just one example of how natural language processing can be used to improve your business and save you money.

What capabilities should your NLU technology have?

Multiple language support

Depending on your business, you may need to process data in a number of languages. Having support for many languages other than English will help you be more effective at meeting customer expectations.

In-depth analysis

A sophisticated NLU solution should be able to rely on a comprehensive bank of data and analysis to help it recognise entities and the relationships between them. It should be able  to understand complex sentiment and pull out emotion, effort, intent, motive, intensity, and more easily, and make inferences and suggestions as a result. It should also have training and continuous learning capabilities built in.

Rapid interpretation and response

Being able to rapidly process unstructured data gives you the ability to respond in an agile, customer-first way. Make sure your NLU solution is able to parse, process and develop insights at scale and at speed.

Ease of usability and integration options

Your NLU solution should be simple to use for all your staff no matter their technological ability, and should be able to integrate with other software you might be using for project management and execution.

Automated actions

Ideally, your NLU solution should be able to create a highly developed interdependent network of data and responses, allowing insights to automatically trigger actions.

Use NLU now with Qualtrics

Qualtrics’ Natural Language Understanding engine, powered by Clarabridge, uses rules based and machine learning techniques to extract, tag and score concepts that are relevant to customer experience analysis such as emotion, effort, intent, profanity and more.

Users can customise many of these elements to more precisely reflect their business, use case(s) and industry. When combined with the original text and associated source and customer metadata, analysts and front line teams can easily understand what customers mean and take insight-led action.

Our solution can:

• Automatically find topics and sentiment in human language text, bringing key drivers of customer and employee experiences to the fore in seconds with iQ™
• Detect emotion, intent, and effort with 150+ industry-specific NLU models that find people’s underlying needs automatically
• Augment customer and employee experience in real-time with automated workflows and triggered actions managed through xFlow
• Enrich your customer and employee Experience ID profiles with notes on emotion, effort, and intensity