Natural language understanding: the secret to content analysis

Computer scientists have been trying to get computers to understand natural, human language since the 1960s. As wave after wave of brilliant innovation synergized with increases in computing power, the idea of Natural Language Processing (NLP) went from the realm of science fiction to being reality, albeit one that needed significant compute resources to produce limited results.

Then, in a rather quick leap, NLP, and Natural Language Understanding (NLU), a subset of NLP, have become so commonplace, we scarcely even notice it anymore. Not convinced? Ask Siri to explain it to you.

The work is not finished, however. Advances continue to come on the market, pushing computers’ language comprehension abilities ever further. New use cases are also emerging. They demonstrate the value of NLU in novel contexts like cognitive search and automated reasoning.

What is natural language understanding?

Natural Language Understanding should be a fairly easy concept to grasp. A computer equipped with NLU capabilities can understand natural language, such as the text of a written document or a spoken sentence. That’s why the technological capability is sometimes referred to as natural language interpretation.

NLU is part of NLP, which deals with the overall process of getting machines and humans to interact using human-like language. NLP contrasts to the standard mode of human-to-machine interaction, wherein the human’s input is translated into a machine language the computer can understand. NLU is the listening side of the equation.

With NLU, the computer comprehends natively human communications. NLU works by combining software logic, linguistics, ML, and AI to parse the unstructured data that comprises natural language.

Where things get a little tricky is in understanding what NLU is capable of—and what it isn’t, yet. For example, it’s one thing to get a computer to understand a sentence like, “I would like to order a bus ticket.” It’s quite another for a computer to accurately comprehend a comment like, “Every man at the party spoke to one woman.” The latter sentence can have two quite different meanings. Did each man at the party speak to a separate woman, or did all the men speak to a single woman? The sentence could have either meaning. Context might help, like if the next sentence read, “There were five men, and five women at the party, each deep in private conversations.”

The predominant modes of NLU today are relatively simple, more “I drove the car” than “I can’t admit I resent my sister-in-law for being a drama queen.” Examples include various Robotic Process Automation (RPA) workloads like ticket and email routing, along with Machine Translation (MT) services like Google Translate.

Some NLU solutions are capable of “automated reasoning” and question answering, a la Siri. The thing to keep in mind is that even the “simple” NLU workloads are quite sophisticated. They represent decades of incremental improvement and the application of a great deal of Artificial Intelligence (AI) and Machine Learning (ML).

How does NLU work

How does NLU work? Although there is a wide range of approaches, most NLU systems employ a standard set of components that work in concert.

The most basic element of the system is a lexicon, which defines the language. A lexicon is more than a dictionary. It puts words and phrases into a structure that NLU can use.

Then, working with a grammar rules component and a semantic guide to comprehension, the system can start to break down the language it is presented and transform it into something meaningful for the user and itself. Collectively, these components are called a “semantic parser.”

The semantic parser performs text categorization. In the sentence, “I would like to order a bus ticket,” the semantic parser is able to identify that “I” refers to the system user, “order” is an action the system user wants to take and “bus ticket” is a product or service to be delivered to the user. In effect, the system is using the basic English subject (noun), verb and object sentence structure and converting it into an expression the computer can use—in this case, to sell a bus ticket.

As one might imagine, getting this to work, even in a simple use case, represents a massive amount of work. Lexicons and accompanying grammar rules engines can take years to develop.