Guide to Natural Language Processing (NLP)

So how does NLP begin to tackle this challenge? Let’s start with a definition and some history first.

What is Natural Language Processing?

Simply speaking, natural language processing refers to any human-to-machine interaction where the computer is able to understand or generate human-like language. This is in contrast to what computers were previously limited to, which was some form of machine language.

Simple as the end result may appear, the actual process of getting a computer to perform NLP represents an extremely complex synergy of different scientific and technical disciplines.

NLP brings together computer science, linguistics, machine learning and artificial intelligence (AI). Depending on how the NLP is designed, the process may also involve statistics and data analysis. These elements, working in concert, make NLP capable of “hearing” or reading natural human speech and accurately parsing the words so the computer can take the action expected by the human user.

Brief History

NLP’s roots are often traced back to the Georgetown experiment in 1954, which translated several Russian sentences into English. This was done using manually coded language rules and dictionary lookups. On the heels of this project, researchers initially thought they would have machine translation “solved” in a matter of years, but soon found it to be a far more difficult task because of the complexities and nuances of language.

Throughout the sixties and seventies, research continued into how to represent linguistic meaning computationally. Notable efforts included Joseph Weizenbaum’s ELIZA program, which simulated a conversation between patient and therapist, and MIT’s SHRDLU, which enabled a user to move blocks around a world using natural language. Here’s an example of a SHRDLU conversation:

Things quickly picked up when statistical models and machine learning algorithms started replacing the need for hand-written, hard-coded rules and grammar. As computers became more powerful and unstructured data became ubiquitous, deep learning and recurrent neural networks further advanced the field, paving the way for virtual assistants like Siri and Alexa. But as anyone with experience using Alexa today can attest, there is still work to be done (Alexa: “I’m sorry, I didn’t understand that.”).

NLP methods and approach

So how do we make computers understand us? The short answer is that it’s complicated–far more complex than this guide will dive into. That said, some basic steps have to happen to translate the spoken word into something machines can understand and respond to. Let’s take a look at what those are.

How does NLP Work?

There is no single way that NLP software functions. However, in general, almost all NLP tools have capabilities that enable them to distinguish syntactic and semantic rules and recognize many different words. This is not an easy thing to do.

Consider the following written enterprise search query: An employee wants to know if the company has declared December 26 a holiday. She types, “Do we get Boxing Day off?” That’s a question that most human beings can easily understand, even if they have to figure out that December 26 is sometimes called “Boxing Day.”

An NLP program will have to deconstruct this question on syntactic and semantic levels to enable the enterprise search engine to return a result. First, it has to recognize that “Do we” is a question reflecting “we,” meaning company employees. Then, it has to parse the words “get” and “off” as referring to getting time off from work versus, say, getting off a plane. It needs to understand what Boxing Day is… to the point where the search engine can respond to the actual query, “Is December 26 a holiday?”

Performing these tasks is an enormous challenge. Human language is highly complex, with English being arguably one of the most difficult.

NLP transforms data into something that a computer can interpret by starting with what is known as “data pre-processing.” In this stage, the NLP tool analyzes syntax and semantics to understand the grammatical structure of the text. It identifies how the words relate to one another in the specific context.

Data pre-processing may utilize tokenization, which breaks piece of text down into semantic units for analysis. The process then tags different parts of speech, e.g., “we” is a noun, “do” is a verb, etc. It could then perform techniques called “stemming” and “lemmatization,” which reduce words to their root forms. The NLP tool might also filter out words like “a” and “the” that doesn’t convey any unique information.

The data pre-processing step generates a clean dataset for precise linguistic analysis. The NLP tool has an algorithm that then interprets the dataset. The algorithm can take one of several forms. The NLP tool uses grammatical rules created by expert linguists with a rule-based approach.

Alternatively, and this is increasingly common, NLP uses machine learning algorithms. These large language models are based on statistical methods that “train” the NLP to understand human language better. Furthermore, the NLP tool might take advantage of deep learning, sometimes called deep structured learning, based on artificial neural networks.

NLP Techniques

The techniques involved in NLP include both syntax analysis and semantic analysis.

Syntax is defined as “the arrangement of words and phrases to create well-formed sentences in a language.” Essentially, syntax is about how sentence structure creates grammatical sense. In terms of NLP, syntactical analysis uses machine learning algorithms to apply grammatical rules to a group of words to derive meaning. It often includes the following processes:

Semantics is defined as the “meaning of a word, phrase, sentence, or text.” This is the most challenging task for NLP and is still being developed. Again, we’ll revisit the “Do we get Boxing Day off?” example. Semantics is the art of understanding that this question is about time off from work for a holiday. This is easy for a human but still difficult for a computer to understand the colloquialisms and shorthand manner of speaking that make up this sentence.

Semantic analysis often involves:

Different semantic classification models can also be used, depending on the end goal:

By topic: sorting text into predefined categories, like “billing questions”, “account information,” etc.

By sentiment: understanding the emotion of content – positive, negative, neutral – to identify how happy or upset a customer is or how people feel about your brand.

By intent: tagging content based on level of interest. Is someone just browsing or ready to buy?

NLP and Machine Learning

As we mentioned earlier in this guide, the NLP field took off when machine learning was added. Machine learning accelerates and automates the text analysis and application of grammar rules. Machine learning algorithms study millions of texts to “learn” about human language, including syntax, semantics, and sentiment.

The more data the algorithm takes in, the more it learns and the better it becomes at analyzing and understanding content. It can apply what it has learned to future NLP analyses without being told what to do or how to do it.

There are two types of machine learning: supervised and unsupervised. With supervised machine learning, the algorithm is “trained” on a set of texts that have been marked up for what it should look for (i.e., parts-of-speech, named entities, relationships, etc.). Then, it’s given text without any instructions to analyze. Common supervised NLP machine learning algorithms include:

Unsupervised machine learning is when you train an algorithm with text that hasn’t been marked up. It uses frameworks like Latent Semantic Indexing (LSI) or Matrix Factorization to guide the learning.

NLP Use Cases

The potential for NLP is ever-expanding, especially as we become more enmeshed with the technology around us. As computers and machines expand their roles in our lives, our need to communicate with them grows. Many are surprised to discover just how many of our everyday interactions are already made possible by NLP. Let’s look at some examples.

NLP applications

NLP applications put NLP to work in specific use cases, such as intelligent search. The technology has many uses, especially in the business world, where people need help from computers in dealing with large volumes of unstructured text data.

For example, a company might benefit from understanding its customers’ opinions of the brand. However, whatever insights regarding the brand are hidden with millions of social media messages. No human being is going to read them all. However, an NLP tool tuned for “sentiment analysis” could get the job done.

Other notable NLP applications include: