Natural Language Processing

From New Media Business Blog

Jump to: navigation, search

Natural Language Processing (NLP) is the ability of a computer program to understand human language as it is spoken or written. It is a very active area of linguistic-computer science research and there is no one universal definition that everyone agrees upon. However, there are many characteristics in NLP which each definition supports.[1]


Natural Language Processing

Natural Language Processing (NLP) is the “theoretically motivated range of computational techniques which analyzes text for the purpose of achieving human-like language processing for a range of tasks or applications.”[2]

To fully understand this definition, it can be further developed:

  • The “range of computational techniques” implies that multiple techniques can be chosen to process and analyze natural language
  • ‘Natural language’ refers to any human language, oral or written; therefore, it is not a computer language (i.e. C, Java) and not numbers or arithmetic
  • ‘Human-like language processing’ reveals that NLP is considered a discipline within Artificial Intelligence (AI), as it attempts to process, understand, and generate language like a human
  • ‘For a range of tasks or applications’ points out that NLP is used to accomplish tasks which stem from understanding human language such as in Virtual Assistants, Search Engines, Chatbots, and Machine Translation [3]

An effective NLP system is able to accurately read or listen to words, comprehend its meaning, determine the appropriate action, and respond back in a language the user will understand. Embedded in NLP are three key components:

  • Language Processing (LP)
    Driven by computational statistics, NLP starts by converting text into data and attempts to learn patterns. It seeks to mimic how humans listen to words and sentences. In other words, LP kickstarts how people and machines can talk to each other naturally. Note that in LP, the machine does not attempt to understand the text. [4]
  • Language Understanding (LU)
    Language Understanding is where the machine comprehends the text and extracts the unstructured content. Unstructured content is comprised of both poorly-defined and flexible rules such as concepts, sentiment, and relations. Imagine how an acronym can have various meanings: OG can mean “old guy” or “original gangster.” Individuals interpret keywords differently based on the concept, sentiment, and context in which this word was used. While humans are able to handle mispronunciations at times, and understand slang, machines are less adept at handling unpredictable inputs. Computers need to convert unstructured content into a well defined form in order to understand and use the data. [5]
    • Speech Understanding
      As a subset of LU, speech understanding concentrates on the ‘sounds’ of language. When an individual talks, it picks up sounds and transcribes them into recognizable words. Once in this form, the same levels of written text language processing are utilized.
  • Language Generation (LG)
    In the final step of NLP, the machine produces a natural language answer based on the data input processed and understood in the previous steps. Gartner’s recent Hype Cycle for BI and Analytics sums up the difference between NLG and NLP well: “whereas NLP is focused on deriving analytic insights from textual data, NLG is used to synthesize textual content by combining analytic output with contextualized narratives.”

To sum up, LP is the reader or listener, LU is the brain, and LG is the writer or speaker. [6]


The goal of NLP, as previously mentioned is “to accomplish human-like language processing.” In other words, NLP wants to achieve Natural Language Understanding (NLU) . However, true NLU is yet to be accomplished. A fully functioning NLU system would be capable of completing the following four steps:

  1. Process the text correctly (spoken or written)
  2. Analyze and translate the text (understanding the unstructured content)
  3. Draw inferences from the text
  4. Communicate analytic insights from the text

Human language is complicated. A machine must understand not only the words, but also the concepts and feelings behind the text.

Recall NLP is the ability of a computer program to understand human language as it is spoken or written. Alan Turing, a notable NLP computer scientist, once said that he was “building a brain.” He proposed a test to determine if a machine can be considered intelligent if a machine “can deceive a human into believing that it is human.” As stated previously, LU is the brain. Up to this point in mankind, creating a machine with the cognition of the human brain has not been done - this is what NLP hopes to achieve. [7]


Human Crafted Rules-Based Sets

Rules-based sets take the text as input and produce a tagged output. In this stage of NLP, a human expert needs to apply their own knowledge to manually construct rules. These rules will then instruct a machine about which words and phrases to look for in text and specific responses to answers or queries.

A common reference to this rule is The Georgetown Experiment , which was the first NLP application. It was governed by rules-based sets, allowing it to automatically translate over sixty Russian sentences into English.

“The spirit is willing, but the flesh is weak” was translated into Russian: дух бодр, плоть же немощна and then back to English, the result was: "The vodka is good, but the meat is rotten.” Regardless of the inaccurate results, the authors claimed that within three or five years, machine translation would be a solved problem. Well publicized by journalists and perceived as a success, the experiment encouraged governments to invest in computational linguistics - allowing advancements in NLP to continue on. [8][9]

Machine Learning

Machine learning is powered by an algorithm that learns to automatically identify useful rules and is the advancement of hand-written rules. Machine learning is used to reproduce known patterns and knowledge, and then automatically apply those results to decision-making and actions. Early machine learning algorithms included decision trees which produced if-then rules.

However, machine learning uses a general learning algorithm to automatically learn such rules through analyzing the plethora of real world examples. Automatic learning procedures utilize statistical inferences to produce models that can understand unfamiliar input. Therefore, the accuracy of these systems are improved by supplying more input data or source to it. [10] [11]

Current Approach


NLP researchers are now developing next generation NLP which can understand the variability and ambiguity behind all languages through cognitive computing and deep learning.

Cognitive computing cognitive computing, utilizes deep learning in conjunction with cognitive science knowledge in order to build NLP systems that simulate human thought processes. Deep learning is a family of algorithms which implement deep networks with unsupervised learning. Similar to how humans learn through experience, cognitive computing helps computers to learn through experience in order to speak, read, and write. As a result, it requires massive amounts of labelled data so that it can make connections between topics. In addition, for the algorithms to start learning, it requires training from human knowledge experts in order to correctly identify relevant correlations.

An example of cognitive computing is IBM Watson. Watson can take in large databases and create relationships within the data, understanding some sentiments behind the text. Afterwards, it further extends its knowledge to make inferences on unknown data.

Current Applications

Search Engines

Who was the U.S. President when the Angels won the World Series? is a complex query that contain sub queries.

Search engines or web-based search engines are software systems that search for information on the World Wide Web. Previously, keyword-based search was used to extract keywords from search questions or queries in order to match content on the World Wide Web.[1] If a user wants to search for information, the question needs to be short and compact to reduce unwanted results. For instance, instead of searching "How tall is Steph Curry?", the user must search "Steph Curry height".

As opposed to keyword-based search, natural language search incorporates a human language query which resembles speaking to a human.[2] The search engine utilizes the mechanic of semantic search where the search engine will take in context and intent of the person in its search parameter.

The same question can be put into an NLP search engine without alteration. This is particularly effective when dealing with longer, more complex queries that reference itself. For example, the query, "Who was the U.S. President when the Angels won the World Series?", is easily answered now, as shown by the photo on the right. However, in the past, the same question had to be broken down into two separate searches:

  1. "When did the Angels win the World Series?"
  2. "U.S President (Year)".

In the first search, we would find out the year that the Angels had won. We would then use that result for the second search. This would lead to a more complex search process if the Angels had won multiple World Series.

Real Time Translation

Speech Translation

Speech translation is a “process by which conversational phrases are instantly translated and spoken aloud in a second language”.

Speech translation involves three software systems:

  • Automatic speech recognition (ASR)
  • Machine translation (MT)
  • Voice synthesis (TTS)

The ASR software processes the input of the speaker and translates the input into text. The text is then translated into a second language through MT. This typically involves phrase-based translation or sentence-based translation, where an entire phrase or sentence is translated at once rather than word-for-word. MT also takes context into consideration before generating a translation. Finally, the translated text is spoken aloud using TTS to best emulate a human speaker.[3]

Each of these technologies requires an extensive amount of data in order to find the best matched data for the intended context to generate an accurate translation.


Existing real-time translation technologies are not perfect. They can only be considered convenient tools for tourists travelling to a foreign country or for students in an academic learning environment.[4] Some real-time translation devices are highly sophisticated because their use is limited to relevant phrases or terms in a particular subject or field (i.e. travelling, medical, business, etc).[5] However, humans talk about a broad range of topics and subjects. The distinct words a speaker uses and the manner in which they are conveyed all depend on the individual. Therefore, real-time translation technologies still require much progress before they are ready for daily use.


Microsoft Translator Speech API is a cloud-based machine translation service that uses REST technology. It can be used to build applications, tools, or any solution requiring multi-language speech translation. Some Microsoft products include Live feature (preview); Presentation Translator add-in, Skype Translator; Microsoft Translator (iOS and Android). Mircrosoft’s technology is already used in telecommunications to translate real-time phone calls, language servicing to subtitle videos and to support deaf communities through sign language applications.[6]

Skype Translator

The Skype Live Translation feature allows for voice and video calls to be translated in eight languages. Instant messaging text can be translated in 50 languages.

Presentation Translator

Presentation Translator is a Microsoft Garage project that displays subtitles directly on a PowerPoint presentation as the presenter speaks. From a selection of 10 languages, speech can be converted into subtitles offered in over 60 languages. Moreover, up to 100 audience members can follow along with the presentation using a QR or five letter code. Students, colleagues and other listeners can see the presentation in their own language on their respective devices (i.e. mobile, tablet or laptop). This helps audience members who have hearing difficulties or struggle with presenter’s language to understand and participate in the presentation. Moreover, the customized speech recognition of Presentation Translator gradually learns the technical terms, jargon and proper nouns relevant to your presentation with use, allowing for more accurate translations.[1]


Google Neural Machine Translation
Machine translation has evolved from word-based machine translation to phrase-based and even sentence-based machine translation.[2] For example, in the past Google :Translate would have translated the idiom “I am feeling blue” word for word to “I am feeling blue”, the colour. However, Google Translate now uses neural machine translation and translates sentence by sentence rather than word by word. :This results in faster and higher quality translations with far greater accuracy.[3] Google will now translate “I am feeling blue” to “I am feeling sad” in another language.

Google Translate
Google Translate
Google Translate is an application that utilizes the cloud to translate written and/or spoken content. Translations are displayed and can be spoken aloud. Transcripts of the translations can also be saved. Google Translate translates 103 languages online and 59 languages offline. In Google Translate's conversation mode, two-way instant speech translation is available in 32 languages.

Google Pixel Buds

Google Pixel Buds are “a pair of wireless earbuds that allow you to listen to media, answer calls, talk to your Assistant, and translate languages”. [1]These earbuds connect only to Google's latest smartphones and Google Assistant. Speech around the user is captured, then the speech is translated using Google Translate and the translation will be heard directly through the earbuds. Pixel buds translate 40 languages in real time.


Ili is a real-time translation gadget designed for travelling by a Japanese company. Ili’s internal all-in-one processor performs speech to text conversion while simultaneously capturing audio data. As ili is equipped with an in-device database, no Wi-Fi is required to perform translations. This device currently translates between four languages: English, Spanish, Japanese and Mandarin. Interestingly, ili is ideally used for one-way communication. By eliminating the need to hand off and teach others how to use the device, the communication process becomes simpler. Moreover, this ensures that the other party keeps their reply brief as ili is not meant to translate specialized language (i.e. business and medical), proper nouns and slang. [1]

Travis Translator
The Travis Translator

Travis is a portable instant translation device that supports over 80 languages. Travis functions offline without Wi-Fi if needed for up to 20 languages. Travis uses artificial intelligence to learn the language the more the user speaks.Travis is equipped with a patent pending voice DSP, using dual noise cancelling microphones and upgrades speakers for a clear audio experience. This device is also compatible with both wired and wireless earphones.

Pilot translator

Pilot is a translating earpiece produced by Waverly Labs in 2016. Pilot and the Pilot app uses speech recognition, machine translation and speech synthesis to translate English, Spanish, French, Italian and Portuguese. Although Pilot currently translates a limited number of languages, Waverly Labs has promised to deliver more languages in the near future. Pilot’s dual noise-cancelling microphones allow Pilot to filter out surrounding background noise and focus on translating the dialogue between Pilot users. If multiple users are present, they can join a group conversation through the Pilot app while talking in their respective languages. The smartphone running the Pilot app can also be used as a speaker to allow everyone to hear the conversation.[2]

Mymanu CLIK
Mymanu CLIK

CLIK are wireless real-time translation earbuds that translates up to 37 languages using an embedded translation system. Users simply need to share their unique passcode on the CLIK app with their conversation partner to communicate locally or over long distances. One-on-one, group, conference and even air translation using cloud-based translation to communicate to others is possible. Moreover, a full transcript of the conversation will be recorded. CLIK is equipped with touch sensor technology and APTX audio codec to provide optimized touch control and a seamless audio experience.[3]

WT2 Real-time Earphone Translator
WT2 Real-time Earphone Translator

WT2 is a real-time, wearable translator made possible through “wireless communication, intelligent algorithms, voice translation and many other technologies”. WT2 is an earphone that translates over five languages through an app. It is perfect for one-on-one conversations and can last up to 6 hours. WT2 has three modes: auto mode, manual mode, and ask mode. These modes account for the conditions of the surrounding environment to optimize the translation experience. Auto mode is suitable for quiet environment. In contrast, manual mode is suitable for noisy environments. Finally, ask mode is design for users who need to make quick inquiries. [4]


Number of Languages
According to the Ethnologue,[5] there are over 7,000 languages being spoken around the world. This number continues to increase as more languages spoken by smaller communities are discovered. The spread of languages across the globe is uneven and a third them are at risk of disappearing. Although technology is rapidly evolving, it will at least take another decade to refine the translation of the main spoken languages. Therefore, it is highly likely that many of the remaining 7,000 will not be translated or will disappear before they are inputted in the translation systems.

Many existing real-time translation technologies, especially software applications require Wi-Fi to operate. However, Wi-Fi connection may not be stable or even exist in less developed countries and rural areas. Moreover, even if Wi-Fi connection is available, it might be a costly add-on. Translation devices such as Travis and ili have taken this issue into consideration and integrated internal translation systems to their devices. As such, devices and applications offering an offline mode are available, but the number of languages translated and the accuracy of the translation is still lacking compared to the online alternative. Since companies are already trying to resolve the internet connectivity issue, it is believed that Wi-Fi will only be considered a short-term obstacle for real-time translation technologies.

Systematic Limitations

  • Speech Recognition:
    Background noise, individual speaking styles such as accents and the speed at which the speaker talks s greatly interfere with speech recognition systems of real-time translation technologies.[6]
  • Speech Synthesis and Prosody:
    Although machine spoken translations greatly resemble natural human vocals, the prosody in speech synthesis still needs improvement. Prosody is the alteration of the speed, pitch and volume of speech to convey an extra, and perhaps critical, channel of meaning. For example, when an individual says “I really want to eat cake.”, that individual will stress the word “really”, to convey how much they desire cake. However, speech synthesis systems lack the contextual understanding to deliberately stress certain words in a sentence, even if they were capable of the act. Therefore, machines may be able to imitate a human’s voice, but not their manner of speech.[7]
  • Pragmatic Understanding:
    Machine translation has seen much progress over the past few decades. Phrase-based and sentence- based translation utilizing neural networks is commonplace. However, even Google’s Neural Machine Translation still struggles with long and complex sentences, and interrelate sentences. This is partially due to a lack of pragmatic understanding. Machine translation systems do not possess the shared knowledge and understanding that groups of individuals use to process the human language they hear. For instance, when asked “Who plays Thor in ‘Thor’?”, an individual will try to recall which actor played the character Thor in the movie Thor. Yet, Siri once responded “I don’t see any movies matching ‘Thor’ playing in Thor, IA, US, today.” Siri was looking for movie, Thor, in the city, Thor. As Thor could be referring to a city, movie, character or even an individual’s name pragmatic understanding is necessary to follow along in a conversation.[8]
Thor the movie, character or location?

As such, real-time translation technologies need to learn more about real-world knowledge through artificial intelligence to acquire common sense and reach an adequate level of pragmatic understanding.[9] Otherwise, machines will continue to fail to understand the context, the underlying meaning, of human dialogue.

Virtual Assistants

The 4 most known Virtual Assistants (left to right): Alexa, Cortana, Siri, Google Now

Having a distinguished voice-controlled virtual assistant (VA) is required to compete in the smartphone industry. Large corporations, such as Apple, Amazon, Google, and Microsoft are developing VA applications in both their smartphones and in-home devices. While VAs are smart, useful, and convenient, their underlying technology raises concerns about privacy and security. It is noteworthy that many VAs are backed up by large corporations, allowing them to utilize the plethora of resources.

In addition to taking voice commands, VAs continually improve through machine learning and artificial intelligence. They are used to make everyday tasks simpler. Users can verbally request VAs to make calls, text, set reminders, alarms, or timers and much more. Many VAs can also link and modify apps in order to listen to music, control household appliances, check weather, etc. Although VAs can accomplish simple tasks, most VAs depend on your Wi-Fi or data connection in order to process certain demands.

Skills Echo users have used at least once


Virtual Assistant release timeline

As a result of their first-mover advantage, Amazon Echo (also known as Alexa) took over and now dominates the home VA market - selling 6 million Echos in 2016. A reason why the Echo sold well was because Amazon marketed it as a speaker with VA functions, rather than a VA station. Although Echo has a smartphone app, it does not carry the same hype as the well-known VAs pre-installed on Apple’s iPhone (Siri) or Samsung (Bixby). The app allows users to change Echo’s wake word from “Alexa” to any name or phrase that they like. Users can also modify their device’s name and location or increase Echo’s skills. [10]

Amazon intended to increase consumer purchases through Echo. Now, Amazon is following through with this plan by paring up Alexa with Amazon Prime to enable voice purchases. After entering the grocery deliver market through its acquisition of Whole Foods, ordering groceries from the kitchen and having it delivered to your home through is a reality. [11]

However, Echo is a home-based device and has fewer features compared to its competitors.[12] Alexa must catch up to its competitors through in-app suggestions; however, the likelihood of users downloading the Alexa app without the device itself is rather unlikely. Amazon lacks a smartphone platform that its competitors such as Google and Apple have. Google Home, which offers a seamless integration between a smartphone and home-based device, may overturn Echo’s position as market leader in the home VA market.


As part of Apple’s iOS operating system, Siri first arrived as an app in early 2010. In Apple’s October 2011 iOS 5 launch, Siri was relaunched in all iPhones and iPod Touch devices. Siri had its start as a spin-off from a project, being developed at the SRI International Artificial Intelligence Center called the CALO project (Cognitive Assistant that Learns and Organizes). At the time, Apple integrated Siri into the iOS system and used Siri as a competitive advantage to encourage consumers to switch to or upgrade their Apple products. [13]

Siri has a variety of basic voice commands and engages with iOS integrated apps, like Apple Music. For an Apple user, Siri works universally on all Apple devices, unlike its competitors. Siri is continuously learning to help answer everyday questions in the most natural of ways. Questions such as, “What is the meaning of life?” or, “Why am I here?” are answered with humour.

Some significant implications of Siri is its inability to communicate with apps such as Instagram and Facebook. Giving the command, “Post my latest photo on Instagram”, is not possible where most competitor VAs are able to do this. Siri cannot use answers from previous questions and relate them to following queries, therefore it cannot hold a continuous conversation. Each question must have structure for Siri to understand, falling behind Google on this aspect. In 2016, Sean O’Kane stated, “Siri’s big upgrades won’t matter if it can’t understand users … it’s still not very good at voice recognition, and when it gets it right, the results are often clunky. And these problems look even worse when you consider that Apple now has full-fledged competitors in this space”.[14] In addition, Apple is falling behind in the home base sector of VAs with its HomePod being pushed back until 2018. [15] Although Siri kickstarted the VA phenomenon, their competitors are overtaking their position as market leader as they have more advanced features based on consumer demand.


Although slightly behind in launching compared to its competitors, Google Assistant sure came out with a bang. Google Assistant was unveiled in May 2016 as part of the Google Home speakers and a new messaging app, Allo, to be able to hold two-way conversations. Google Assistant improved off of the Google Now, a one-way VA which launched in July 2012. Google Assistant started off only on Google Pixel smartphones, but in early 2017, it was made available on English-speaking Android phones running Android Marshmallow or Nougat.[16] Google also plans to roll out future versions of Google Assistant that will available on Android tablet, Android Wear 2.0, Android TV, Google Pixel Buds, and even Android Auto.[17] In a bid to be more competitive, Google is also coming out with Google Assistant for iOS systems as a separate app.

It is noteworthy that Google Assistant is powered by Google, the world’s largest search engine. As a result, it has powerful data-grabbing potential when it comes to simple commands such as requesting news briefings. It works with some third-party apps, like Spotify and WhatsApp, to perform a variety of commands. Actions on Google, which Google launched in December 2016,[18] allows software developers to advance their Google Assistant the way they want.

The Google Home must be activated each time you have a question with the wake word, so seamless conversation is difficult. This is different from the Google Assistant smartphone app that does allow for a two-way conversation. It is also behind Amazon in the amount of third-party partnerships it has.


Samsung’s Bixby is new to the VA market, launching with the Samsung Galaxy S8 and S8 Plus in April 2017. Samsung is trying to join the game of VAs in response to Google’s Assistant and Apple’s Siri. Bixby is currently only available in English-speaking countries.[19] Bixby is relatively new and is still in the process of constant development. Its processing speed is almost slower than manual text input. [20] However, it has distinguishing features such as enabling more in-app hands-free options. For example, when doodling with a stylus, a voice command can change the colour or size. [21]

Samsung Bixby Voice first look


Cortana recognizing Clippy as her mentor

Although Microsoft’s Cortana is a well-known VA in the market, it is not widely used. However, it has similar functions to its competitors and has potential to improve as a Microsoft product. Launched in 2013, Cortana operates on Microsoft smartphones and products such as Invoke, Windows 10 computers, Windows Mixed Reality, and soon Amazon Echo.[1] Cortana is the evolution and development of Clippy, the useful paperclip tool that assisted Microsoft products users. Cortana works well with basic tasks and notably uses Bing to search the web. It is very much integrated into Microsoft Edge [2] and can easily search the Internet for anything; however, it only allows Cortana to use Edge and Bing as browsers. It stores all personal information into an editable Notebook that users can manipulate and use to safeguard their privacy. [3] Microsoft’s Cortana-based Invoke, was launched in October 2017 into an already Amazon-dominated market: “Microsoft is banking that integration with its Office 365 applications will make Invoke attractive for many people who use Office at work.” [4] Cortana is only compatible with Windows computers, which is a huge downside, since most app developers and VA users are on smartphones. Its capabilities are very limited and it is hard to access on iPhone and Android, which both already have their respective VAs.


Some of the limitations for all VAs include:[5]

  • Cost vs. savings: As a developing technology, consumers have a lack of knowledge on the benefits of VA, therefore, hesitant to purchase a device that costs $80-$200.
  • Security: Corporations reassure consumers with voice recording encryption, but as this information is transmitted over the Internet, hackers can easily intercept private data.
  • Imperfect voice recognition: An example of voice recognition struggling to interpret speech, where accents interact with various VAs:
  • Lack of privacy: As VAs hear and record every conversation you have. It is then sent to a database which stores all of your personal information.

Virtual assistants recognizing accents

Female Dominant Voices in VA

Available voice gender for 4 popular Virtual Assistants

Interestingly, the main VAs, Siri, Alexa, Cortana, Bixby, and Google Assistant, all have female voices by default and many of them have female-sounding names. In 2017, the results from two studies conducted by The Wall Street Journal concluded that a female voice for VAs are preferred over male voices as it is believed to be more welcoming and understanding.[1] Another reason for why VA voices are predominantly female is due to the programmers who made them. In addition, VAs are marketed as a “helper.” They are used to help with activities such as cooking, shopping and other tasks associated with females. Today, programmers are predominantly Caucasian males, only 9 per cent of the engineering workforce[2] is comprised of women; thus, creating systemic bias. Unlike the others, Google programmed and named their assistant, “OK Google” to be more gender neutral. It can be noted that currently the only known VA male voice is IBM’s Watson.

The lack of diversity in the technology industry


Sophia, a complex chatbot

Chatbots are computer programs that can conduct conversations through auditory or textual methods.

First-Generation Chatbots

Older chatbots used a set of rules in order to generate responses. These bots were less likely to make grammatical errors as their sentences were hard-set handwritten to respond to a question or request. These answers were stored in a database for responses to a question or request. However if the query or input is out of the ruleset, they are unable to generate a response. [3].

Current Chatbots

Newer chatbots use deep learning to analyze and produce responses, making them more adaptive to human input and context. However, the drawback is that they require much more training time and material before they can accurately respond. [4]

NLP chatbots go through 3 steps in order to conduct a conversation with a human:[5]

  1. Converting human language into input for the chatbot. This process extracts the text or voice for the bot to use
  2. Applying NLP to return semantic and context
  3. Return the correct response and/or action through AI

Business Context

Millennials are open to making purchases from chatbots

With the integration of natural language recognition, chatbots can interpret textual inputs and understand inquiries more effectively. As a result, this simulates conversations and provides intelligent human-like responses which are applicable in the business world. [6]

The chatbot is always ready to answer your questions, they will not make you wait. Chatbots can be used as a customer service representative in place of a human operator to decrease the waiting time for customers. In addition to saving time, it saves the company cost and resources as labour can be used for higher level processes that require greater skills. Such cost savings can increase margins.

By directly conversing with the customer, chatbots can monitor and store customer behaviour such as buying preferences, and items purchased, for further analysis. With this data, it can assist the marketing team in personalizing offers and designing content that is appealing to the target audience. As a result, chatbots can personalize the customer experience by building a customer profile and maintain a strong relationship. [7]

The North Face Expert Shopper XPS
2017 XPS

The North Face Expert Shopper XPS is a prime example of a chatbot acting as a customer service agent where XPS will converse with a customer as a sales representative. XPS will suggest different jackets depending on specific characteristics chosen by the customer, such as the season, purpose, etc., instead of having customers manually filter through items. [1]. This chatbot applies IBM Watson’s NLP capabilities into its algorithm for text and semantic recognition.[2]


Although NLP technologies can be very helpful, a concerning issue is that it collects and stores the user’s personal data. NLP technologies require audio or text input to operate. As such, NLP users are always at risk of having the details of their conversation recorded and sent to tech companies. However, NLP technologies such as VAs intrude on privacy at a whole new level. For example, VAs analyze its user’s voice, speech patterns, the type of documents that are read often and other needs that make up its user’s habits.[3] VAs gather personal search and email history in order to provide information customized towards your needs and personality. VAs attempts to familiarize itself with its users, like with a friend. It is found that for both VA users and chatbot users do less pre-purchase research and decrease in-store purchases as consumers as they are reliant on these technologies.[4] VAs and chatbots play a significant role in selecting suitable consumer products and showing relevant product information findings to its users. This could greatly influence the consumer retail landscape. However, privacy concerns remain an issue for consumers.

Consumers must understand they will sacrifice their privacy for the convenience of using NLP related technologies. The Internet and many other emerging technologies have greatly desensitized people from this loss of privacy. As globalization continues, it is believed that technological trends will further erode privacy to provide increased convenience and customization. For instance, even VAs for dogs are available on the market.[5] NLP technologies are moving beyond getting to know humans, they are even trying to learn about humans’ closest companions, pets.

The Future of NLP

Adding Structure to Unstructured Data

The evolution of New Media technologies allows more user-generated content to be created, increasing the amount of unstructured data. This consists of text and multimedia content in the form of e-mails, videos, images, and many more. In fact, 80% of data today is unstructured, and failure to manage this data could result in a greater opportunity cost for businesses. [6]

The future of NLP is shifting towards NLU, a subset of NLP that focuses on a machine’s ability to understand what we say and make sense of data in a meaningful way. This includes learning from experience in regards to what we did not say and understanding what our actual intent was in that specific context. By analyzing language patterns to understand text, it performs sentiment analysis to determine the tone of the text and its significance.

With the advancement in cognitive technologies such as deep learning and cognitive computing, NLP will bridge the gap between human communication and digital data. Historically, we have learned about the language of machines, but now, they are learning human language - our roles have reversed. NLP can disambiguate human language and it will streamline communication between humans and machines.

Integration of NLP with Big Data

Supporting Invisible UI

User interface is the means by which the user and computer system interact. NLP allows for a seamless and direct interaction, where the user can speak simply and the machine will be able to understand what they say, rather than how they say it. For example, the Amazon Echo interrupts and contributes to conversations through notifications and relevant product suggestions. [7]

Smarter Search

The advancement in search capabilities allows one to search using conversational language rather than keywords or topics. Smarter search is a functionality of “searching as you would talk”, similar to querying a virtual assistant.[8]

Business Intelligence into Conversation

Big Data on its own is not useful for decision-making until it is analyzed and that is where Business Intelligence (BI) comes in. BI tools supports decision-making, optimizes internal business processes, and increases the operational efficiency. By turning BI into a conversation with NLP, the future of BI seems promising. An example of how this works is Analytics Advisor, powered by Alexa.

Accessibility of Data

NLP will make data more accessible and user-friendly for all involved in the business process. Instead of querying databases and interacting with complex systems, big data can be queried through speech and human-like language. This makes it possible for non-technical people to interact with data efficiently to support their decisions in real time.[9]

Understanding of Query

NLP will make BI more insightful with the maturity of technology, especially in AI. It will allow the computer to “understand” the query and provide a direct natural language answer to the question asked. Once the bot learns the semantic relations and inferences of the question, it will automatically filter and organize the data. Then, it will present the user with an intelligent answer. Rather than showing data or the raw search results, it enables the user to converse with the bot meaningfully to make better business decisions.[1]

Intelligence from Unstructured Data

NLP helps to harness unstructured data by uncovering patterns in the the scattered data that can be analyzed for relevant information. The accuracy which machines understand human language through sentiment analysis allows it to extract information from the surrounding data and make it more invaluable. A tool that companies can use to uncover insights from unstructured data is IBM’s Watson Natural Language Understanding, which has the capabilities to understand sentiment and emotion.

NLP and Social Media

With its ability to analyze patterns to understand text, NLP will continue to play a significant role in social media. Whether it is a tweet on Twitter or a comment on Facebook, NLP can determine the tone of the text and categorize it as either positive, negative, or neutral. The results from this analysis can assist companies in determining customer brand perception and the best way to approach any negativity. However, it is important to note that the language varies depending on the medium, and the way people express themselves differs. This makes it difficult for a machine to decipher the message accurately. For example, sarcasm creates complexity for a machine to interpret, even if it has good grasp of the context. In the foreseeable future, the advancements in NLP continue to struggle with gaining human intuition.[1]

NLP and Healthcare Industry

NLP is also used with unstructured data in the healthcare industry. Some specific tasks of NLP systems include summarizing long narrative texts and mapping data elements from unstructured text to its appropriate structured field. NLP is working towards bridging the gap between the data available and the limited cognitive capacity of the mind. [2]

Virtual to Personal Assistants

In the near future, VAs will not only take orders, but they will also have ideas of their own. They will become personal assistants, helping the user with their goals. For example, if a ‘personal assistant VA’ knows that it will rain later in the day, it will suggest that the user should bring an umbrella. By understanding the user’s needs through NLP, personal assistants will become more proactive in communicating with humans, influencing the decisions humans make. They will support and mentor us. Based on all the structured and unstructured data it gathers and stores, they will know their users better than anyone else in this world, taking a more human-like approach. [3]


Alicia Chan Christina Thompson Jessie Zhang Natalie Lee Kean Tran
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada
Beedie School of Business
Simon Fraser University
Burnaby, BC, Canada



Authors: Alicia, Christina, Jessie, Kean, Natalie

Personal tools