In our work, we have the pleasure of meeting many entrepreneurs who want to “do something with NLP”, and organizations who want to count NLP in their digital stack. Indeed, NLP is cool and trendy – especially since it deals with language, a natural and powerful capacity of our brains. But when it comes to business, using NLP for the sake of NLP is not a good idea – it is a tool and should be implemented with a specific use case in mind. In the right contexts, NLP can increase productivity or enhance existing knowledge. And, when married with specific expertise and intuition from the actual business domain, it can activate your creative muscles and trigger disruptive ideas and completely new ways of doing things. The following chart shows the variety of business functions that are affected by NLP, based on our analysis of related online news and discussions:

Business functions discussed in the context of NLP (based on online texts 2019)

In the following, we explain three overarching goals for implementing NLP inside a business and share our experience about priorities and common pitfalls for each category.

Streamline existing tasks

You can save costs with NLP by increasing the productivity of your work with existing text data. NLP can be used to streamline routines that involve texts. This scenario is not only easy to understand, but also easy to measure in terms of ROI – you simply count the cost in man-hours used for a specific task “before” and “after” the implementation of NLP.

A prominent and familiar example is customer support. Tasks in customer support are often focussed around a small number of variables such as products and processes. These are perfectly known inside the business, but may not be familiar to external customers. On the receiving side, NLP can be applied to analyse and classify standard requests such as calls and complaint emails. Responses can be automated with conversational AI, as implemented in chatbots and virtual assistants. Algorithms such as sentiment analysis allow to evaluate large quantities of customer feedback, thus giving the business the opportunity to react and win on customer centricity.

There are many other examples for NLP automation, such as the use of machine translation software and intelligent document analysis. The nice thing about most of these applications is that training data is already available in the organization – the challenge is to set up a stable, sustainable supply of clean data and a feedback loop via your employees. The actual NLP implementation can be based on standard algorithms, such as sentiment analysis and entity recognition, customized with business-specific knowledge. This customization is often lexical. For example, to tailor an entity extraction algorithm to parse product failure complaints, it will be useful to “teach” the entity recognizer which products and features are likely to fail. Finally, in production mode, human verification will still be needed for those cases where the NLP algorithms are not confident about their output.

Support your decisions with better information

This second area allows to enhance existing data analytics use cases with the power of text data. Most of us heard that 80% of business-relevant data exists in unstructured form. However, with NLP just entering the “live” business arena, mainstream analytical techniques still focus on statistical and quantitative analysis of structured data – i.e., the “other” 20%. By adding unstructured data sources to the equation, a business can improve the quality and granularity of the generated results. Ultimately, NLP generates a unique information advantage and paves the way to better decisions.

An example where this scenario applies is the screening of target companies for M&A transactions. The “traditional” target screening process is highly structured and quantitative. It focusses on basic factors such as location and area of the business, formal criteria (legal form, shareholder structure) and, of course, financial indicators such as revenue and profitability. Many of the less tangible, but central aspects of a business – for example, its intellectual property, public image and the quality of the team – don’t surface in the result and have to be manually investigated on the basis of additional data sources. NLP allows to leverage a variety of text data that contains information about a company – social media, business news, patents etc. – to efficiently extract this information for a large number of companies.

NLP can enhance decision making in all areas of market intelligence, such as trend monitoring, consumer insight and competitive intelligence. In general, use cases in this category require a more involved layer of business logic. While NLP is used to structure large quantities of data, additional knowledge of the business and its context has to be applied to make sense of this data. The M&A screening example first requires a definition of the goals of an M&A transaction and, from there, the relevant parameters: if the goal is to expand B2C sales into a different geography, the perception of the target company by consumers is crucial. On the other hand, the acquisition of a complementary technology will direct the focus on the intellectual assets of the target. The relevant parameters for each goal have to be formulated by the business-side team and can then be coded into a machine-usable form.

Conquering the greenfields

So far, we have seen relatively defensive approaches to improving what is already being done. But NLP can also trigger bold new “ways of doing things” and lead to high-impact applications that might justify whole new businesses. This journey requires the right equipment – not only solid domain knowledge, but also market expertise and the ability to find sweet spots at the intersection of technology and market opportunity.

As an example, NLP can be applied in the mental health area to analyze the mental and emotional state of a person. This can be used to identify endangered individuals, such as individuals suffering from severe depression and suicide risk. Traditionally, these individuals are identified and treated upon a proactive doctor visit. Naturally, the more “passive” cases are rarely recognized in time. NLP techniques such as sentiment and emotion analysis can be applied on social media to screen the mental and emotional states of users, thus pointing out individuals that are in a high-risk state for further support.

Further examples for disruptive use cases can be found in various industries, such as drug discovery in healthcare and automatic broadcasting in media. Venturing in this space requires a high confidence and competence in one’s own industry. As everywhere else, disruptions are often pioneered by start-ups whose flexibility and innovation focus give rise to fruitful intersections between business model and technology. However, with the right amount of technological competence and a technology-driven mindset, incumbents can also strive in these areas, massively capitalizing on existing assets such as data, market expertise and customer relations.

In the end – things don’t come for free. NLP has the potential to save costs, improve decision making and disrupt any language-intensive area. To get on this path, businesses need crystal-clear formulations of their goals and use cases and the willingness to customize out-of-the-box NLP offerings to their specific knowledge base. Those who get it right will not only reap the benefits of specific projects down the road, but also uncover new use cases and strategic potentials for NLP throughout the whole organization.

In this article, we review major research trends in the animated NLP space and formulate some implications for the business perspective. The article is backed by a statistical and NLP-based analysis of papers from ACL, the Annual Conference of the Association of Computational Linguistics, which is the major international conference for NLP and computational linguistics.

1. Motivation

When compared to other species, natural language is one of the primary USPs of the human mind. NLP, a major buzzword in today’s tech discussion, deals with how computers can understand and generate language. The rise of NLP in the past decades is backed by a couple of global developments – the universal hype around AI, exponential advances in the field of Deep Learning and an ever-increasing quantity of available text data. But what is the substance behind the buzz? In fact, NLP is a highly complex, interdisciplinary field that is constantly supplied by high-quality fundamental research in linguistics, math and computer science. The ACL conference brings these different angles together. As the following chart shows, research activity has been flourishing in the past years:

No alt text provided for this image

Figure 1: Paper quantity published at the ACL conference by years

In the following, we summarize some core trends in terms of data strategies, algorithms, tasks as well as multilingual NLP. The analysis is based on ACL papers published since 1998 which were processed using a domain-specific ontology for the fields of NLP and Machine Learning.

2. Data: working around the bottlenecks

The quantity of freely available text data is increasing exponentially, mainly due to the massive production of Web content. However, this large body of data comes with some key challenges. First, large data is inherently noisy. Think of natural resources such as oil and metal – they need a process of refining and purification before they can be used in the final product. The same goes for data. In general, the more “democratic” the production channel, the dirtier the data – which means that more effort has to be spent on its cleaning. For example, data from social media will require a longer cleaning pipeline. Among others, you will need to deal with extravagancies of self-expression like smileys and irregular punctuation, which are normally absent in more formal settings such as scientific papers or legal contracts.

The other major challenge is the labeled data bottleneck: strictly speaking, most state-of-the-art algorithms are supervised. They not only need labeled data – they need Big Labeled Data. This is especially relevant for the advanced, complex algorithms of the Deep Learning family. Just as a child’s brain first needs a max of input before it can learn its native language, to go “deep”, an algorithm first needs a large quantity of data to embrace language in its whole complexity.

Traditionally, training data at smaller scale has been annotated manually. However, dedicated manual annotation of large datasets comes with efficiency trade-offs which are rarely acceptable, especially in the business context.

What are the possible solutions? On the one hand, there are some enhancements on the management side, incl. crowd-sourcing and Training Data as a Service (TDaaS). On the other hand, a range of automatic workarounds for the creation of annotated datasets have also been suggested in the machine learning community. The following chart shows some trends:

No alt text provided for this image

Figure 2: Discussion of approaches for creation and reuse of training data (amounts of mentions normalised by paper quantity in the respective year)

Clearly, pretraining has seen the biggest rise in the past five years. In pre-training, a model is first trained on a large, general dataset and subsequently tweaked with task-specific data and objectives. Its popularity is largely due to the fact that companies such as Google and Facebook are making huge models available out-of-the-box to the open-source community. Especially pre-trained word embeddings such as Word2Vec, FastText and BERT allow NLP developers to jump to the next level. Transfer learning is another approach to reusing models across different tasks. If the reuse of existing models is not an option, one can leverage a small quantity of labeled data to automatically label a larger quantity of data, as is done in distant and weak supervision – note, however, that these approaches usually lead to a decrease in the labeling precision.

3. Algorithms: a chain of disruptions in Deep Learning

In terms of algorithms, research in recent years has been strongly focussed on the Deep Learning family:

No alt text provided for this image

Figure 3: Discussion of Deep Learning algorithms (amounts of mentions normalised by paper quantity in the respective year)

Word embeddings are clearly taking up. In their basic form, word embeddings were introduced by Mikolov et al. (2013). The universal linguistic principle behind word embeddings is distributional similarity: a word can be characterized by the contexts in which it occurs. Thus, as humans, we normally have no difficulty completing the sentence “The customer signed the ___ today” with suitable words such as “deal” or “contract”. Word embeddings allow to do this automatically and are thus extremely powerful for addressing the very core of the context awareness issue.

While word2vec, the original embedding algorithm, is statistical and does not account for complexities of life such as ambiguity, context sensitivity and linguistic structure, subsequent approaches have enriched word embeddings with all kinds of linguistic information. And, by the way, you can embed not only words, but also other things such as senses, sentences and whole documents.

Neural Networks are the workhorse of Deep Learning (cf. Goldberg and Hirst (2017) for an introduction of the basic architectures in the NLP context). Convolutional Neural Networks have seen an increase in the past years, whereas the popularity of the traditional Recurrent Neural Network (RNN) is dropping. This is due, on the one hand, to the availability of more efficient RNN-based architectures such as LSTM and GRU. On the other hand, a new and pretty disruptive mechanism for sequential processing – attention – has been introduced in the sequence-to-sequence (seq2seq) model by Sutskever et al. (2014). If you use Google Translate, you might have noticed the leapfrog in the translation quality a couple of years ago – seq2seq was the culprit. And while seq2seq still relies on RNNs in the pipeline, the transformer architecture, another major advance from 2017, finally gets rid of recurrence and completely relies on the attention mechanism (Vaswani et al. 2017).

Deep Learning is a vibrant and fascinating domain, but it can also be quite intimidating from the application point of view. When it does, keep in mind that most developments are motivated by increased efficiency at Big Data scale, context awareness and scalability to different tasks and languages. For a mathematical introduction, Young et al. (2018) present an excellent overview of the state-of-the-art algorithms.

4. Consolidating various NLP tasks

When we look at specific NLP tasks such as sentiment analysis and named entity recognition, the inventories are much steadier than for the underlying algorithms. Over the years, there has been an gradient evolution from preprocessing tasks such as stemming over syntactic parsing and information extraction to semantically oriented tasks such as sentiment/emotion analysis and semantic parsing. This corresponds to the three “global” NLP development curves – syntax, semantics and context awareness – as described by Cambria et al. (2014). As we have seen in the previous section, the third curve – the awareness of a larger context – has already become one of the main drivers behind new Deep Learning algorithms.

From an even more general perspective, there is an interesting trend towards task-agnostic research. In Section 2, we saw how the generalization power of modern mathematical approaches has been leveraged in scenarios such as transfer learning and pre-training. Indeed, modern algorithms are developing amazing multi-tasking powers – thus, the relevance of the specific task at hand decreases. The following chart shows an overall decline in the discussion of specific NLP tasks since 2006:

No alt text provided for this image

Figure 4: Amount of discussion of specific NLP tasks

5. A note on multilingual research

With globalization, going international becomes an imperative for business growth. English is traditionally the starting point for most NLP research, but the demand for scalable multilingual NLP systems increases in recent years. How is this need reflected in the research community? Think of different languages as different lenses through which we view the same world – they share many properties, a fact that is fully accommodated by modern learning algorithms with their increasing power for abstraction and generalization. Still, language-specific features have to be thoroughly addressed especially in the preprocessing phase. As the following chart shows, the diversity of languages addressed in ACL research keeps increasing:

No alt text provided for this image

Figure 5: Frequent languages per year (> 10 mentions per language)

However, just as seen for NLP tasks in the previous section, we can expect a consolidation once language-specific differences have been neutralized for the next wave of algorithms. The most popular languages are summarised in Figure 6.

No alt text provided for this image

Figure 6: Languages addressed by ACL research

For some of these languages, research interest meets commercial attractiveness: languages such as English, Chinese and Spanish bring together large quantities of available data, huge native speaker populations and a large economic potential in the corresponding geographical regions. However, the abundance of “smaller” languages also shows that the NLP field is generally evolving towards a theoretically sound treatment of multilinguality and cross-linguistic generalisation.

Summing up

Spurred by the global AI hype, the NLP field is exploding with new approaches and disruptive improvements. There is a shift towards modeling meaning and context dependence, probably the most universal and challenging fact of human language. The generalisation power of modern algorithms allows for efficient scaling across different tasks, languages and datasets, thus significantly speeding up the ROI cycle of NLP developments and allowing for a flexible and efficient integration of NLP into individual business scenarios.

Follow us for a review of ACL 2019 and more updates on NLP trends!


  • E. Cambria and B. White (2014). Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]. Comp. Intell. Mag. 9, 2.
  • J. Devlin, M. Wei, K. Lee and K. Toutanova (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
  • Y. Goldberg and G. Hirst (2017). Neural Network Methods in Natural Language Processing. Morgan & Claypool Publishers.
  • T. Mikolov et al. (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems – vol. 2 (NIPS’13).
  • R. Prabhavalkar, K. Rao, Kanishka, T. Sainath, B. Li, L. Johnson and N. Jaitly (2017). A Comparison of Sequence-to-Sequence Models for Speech Recognition. 939-943. 10.21437/Interspeech.2017-233.
  • I. Sutskever, O. Vinyals, and Q. V. Le (2014). Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems – vol. 2 (NIPS’14).
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser and I. Polosukhin (2017). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17).
  • T. Young, D. Hazarika, S. Poria and E. Cambria (2018). Recent Trends in Deep Learning Based Natural Language Processing. In IEEE Computational Intelligence Magazine – vol. 13.

In the past years, the tech world has seen a surge of Natural Language Processing (NLP) applications in various areas, including adtech, publishing, customer service and market intelligence. According to Gartner’s hype cycle, NLP has reached the peak of inflated expectations in 2018. Many businesses see it as a “go-to” solution to generate value from the 80% of business-relevant data that comes in unstructured form. To put it simply – NLP is wildly adopted with wildly variable success.

In this article, I share some practical advice for the smooth integration of NLP into your tech stack. The advice summarizes the experience I have accumulated on my journey with NLP — through academia, a number of industry projects, and my own company which develops NLP-driven applications for international market intelligence. The article does not provide technical details but focusses on organisational factors including hiring, communication and expectation management.

Before starting out on NLP, you should meditate on two questions:

1. Is a unique NLP component critical for the core business of our company?

Example: Imagine you are a hosting company. You want to optimise your customer service by analysing incoming customer requests with NLP. Most likely, this enhancement will not be part of your critical path activities. By contrast, a business in targeted advertising should try to make sure it does not fall behind on NLP — this could significantly weaken its competitive position.

2. Do we have the internal competence to develop IP-relevant NLP technology?

Example: You hired and successfully integrated a PhD in Computational Linguistics with the freedom to design new solutions. She will likely be motivated to enrich the IP portfolio of your company. However, if you are hiring middle-level data scientists without a clear focus on language that need to split their time between data science and engineering tasks, don’t expect a unique IP contribution. Most likely, they will fall back on ready-made algorithms due to lack of time and mastery of the underlying details.

Hint 1: if your answers are “yes” and “no” — you are in trouble! You’d better identify technological differentiators that do match your core competence.

Hint 2: if your answers are “yes” and “yes” — stop reading and get to work. Your NLP roadmap should already be defined by your specialists to achieve the business- specific objectives.

If you are still there, don’t worry – the rest will soon fall in place. There are three levels at which you can “do NLP”:

  1. Black belt level, reaching deep into mathematical and linguistic subtleties
  2. Training & tuning level, mostly plugging in existing NLP/ML libraries
  3. Blackbox level, relying on “buying” third-party NLP

The black belt level

Let’s elaborate: the first, fundamental level is our “black belt”.  This level comes close to computational linguistics, the academic counterpart of NLP. The folks here often split into two camps — the mathematicians and the linguists. The camps might well befriend each other, but the mindsets and the way of doing things will still differ.

The math guys are not afraid of things like matrix calculus and will strive on details of newest methods of optimisation and evaluation. At the risk of leaving out linguistic details, they will generally take the lead on improving the recall of your algorithms. The linguists were raised either on highly complex generative or constraint-based grammar formalisms, or alternative frameworks such as cognitive grammar. These give more room to imagination but also allow for formal vagueness. They will gravitate towards writing syntactic and semantic rules and compiling lexica, often needing their own sandbox and taking care of the precision part. Depending on how you handle communication and integration between the two camps, their collaboration can either block productivity or open up exciting opportunities.

In general, if you can inject a dose of pragmatism into the academic perfectionism you can create a unique competitive advantage. If you can efficiently combine mathematicians and linguists on your team — even better! But be aware that you have to sell them on an honest vision — and then, follow through. Doing hard fundamental work without seeing its impact on the business would be a frustrating and demotivating experience for your team.

The training & tuning level

The second level involves the training and tuning of models using existing algorithms. In practice, most of the time will be spent on data preparation, training data creation and feature engineering. The core tasks — training and tuning — do not require much effort. At this level, your people will be data scientists pushing the boundaries of open-source packages, such as nltk, scikit-learn, spacy and tensorflow, for NLP and/or machine learning. They will invent new and not always academically justified ways of extending training data, engineering features and applying their intuition for surface-side tweaking. The goal is to train well-understood algorithms such as NER, categorisation and sentiment analysis, customized to the specific data at your company.

The good thing here is that there are plenty of great open-source packages out there. Most of them will still leave you with enough flexibility to optimize them to your specific use case. The risk is on the side of HR — many roads lead to data science. Data scientists are often self-taught and have a rather interdisciplinary background. Thus, they will not always have the innate academic rigour of level 1 scientists. As deadlines or budgets tighten, your team might get loose on training and evaluation methods, thus accumulating significant technical debt.

The blackbox level

On the third level is a “blackbox” where you buy NLP. Your developers will mostly consume paid APIs that provide the standard algorithm outputs out-of-the-box, such as Rosette, Semantria and Bitext (cf. this post for an extensive review of existing APIs). Ideally, your data scientists will be working alongside business analysts or subject matter experts. For example, if you are doing competitive intelligence, your business analysts will be the ones to design a model which contains your competitors, their technologies and products.

At the blackbox level, make sure you buy NLP only from black belts! With this secured, one of the obvious advantages of outsourcing NLP is that you avoid the risk of diluting your technological focus. The risk is a lack of flexibility — with time, your requirements will get more and more specific. The better your integration policy, the higher the risk that your API will stop satisfying your requirements. It is also advisable to invest into manual quality assurance to make sure the API outputs deliver high quality.

Final Thoughts

So, where do you start? Of course, it depends — some practical advice:

  • Talk to your tech folks about your business objectives. Let them research and prototype and start out on level 2 or 3.
  • Make sure your team doesn’t get stuck in low-level details of level 1 too early. This might lead to significant slips in time and budget since a huge amount of knowledge and training is required.
  • Don’t hesitate — you can always consider a transition between 2 and 3 further down the path (by the way, this works in any direction). The transition can be efficiently combined with the generally unavoidable refactoring of your system.
  • If you manage to build up a compelling business case with NLP — welcome to the club, you can use it to attract first-class specialists and add to your uniqueness by working on level 1!

About the author: Janna Lipenkova holds a PhD in Computational Linguistics and is the CEO of Anacode, a provider of tech-based solutions for international market intelligence. Find out more about our solution here

Author: Janna Lipenkova

Market research surveys typically consist of two types of questions, namely “closed” and “open” questions. Closed questions limit the possible range of responses and result in structured data. By contrast, “open” questions allow the respondent to reply with free text, expressing their full-fledged, authentic opinion. This flexibility makes open questions attractive in market research: given the right wording, an open question can trigger responses that are broader, deeper and provide more authentic insight than the rigid, black-and-white multiple-choice question. 

Challenges with the analysis of open-ended questions

Why, then, are open-ended questions not widely adopted in market research? One of the reasons is that they are difficult to analyze due to their unstructured character. Most researchers use manual coding, where open questions are manually structured into a classification scheme (the “coding frame”) according to the topics they represent. The following table shows some examples:

Manual coding comes with several issues:

  • High cost: manual coding is labor-intensive and thus expensive.
  • Errors: depending on their mental and physical shape, human coders make mistakes or provide inconsistent judgments at different points in time.  
  • Subjectivity: due to the inherent ambiguity and the subtleties involved in human language, different people might code the same response in different ways. 

Last but not least, coding hundreds or even thousands of open-end questions can be a frustrating endeavor. In today’s world, where AI is used to automate and optimize virtually every repetitive task, it seems natural to turn to automated processing to eliminate the monotone parts of coding. Beyond the several benefits of automation, this also creates time for more involved challenges that require the creativity, experience and intellectual versatility of the human brain. 

Using Natural Language Processing as a solution

Natural Language Processing (NLP) automates the manual work researchers do when they code open-ended questions. It structures a text according to the discussed topics and concepts as well as other relevant metrics, such as the frequency, relevance and sentiment. Beyond speeding up the coding process, NLP can be used to discover additional insights in the data and enrich the end result. The capacity of a machine to look at a large dataset as a whole and discover associations, regularities and outliers is larger than that of the human brain. 

Three algorithms – topic modeling and classification, concept extraction and sentiment analysis – are particularly useful in the coding process. 

Topic modeling and classification

Topic modeling detects abstract topics in the text. Topic modeling is an unsupervised learning method similar to clustering, and learns lexical similarities between texts without a predefined set of classes. Thus, it is particularly useful in the initial stage of the construction of a coding frame. The following word cloud shows words that are frequently mentioned in texts about comfort:

Topic classification is similar to topic modeling. However, it works with a given coding frame and classifies each text into one of the predefined classes. This means, that it can be used for coding after the coding frame has been constructed.

Concept extraction

Concept extraction matches concrete strings in the text. Whereas topic modeling and classification work with – often implicit – lexical information distributed everywhere in the text, concept extraction matches the exact words and phrases that occur in the text. On a more advanced level, concept extraction also uses the structure of the lexicon and can deal with lexical relations, such as:

  • Synonymy: EQUALS-relationship, e. g. VW EQUALS Volkswagen 
  • Hypernymy: IS-A-relationship, e. g. Sedan IS-A Vehicle
  • Meronymy: PART-OF relationship, e. g. Engine PART-OF Car

Concept extraction usually focuses on nouns and noun phrases (engine, Volkswagen). In the context of evaluations (open-ended questions), it is also useful to extract concepts that are “hidden” in adjectives (fast ➤ Speed, cozy ➤ Comfort) and verbs (overpay ➤ Price, fail ➤ Reliability).

In terms of implementation, there are two main approaches to concept extraction: the dictionary-based approach and the machine-learning approach. The dictionary-based approach works with predefined lists of terms for each category (also called “gazeteers”). The machine-learning approach, on the other hand, learns concepts of specific types from large quantities of annotated data. As a rule of thumb, the smaller and more specific the available dataset, the more efficient the use of pre-defined lists of concepts and linguistic expressions. 

Sentiment analysis

Sentiment analysis detects whether a given text has a positive or a negative connotation. Sentiment analysis can be further detailed to the level of individual aspects mentioned in a text, thus allowing to detect mixed sentiments on the phrase level: 

“Classy and reliable, but expensive.”

Sentiment analysis operates on an emotional, subjective and often implicit linguistic level. This subtlety raises several challenges for automation. For example, sentiment analysis is highly context dependent: a vacuum cleaner that sucks would probably get a neutral-to-positive sentiment; by contrast, the internet connection in a car normally shouldn’t “suck”. Another complication is irony and sarcasm: on the lexical level, ironic statements often use vocabulary with a clear polarity orientation. However, when put into the surrounding context, this polarity is inversed:

“Really great engineering… the engine broke after only three weeks!”

Irony is mostly detected from anomalies in the polarity contrasts between neighboring text segments. For instance, in the example above, “really great engineering” gets a strong positive sentiment which radically clashes with the negative feel of “the engine broke after only three weeks”. Since the two phrases are directly juxtaposed without a conjunction such as “but” or “although”, the machine is able to recognize the overall negative connotation. 

Combining Human and Artificial Intelligence

Summing up, using NLP for the coding of open-ended questions leverages several benefits of automation: it speeds up the process and saves human labor on the “frustrating” side of things. It achieves better consistency and objectivity, mitigating the effects of human factors such as fatigue and inconsistent judgment. Finally, the ability of the machine to process large data quantities at a high level of detail allows a level of granularity that might be inaccessible to the human brain. 

While it is out of question that NLP automation increases the efficiency of verbatim coding, keep in mind that current AI technology is not perfect and should always have a human in the driving seat. Methods such as NLP can process large quantities of data in no time, but they do not yet capture the full complexity of language. A combination of high-quality NLP with a carefully engineered process for continuous optimization will ensure a rewarding journey towards in-depth understanding of the opinions, needs and wants of end consumers.   

In the current flood of Business Intelligence and insight tools, there is a phrase causing users to abandon the fanciest tools and leading to serious self-doubt for the provider – the “so what?” question. Indeed, your high-quality analytics application might spit out accurate, statistically valid data and package them into intuitive visualisations – but if you stop there, your data has not yet become a basis for decision and action. Most users will be lost or depend on the help and expertise of a business translator, thus creating additional bumps on their journey to data-driven action.

In this article, we focus on applications of Web-based Text Analytics – not “under-the-hood” technological details, but the practical use of Text Analytics and Natural Language Processing (NLP) to answer central business questions. Equipped with this knowledge, you will be able to tap into the full power of Text Analytics and fully benefit from large-scale data coverage and machine intelligence. A real-time mastery of the oceans of data floating on the Web will allow you to make your market decisions and moves with ease and confidence.


1. The basics

Before diving into details, let’s first get an understanding of how Text Analytics works. Text Analytics starts out with raw, semi-structured data – text combined with some metadata. The metadata have a custom format, although some fields, such as dates and authors, are pretty consistent across different data sources. The first step is a one-by-one analysis of these datapoints, resulting in a structured data basis with a unified schema. Even more important than the structuring is the transformation of the data from qualitative to quantitative. This transformation enables the second step – aggregation, which condenses a huge number of structured representations into a small number of consolidated and meaningful analyses, ready for visualization and interpretation by the end user.

2. Answering questions with Text Analytics

A number of questions can be answered with Text Analytics and NLP. Let’s start with the basics – what do users talk about and how do they talk about it? We’ll be providing examples from the Chinese social media landscape on the way.

First, the what – what is relevant, popular or even hot? This question can be answered with two algorithms:

  • Text categorisation classifies a text into one or multiple predefined categories. The category doesn’t need to explicitly be named in the text – instead, the algorithm takes words and their combinations as cues (so-called features) to recognise the category of the text. Text categorisation is a coarse-grained algorithm and thus well-suited for initial filtering or getting an overview over the dataset. For example, the following chart shows the categorisation of blog articles around the topic of automotive connectivity:
  • Concept extraction digs more into depth and identifies concepts such as brands, companies, locations and people that are directly mentioned in the text. Thus, it can identify multiple concepts of different types, and each concept can occur multiple times in the text. For example, the following chart shows mention frequencies for the most common automotive brands in the Chinese social web in February 2018:

Using time series analysis in the aggregation, text categorisation and concept extraction can be used to identify upcoming trends and topics. Let’s look into the time development for Volkswagen, the most frequent auto brand:

Once we have identified what people talk about, it is time to dig deeper and understand howthey talk about it. Sentiment analysis allows to analyze how the topics and concepts are perceived by customers and other stakeholders. Again, sentiment analysis can be applied at different levels: whole texts can be analysed for an initial overview. At an advanced stage, sentiment analysis can be applied to specific concepts to answer more detailed questions. Thus, competitor brands can be analysed for sentiment to determine the current rank of one’s own brand. Products can be analysed to find out where to focus improvement efforts. And finally, product features are analysed for sentiment to understand how to actually make improvements. As an example, the following chart shows the most positively perceived models for Audi in the Chinese web:

3. From insights to actions

Insights from Web-based Text Analytics can be directly integrated into marketing activities, product development and competitive strategy.

Marketing intelligence

By analysing the contexts in which your products are discussed, you learn the “soft” facts which are central for marketing success, such as less tangible connotations of your offering – these can be used as hints to optimise your communication. You can also understand the interest profile of your target crowd and use it to improve your story and wording. Finally, Text Analytics allows to monitor the response to your marketing efforts in terms of awareness, attitude and sentiment.

Product intelligence

With Text Analytics, you can zoom in on awareness and attitudes about your own products and find out their most relevant aspects with concept extraction. Using sentiment analysis, you can compare the perception of different products amongst each other and focus on their fine-grained features. Once you place your products and features on a positive-negative scale, you know where to focus your efforts to maximise your strengths and neutralise your weaknesses.

Competitive intelligence

Your brand doesn’t exist in a vacuum – let’s broaden our research scope. Text Analytics allows you to answer the above questions not only for your own brand, but also for your competitors. Thus, you will learn about the marketing, positioning and branding of your competitors to better differentiate yourself and present your USPs in a sharp and convincing manner. You can also analyse competitor products to learn what they did right – especially on those features where your own company went wrong. And, in a more strategic perspective, Text Analytics allows you to monitor technological trends to respond early to market developments.

So what?

How to show that your findings are not only accurate and correct, but also relevant to business success? Using continuous real-time monitoring, you can track your online KPIs and validate your actions based on the response of the market. Concept extraction can be used to measure the changes in brand awareness and relevance, whereas sentiment analysis shows how brand, product and product feature perceptions have improved based on your efforts.

With the right tools, Text Analytics can be efficiently used in stand-alone mode or as a complement to traditional field research. In the age of digitalisation, it allows you to listen to the voice of your market on the Web and turns your insight journey into an engaging path to actionable, transparent market insights.


Get in touch with Anacode’s specialists to learn how your business challenges can turn into opportunities with Text Analytics.