Welcome to the second AI Spy, a glossary of AI-related concepts. I’m going to delve into the world of Natural Language Processing (NLP), a subfield of machine learning that enables computers to understand and generate human language. You’ll learn how NLP works, and its applications, and get hands-on with a popular NLP library (if you fancy it).
What Is NLP
Talking in binary doesn’t come naturally to humans, so programming languages and ways we interact with machines have grown to more closely mimic our everyday speech. The best example of this is Natural Language Processing (NLP). The clue is in the name.
Being able to classify and analyse vast amounts of data is an ongoing challenge, so it’s not surprising that NLP has emerged as a powerful tool that enables people to extract valuable insights. It’s used across a wide range of industries, like marketing, customer service, translation, and many more.
You’ll have no doubt come across NLP many times already with chatbots, which everybody hated until ChatGPT was thrust onto the scene and caused a bit of a fuss with its powerful use of language analysis and text generation.
It might seem like ChatGPT just appeared overnight, but it took us a while to get here.
A Brief History of NLP
NLP has come to fruition over the years thanks to the collective skills of researchers from various disciplines, including computer science, linguistics, and artificial intelligence.
The need for NLP was identified by researchers in the post-war 1940s as a desire for machine translation. The scope started to broaden to the wider challenge of enabling computers to understand and generate human language. It wasn’t until 1958 when pioneering linguist (just one of his many hats) Noam Chomsky identified significant issues - that the language models at the time could not distinguish between grammatically correct and incorrect sentences in the same way humans do.
The techniques to improve the accuracy and capabilities of NLP have evolved over the years. Most recently, it has taken a neural network-based approach, but in the past, it was more rule-based or statistical. Neural networks are heavily data-driven, and the proliferation of large datasets and a steady increase in computational power are just a couple of reasons why it has become more feasible for NLP to rely on this approach.
How Does NLP Work
The important thing is that NLP models can learn directly from data. The more data they get, the better they can become.
There are several steps taken for NLP models to achieve their output.
- Text is converted to numbers. Computers and thus neural network models prefer numbers, so words get converted into numerical representations, called vectors.
- The neural network model gets fed. These numerical word vectors are fed into a model. This so-called model is like a complex mathematical function that can learn from examples. There are a few different types of models, but transformer models are the ones that are so hot right now.
- Pre-training. The data-hungry models chow down on huge amounts of text taken from various sources like books or websites. This gives them an arguably well-balanced diet before they’re trained on any specific task.
- Learning from data and fine-tuning. As mentioned, more data is better, so if the neural network is shown many examples of texts and their corresponding labels, the model learns to recognise patterns. For example, sentiment analysis tasks need to know what positive or negative sentiments are.
- Inference and predictions. Finally, when the model is given a new piece of text, it can use everything it has learned to make a prediction and output something relevant to the task.
It’s not just sentiment analysis. There’s a wide range of tasks NLP models can perform.
What Can NLP Do
In a rapidly evolving field, NLP has numerous applications:
- Language generation. As much as everybody seems to hate customer service chatbots, they’re out there and growing in popularity. NLP is used to understand the customer’s question and generate responses. Then there’s the general-purpose chatbot, ChatGPT, which showcased the powers of NLP to a wider audience with its ability to generate or summarise entire essays.
- Language translation. Google Translate and Microsoft Translator use NLP to translate languages in real time.
- Text analysis. Tools like Hootsuite can analyse sentiment on social media. Sentiment analysis can also provide summaries of user reviews, as seen on Amazon. It’s also possible to recognise entities (names of people, places).
- Voice recognition. It’s not just text recognition. NLP is used in virtual assistants like Amazon’s Alexa and Apple’s Siri. It can also transcribe audio, such as the transcripts automatically generated from YouTube videos.
NLP has some pretty big implications, including but not limited to extracting information from medical records and classifying documents, financial institutions using it to gauge market sentiment from social media posts to make stock market predictions, smart email filtering, enhanced search functionality on e-commerce platforms to help users find what they’re looking for.
But NLP has been hiding in plain sight for decades - those spell checkers and grammar tools we take for granted use natural language processing techniques under the hood!
I’ve scratched the surface with the history and capabilities of NLP, so I hope this whistlestop tour has given you a good grasp of what NLP is all about, and that you’re now as interested as I am to see how this rapidly evolving field develops.
BONUS: How To Use a NLP Library
I was going to headline that ‘Gensim Impact’ but decided against it.
Over the years a number of NLP libraries for programming languages (well, mostly Python) have appeared, including NLTK, spaCy, Gensim, Stanza, and TextBlob. But there’s one emerging king which is the Transformers library by 🤗Hugging Face, which has a number of different pipelines (APIs) for you to play with.
This section presumes a basic level of Python literacy. If you don’t know what Python is, it may not be for you.
- Activate your Python environment and install Transformers and Numpy.
pip install transformers numpy- Install a machine learning library.
If you don’t have a machine learning library installed already you’ll need one of them too, so either install PyTorch or TensorFlow. Getting TensorFlow to recognise my GPU was a hassle so I wrote a guide on working with TensorFlow in Docker.
PyTorch, however, seems to be the boss these days so if you want what’s hot right now, get the installation command from PyTorch’s website. Tip: If using pip3 in the command doesn’t work for some reason, change it to pip instead without the 3. That worked for me although I have no idea why.
- Write/copy’n’paste some code.
This uses Transformers’ ‘sentiment analysis’ pipeline to determine if the sentiment is positive or negative.
from transformers import pipeline
def perform_sentiment_analysis(
text, model="distilbert-base-uncased-finetuned-sst-2-english", revision="main"
):
nlp = pipeline(
"sentiment-analysis",
model=model,
revision=revision,
)
result = nlp(text)[0]
return f"label: {result['label']}, with score: {round(result['score'], 4)}"
print(perform_sentiment_analysis("I love this movie!"))
print(perform_sentiment_analysis("I hate this movie!"))This ‘question-answering’ code uses a version of the BERT model which has been fine-tuned on the SQuAD (Stanford Question Answering Dataset) dataset. It takes a context and question as the input and returns its answers.
from transformers import pipeline
def answer_question(context, question):
nlp = pipeline(
"question-answering",
model="distilbert-base-uncased-distilled-squad",
revision="main",
)
result = nlp(question=question, context=context)
return f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
context = r"""
The Eiffel Tower is a wrought iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower. Constructed from 1887 to 1889 as the entrance to the 1889 World's Fair, it was initially criticized by some of France's leading artists and intellectuals for its design, but it has become a global cultural icon of France and one of the most recognizable structures in the world. The Eiffel Tower is the most-visited paid monument in the world; an estimated 6.91 million people ascended it in 2015.
"""
print(answer_question(context, "Who designed the Eiffel Tower?"))
print(answer_question(context, "When was the Eiffel Tower constructed?"))
print(answer_question(context, "How many people visited the Eiffel Tower in 2015?"))This little ‘text-generation’ experiment with little baby GPT2 came out with something very, um, interesting. Feel free to experiment with different models and inputs.
from transformers import pipeline
def generate_text(prompt, max_length=100):
text_generator = pipeline("text-generation", model="gpt2")
result = text_generator(prompt, max_length=max_length, do_sample=True)
return result[0]["generated_text"]
prompt = "In a world where AI has become an integral part of our daily lives, "
print(generate_text(prompt))Its output for me was this cheery missive: “In a world where AI has become an integral part of our daily lives, ills have become increasingly visible as AI continues to reshape the world of business as well. While corporations have a responsibility to care about how their work lives impact everyone else, it is important for organizations to be aware of the fact that their employees are underpaid, understaffed, and underpaid at work. At Apple we are making a conscious effort to address these factors. That said, some workers may not know this”
Makes you think.
Sources
https://cs.stanford.edu/people/eroberts/courses/soco/projects/2004-05/nlp/overview_history.html
https://docs.cohere.com/docs/history-of-nlp
https://exaud.com/noam-chomsky-computer-science/
https://www.dataversity.net/a-brief-history-of-natural-language-processing-nlp/
https://aws.amazon.com/what-is/neural-network/
https://towardsai.net/p/machine-learning/modern-nlp-a-detailed-overview-part-1-transformers
https://www.datacamp.com/blog/what-is-natural-language-processing
https://huggingface.co/docs/transformers/main_classes/pipelines