Artificial Intelligent Chatbot

Presenting by Madhan Kumar Selvaraj

Globally 30% of the customer service is handling by the Chatbot. Its market size is projected to grow from $2.6 billion in 2019 to $9.4 billion by 2024. Many of the people have a thought that machines will grab their job. But the reality is that the machine assist humans to do the complicated woks in a easy manner.

What this blog is about?

This blog helps you to create an Artificial intelligent Chatbot by deploying it in the Python Flask web framework and the Heroku cloud platform. Here I am doing this chatbot to assist a coaching institute to respond to the students regarding the course details and also it can answer questions apart from the course.

Everyone knows about Google assistant, Apple's Siri, Microsoft's Cortona, Amazon's Alexa. Our Chatbot also works like the same and it is having limitation because we are using limited pattern and response. If we add more pattern and response, it can response to all our queries.

Chatbot

A chatbot is an artificial intelligence (AI) software that can simulate a conversation with a user in natural language. Chatbots are not very new, one of the foremost of this kind is ELIZA, which was created in the early 1960s and is worth exploring.

• Rule-Based Approach

• In this approach, a bot is trained according to rules. Based on this a bot can answer simple queries but sometimes fails to answer complex queries.

• Self-Learning Approach

• These bots follow the machine learning approach which is rather more efficient.

Technology used here

1. NLP (Natural Language Processing)
2. CNN (Convolution Neural Network)
3. Flask web framework
4. Libraries for web scraping

Prerequisites

• Python Flask web framework
• Keras
• Tensorflow
• Heroku cloud platform
  

The workflow of this project

1. Create a pattern and response in the JSON file to help the machine to learn using a deep learning algorithm
2. Train the model by using the pattern and response file
3. Create a logic to scrape data from the Google search result
4. Develop a Flask framework for user interaction
   
5. Populate the result in the webpage by using our model
6. Deploy it in the heroku cloud platform
   

An issue with the text data

The first step of creating a chatbot is by training the model by using some algorithms. Here we are dealing with the text data and our machine knows only 0's and 1's. Even for the complex data like the image we are having a pixel value that is in the format of number and have a look in this link. For the text data, we need to use the technique called Bag of words to convert the text to numbers. I'll explain everything in the later part of the blog.

Firstly, we will create a pattern and response to feed input to our model to learn. Here I added a basic text pattern and their response in the JSON file.Change this file based on your requirement.

{"intents": [
        {"tag": "greeting",
         "patterns": ["Hi there", "How are you", "Is anyone there?","Hey","Hola", "Hello", "Good day","Good morning","Good afternoon","Good evening"],
         "responses": ["Hi","Hello, thanks for your greeting", "Good to see you again", "Hi there, how can I help?"],
         "context": [""]
        },
        {"tag": "goodbye",
         "patterns": ["Bye", "See you later", "Goodbye", "Nice chatting to you bye", "Till next time"],
         "responses": ["See you!", "Have a nice day", "Bye! Come back again soon."],
         "context": [""]
        },
        {"tag": "thanks",
         "patterns": ["Thanks", "Thank you", "helpful", "Awesome thanks", "Thanks for helping me"],
         "responses": ["Happy to help!", "Any time!", "My pleasure"],
         "context": [""]
        },
        {"tag": "noanswer",
         "patterns": [""],
         "responses": ["Sorry, can't understand you", "Please give me more info", "Not sure, I understand"],
         "context": [""]
        },
		 {"tag": "identity",
         "patterns": ["Who are you?","What is your name?"],
         "responses": ["You can call me as Asura", "I am Asura"],
         "context": [""]
        },
		 {"tag": "work",
         "patterns": ["what you can do for me?", "what you know?", "what is the use of you?","How you can help me?"],
         "responses": ["I am here to guide you for Data science course","Assist you for Data science course"],
         "context": [""]
        },
		 {"tag": "datascience",
         "patterns": ["Tell me about data science","Use of learning data science"],
         "responses": ["It is field of study that gives computers the ability to learn without being explicitly programmed"],
         "context": [""]
        },
		 {"tag": "dsgrowth",
         "patterns": ["What is the growth of data science","Scope of data science"],
         "responses": ["LinkedIn data shows that hiring growth has grown 74% in the past four years", "Data science is going to be one of the most sought after profession after a while just like app developers in the last few years","Career growth in Data Science can be well estimated by the fact that Harvard Business Review has called it the 'Sexiest job of the 21st century'"],
         "context": [""]
        },
		 {"tag": "coursedetail",
         "patterns": ["Tell me about the course","what about the course"],
         "responses": ["Data science course package contains statistics, machine learning and deep learning. It takes half a year to complete the course and you can have a course in Chennai, Bangalore, Mumbai, Delhi and Kolkatta"],
         "context": [""]
        },
		{"tag": "coursefee",
         "patterns": ["What is the course fees","what is the fees","How much need to pay"],
         "responses": ["You can check our website for further reference! https://madhankumarselvaraj.blogspot.com/"],
         "context": [""]
        }
   ]
}

Data preprocessing

Natural Language Took Kit (NLTK) contains a lot of text preprocessing techniques like stopwords, stemming, lemmatizer, chunking, chinking. Stopwords are words that do not contain enough significance to be used without our algorithm. We would not want these words to take up space in our database, or taking up the valuable processing time. For this, we can remove them easily by storing a list of words that you consider to stop words.
Tokenization is the act of breaking up a sequence of strings into pieces such as words, keywords, phrases, symbols and other elements called tokens. Tokens can be individual words, phrases or even whole sentences. In the process of tokenization, some characters like punctuation marks are discarded. Here we are going to use the lemmatizer to convert the word into a meaning word. For example, the word like playing, played, playful will be converted to play.

We need to separate all the unique words and their respective classes and both the pattern and tags added in the document variable.

for intent in intents['intents']:
    for pattern in intent['patterns']:
        #tokenize each word
        w = nltk.word_tokenize(pattern)
        words.extend(w)
        #add documents in the corpus
        documents.append((w, intent['tag']))
        # add to our classes list
        if intent['tag'] not in classes:
            classes.append(intent['tag'])
            
# lemmatize, lower each word and remove duplicates
words = [lemmatizer.lemmatize(w.lower()) for w in words if w not in ignore_words]
words = sorted(list(set(words)))
# sort classes
classes = sorted(list(set(classes)))

Bag of words

Bag of Words (BOW) is a method to extract features from text documents. These features can be used for training machine learning algorithms. It creates a vocabulary of all the unique words occurring in all the documents in the training set. In simple terms, it’s a collection of words to represent a sentence with word count and mostly disregarding the order in which they appear. The length of the vector will always be equal to vocabulary size.

for doc in documents:
    # initialize our bag of words
    bag = []
    # list of tokenized words for the pattern
    pattern_words = doc[0]
    # lemmatize each word - create base word, in attempt to represent related words
    pattern_words = [lemmatizer.lemmatize(word.lower()) for word in pattern_words]
    # create our bag of words array with 1, if word match found in current pattern
    for w in words:
        bag.append(1) if w in pattern_words else bag.append(0)
    # output is a '0' for each tag and '1' for current tag (for each pattern)
    output_row = list(output_empty)
    output_row[classes.index(doc[1])] = 1
    training.append([bag, output_row])

Training the model

I took the pattern value as the training input and their classes as the training output data. I used the Convolution Neural Network (CNN) with two hidden layers and 128 neurons in the first layer and 64 neurons in the second layer. Used the dropout operation to make it a sparse network instead of a deep network to avoid the problem of overfitting. Activation function Rectified Linear Unit (ReLU) is used for the neural network and the Softmax function for the output layer and Stochastic Gradient Descent (SGD) optimizer is used instead of gradient descent. Have a look at my blog for detailed information.

# Create model - 3 layers. First layer 128 neurons, second layer 64 neurons and 3rd output layer contains number of neurons
# equal to number of intents to predict output intent with softmax
model = Sequential()
model.add(Dense(128, input_shape=(len(train_x[0]),), activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(len(train_y[0]), activation='softmax'))
# Compile model. Stochastic gradient descent with Nesterov accelerated gradient gives good results for this model
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
#fitting and saving the model 
hist = model.fit(np.array(train_x), np.array(train_y), epochs=200, batch_size=5, verbose=1)

In addition to responding only based on our pattern, I included web scraping to extract the data from the google result based on the user query. Here I added Wikipedia, wikihow website to search the query based on the condition. Xpath in the script is the path of the data of Google's response page. I am not using any Google API because it is taking much time to fetch the response for our query.

import requests
from lxml import html
meta_data = "https://www.google.com/search?client=firefox-b-d&q="
def scrape_data(raw_text, text_type):
    input_type = text_type
    join_data = "
  "
    check_text = raw_text.split(" ")[-1]
    raw_text = raw_text.replace(" ","+")
    if check_text == "here" or check_text == "now" or text_type == "nearby":
        text_type = "" 
    raw_text = meta_data + raw_text + text_type
    text_type = input_type
    pageContent=requests.get(raw_text)
    tree = html.fromstring(pageContent.content)
    if check_text == "here" or check_text == "now":
        response = tree.xpath('.//div[@class="BNeawe iBp4i AP7Wnd"]/text()')[0]
    elif check_text == "link":
        response = tree.xpath('.//div[@class="X7NTVe"]//a/@href')[:6]
        response = response[::2]
        response = join_data.join(response)
    elif text_type == "nearby":
        response = tree.xpath('.//div[@class="BNeawe deIvCb AP7Wnd"]//text()')[1:4]
        response = join_data.join(response)
    else:
        response = tree.xpath(('.//div[@class="BNeawe s3v9rd AP7Wnd"]/text()'))[0]
    return response

I added the way of extracting the data from the Google search webpage by using the scraping technique.

Result for any one word answer

Search result for nearby query

Deploying in Flask web framework

I used the Flask framework in my previous blogs and have a look at it. I added CSS and HTML file and you can refer it in my GitHub repository. The use of the Flask in our project is to get the request from the user and then responding by getting an answer from our model.

from flask import Flask, render_template, request
from chatbot_model import chatbot_response
from scrape import scrape_data
check_wikipedia1 = ['what', 'is']
check_wikipedia2 = ['who', 'is']
check_wikihow = ['how', 'to']
app = Flask(__name__)
@app.route("/")
def home():
    return render_template("index.html")
@app.route("/get")
def get_bot_response():
    user_request = request.args.get('msg')  # Fetching input from the user
    user_request = user_request.lower()
    if len(user_request.split(" ")) > 1:
        check_search = user_request.split(" ")[0]
        if check_search == 'google':
            user_request = user_request.replace("google","")
            user_request = user_request.translate ({ord(c): "" for c in "!@#$%^&*()[]{};:,./<>?\|`~-=_+"})
            check_query = user_request.split(" ")[1]
            check_text = user_request.split(" ")[1:3]
            if check_text == check_wikipedia1 or check_text == check_wikipedia2:
                response = scrape_data(user_request, "wikipedia")
            elif check_text == check_wikihow:
                response = scrape_data(user_request, "wikihow")
            elif check_query == "nearby":
                response = scrape_data(user_request, "nearby")
            else:
                response = scrape_data(user_request, "")               
        else:
            response = chatbot_response(user_request)                
    else:
        response = chatbot_response(user_request)    
    return response
if __name__ == "__main__":
    app.run(threaded=False)

Output

I named the institute by the Elon Musk's son name  X Æ A-12 and called my chatbot as Asura. You can check the conversation between myself and chatbot video at the beginning of the blog.

Deployment in Heroku cloud

Heroku is a container-based cloud Platform as a Service (PaaS). Developers use Heroku to deploy, manage, and scale modern apps. Our platform is elegant, flexible, and easy to use, offering developers the simplest path to getting their apps to market. Check this video for the deployment purpose.

Check the chatbot app deployed in Heroku cloud

• https://chatbot-asura.herokuapp.com/
  

Complete code

• Deploying in local machine script
• Deploying in Heroku cloud platform script
  

Finally, we learned how to create an AI chatbot based on our requirements and deployed in the Flask web framework and Heroku cloud for user interaction. Take this project as a model and add additional features that you like. I hope you like this blog.

P.S - Don't make use of all your time for others. Schedule a few time for you by doing your hobbies like reading books, doing craftworks, drawing, watching movies. This makes some sense of satisfaction and makes you love yourself. Have a smile on your face every day.