Review-driven recommendation

-

 Presenting by Madhan Kumar Selvaraj


 After a very long time, I am blogging about new project named Review-driven recommendation. In this Artificial Intelligence (AI) era, data is a called as new gold. Now it's very common to buy any products and choosing a restaurant to eat we are checking the other user reviews. Once we are satisfied with their review we'll choose that particular product.

Real-world issue

  • Let's assume we are planning to buy an ABC brand model mobile. But the product review is available at many platforms like Google reviews, MouthShut, yelp and lot more. 
  • We need to check all the reviews from the users and from different platforms to get most of it. It'll take more time and effort.

 To fix this issue, we are extracting user reviews from all the platforms and using some NLP models to get the top best products list with some visualization.

Project roadmap


The workflow of the project

  1. Scraping user reviews from the multiple platforms
  2.  Extracted data are combined and stored in the JSON format
  3. Preprocessing the extracted data and applying spacy model
  4. Result data are visualized to get the best products

Technologies used here

  • Python
  • web scraping
  • pandas
  • NLTK
  • Spacy
  • Plotly

 Review-driven recommendation

Scraping the data

  • To scrape the reviews from the Google reviews, we need some pre-requisites for the web scraping that you can refer from my previous blogs
  • We are having 2 parts in scraping
    1. Getting top suggestion from the Google Maps
    2. Using the suggestion data, we are scraping its data
      •  
  • Below code helps to get the top recommendation from the Google Map
def start_requests(self):
    self.params["q"] = f"{self.search_for} at {self.place}"
    response = requests.get('https://www.google.com/maps', params=self.params, headers=self.headers)
    if response.ok and response.content:
        tree = html.fromstring(response.content)
        try:
            rows_block = tree.xpath('//link[@rel="shortcut icon"]/following-sibling::script[1]')[0].text
            results_list = re.findall(r'''"SearchResult.TYPE_.*?],\\+"(.*?)\\+"''', rows_block)
            print(results_list)
            if results_list:
                self.review_requests([results_list[1]])
            if len(self.all_reviews) > 0:
                filename = f"{self.output_filename}_{str(datetime.date.today())}.json"
                if not os.path.exists(self.output_folder_name):
                    os.mkdir(self.output_folder_name)
                if os.path.exists(f"{self.output_folder_name}/{filename}"):
                    os.remove(f"{self.output_folder_name}/{filename}")
                with open(f"{self.output_folder_name}/{filename}", "a+") as new_file:
                    for each_row in self.all_reviews:
                        new_file.write(json.dumps(each_row))
                        new_file.write("\n")
            print(f"Total number of reviews fetched across all the links is {len(self.all_reviews)}")
        except Exception as error:
            print(error)
  • Similar to Google review, MouthShut reviews also scraped and stored in the JSON format and those code will be available at Github.

 Text preprocessing

  • Import all the necessary libraries that required to preprocess the text
import nltk
import spacy
import os
import pandas as pd
import contractions

nltk.download("words")
from datetime import datetime
from plotly.subplots import make_subplots
import plotly.graph_objects as go
from nltk.tokenize.treebank import TreebankWordTokenizer
from nltk.tokenize.treebank import TreebankWordDetokenizer
  • We are not using any in-built data to train our model. Instead, we'll use a set of positive and negative words to preprocess the extracted reviews
  • positive_words={'absolutely','accepted','acclaimed','accomplished','accomplishment','admirable','amazing','authentic','abundant','abundance','achiever','affluent','astonishing','awesome','adorable','beaming','beauty','beautiful','believe','beloved','beneficial','bestfriend','blessed','bliss','brave','breathtaking','bright','brilliant','brilliance','bravo','calm','calming','capable','captivating','caring','celebrate','certain','champ','champion','charitable','charm','charming','comfortable','committed','compassion','compassionate','confident','congratulations','correct','courage','courageous','courteous','creative','creativity','credible','credibility','daring','darling','dazzling','dear','dearest','dedicated','delicious','delight','delightful','....'}
  • negative_words={'abrasive','apathetic','controlling','dishonest','impatient','anxious','betrayed','disappointed','embarrassed','jealous','abysmal','bad','callous','corrosive','damage','despicable','donot','enraged','fail','gawky','haggard','hurt','icky','insane','jealous','lose','malicious','naive','not','objectionable','pain','questionable','reject','rude','sad','sinister','stuck','tense','ugly','unsightly','vice','wary','yell','zero','adverse','banal','cannot','corrupt','damaging','detrimental','dreadful','eroding','faulty','ghastly','hard','hurtful','ignorant','insidious','junky','lousy','mean','nasty','noxious','odious','perturb','....'}
  • Some preprocessing steps includes
  1.  dropping unwanted columns
  2. Converting the ratings greater than 3 as positive and remaining as negative
  3. Expanding the words like don't to donot by using the spacy library
df_dropped = dataframe.drop(
    ["user", "review_date", "images", "extraction_date", "review_availability", "address", "url", 'name'], axis=1)
df_filter = df_dropped.copy()
df_filter = df_filter[df_filter.review_text != ""]

df_filter["rating"] = df_dropped["rating"].str.replace(" out of 5,", "").str.replace("Rated ", "")
df_filter.rating = df_filter.rating.astype(float).astype(int)
df_filter["rating"].value_counts()
df_filter["bi_ratings"] = df_filter["rating"].apply(lambda x: "Positive" if x > 3.0 else "Negative")
df_filter.bi_ratings.value_counts()
df_filter = df_filter[df_filter.columns[[3, 1, 2, 0, 4]]]
df_filter["contractions_review"] = df_filter["review_text"].apply(lambda x: contractions.fix(x))
  •  By using our own model, we are grouping the positive and negative words from the reviews together, and we are finding some insights on it.
for hospital_name, group_df in hospital_name_groupby:
    try:
        temp = group_df[group_df['positive_words'].str.len() != 0].copy()
        temp['positive_words'] = temp['positive_words'].apply(lambda x: ",".join(list(filter(None, x))))
        words_df = temp.groupby(level='rating')['positive_words'].apply(','.join).reset_index()
        temp = group_df[group_df['negative_words'].str.len() != 0]
        temp['negative_words'] = temp['negative_words'].apply(lambda x: ",".join(list(filter(None, x))))
        words_df["negative_words"] = temp.groupby('rating')['negative_words'].apply(','.join).reset_index()[
            "negative_words"]
        words_df1 = words_df.assign(negative_words=words_df.negative_words.str.split(","))
        words_df1 = words_df1.assign(positive_words=words_df1.positive_words.str.split(","))
        words_df1.set_index("rating", inplace=True)
        positive_df = pd.DataFrame(words_df1["positive_words"].dropna()).loc[:, "positive_words"].apply(
            lambda x: pd.Index(x).value_counts().nlargest(10))
        negative_df = pd.DataFrame(words_df1["negative_words"].dropna()).loc[:, "negative_words"].apply(
            lambda x: pd.Index(x).value_counts().nlargest(10))
        positive_df.fillna(0, inplace=True)
        negative_df.fillna(0, inplace=True)
        positive_df = positive_df.T
        negative_df = negative_df.T
        positive_df = positive_df.loc[:, positive_df.columns.isin([4, 5])]
        positive_df = pd.DataFrame(positive_df.sum(axis=1))
        positive_df.reset_index(inplace=True)
        positive_df.rename(columns={0: "value", "index": "reviews"}, inplace=True)
        positive_df = positive_df[positive_df["value"] > 5].sort_values(by='value', ascending=False)
        positive_df = positive_df[positive_df["reviews"] != ""]
        positive_df.reset_index(drop=True, inplace=True)
        negative_df = negative_df.loc[:, negative_df.columns.isin([1, 2, 3])]
        negative_df = pd.DataFrame(negative_df.sum(axis=1))
        negative_df.reset_index(inplace=True)
        negative_df.rename(columns={0: "value", "index": "reviews"}, inplace=True)
        negative_df = negative_df[negative_df["value"] > 5].sort_values(by='value', ascending=False)
        negative_df = negative_df[negative_df["reviews"] != ""]
        negative_df.reset_index(drop=True, inplace=True)
        if not positive_df.empty and positive_df.shape[0] > 3 and not negative_df.empty and negative_df.shape[0] > 0:
            print("\n", hospital_name, "\n")
            subplot_fig.add_trace(
                go.Pie(labels=positive_df.reviews, values=positive_df.value, textinfo='label+value+percent',
                       insidetextorientation='radial'), row=1, col=1)
            subplot_fig.add_trace(
                go.Pie(labels=negative_df.reviews, values=negative_df.value, textinfo='label+value+percent',
                       insidetextorientation='radial'), row=1, col=2)
            # subplot_fig.show()
            subplot_fig.write_image(f"{hospital_name}.png")
        elif not positive_df.empty and positive_df.shape[0] > 3:
            fig = go.Figure(
                data=[go.Pie(labels=positive_df.reviews, values=positive_df.value, textinfo='label+value+percent',
                             insidetextorientation='radial', title=f"{hospital_name} -> Positive reviews")])
            # fig.show()
            fig.write_image(f"{hospital_name}.png")
        elif not negative_df.empty and negative_df.shape[0] > 3:
            fig = go.Figure(
                data=[go.Pie(labels=negative_df.reviews, values=negative_df.value, textinfo='label+value+percent',
                             insidetextorientation='radial', title=f"{hospital_name} -> Negative reviews")])
            # fig.show()
            fig.write_image(f"{hospital_name}.png")
    except:
        pass 

Sample visualization report




Ready-made code

Code for this project available in the below link
 Finally, we created our own Review-driven recommendation application. Still, there is a lot of improvement needed in this application. Make this project as a reference and update it as per your taste. Happy coding!

Comments

Post a Comment

Popular posts from this blog

Artificial Intelligent Chatbot

-
Image
Presenting by Madhan Kumar Selvaraj Globally 30% of the customer service is handling by the Chatbot. Its market size is projected to grow from $2.6 billion in 2019 to $9.4 billion by 2024. Many of the people have a thought that machines will grab their job. But the reality is that the machine assist humans to do the complicated woks in a easy manner. What this blog is about? This blog helps you to create an Artificial intelligent Chatbot by deploying it in the Python Flask web framework and the Heroku cloud platform. Here I am doing this chatbot to assist a coaching institute to respond to the students regarding the course details and also it can answer questions apart from the course. Everyone knows about Google assistant, Apple's Siri, Microsoft's Cortona, Amazon's Alexa. Our Chatbot also works like the same and it is having limitation because we are using limited pattern and response. If we add more pattern and response, it can response to all our queries. Chatbot A c
Read more

Whatsapp chat book

-
Image
 Presenting by Madhan Kumar Selvaraj More than 2 billion people are using the Whats-app application as per the report . It replaced all of our chatting applications and most of our formal, as well as informal chats, are happening through the Whats-app. Those chats are between our family, friends, spouse, our loved ones. I got an idea of printing the chat in the form of a book or hard copy kind of thing to make our bond as a lifetime memory. Real-world issue One of the easiest ways to convert the chat into PDF format is by taking the long screenshot of our Whats-app chat in the application and converting it into PDF. The problem is that Whats-app deletes messages frequently in the application and we can't able to change the font type, size, color. Quality of taking the long screenshot is not that efficient to convert into the printable format and most of the applications are proprietary. Some of the websites like Zapptales are providing options to convert the exported Whats-app ch
Read more

Detecting stranger through CCTV camera and alerting the owner

-
Image
Presenting by Madhan Kumar Selvaraj In this blog, we are going to see some interesting stuff related to detection. Globally there are around 25 million CCTV cameras in the house. Sadly, for every 3 minutes robbery, theft and burglary are happening. Real-world issue Even though we have access to live streaming CCTV video footage through the internet. But most of the time we failed to check the video footage at the time of the robbery. So, there is a need for a smart alert system. One-liner of the project Detecting the human subject other than the family members through the CCTV footage and alerting the house owner through the mail. Technologies used here OpenCV computer vision Facenet CNN model MTCNN (Multi-task Cascaded Convolutional Neural Networks) Support Vector Machine algorithm Matplotlib library The workflow of the project Focusing only on the human subject in the CCTV footage other than animals, birds, etc Take a screenshot of the video once we
Read more