Exploring multilingual reviews for aspect-based sentiment analysis using Lexicon and BERT

Abstract

Keywords: BERT; Lexicon; SentiWordNet; Sentiments; Social Media

Introduction

The information revolution is one of the most prominent features of the recent era. Nowadays, the world has become a global village. The social network has emerged as an essential part of everyone’s life. Social media has gained popularity after the massive eruption of internet use in various fields of life. Social media is vital in guiding people about specific tendencies, such as individuals’ opinions on social, political, religious, and economic fields [1]. There are various platforms for social media, such as Twitter, Facebook etc. These platforms give insight into how individuals perceive things and the information available [2].

These are also helpful in accumulating and organizing data for particular organizations related to people’s decisions. Organizations gather data on how their workers and customers think of their services or products. The constant development of analytical tools provides more opportunities to analyze this data for decision-making. For this reason, different methods of analysis have been introduced. One of the most used is sentiment analysis (SA). Sentiment analysis analyzes emotions, sentiments, and opinions from text data using natural language processing tools [3, 4]. Sentiments represent people’s viewpoints, such as negative, positive, or maybe neutral. SA is divided into three different types based on sentences, documents, and

*Corresponding author. *E-mail address: rizwanrana315@gmail.com This paper was recommended for publication in revised form by Editor-in-Chief Ahmet Selim Dalkilic Published by Yıldız Technical University Press, İstanbul, Turkey Copyright 2021, Yıldız Technical University. This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

words (aspects) [5]. Document-level sentiment analysis scrutinizes the whole document and then decides whether the document is positive or negative [6]. Sentence-level sentiment analysis involves analyzing individual sentences and determining whether each sentence expresses a positive or negative sentiment [7]. In contrast, aspect-based sentiment analysis (ABSA) is a more recent type that aims to identify sentiment polarities with a specific focus on particular aspects [8]. Aspects or features are concepts upon which an individual communicates his thoughts or ideas. For example, consider a comment like “my phone’s camera is outstanding, although its battery drains too fast, but the speed and performance of the phone is amazing.” In this review, the user has expressed his opinion about different aspects of his mobile, i.e., battery, speed, performance, and camera. We can infer that opinions about cameras, speed, and performance are positive, while opinion about the battery is negative. Aspect-based sentiment analysis (ABSA) offers several advantages over traditional sentiment analysis techniques by providing a more fine-grained and nuanced understanding of sentiment within specific aspects. For example, A major online retailer wants to analyze customer reviews for electronic products. Traditional sentiment analysis may classify a review as positive or negative without capturing sentiments related to specific aspects like battery life, display quality, or customer service. ABSA enables the retailer to extract aspect-level sentiment, providing insights into which aspects are driving customer satisfaction or dissatisfaction. This sentiment analysis is also helpful for manufacturers to understand the public’s perception of specific product features and make data-driven decisions for future iterations. Another example from healthcare, a pharmaceutical company wants to monitor sentiments regarding their new medicine in online forums. Traditional sentiment analysis may provide an overall sentiment, but ABSA allows the company to focus on specific aspects like effectiveness, side effects, or ease of use. It helps them understand the strengths and weaknesses of the medicine in detail. Extensive research efforts have been devoted to ABSA, which remains a difficult and challenging research area because of the complexity of opinions and diversity of aspects [9]. In ABSA, it is tough to accurately identify the sentiment of the aspect because sometimes multiple aspects may be present in the review with various sentiments. Various existing techniques only split the sentiment words into two groups (positive and negative) without considering the polarity strength [5]. Existing studies of ABSA are mainly focused on reviews and tweets in English. Some work has been done on the Turkish [10], Arabic [11], and Spanish [12] languages, but their techniques are language oriented. These techniques are only developed for specific languages. Analysis of these multilingual reviews and tweets is one of the significant tasks of sentiment analysis, as it explores the sentiments and opinions of people of different

cultures and localities [13]. Moreover, the efficacy of extant polarity calculation techniques must be enhanced [14]. The following are the main contributions of our paper: • Multilingual conversing was introduced to convert other languages’ reviews into English. • A novel approach, utilizing a lexicon-based method in conjunction with BERT, has been implemented to determine the polarity of sentences or reviews. • The improved accuracy of the proposed technique is compared to the currently used methods. The rest of the paper is structured as follows: The analysis of the most cutting-edge methodologies for conducting sentiment analysis is presented in Section 2. Section 3 demonstrates the proposed ABSA model involving data pre-processing, aspect extraction, aspect engineering, and polarity calculation. Section 4 showcases the experimental outcomes and comparisons with existing approaches. Conclusions and future work are discussed in the last section. Background Studies Over the past decade, extensive research has been conducted in sentiment analysis, with various techniques employed in classification algorithms for diverse applications. The following section provides an overview of several different methods of ABSA. ABSA consists of two tasks, Aspect-based feature extraction and classification of sentiments. Banjar et al. presented a technique for determining the sentiment analysis from a user’s text data in the form of tweets [13]. Part of Speech tagger with Aspect Co-Occurrence equipped with semantic similarity is employed for aspect extraction and refinement. A novel match pattern approach is applied for sentiment polarity calculation. Janjua et al. introduce a hybrid approach for ABSA using neural networks instead of traditional machine learning (ML) algorithms [14]. Their approach consists of rule mining to identify aspects and a neural network to classify data. Another study shows the creation of a search engine that can analyze the opinions expressed by people in their tweets and reviews [15]. Using ABSA, Tran et al. proposed a unique approach that helps hotel personnel receive more accurate and exact information about their clients and the services they provide [16]. The authors built a modified version of the BiLSTM-CRF model to extract features with their respective polarity. In the end, LDA modeling is used to structure the topics. The sentiment analysis approach suggested by Nawaz et al. [17] aims to enhance the quality of goods by analyzing public feedback. POS tagger using Visuwords that extracts and reduces aspects. Recommendations and non-recommendation are assigned to reviews using Relational Classifiers. Mowlaei et al. proposed improvements in ABSA’s two lexicon methods: Adaptive Lexicon learning using a Genetic Algorithm and Aspect Based Frequency Based Sentiment Analysis[18].

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

These approaches use statistical methods, lexicon and Genetic algorithms. Devlin [19], BERT is a stack of multilayered transformer encoders and learns the bidirectional representation. BERT utilizes the pre-trained models for text labeling and is fine-tuned for any specific task. The results demonstrate that a language model trained in both directions may have a greater understanding of language context and flow than models which are trained as single-direction language models. There are evolving tasks related to sentiment prediction. Pak et al. [20] use Twitter for sentiment analysis, then perform linguistic analysis of the collected corpus. A multinomial Naive Bayes is built on the collected corpus sentiment classifier. This classifier outperforms the SVM classifier. However, the model is only composing one language. This model can also be used to compare different language corpus. Xu [21] achieved 78% accuracy on BERT-PT, classifying the sentiment as positive, negative, and neutral. Gao [22] presented the three versions of BERT, which outperformed sentiment classification and achieved an accuracy of 76%. Da Li et al. [23] use the lexicons with neural networks for Chinese social media analysis of sentiments. Their method improves the sentiment polarity on Weibo. Wu [24] presented the Knowledge-aware Dependency Graph Network (KDGN), which combines domain knowledge, dependency labels, and syntax path into the dependency graph. The experimental outcomes on benchmark datasets reveal that KDGN surpasses prior baseline approaches for ABSA, emphasizing the significance of domain knowledge and dependency labels. Another study presented a model for text sentiment analysis known as KSCB (K-means++, SMOTE, CNN, and Bi-LSTM) [25]. It tackles the issues of unbalanced class distribution and unlabeled datasets faced by current machine-learning approaches. The results of the experiments demonstrate that KSCB outperforms five other baseline methods in the classification of text sentiment. Transfer learning and pre-trained models have revolutionized sentiment analysis by providing a powerful foundation of language understanding and enabling more

accurate, efficient, and adaptable sentiment analysis models. They have reduced the need for extensive training data, improved performance, and facilitated domain adaptation. Qian [26] introduced an interactive capsule network, which expands the segmented approach. This network incorporates two BERT models as encoders to process contexts individually and then uses interactive attention and a dynamic routing algorithm to learn the connection between them. Another research employed BERT’s transfer learning capability to enhance sentiment analysis decision-making via a CNN-BiLSTM model [27]. The results showed that the proposed method obtained superior binary classification performance in sentiment analysis, outperforming all other embedding methods and algorithms. Table 1 illustrates the strengths and weaknesses of lexicons-based, machine-learning and deep-learning models for ABSA. Considering state-of-the-art techniques of SA for reviews and tweets, it is observed that existing techniques do not optimize the aspects. Some techniques use explicit aspects, while some use implicit aspects. Traditional sentiment analysis techniques are focused on single-language reviews. Sentiment analysis in a single language raises the possibility of missing important information in texts written in other languages. In addition, these approaches are also domain-oriented. Therefore, building a system that identifies the sentiments from data in multiple languages and calculates the polarity of sentiments with better accuracy is desirable. Proposed Model This research uses hybrid computational algorithms of lexicons and BERT to build an integrated model to preprocess online reviews, extract features, classify, and visualize the sentiment analysis results. The proposed model comprises data preprocessing, aspect extraction, aspect engineering, and polarity calculation. Different stages of the proposed model are described in the following subsections. Figure 1 illustrates the proposed model. Data Collection Three different datasets of multiple domains are used. The first is a benchmark dataset of Multilingual reviews of

Table 1. Strengths and weakness of ABSA approaches ABSA approaches

Less expensive No need for training data, primarily if companies use a dictionary-based approach

Requires extensive language resources, which are not always accessible. There are no dictionaries for many languages.

Time-intensive Require a substantial amount of labeled training data.

Automatically learn meaningful representations of textual data Accurate

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

various products extracted from Amazon [28]. These products are books, toys, cameras and watches, etc. It contains English, German, Japanese, Spanish, Chinese and French reviews. The Secord dataset is from a domain movie containing only English-language reviews [29]. The Last dataset contains tweets about news and events in different languages, collected using Twitter API [30]. Table 2 shows the breakdown of datasets.

Multilingual Conversion One of the most attractive discoveries of man is language. No matter if it is spoken or written, it is our intellect that provides a mode of communication. It is the only language used to express our ideas, emotions, and feelings. In the age of information technology, people can interact with each other using many methods, such as social media, etc., resulting in a huge account of resources that reshape the world of sentiment analysis. Existing literature shows that

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

Sie funktionieren auf jeden Fall! Sie sind auch unglaublich süß und gut verpackt.

They definitely work! They’re also incredibly cute and well-packaged.

Acheté principalement pour visionner des videos ou des séries. Malheureusement l’écran reflète tout, c’est un vrai miroir. Extremement gênant.

Purchased primarily for viewing videos or series. Unfortunately the screen reflects everything, it is a real mirror. Extremely annoying.

Me encanta, hace todo lo que quiero, no puedo creer lo genial que es

I love it it does everything I want can’t believe how great it is.

The product arrived promptly as stated by Amazon. Drawn Together is very funny. Not for the faint hearted. If you like your humour crude and full frontal this is the season dvd for you. I loved it, very funny.

The product arrived promptly as stated by Amazon. Drawn Together is very funny. Not for the faint hearted. If you like your humour crude and full frontal this is the season dvd for you. I loved it, very funny.

multilingual analysis is one of the most expensive tasks. Due to the availability of sophisticated tools for conversion to English by machine learning algorithms, its cost reduces. Attempt to apply machine translation in various NLP tasks have not been widely used due to the poor quality of translated texts; however, recent improvements in machine translation have inspired such attempts. The most modern and intelligent technique, “Google Language Translator” [31], is used for multilingual analysis. Google Language Translator supports more than 100 languages from Afrikaans to Zulu. This model is pre-trained. Another important aspect of Google translator is to identify the language accurately. Google Rest API is used for this purpose. The conversion of some multilingual reviews into the English language is shown in Table 3. Data Pre-processing Due to the inherent noise in review data, it was necessary to conduct a data cleaning process before conducting any subsequent analysis. The primary goal of this step is to transform the raw data into a format that is amenable to further operations, thereby enabling the derivation of meaningful results from it [32]. Here is some of the importance of text preprocessing for sentiment analysis: • Improves Accuracy: Text preprocessing helps to remove irrelevant and redundant information from the text, which can improve the accuracy of SA results. By removing stop words, punctuation, and other non-essential elements, the machine learning model can focus on the most important features of the text. • Reduces Noise: Text data often contains noise, which misspellings, slang, or grammatical errors can cause. Preprocessing techniques such as spell-checking and removing special characters can help reduce noise and improve data quality. • Normalizes Data: Text preprocessing can also help to normalize data by converting all text to lowercase and

removing accents. This can ensure consistency in the data and avoid errors caused by uppercase and lowercase letters. • Increases Efficiency: Text preprocessing can help reduce the dataset’s size by removing irrelevant and redundant information. It can improve the efficiency and reduce the time required for analysis. The Porter stemming method removes frequent morphological ends from words [33]. For example, playing is changed to play. Stopwords like ‘is’, ‘am’ etc. are commonly used words in a text which do not carry important meanings. Usually, stopwords are Determiners, Coordinating conjunctions and Prepositions. The term-based Random Sampling (TBRS) method is used to remove stopwords [34]. Further, all the punctuation and symbols in the text are removed and at the end, the text is converted into lowercase. These preprocessing steps enhance the performance of sentiment analysis. Aspect Extraction Aspect extraction involves identifying and extracting specific aspects or features from a review. The selection of relevant aspects and their representation is crucial. The quality of aspect extraction can significantly affect the performance of SA models. Poorly defined or inaccurate aspects can lead to biased or unreliable sentiment predictions. Additionally, the complexity of aspect extraction can influence the model’s performance, as accurately identifying aspects from unstructured text can be challenging. The Part of Speech tagger finds aspects from pre-processed reviews in our proposed model. Part of speech is the classes of words with the same grammatical properties. Grammars of natural language have many classes of words. The parts of speech are verbs, nouns, adjectives, etc. The Part of speech tagger takes the text and assigns parts of speech classes (nouns, adjectives, etc) to each word. In

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

reviews, aspects are usually nouns and noun phrases are present in reviews [35]. Further, nouns are divided into different classes, like proper and improper nouns. Proper nouns do not majorly contribute to sentiment analysis, so only improper nouns as aspects are considered. Three taggers: Stanford Loglinear Part-Of-Speech Tagger, CTs Part of Speech Tagger, and Morph Adorner Part of Speech Tagger [36] are used for filtering aspects from reviews. Aspect Engineering Bidirectional Encoder Representations from Transformers (BERT) is a deep learning technique based on transformer that was developed by Google [37]. Two BERT models are available: BERT (base) and BERT (large). BERT (base) comprises twelve bidirectional self-attention heads with twelve encoders, while BERT (large) comprises sixteen bidirectional self-attention heads with twenty-four encoders. Both models have undergone pre-training utilizing an extensive dataset comprising 2.5 billion words from the English Wikipedia and 800 million words extracted from the books corpus. The input and output elements are interlinked in the BERT model, and the weights between these elements are calculated based on their connectivity. In the proposed technique, BERT (base) is employed, which consists of twelve bidirectional self-attention heads and twelve encoders. During training, the weights are adjusted using a loss function to build an optimal BERT model that minimizes the loss. The outputs of the model in forward propagation are the probabilities of possible labels, which are compared with target labels. The loss function computes a penalty for every discrepancy between the model’s predictions and the desired target label. Polarity Calculation Polarity is calculated using the lexicon SentiWordNet 3.0 [38]. SentiWordNet is a lexical resource that contains weightings for neutral, positive and negative sentiments attached to various aspects. SentiWordNet is a

lexical resource for sentiment analysis that is publicly available. [39]. SentiWordNet 3.0 is the upgraded version of SentiWordNet. There are many words in the English language whose orientation differs in sentences. The same words are used as nouns, adjectives, and verbs in the same sentence. For example, the word “Risk” is used as a noun as well as a verb. Part of speech tells whether the word is a noun, adjective, verb etc. Table 4 shows some examples of SentiWordNet tags with scores. The polarity of review/tweets is calculated using: Sum of the polarity of all the aspects present in the review divided by the total number of aspects present in the review. It is modeled in equation 1, where ‘w’ are aspects and N represents the total number of aspects.

Results And Discussion

The proposed technique is evaluated using accuracy, precision, and recall. In the case of sentiment analysis, accuracy was defined as “the percentage of total reviews that correctly predict the sentiment class over the total reviews which are present in the dataset”. Precision is defined as the ratio of the total number of correct outcomes to the overall number of outcomes returned and recall refers to the ratio of the total number of correct outcomes to the number of outcomes that should have been returned. The obtained dataset undergoes pre-processing to remove unwanted information like HTML tags, smileys, etc. Following that is the “Multilingual Conversion” module, which verifies and converts all reviews and tweets into a format that is understandable by our algorithm. Aspects are extracted from reviews and tweets. The aspects and opinion words are then subjected to BERT for aspect engineering.

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

Table 5. Experimental results in term of precision, recall and accuracy #

Finally, the Lexicon “SentiWordNet” is utilized to calculate polarity. BERT only accepts fixed-length sentences, so these sequences are stored as reviews. The graph (Fig. 2) shows the dataset’s sentence length from maximum to minimum after pre-processing. 160 is the maximum length of tokens which consider for further experiments. The results of the proposed technique are tabulated in Table 5. The proposed technique achieved 91.07% precision, 89.24% recall and 90.72% accuracy on the product review dataset. It achieved 88.05% precision, 87.24% recall, 89.45% accuracy on movie reviews datasets and achieved 89.56% precision, 88.89% recall, 90.25% accuracy on the Twitter dataset. The listed results show our technique yielded improved accuracy, precision and recall. To examine the impact of multilingual conversion, a comparison is presented of the proposed technique executed on multiple domain datasets with multilingual conversion and without multilingual conversion. Four comparisons in terms of precision (%), recall (%), accuracy (%) and F-measure (%) are shown in Figure 3. Comparisons show results are improved on products and tweet datasets with multilingual conversion up to 16% to 20%. Movie review results are the same because they

contain all reviews in English. Also, comparisons show the importance of multilingual reviews, as these reviews also express sentiments and opinions about any product, movie, etc. The conversion of these multilingual reviews into English makes it possible for our proposed technique to process these reviews, extract sentiments, and calculate the polarity. Table 6 reports the effect of removing HTML tags, stop words and the importance of filtering the words during the pre-processing step; as discussed in section 3.3, this was absent from many previous studies. Another comparison was presented to compare the proposed technique (Lexicon+BERT) with three previous works: the model using RNN+Capsule [40], BERT+TD [41], GGCN+BERT [42], BERT+extended context [43], SenticNet + GCN [44]. A comparison in terms of precision, recall and accuracy of these techniques with the proposed technique is shown in Figure 3. The improved performance of the proposed aspect extraction, optimization, and polarity calculation suggests the high relevance of aspects and multilanguage analysis. Another observation from the ablation study is the importance of BERT. It improves the performance of the proposed ABSA technique.

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

Figure 3. (a) Comparison with and without multilingual in terms of Precision (b) Comparison in terms of recall (c) Comparison in terms of accuracy (d) Comparison in terms of F-measure.

Sigma J Eng Nat Sci, Vol. 42, No. 5, pp. 1469−1479, October, 2024

Figure 4. Comparison of the proposed technique with baseline approaches.

Data Availability Statement

A novel technique has been proposed for sentiment polarity estimation using BERT, part of speech taggers, and SentiWordNet. Our proposed technique is equipped with “Multilingual Conversion” which incorporates improvements to its ability to convert multilanguage reviews and tweets into English. The proposed technique is applied to each word of the opinionated review/tweets of multiple domain datasets to classify the reviews. The results experimentally projected the progress of the work in terms of recall, precision and accuracy. Future efforts will need to combine existing research with the most recent advancements in deep learning techniques to enable more accurate complex aspect extraction and sentiment analysis. By leveraging the power of deep learning, it is possible to overcome the complexities inherent to sentiment analysis tasks and obtain highly refined and nuanced results. Thus, a concerted effort towards integrating existing research with cutting-edge deep learning techniques would be a crucial step in advancing sentiment analysis. The main challenge in multilingual sentiment analysis is the lack of lexical resources. Future research should create a comprehensive multilingual corpus covering various languages to overcome this. This multilingual corpus would serve as a valuable resource for sentiment analysis tasks across different languages, enabling researchers to overcome the current limitations and enhance the accuracy and effectiveness of multilingual sentiment analysis algorithms and models.

The authors confirm that the data that supports the findings of this study are available within the article. Raw data that support the finding of this study are available from the corresponding author, upon reasonable request.

Conflict Of Interest

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics

There are no ethical issues with the publication of this manuscript.

Share and Cite