Classification of Turkish tweets by document vectors and investigation of the effects of parameter c
* Author to whom correspondence should be addressed.
Sigma Journal of Engineering and Natural Sciences 2020, Vol. 38, Issue 3, pp. 1581-1592; doi.org/10.62051/ytu.sigma-journal-of-engineering-and-natural-sciences-classification-of-turkish-tweets-by-document-vectors-and-investigation-of-the-ef
Abstract
Keywords: Text classification; natural language processing; document vectors; doc2vec; sentiment analysis; deep learning.
1. Introduction
Natural language processing is to collect information regarding the comprehension of human language and development of the ability to use it, so that it can be used to understand and process tasks, and to develop tools and techniques to use in this manner [1]. Natural language processing has developed as a branch of artificial intelligence and has become increasingly popular with the development of social media and web technologies. It contributes to the solution of many language-related problems, such as automated translation systems, question-and-answer systems, summarizing a topic, semantic role labeling, sentiment analysis, correction of spelling errors, and text classification. Text classification (TC), is the assignment of texts formed in the natural language to predetermined groups by an expert. TC studies have started with the works of Maron in the 1960s [2]. However, since the area has become more and more popular since the 1990s, the development of technology and the subsequent access to strong sources of hardware, it has become a prominent sub-discipline of information systems [3]. TC is being used in many areas *
Corresponding Author: e-mail: metinbilgin@uludag.edu.tr, tel: (224) 275 52 63
such as the automatic classification of text-based documents, filtering of unwanted messages, developing results of search engines, determination of ideas, or author recognition [4]. Sentiment analysis (SA) is the ability to express sentiments, thoughts, ideas, and experiences regarding any topic on social media such as Facebook, Google+, Twitter, etc. SA is the determination of the sentiment reflected in the texts [5]. As an example, we can give positive or negative comments regarding a product, or the determination of the mood of users, and political ideas of the society [6]. Sentiment analysis studies categorize the content of every part of message content in the database into two (e.g., positive, negative) or more (e.g., very good, good, satisfactory, bad, very bad) categories; thus, SA can be considered a type of TC that in every message, the dominant sentiment represents a category [7]. In Zhang et al.'s work, positive-negative-tagged sentiment analysis was conducted on 100,000 Chinese interpretations of a product obtained from the trial using various classification and learning methods. In the study of two separate experiments by Lexicon and part of speech, the combination of Word2Vec and SVM perf was 89.95% on lexicon basis and 90.30% on part speech [8]. Dickinson et al. emotion analysis was conducted using n-gram and Word2Vec methods to investigate how a firm's stock price change affected the sentiment of the tweets about the product. As a result of the study, 65% success was achieved for the n-gram method and 75% success was achieved in the word2vec method [9]. Tang et al. conducted another emotion analysis on the twitters. Unlike the others in this study, it was tried to show how various word embedding methods changed the success of the system [10]. Polpinij et al. conducted another empirical analysis of hotel reviews. In this study where the Word2Vec method is used, SVM is used as a learning algorithm [11]. Şahin, on the Turkish texts, Word2Vec, and SVM based classification process were done. It was investigated how the processing of root or attached words affected the system performance in the study in which 22,729 Turkish documents were used [12]. Xue et al. performed emotion analysis using Word2Vec on Sina Weibo, a Chinese microblogging site. The other side of the work is a model for building an emotion dictionary using Word2Vec. Later, it was aimed to determine the sentiment tendency of documents using this dictionary. As a result of the study, success was obtained as 0.94 for positive documents, 0.96 for negative documents, and 0.85 for neutral documents [13]. Bilgin and Şentürk, on the other hand, have implemented semantic supervised sentiment analysis on Twitter for both English and Turkish. Two versions of the Doc2Vec algorithm, DBoW, and DB, were used in this study. As a result, DBoW showed higher performance than DM [14]. For the sentiment analysis they have made, Bilgin and Köktaş have used word vectors on a three-class data set. They have tested the performance of the system using two different machine learning methods, based on a Word2Vec approach [15]. Bilgin and Köktaş have carried out a study on sentiment analysis and have used 3000 Turkish tweets. They had used Term Weighted and Word Vectors as methods. Word Vectors were shown to be better than Term Weight in this study [16]. As well as these studies, lots of studies were carried out on Turkish Tweets [17-20]. This study is on Doc2Vec methods that are improved after Word2Vec. Doc2Vec is a novel approach which little study in it. One of the novel aspects of this study that the parameter changes are investigated on the accuracy metric. Also, 2 different methods of Doc2Vec are detailed examined, and present the pros and cons of the methods. To use of Turkish tweets created a reference point for studies on Turkish. Another novel aspect is one of the first studies to use Doc2Vec for Turkish.
2.1. Doc2Vec
The method of word vectors was suggested to represent words as vectors in an n-dimensional space, and thus calculate the distances between words to determine the semantic similarities [21]. Word2Vec is an approach that uses a neural-network approach to embed words. This model is trained with a large set of texts and forms a unique vector for every word. The attribute of these unique vectors is that words with similar meanings form similar vectors. It has two methods: CBoW (Continuous Bag of Words) and Skip-gram. CBoW uses the context around a word to guess the word, whereas Skip-gram tries to guess the words by encircling the words that have a fixed window size. Skip-Gram can produce better results for sparsely used words [22]. CBoW and Skip-Gram methods are illustrated in Figure 2.
Figure 1. Word2Vec (CBoW and Skip-gram) [14] Word2Vec has a version that is developed to embed the word sequence, called Paragraf2Vec in some sources. The purpose of this model is to use the word vectors model to represent larger blocks of text (sentence, paragraph, or document) as vectors [23]. The model that uses unsupervised learning has two different methods named DBoW and DM. DBoW uses the same method as Skip-gram, whereas DM uses the same method as CBoW. Having more parameters and consequently being more complex, DM can produce better results compared to DBoW. DBoW is simpler compared to DM and instead of the sequence of words, it is based on the sequence of words in a paragraph. Different from Skip-gram, DBoW replaces the document vector that represents the input. Differently from CBoW, DM uses the document vector as well as contextual words to guess the target word [24]. DM and DBoW methods are demonstrated in Figure 3. Each of DBoW and DM architectures uses one of HS (hierarchical softmax) or NS training algorithms for learning. While HS training algorithm generally produces good results in lowfrequency words, NS produces better results in high- frequency words. The parameters used for the Doc2Vec algorithm are given in Table 1.
Figure 2. Doc2Vec (DM and DBoW) [14] Table 1. Parameters and Tasks for Doc2Vec Parameter -size window -hs -negative -alpha dm
Task Length of vector Number of the neighbor node Learning Algorithm (hs=1 HS, hs=0 NS) Number of Negative samples Learning Rate (alpha=0.025) Method (dm=1 DM, dm=0 DBoW)
A software was developed using the Gensim library in the Python programming language for the work to be done. The software we developed is designed to be supervised learning and able to train the system. Training and testing steps have been carried out by using DBoW and DM algorithms. Also, the accuracy value for the test set was calculated with the software, and confusion matrices were created.
2.2. Dataset
A three-class dataset, created by Amasyalı et al., was used in the study [25]. Randomly 10.000 tweets were selected for train and 2.500 tweets for the test from approximately 12.500 tweets. The information concerning the dataset is given in Table 2. Table 2. Dataset’s Properties Train Set Test Set
3. Results And Discussion
In this study, different methods of the Doc2Vec algorithm were used for the classification of tweets we had. The optimum success of the system was tried to be measured by two different methods, including DM and DBoW, by making changes on parameters, and the effects of
parameter changes on success were investigated. The results are given in Table 3-4, and the graphs of the results in Figure 3-10. Table 3. The Results for accuracy metric (-size =100) DBoW -negative 1 2 3 4 5 DM -negative 1 2 3 4 5
42.48. 41.68
The graph of the accuracy values obtained when DBoW and DM methods used the parameters of size=100, negative=1-5, window=3- 7 for HS ve NS architectures is given in Figure 3-6.
Table 4. The Results for (-size =500) DBoW -negative 1 2 3 4 5 DM -negative 1 2 3 4 5
39.48. 40.04
Since HS learning algorithm was applied for the DBoW method; increasing and decreasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused oscillation in the accuracy value.
Figure 4. The Results for DBoW-NS (-size=100) Since NS learning algorithm was applied for the DBoW method; increasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused little oscillation in the accuracy value.
Since HS learning algorithm was applied for DM method; increasing and decreasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused changeable oscillation in the accuracy value.
Figure 6. The Results for DM-NS (-size=100) Since NS learning algorithm was applied for DM method; increasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused oscillation increasingly in the accuracy value.
Since HS learning algorithm was applied for the DBoW method; similarly accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused changeable oscillation in the accuracy value.
Figure 8. The Results for DBoW-NS (-size=500) Since NS learning algorithm was applied for the DBoW method; increasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused little changeable oscillation in the accuracy value.
Since HS learning algorithm was applied for the DM method; decreasing accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused changeable oscillation in the accuracy value.
Figure 10. The Results for DM-NS (-size=500) Since NS learning algorithm was applied for the DM method; likely accuracy values were obtained depending on the number of negative samples and on the change in the window. The change in the window by the same number of negative samples caused oscillation in the accuracy value. In the study in which vector length variation was investigated; DBoW reacted negatively to the increase in the size, and accuracy value decreased. Although NS architecture managed to produce better values when the window parameter was increased along with the size parameter, these values were not better than those obtained with size=100. Although similar results were obtained in DBoW with HS and NS, with size = 100 and negative = 1; NS architecture produced better results as long as the negative value increased and even DBoW reached the highest accuracy value of 58.04 with the parameters of size=100, negative=5, window=4. In DM, both HS and NS reacted similarly to the size increase and produced similar results. The oscillation slightly increased in both DM-HS and DM-NS when size=500 in comparison to when size=100. When particularly an evaluation, independent of size, is made, it can be said that the accuracy rate of DM-NS was found 40.2, and this rate didn’t oscillate much from the changes in both negative and window parameters, and it produced the same results. Considering the findings of the study, it can be said that; DBoW achieved higher accuracy values than DM. DBoW-NS reacted more negatively to the increase in size than DBoW-HS. As a result of the study, it was seen that the highest accuracy value for DBoW was obtained both when size=100, window=3-4-5, negative=3-4-5, and when NS architecture was used. DM gave similar reactions to the changes in size, window, and negative parameters and produced similar accuracy values without oscillating so much. The highest accuracy value of 42.48 was obtained with size=100, window=3,negative=1. The reason why better results were obtained with DBoW method than DM method is that the calculation cost of DBoW is less in small spaces and in parallel to that, higher accuracy values
are expected to be obtained. As space, where processing was done, grew, process complexity increased, and accordingly, the accuracy values of DBoW approximated to DM. The same situation is valid for HS and NS architectures. NS produced better results than HS as the size parameter increased. As the size parameter increased, the values obtained by the DBoW method for NS were less than those obtained for HS. In the DM method, a little change occurred in the parameter-dependent accuracy values.
4. Conclusions
In this study, text classification was performed through the data obtained from twitter using two different methods of the Doc2Vec algorithm. A train was given using supervised learning infrastructure and then tests were conducted for a 3-class dataset by using the Gensim library in Python programming language. SVM algorithm used for the classification. Different learning architectures and different parameter values were used to find the most appropriate parameter range for the system. The success of the system was tried to be increased with these changes. As a result of the study, it was seen that better results were obtained by DBoW method than DM method. Higher accuracy values were obtained by NS architecture than HS. The accuracy of the optimized parameter values obtained in this study will be tested in future studies to be performed on the data sets of different sizes.
Share and Cite
BİLGİN, M. Classification of Turkish tweets by document vectors and investigation of the effects of parameter c. Sigma Journal of Engineering and Natural Sciences 2020, Vol. 38, pp. 1581-1592. https://doi.org/10.62051/ytu.sigma-journal-of-engineering-and-natural-sciences-classification-of-turkish-tweets-by-document-vectors-and-investigation-of-the-ef