Credit Scoring By Artificial Neural Networks Based Cross-Entropy and Fuzzy Relations

Abstract

Keywords: Credit scoring; artificial neural networks; cross-entropy; fuzzy relations; gradient based algorithms.

1. Introduction

The value of revolving credit outstanding in the whole world economy has grown rapidly depending on many economic factors. Today, the financial institutions still continue to diversify the credit products to reach more customers in the context of the economic perspective. Apparently, this intention brings about various challenges such as evaluating the risk levels and credit scores of customers, assessing the credit amounts, the follow-up on payments, the legal processes, etc. Apart from the credit scoring and risk evaluation, most of workloads are directly interested in the management issues that can be accomplished via the corporate governance structure thoroughly. However, the financial institutions make a special effort for the systematic risk related to the credit scoring. Essentially, the credit scoring is an efficient method to measure the systematic risk when financing the individual customers as well as the small and mediumsized enterprises (SMEs). For this reason, the researchers and experts focus on improving more efficient procedures for the credit scoring process in terms of estimating more robust models using the statistical, operations research, artificial intelligence (AI), machine learning and hybrid approaches in the automated and consistent manner. Generally, the credit scoring problem deals

Corresponding Author: e-mail: ozan.kocadagli@msgsu.edu.tr, tel: (212) 246 00 11 /5511

with two categories: “application scoring” and “behavioral scoring”. The former is interested in classifying the credit applicants into various risk groups and making a decision about whether any credit application is worthy to approve or not, but the latter deals with making an inference using the payment histories of customers [1]. This inference includes the critical strategies about credit limits of costume rs, payment difficulties, bankruptcy, etc. Besides, these inferential models also predict future purchasing behavior or credit status of customers [2]. Generally, it´s not easy to determine which variables should be handled to evaluate the credit risks since this kind of information is rarely shared by financial institutes. However, the modelling structure can be learned from our experience of financial markets, and more implicitly from the Fair, Isaac and Company (FICO) method, VantageScore, Ficth, Moody’s and Standart & Poor’s Ratings. Today, the FICO model or its different versions are still used by the majority of financial institutions. Even so, the financial institutes still continue to develop their credit scoring procedures depending on the number of customers, economic factors, industrial field being used the credit, etc., because the developed models should meet all the demands in this dynamical and stochastic environment. Today’s technology allows the supercomputers to make more data processing due to more powerful processor and RAM capacities. Hence, this development prompts the researchers to use more efficient decision support systems those will be able to cope with big data sets as well as model complexity. In the literature, the researchers mostly have focused on hybridizing artificial neural networks (ANNs), support vector machines (SVM), regression trees, expert systems with meta-heuristic algorithms, Bayesian and fuzzy theories. Specifically, the deep learning which can be considered as an extension of ANNs where there are much larger layer sets and various combination activation functions is very popular research field nowadays. Even so, in the context of credit scoring literature, there are different approaches such as Discriminant Analysis [3,4], logistic regression analysis [5], Decision Tree [6,7,8,9], linear and nonlinear programming, decision support systems, ANNs, SVMs [5,10,11,12, 13, 14,15,16,17], Bayesian Networks [17,18, 19], Expert Systems and hybrid approaches [3,5,20,21,22,23,24,25], etc. Recently the ensemble methods are also applied to credit scoring procedure [5], [7], [15], [16], [21], [26], [27], [28], [29], [30], [31], [32], [33]. Generally, these procedures present different ensemble methods such as AdaBoost, Bagging, Random Subspace, Stacking, Decorate and Rotation Forest with the following base classifiers: 1-nearest neighbor, LogR, multilayer perceptron, radial basis function, gamma, clustering-launched classification, SVM and C4.5 decision tree. This study introduces to a novel procedure that allows the experts to estimate the efficient credit scoring models. Specifically, this procedure hybridizes the fuzzy relations with ANNs based cross-entropy and information criteria. While the fuzzy relation is used for reducing the dimension of feature matrix, ANNs present a natural and flexible way to model the highdimensional and excessive non-linear systems without any restrictive assumption [34,35,36,37]. Thus, they are capable of give superior performance to the classical approaches. Also, they can be hybridized with other artificial intelligent techniques practically [36]. Therefore, ANNs are widely used in credit scoring problems because of their flexible structures [1,7,22,29,39,40,41,42,43,44,45,46,47].

2. Motivation And Overview

According to the literature, the hybrid approaches provide superior performance against the classical techniques. However, they bring out many shortcomings such as the control of tuning parameters, model complexity, time-consuming, processor and memory allocation, etc. In the context of ANNs, the researches suffer from the selection of risk function, model complexity, early-stopping, cross-validation, time-consuming [34,35]. Generally, they intend to minimize Mean Squared Error (MSE) at the classification problems such as the credit scoring. However,

this approach is not inherently convenient for classification problem, because the output vector of ANNs consists of categorical variables. Specifically, the case of binary number {0, 1} denotes whether any credit application is worthy to approve or not [1,48]. To get rid of this inconvenience, various classifiers can be used such as cross-entropy, Kullback Leibler divergence, Bayesian classifiers and information criteria instead of MSE [29,38,40,43,49,50]. In this study, as a classifier in ANNs, the cross-entropy measure is preferred, because it is more convenient to represent a binary structure such as whether any credit application is worthy to approve or not. To estimate a robust model by ANNs, another issue is complexity which is closely related to the over/lower fitting to training data [51,52]. For this reason, reducing the dimension of feature matrix plays important role to control the model complexity as well as determining the number of neurons in the hidden layer(s). However, the model complexity is mostly overlooked by researchers due to time consumption. In this study, to reduce the dimension of feature matrix, the fuzzy relations are proposed against traditional feature selection methodologies. This approach allows using various metrics to create a relation matrix among features. Thus, the equivalence classes over the fuzzy relations can be defined by means of various fuzzy composition operators where these classes consist of certain feature(s) with same characteristics. In this perspective, each class can be considered as a component or main factor as similar to the principal component analysis and factor analysis [38]. To determine the efficient number of neurons in the hidden layer, information criteria is a practical approach; otherwise the trial and error apparently causes waste of time [40,53,54,55,56]. For this reason, in the proposed procedure, the number of neuron is determined by Akaike Information Criterion (AIC), Corrected AIC (AICc) and Bayesian Information Criterion (BIC). In addition, to control the model complexity, early stopping approach based on the cross-validation is used. In the context of training ANNs, the best practical way is to use gradient based algorithms such as Quasi-Newton known as Broyden, Fletcher, Goldfarb, and Shanno (BFGS), LevenbergMarquardt (L-M), Scaled Conjugate Gradient (SCG), Gradient Descent (GD) and GD with momentum (GDwM). Particularly, BFGS utilizes the approximated Hessian matrix and update it each iteration instead its original one. This configuration allows solving the sophisticated unconstrained problems. L-M algorithm runs a non-negative damping parameter where its larger values make the algorithm closer to Gauss-Newton Method whereas its smaller values make it closer the GD. By means of efficient damping parameters, L-M is capable of very fast searching in the high dimensional parameter cases and non-linear least square problems. By using a step size scaling mechanism SCG avoids time consuming for line-search per learning iteration, thus it makes the algorithm much faster. GD minimizes the error function by performing a search using the learning rate parameter in the direction of the gradient vector through the solution space. However, updating the learning rate at each iteration is not practical and efficient approach for the high dimensional systems like ANNs. In addition, this algorithm mostly stuck in the local optima due to the use of gradient vector. To overcome this situation, GDwM can be preferred. Unlike GD, this algorithm utilizes a momentum constant together with the learning rate at each iteration; thus it gains a feature that is not trapped to the local optima [35,58,60]. In the light of the above, examining the pros and cons of gradient based algorithms plays important role to estimate the models from ANNs in the most accurate and best manner. For this reason, training ANNs via various gradient based algorithms provides an efficiency for the estimation procedure. To summarize, the main purpose of this paper is to introduce an efficient approach that allows the decision makers to estimate the robust classification models for the credit scoring using ANNs based cross entropy and information criteria as well as fuzzy relations. Thus, the decision makers will be able to make more accurate decision about the credit approvals of the individual customers. Also, this procedure can be easily applied to another credit data set as well. To present the proposed approach, this paper is constructed as following. Section 3 introduces the fuzzy

relation methodology in the context of reducing the dimension of feature matrix. Section 4 deals with the architecture of ANNs based cross-entropy and information criteria. Section 5 includes the credit scoring implementations where two benchmark data sets are handled. Section 6 is allocated for the results of analysis. Finally, the analysis results and outputs are discussed and interpreted in detail in the Section 7.

3. Reduction The Dimensionality Of Features Using Fuzzy Relations

3.1.1. Fuzzy Cartesian Products and Relations Let 𝐴̃1 , 𝐴̃2 , … , 𝐴̃𝑝 be fuzzy sets in universes, 𝑋1 , 𝑋2 , … , 𝑋𝑝 , respectively. Thus, Cartesian product 𝐴̃1 × 𝐴̃2 × … × 𝐴̃𝑝 denotes a fuzzy relation in 𝑋1 × 𝑋2 × … × 𝑋𝑝 . The membership function of this fuzzy relation can be characterized as 𝜇𝐴̃1× 𝐴̃2×…×𝐴̃𝑝 (𝑥1 , 𝑥2 , … , 𝑥𝑝 ) = 𝜇𝐴̃1 (𝑥1 ) ∗ 𝜇𝐴̃2 (𝑥2 ) ∗ … ∗ 𝜇𝐴̃𝑝 (𝑥𝑝 )

where “ * “ is one of t–norm operators. In practice, t-norm operator is selected as the fuzzy intersection “min”. Further information related to various t-norm operators can be found in [60],[62]. For a two-dimensional space (p = 2), a fuzzy relation R on 𝑋 × 𝑋 is a fuzzy set on the Cartesian product 𝑋 × 𝑋 where 𝑋 = [𝑥1 , 𝑥2 , … , 𝑥𝑚 ] corresponds to a sample data set with n samples and m features. Here, each data sample (or observation) can be showed as a row vector with 𝑥𝑖 = {𝑥𝑖1 , 𝑥𝑖2 , … , 𝑥𝑖𝑚 } i = 1, 2, …, n (2) Each data sample is an m-dimensional vector that has mostly different units, so each feature vector should be normalized or standardized to a unified scale before the classification procedure [35]. Mathematically, a relation matrix 𝑅̃ on 𝑋 × 𝑋 can be constituted by various methods such as cosine amplitude, max-min, exponential similarity coefficient, correlation coefficient, scalar product, and nonparametric approaches [59,60,61,62,63,64]: 𝑟𝑖𝑗 = 𝑅̃(𝑥𝑖 , 𝑦𝑗 ) i, j = 1, 2, …., n (3) In this study, to assign membership values in 𝑅̃, a new approach based Hausdorff metric is proposed as well as cosine amplitude, max-min and correlation coefficient. For a correlation coefficient approach as similar to “Correlation Coefficient” in statistics, a fuzzy relation from 𝑋 to 𝑋 is evaluated by 𝑟𝑖𝑗 =

where 𝑥̅𝑖 and 𝑥̅𝑗 are averages of features 𝑖𝑡ℎ and 𝑗𝑡ℎ , 𝑥𝑖𝑘 and 𝑥𝑗𝑘 are 𝑘 𝑡ℎ observations of features 𝑖𝑡ℎ and 𝑗𝑡ℎ , respectively. By means of cosine-amplitude, 𝑅̃ can be defined as 𝑟𝑖𝑗 =

In terms of being alternative to the other metrics, 𝑅̃ can be constituted by Hausdorff distance (dH ) ij between features xi and xj [65]. From the characteristic of Hausdorff metric, the distances d i,j and dj,i between xi and xj might not be equal each other. Therefore, the relative distance is to evaluate as the average of d i,j and dj,i, dH ij = di,j+d ji/2. Let d min and dmax be minimum and maximum distances among observations of xi and xj ,respectively; then the membership values of 𝑅̃ can be evaluated as follow [38]: rij = 1 −

3.1.2. Composition In order to get the certain number of classes on the fuzzy relations, the composition operations are required. Similar to the classical relations, the composition operations can be applied to the fuzzy relations too. For instance, let 𝑋 × 𝑌 and 𝑌 × 𝑍 be the fuzzy relations on 𝑅̃ and 𝑆̃, respectively. Regarding to 𝑅̃ and 𝑆̃, any 𝑇̃ = 𝑅̃ ∘ 𝑆̃ are constituted by means of certain composition operations on 𝑋 × 𝑍. To do this, two well-known composition operations, max – min and max – product, can be used. The other composition operators can be found in [59,60, 61,62,63]. Generally, the fuzzy max – min and max-product compositions are defined by set-theoretic and membership notations as follows: 𝜇 𝑇̃ (𝑥, 𝑧) = ⋁𝑦 𝜖 𝑌(𝜇𝑅̃ (𝑥, 𝑦) ⋀ 𝜇𝑆̃ (𝑦, 𝑧)) or 𝜇 𝑇̃ (𝑥, 𝑧) = max − 𝑚𝑖𝑛𝑦 ∈ 𝑌 ( 𝜇𝑅̃ (𝑥, 𝑦) , 𝜇𝑆̃ (𝑦, 𝑧))

𝜇 𝑇̃ (𝑥, 𝑧) = ⋁𝑦 𝜖 𝑌(𝜇𝑅̃ (𝑥, 𝑦) ∗ 𝜇𝑆̃ (𝑦, 𝑧)) or 𝜇 𝑇̃ (𝑥, 𝑧) = 𝑚𝑎𝑥𝑦 ∈ 𝑌 (𝜇𝑅̃ (𝑥, 𝑦) ∗ 𝜇𝑆̃ (𝑦, 𝑧))

From the above equations, it is noted that if we let 𝑆̃ = 𝑅̃, then 𝑇̃ = 𝑅̃ °𝑅̃ will be defined on X×X [38]. In this study, to get more significant classes during the reduction of feature matrix, both composition operations are utilized. 3.1.3 Fuzzy Tolerance and Equivalence Relations Let 𝑅̃ be a fuzzy relation defined on a single universe X. In this case, there are some important algebraic properties: Reflexivity, Symmetry and Transitivity. These properties can be showed as matrix relations as follows [38]: Reflexivity: For ∀ 𝑥𝑖 ∈ 𝑋 , (𝑥𝑖 , 𝑥𝑖 ) 𝜖 𝑅̃ (10) Symmetry:

or 𝜇𝑅̃ (𝑥𝑖 , 𝑥𝑖 ) = 1 For ∀(𝑥𝑖 , 𝑥𝑗 ) 𝜖 𝑅̃ and (𝑥𝑗 , 𝑥𝑖 ) 𝜖 𝑅̃

For ∀(𝑥𝑖 , 𝑥𝑗 ), (𝑥𝑗 , 𝑥𝑘 ) 𝜖 𝑅̃ , 𝑅̃ (𝑥𝑖 , 𝑥𝑗 ) ⋀ 𝑅̃(𝑥𝑗 , 𝑥𝑘 ) ≤ 𝑅̃ (𝑥𝑖 , 𝑥𝑘 )

where 𝜆 ≥ min[𝜆1 , 𝜆2 ] . It can be shown that any fuzzy tolerance relation, 𝑅̃𝑡 (also called a proximity relation) on a universe 𝑋 is a relation that exhibits only the properties of reflexivity and symmetry. 𝑅̃𝑡 , can be reformed into an equivalence relation 𝑅̃𝑒 (or similarity) by at most (m-1) compositions with itself, where m is the size of square matrix 𝑅̃𝑡 . That is; 𝑅̃𝑡𝑚−1 = 𝑅̃𝑡 °𝑅̃𝑡 ° … °𝑅̃𝑡 = 𝑅̃𝑒 (13)

Unlike the fuzzy tolerance relation, an equivalence relation has also transitivity properties as well as reflexivity and symmetry. Besides, it can be proven that a fuzzy equivalence relation 𝑅̃𝑒 , has equivalence classes on 𝑅̃. For any given 𝜆-cut level (𝜆 𝜖 [0,1]) and 𝑟𝑖𝑗 = 𝑅̃(𝑥𝑖 , 𝑥𝑗 ) ≥ 𝜆 (i, j = 1, 2, …, m), the equivalence classes of 𝑥𝑖 𝜖 𝑋 is determined as 𝑅̃ [𝑥𝑖 ]𝜆 = {𝑥𝑖 |𝑅̃[𝑥𝑖 ] ≥ 𝜆} = {𝑥𝑖 |𝑥𝑖 𝑅̃𝜆 𝑥𝑗 } (14) where 𝑥𝑖 𝑅̃𝜆 𝑥𝑗 expresses that 𝑥𝑗 (j = 1, 2, …,m) drops into the same class with 𝑥𝑖 with respect to 𝑅̃𝜆 . Thus, for any given 𝜆 𝜖 [0,1], the equivalence classes of 𝑅̃ give a partition of X called as λpartition of X. Consequently, let 𝑅̃𝜆 = {(𝑥𝑖 , 𝑥𝑗 )|𝜇𝑅̃ (𝑥𝑖 , 𝑥𝑗 ) ≥ 𝜆} define the λ-cut relation of 𝑅̃ , then the columns (features) situated in the same class will be determined with respect to the 𝜆-cut level. That is, if 𝜇𝑅̃ (𝑥𝑖 , 𝑥𝑗 ) ≥ 𝜆, then the value on the 𝑖𝑡ℎ row and 𝑗𝑡ℎ column of 𝑅̃𝑒 is replaced with 1; otherwise 0. After this assignment called defuzzification, the new matrix consists of only two numbers: 0 and 1. Thus, if all of the values on different columns of defuzzified matrix are equal to each other, they are assigned into same class.

Measure

In this study, to estimate the credit scoring model, the feed-forward ANNs with three layers are used. As seen in Fig.1, the first layer receives the feature vectors as inputs. The hidden layer includes the p number of neurons. The last layer gives output yc having totally k number of classes yc = [o1 , o2 , … , ok] where c shows the indices of classes (c = 1,2, … , k). Generally, output yc belongs to any one of k classes: yc ϵ[{1,0,0, , … ,0}, {0,1,0, … ,0}, {0,0,1, … ,0}, … , {0,0,0, … ,1}] (15) Class 1

Thus, any component oc is evaluated as follow: oc = f(W Ι , W ΙΙ , x) =

For a specific case, k = 2, yc consists of only 0’s or 1’s (binary values). Essentially, Eq. (16) is the soft-max function (known as a logistic function for k = 2) where W I : A matrix consists of all the weights among inputs and neurons in the hidden layer. W II : A matrix consists of all the weight values among the neurons in hidden and output layers. x: A vector consists of the values of all the features in any state. bI : A vector consists of all the bias values for the activation functions in the hidden layer, bΙ = [b1I bI2 ... bIs ]. Here, as an activation function, the tangent hyperbolic is used. bII : A vector consists of all the bias values in the soft-max function in the output layer, bΙI = [b1II II bII 2 ... bk ] [38].

Figure 1. The structure of a feed-forward ANN According to the above definitions, 𝑊 𝐼 is structured as follow W Ι = [w1I w2I … wiI …

where 𝑤𝑖 is a vector, defined as wiI = [wi,1 wi,2 … wi,r ] (i = 1, 2, ..., s). Apparently, 𝑤𝑖 includes all the weights between the neuron 𝑖 𝑡ℎ and all the inputs. Similarly, 𝑊 𝛪𝐼 is defined as follow: W ΙI = [w1II w2II … wcII … wkII ]′

Eq. (16), A(w I x + bI ): Rs → Rs represents a vector function that includes the outputs of all the activation functions as follow [1]. A(W Ι x + bΙ ) = G (A1 (w1I x + b1I ), A2 (w2I x + bI2 ), … , As (wsI x + bIs ))

In this framework, Aj (j=1,2,…, s) is defined as the 𝑗𝑡ℎ tangent hyperbolic function in the hidden layer: e

In order to assign the output yc = [o1 , o2 , … , ok ] to only one of k classes, the components of 𝑦𝑐 are transformed into the binary numbers {0, 1} as follow: 1, if oc = max [o1 , o2 , … , ok ], χoc = { (23) 0, Otherwise. From Eq. (16), the soft-max function of any component can be considered as the posterior probability of a certain class given any component 𝑜𝑐 :

where Ct is one of the predetermined classes. Hence, maximizing the posterior probability in Eq. (24) is equivalent to minimizing the following cross-entropy measure: k E(W Ι , W ΙΙ ) = − ∑N i=1 ∑t=1 yi,c (t)logP(Ct /oc )

6. Results And Discussion

In analysis, to control the model complexity in addition to diminishing memory and time consumption, the feature matrix was reduced by the fuzzy relation procedure. To improve the classification performance of ANNs, they were trained by various gradient based algorithms considering two risk functions: cross-entropy and MSE. In the context of generalization and estimating more robust models, the whole data set was portioned into three subsets considering cross-validation methodology: training, validation and test. Training of ANNs was stopped automatically as soon as the error level over validation data grows. Thus, this approach allows estimating more accurate models for ANNs. In analysis, the efficient number of neurons was determined by information criteria that automatically penalize the excessive complex models. According to analysis results, the gradient based algorithms showed superior performances to each other with respect to different configurations. Especially, ANNs with cross-entropy provides better performances than ANNs with MSE. Besides, the procedure based the fuzzy relation have exposed significant components by reducing the dimension of original feature matrix. Thus, this approach helps to control the complexity of model at the excessive number of inputs. As seen in the above tables, it can be seen that all the algorithms are able to achieve plausible accuracy ratios for training data; however, their efficiencies reduce a little bit for test data. Actually, this reduction is acceptable, because test data can be considered as new individual credit applications, and it is not introduced to ANNs before. In addition, the training procedure utilized validation data set for estimating more general models generalization. From the analysis results, the best configurations of training algorithms are given in Table 9. Table 9. Comparison of the best configurations for all of benchmark data sets Fuzzy Training Test Benchmark Cross Algorithm Metric/ Neuron data data Data Entropy Features

7. Conclusions

The proposed approach provides substantial advantages in terms of feature selection with fuzzy relations, training ANNs and controlling complexity. Firstly, the fuzzy relations require less memory and time consumption by reducing the dimension of dataset as well as controlling the model complexity. In addition, the fuzzy relations allow the researchers to use different metrics for establishing the similarity between features. Secondly, the information criteria help to determine the efficient number of neurons in the hidden layer. Thus, ANNs based cross-entropy 866

and MSE bring out more reliable and robust models in terms of classification accuracy and model complexity. In addition, the proposed procedure can be easily adapted to various training approaches according to the structure of activation functions and risk functions. As a result, the proposed procedure provides the best model configuration for ANNs in terms of reliability and complexity. In addition, this procedure can be easily applied to another credit data set whose predetermined classes consist of different cases. In spite of superior advantages, the proposed approach requires an expert knowledge, because it composes of many steps such as preparing data matrix, extracting to features, reducing the dimension of feature matrix, training ANNs, selecting the best model configurations, interpreting and discussing the results. Therefore, an interdisciplinary collaboration is inevitable to construct this kind of expert system thoroughly. In this context, the software of proposed procedure can help users to control this complicated system easily, and reduce time consumption related to adjusting some parameters or functional structures. To validate the robustness of proposed approach, more comprehensive data sets should be handled, but this a challenge due to some legal restrictions. In addition, many benchmark data sets have not enough to explanatory features related to customer information. Hence, these challenges prompt us to make collaboration with experts and financial institutions in terms of making deeply analysis with more comprehensive credit data sets. As a future direction, we are planning to make collaboration with the financial institutes to reach more comprehensive credit data sets. Also, the studies related to modifying the proposed procedure will be continued in terms for developing more efficient algorithms and estimating more robust models via novel hybrid artificial intelligence techniques. In addition, a user-friendly interface will be designed for developed procedures by MATLAB.

Share and Cite