Implementation of an Ms Excel Tool for Backpropagation Neural Network Algorithm in Environmental Eng

2.1. Neural Network Algorithm

Fig. 1 shows general structure of a multilayer neural network in which l layers exists. The first and the last layers (1st and lth layers) are input and output layers. Each layer has a number of neurons ni (n1, n2, ... nl–1, nl, respectively for each layer). These neurons are connected to each other by weights wi(j, k). Besides, each neuron (except those in the input layer) has a bias bi(j).

Figure 1. General structure of an artificial neural network A backpropagation neural network starts with selecting a set of training examples from the data set which consists of n1 independent variables (x) and nl dependent variables (y). Note that the numbers of independent and dependent variables are equal to the numbers of input and output neurons, respectively. The number of data points in the data set is m. The set of training examples is a fraction of the data set, which is usually taken as 0.60 to 0.80. The BPNN algorithm requires forward– and backward– runs for each training example during the learning process. The forward run starts with setting the activation for the input layer neurons a1 to the values of the independent variables. From this point forward, this set of values is not updated during the learning process. In the forward run, the next step is to calculate the input sums (z2) and activations (a2) of all neurons in the second layer (l = 2). The calculation process is performed for each layer until the activations of the neurons in the output layer is finally computed. Following equations are used for this purpose:

z i  w i a i 1  b i i = 2, 3, ... l a i  f z i i = 2, 3, ... l

where f is a vectorizing function, which, when applied to every element j of a vector v, produces f(v)j = f(vj). The activations of the neurons in the output layer are used to compute the cost function C using the following quadratic function: l

The cost function is the main criteria in the learning process. It reflects total discrepancy between values of the dependent variable (y, also called target values) and the predicted values which is al. The output error δl is calculated as follows: 253

S. Demir, N. Manav Demir, A. Karadeniz, H. Civelek Yörüklü / Sigma J Eng & Nat Sci 36 (1), 251-260, 2018

where  a C is the derivative of the cost function with respect to the activations of the neurons in the output layer,

f  is derivative of the activation function with respect to its input,

and  is the Hadamard product of two vectors. The output error is propagated back to the second layer of the network using the following formula:

In the backward run, the network calculates the error for each layer and updates the biases and weights using the gradient of the cost function:

where η is the learning rate. In the backward run, all of the weights and biases in the network are updated based on the error calculated between the target value and the predicted value for each dependent run. The complete set of forward and backward runs for each example in the training set is called an epoch. The network is trained for a predefined number of epochs until the learning process ends.

2.2. MS Excel

Visual Basic for Applications (VBA) is an event-driven programming language and is a derivative of Microsoft Visual Basic 6 (VB6). VBA is embedded in Microsoft Office software like Excel and is compiled to packed code (P-code). The P-code is then stored as a separate stream in an Excel file, independent of the spreadsheet stream. This property of VBA simplifies the problem of compiling the program into an executable file and of preparing a distribution package, if necessary, usually associated with stand-alone computer programming platforms, like VB6. Currently, VBA 7.1 is available through later versions Microsoft Office, however, there are still a great number of Microsoft Office 2010 users. Therefore, the codes and spreadhseets described in this paper is limited to MS Excel 2010 along with VBA 7.0. An Excel VBA program was implemented that employs the backpropagation algorithm given in previous section. A screenshot of the Excel VBA program is shown in Fig. 2a. The software allows the user select the number of input and output neurons, the number of hidden layers, the activation function of hidden layers and output layer, the steepness coefficients for the activation functions, percent of training data, learning rate, and the number of epochs. Thirteen activation functions were embedded in the software along with their derivatives, namely, linear, rectified linear, logistic, symmetric logistic, arctangent, hyperbolic tangent, Gaussian, symmetric Gaussian, softplus, Elliot, symmetric Elliot, sinusoid, and sinc functions [14]. The software was prepared in 254

modular structure, in which the user can go through the code to add new activation functions (Fig. 2b).

(a) (b) Figure 2. Screenshots of the Excel VBA program. a. sheet view, b. code view

2.3. Test Case: Darcy Friction Factor

Fast and accurate calculation of Darcy friction factor is of importance for environmental engineers when dealing with large piping systems such as water distribution networks since it takes long times to complete calculations because of iterative scheme of solution. Colebrook [15] proposed an implicit formulation as an accurate alternative to well-known Moody’s diagram for calculating Darcy friction factor. With the development of computers, a great number of papers have been dedicated to formulating explicit correlations of friction factor for computerized applications including [16-20], each of which provides accuracy and calculation speed to certain degrees. In this study, 375 data points of Darcy friction factor by implicit Colebrook equation were used to test the performance of the implemented BPNN tool. The relative roughness (RR) and Reynolds numbers (RE) were between 1*10-6 to 0.05, and 4*103 to 1*108, respectively. Since friction factor is traditionally expressed as logarithmic functions of RR and RE, natural logarithms of RR and RE were used as the inputs to estimate the friction factors (f).

2.4. Test Case: Oxygen Solubility

Solubility of oxygen in water (C) that is in contact with air at a given atmospheric pressure and temperature governs the rate at which water gains oxygen. Therefore, it determines the capacity of a neutral water body to aerobically neutralize organic pollutants and is an important parameter for environmental engineers to assess water quality. For example, dissolved oxygen concentration needs to be kept at a sufficient level in order to properly operate of aerobic processes [21]. On the other hand, under anaerobic conditions, an increase in dissolved oxygen concentration negatively affects the efficiency of the treatment [22]. 255

S. Demir, N. Manav Demir, A. Karadeniz, H. Civelek Yörüklü / Sigma J Eng & Nat Sci 36 (1), 251-260, 2018

Solubility of oxygen is mainly a function of water temperature (T) and salinity (S) at a given atmospheric pressure. Although salinity is of minor concern for fresh water bodies compared to temperature, in some cases it becomes the major concern considering the fact that wastewater streams eventually find their way to seas. In this study, solubilities of oxygen in fresh and salt water was used to test performance of the implemented BPNN tool. For this purpose, oxygen solubilities at various water temperatures (between 0 and 45°C by 1°C) and various salinities (0.0, 9.0, 18.1, 27.1, 36.2 and 45.2 ppt) were used. A total of 276 data points were employed to train the BPNN tool.

3.1. Darcy Friction Factor

Darcy friction factors were calculated using the implicit Colebrook equation for relative roughness 1*10-6 ≤ RR ≤ 0.05, and Reynolds number 4*103 ≤ RE ≤ 1*108. A total of 375 data points were used. Natural logarithms of RR and RE-1 (ln[RR] and ln[RE-1]) were used as inputs and f values by Colebrook equation were used as target values. Randomly selected 70% of data were used for training. The learning rate was 0.75 and the number of epochs was 5000. Ten neurons were employed in the single hidden layer with logistic function as the activation function. Since the BPNN tool randomly selects the training set, the tool was run for 25 times to ensure the convergence to the global minimum of the cost function. The highest and the lowest values of relative mean square errors (RMSE) from 25 runs were calculated as 1.05*10-3 and 2.98*10-3, respectively, with an average of 1.53*10-3±0.59*10-3. The calculated coefficients of determination were between 0.9977 and 0.9988 with an average value of 0.9983±0.0003. Fig. 3.a shows correlation plot between Colebrook- and BPNN-friction factors for best BPNN performance. The slope and intercept of a regression line between calculated values of friction factor were predicted as 1.0035 and 0.0005 showing that the BPNN results agree well with the Colebrook values. Another VBA code was also written to calculate friction factors by BPNN with the trained-weights and the calculated values of friction factors are shown in Fig. 3.b. The speed of calculation of friction factor is another concern when dealing with large pipe systems. Darcy friction factor by BPNN was calculated 106 times to test the speed of BPNN prediction method. The speed of BPNN prediction was satisfactory (calculations were completed in around 15 seconds on a computer Intel Core i5-2500K CPU on 3.3GHz and 16 GB RAM at 2133 MHz).

y0,08 = 1,0035x + 0,0005 R² = 0,9988 0,07 0,06 0,05 0,04 0,03 0,02 0,01 0 0

Figure 3. BPNN results. a. correlation plot between calculated friction factors by Colebrook equation and by BPNN, b. Calculated friction factors by BPNN as a function of RR and RE.

3.2. Oxygen Solubility

Solubility of oxygen at water temperatures (T) between 0 and 45°C and salinities (S) between 0 and 45.1 ppt were used for training BPNN tool. The neural network consisted of one input layer with two neurons (T and S), one hidden layers with ten neurons, and one output layer with single neuron (C). Sinc function was used as the activation function. The number of epochs was set to 5000 and the learning rate was 0.75. Randomly selected 70% of 276 data points were used for training. The tool was run for 25 times to ensure the convergence to the global minimum of the cost function. The highest and the lowest values of relative mean square error (RMSE) between target and predicted values of oxygen solubility were calculated as 1.52*10-4 and 9.06*10-4, respectively. Average RMSE from 25 runs was 4.78*10-4±2.33*10-4. Average coefficient of determination from 25 runs was calculated as 0.9998±0.0000. Experimental and predicted values of oxygen solubility are shown in Fig. 4. Fig. 4.a is a correlation plot between experimental and predicted values. The slope and intercept of the regression line were calculated as 0.9951 and 0.0680, respectively, which, along with satisfactorily low RMSE values, indicates that BPNN-predicted values agree well with the experimental values. Solubilities of oxygen at salinities other than 257

S. Demir, N. Manav Demir, A. Karadeniz, H. Civelek Yörüklü / Sigma J Eng & Nat Sci 36 (1), 251-260, 2018

those employed in training set were calculated using the trained BPNN and is shown as a function of temperature and salinity in Fig. 4.b. Smooth curves of oxygen solubilities supports the predicted values.

Figure 4. BPNN results. a. correlation plot between experimental and BPNN solubilities of oxygen, b. Calculated solubilities of oxygen by BPNN as a function of temperature and salinity.