Expert Systems With Applic Ations: Jonathan L. Ticknor
Expert Systems With Applic Ations: Jonathan L. Ticknor
of Pages 6, Model 5G
25 April 2013
5
6
9
10
a r t i c l e i n f o a b s t r a c t
1
2 2
0
13
14 Keywords: In this paper a Bayesian regularized artificial neural network is proposed as a novel method to forecast 21
15 Bayesian regularization financial market behavior. Daily market prices and financial technical indicators are utilized as inputs 22
16 Neural network to predict the one day future closing price of individual stocks. The prediction of stock price movement 23
17 Stock prediction is generally considered to be a challenging and important task for financial time series analysis. The accu- 24
18 Overfitting
19 rate prediction of stock price movements could play an important role in helping investors improve stock 25
returns. The complexity in predicting these trends lies in the inherent noise and volatility in daily stock 26
price movement. The Bayesian regularized network assigns a probabilistic nature to the network weights, 27
allowing the network to automatically and optimally penalize excessively complex models. The proposed 28
technique reduces the potential for overfitting and overtraining, improving the prediction quality and 29
generalization of the network. Experiments were performed with Microsoft Corp. and Goldman Sachs 30
Group Inc. stock to determine the effectiveness of the model. The results indicate that the proposed 31
model performs as well as the more advanced models without the need for preprocessing of data, season- 32
ality testing, or cycle analysis. 33
Ó 2013 Published by Elsevier Ltd. 34
35
36
37 1. Introduction parametric and nonparametric models for predicting stock market 59
trends and returns. The empirical results suggested that the prob- 60
38 Stock price prediction has recently garnered significant interest abilistic neural network (classification model) outperforms the 61
39 among investors and professional analysts. The prediction of stock standard feedforward neural network (level estimation model) in 62
40 market trends is very complex, owing to the inherent noisy envi- predicting stock trends. 63
41 ronment and high volatility related to daily market trends. The One drawback of using standard back propagation networks is 64
42 complexity of the market and stock price movement is related to the potential for overfitting the training data set, which results in 65
43 a number of factors including political events, market news, quar- reduced accuracy on unknown test sets. To address the potential 66
44 terly earnings reports, and conflicting trading behavior. Traders of- overfitting of neural network weights, some researchers have 67
45 ten rely on technical indicators based on stock data that can be developed hybridized neural network techniques (Chu, Chen, 68
46 collected on a daily basis. Despite the availability of these indica- Cheng, & Huang, 2009; Leigh, Purvis, & Ragusa, 2002; Oh & Kim, 69
47 tors, it is often difficult to predict daily to weekly trends in the 2002). Hassan, Nath, and Kirley (2007) utilized a hybrid model that 70
48 market. included Hidden Markov Model (HMM), Artificial Neural Networks 71
49 In the past two decades, a large body of research for predicting (ANN), and genetic algorithm (GA) to predict the price of three ma- 72
50 stock market returns has been developed. This body of knowledge jor stocks. The results of their study indicated that the next day 73
51 contains many artificial intelligence (AI) approaches, namely artifi- stock price could be predicted to within a 2% of the actual value, 74
52 cial neural networks (ANN). A number of these networks have uti- a significant improvement over other standard models. 75
53 lized feedforward neural networks to predict stock trends (Baba & In recent years, a number of techniques including fuzzy logic, 76
54 Kozaki, 1992; Chenoweth & Obradovic, 1996; Fernandez- genetic algorithms, and biologically based algorithms have been 77
55 Rodriguez, Gonzalez-Martel, & Sosvilla-Rivebo, 2000; Ghiassi, utilized to improve feedforward neural networks (Armano, 78
56 Saidane, & Zimbra, 2005; Hamzacebi, Akay, & Kutay, 2009; Marchesi, & Murru, 2004; Atsalakis & Valvanis, 2009; Blanco, Del- 79
57 Lendasse, De Bodt, Wertz, & Verleysen, 2000; Roh, 2007; Walezak, gado, & Pegalajar, 2001; Chang & Liu, 2008; Chang, Wang, & Zhou, 80
58 1999). Leung, Daouk, and Chen (2000) analyzed a number of 2012; Chen, Yanga, & Abraham, 2006; Ritanjali & Panda, 2007; Yu- 81
dong & Lenan, 2009). Neural networks and support vector ma- 82
chines were compared by Kara, Boyacioglu, and Baykan (2011) to 83
⇑ Tel.: +1 919 660 5030. predict stock price index movement, the results of which showed 84
E-mail address: [email protected] the superiority of neural networks. These new techniques have im- 85
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013
ESWA 8515 No. of Pages 6, Model 5G
25 April 2013
86 proved the capability of the original back propagation neural net- Hagan (1997) assumed that the noise in the data was Gaussian, 135
87 work with some success and reduced some of the drawbacks of and with this assumption were able to determine the probability 136
88 ANNs related to noise fitting. density function for the weights. The optimization of the regulariza- 137
89 In this paper the feedforward artificial neural network coupled tion parameters a and b require solving the Hessian matrix of F(w) 138
90 with Bayesian regularization is introduced as a novel approach to at the minimum point wMP. Forsee and Hagan (1997) proposed a 139
91 predicting stock market trends. The networks ability to predict Gauss–Newton approximation to the Hessian matrix which is pos- 140
92 stock trends is shown for two major United States stocks and is sible if the Levenburg–Marquardt training algorithm is used to lo- 141
93 compared to another advanced hybrid technique. The simulations cate the minimum. This technique reduces the potential for 142
94 indicate that the weight optimization technique offers an arriving at local minima, thus increasing the generalizability of 143
95 enhanced level of performance compared to other advanced tech- the network. 144
96 niques. A brief discussion of Bayesian regularization is included to The novelty of this technique is the probabilistic nature of the 145
97 provide a brief background on the equations governing the network weights in relation to the given data set and model frame- 146
98 technique and its potential applications in financial time series work. As a neural network grows in size through additional hidden 147
99 forecasting. layer neurons, the potential for overfitting increases dramatically 148
and the need for a validation set to determine a stopping point is 149
100 2. Bayesian regularization of neural networks crucial. In Bayesian regularized networks, overly complex models 150
are penalized as unnecessary linkage weights are effectively driven 151
101 The back propagation neural network is a very popular to zero. The network will calculate and train on the nontrivial 152
102 technique in the field of ANN which relies on supervised learning, weights, also known as the effective number of parameters, which 153
103 typically through the use of a gradient descent method to reduce a will converge to a constant as the network grows (Burden & 154
104 chosen error function (e.g. mean squared error). Fig. 1 provides a Winkler, 2008). The inherent noise and volatility of stock markets 155
105 general architecture for a feedforward neural network, including introduces a high probability of overfitting and overtraining for 156
106 the three main layers of the model: input, hidden, and output. general back propagation networks. These more parsimonious net- 157
107 The connection between neurons in Fig. 1 in each layer is termed works reduce the chance of overfitting while eliminating the need 158
108 a link. This link is stored as a weighted value which provides a for a validation step, thus increasing the available data for training. 159
Table 1
ANN parameters for stocks in initial experiment.
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013
ESWA 8515 No. of Pages 6, Model 5G
25 April 2013
191 the output data to obtain the stock value in the appropriate units in this study. A general search of the existing literature helped de- 225
192 before comparison to actual data. duce the major indicators to be used in this study (Yudong & Lenan, 226
2009; Kara et al., 2011; Kim & Han, 2000). The indicators used in 227
193 4. Simulation experiments this study, including their formulae, are summarized in Table 2. 228
All of these indicators are computed from the raw data collected 229
Williams R% Hn C t
Hn Ln 100
Stochastic K% C t Ltn
100
HHtn LLtn
219 4.2. Technical indicators Pn1
Stochastic D% i¼0
K ti %
n
220 Six technical indicators were used in this study as part of the in- Ct is the closing price, Lt is the low price, Ht is the high price at time t.
221 put variables to the network. Technical indicators are often used by Smoothing factor, a = 2/(1 + h), h is the time period for h day moving average (i.e. 5
222 investors in the stock market as a potential mechanism for predict- or 10).
223 ing market trends (Kim, 2003). A number of indicators have been HHt and LLt are highest high and lowest low in the last t days.
Upt means upward price change; Dwt means downward price change at time t.
224 developed, though only a subset of the major indicators were used
Fig. 2. One day future stock price versus trading day: model prediction and actual stock price.
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013
ESWA 8515 No. of Pages 6, Model 5G
25 April 2013
244 value. The Bayesian regularized network provides on average a indicates a line with a slope of one, which would indicate a perfect 256
245 >98% fit to future stock prices for both stocks over the full trading fit. The reduced error in the training data is expected, however, the 257
246 period. The technology and banking stocks were chosen because of limited spread in the testing data provides evidence that the model 258
247 the differences in industry market behavior and volatility. is providing appropriate generalization. 259
248 Microsoft stock was highly volatile over the length of the trading
249 period as well as in daily trading. The proposed model was able 5.3. Performance metric 260
250 to handle this noise and volatility without overfitting the data
251 which can be seen in the fit for testing data (points beyond day The performance of the Bayesian regularized artificial neural 261
252 600). This result provides evidence that the model can handle large network was measured by computing the mean absolute percent- 262
253 datasets with significant noise and volatility while maintaining age error (MAPE). This performance metric has been used in a 263
254 generalizability. Figs. 3 and 4 shows the results of this experiment number of studies and provides an effective means of determining 264
255 plotted as target stock price versus predicted price. The dashed line the robustness of the model for predicting daily trends (Chang & 265
Fig. 3. Comparison of microsoft target and predicted stock price – one day future value.
Fig. 4. Comparison of Goldman Sachs target and predicted stock price – one day future value.
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013
ESWA 8515 No. of Pages 6, Model 5G
25 April 2013
Table 3
Q3 Forecast accuracy of the Bayesian regularized ANN.
Stock Training MAPE (%) Test set MAPE (%) Total MAPE (%)
Microsoft (MSFT) 1.0494 1.0561 1.0507
Goldman Sachs (GS) 1.5235 1.3291 1.4860
Table 4
Forecast accuracy comparison with Hassan et al. (2007).
Stock name Mean absolute % error (MAPE) in forecast for 91 test dataset
Proposed Bayesian ANN Fusion model with weighted averagea ARIMA modela
Apple Inc. (AAPL) 1.9688 1.9247 1.8009
IBM Corporation (IBM) 0.7441 0.8487 0.9723
a
Models from Hassan et al. (2007)
Fig. 5. One day future stock price prediction for comparison with Hassan et al. (2007).
266 Liu, 2008; Chang et al., 2012; Hassan et al., 2007). This metric is regularized network can predict the stock price to within 98% 287
267 first calculated by finding the absolute value of the deviation accuracy. 288
268 between the actual stock price and the predicted stock price. This In order to test the efficacy of this approach, the network was 289
269 value is then divided by the actual stock price and multiplied by tested against the same data set used in Hassan et al. (2007) to di- 290
270 100 to determine the percentage error of the individual data point. rectly compare the two models. After a thorough analysis, the net- 291
271 This procedure is performed over the entire trading space, work architecture for this model included twenty hidden neurons, 292
272 summed, and then divided by total trading days to arrive at the as opposed to the five neurons in the first experiment. The results 293
273 MAPE value. The MAPE value is computed using the following of this experiment are shown in Table 4 and plotted in Fig. 5. The re- 294
274 equation: sults show that Bayesian regularized network forecasts are as good 295
275 as both the fusion model and the ARIMA model in Hassan et al. 296
Pr
i¼1 ðabsðyi pi Þ=yi Þ (2007). The data from Tables 3 and 4 showed that the proposed mod- 297
277 MAPE ¼ 100% ð4Þ
r el is an effective technique for estimating next day closing prices and 298
278 where r is the total number of trading days, yi is the actual stock can provide a simplified financial time series prediction tool com- 299
279 price on day i, and pi is the predicted stock price on day i. pared to other similar advanced neural network techniques. 300
281 Table 3 presents the experimental results for the two stocks (GS In this paper a novel time series forecasting tool was intro- 302
282 and MSFT) chosen for this study. The MAPE value was calculated duced. The model combines Bayesian regularization with the 303
283 for the training, testing, and total data sets to monitor the Levenberg–Marquardt algorithm to forecast the movement of 304
284 effectiveness of the model at predicting unknown data. It is inter- stock prices. The results of this model indicate that this tool re- 305
285 esting to note the small MAPE value for each stock during the duces the potential for overfitting and local minima solutions that 306
286 testing phase. These values indicate that on average, the Bayesian commonly plaque neural network techniques. The probabilistic 307
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013
ESWA 8515 No. of Pages 6, Model 5G
25 April 2013
308 nature of the network weights allows the investor to safely expand Chu, H. H., Chen, T. L., Cheng, C. H., & Huang, C. C. (2009). Fuzzy dual-factor time- 353
series for stock index forecasting. Expert Systems with Applications, 36, 354
309 the network without increasing the risk of overfitting through the 355
165–171.
310 use of the effective number of parameters. The proposed model Fernandez-Rodriguez, F., Gonzalez-Martel, C., & Sosvilla-Rivebo, S. (2000). On the 356
311 showed strong generalization with respect to recent stock trends profitability of technical trading rules based on artificial neural networks: 357
Evidence from the Madrid stock market. Economics Letters, 69, 89–94. 358
312 for two companies in varying market domains. To evaluate the effi- 359
Forsee, F.D., & Hagan, M.T. (1997). Gauss-Newton approximation to Bayesian
313 cacy of this model, the network was compared to an advanced hy- learning. In 1997 IEEE international conference on neural networks (Vols. 1–4, pp. 360
314 brid neural network model by Hassan et al. (2007). The results 1930–1935). Houston, TX, USA. 361
Garson, G. D. (1991). Interpreting neural-network connection weights. AI Expert, 6, 362
315 indicate that the forecasting ability of the model in this study is
47–51. 363
316 comparable, and potentially superior, to the fusion model and AR- Ghiassi, M., Saidane, H., & Zimbra, D. K. (2005). A dynamic artificial neural network 364
317 IMA model. Prediction of stock market trends is very important for model for forecasting time series events. International Journal of Forecasting, 21, 365
318 the development of effective trading strategies. This model is a no- 341–362. 366
Goh, A. T. C. (1995). Back-propagation neural networks for modeling complex 367
319 vel approach to solving this problem type and can provide traders a systems. Artificial Intelligence in Engineering, 9, 143–151. 368
320 reliable technique for predicting future prices. It is important to Hamzacebi, C., Akay, D., & Kutay, F. (2009). Comparison of direct and iterative 369
321 note that addition or substitution of other technical indicators artificial neural network forecast approaches in multi-periodic time series 370
forecasting. Expert Systems with Applications, 36, 3839–3844. 371
322 could improve the quality of this model in future applications. Harrington, P. B. (1993). Sigmoid transfer functions in backpropagation neural 372
networks. Analytical Chemistry, 65, 2167–2168. 373
323 Acknowledgements Hassan, M. R., Nath, B., & Kirley, M. (2007). A fusion model of HMM, ANN, and GA for 374
stock market forecasting. Expert Systems with Applications, 33, 171–180. 375
Kara, Y., Boyacioglu, M. A., & Baykan, O. K. (2011). Predicting direction of stock price 376
324 This work was supported by the Department of Energy Office of index movement using artificial neural networks and support vector machines: 377
325 Science Graduate Fellowship Program (DOE SCGF), made possible The sample of the Istanbul stock exchange. Expert Systems with Applications, 38, 378
5311–5319. 379
326 in part by the American Recovery and Reinvestment Act of 2009, 380
Kim, K. (2003). Financial time series forecasting using support vector machines.
327 administered by ORISE-ORAU under contract no. DE-AC05- Neurocomputing, 55, 307–319. 381
328 06OR23100. The views expressed in this work do not represent Kim, K., & Han, I. (2000). Genetic algorithms approach to feature discretization in 382
artificial neural networks for prediction of stock price index. Expert Systems with 383
329 those of the Department of Energy or the Office of Science and
Applications, 19, 125–132. 384
330 are solely those of the author. Leigh, W., Purvis, R., & Ragusa, J. M. (2002). Forecasting the NYSE composite index 385
with technical analysis, pattern recognizer, neural network, and genetic 386
331 References algorithm: A case study in romantic decision support. Decision Support 387
Systems, 32, 361–377. 388
Lendasse, A., De Bodt, E., Wertz, V., & Verleysen, M. (2000). Non-linear financial time 389
332 Armano, G., Marchesi, M., & Murru, A. (2004). A hybrid genetic-neural architecture
series forecasting application to the Bel 20 stock market index. European Journal 390
333 for stock indexes forecasting. Information Sciences, 170, 3–33.
of Economical and Social Systems, 14, 81–91. 391
334 Atsalakis, G. S., & Valvanis, K. P. (2009). Surveying stock market forecasting
Leung, M. T., Daouk, H., & Chen, A. S. (2000). Forecasting stock indices: A comparison 392
335 techniques – Part II: Soft computing methods. Expert Systems with Applications,
of classification and level estimation models. International Journal of Forecasting, 393
336 36, 5932–5941.
16, 173–190. 394
337 Baba, N., & Kozaki, M. (1992). An intelligent forecasting system of stock price using
MacKay, D. J. C. (1992). A practical Bayesian framework for backpropagation 395
338 neural networks. In Proceedings of the IEEE international joint conference on
networks. Neural Computation, 4, 448–472. 396
339 Q2 neural networks (pp. 371–377). 397
Oh, K. J., & Kim, K. J. (2002). Analyzing stock market tick data using piecewise
340 Blanco, A., Delgado, M., & Pegalajar, M. C. (2001). A real-coded genetic algorithm for
nonlinear model. Expert Systems with Applications, 2, 249–255. 398
341 training recurrent neural networks. Neural Networks, 14, 93–105.
Ritanjali, M., & Panda, G. (2007). Stock market prediction of S&P 500 and DJIA using 399
342 Burden, F., & Winkler, D. (2008). Bayesian regularization of neural networks.
bacterial foraging optimization technique. In 2007 IEEE congress on evolutionary 400
343 Methods in Molecular Biology, 458, 25–44.
computation (pp. 2569–2579). 401
344 Chang, P., & Liu, C. (2008). A TSK type fuzzy rule based system for stock price
Roh, T. H. (2007). Forecasting the volatility of stock price index. Expert Systems with 402
345 prediction. Expert Systems with Applications, 34, 135–144.
Applications, 33, 916–922. 403
346 Chang, P., Wang, D., & Zhou, C. (2012). A novel model by evolving partially
Walezak, S. (1999). Gaining competitive advantage for trading in emerging capital 404
347 connected neural network for stock price trend forecasting. Expert Systems with
markets with neural networks. Journal of Management Information Systems, 16, 405
348 Applications, 39, 611–620.
178–194. 406
349 Chen, Y., Yanga, B., & Abraham, A. (2006). Flexible neural trees ensemble for stock
Yudong, Z., & Lenan, W. (2009). Stock market prediction of S&P 500 via combination 407
350 index modeling. Neurocomputing, 70, 697–703.
of improved BCO approach and BP neural network. Expert Systems with 408
351 Chenoweth, T., & Obradovic, Z. (1996). A multi-component nonlinear prediction
Applications, 36, 8849–8854. 409
352 system for the S&P 500 index. Neurocomputing, 10, 275–290.
410
Please cite this article in press as: Ticknor, J. L. A Bayesian regularized artificial neural network for stock market forecasting. Expert Systems with Applications
(2013), http://dx.doi.org/10.1016/j.eswa.2013.04.013