Detecting and Analysing Candlestick

Pattern in Stock Market Using Deep

Learning

CAPSTONE PROJECT

Submitted by

Karthick Raja V
Reg. No. 22MDT1083

in partial fulfillment for the award of the degree of

M.Sc. Data Science

Department of Mathematics
School of Advanced Sciences
Vellore Institute of Technology, Chennai Campus
Vandalur - Kelambakkam Road, Chennai - 600 127
April - 2024
 DECLARATION

I hereby declare that the Project entitled Detecting and Analysing Candle-
stick Pattern in Stock Market Using Deep Learning submitted by me to the
Department of Mathematics, School of Advanced Sciences, Vellore Institute of
Technology, Chennai Campus, 600 127 in partial fulfillment of the requirements
of the award of the degree of Master of Science in Data Science is a bona-fide
record of the work carried out by me under the supervision of Dr. Saroj Kumar
Dash. I further declare that the work reported in this project, has not been submit-
ted and will not be submitted, either in part or in full, for the award of any other
degree or diploma of this institute or of any other institute or University.

Place: Chennai Signature of Student

Date: Karthick Raja V
 CERTIFICATE

This is to certify that the project entitled Detecting and Analysing Candle-
stick Pattern in Stock Market Using Deep Learning is prepared and submitted
by Karthick Raja V (Reg. No. 22MDT1083) to Vellore Institute of Technology,
Chennai Campus, in partial fulfillment of the requirement for the award of the de-
gree of Master of Science in Data Science is a bona-fide record carried out under
my guidance. The project fulfills the requirements as per the regulations of this
University and in my opinion meets the necessary standards for submission. The
contents of this report have not been submitted and will not be submitted either
in part or in full, for the award of any other degree or diploma and the same is
certified.

Place: Chennai Signature of Guide

Date: Dr. Saroj Kumar Dash
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning iii

Acknowledgement

With immense pleasure and a deep sense of gratitude, I wish to express

my sincere thanks to my supervisor Dr. Saroj Kumar Dash, Professor Grade 2,
School of Advanced Sciences, Vellore Institute of Technology (VIT), Chennai for
his motivation and continuous encouragement, this project would not have been
successfully completed.
I am grateful to the Chancellor of VIT, Dr. G.Viswanathan, the Vice
Presidents, the Vice Chancellor and the Pro Vice Chancellor for motivating me to
carry out the project at Vellore Institute of Technology, Chennai.
I express my sincere thanks to Dr. S. Mahalakshmi, Dean, School of
Advanced Sciences, VIT, Chennai for her support and encouragement.
I wish to extend my profound sense of gratitude to my parents for provid-
ing me the moral support and encouragement whenever required.

Place: Chennai Karthick Raja V

Date: 22MDT1083

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning iv

Abstract

Given that the stock market operates in a dynamic landscape where vast amounts
of information are generated every second, accurately forecasting prices poses
a significant challenge. The unpredictable nature of stock market fluctuations
often means that conventional approaches to market analysis rely on predicting
future prices, a strategy that carries the risk of inaccuracy. This master’s thesis
now shifts its focus towards the identification and interpretation of candlestick
patterns, visual representations of historical price movements, rather than solely
on the speculative act of price forecasting.
For traders and investors, analyzing candlestick patterns is essential to the
decision-making process. Conventional techniques for identifying and evaluat-
ing these patterns frequently rely on labor-intensive manual interpretation or rule-
based systems that are prone to bias. This master’s thesis suggests an innovative
method for automatically identifying and analyzing candlestick patterns in histor-
ical stock market data by utilizing deep learning techniques, most especially the
You Only Look Once (YOLO) v5 algorithm.
The primary objective of this research is to develop and evaluate a deep learning-
based framework capable of accurately identifying various candlestick patterns
from raw candlestick images. YOLO v5, a state-of-the-art object detection algo-
rithm known for its efficiency and accuracy, will be adapted and trained using a
comprehensive dataset of candlestick images derived from historical stock market
data.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning v

Contents

Declaration i

Certificate ii

Acknowledgement iii

Abstract iv

1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 Candle Sticks: . . . . . . . . . . . . . . . . . . . . . . . 7
1.1.2 Price Action : . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Motivation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Patterns: . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 13

2 Literature Review 16
2.1 Understanding market patterns . . . . . . . . . . . . . . . . . . . 16
2.2 Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Yolo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3 Methodology 21
3.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1.1 Data Source Selection . . . . . . . . . . . . . . . . . . . 21
3.1.2 Pattern Definition: . . . . . . . . . . . . . . . . . . . . . 22
3.1.3 Image Collection: . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Image Labeling: . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2.1 Data Class Count: . . . . . . . . . . . . . . . . . . . . . 27

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning vi

3.2.2 Roboflow: . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3 Data Split: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4 Results and Discussions 35

4.1 Evaluation Metrics: . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.2 Results: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.2.1 F1 score . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.2.2 Confusion matrix: . . . . . . . . . . . . . . . . . . . . . 39
4.2.3 Training and Validation Results: . . . . . . . . . . . . . . 40
4.2.4 Precision-Recall Curve: . . . . . . . . . . . . . . . . . . 42
4.2.5 Precision Curve and Recall curve . . . . . . . . . . . . . 44
4.3 Prediction: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5 Summary and Future Work 48

5.1 Summary: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.2 Future Scope: . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

A Code Used 50
REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning vii

List of Tables

3.1 Table with Class and Counts . . . . . . . . . . . . . . . . . . . . 27

3.2 Data split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning viii

List of Figures

1.1 candle stick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2 candlestick chart . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Long Body candlestick . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Long Shadow candlestick . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Engulfing candlestick . . . . . . . . . . . . . . . . . . . . . . . . 9
1.6 Tatasteel Weekly chart . . . . . . . . . . . . . . . . . . . . . . . 10
1.7 Double top and bottom . . . . . . . . . . . . . . . . . . . . . . . 12
1.8 Head and shoulder . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.9 Inverse Head and Shoulders . . . . . . . . . . . . . . . . . . . . . 14
1.10 Triangle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.1 TradingView site . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.2 screenshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 example data . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4 Annotation using Labelimg . . . . . . . . . . . . . . . . . . . . . 25
3.5 Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.6 Multiclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.7 data import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.8 bounding box . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.9 Yolov5 overview . . . . . . . . . . . . . . . . . . . . . . . . . . 33

4.1 F1 score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.2 confusion matrix . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.3 results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.4 PR curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.5 Precision Curve . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.6 Recall Confidence Curve . . . . . . . . . . . . . . . . . . . . . . 45
4.7 chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning ix

4.8 predicted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 1

Chapter 1

Introduction

Stock Market a marketplace of ownership and unpredictable charm. The stock

market stands as a multifaceted and ever-evolving system, playing a pivotal role
as the bedrock of the global economy. At its heart, it serves as a platform for
businesses to acquire funds by selling ownership holdings, while also allowing
investors to participate in their growth. Aside from its essential purpose, the stock
market has a fascinating attraction.The stock market is more than just buying and
selling company shares (stocks). While equities are a primary instrument, there
are others that help to drive total market activity. Here are some of the main
instruments traded in the stock market:
• Equities (Stocks): Equities represent ownership in a corporation. When you
purchase stock, you acquire a share of the company.

• Exchange-Traded Funds (ETFs): ETFs are collections of securities that

trade like individual stocks. They may track specific indexes, sectors, or
investment strategies.

• Derivatives: Derivatives are contracts whose value derives from an under-

lying asset, such as stocks, bonds, commodities, or currencies. Examples
include options, futures, and swaps.

• Bonds: Bonds are debt securities issued by corporations or governments.

Investors lend money to the issuer in exchange for periodic interest pay-
ments and the repayment of the principal amount at maturity.
In the stock market,there are two primary sorts of players with different aims and
strategies: traders and investors.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 2

Investors
• Focus: Long-term Growth (Years)
• Goal: Build money and attain financial goals throughout time.
• Investment Strategy: Concentrate on the underlying value of businesses and
their potential for future growth. They usually analyze a company’s funda-
mentals (financial health, industry developments, etc.) and hold stocks for
months or even years.
• Examples: Individuals saving for retirement, mutual funds, pension funds.

Traders
• Focus: Short-term Gains (Days, Weeks, Months)
• Goal: Profit from short-term fluctuations in stock prices.
• Trading Strategy: Use price charts, technical indicators, and market senti-
ment to discover buying and selling opportunities. They may adopt a variety
of methods, such as day trading or swing trading, with the goal of capitaliz-
ing on market swings in a shorter timeframe.
• Examples: Day traders, hedge funds (sometimes), individual investors using
short-term trading strategies.
In the field of stock market analysis, there are two primary approaches: fun-
damental analysis and technical analysis. Each strategy employs a specific set of
tools and targets a different segment of the market.
1. Fundamental Analysis
Fundamental analysis is a strategy that investors use to determine the in-
herent value of a stock. It’s similar to determining the genuine value of
a property before purchasing it, except instead of focusing on the house’s
condition and location, you’re examining a company’s financial health and
future possibilities.
• Financial Statements: These are the foundation of basic analysis. In-
vestors look at a company’s balance sheet, income statement, and cash
flow statement to determine its financial health, profitability, debt lev-
els, and cash flow creation.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 3

• Management Team: The quality and expertise of the management

team are critical to a company’s success. Investors consider the man-
agement team’s track record, vision for the firm, and capacity to tackle
problems.
• Competitive Advantage: What sets the firm apart from its competi-
tors? Is there a strong brand, unique technology, or cost advantage?
• Industry Trends: Understanding the general health and development
prospects of the industry in which a business works is critical. A strong
industry may raise even ordinary enterprises, but a falling sector can
bring down even the best-run organizations.
• Regulations: Government rules have a substantial influence on a com-
pany’s profitability. Investors should be aware of any possible regula-
tory impediments that may impact the sector.
• Economic Growth: A robust economy often results in increased busi-
ness profits, but a poor economy might reduce earnings.

2. Technical Analysis
Technical analysis is a technique used by investors and traders to assess as-
sets and uncover trading opportunities by studying statistical trends derived
from trading activity, such as price movements and volume. It contrasts with
basic analysis, which concentrates on a company’s financial health, indus-
try, and broader economic conditions.It argue that a security’s market price
already incorporates all essential information (stock, bond, etc.). This cov-
ers information such as corporate finances, industry trends, and economic
statistics.Technical analysts contend that a security’s market price already
includes all relevant information (stock, bond, etc.). This includes informa-
tion on business finances, industry trends, and economic data.

• Charts: Price charts provide the core of technical analysis. They visu-
ally depict a security’s price movement over time, allowing analysts to
see trends, patterns, and probable support and resistance levels.
• Technical Indicators:These are mathematical calculations based on price
and volume data that seek to identify trends, alert possible buying and
selling opportunities, and assess market mood. Moving averages, rel-
ative strength index (RSI), and MACD are three typical examples.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 4

• Support and Resistance: Support is a price level at which a stock often

attracts purchasing interest, keeping it from dropping further. In con-
trast, resistance is a price level at which the stock is likely to receive
selling pressure, inhibiting its upward rise. Identifying these levels
might be useful when making trading choices.

1.1 Background
Several time series models and neural network algorithms attempt to forecast
stock values but frequently fail.As trading and investing are zero-sum games in
which one loses so that another gains, if an algorithm can forecast the price, ev-
eryone can use it to win, and no one will lose, but the market does not operate
that way.Instead of anticipating prices, we may recognize price action patterns
in the market.The Business Standard news release contains statistics on financial
literacy in India. According to the announcement, a 2022 research conducted by
UK-based IBS Intelligence (IBSi) revealed that just 27% of India’s adult popu-
lation is financially educated. This emphasizes the need for improved financial
education activities in the country, particularly given students’ poor baseline fi-
nancial literacy (16.7%), as mentioned in the same article Standard (2023).

Candlestick Charts
Candlestick charts are a common method of visualizing stock market price changes.
They provide a more useful image than merely lines on a chart by including the
opening, closing, high, and low values of an asset (stock, bond, etc.) over a certain
time period (day, week, month).
Anatomy of candlestick:

• Body: The thickness of the candlestick shows the difference between the
opening and closing prices as shown in Fig 1.1.

• Wick (or Shadow): The thin lines extending from the top and bottom of the
body show the period’s highest and lowest prices. A lengthy upper wick
indicates that the price rose far above the opening price before dropping
back down. In contrast, a lengthy lower wick shows that the price dropped
much below the opening price before rebounding.

Bullish and Bearish Candles:

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 5

Figure 1.1: candle stick

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 6

• Bullish candle:it indicates a price gain over a particular term. The body
is usually filled (colored) and appears green, blue, or black, depending on
the charting platform. The upper wick is often short or missing, whilst the
bottom wick varies in length.
• Bearish Candle: A bearish candle represents a price drop over time. The
body is usually empty (unfilled) and has the same colors as bullish candles.
The top wick might vary in length, but the bottom wick usually extends
below the closing price.

Figure 1.2: candlestick chart

A collection of several types of candles creates a candlestick chart as seen

in Fig 1.2.Candlestick charts have a lengthy history, beginning in Japan in the
18th century. They were originally used by rice dealers to track price varia-
tions in rice. The credit for popularizing candlestick charts goes to Munehisa
Homma, a Japanese rice dealer who dealt in the Osaka rice market in the 18th
century. Homma studied market psychology and price movements and devised
a way to graphically portray them using what are now known as candlestick
charts..Homma’s method entailed documenting the opening and closing prices of
rice, as well as the highest and lowest prices for a given time period. He utilized
this information to develop a graphical depiction that we now know as candlestick
charts. Originally, these charts were made by hand.In the late twentieth century,
an American technical analyst named Steve Nison introduced candlestick charts to
the Western world. Nison released two publications in the early 1990s, ”Japanese
Candlestick Charting Techniques” and ”Beyond Candlesticks,” that introduced
Western traders and investors to candlestick charting.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 7

Nison’s publications not only taught the fundamentals of candlestick charting,

but also detailed many candlestick patterns and their interpretations. Traders im-
mediately grasped the importance of candlestick charts for monitoring price fluc-
tuations and finding possible trading opportunities.Since then, candlestick charts
have been one of the most used tools for technical analysis. They are often uti-
lized by traders and investors in a variety of financial markets, including stocks,
currency, commodities, and cryptocurrencies. Candlestick charts give a visual
picture of market mood, trend direction, and potential reversals, making them a
useful tool for market analysis and decision making.

Figure 1.3: Long Body candlestick

1.1.1 Candle Sticks:

There are several kinds of candles with varied meanings. Here are some common
candlestick shapes along with their meanings:
• Long-Body Candlesticks:
– Long White Candlestick: This candlestick has a lengthy white (or
green) body, signifying strong buying pressure during the trading ses-
sion. The longer the body, the stronger the buying urge.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 8

– Long Black Candlestick: In contrast, a lengthy black (or red) candle-

stick indicates significant selling pressure. The longer the body, the
stronger the selling pressure.Can be seen in Fig 1.3.

• Short-Body Candlesticks:

– Doji: A doji occurs when the starting and closing prices are almost
similar, resulting in a tiny or nonexistent body. It suggests a lack of
market consensus.
– Spinning top :it a short body but extended upper and lower shadows.
It implies that buyers and sellers are at odds or that they are unable to
decide.

• Candlesticks With Long Shadows:

Figure 1.4: Long Shadow candlestick

– Hammer: A hammer candlestick has a tiny body toward the top of the
candle and a long, downward shadow. It indicates a probable upward
reversal following a decline.
– Inverted hammer : It is similar to a hammer, except it appears at the
bottom of a downtrend. It indicates a probable upward reversal.

• Multiple Candlestick Patterns:

– Engulfing Pattern: This pattern is formed when the body of one candle
entirely engulfs the body of the previous candle. A bullish engulfing

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 9

Figure 1.5: Engulfing candlestick

pattern emerges at the bottom of a downtrend and indicates a potential

upward reversal, whereas a bearish engulfing pattern forms at the top
of an uptrend and indicates a potential downward reversal. Fig 1.5.
– Three Black Crows: This bearish pattern consists of three extended
black candlesticks that close sequentially downward. It signals a sig-
nificant downturn.
– Three White Soldiers: This bullish pattern is made up of three lengthy
white candlesticks that close successively higher. It suggests a signifi-
cant upswing.

1.1.2 Price Action :

Price action is the movement of an asset’s price over a specified time period, as
displayed on a chart Fig 1.6. It is a key idea in technical analysis, a technique
used by traders and investors to estimate future price movements using past data.
Price action analysis focuses on price changes rather than indications or economic
factors.In essence, price action represents the aggregate activity of market players,
including buyers, sellers, and speculators, in response to a variety of variables such

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 10

Figure 1.6: Tatasteel Weekly chart

as economic news, geopolitical events, supply and demand dynamics, and investor
emotion.Traders that use price action analysis look for patterns, trends, and other
forms in price charts to discover trading opportunities. They monitor how prices
move, whether they create identifiable patterns such as triangles, flags, or head
and shoulders, and if they adhere to important levels of support and resistance

1.2 Motivation:
Candlestick charts may reflect price changes over a wide range of periods, from
seconds to years, depending on trading frequency and length of interest. Here’s
how candlestick charts may be used at various time frames:

• Seconds/Minutes/Hours: In intraday trading, traders frequently utilize can-

dlestick charts with short periods of seconds, minutes, or hours to monitor
price changes inside a single trading session. These charts give insights into
short-term market swings, allowing traders to make judgments over shorter
time periods.

• Days: Traders and investors commonly use daily candlestick charts to mon-
itor price changes inside a single trading day. Each candlestick illustrates
the price activity that occurred on a single trading day, including the open-
ing, closing, high and low values. Daily charts are often utilized in swing
and position trading methods.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 11

• Weeks: Weekly candlestick charts give a more comprehensive perspective

of price changes by reflecting the price activity over a full trading week.
Each candlestick on a weekly chart represents the price change between
Monday’s start and Friday’s close. Weekly charts can help you spot long-
term trends and make intelligent trading decisions.

• Months/Years: Monthly and yearly candlestick charts provide even more

comprehensive views, summarizing price changes over longer time peri-
ods. Long-term investors often use these charts to discern market trends
and patterns. Monthly and annual charts can assist investors make long-
term judgments about asset allocation and portfolio management.

• Custom periods: Certain trading systems allow users to create candlestick

charts that reflect specified periods, such as quarterly or semi-annual charts.
Custom periods are useful for examining price changes in relation to certain
market events or economic cycles.
Regardless of time period, the fundamental concepts of candlestick anal-
ysis remain consistent. Traders use candlestick patterns and formations to
spot prospective trends, reversals, and trading opportunities, whether they’re
looking at short-term intraday charts or longer-term monthly charts.

1.2.1 Patterns:
The world of technical analysis offers a vast array of price action patterns,
each with its own interpretation and potential implications.Some of the fre-
quently occurring patters are

– Double bottom (W-bottom): A double bottom (Fig 1.7) occurs when

the price strikes a low, bounces back, revisits a similar low, and then
recovers again. It indicates a probable upward reversal.
– Double Top (M-Top): Unlike the double bottom, this bearish reversal
pattern features two successive highs at about the same price level,
with a trough (middle bottom) in between. Confirmation happens
when the price falls below the support level created by the lows of
the two peaks.
– Rounding Top: This bearish reversal pattern looks like a dome or
rounded hill, indicating a gradual loss of upward momentum. Confir-

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 12

Figure 1.7: Double top and bottom

mation comes when the price falls below the support level established
prior to the rounded top formation.
– Head and Shoulders(Fig 1.8): This pattern usually appears at the apex
of an upswing and consists of three peaks. The central peak (the head)
is taller than the two adjacent peaks (the shoulders). It indicates a
possible trend reversal from bullish to negative.
– Inverse Head and Shoulders: The inverse of the head and shoulders
pattern, it usually appears at the bottom of a downturn and indicates a
possible turnaround from bearish to bullish.Seen in Fig 1.9.
– Triangle: In a symmetrical triangle pattern, the top and lower trend-
lines converge to form a triangular shape. This pattern implies a period
of market indecision, with prices making lower highs and higher lows.
Traders predict a breakout from the triangular formation, but the di-
rection is not decided. Breakouts above the top trendline indicate a
bullish continuation, while those below the lower trendline indicate a
bearish continuation.Fig 1.10.
– Ascending/Descending Wedge: Like triangles, wedges are generated
by converging trend lines. An ascending wedge pushes upward and

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 13

Figure 1.8: Head and shoulder

can be bullish or negative (depending on the circumstances). A sink-

ing wedge, on the other hand, slopes downward and is often seen as
bearish. Breakouts from wedges frequently indicate a continuation of
the previous trend.

1.2.2 Motivation
Many traders and investors use algorithms to try to anticipate market or stock
prices, but this is impossible; if an algorithm can predict the price, everyone in
the market wins, and trading is a zero-sum game, thus it is useless.Instead of
anticipating prices, we can spot price movement patterns, which gives us an ad-
vantage in the market and allows us to make better educated decisions about trades
and investments. It will also assist many people without market understanding in

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 14

Figure 1.9: Inverse Head and Shoulders

identifying trends through the use of algorithms.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 15

Figure 1.10: Triangle

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 16

Chapter 2

Literature Review

2.1 Understanding market patterns

Adam Grimes (2012) discusses the art and science of technical analysis. Science
focuses on variables that can be measured and tested. Teaching a science fre-
quently entails creating a body of knowledge, procedures, and data approaches,
as well as researching what is known and knowable. Art is often seen as subjec-
tive and imprecise, yet this is not entirely accurate. Both are interrelated and
cannot exist alone. Scientific progress is based on both logical and inductive
leaps to solve philosophical and epistemological issues at the edges of knowl-
edge. Art uses scientifically quantifiable tools and procedures, but it also symbol-
izes a different way of knowing.What distinguishes it from usual logical proce-
dures.Science and art are intrinsically linked: science without art is sterile, and art
without science is soft and incomplete. This is especially true while researching
the current financial markets.

2.2 Deep Learning

HyunSik Sim, HaeIn Kim, and JaeJoon Ahn Sim et al. (2019) investigate the pos-
sible use of deep learning techniques in picture identification for stock market
prediction. Stock market forecasting is a challenging assignment for investors.
In this paper, they introduce a convolutional neural network (CNN)-based stock
price prediction model to evaluate the use of innovative learning methodologies in
the stock market. When CNN was utilized, 9 technical indicators were chosen as

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 17

forecasting model predictors, and these indicators were converted into time series
graph images. To assess the usefulness of deep learning for image recognition
in financial markets, the projected accuracies of the proposed model were com-
pared to those of a standard artificial neural network (ANN) and a support vector
machine (SVM) model. The experimental results suggest that CNN may be an
effective tool for developing stock prediction models.

Candlestick charts, as explained by Chen & Tsai (2020), display high, low,
opening, and closing prices over a certain time period. Candlestick patterns form
when human actions and reactions are patterned and frequently imitated. These
patterns capture information about the candles. Thomas Bulkowski’s Encyclo-
pedia of Candlestick Charts includes 103 candlestick designs. Traders use these
patterns to determine when to enter and exit. Candlestick pattern classification
techniques make it simpler to visually recognize these patterns. To show its po-
tential, we propose a two-step approach for spotting candlestick patterns automati-
cally. In the first stage, the time series is encoded as various types of images using
the Gramian Angular Field (GAF).In the second stage, a Convolutional Neural
Network (CNN) is used to learn eight different types of candlestick patterns from
GAF pictures. This publication refers to the approach as GAF-CNN. The study
found that their method surpassed the LSTM model in recognizing eight types of
candlestick patterns in real-world data with an average accuracy of 90.7%.

Chen et al. (2020),adopt a Gramian Angular Summation Field (GASF) time

series encoding method to emphasize time series properties in the CNN model.The
report refers to the method as a GASF-CNN model. Initially, a GASF-CNN model
was trained to properly capture the eight key candlestick patterns (90.0%).This
study has two important contributions. The work introduces a GASF-CNN model
for spotting candlesticks in financial vision research as an innovation.The study
proposes a modified local search adversarial strategy to explain the GASF-CNN
model’s ability to recognize different candlestick patterns.

Ramadhan et al. (2022) conducted research on the effectiveness of candle-

stick patterns in predicting trading positions. They utilized Gramian Angular
Field (GAF) to represent candlestick patterns visually and then applied Convo-
lutional Neural Network (CNN) and Long Short-Term Memory (LSTM) models
to forecast closing prices. The trading positions involved buying and selling with a
hold period of several hours. Their results showed that CNN accurately predicted
3-hour and 5-hour GAF candlestick patterns with accuracies of 90% and 93%,

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 18

respectively. Using a 10-hour look back, LSTM successfully forecasted closing

price trends with a Root Mean Square Error (RMSE) score of 155.458 and a Mean
Absolute Percentage Error (MAPE) of 0.9754%. Additionally, when combining
CNN-LSTM models with a three-hour hold time, the accuracy of detecting 85
candlestick patterns in test data reached 82.7%, compared to 60% accuracy when
using CNN alone. This combination of CNN-LSTM models enhances candlestick
pattern prediction and provides more profitable trading opportunities compared to
using CNN alone .

2.3 Yolo
According to Huang et al. (2021), an effective and efficient object detector is es-
sential for practical applications. To solve these two issues, they extensively eval-
uate a set of current improvements that improve the performance of PP-YOLO
while keeping the infer time nearly constant. This project will use an incremental
ablation study to investigate a set of enhancements and experimentally assess their
impact on final model performance. Things they tried that didn’t work will also be
discussed. They increased the performance of PP-YOLO on COCO2017 test-dev
by incorporating numerous beneficial adjustments, boosting it from 45.9% mAP
to 49.5%.

Serdar Birogul, Günay Temür, and Utku Kose (2020)Birogul et al. (2020) cre-
ated an algorithm that analyzes one-year candlestick charts of stocks traded on
Borsa İstanbul (BIST) from 2000 to 2018. The model’s ability to make a ”Buy-
Sell” decision based just on candlestick charts indicates potential for reliable fore-
casting. The purpose is to improve the ”Buy-Sell” decision-making process by
offering foresight alongside other basic and technical assessments. It may also be
utilized to make independent financial decisions. The test findings clearly demon-
strate the importance of this foresight on achievement. The model had an 85%
success rate and generated a 100% profit. Furthermore, the method is applicable
to any time series that can be represented using candlestick charts.

Kim et al. (2022) highlight the significance of the SMD (Singapore Maritime
collection) as a valuable resource for training deep neural networks (DNN) in mar-
itime object detection, despite its issues with noisy labels and imprecise bounding
boxes. Their research focuses on rectifying these annotations to create a more reli-

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 19

able benchmark dataset named SMD-Plus. Additionally, they propose augmenta-

tion strategies tailored specifically for SMD-Plus, including online transformation
using Copy & Paste to address class imbalance and the mix-up approach inte-
grated with fundamental augmentation techniques for YOLO-V5. Experimental
results demonstrate that the modified YOLO-V5 with SMD-Plus achieves superior
performance in detection and categorization compared to the original YOLO-V5
model.

Yang et al. (2020) all utilize YOLOV5 to detect faces, even while they are
wearing masks. Standing in front of the camera allows you to identify people
and unlock the gate. This reduces the need for human crowd management while
saving time and resources. Our experiment had a success percentage of 97.9%.
They compared our findings to other standard machine learning algorithms. Fur-
thermore, a snapshot of someone wearing a mask but not covering their nose is
clearly identified.Given the global effect of COVID-19, we believe this method
can successfully reduce exposure distances while improving surveillance.

Yao et al. (2021) introduced a high-precision method for detecting flaws in

kiwifruit utilizing YOLOv5. They curated a unique dataset comprising four types
of faults in kiwifruit, marking the first instance of such a dataset in the realm of
agricultural products post-manufacturing. Additionally, this marks the inaugu-
ral application of the YOLOv5 network to crops. Enhancements to the YOLOv5
model included the incorporation of a target detection layer and SELayer to aug-
ment feature extraction capabilities. Furthermore, they enhanced model accuracy
and convergence by transitioning from the DIoU loss function to the CIoU func-
tion, resulting in a 9% increase in mean Average Precision ([email protected]) compared
to the original YOLOv5 model. With the ability to recognize a single picture
within 0.1 seconds on a GPU 1050Ti, coupled with its robustness in various envi-
ronmental conditions, the model presents efficient and intelligent post-production
reprocessing strategies for farmers.

Horvat & Gledec (2022) study delves into the prominence of YOLOv5 as a
prevalent and contemporary iteration within deep learning neural networks, partic-
ularly within computer vision tasks. The YOLO algorithm’s increasing popularity
stems from its efficiency in handling complex data scenarios, making it a favored
choice among data scientists. This paper aims to conduct a comparative analysis
of various iterations of the YOLOv5 model using common image datasets. By
doing so, it seeks to provide researchers with refined recommendations for select-

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 20

ing the most suitable model tailored to specific problem categories. The analysis
considers factors such as performance, accessibility, and ease of use, particularly
when integrated with popular programming languages like Python.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 21

Chapter 3

Methodology

3.1 Data Collection

3.1.1 Data Source Selection
Choosing an acceptable data source for candlestick chart pictures entails locating
a trustworthy source of historical financial data that provides the requisite OHLC
(Open, High, Low, Close) values for various financial assets.In this research study,
we will use the trading view website as our major source of data.

Trading view
TradingView is a popular web-based platform for traders and investors that offers
a variety of tools and features for researching financial markets, exchanging trad-
ing ideas, and placing trades. Here are some important aspects of TradingView.Fig
3.1.

• Charting Tool: TradingView provides powerful charting and technical anal-

ysis capabilities. It enables users to build bespoke charts for a variety of
asset classes, including stocks, FX, cryptocurrencies, and more. Users may
use a variety of technical indicators and sketching tools to study market
movements and spot possible trading opportunities.

• Market Data: The website gives users access to real-time and historical
market data for a variety of financial products, such as equities, indices,
FX, cryptocurrencies, and commodities. Users may get price quotations,
volume data, and other pertinent information for their favorite markets.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 22

• Screeners and Alerts: TradingView provides customisable screeners and

alerts to assist users find trading opportunities based on specified criteria
such as price movements, technical indicators, and fundamental data. Users
can create alerts to get notifications when specific circumstances are satis-
fied.

Figure 3.1: TradingView site

3.1.2 Pattern Definition:

To train a deep learning (DL) model to identify chart patterns in financial data, it’s
essential to select specific patterns that are commonly recognized by traders and
analysts. Determine the chart patterns we want the DL model to identify .Gather
reference photographs for each pattern that illustrate variances to guarantee com-
plete coverage.

Patterns used for this paper:

• Head and Shoulder
• Inverted head and shoulder
• Double top
• Double bottom
• Triangle

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 23

3.1.3 Image Collection:

Figure 3.2: screenshots

When it comes to selecting time frames for your image collection, it’s impor-
tant to carefully consider the periods that best align with the specific trends you are
aiming to target. You might opt for daily intervals to capture short-term fluctua-
tions or weekly snapshots for a broader perspective. Moving on to stock selection,
diversifying your portfolio with a mix of equities that cover different industries
and exhibit distinct historical patterns is key. This diversified approach enables
your model to build a robust understanding of market behaviors and trends. As
for image capture, it is advisable to take a meticulous Manual image data col-
lection by taking screenshots can be a time-consuming and challenging process,
particularly in scenarios where there is a significant volume of data to be captured,
such as with extensive datasets or instances where meticulous and accurate label-
ing is essential to maintain the integrity and reliability of the collected information
approach by either manually capturing screenshots or exporting charts as images,
ensuring consistent quality and choosing appropriate file formats like PNG or JPG
to maintain clarity and compatibility. manual image collection is a major bottle-
neck in this process. It can be incredibly time-consuming, especially if you need
a large and diverse dataset.

• Repetitive Task: Engaging in a repetitive work, such as manual data collec-

tion, typically needs persons to do tedious acts such as browsing web pages,

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 24

Figure 3.3: example data

taking screenshots, and transcribing information. These duties can become

monotonous owing to their repetitive nature, which requires a significant
commitment of time, particularly when managing large datasets. The pro-
cedure may include going through several sources, methodically acquiring
pertinent information, analyzing it, and arranging it systematically. As a
result, doing such jobs frequently requires prolonged focus and attention to
detail.

• Search and Selection Process: When beginning the search and selection
process, individuals frequently face the issue of discovering relevant data,
which can take a substantial amount of time. This is especially true when
the data is spread out among several sources or requires precise selection
based on predetermined criteria. This implies that researchers or data col-
lectors may have to filter through massive amounts of information in order
to identify the most relevant data points for their requirements. In essence,
the procedure can be complex and time-consuming due to the need for care-
ful investigation and judgment in finding essential data.

• Data Entry and Annotation: it is the critical act of putting acquired data
into a database or spreadsheet, which is then properly annotated with rele-
vant information. This portion of the data processing journey can be time-
consuming, especially when dealing with data that requires manual format-

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 25

ting or comments that must be carefully inserted by hand. The attention

to detail necessary while entering and enriching data is critical to ensuring
correctness and facilitating subsequent analysis and understanding of the
dataset.

3.2 Image Labeling:

LabelImg, a widely used open-source application, is chosen by developers and
academics due to its efficient annotation capabilities with bounding boxes. This
tool is widely used in machine learning and computer vision projects that require
precise picture labeling to improve object recognition accuracy in a variety of
applications.

• Bounding Box Annotation: LabelImg, a powerful annotation tool, empow-

ers users by providing the capability to easily create precise rectangular
boxes encasing specific objects within an image. These carefully drawn
boxes function as pivotal markers that not only outline the spatial attributes
of the objects but also offer crucial information regarding their dimensions,
aiding in the comprehensive learning process for the machine learning model.Fig.3.4.

Figure 3.4: Annotation using Labelimg

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 26

• Labeling: the process of labeling, you have the option to allocate a des-
ignated label, such as text, to every object that receives a bounding box.
These labels play a crucial role in enabling the model to comprehend the
meaning and identity of the object contained within the box. Examples of
these labels include ’Head and shoulders top’, ’head and shoulders bottom’,
’triangle’, ’W bottom’, and ’M top’ , each serving as a distinctive marker to
guide the model’s recognition capabilities effectively.

• Predefined Labels: LabelImg offers a convenient feature that allows you to

set up a designated collection of predefined labels seen in Fig 3.5. These
labels are recommended by the tool as you annotate, which helps to simplify
and accelerate the process of annotation, making it more efficient and time-
saving for users.

Figure 3.5: Classes

• LabelImg Multiple Label Support: supports the annotation of images con-

taining multiple objects, providing the flexibility to assign distinct labels to

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 27

each object individually. This feature enables users to accurately catego-

rize and differentiate various elements within an image, making it a versa-
tile tool for tasks requiring precise object identification and labeling. With
LabelImg’s capability to handle diverse labeling requirements, users can ef-
ficiently annotate complex images, streamlining the process of generating
labeled datasets for training machine learning models or conducting visual
analysis.

3.2.1 Data Class Count:

For this research, I employed a total of five different patterns, each with cor-
responding count values clearly outlined in the accompanying table 3.1. This
method allowed for a comprehensive analysis of the data, providing a structured
approach to the study’s findings and enhancing the overall depth of the research
process.

Table 3.1: Table with Class and Counts

Class Count
Head and shoulder top 1371
Head and shoulder bottom 1185
M top 1824
W bottom 1480
Triangle 401

In the intricate world of stock market analysis, it is not uncommon to find a

single stock chart image revealing a multitude of chart patterns all at once. For
instance, within one image displaying a head and shoulders pattern, one might also
observe the emergence of a bullish triangle breakout. This complex intersection
of multiple chart patterns within a single chart image serves as a testament to the
intricate nature of technical analysis in the realm of trading and investment.Fig
3.6.

3.2.2 Roboflow:
Roboflow stands out as an all-encompassing platform tailored specifically to en-
hance the efficiency of constructing computer vision applications. Eager to sim-
plify and optimize workflow, it boasts a diverse array of tools that cover every

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 28

Figure 3.6: Multiclass

stage of the development process. Whether it’s facilitating data collection and an-
notation tasks or aiding in model training and deployment endeavors, Roboflow
serves as a comprehensive solution. Now, let’s delve into a detailed exploration of
its essential features and capabilities.

• Data Management:

– Uploading Datasets:Roboflow offers multiple methods for uploading

datasets for your machine vision project. Roboflow’s platform pro-
vides a smooth and intuitive experience, whether you choose a user-
friendly online interface where you can effortlessly drag and drop pho-
tos and videos, or the efficiency of a command-line interface (CLI)
optimized for processing huge datasets with speed and precision. You
may smoothly transmit your data and multimedia resources, enhanc-
ing your computer vision efforts using Roboflow’s numerous tools and
functions.
– Annotations:Roboflow supports several different annotation formats,
including popular ones like COCO, YOLO, Pascal VOC, and Labe-
lImg. This adaptability enables you to work effortlessly with a variety
of datasets. Whether you wish to upload existing annotations or utilize
the platform’s user-friendly built-in capabilities for exact data labeling,
Roboflow simplifies the annotation process.

• Data import

– To export your dataset, first locate the dataset version you want to
download and then click on the ”Export” button available next to it.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 29

Within the export options menu, ensure to select ”YOLOv5 PyTorch”

as the designated format for the export process.

• Download

– By accessing the provided Code Snippet Fig 3.7, you will be pre-
sented with Python code specifically tailored for seamless integration
into your project. This resource enables you to efficiently retrieve the
dataset via Roboflow’s API, streamlining your development process.
This functionality proves invaluable for incorporating dataset down-
loads as an integral component of your daily workflow.

Figure 3.7: data import

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 30

3.3 Data Split:

Dividing image data into 80% for training, 13% for validation, and 7% for test-
ing is a common approach in deep learning model training. This segmentation is
crucial for enhancing the model’s performance and reliability. Initially, assigning
80% of the data to the training set enables the model to grasp intricate patterns
and subtleties within the dataset, laying a robust groundwork.Second, the valida-
tion set, which makes up 13% of the data, serves as a checkpoint during training,
preventing the model from becoming unduly specialized on the training data—a
condition known as overfitting. Overfitting can dramatically reduce the model’s
generalizability and practical value.Finally, the remaining 7% chosen for the test
set is critical in giving an objective assessment of the model’s performance af-
ter training. This set provides a trustworthy signal of how well the model will
perform in practical, real-world applications since it evaluates the model on previ-
ously unknown data. In summary, this tripartite data allocation technique not only
boosts generalizability but also allows for precise evaluation, which improves the
model’s performance and applicability in a variety of settings. The split is ex-
plained in table 3.2.

Table 3.2: Data split

Split proportion count
Train 80% 5318
Valid 13% 869
Test 7% 449

3.4 Model
YOLOv5
The ”You Only Look Once” technique approaches object recognition holistically,
examining all input photos at once. This is distinct from the usual ”sliding win-
dow” technique, in which certain portions of the picture are treated independently
and often through numerous steps of categorization. This modern technique con-
siderably boosts processing efficiency by reviewing the entire image at once, elim-
inating the need for many separate assessments and resulting in speedier findings.
Furthermore, this technology improves detection accuracy by adding the total vi-
sual context into the neural network, resulting in a more complete comprehension

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 31

of the scene under analysis. With this global context available throughout the
network, the model can make more informed and precise decisions, resulting in
improved performance in object detection tasks.

Figure 3.8: bounding box

YOLO (You Only Look Once) is a cutting-edge object identification tech-

nique that relies heavily on bounding boxes to effectively identify and categorize
items in photos. In practice, YOLO uses bounding boxes as precise rectangular
frames that essentially serve as markers to precisely locate and classify objects
with exceptional efficiency and accuracy, making it the preferred choice for tasks
requiring quick and precise object localization and recognition in image process-
ing apps.Fig 3.8.

• Divide the Image:

– YOLO splits the supplied picture into a grid of squares. This grid aids
in localizing things inside the picture.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 32

• Bounding Box Prediction by Cell:

– In contrast to certain object detectors that employ a sliding window

technique, YOLO predicts bounding boxes and their associated class
probabilities for each grid cell.
– This means that each cell predicts many outcomes:
* A bounding box (coordinates for a rectangle surrounding the item)
* Confidence score (probability that a box contains an item)
* Class probabilities (the possibility of an object belonging to a cer-
tain class)

• Anchor Boxes (predefined shapes):

– To boost efficiency, YOLO uses predetermined forms known as anchor

boxes. These serve as references for the expected bounding boxes.
– During training, the model learns how to adapt these anchor boxes to
accommodate the forms of various objects.
– To accommodate for object differences, you can declare different an-
chor box sizes and ratios ahead of time.

• Choosing the Best Bounding Box:

– YOLO anticipates several bounding boxes per cell owing to anchor

boxes.
– To avoid duplication, a method called Non-Maximum Suppression
(NMS) is utilized.
– NMS evaluates the confidence ratings of overlapping bounding boxes
and eliminates the redundant ones, leaving just the most confident pre-
diction for each item.

The YOLOv5 network design is made up of three key components: the back-
bone, neck, and output seen Fig 3.9. Initially, the input terminal is in charge of
performing critical data preparation activities including mosaic data augmenta-
tion and adaptive image filling. To efficiently adapt to varied datasets, YOLOv5
features adaptive anchor frame computation right on the input, allowing for auto-
mated modification of the anchor frame size as the dataset changes. Moving on
to the backbone, it acts as a convolutional neural network, efficiently combining

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 33

and building picture features at various degrees of granularity.By implementing

techniques such as cross-stage partial networks and spatial pyramid pooling, the
backbone can successfully extract feature maps of varying sizes from the input
image through a series of convolution and pooling operations.

Figure 3.9: Yolov5 overview

The BottleneckCSP design is a crucial technique for optimizing computational

operations, leading to significantly faster inference speeds. Additionally, the SPP
framework plays a vital role in extracting features from multiple scales within a
single feature map. This innovative approach results in the generation of three-
scale feature maps, thereby enhancing detection accuracy. The neck neural net-
work acts as a sophisticated array of interconnected layers designed to efficiently
merge and process image data before forwarding it for predictive analysis.
This network architecture heavily relies on the feature pyramid topologies of
FPN and PAN. Specifically, the FPN architecture facilitates the transfer of high-
level semantic information from higher-level feature maps to lower levels. In
contrast, the PAN design excels in conveying precise localization characteristics
from lower to higher layers of the network. The synergistic integration of FPN
and PAN structures enhances the feature extraction process across various net-
work layers during Backbone fusion, significantly boosting the system’s overall
detection performance.
Furthermore, the head output is primarily responsible for identifying targets of
different sizes distributed across the feature maps, enabling a more comprehensive

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 34

detection approach.
The YOLOv5 model encompasses a range of architectures, including YOLOv5n,
YOLOv5s, YOLOv5m, YOLOv5l, YOLOv5x, YOLOv5n6, YOLOv5s6, YOLOv5m6,
YOLOv5l6, and YOLOv5x6 + TTA (Jocher, 2020a). However, the academic fo-
cus typically centers on the initial five variations: nano, small, medium, big, and
xlarge. These models differ primarily in the number of feature extraction modules
and convolution kernels, influencing practical considerations regarding computa-
tional resources and the overall neural network parameters.
The specified sizes for YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and
YOLOv5x are 4 MB, 14 MB, 41 MB, 89 MB, and potentially up to 166 MB,
respectively. These details are pivotal for users to make informed decisions re-
garding the model that best suits their computational and performance needs.
For our pattern detection model, we opted to utilize the YOLOv5s model with
pre-trained weights. During the training phase, the model ingests the data and
saves the weights to a file for future predictions, ensuring optimized accuracy in
pattern recognition. Throughout the training process, various metrics are com-
puted, and these results, alongside insightful plots and visualizations, are meticu-
lously archived in a dedicated folder. This comprehensive repository of training
outcomes serves as a valuable resource for analyzing the model’s performance
and refining its capabilities to enhance pattern detection accuracy in subsequent
iterations.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 35

Chapter 4

Results and Discussions

4.1 Evaluation Metrics:

When evaluating the effectiveness of a YOLO model, it is critical to look beyond
just classifying findings as ”correct” or ”incorrect.” The assessment approach in-
cludes assessing both the accuracy of item categorization and the precision of
object localization inside the bounding box. Metrics must be chosen to fully mea-
sure these dual components. This allows the assessment to adequately reflect
the model’s capacity to reliably recognize things in their proper spatial context.
Having a range of assessment criteria allows for a better understanding of how
effectively the YOLO model is operating.

1. Intersection over Union (IoU):

• Intersection over Union (IoU) is a crucial metric in object detection

tasks as it quantifies the spatial agreement between a predicted bound-
ing box and the actual ground truth bounding box, depicting how ac-
curately the object’s location is captured in the image.
• The IoU, computed by dividing the intersection area of the two boxes
by their union area, acts as a measure of localization accuracy, where
a higher IoU value signifies a more precise alignment between the
predicted and ground truth bounding boxes.
The formula,
Area of Intersection
IOU = (4.1)
Area of Union

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 36

Where:

Area of Intersection = Area of the overlap between the predicted bounding box and the gr
bounding box
Area of Union = Total area covered by both the predicted bounding box and the gro
bounding box

2. Precision:

• Precision is a crucial metric in evaluating the performance of your

model as it directly assesses the accuracy of the detection it gener-
ates. By definition, precision essentially quantifies the quality of the
positive predictions by delving into the ratio of accurate detection to
the overall detection produced by your model, encompassing both true
positives and false positives. This insightful metric gives you a clear
understanding of how effectively your model discerns and labels rele-
vant data points.Formula for precision is given below,
The formula for precision in a classification task is given by:

True Positives
Precision = (4.2)
True Positives + False Positives
3. Recall:

• Recall is a crucial metric as it measures the model’s ability to correctly

identify all objects in an image. This metric indicates the proportion
of true positives correctly detected relative to the total actual objects
present in the image, considering both the correctly identified objects
and those missed by the model. By calculating recall, you gain insight
into how effectively the model captures the entirety of the objects of
interest within the image.
• The formula for recall in a classification task is given by:

True Positives
Recall = (4.3)
True Positives + False Negatives
4. Mean Average Precision (mAP):

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 37

• The Mean Average accuracy (mAP) metric is important for evaluat-

ing the performance of YOLO models because it provides a complete
measure that takes into account both accuracy and recall at different
Intersection over Union (IoU) thresholds. Essentially, mAP evaluates
how well the model predicts bounding boxes with varying confidence
ratings, weighing accuracy and recall.
• The approach entails computing accuracy and recall at each confi-
dence level using IoU measurements and displaying these results to
create an accuracy-Recall curve.
• The Average Precision (AP) is then calculated to characterize the de-
tection quality at a given IoU threshold, with mAP indicating the aver-
age AP over several IoU thresholds, which generally ranges from 0.5
to 0.95. A greater mAP indicates better overall detection performance.
• The formula for mean Average Precision (mAP) across multiple classes
is given by:

N
1 X
mAP = APi (4.4)
N i=1

Where N is the total number of classes or categories, and APi is the

Average Precision for class i.

5. F1 Score:

• The F1 score, which is a harmonic mean of accuracy and recall, serves

as a single metric that captures the trade-off between these two critical
assessment criteria.
• The F1 score provides a thorough evaluation of a model’s performance
in settings where both accuracy and recall are equally important.
• This balanced evaluation is especially useful in activities like infor-
mation retrieval, medical diagnostics, and anomaly detection, where
maintaining an ideal balance of precision and recall is critical for suc-
cessful decision-making and performance assessment.
• The formula for F1 score is given by:

Precision × Recall
F1 Score = 2 ×
Precision + Recall

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 38

(4.5)

4.2 Results:
4.2.1 F1 score

Figure 4.1: F1 score

The graph 4.1 illustrates the relationship between precision and recall as the
confidence threshold varies. Precision, which measures the proportion of correctly
identified instances among all detected instances, is plotted on the y-axis with
values ranging from 0 to 1. Recall, on the other hand, denotes the proportion of
correctly identified instances among all actual instances and appears on the x-axis
also ranging from 0 to 1.
The ’all classes 0.73 at 0.388’ curve represents the combined precision-recall
outcomes for all classes considered concurrently. For a confidence threshold of
0.388, the model attains a precision rate of 0.73 across all classes. This con-

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 39

veys that when the model’s predictions have a confidence score surpassing 0.388,
around 73
Each distinct curve corresponds to the precision-recall trade-off for individual
classes; however, due to the graph’s complexity, identifying the specific classes
represented by each curve may be challenging. These curves provide insights
into how precision and recall fluctuate per class as the confidence threshold is
adjusted, offering valuable information on model performance across different
class categories.

4.2.2 Confusion matrix:

Figure 4.2: confusion matrix

A confusion matrix Fig 4.2 is an excellent tool for demonstrating the perfor-
mance of classification models. The rows of the matrix represent the actual pricing
patterns seen in the picture data, which reflect the ground truth information uti-
lized for evaluation. On the other side, the columns show the pricing patterns
predicted by the YOLOv5 model for each image. The meticulous investigation of

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 40

this confusion matrix yields important insights into the YOLOv5 model’s accu-
racy in recognizing the complicated pricing patterns inside the picture data. This
study provides a comprehensive view of where the model succeeds and where
it may need improvement.By scrutinizing and interpreting the confusion matrix,
one can pinpoint specific areas for improvement, whether it be through refining the
training data utilized or tweaking the architecture of the model itself. Ultimately,
such insights derived from the confusion matrix are invaluable in bolstering the
overall performance and efficacy of the YOLOv5 model in its task of identifying
and categorizing price patterns within the image data.There seems to be a decent
amount of correct detections along the diagonal, suggesting the model can identify
some price patterns reasonably well.There are also non-zero values off-diagonal,
indicating both false positives and false negatives. The severity of these errors
depends on the specific application. In price pattern detection, missing a crucial
pattern (false negative) might be more critical than a false positive.

4.2.3 Training and Validation Results:

General observations:
• Convergence: All training losses (box loss, obj loss, and cls loss) tend to
decrease over time, indicating that the model is learning and improving ac-
curacy in box placement, object identification, and classification in Fig 4.3.

Figure 4.3: results

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 41

• Smooth Curves: Smooth loss curves, especially during training, indicate a

reliable process with few abnormalities. These continually declining curves
show that the model’s training is proceeding smoothly and steadily, indicat-
ing a strong learning process without substantial disturbances.

• Slight Validation Rise: Validation loss shows an initial fall, followed by a

plateau or modest increase. This occurrence might be linked to overfitting,
a typical problem in machine learning in which the model concentrates too
much on the complexities of the training data, resulting in poor general-
ization to new data. The modest increase in validation loss shows that the
overfitting issue must be addressed in order to get better performance on
new, previously unknown datasets.

Training Losses
• Box Loss: Beginning at 0.08, it gradually drops to roughly 0.03, indicating
an increase in bounding box prediction accuracy.

• Object Loss: Starts at 0.024 and falls to roughly 0.014, suggesting improved
object recognition capabilities with time.

• Class Loss: Begins at 0.04 and decreases to around 0.01, indicating im-
proved item categorization accuracy as training continues.

Validation Losses
• Overall Loss: Initially higher than training loss, but gradually lowers until
leveling out or slightly rising later.

• Object Loss: Follows a similar pattern as overall validation loss.

• Class Loss: Stays generally constant throughout.

Metrics:
• Precision and Recall: In terms of measurements, precision and recall levels
throughout training sessions are constantly around 0.7. This level implies
that the model strikes a respectable balance between producing correct de-
tections and reducing the incidence of false positives, demonstrating the
training process’s success in maintaining a high degree of performance.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 42

• mAP: The model’s mean Average Precision (mAP) is around 0.75, which
remains consistent within the 0.5:0.95 limits. This performance demon-
strates the model’s ability to properly recognize items across multiple thresh-
olds, as well as its efficacy in capturing a diverse range of objects with con-
stant precision.

4.2.4 Precision-Recall Curve:

A precision-recall curve is a powerful visual depiction that illustrates the delicate
balance between precision, which aims to minimize false positives, and recall,
which seeks to minimize false negatives, in the context of a binary classification
model. Specifically within object detection tasks, this curve serves as a valuable
tool to assess how effectively the model is performing across different classes of
objects that it has been specifically trained to identify and locate.

Figure 4.4: PR curve

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 43

Interpretation:
In the context of interpreting the curve in Fig 4.4, it’s important to understand the
information conveyed by the x-axis and y-axis. The x-axis, representing recall,
gives insights into the percentage of true positive detections relative to the actual
positive cases in the dataset. On the other hand, the y-axis, depicting precision,
offers a view of the accuracy in identifying true positive detections among all
the model’s positive predictions. These axes together provide a comprehensive
understanding of the model’s performance.

• Head and shoulders top/bottom: These curves appear to have the best accu-
racy and recall values, indicating that the model detects these items with few
false positives or negatives. Their precisions are around 0.85, which means
that 85% of the detections for these classes are true positives, and their re-
calls are near to 1.0, suggesting that the model discovers the majority of the
real objects in the dataset.

• M Head: This curve also implies strong performance, with an accuracy of

roughly 0.82 and a recall of about 0.8.

• Triangles have a lesser accuracy (about 0.6), which means there may be
more false positives than real positives. The recall is also lower than other
classes, around 0.6, indicating the model might miss some actual triangles.

• W Bottom: This curve has a precision of 0.78 and a recall of around 0.8,
indicating a fair balance of accuracy and recall.

• All classes (mAP at 0.5): This figure, about 0.77, indicates the model’s
mean average accuracy (mAP) at an intersection-over-union (IoU) thresh-
old of 0.5. The IoU metric measures how well a projected bounding box
overlaps with the ground truth bounding box for an item. A 0.5 IoU thresh-
old indicates that at least half of the predicted bounding box’s area must
coincide with the ground truth box for a correct detection. The mAP mea-
sure aggregates precision-recall performance across all classes at various
IoU levels, resulting in a single score for evaluating the model’s overall de-
tection capacity.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 44

Figure 4.5: Precision Curve

4.2.5 Precision Curve and Recall curve

Precision Confidence Curve:
• Curve Trend: The curve starts at a high precision value (around 0.85) for
low confidence thresholds and gradually decreases as the confidence thresh-
old increases. This suggests that the model is more precise (fewer false pos-
itives) when its confidence in detections is lower, but precision drops as the
model becomes more selective (higher confidence thresholds).

• Confidence Range: The x-axis covers a confidence range of 0.2 to 1.0. This
indicates that the model considers detections across a wide range of confi-
dence levels. No Separate

• Head and Shoulders Curves: Notably high precision and recall values, sug-
gesting excellent model capability in identifying these patterns can be seen
in Fig 4.5.

• Triangle Curve: Lower scores indicate potential for refining model training
or adjusting confidence thresholds to improve triangle detection accuracy.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 45

• mAP of 1.00: While impressive, it’s achieved at a very high confidence

threshold (0.977), potentially missing some true patterns. Examining model
performance across broader confidence ranges is crucial.

Figure 4.6: Recall Confidence Curve

Recall Confidence Curve:

• Head and Shoulders (top/bottom): Maintain good recall even at greater con-
fidence levels, demonstrating exceptional recognition of these patterns.
• M Head: Displays good performance with a steady reduction in recall as
confidence grows.
• Triangle: Has a lower recall curve, which drops quicker as confidence in-
creases. This shows that the model struggle to identify triangles with high
confidence.
• W Bottom: Has a reasonable recall curve that strikes a balance between
sustaining recollection and boosting confidence.The recall confidence curve
can be seen in Fig 4.6.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 46

4.3 Prediction:

Figure 4.7: chart

The sophisticated model, which has undergone rigorous training on an exten-

sive dataset consisting of diverse price pattern examples accompanied by bound-
ing boxes, is now proficient in scrutinizing the chart for these intricate patterns.
Through a meticulous process, the model is able to meticulously recognize spe-
cific price movements within the chart and effectively highlight them by demar-
cating with bounding boxes. These distinct boxes serve as a visual aid on the
chart, enclosing the pertinent sections that epitomize the identified price patterns.
This approach not only enhances the clarity of the identified patterns but also en-
ables one to swiftly discern the precise locations where these formations appear
on the chart. Such visualization plays a crucial role in facilitating a comprehen-
sive understanding of the chart dynamics, aiding in robust technical analysis, and
potentially illuminating strategic insights for making well-informed trading deci-
sions. Ultimately, this advanced tool empowers users to leverage the visual repre-
sentations provided by the bounding boxes to deepen their analysis and navigate
the complexities of market trends with enhanced precision and confidence.When
an candlestick chart Fig 4.7 is fed into the model it gives us an chart with detected
pattern using bounding boxes Figure 4.8.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 47

Figure 4.8: predicted

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 48

Chapter 5

Summary and Future Work

5.1 Summary:
In the dynamic world of the financial market, technical analysis serves as a critical
tool for traders to pinpoint key levels and strategic trade opportunities, facilitat-
ing optimal entry and exit points. Within this landscape, the deep learning model
emerges as a valuable resource, providing substantial assistance and guidance.
By integrating both technical analysis and the innovative yolo model, traders can
enhance their decision-making processes, harnessing the power of data-driven in-
sights to capitalize on market movements and trends. With its ability to identify
potential trades with precision and efficiency, the yolo model empowers traders
to navigate the complexities of the financial market with greater confidence and
success. . This YOLOv5 model, with its cutting-edge capabilities, shows signif-
icant potential in revolutionizing the process of price pattern detection within the
financial market. By leveraging its advanced algorithms and being trained on ex-
tensive historical data, this technology can efficiently detect intricate price action
patterns on charts. Upon detection, the model not only pinpoints these patterns
but also conveniently outlines them with precise bounding boxes, creating a user-
friendly visual aid directly on the chart. The implementation of this model brings
tremendous advantages to traders at all levels: novice traders benefit from sim-
plified pattern recognition, while seasoned traders save precious time that would
otherwise be spent scrutinizing charts manually. Picture the convenience of hav-
ing the YOLOv5 model instantaneously recognize potential trading opportunities,
enabling traders to redirect their focus towards more in-depth analysis and mak-
ing well-informed decisions. With its potential to streamline and enhance trading

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 49

strategies, this technology stands as a beacon of innovation that could potentially

transform how traders navigate the complexities of the financial market landscape.

5.2 Future Scope:

In this study, just a few price action patterns were used. It is crucial to note
that there are various more patterns used by market traders that may be useful to
include in the research. By increasing the dataset size, the model may be trained
more effectively, boosting the accuracy of the findings. Furthermore, there are a
variety of sophisticated variants of the YOLO (You Only Look Once) algorithm
available, which might significantly improve detection capabilities. It would be
beneficial to investigate these different alternatives in order to make the detecting
procedure more efficient and precise. By incorporating a broader range of price
action patterns and increasing the dataset size, the research findings are likely to
be more comprehensive and insightful. Additionally, experimenting with different
versions of the YOLO algorithm can open up new possibilities for improving the
overall detection system.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
 Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 50

Appendix A

Code Used

1 # Clone the repository from the GitHub link provided

2 !git clone https://github.com/ultralytics/yolov5
3

5 # Change the current directory to the cloned

repository
6 %cd yolov5
7

8 # Install the required dependencies

9 listed in the requirements.txt file
10 %pip install -qr requirements.txt # Installing
dependencies
11

12 # Additionally, install the roboflow package quietly

13 %pip install -q roboflow
14

15

16 # Import necessary libraries

17 import torch
18 import os
19 from IPython.display import Image, clear_output
20 # for displaying images
21

22 # Print setup information including Torch

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
 Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 51

23 version and GPU availability

24 print(f"Setup complete. Using torch {torch.__version__
}
25 ({torch.cuda.get_device_properties(0).
26 name if torch.cuda.is_available() else ’CPU’})")
Listing A.1: Cloning Yolov5 repo

1 # Import Roboflow library and download

2 dataset
3 from roboflow import Roboflow
4

5 # Initialize Roboflow with API key and access the

6 project and dataset
7 rf = Roboflow(api_key="76GCjbrULTdFgDoS5Gc0")
8 project = rf.workspace("mod5gen20-cnkzt").
9 project("stocks-2ulc2")
10 dataset = project.version(1).download("yolov5")
Listing A.2: Importing Dataset

1 # Set up environment variable for dataset directory

2 os.environ["DATASET_DIRECTORY"] = "Dl/repo/contents/
datasets"
Listing A.3: set up environment

2 # Train the model using specified parameters and

dataset
3 !python train.py --img 416 --batch 16 --epochs 100
4 --data {dataset.location}/data.yaml
5 --weights yolov5s.pt --cache
Listing A.4: Training

1 # Start TensorBoard to monitor training progress

2 %load_ext tensorboard
3 %tensorboard --logdir runs
Listing A.5: Results of training

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
 Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 52

1 # Test the trained model using detect.py script

2 !python detect.py --weights runs/train/exp/
3 weights/best.pt
4 --img 416 --conf 0.1 --source
5 {dataset.location}/test/images
Listing A.6: Testing

1 # Display inference results on all test images

2 import glob
3 from IPython.display import Image, display
4

5 for imageName in glob.glob(’/content/yolov5

6 /runs/detect/exp/*.jpg’): # assuming JPG
7 display(Image(filename=imageName))
8 print("\n")
Listing A.7: Inference of Testing

1 # Perform inference on a new image named "cams.png"

2 located in the "/content" directory
3 new_image_path = "/content/cams.png"
4 !python detect.py --source $/content
5 /cams.png --weights /content
6 /yolov5/runs/train/exp/weights/best.pt
7

8 # Display the annotated image

9 annotated_image_path = os.path.join("/content"
10 , os.path.basename
11 ("/content/cams.png"))
12 display(Image(filename=annotated_image_path))
Listing A.8: New Detection

1 # Perform object detection on an image and

2 display the annotated result
3 from PIL import Image
4

5 results = model(’/content/cams.png’)

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
 Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 53

6 # Perform object detection on an image

7

8 # Iterate through each detection resul

9 t in the list and show the annotated image
10 for result in results:
11 result.show()
Listing A.9: Detection Result

1 # Export the model’s weights for future use

2 from google.colab import files
3 files.download(’./runs’)
Listing A.10: Saving weights

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 54

Bibliography

Birogul, S., Temür, G. & Kose, U. (2020), ‘Yolo object recognition algorithm and
“buy-sell decision” model over 2d candlestick charts’, IEEE access 8, 91894–
91915.
Chen, J.-H., Chen, S. Y.-C., Tsai, Y.-C. & Shur, C.-S. (2020), ‘Explainable deep
convolutional candlestick learner’, arXiv preprint arXiv:2001.02767 .
Chen, J.-H. & Tsai, Y.-C. (2020), ‘Encoding candlesticks as images for pat-
tern classification using convolutional neural networks’, Financial Innovation
6(1), 26.
Grimes, A. (2012), The art and science of technical analysis: market structure,
price action, and trading strategies, Vol. 544, John Wiley & Sons.
Horvat, M. & Gledec, G. (2022), A comparative study of yolov5 models per-
formance for image localization and classification, in ‘33rd Central European
Conference on Information and Intelligent Systems (CECIIS)’, p. 349.
Huang, X., Wang, X., Lv, W., Bai, X., Long, X., Deng, K., Dang, Q., Han, S.,
Liu, Q., Hu, X. et al. (2021), ‘Pp-yolov2: A practical object detector’, arXiv
preprint arXiv:2104.10419 .
Kim, J.-H., Kim, N., Park, Y. W. & Won, C. S. (2022), ‘Object detection and
classification based on yolo-v5 with improved maritime dataset’, Journal of
Marine Science and Engineering 10(3), 377.
Ramadhan, A., Palupi, I. & Wahyudi, B. A. (2022), ‘Candlestick patterns recogni-
tion using cnn-lstm model to predict financial trading position in stock market’,
Journal of Computer System and Informatics (JoSYC) 3(4), 339–347.
Sim, H. S., Kim, H. I., Ahn, J. J. et al. (2019), ‘Is deep learning for image recog-
nition applicable to stock market prediction?’, Complexity 2019.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus
Detecting and Analysing Candlestick Pattern in Stock Market Using Deep Learning 55

Standard, B. (2023), ‘Spreading the financial literacy wave across india - national
finance olympiad 2023’, Business Standard Press Release .
URL: https://www.business-standard.com/content/press-releases-
ani/spreading-the-financial-literacy-wave-across-india-national-finance-
olympiad-2023-1231221007321 .html

Yang, G., Feng, W., Jin, J., Lei, Q., Li, X., Gui, G. & Wang, W. (2020), Face mask
recognition system with yolov5 based on image recognition, in ‘2020 IEEE 6th
International Conference on Computer and Communications (ICCC)’, IEEE,
pp. 1398–1404.

Yao, J., Qi, J., Zhang, J., Shao, H., Yang, J. & Li, X. (2021), ‘A real-time detection
algorithm for kiwifruit defects based on yolov5’, Electronics 10(14), 1711.

Divison of Mathematics, School of Advanced Sciences, Vellore Institute of Technology , Chennai Campus