2018 IEEE Conference on Multimedia Information Processing and Retrieval

Deep Learning of Path-based Tree Classifiers for Large-Scale Plant Species

Identification

Haixi Zhang∗ , Guiqing He∗ , Jinye Peng∗ , Zhenzhong Kuang† , Jianping Fan‡
∗ School of Electrical and Information
Northwestern Polytrchnical University, Xi’an, Shanxi, China
Email:guiqing [email protected]
† School of Computer Science and Technology
Hangzhou Dianzi University, Zhejiang, China
Email: [email protected]
‡ School of Computer Science
University of North Carolina at Charlotte, Charlotte, USA
Email: [email protected]

Abstract tours the gardens in the community, the question may arise:
In this paper, a deep learning framework is devel- what is the name of that plant? Intrigued by a particular
oped to enable path-based tree classifier training for herbaceous or woody plant, the person may wonder when
supporting large-scale plant species recognition, where it blooms, its flower color, size and bloom time, cultural
a deep neural network and a tree classifier are jointly
requirements, and its commercial availability. Will it attract
trained in an end-to-end fashion. First, a two-layer plant
taxonomy is constructed to organize large numbers of pollinators, such as insects and birds? Is it an introduced
plant species and their genus hierarchically in a coarse- (exotic) species or is it native to the region or state? Is the
to-fine fashion. Second, a deep learning framework is plant on the state or federal invasive species list? In other
developed to enable path-based tree classifier training, situations people will need to identify the plant species to
where a tree classifier over the plant taxonomy is
determine if internal or external exposure would cause harm.
used to replace the flat softmax layer in traditional
deep CNNs. A path-based error function is defined Image-based plant recognition has been a really popular
to optimize the joint process for learning deep CNN research area recently[2–8]. For large-scale plant species
and tree classifier, where back propagation is used to identification, some of these plant species may have strong
update both the classifier parameters and the network inter-species visual similarities, thus it is unreasonable to
weights simultaneously. We have also constructed a
ignore such inter-species visual similarities completely and
large-scale plant database of Orchid family for algorithm
evaluation. Our experimental results have demonstrated learn their inter-related classifiers independently. In most
that our path-based deep learning algorithm can achieve existing deep learning schemes [18–20], softmax is used
very competitive results on both the accuracy rates and and the inter-task correlations are completely ignored, as
the computational efficiency for large-scale plant species a result, the process for learning the deep CNNs may be
recognition.
pushed away from the global optimum because the gradients
Keywords-path based; tree classifier; plant speices recogni-
of the objective function are not uniform for all the object
tion; plant taxonomy
classes and such learning process may distract on discerning
I. I NTRODUCTION some object classes that are hard to be discriminated. In
order to leverage traditional deep CNNs[18, 20] for large-
Plants are enormously important to human welfare be-
scale plant species identification application, one potential
cause they are a source of food, clothing, housing materials,
solution for eliminating such distraction is to group the
medicines, and more besides. In the past, plant species
visually-similar plant species together(e.g., such visually-
identification was the sole domain of taxonomists, botanists,
similar plant species in the same group will have similar
and other professionals who identified the plants of interest
learning difficulties and the gradients of their joint objective
by comparing them with previously collected specimens or
function will be more uniform), thus it is very attractive
by using books or identification manuals [1]. Obviously,
to invest whether the inter-task correlations (inter-species
learning about unknown plants is also an exciting venture
similarities) can be leveraged to improve the learning of
for amateur gardeners and outdoor enthusiasts. When anyone
the deep CNNs for large-scale plant species recognition
hikes, visits public gardens, wanders around campus, or just
application.
*Guiqing He is the corresponding author(email: guiqing [email protected]. Another critical issue for large-scale plant species recog-
cn) nition is how to reduce huge computational cost at test time.

0-7695-6354-6/18/$31.00 ©2018 IEEE 25

DOI 10.1109/MIPR.2018.00013
Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.
The computational cost for the flat approach such as N-way The hierarchical approach is very attractive for supporting
softmax grows linearly with the number of plant species large-scale visual recognition, and the problem of hierarchi-
to be recognized. Thus such flat approach may become cal classifier training has extensively been investigated in
computationally expensive. One way to address this issue the past decades [24–27]. Most existing approaches leverage
is to integrate a tree structure to organize large numbers hand-crafted features for hierarchical classifier training, thus
of plant species hierarchically in a coarse-to-fine fashion, it is very attractive to invest how deep features [18–22] can
and such hierarchical approach can significantly reduce be leveraged to improve hierarchical classifier training.
the computational complexity at test time. In addition, the One way to reduce the computational cost for large-scale
underlying tree structure may provide an explicit solution visual recognition is to organize large numbers of atomic
for identifying the inter-related learning tasks automatically: object classes hierarchically in a tree structure. Such tree
the classifier training tasks for the sibling plant species under structures can be categorized into three types: (a) hierar-
the same genus are typically inter-related. chical tree of binary classifiers [24]; (b) semantic ontology
Based on these observations, a path-based learning algo- [28, 29]); and (c) label tree or visual hierarchy [25–27]. The
rithm is developed to enable joint learning of deep CNNs hierarchical tree of binary classifiers may seriously suffer
(for feature extraction) and tree classifier (over plant taxon- from huge computational cost for determining the optimal
omy) in an end-to-end fashion. This paper is organized as binary classifier for each non-leaf node.
follows: In section 2, we present a brief review of related III. O RCHID 2608 DATASET
work; In section 3, we briefly introduce our new dataset:
Because the available plant datasets are usually not big
Orchid 2608; In section 4, a hierarchical path-based learning
enough for our task, we have constructed a new dataset
algorithm is developed to enable joint learning of deep
which contains nearly 2 million orchid flower images. All
CNNs and tree classifier for supporting large scale plant
the images in this dataset are automatically crawled from
species identification. Our experimental results are given in
search engines such like Flickr, Google and Bing through
section 5 and we conclude this paper in section 6.
keywords for plant species interpretation. These keywords
are collected from a professional plant taxonomy website
II. R ELATED W ORK called ”The Plant List”. We first crawl all the species names
In this section, we briefly review the most relevant re- under Orchidaceae from the ”The Plant List”, then we use
searches on: (1) image-based plant identification [2–9]; and a python-based crawler to get images from different search
(2) tree classifier training [24–27]. engines through these keywords of Orchid species. We can
Image-based plant identification has been an area of grow- get enough images for only 2608 Orchid species among all
ing interest [11, 12, 14]. The typical approach is to identify 20,000 orchid species. After that, a clustering-based rubbish
the plants with leaves, such as the recent smartphone- image filter algorithm is used to filter out the noisy images.
based plant identification system called Leafsnap [3], which Finally all images are manually cropped and labeled.
involves digitizing the plant image of a solitary flattened leaf Compared with the two most popular existing plant
and then comparing it with a database of previously scanned datasets: Flower 101[9] and PlantClef 2016[7], our dataset
leaves from some known plant species. By comparing the contains more speices (2608) and all these species are under
test plant image with the template images in the database, Orchid family. So our dataset(Orchid 2608) is much bigger
Leafsnap [3] does not pay enough attentions to both large and more specific than Flower 101 and PlatClef 2016, which
intra-species visual variations and strong inter-species visual means Orchid 2608 is more suitable for evaluating our
similarities which are typical in the botanical domain [13]. proposed algorithm for large-scale fine-grained plant species
To overcome this impediment, in this paper, a deep path- identification.
based learning algorithm is developed to leverage large- IV. PATH - BASED D EEP L EARNING A LGORITHM
scale plant images to jointly learn more representative deep
networks and more discriminative multi-task softmax in an
end-to-end fashion. iPlant [10] is a project designed to
support the computational needs of the research community
and facilitate progress toward solutions of major problems
in plant science. Pl@ntNet [11](idenitfy.plannet-project.org)
is an iOS/Android app that uses visualization software
to recognize 11,000 plant species when the images offer Figure 1. The flow chart for path-based deep network with plant taxonomy
enough clarity for comparison with the species in their based tree classifier
reference database. The project called “What Tree is That?”
provides a step-by-step guide to identify more than 250 trees As shown in figure(1), a path-based hierarchical deep
[12]. learning algorithm is developed for jointly training feature

26

Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.
extraction network and structure tree classifier over fixed B. Path-based Prediction
plant taxonomy. Our path based deep learning algorithm Given an input image x, we got the final prediction
have following key contributions: (1) leveraging the plant through searching max probability among all paths from
taxonomy help organizing large numbers of plant species root node to leaf nodes. The probability of a single path is
and their most relevant genus in a coarse-to-fine fashion; determined by the genu probability(PG) and the inner-group
(2) using a tree classifier over the plant taxonomy to replace species probability(PI). The path probability is calculated as:
the N-way flat softmax classifier in deep network; (3) using
a path-base back propagation method and a path-based loss P (x) = max pG Gg
g ∗ max ps (x) (1)
gG sGg
function to address the issue of inter-level error propagation;
(4) enabling joint learning of deep network (for feature and PG and PI is defined as
extraction) and tree classifier in an end-to-end fashion.  
Compared with traditional approaches, our path-based P G(x|lg ) = max pG
g (x) = l (2)
gG
learning algorithm has following improvements: (1) A tree  
classifier over the plant taxonomy is used to replace the P I(x|lg , ls ) = max pGg
s (x) = l (3)
N-way flat softmax classifier, and the inter-species visual sGg
similarities are leveraged for training the inter-related clas-
where pG Gg
g and ps represent the probability output at genu
sifiers for the sibling plant species under the same genus.
and specie level for an input image, lg and ls represent genu
In this paper, a two-layer plant taxonomy (genus layer and
and specie groundtruth label respectively.
species layer) is constructed to organize large numbers of
plant species and their most relevant genus hierarchically in C. Path-based Training
a coarse-to-fine fashion. (2) A path-based back propagation
A bottom-up approach is developed to jointly train the
method is used to achieve joint learning of deep network and
deep feature extraction network and the tree classifier over
tree classifier in an end-to-end fashion, so that the critical
fixed plant taxonomy. It is worth noticing that we consider
issue of inter-error propagation can be addressed effectively.
the prediction process as a path travel from genu node to
specie node. Similarly, we consider every single path as
A. Two-layer Tree Structure an entity during training process, which means parameters
on the same path should be updated at same time during
backpropagation. Our algorithm is used to learn the path
classifier ft (x) = (Wg + W s)T x + b, where sSg .
However, when misclassification happens on high-level
node, it is impossible to make right prediction on low-
level node. This means error will propagate from high-
level node to low-level node. Due to these observations, we
proposed a path-based training method which only update
parameters on paths which are relevant to prediction during
each iteration. As shown in figure(3), when right prediction
is made on genu level, only one single path need to be
updated because the prediction result on genu layer will not
affect the prediciton result on specie layer. On the contrary,
when misclassification happens on genu layer, prediction on
specie layer will be affected. So all specie nodes relevant
to the misclassified genu node needed to be updated. The
Figure 2. Two layer orchid ontology path-based objective function is formulated as:

As shown in figure(2), we construct a fixed two-layer L = min −log((Wg + Ws )T x + b) (4)
tree classifier for Orchid 2608. It is easy to understand gG,sSg

that we can get a fixed tree structure from plant taxonomy where Sg represents the relevant nodes with predicted
which can perfectly organize the large number of plant genu node.
species in a coarse to fine fashion. The high-level layer So the partial difference(gradients) of L are determined
includes 158 nodes represent 158 genus, and the low-level as:
layer include 2608 speice nodes. This fixed tree taxonomy
perfectly provide a good environment to determine the inter- ∂L ∂L ∂z(x)
= ∗ ∗ Ψ(g) (5)
species relations. ∂wg ∂z(x) ∂wg

27

Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.
 ∂L ∂L ∂z(x) MTCDCNN[39]). Our comparison experiment focus on as-
= ∗ ∗ Ψ(s) (6)
∂ws ∂z(x) ∂wg sessing: (1) whether our path-based deep network can pro-
vide a better solution for jointly learning the deep network
wherez(x) = sof tmax(f (x)), Ψ(g) and Ψ(s) are used and the tree classifier; (2) whether our path-based deep
to determine nodes need to be update. Both Ψ(g) and Ψ(s) learning algorithm can achieve higher performance on large-
are constructed based on our plant taxonomy. scale plant identification.
The corresponding gradients for the joint objective func-
tion are backpropagated through the deep networks at the GA IA PA
Caffenet - - 69.69
species level to fine-tune the weights.
MTCDCNN 90.35 79.63 71.95
PathNet 89.86 80.44 72.28
HDCNN - - 71.12
Table I
C LASSIFICATION ACCURACY OF OUR PATH - BASED DEEPLEARNING
ALGORITHM ON O RCHID 2608 IMAGE SET

As shown in Table 1, one can see that our path based

approach get the best performance(72.28% on PA). Besides,
both our method and MTCDCNN [39] get better perfor-
mance than the Caffenet[18, 19], provides tree classifier
can help improve performance on large-scale fine-grained
classification task. The reason why tree classifier works
better lies in the tree classifier can provide a better en-
vironment for determining the inter-related learning tasks.
Because each groups contains a smaller number of plant
speices, the gradient of loss function are much more uniform
and our algorithm can obtain the global optium effectively.
By explicitly levageing the inter-class visiual similarities,
our path-based algorithm can achieve better performace on
distinguishing the semantically related object classes in the
same group, which are usually hard to be distinguished.
To further display the classification results on each level of
the plant taxonomy, we plot the average accuracy curves on
Figure 3. Path-based backpropagation flow chart
both genu level and the specie level in Figure 4a. It is easy
to see that, both our method and MTCDCNN can achieve
really high accuracy on genu-level. Also the inner-genu
V. E XPERIMENT R ESULTS accuracy rates(for distinguish fine-grained species within
We have evaluated our path-based deep learning algorithm the same genu) are shown in Figure 4b. Besides, although
on Orchid 2608, which have 2608 fine-grained species, both our method and MTCDCNN performs well on genu-
categorized into 158 coarse genus by fixed plant taxonomy. level, we can see our method can get higher accuracy on
Orchid 2608 contains over 1,800,000 images for training final prediction. Compared with AlexNet and MTCDCNN,
and 180,000 images for testing. The impletation of our deep our path-based method has achieved competitive results.
network is based on Alexnet[18] with 8 layers. Given a two- One can observe that our path-based algorithm can achieve
layer plant taxonomy structure, different methods[18, 38, 39] commpetitive accuracy on group based plant recognition.
are used for jointly training of deep network and tree By grouping the semantic-related object classes with similar
classifier. In addition to the common used global flat ac- learning complexity into the same group, the gradients
curacy(FA), we also pay attention to the path-based accu- of their objective function are more uniform, our path-
racy(PA) which is determined by the group accuracy(GA) based algorithm can effectively achieve global optimum abd
and the inner group accuracy(IA), where IA represnets the result in more discriminative classifiers to distinguish such
accuracy rate for recognizing the fine-grained object classes semantically-related object classes more effectively.
within the same group. These experimental results have demonstrated that, by
To evaluate effectiveness of our path-based algorithm, categorizing large-number of fine-grained plant species into
we compared our method with two different baselines many coarse genus according to visual similarity, our plant
(Caffenet [18, 19] without inter-species relations and taxonomy based tree classifier can provide stronger dis-
criminative power for determining the inter-species learning

28

Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.
 R EFERENCES

[1] R. S. Cowan, M.W. Chase, W. J. Kress, V. Savolainen,

“300,000 species to identify: problems, progress, and
prospects in DNA barcoding of land plants”, Taxon,
2006.
[2] P.N. Belhumeur, D. Chen, S. Feiner, D. W. Jacobs,
W. J. Kress, H. Ling, I. Lopez, R. Ramamoorthi, S.
Sheorey, S. White, L. Zhang, “Searching the worlds
herbaria: A system for visual identification of plant
species”, ECCV, pp. 116-129, 2008.
(a) Genu Accuracy [3] N. Kumar, P.N. Belhumeur, A. Biswas, D. W. Jacobs,
W. J. Kress, I. C. Lopez, J. V. Soares, ”Leafsnap: A
computer vision system for automatic plant species
identification”, ECCV, pp. 502-516, 2012.
[4] H. Goeau, P. Bonnet, A. Joly, I. Yahiaoui, D.
Barthelemy, N. Boujemaa, J. Molino, “The Image-
CLEF 2012 plant identification tasks”, Technical Re-
port, INRIA, 2012.
[5] M. Chopra, “TreeID: An image recognition system
for plant species identification”, Tech. report, Stanford
University.
[6] J.D. Wegner, S. Branson, D. Hall, K. Schindler, P.
Perona, “Cataloging public objects using aerial and
street-level images urban trees,” IEEE CVPR, 2016.
[7] T. Saitoh, T. Kaneko, “Automatic recognition of wild
flowers”, IEEE ICPR, 2000.
(b) Specie Accuracy [8] M.E. Nilsback and A. Zisserman, “A visual vocabulary
Figure 4. Comparison on the GA and PA curves between path-based for flower classification”, IEEE CVPR, 2006.
method and MTCDCNN [9] M.E. Nilsback and A. Zisserman, “Delving deeper into
the whorl of flower segmentation”, Image and Vision
Computing, vol.28, no.6, pp.1049-1062, 2010.
missions and distinguish fine-grained plant species, which [10] S. A. Goff, et al., “The iPlant Collaborative: Cyber-
means our path-based algorithm may not be disturbed on infrastructure for Plant Biology”, Frontiers in Plant
such hard species which hard to be discriminated. Besides, Science, vol. 2, 2011.
according to the performance between our method and [11] Pl@ntNet: idenitfy.plannet-project.org
MTCDCNN, the error propagation between different levels [12] What Tree is That?: www.arborday.org/trees/whatTree
can be declined in our path-based training progress. [13] Tera Botanica, www.tela-botanica.org
VI. C ONCLUSION [14] J. Cope, D. Corney, J. Clark, P. Remagnino, P. Wilkin,
“Plant species identification using digital morphomet-
A path-based deep learning algorithm is developed to
rics: A review”, Expert Systems with Applications,
jointly training the deep neural network and the tree classifier
vol.39, pp.7562-7573, 2012.
over a fixed plant taxonomy. We also construct a big
[15] A.R. Sfar, F. Palaiseau, N. Boujemaa, D. Geman, “Van-
plant dataset suit for large-scale plant identification task.
tage feature frames for fine-grained categorization”,
Experiments on our new datasets provide that our path-based
CVPR, 2013.
deep network can achieve very competitive results on both
[16] R. Farrell, O. Oza, N. Zhang, V.I. Morariu, T. Dar-
the accuracy rates and the computational efficiency for large-
rell, L.S. Davis, “Birdlets: Subordinate categorization
scale plant identification.
using volumetric primitives and pose-normalized ap-
ACKNOWLEDGMENT pearance”, IEEE ICCV, pp. 161-168, 2011.
This work is partly supported by National Natural Science [17] G. Martinez-Munoz, W. Zhang, N. Payet, S. Todorovic,
Foundation of China(No.61772161 and No.61402368), the N. Larios, A. Yamamuro, D. Lytle, A. Moldenke, E.
National Aerospace Science and Technology Foundataion, Mortensen, R. Paasch, L. Shapiro, T.G. Dietterich,
the Fundmental Research Fund for the Central Unviersities. “Dictionary-free categorization of very similar objects
via stacked evidence tree”, IEEE CVPR, 2009.

29

Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.
[18] A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet [37] J. Wang, X. Shen, W. Pan, On large margin hierarchical
classification with deep convolutional neural networks, classification with multiple paths, Journal of the Amer-
NIPS, 2012. ican Statistical Association, vol.104, no.487, 2009.
[19] J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, [38] Yan, Zhicheng, et al. HD-CNN: hierarchical deep
E. Tzeng, T. Darrell, DeCAF: A deep convolutional convolutional neural networks for large scale visual
activation feature for generic visual recognition, ICML, recognition. Proceedings of the IEEE International
2014. Conference on Computer Vision. 2015.
[20] K. Simonyan, A. Zisserman, “Very deep convolutional [39] Kuang Z, Li Z, Zhao T, et al. Deep Multi-task Learning
networks for large-scale image recognition”, ICLR, for Large-Scale Image Classification[C]//Multimedia
2015. Big Data (BigMM), 2017 IEEE Third International
[21] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Conference on. IEEE, 2017: 310-317.
Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, [40] Kontschieder, Peter, et al. ”Deep neural decision
“Going deeper with convolutions”, IEEE CVPR 2015. forests.” Proceedings of the IEEE International Con-
[22] Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, “Gradient- ference on Computer Vision. 2015.
based learning applied to document recognition”, [41] Wang, Jiang, et al. ”Cnn-rnn: A unified framework
Proc.IEEE, 1998. for multi-label image classification.” Proceedings of
[23] T. Gao, D. Koller, Discriminative learning of re- the IEEE Conference on Computer Vision and Pattern
laxed hierarchy for large-scale visual recognition, IEEE Recognition. 2016.
ICCV, pp. 2072-2079, 2011.
[24] G. Griffin, P. Perona, Learning and using taxonomies
for fast visual categorization, IEEE CVPR, 2008.
[25] S. Bengio, J. Weston, D. Grangier, Label embedding
trees for large multi-class tasks, NIPS, 2010.
[26] J. Fan, N. Zhou, J. Peng, L. Gao, “Hierarchical learning
of tree classifiers for large-scale plant species identifi-
cation”, IEEE Trans. Image Processing, vol. 24, no.11,
pp.4172-4184, 2015.
[27] J. Fan, Y. Gao, H. Luo, “Integrating concept ontology
and multi-task learning to achieve more effective clas-
sifier training for multi-level image annotation”, IEEE
Trans. on Image Processing, vol. 17, no.3, pp.407-426,
2008.
[28] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li,
“ImageNet: A large-scale hierarchical image database”,
IEEE CVPR, pp. 248-255, 2009.
[29] A. Zweig, D. Weinshall, Hierarchical regularization
cascade for joint learning, ICML, 2013.
[30] L. Cai, T. Hofmann, Hierarchical document categoriza-
tion with support vector machines, ACM CIKM, 2004.
[31] B. Shahbaba, R. Neal, Improving classification when
a class hierarchy is available using a hierarchy-based
prior, Bayesian Analysis, vol.2, pp.221-238, 2007.
[32] D. Koller, M. Sahami, Hierarchically classifying doc-
uments using very few words, ICML, 1997.
[33] A.K. McCallum, R. Rosenfeld, T. Mitchell, A.Y. Ng,
Improving text classification by shrinkage in a hierar-
chy of classes, ICML, 1998.
[34] O. Dekel, J. Keshet, Y. Singer, Large margin hierarchi-
cal classification, ICML, 2004.
[35] D. Zhou, L. Xiao, M. Wu, Hierarchical classification
via orthogonal transfer, ICML, 2011.
[36] M. Sun, W. Huang, S. Savarese, Find the best path: an
efficient and accurate classifier for image hierarchies,
IEEE ICCV, 2013.

30

Authorized licensed use limited to: UNIVERSITY OF JORDAN. Downloaded on October 17,2023 at 09:17:28 UTC from IEEE Xplore. Restrictions apply.