Association Rule Learning
Association Rule Learning
Association rule learning is a rule-based machine learning method for discovering interesting relations
between variables in large databases. It is intended to identify strong rules discovered in databases using
some measures of interestingness.[1]
Based on the concept of strong rules, Rakesh Agrawal, Tomasz Imieliński and Arun Swami[2] introduced
association rules for discovering regularities between products in large-scale transaction data recorded by
point-of-sale (POS) systems in supermarkets. For example, the rule
found in the sales data of a supermarket would indicate that if a customer buys onions and potatoes
together, they are likely to also buy hamburger meat. Such information can be used as the basis for
decisions about marketing activities such as, e.g., promotional pricing or product placements.
In addition to the above example from market basket analysis association rules are employed today in
many application areas including Web usage mining, intrusion detection, continuous production, and
bioinformatics. In contrast with sequence mining, association rule learning typically does not consider
the order of items either within a transaction or across transactions.
Contents
Definition
Useful Concepts
Support
Confidence
Lift
Conviction
Alternative measures of interestingness
Process
History
Statistically sound associations
Algorithms
Apriori algorithm
Eclat algorithm
FP-growth algorithm
Others
ASSOC
OPUS search
Lore
Other types of association rule mining
See also
References
Bibliographies
Definition
Following the original definition by Agrawal,
Example database with 5 transactions and 5 items
Imieliński, Swami[2] the problem of
association rule mining is defined as: transaction ID milk bread butter beer diapers
1 1 1 0 0 0
Let be a set of binary
2 0 0 1 0 0
attributes called items.
3 0 0 0 1 1
Let be a set of 4 1 1 1 0 0
transactions called the database.
5 0 1 0 0 0
, where .
In Agrawal, Imieliński, Swami[2] a rule is defined only between a set and a single item, for
.
Every rule is composed by two different sets of items, also known as itemsets, and , where is
called antecedent or left-hand-side (LHS) and consequent or right-hand-side (RHS).
To illustrate the concepts, we use a small example from the supermarket domain. The set of items is
and in the table is shown a small database containing the
items, where, in each entry, the value 1 means the presence of the item in the corresponding transaction,
and the value 0 represents the absence of an item in that transaction.
An example rule for the supermarket could be meaning that if butter and
bread are bought, customers also buy milk.
Note: this example is extremely small. In practical applications, a rule needs a support of several hundred
transactions before it can be considered statistically significant, and datasets often contain thousands or
millions of transactions.
Useful Concepts
In order to select interesting rules from the set of all possible rules, constraints on various measures of
significance and interest are used. The best-known constraints are minimum thresholds on support and
confidence.
Support
Support is an indication of how frequently the itemset appears in the dataset.
The support of with respect to is defined as the proportion of transactions in the dataset which
contains the itemset .
Confidence
Confidence is an indication of how often the rule has been found to be true.
The confidence value of a rule, , with respect to a set of transactions , is the proportion of the
transactions that contains which also contains .
Note that means the support of the union of the items in X and Y. This is somewhat
confusing since we normally think in terms of probabilities of events and not sets of items. We can
rewrite as the probability , where and are the events that a
transaction contains itemset and , respectively. [4]
Lift
The lift of a rule is defined as:
or the ratio of the observed support to that expected if X and Y were independent.
If the rule had a lift of 1, it would imply that the probability of occurrence of the antecedent and that of
the consequent are independent of each other. When two events are independent of each other, no rule
can be drawn involving those two events.
If the lift is > 1, that lets us know the degree to which those two occurrences are dependent on one
another, and makes those rules potentially useful for predicting the consequent in future data sets.
If the lift is < 1, that lets us know the items are substitute to each other. This means that presence of one
item has negative effect on presence of other item and vice versa.
The value of lift is that it considers both the support of the rule and the overall data set.[3]
Conviction
All-confidence[7]
Collective strength[8]
Leverage[9]
Several more measures are presented and compared by Tan et al.[10] and by Hahsler.[4] Looking for
techniques that can model what the user has known (and using these models as interestingness measures)
is currently an active research trend under the name of "Subjective Interestingness."
Process
Association rules are usually required to satisfy a user-specified minimum support and a user-specified
minimum confidence at the same time. Association rule generation is usually split up into two separate
steps:
An early (circa 1989) use of minimum support and confidence to find all association rules is the Feature
Based Modeling framework, which found all rules with and greater than user
defined constraints.[17]
Algorithms
Many algorithms for generating association rules have been proposed.
Some well-known algorithms are Apriori, Eclat and FP-Growth, but they only do half the job, since they
are algorithms for mining frequent itemsets. Another step needs to be done after to generate rules from
frequent itemsets found in a database.
Apriori algorithm
Apriori[13] uses a breadth-first search strategy to count the support of itemsets and uses a candidate
generation function which exploits the downward closure property of support.
Eclat algorithm
Eclat[14] (alt. ECLAT, stands for Equivalence Class Transformation) is a depth-first search algorithm
based on set intersection. It is suitable for both sequential as well as parallel execution with locality-
enhancing properties.[20][21]
FP-growth algorithm
FP stands for frequent pattern.[22]
In the first pass, the algorithm counts the occurrences of items (attribute-value pairs) in the dataset of
transactions, and stores these counts in a 'header table'. In the second pass, it builds the FP-tree structure
by inserting transactions into a trie.
Items in each transaction have to be sorted by descending order of their frequency in the dataset before
being inserted so that the tree can be processed quickly. Items in each transaction that do not meet the
minimum support requirement are discarded. If many transactions share most frequent items, the FP-tree
provides high compression close to tree root.
Recursive processing of this compressed version of the main dataset grows frequent item sets directly,
instead of generating candidate items and testing them against the entire database (as in the apriori
algorithm).
Growth begins from the bottom of the header table i.e. the item with the smallest support by finding all
sorted transactions that end in that item. Call this item .
A new conditional tree is created which is the original FP-tree projected onto . The supports of all nodes
in the projected tree are re-counted with each node getting the sum of its children counts. Nodes (and
hence subtrees) that do not meet the minimum support are pruned. Recursive growth ends when no
individual items conditional on meet the minimum support threshold. The resulting paths from root to
will be frequent itemsets. After this step, processing continues with the next least-supported header
item of the original FP-tree.
Once the recursive process has completed, all frequent item sets will have been found, and association
rule creation begins.[23]
Others
ASSOC
The ASSOC procedure[24] is a GUHA method which mines for generalized association rules using fast
bitstrings operations. The association rules mined by this method are more general than those output by
apriori, for example "items" can be connected both with conjunction and disjunctions and the relation
between antecedent and consequent of the rule is not restricted to setting minimum support and
confidence as in apriori: an arbitrary combination of supported interest measures can be used.
OPUS search
OPUS is an efficient algorithm for rule discovery that, in contrast to most alternatives, does not require
either monotone or anti-monotone constraints such as minimum support.[25] Initially used to find rules
for a fixed consequent[25][26] it has subsequently been extended to find rules with any item as a
consequent.[27] OPUS search is the core technology in the popular Magnum Opus association discovery
system.
Lore
A famous story about association rule mining is the "beer and diaper" story. A purported survey of
behavior of supermarket shoppers discovered that customers (presumably young men) who buy diapers
tend also to buy beer. This anecdote became popular as an example of how unexpected association rules
might be found from everyday data. There are varying opinions as to how much of the story is true.[28]
Daniel Powers says:[28]
In 1992, Thomas Blischok, manager of a retail consulting group at Teradata, and his staff
prepared an analysis of 1.2 million market baskets from about 25 Osco Drug stores.
Database queries were developed to identify affinities. The analysis "did discover that
between 5:00 and 7:00 p.m. that consumers bought beer and diapers". Osco managers did
NOT exploit the beer and diapers relationship by moving the products closer together on the
shelves.
Contrast set learning is a form of associative learning. Contrast set learners use rules that differ
meaningfully in their distribution across subsets.[30][31]
Weighted class learning is another form of associative learning in which weight may be assigned to
classes to give focus to a particular issue of concern for the consumer of the data mining results.
High-order pattern discovery facilitate the capture of high-order (polythetic) patterns or event
associations that are intrinsic to complex real-world data. [32]
K-optimal pattern discovery provides an alternative to the standard approach to association rule
learning that requires that each pattern appear frequently in the data.
Approximate Frequent Itemset mining is a relaxed version of Frequent Itemset mining that allows
some of the items in some of the rows to be 0.[33]
Interval Data Association Rules e.g. partition the age into 5-year-increment ranged
Sequential pattern mining discovers subsequences that are common to more than minsup sequences in
a sequence database, where minsup is set by the user. A sequence is an ordered list of transactions.[34]
Subspace Clustering, a specific type of Clustering high-dimensional data, is in many variants also based
on the downward-closure property for specific clustering models.[35]
Warmr is shipped as part of the ACE data mining suite. It allows association rule learning for first order
relational rules.[36]
See also
Sequence mining
Production system (computer science)
Learning classifier system
Rule-based machine learning
References
1. Piatetsky-Shapiro, Gregory (1991), Discovery, analysis, and presentation of strong rules, in
Piatetsky-Shapiro, Gregory; and Frawley, William J.; eds., Knowledge Discovery in
Databases, AAAI/MIT Press, Cambridge, MA.
2. Agrawal, R.; Imieliński, T.; Swami, A. (1993). "Mining association rules between sets of
items in large databases". Proceedings of the 1993 ACM SIGMOD international conference
on Management of data - SIGMOD '93. p. 207. CiteSeerX 10.1.1.40.6984 (https://citeseerx.i
st.psu.edu/viewdoc/summary?doi=10.1.1.40.6984). doi:10.1145/170035.170072 (https://doi.
org/10.1145%2F170035.170072). ISBN 978-0897915922.
3. Hahsler, Michael (2005). "Introduction to arules – A computational environment for mining
association rules and frequent item sets" (https://mran.revolutionanalytics.com/web/package
s/arules/vignettes/arules.pdf) (PDF). Journal of Statistical Software.
4. Michael Hahsler (2015). A Probabilistic Comparison of Commonly Used Interest Measures
for Association Rules. http://michael.hahsler.net/research/association_rules/measures.html
5. Hipp, J.; Güntzer, U.; Nakhaeizadeh, G. (2000). "Algorithms for association rule mining --- a
general survey and comparison". ACM SIGKDD Explorations Newsletter. 2: 58–64.
CiteSeerX 10.1.1.38.5305 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.38.53
05). doi:10.1145/360402.360421 (https://doi.org/10.1145%2F360402.360421).
6. Brin, Sergey; Motwani, Rajeev; Ullman, Jeffrey D.; Tsur, Shalom (1997). "Dynamic itemset
counting and implication rules for market basket data". Proceedings of the 1997 ACM
SIGMOD international conference on Management of data - SIGMOD '97. pp. 255–264.
CiteSeerX 10.1.1.41.6476 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.64
76). doi:10.1145/253260.253325 (https://doi.org/10.1145%2F253260.253325). ISBN 978-
0897919111.
7. Omiecinski, E.R. (2003). "Alternative interest measures for mining associations in
databases". IEEE Transactions on Knowledge and Data Engineering. 15: 57–69.
CiteSeerX 10.1.1.329.5344 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.329.
5344). doi:10.1109/TKDE.2003.1161582 (https://doi.org/10.1109%2FTKDE.2003.1161582).
8. Aggarwal, Charu C.; Yu, Philip S. (1998). "A new framework for itemset generation".
Proceedings of the seventeenth ACM SIGACT-SIGMOD-SIGART symposium on Principles
of database systems - PODS '98. pp. 18–24. CiteSeerX 10.1.1.24.714 (https://citeseerx.ist.
psu.edu/viewdoc/summary?doi=10.1.1.24.714). doi:10.1145/275487.275490 (https://doi.or
g/10.1145%2F275487.275490). ISBN 978-0897919968.
9. Piatetsky-Shapiro, Gregory; Discovery, analysis, and presentation of strong rules,
Knowledge Discovery in Databases, 1991, pp. 229-248
10. Tan, Pang-Ning; Kumar, Vipin; Srivastava, Jaideep (2004). "Selecting the right objective
measure for association analysis". Information Systems. 29 (4): 293–313.
CiteSeerX 10.1.1.331.4740 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.331.
4740). doi:10.1016/S0306-4379(03)00072-3 (https://doi.org/10.1016%2FS0306-4379%280
3%2900072-3).
11. Tan, Pang-Ning; Michael, Steinbach; Kumar, Vipin (2005). "Chapter 6. Association Analysis:
Basic Concepts and Algorithms" (http://www-users.cs.umn.edu/~kumar/dmbook/ch6.pdf)
(PDF). Introduction to Data Mining. Addison-Wesley. ISBN 978-0-321-32136-7.
12. Jian Pei; Jiawei Han; Lakshmanan, L.V.S. (2001). "Mining frequent itemsets with convertible
constraints". Proceedings 17th International Conference on Data Engineering. pp. 433–442.
CiteSeerX 10.1.1.205.2150 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.205.
2150). doi:10.1109/ICDE.2001.914856 (https://doi.org/10.1109%2FICDE.2001.914856).
ISBN 978-0-7695-1001-9.
13. Agrawal, Rakesh; and Srikant, Ramakrishnan; Fast algorithms for mining association rules
in large databases (http://rakesh.agrawal-family.com/papers/vldb94apriori.pdf) Archived (htt
ps://web.archive.org/web/20150225213708/http://rakesh.agrawal-family.com/papers/vldb94
apriori.pdf) 2015-02-25 at the Wayback Machine, in Bocca, Jorge B.; Jarke, Matthias; and
Zaniolo, Carlo; editors, Proceedings of the 20th International Conference on Very Large
Data Bases (VLDB), Santiago, Chile, September 1994, pages 487-499
14. Zaki, M. J. (2000). "Scalable algorithms for association mining". IEEE Transactions on
Knowledge and Data Engineering. 12 (3): 372–390. CiteSeerX 10.1.1.79.9448 (https://cites
eerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.79.9448). doi:10.1109/69.846291 (https://do
i.org/10.1109%2F69.846291).
15. Hájek, P.; Havel, I.; Chytil, M. (1966). "The GUHA method of automatic hypotheses
determination". Computing. 1 (4): 293–308. doi:10.1007/BF02345483 (https://doi.org/10.100
7%2FBF02345483).
16. Hájek, Petr; Rauch, Jan; Coufal, David; Feglar, Tomáš (2004). "The GUHA Method, Data
Preprocessing and Mining". Database Support for Data Mining Applications. Lecture Notes
in Computer Science. 2682. pp. 135–153. doi:10.1007/978-3-540-44497-8_7 (https://doi.or
g/10.1007%2F978-3-540-44497-8_7). ISBN 978-3-540-22479-2.
17. Webb, Geoffrey (1989). "A Machine Learning Approach to Student Modelling". Proceedings
of the Third Australian Joint Conference on Artificial Intelligence (AI 89): 195–205.
18. Webb, Geoffrey I. (2007). "Discovering Significant Patterns". Machine Learning. 68: 1–33.
doi:10.1007/s10994-007-5006-x (https://doi.org/10.1007%2Fs10994-007-5006-x).
19. Gionis, Aristides; Mannila, Heikki; Mielikäinen, Taneli; Tsaparas, Panayiotis (2007).
"Assessing data mining results via swap randomization". ACM Transactions on Knowledge
Discovery from Data. 1 (3): 14–es. CiteSeerX 10.1.1.141.2607 (https://citeseerx.ist.psu.edu/
viewdoc/summary?doi=10.1.1.141.2607). doi:10.1145/1297332.1297338 (https://doi.org/10.
1145%2F1297332.1297338).
20. Zaki, Mohammed Javeed; Parthasarathy, Srinivasan; Ogihara, Mitsunori; Li, Wei (1997).
"New Algorithms for Fast Discovery of Association Rules": 283–286.
CiteSeerX 10.1.1.42.3283 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.32
83). hdl:1802/501 (https://hdl.handle.net/1802%2F501).
21. Zaki, Mohammed J.; Parthasarathy, Srinivasan; Ogihara, Mitsunori; Li, Wei (1997). "Parallel
Algorithms for Discovery of Association Rules". Data Mining and Knowledge Discovery. 1
(4): 343–373. doi:10.1023/A:1009773317876 (https://doi.org/10.1023%2FA%3A100977331
7876).
22. Han (2000). Mining Frequent Patterns Without Candidate Generation. Proceedings of the
2000 ACM SIGMOD International Conference on Management of Data. SIGMOD '00.
pp. 1–12. CiteSeerX 10.1.1.40.4436 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=1
0.1.1.40.4436). doi:10.1145/342009.335372 (https://doi.org/10.1145%2F342009.335372).
ISBN 978-1581132175.
23. Witten, Frank, Hall: Data mining practical machine learning tools and techniques, 3rd edition
24. Hájek, Petr; Havránek, Tomáš (1978). Mechanizing Hypothesis Formation: Mathematical
Foundations for a General Theory (http://www.cs.cas.cz/hajek/guhabook/). Springer-Verlag.
ISBN 978-3-540-08738-0.
25. Webb, Geoffrey I. (1995); OPUS: An Efficient Admissible Algorithm for Unordered Search,
Journal of Artificial Intelligence Research 3, Menlo Park, CA: AAAI Press, pp. 431-465
online access (http://webarchive.loc.gov/all/20011118141304/http://www.cs.washington.ed
u/research/jair/abstracts/webb95a.html)
26. Bayardo, Roberto J., Jr.; Agrawal, Rakesh; Gunopulos, Dimitrios (2000). "Constraint-based
rule mining in large, dense databases". Data Mining and Knowledge Discovery. 4 (2): 217–
240. doi:10.1023/A:1009895914772 (https://doi.org/10.1023%2FA%3A1009895914772).
27. Webb, Geoffrey I. (2000). "Efficient search for association rules". Proceedings of the sixth
ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '00.
pp. 99–107. CiteSeerX 10.1.1.33.1309 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=
10.1.1.33.1309). doi:10.1145/347090.347112 (https://doi.org/10.1145%2F347090.347112).
ISBN 978-1581132335.
28. "DSS News: Vol. 3, No. 23" (http://www.dssresources.com/newsletters/66.php).
29. Ramezani, Reza, Mohamad Sunni ee, and Mohammad Ali Nematbakhsh; MRAR: Mining
Multi-Relation Association Rules, Journal of Computing and Security, 1, no. 2 (2014)
30. GI Webb and S. Butler and D. Newlands (2003). On Detecting Differences Between Groups
(http://portal.acm.org/citation.cfm?id=956781). KDD'03 Proceedings of the Ninth ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining.
31. Menzies, T.; Ying Hu (2003). "Computing practices - Data mining for very busy people".
Computer. 36 (11): 22–29. doi:10.1109/MC.2003.1244531 (https://doi.org/10.1109%2FMC.
2003.1244531).
32. Wong, A.K.C.; Yang Wang (1997). "High-order pattern discovery from discrete-valued data".
IEEE Transactions on Knowledge and Data Engineering. 9 (6): 877–893.
CiteSeerX 10.1.1.189.1704 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.189.
1704). doi:10.1109/69.649314 (https://doi.org/10.1109%2F69.649314).
33. Liu, Jinze; Paulsen, Susan; Sun, Xing; Wang, Wei; Nobel, Andrew; Prins, Jan (2006).
"Mining Approximate Frequent Itemsets in the Presence of Noise: Algorithm and Analysis".
Proceedings of the 2006 SIAM International Conference on Data Mining. pp. 407–418.
CiteSeerX 10.1.1.215.3599 (https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.215.
3599). doi:10.1137/1.9781611972764.36
(https://doi.org/10.1137%2F1.9781611972764.36). ISBN 978-0-89871-611-5.
34. Zaki, Mohammed J. (2001); SPADE: An Efficient Algorithm for Mining Frequent Sequences,
Machine Learning Journal, 42, pp. 31–60
35. Zimek, Arthur; Assent, Ira; Vreeken, Jilles (2014). Frequent Pattern Mining. pp. 403–423.
doi:10.1007/978-3-319-07821-2_16 (https://doi.org/10.1007%2F978-3-319-07821-2_16).
ISBN 978-3-319-07820-5.
36. King, R. D.; Srinivasan, A.; Dehaspe, L. (Feb 2001). "Warmr: a data mining tool for chemical
data". J Comput Aided Mol Des. 15 (2): 173–81. Bibcode:2001JCAMD..15..173K (https://ui.
adsabs.harvard.edu/abs/2001JCAMD..15..173K). doi:10.1023/A:1008171016861 (https://do
i.org/10.1023%2FA%3A1008171016861). PMID 11272703 (https://pubmed.ncbi.nlm.nih.go
v/11272703).
Bibliographies
Annotated Bibliography on Association Rules (http://michael.hahsler.net/research/bib/associ
ation_rules/) by M. Hahsler
Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. By using
this site, you agree to the Terms of Use and Privacy Policy. Wikipedia® is a registered trademark of the Wikimedia
Foundation, Inc., a non-profit organization.