SlideShare a Scribd company logo

Introduction to Apache Accumulo

•
Aaron Cordova, profile picture
Aaron Cordova

Apache Accumulo is a distributed, scalable NoSQL database modeled after Google's Bigtable, designed to manage large amounts of unstructured data with dynamic configurations for data layout and security. The document outlines its architecture, which includes features such as automatic load balancing, dynamic partitioning, and a high-level Java API, along with effective data security measures and a lifecycle for managing data versions. Accumulo is capable of handling trillions of records while providing fast query responses and supports various applications through its unique data model and iterator functionalities.

Technology
1 of 110
Downloaded 314 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
Apache Accumulo Introduction
Introduction • Aaron Cordova • Founded Accumulo project with several others • Led development through release 1.0 • aaron@tetraconcepts.com
Agenda • Introduction • Data Model • API • Architecture - scaling, recovery • Security • Data-lifecycle • Applications
Introduction
History • Began writing in summer of 2008, after comparing design goals with BigTable paper and existing implementations Hbase, Hypertable • Released internal version 1.0 summer of 2009. • September 2011 accepted as an Apache Incubator project. Doug Cutting, founder of Hadoop, was the Champion Sponsor • Feb 2012 1.4 Released • March 2012 graduates to a top level Apache project • V 1.5 due out soon
Introduction • Accumulo is a sparse, distributed, sorted, multi-dimensional map • Modeled after Google’s BigTable design • Scales to trillions of records and 100s of Terabytes • Features automatic load balancing, high-availability, dynamic control over data layout
Data Model
Data Model Key Column Value row ID Timestamp Family Qualifier Visibility
Data Model (Logical 2D table structure) attribute: attribute: purchases: returns:hat age phone sneakers bill 49 555-1212 $100 - george 38 - $80 $30
Physical layout (sorted keys) row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
High-level API
Accumulo API • To use Accumulo, must write a an application using the Accumulo Java client library. There is no SQL (hence NoSQL) • Data is packaged into Mutation objects which are added to a BatchWriter which sends them to TabletServers • Clients can scan a set of key value pairs by specifying optional start and end keys (Range) and obtaining a Scanner. Iterating over the scanner returns sorted key value pairs for that range. Each scan takes milliseconds to start. • Can scan over a subset of the columns • Can send a set of Ranges to a BatchScanner, get matching key value pairs, unsorted
Insert row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Insert bill attribute phone private Jun 2010 555-1212 row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Insert row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Scan - Full key lookup bill attribute phone private Jun 2010 row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Scan - Single row bill row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Scan - Multiple Rows bill - will row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Scan - Multiple Rows, Selected Columns bill - will, fetch purchases row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Architecture - Scaling and Recovery
Performance Accumulo BigTable circa 2006 Cassandra • Accumulo ‘scales’ because aggregate read and write 10000 performance increase as more machines are added, and Thousands of writes per second because individual reads/write performance remains very good 1000 even with trillions of key-value pairs already in the system • Sources: http://www.slideshare.net/ acordova00/accumulo-on-ec2 100 http://techblog.netflix.com/2011/11/ benchmarking-cassandra-scalability-on.html http://static.googleusercontent.com/ external_content/untrusted_dlcp/ research.google.com/en/us/archive/bigtable- 10 osdi06.pdf 1 16 64 256 1024 Number of machines
Accumulo Prerequisites • One to hundreds of computers with local hard drives, connected via ethernet • Password-less SSH access • Local directory for write-ahead logs • Hadoop and ZooKeeper installed, configured, and running
Architecture Accumulo ZooKeeper HDFS MapReduce
Architecture: HDFS HDFS NameNode DataNodes File
Architecture: HDFS HDFS NameNode DataNodes Block 1 Block 2
Architecture: HDFS HDFS NameNode DataNodes
Architecture: Tables Accumulo Master Tablet Servers Table
Architecture: Tables Accumulo Master Tablet Servers P1 P2 P3
Architecture: Tables Accumulo Master Tablet Servers
Architecture: Writes P1 Mem Table File1 HDFS
Architecture: Writes P1 Mem Client Table Write-ahead Log File1 HDFS
Architecture: Writes P1 Mem Table Write-ahead Log File1 File 2 HDFS
Architecture: Writes P1 Mem Table X Write-ahead Log File1 File 2 HDFS
Architecture: Splits row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Architecture: Splits Accumulo Master Tablet Servers
Architecture: Splits Accumulo Master Tablet Servers
Architecture: Splits Accumulo Master Tablet Servers
Sorted keys - dynamic partitioning • Because keys are sorted, tables can be partitioned based on the data • partitions (tablets) are uniform in size, regardless of data distribution, (as long as single rows are smaller than the partition size) • not based on the number of servers • Can add /remove / fail servers at any time, the system is always automatically balanced
Partitioning Contrast • Some relational databases allow partitioning. May require users to choose a field or two on which to partition. Hopefully that field is uniformly distributed • Hash-based systems (default Cassandra, CouchDB, Riak, Voldemort) avoid this problem, but at the cost of range scans. Some support range scans via other means. • Many systems couple partition storage with partition service, requiring data movement to rebalance partition service (MongoDB, Cassandra, etc)
Architecture: Reads P1 Mem Client Table Merge File1 File 2
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers Master reassigns DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers Replay Write-ahead Log DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
Metadata Hierarchy metadata table root md1 md2 md3 user tables user1 user2 index1 index2
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Client
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Client knows zookeeper, finds root tablet Client
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Scan root tablet find metadata tablet that describes the user table we want Client
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Read location info of tablets of user table and cache it Client
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Read directly from server holding the tablets we want Client
Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Find other tablets via cache lookups Client
Security
Security • Design and Guarantees • Data Labeling • Authentication • User Configuration
Data Security • Accumulo will only return cells whose visibility labels are satisfied by user credentials presented at Scan time • Two necessary conditions • Correctly labeling data on ingest • Presenting right user credentials
Security Labels Extension of BigTable data model column row ID timestamp value family qualifier visibility
Column Visibility row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
Security Label Syntax • A & B - both A and B required • A | B - must have either A or B • (A | B) & C - must have C and A or B • A | (B & C) - must have A or both B and C • A & (B | (C & D))
Security Label Example • Drive needs: • license&over15 • Join military: • (over17|(over16&parentConsent)) & (greencard| USCitizen) • Access to Classified data • TS&SI&(USA|GBR|NZL|CAN|AUS)
Security Model Security Perimeter Accumulo auths data auths Trusted Client Auth Service verify ID, password, cert data User
Trusted Client Responsibility • Ensure that credentials belong to the user • Ensure that the user is authenticated
Application Authorization • Trusted Client applications must have max authorizations set before they can be passed • The Trusted Client limits the set of authorizations by application
Application Authorization Example • Data may be labeled with any combination of the following: { personal, research, finance, diet, cancer } • We wish to limit certain applications to a subset
Example Table row colF ColQ col vis value row0 name - personal|finance John row0 age - personal|research 49 row0 phone - personal|finance 555-1212 row0 owed - personal|finance $5440 row0 diagnosis - personal|(research & melanom cancer) a row0 diagnosis - personal|(research & diet) diabetes
Application Authorizations Cancer Research: cancer diagnoses, age Diabetes Research: diet info, age Accounting System: balance, name, phone Personal Records Management: all
Security Model Security Perimeter Accumulo Cancer Auth Service Research App ID, password, cert Researcher
Security Model Security Perimeter Accumulo Cancer Auth Service Research App verify ID, password, cert Researcher
Security Model Security Perimeter Accumulo research, cancer, diabetes Cancer Auth Service Research App verify ID, password, cert Researcher
Security Model Security Perimeter Accumulo research, cancer Cancer Auth Service Research App ID, password, cert Researcher
Security Model Security Perimeter Accumulo research, data cancer Cancer Auth Service Research App ID, password, cert Researcher
Security Model Security Perimeter Accumulo research, data cancer Cancer Auth Service Research App ID, password, cert data Researcher
Data life-cycle
Data Model Key Column Value row ID Timestamp Family Qualifier Visibility
Versions What can we do with multiple versions of the same data? rowID family qualifier timestamp value row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
Iterators • Mechanism for adding online functionality to tables • Aggregation (called Combiners) • Age-Off • Filtering (including by security label)
Versioning Iterators rowID family qualifier timestamp value row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
Filtering Iterators • Age Off • RegEx • Arbitrary filtering
Age Off • Can specify a particular date - e.g. delete everything older than July 1, 2007 • Can specify a time period - e.g. delete everything older than 6 months
Age-Off Current Time: 1103 rowID family qualifier timestamp value row1 fam1 qual1 1005 2 K/V pair is more than row1 fam1 qual1 1004 5 100 sec. old row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
Age-Off Current Time: 1104 rowID family qualifier timestamp value K/V pair is more than row1 fam1 qual1 1005 2 100 sec. old row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
Age-Off Current Time: 1105 K/V pair is more than rowID family qualifier timestamp value 100 sec. old row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
Manual Deletes • Can insert ‘deletes’. They are inserted like other key-value pairs, any keys with an older timestamp is suppressed from reads • Compactions write non-deleted data to new files • Old files are then removed from HDFS • To ensure data is deleted from disk, • write deletes (they are now absent from query results) • compact (can compact a particular range of a table if it’s large)
Garbage Collection • Garbage collector compares the files in HDFS with the set of files currently active • When files are no longer on the active list, GC waits for a while, then deletes from HDFS
Applications • Fast lookups / scan on extremely large tables with flexible schemas, varying security • Large index across heterogeneous data sets • Continuous Summary Analytics via Iterators • Secure Storage of key value pairs for MapReduce jobs
Where does your data come from? • BigTable was designed to store data for web applications serving millions of users. Web application creates all the data. Many NoSQL databases are designed solely for this purpose. Accumulo can certainly support that. • However, many organizations have lots of data from various sources. Different schema, different security levels. Bringing them together for analysis is very valuable. Accumulo can support this too.
Indexing and queries • BigTable data model supports building a wide variety of indexes • Simple strings, numbers, geo points, ip addresses, etc • Each has to be coupled with query code • New applications should examine their data access use cases, indexes and query code to accomplish those can then be written • Best applications are constructed so each user request is a single scan, or a small number of scans
Compared to MapReduce • Hadoop’s HDFS stores simple files. Usually unsorted. • MapReduce is designed to process all or most of the files at once. • Accumulo maintains a set of sorted files in HDFS • Accumulo scans are designed to access a small portion of the data quickly. • Fairly complementary
Tough use case • Ran MapReduce on some input data set to create a large result set. • Now have a few new records, want to update the result set • MapReduce has to process all the data again, have to wait • Accumulo allows users to perform a limited set of operations to update a result set incrementally, using Iterators • Result sets are always up to date, immediately after insert
Combiners row col fam col qual col vis time value bill perf June_calls P June 1 9 bill perf June_calls P June 4 3 bill perf July_calls P July 3 4 bill perf July_calls P July 11 7 bill perf August_calls P Aug 12 5 bill perf August_calls P Aug 29 2
Combiners row col fam col qual col vis time value bill perf June_calls P - 12 bill perf July_calls P - 11 bill perf August_calls P - 7
Combiners • Almost equivalent to Reduce of MapReduce except: • Cannot assume we have seen all the values for a particular key • Exactly equivalent to a Combiner function
Combiners • Useful Combiners: • Event count (StringSummation or LongSummation aggregator) • Event hour occurrence histogram (NumArraySummation aggregator) • Event duration histogram (NumArraySummation aggregator)
Conceptual Graph Representation b d a c e g f
Edge table row col fam col qual col vis time value a edge f 1.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0 g edge e 1.0 g edge f 1.0
Edge Weights • Summing Combiners are typically used to efficiently and incrementally update edge weights • See SummingCombiner
Edge table Incoming: a, edge, f, 1.0 row col fam col qual col vis time value a edge f 1.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge table row col fam col qual col vis time value a edge f 2.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge table Incoming: c, edge, b, 6.0 row col fam col qual col vis time value a edge f 2.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge table row col fam col qual col vis time value a edge f 2.0 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge table Incoming: a, edge, f, 2.3 row col fam col qual col vis time value a edge f 2.0 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge table row col fam col qual col vis time value a edge f 4.3 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
Edge Table Applications • Graph Analytics - traversal, neighbors, connected components • Neighborhood = feature vector. Vector-based machine learning techniques. Nearest neighbor search, clustering, classification • Automated dossiers, fact accumulation - ‘tell me everything we know about X’ in a single scan • Find entities based on features - ‘show me everyone who has feature value > x’ or ‘with < 5 neighbors of type k’
RDF Triples row col fam col qual col vis time value DC is_capital_of USA 1.0 Don vacations_in Arctic 7.0 Don is_employed_by MI6 1.0 Sean has_status “007” 1.0 Sean starred_with Ursula 1.0 Sean starred_with Anya 0.7 Sean starred_with Teresa 0.3
RDF Triples - RYA • See RYA project : http://www.usna.edu/Users/cs/adina/research/ Rya_CloudI2012.pdf
Additional Training
Additional Training • Talked about the basics today • 3 days of developer training with hands on examples covering • installation, configuration, read / write API, MapReduce, security, table configuration, indexing specific types, querying index tables, combiners, custom iterators, table constraints, storing relational data, joins, high performance considerations, document-partitioned indexing (text search), machine learning, object persistence • 2 days of administrator training covering • hardware selection, process assignment, troubleshooting, maintenance, replication and high availability, cluster modification, failure handling
Next Scheduled Training Sessions • March 5-7 Columbia MD • April 9-11 Columbia MD • http://www.tetraconcepts.com/training • aaron@tetraconcepts.com • brian@tetraconcepts.com

More Related Content

PDF
Lý thuyết giấu tin trong file audio
3M company
 
PDF
Ripe71 FastNetMon open source DoS / DDoS mitigation
Pavel Odintsov
 
PDF
Giáo trình bảo mật thông tin
jackjohn45
 
RTF
Phân tích và thiết kế FTTH trên GPON
freeloadtailieu
 
PDF
Ficha Técnica HP ProLiant DL380 Gen9
RaGaZoMe
 
DOCX
Đề tài: Thiết kế mạng truy nhập GPON dựa trên phần mềm mô phỏng
Dịch vụ viết bài trọn gói ZALO 0917193864
 
PPTX
Báo cáo dropbox
Kinny_Nguyen
 
PPTX
Fhrp notes
Krunal Shah
 
Lý thuyết giấu tin trong file audio
3M company
 
Ripe71 FastNetMon open source DoS / DDoS mitigation
Pavel Odintsov
 
Giáo trình bảo mật thông tin
jackjohn45
 
Phân tích và thiết kế FTTH trên GPON
freeloadtailieu
 
Ficha Técnica HP ProLiant DL380 Gen9
RaGaZoMe
 
Đề tài: Thiết kế mạng truy nhập GPON dựa trên phần mềm mô phỏng
Dịch vụ viết bài trọn gói ZALO 0917193864
 
Báo cáo dropbox
Kinny_Nguyen
 
Fhrp notes
Krunal Shah
 

What's hot (6)

PDF
Giáo trình Quản trị mạng
Tran Tien
 
PPTX
Forti wifi
Lan & Wan Solutions
 
PDF
Báo cáo quản lý cửa hàng máy tính
thuvienso
 
PDF
Giao thức ftp, smtp
Little April
 
PDF
Bài giảng an toàn ứng dụng web và csdl PTIT
NguynMinh294
 
PPTX
Hpe Intelligent Management Center
Aruba, a Hewlett Packard Enterprise company
 
Giáo trình Quản trị mạng
Tran Tien
 
Forti wifi
Lan & Wan Solutions
 
Báo cáo quản lý cửa hàng máy tính
thuvienso
 
Giao thức ftp, smtp
Little April
 
Bài giảng an toàn ứng dụng web và csdl PTIT
NguynMinh294
 
Hpe Intelligent Management Center
Aruba, a Hewlett Packard Enterprise company
 
Ad

Viewers also liked (20)

PPTX
Introduction to Apache Accumulo
Jared Winick
 
PPTX
An Introduction to Accumulo
Donald Miner
 
PDF
Apache Accumulo and the Data Lake
Aaron Cordova
 
PDF
Machine Learning & Graph Processing w/ Spark and Accumulo
Rahul Singh
 
PDF
Accumulo Summit 2015: Alternatives to Apache Accumulo's Java API [API]
Accumulo Summit
 
PDF
Faster Python
Anoop Thomas Mathew
 
PDF
Reversing the dropbox client on windows
extremecoders
 
PPT
Euro python2011 High Performance Python
Ian Ozsvald
 
PDF
Odessapy2013 - Graph databases and Python
Max Klymyshyn
 
KEY
Accumulo on EC2
Aaron Cordova
 
PDF
Sqrrl September Webinar: Cell-Level Security
Sqrrl
 
PDF
Accumulo Summit 2014: Four Orders of Magnitude: Running Large Scale Accumulo ...
Accumulo Summit
 
PDF
Accumulo design
scsorensen
 
PDF
Accumulo Summit 2015: Tracing in Accumulo and HDFS [Internals]
Accumulo Summit
 
PPTX
Accumulo meetup 20130109
Sqrrl
 
PDF
Beyond tf idf why, what & how
lucenerevolution
 
PDF
Accumulo Summit 2016: Accumulo in the Enterprise
Accumulo Summit
 
PDF
Large Scale Accumulo Clusters
Aaron Cordova
 
PDF
Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?
Accumulo Summit
 
PDF
Inside the ANN: A visual and intuitive journey to understand how artificial n...
XavierArrufat
 
Introduction to Apache Accumulo
Jared Winick
 
An Introduction to Accumulo
Donald Miner
 
Apache Accumulo and the Data Lake
Aaron Cordova
 
Machine Learning & Graph Processing w/ Spark and Accumulo
Rahul Singh
 
Accumulo Summit 2015: Alternatives to Apache Accumulo's Java API [API]
Accumulo Summit
 
Faster Python
Anoop Thomas Mathew
 
Reversing the dropbox client on windows
extremecoders
 
Euro python2011 High Performance Python
Ian Ozsvald
 
Odessapy2013 - Graph databases and Python
Max Klymyshyn
 
Accumulo on EC2
Aaron Cordova
 
Sqrrl September Webinar: Cell-Level Security
Sqrrl
 
Accumulo Summit 2014: Four Orders of Magnitude: Running Large Scale Accumulo ...
Accumulo Summit
 
Accumulo design
scsorensen
 
Accumulo Summit 2015: Tracing in Accumulo and HDFS [Internals]
Accumulo Summit
 
Accumulo meetup 20130109
Sqrrl
 
Beyond tf idf why, what & how
lucenerevolution
 
Accumulo Summit 2016: Accumulo in the Enterprise
Accumulo Summit
 
Large Scale Accumulo Clusters
Aaron Cordova
 
Accumulo Summit 2014: Benchmarking Accumulo: How Fast Is Fast?
Accumulo Summit
 
Inside the ANN: A visual and intuitive journey to understand how artificial n...
XavierArrufat
 
Ad

Similar to Introduction to Apache Accumulo (20)

PDF
Balancing Replication and Partitioning in a Distributed Java Database
Ben Stopford
 
PPTX
Microsoft Openness Mongo DB
Heriyadi Janwar
 
PDF
Intro to NoSQL and MongoDB
DATAVERSITY
 
PDF
Common MongoDB Use Cases
DATAVERSITY
 
PDF
Accumulo design
Koverse, Inc.
 
PPTX
Common MongoDB Use Cases Webinar
MongoDB
 
PPTX
Hadoop World 2011: Hadoop and RDBMS with Sqoop and Other Tools - Guy Harrison...
Cloudera, Inc.
 
PPTX
HBase and Accumulo | Washington DC Hadoop User Group
Cloudera, Inc.
 
PPTX
H base vs hive srp vs analytics 2-14-2012
reedshea
 
PDF
What Every Developer Should Know About Database Scalability
jbellis
 
PPTX
TriHUG January 2012 Talk by Chris Shain
trihug
 
PDF
Apache HBase: Introduction to a column-oriented data store
Christian Gügi
 
PDF
Common MongoDB Use Cases
DATAVERSITY
 
PPT
Data warehousing
Bhaskar Pathak
 
PPTX
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
lisapaglia
 
PPT
Data Kinetics Products
sheena82
 
PPTX
A Practical Look at the NOSQL and Big Data Hullabaloo
Andrew Brust
 
PDF
Olap scalability
lucboudreau
 
PPTX
NoSQL and The Big Data Hullabaloo
Andrew Brust
 
PPTX
Cassandra 2012 scandit
Charlie Zhu
 
Balancing Replication and Partitioning in a Distributed Java Database
Ben Stopford
 
Microsoft Openness Mongo DB
Heriyadi Janwar
 
Intro to NoSQL and MongoDB
DATAVERSITY
 
Common MongoDB Use Cases
DATAVERSITY
 
Accumulo design
Koverse, Inc.
 
Common MongoDB Use Cases Webinar
MongoDB
 
Hadoop World 2011: Hadoop and RDBMS with Sqoop and Other Tools - Guy Harrison...
Cloudera, Inc.
 
HBase and Accumulo | Washington DC Hadoop User Group
Cloudera, Inc.
 
H base vs hive srp vs analytics 2-14-2012
reedshea
 
What Every Developer Should Know About Database Scalability
jbellis
 
TriHUG January 2012 Talk by Chris Shain
trihug
 
Apache HBase: Introduction to a column-oriented data store
Christian Gügi
 
Common MongoDB Use Cases
DATAVERSITY
 
Data warehousing
Bhaskar Pathak
 
"Navigating the Database Universe" by Dr. Michael Stonebraker and Scott Jarr,...
lisapaglia
 
Data Kinetics Products
sheena82
 
A Practical Look at the NOSQL and Big Data Hullabaloo
Andrew Brust
 
Olap scalability
lucboudreau
 
NoSQL and The Big Data Hullabaloo
Andrew Brust
 
Cassandra 2012 scandit
Charlie Zhu
 

Recently uploaded (20)

PDF
BLW VOCATIONAL TRAINING SUMMER INTERNSHIP REPORT
codernjn73
 
PDF
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
PPTX
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
PPTX
ChatGPT's Deck on The Enduring Legacy of Fax Machines
Greg Swan
 
PDF
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
PDF
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
PDF
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
PDF
Doc9.....................................
SofiaCollazos
 
PPTX
Comunidade Salesforce São Paulo - Desmistificando o Omnistudio (Vlocity)
Francisco Vieira Júnior
 
PDF
Brief History of Internet - Early Days of Internet
sutharharshit158
 
PPTX
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
PDF
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
PDF
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
PDF
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
PDF
REPORT: Heating appliances market in Poland 2024
SPIUG
 
PDF
Software Development Company | KodekX
KodekX
 
PPTX
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
PPTX
How to Build a Scalable Micro-Investing Platform in 2025 - A Founder’s Guide ...
Third Rock Techkno
 
PDF
This slide provides an overview Technology
mineshkharadi333
 
PDF
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 
BLW VOCATIONAL TRAINING SUMMER INTERNSHIP REPORT
codernjn73
 
SparkLabs Primer on Artificial Intelligence 2025
SparkLabs Group
 
The-Ethical-Hackers-Imperative-Safeguarding-the-Digital-Frontier.pptx
sujalchauhan1305
 
ChatGPT's Deck on The Enduring Legacy of Fax Machines
Greg Swan
 
Data_Analytics_vs_Data_Science_vs_BI_by_CA_Suvidha_Chaplot.pdf
CA Suvidha Chaplot
 
Unlocking the Future- AI Agents Meet Oracle Database 23ai - AIOUG Yatra 2025.pdf
Sandesh Rao
 
Cloud-Migration-Best-Practices-A-Practical-Guide-to-AWS-Azure-and-Google-Clou...
Artjoker Software Development Company
 
Doc9.....................................
SofiaCollazos
 
Comunidade Salesforce São Paulo - Desmistificando o Omnistudio (Vlocity)
Francisco Vieira Júnior
 
Brief History of Internet - Early Days of Internet
sutharharshit158
 
OA presentation.pptx OA presentation.pptx
pateldhruv002338
 
Automating ArcGIS Content Discovery with FME: A Real World Use Case
Safe Software
 
Presentation about Hardware and Software in Computer
snehamodhawadiya
 
CIFDAQ's Market Wrap : Bears Back in Control?
CIFDAQ
 
REPORT: Heating appliances market in Poland 2024
SPIUG
 
Software Development Company | KodekX
KodekX
 
What-is-the-World-Wide-Web -- Introduction
tonifi9488
 
How to Build a Scalable Micro-Investing Platform in 2025 - A Founder’s Guide ...
Third Rock Techkno
 
This slide provides an overview Technology
mineshkharadi333
 
Using Anchore and DefectDojo to Stand Up Your DevSecOps Function
Anchore
 

Introduction to Apache Accumulo

  • 2. Introduction • Aaron Cordova • Founded Accumulo project with several others • Led development through release 1.0 • [email protected]
  • 3. Agenda • Introduction • Data Model • API • Architecture - scaling, recovery • Security • Data-lifecycle • Applications
  • 5. History • Began writing in summer of 2008, after comparing design goals with BigTable paper and existing implementations Hbase, Hypertable • Released internal version 1.0 summer of 2009. • September 2011 accepted as an Apache Incubator project. Doug Cutting, founder of Hadoop, was the Champion Sponsor • Feb 2012 1.4 Released • March 2012 graduates to a top level Apache project • V 1.5 due out soon
  • 6. Introduction • Accumulo is a sparse, distributed, sorted, multi-dimensional map • Modeled after Google’s BigTable design • Scales to trillions of records and 100s of Terabytes • Features automatic load balancing, high-availability, dynamic control over data layout
  • 8. Data Model Key Column Value row ID Timestamp Family Qualifier Visibility
  • 9. Data Model (Logical 2D table structure) attribute: attribute: purchases: returns:hat age phone sneakers bill 49 555-1212 $100 - george 38 - $80 $30
  • 10. Physical layout (sorted keys) row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 12. Accumulo API • To use Accumulo, must write a an application using the Accumulo Java client library. There is no SQL (hence NoSQL) • Data is packaged into Mutation objects which are added to a BatchWriter which sends them to TabletServers • Clients can scan a set of key value pairs by specifying optional start and end keys (Range) and obtaining a Scanner. Iterating over the scanner returns sorted key value pairs for that range. Each scan takes milliseconds to start. • Can scan over a subset of the columns • Can send a set of Ranges to a BatchScanner, get matching key value pairs, unsorted
  • 13. Insert row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 14. Insert bill attribute phone private Jun 2010 555-1212 row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 15. Insert row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 16. Scan - Full key lookup bill attribute phone private Jun 2010 row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 17. Scan - Single row bill row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 18. Scan - Multiple Rows bill - will row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 19. Scan - Multiple Rows, Selected Columns bill - will, fetch purchases row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 20. Architecture - Scaling and Recovery
  • 21. Performance Accumulo BigTable circa 2006 Cassandra • Accumulo ‘scales’ because aggregate read and write 10000 performance increase as more machines are added, and Thousands of writes per second because individual reads/write performance remains very good 1000 even with trillions of key-value pairs already in the system • Sources: http://www.slideshare.net/ acordova00/accumulo-on-ec2 100 http://techblog.netflix.com/2011/11/ benchmarking-cassandra-scalability-on.html http://static.googleusercontent.com/ external_content/untrusted_dlcp/ research.google.com/en/us/archive/bigtable- 10 osdi06.pdf 1 16 64 256 1024 Number of machines
  • 22. Accumulo Prerequisites • One to hundreds of computers with local hard drives, connected via ethernet • Password-less SSH access • Local directory for write-ahead logs • Hadoop and ZooKeeper installed, configured, and running
  • 23. Architecture Accumulo ZooKeeper HDFS MapReduce
  • 24. Architecture: HDFS HDFS NameNode DataNodes File
  • 25. Architecture: HDFS HDFS NameNode DataNodes Block 1 Block 2
  • 26. Architecture: HDFS HDFS NameNode DataNodes
  • 27. Architecture: Tables Accumulo Master Tablet Servers Table
  • 28. Architecture: Tables Accumulo Master Tablet Servers P1 P2 P3
  • 29. Architecture: Tables Accumulo Master Tablet Servers
  • 30. Architecture: Writes P1 Mem Table File1 HDFS
  • 31. Architecture: Writes P1 Mem Client Table Write-ahead Log File1 HDFS
  • 32. Architecture: Writes P1 Mem Table Write-ahead Log File1 File 2 HDFS
  • 33. Architecture: Writes P1 Mem Table X Write-ahead Log File1 File 2 HDFS
  • 34. Architecture: Splits row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 35. Architecture: Splits Accumulo Master Tablet Servers
  • 36. Architecture: Splits Accumulo Master Tablet Servers
  • 37. Architecture: Splits Accumulo Master Tablet Servers
  • 38. Sorted keys - dynamic partitioning • Because keys are sorted, tables can be partitioned based on the data • partitions (tablets) are uniform in size, regardless of data distribution, (as long as single rows are smaller than the partition size) • not based on the number of servers • Can add /remove / fail servers at any time, the system is always automatically balanced
  • 39. Partitioning Contrast • Some relational databases allow partitioning. May require users to choose a field or two on which to partition. Hopefully that field is uniformly distributed • Hash-based systems (default Cassandra, CouchDB, Riak, Voldemort) avoid this problem, but at the cost of range scans. Some support range scans via other means. • Many systems couple partition storage with partition service, requiring data movement to rebalance partition service (MongoDB, Cassandra, etc)
  • 40. Architecture: Reads P1 Mem Client Table Merge File1 File 2
  • 41. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 42. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 43. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 44. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 45. Architecture: Recovery Accumulo Master Tablet Servers Master reassigns DataNodes NameNode
  • 46. Architecture: Recovery Accumulo Master Tablet Servers Replay Write-ahead Log DataNodes NameNode
  • 47. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 48. Architecture: Recovery Accumulo Master Tablet Servers DataNodes NameNode
  • 49. Metadata Hierarchy metadata table root md1 md2 md3 user tables user1 user2 index1 index2
  • 50. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Client
  • 51. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Client knows zookeeper, finds root tablet Client
  • 52. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Scan root tablet find metadata tablet that describes the user table we want Client
  • 53. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Read location info of tablets of user table and cache it Client
  • 54. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Read directly from server holding the tablets we want Client
  • 55. Architecture: Lookup Accumulo Master Tablet Servers ZooKeeper Find other tablets via cache lookups Client
  • 57. Security • Design and Guarantees • Data Labeling • Authentication • User Configuration
  • 58. Data Security • Accumulo will only return cells whose visibility labels are satisfied by user credentials presented at Scan time • Two necessary conditions • Correctly labeling data on ingest • Presenting right user credentials
  • 59. Security Labels Extension of BigTable data model column row ID timestamp value family qualifier visibility
  • 60. Column Visibility row col fam col qual col vis time value bill attribute age public Jun 2010 49 bill attribute phone private Jun 2010 555-1212 bill purchases sneakers public Apr 2010 $100 george attribute age private Oct 2009 38 george purchases sneakers public Nov 2009 $80 george returns hat public Dec 2009 $30
  • 61. Security Label Syntax • A & B - both A and B required • A | B - must have either A or B • (A | B) & C - must have C and A or B • A | (B & C) - must have A or both B and C • A & (B | (C & D))
  • 62. Security Label Example • Drive needs: • license&over15 • Join military: • (over17|(over16&parentConsent)) & (greencard| USCitizen) • Access to Classified data • TS&SI&(USA|GBR|NZL|CAN|AUS)
  • 63. Security Model Security Perimeter Accumulo auths data auths Trusted Client Auth Service verify ID, password, cert data User
  • 64. Trusted Client Responsibility • Ensure that credentials belong to the user • Ensure that the user is authenticated
  • 65. Application Authorization • Trusted Client applications must have max authorizations set before they can be passed • The Trusted Client limits the set of authorizations by application
  • 66. Application Authorization Example • Data may be labeled with any combination of the following: { personal, research, finance, diet, cancer } • We wish to limit certain applications to a subset
  • 67. Example Table row colF ColQ col vis value row0 name - personal|finance John row0 age - personal|research 49 row0 phone - personal|finance 555-1212 row0 owed - personal|finance $5440 row0 diagnosis - personal|(research & melanom cancer) a row0 diagnosis - personal|(research & diet) diabetes
  • 68. Application Authorizations Cancer Research: cancer diagnoses, age Diabetes Research: diet info, age Accounting System: balance, name, phone Personal Records Management: all
  • 69. Security Model Security Perimeter Accumulo Cancer Auth Service Research App ID, password, cert Researcher
  • 70. Security Model Security Perimeter Accumulo Cancer Auth Service Research App verify ID, password, cert Researcher
  • 71. Security Model Security Perimeter Accumulo research, cancer, diabetes Cancer Auth Service Research App verify ID, password, cert Researcher
  • 72. Security Model Security Perimeter Accumulo research, cancer Cancer Auth Service Research App ID, password, cert Researcher
  • 73. Security Model Security Perimeter Accumulo research, data cancer Cancer Auth Service Research App ID, password, cert Researcher
  • 74. Security Model Security Perimeter Accumulo research, data cancer Cancer Auth Service Research App ID, password, cert data Researcher
  • 76. Data Model Key Column Value row ID Timestamp Family Qualifier Visibility
  • 77. Versions What can we do with multiple versions of the same data? rowID family qualifier timestamp value row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
  • 78. Iterators • Mechanism for adding online functionality to tables • Aggregation (called Combiners) • Age-Off • Filtering (including by security label)
  • 79. Versioning Iterators rowID family qualifier timestamp value row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
  • 80. Filtering Iterators • Age Off • RegEx • Arbitrary filtering
  • 81. Age Off • Can specify a particular date - e.g. delete everything older than July 1, 2007 • Can specify a time period - e.g. delete everything older than 6 months
  • 82. Age-Off Current Time: 1103 rowID family qualifier timestamp value row1 fam1 qual1 1005 2 K/V pair is more than row1 fam1 qual1 1004 5 100 sec. old row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
  • 83. Age-Off Current Time: 1104 rowID family qualifier timestamp value K/V pair is more than row1 fam1 qual1 1005 2 100 sec. old row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
  • 84. Age-Off Current Time: 1105 K/V pair is more than rowID family qualifier timestamp value 100 sec. old row1 fam1 qual1 1005 2 row1 fam1 qual1 1004 5 row1 fam1 qual1 1003 3 row1 fam1 qual1 1002 2 row1 fam1 qual1 1001 7
  • 85. Manual Deletes • Can insert ‘deletes’. They are inserted like other key-value pairs, any keys with an older timestamp is suppressed from reads • Compactions write non-deleted data to new files • Old files are then removed from HDFS • To ensure data is deleted from disk, • write deletes (they are now absent from query results) • compact (can compact a particular range of a table if it’s large)
  • 86. Garbage Collection • Garbage collector compares the files in HDFS with the set of files currently active • When files are no longer on the active list, GC waits for a while, then deletes from HDFS
  • 87. Applications • Fast lookups / scan on extremely large tables with flexible schemas, varying security • Large index across heterogeneous data sets • Continuous Summary Analytics via Iterators • Secure Storage of key value pairs for MapReduce jobs
  • 88. Where does your data come from? • BigTable was designed to store data for web applications serving millions of users. Web application creates all the data. Many NoSQL databases are designed solely for this purpose. Accumulo can certainly support that. • However, many organizations have lots of data from various sources. Different schema, different security levels. Bringing them together for analysis is very valuable. Accumulo can support this too.
  • 89. Indexing and queries • BigTable data model supports building a wide variety of indexes • Simple strings, numbers, geo points, ip addresses, etc • Each has to be coupled with query code • New applications should examine their data access use cases, indexes and query code to accomplish those can then be written • Best applications are constructed so each user request is a single scan, or a small number of scans
  • 90. Compared to MapReduce • Hadoop’s HDFS stores simple files. Usually unsorted. • MapReduce is designed to process all or most of the files at once. • Accumulo maintains a set of sorted files in HDFS • Accumulo scans are designed to access a small portion of the data quickly. • Fairly complementary
  • 91. Tough use case • Ran MapReduce on some input data set to create a large result set. • Now have a few new records, want to update the result set • MapReduce has to process all the data again, have to wait • Accumulo allows users to perform a limited set of operations to update a result set incrementally, using Iterators • Result sets are always up to date, immediately after insert
  • 92. Combiners row col fam col qual col vis time value bill perf June_calls P June 1 9 bill perf June_calls P June 4 3 bill perf July_calls P July 3 4 bill perf July_calls P July 11 7 bill perf August_calls P Aug 12 5 bill perf August_calls P Aug 29 2
  • 93. Combiners row col fam col qual col vis time value bill perf June_calls P - 12 bill perf July_calls P - 11 bill perf August_calls P - 7
  • 94. Combiners • Almost equivalent to Reduce of MapReduce except: • Cannot assume we have seen all the values for a particular key • Exactly equivalent to a Combiner function
  • 95. Combiners • Useful Combiners: • Event count (StringSummation or LongSummation aggregator) • Event hour occurrence histogram (NumArraySummation aggregator) • Event duration histogram (NumArraySummation aggregator)
  • 97. Edge table row col fam col qual col vis time value a edge f 1.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0 g edge e 1.0 g edge f 1.0
  • 98. Edge Weights • Summing Combiners are typically used to efficiently and incrementally update edge weights • See SummingCombiner
  • 99. Edge table Incoming: a, edge, f, 1.0 row col fam col qual col vis time value a edge f 1.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 100. Edge table row col fam col qual col vis time value a edge f 2.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 101. Edge table Incoming: c, edge, b, 6.0 row col fam col qual col vis time value a edge f 2.0 c edge b 1.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 102. Edge table row col fam col qual col vis time value a edge f 2.0 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 103. Edge table Incoming: a, edge, f, 2.3 row col fam col qual col vis time value a edge f 2.0 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 104. Edge table row col fam col qual col vis time value a edge f 4.3 c edge b 7.0 c edge d 1.0 d edge b 1.0 d edge e 1.0 e edge d 1.0 f edge g 1.0
  • 105. Edge Table Applications • Graph Analytics - traversal, neighbors, connected components • Neighborhood = feature vector. Vector-based machine learning techniques. Nearest neighbor search, clustering, classification • Automated dossiers, fact accumulation - ‘tell me everything we know about X’ in a single scan • Find entities based on features - ‘show me everyone who has feature value > x’ or ‘with < 5 neighbors of type k’
  • 106. RDF Triples row col fam col qual col vis time value DC is_capital_of USA 1.0 Don vacations_in Arctic 7.0 Don is_employed_by MI6 1.0 Sean has_status “007” 1.0 Sean starred_with Ursula 1.0 Sean starred_with Anya 0.7 Sean starred_with Teresa 0.3
  • 107. RDF Triples - RYA • See RYA project : http://www.usna.edu/Users/cs/adina/research/ Rya_CloudI2012.pdf
  • 109. Additional Training • Talked about the basics today • 3 days of developer training with hands on examples covering • installation, configuration, read / write API, MapReduce, security, table configuration, indexing specific types, querying index tables, combiners, custom iterators, table constraints, storing relational data, joins, high performance considerations, document-partitioned indexing (text search), machine learning, object persistence • 2 days of administrator training covering • hardware selection, process assignment, troubleshooting, maintenance, replication and high availability, cluster modification, failure handling
  • 110. Next Scheduled Training Sessions • March 5-7 Columbia MD • April 9-11 Columbia MD • http://www.tetraconcepts.com/training • [email protected] • [email protected]