GeoMesa: Scalable Geospatial Analytics
Download as PPTX, PDF6 likes2,501 views
GeoMesa is an open-source project that provides scalable geospatial analytics on large datasets. It allows querying and analyzing data stored in Apache Accumulo using a geospatial index. GeoMesa implements the GeoTools API and supports point, line, polygon, raster, and time-enabled data through flexible space-filling curves. It enables distributed computation and analytics through features like multi-step query planning, secondary indexes, and integration with frameworks like Spark and streaming APIs. The project is developed and supported by a community including LocationTech.
![[why] which X (points) are close to location Y?
• hundreds: PostgreSQL and brute force
– full table scan
• hundreds of thousands: PostgreSQL and PostGIS
– GeoTools API
– GiST (think R-trees)
• hundreds of millions: a funny thing happens as you collect much more data...](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-5-320.jpg)
![[why] dissolution of large-volume data](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-6-320.jpg)
![[why] perhaps SQL is the bottleneck?
• NoSQL databases, such as Apache Accumulo
• trade ACID for distributed processing, storage
• but there’s no PostGIS for Accumulo, so how does the canonical diagram of an Accumulo (key,
value) pair help us answer some simple questions...](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-7-320.jpg)
![[why] questions that ought to be easy for an index to answer
• easy question: Which comes first, “Ontario” or “Quebec”?](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-8-320.jpg)
![[why] questions that ought to be easy for an index to answer
• easy question: Which comes first, “Ontario” or “Quebec”?
• similar question: Which comes first, or ?](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-9-320.jpg)
![[why] questions that ought to be easy for an index to answer
• easy question: Which comes first, “Ontario” or “Quebec”?
• similar question: Which comes first, or ?
• simplify, and think only of representative cities, and think of them strictly as points](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-10-320.jpg)
![[why] geohashing](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-11-320.jpg)
![[why] geohashing](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-12-320.jpg)
![[why] geohashing
City Coordinates (courtesy Wikipedia) Geohash
Ottawa 45°25′15″N 75°41′24″W f244m
Montréal 45°30′N 73°34′W f25dv
Charlottesville (Virginia, USA) 38°1′48″N 78°28′44″W dqb0q
● Two unique orders:
○ Order by name: Charlottesville, Montréal, Ottawa
○ Order by longitude or latitude or geohash: Charlottesville, Ottawa, Montréal
● Lexicoding location -> geohash provides a deterministic, repeatable ordering
○ with this, we can index, store, and query points by lexicographic ranges](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-13-320.jpg)
![[why] build-versus-buy remorse
• PostgreSQL+PostGIS has some nice functions
– geometric predicates
– secondary indexes
– standard GeoTools API
• some of our data are (multi) lines, (multi) polygons
• time is often more than a secondary consideration
• sometimes, analysis work needn’t be done on the same old client
– distributed across the tablet servers?
– using tools like Spark?
– streaming?](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-14-320.jpg)
![[why] synthesis](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-15-320.jpg)
![[how] GeoMesa features
• GeoTools API
• sharding distributes queries uniformly
• flexible SFC can incorporate time
• supports (multi) point, (multi) line, (multi) polygon geometries
• secondary indexes and a multi-stage query planner
• burgeoning raster support via WCS
• GeoServer as a plugin-based GUI
• WPS standards for computation (and function chaining)](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-17-320.jpg)
![[how] GeoTools API](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-18-320.jpg)
![[how] sharding](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-19-320.jpg)
![[how] space-filling curve progression
%~#s%3#r%0,3#gh%yyyyMM#d::%~#s%3,2#gh::%~#s%5,2#gh%HHmm#d%id](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-20-320.jpg)
![[how] multi-step query planning](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-21-320.jpg)
![[how] multi-step query planning](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-22-320.jpg)
![[how] non-point geometries](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-23-320.jpg)
![[how] rasters + GeoWave integration](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-24-320.jpg)
![[how] supporting other frameworks](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-25-320.jpg)
![[how] GeoServer as a plug-in GUI](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-26-320.jpg)
![[how] Web Processing Service
• WPS is another OGC standard
• Think of it as an abstract function definition, mapping input types to output types, and defining
the computation that occurs between the two.
• WPS processes can be chained.
• This provides for a natural extension mechanism to GeoMesa.](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-27-320.jpg)
![[how] synthesis
Those are merely the highlights of some of GeoMesa’s current features…
… so what?](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-28-320.jpg)
![[what] distributing computation](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-30-320.jpg)
![[what] queries that interpolate both position and time](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-31-320.jpg)
![[what] K-nearest neighbor](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-32-320.jpg)
![[what] clustering (DBSCAN)](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-33-320.jpg)
![[what] near-real-time streaming track analytics with web sockets](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-34-320.jpg)
![[what] track viewer utility](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-35-320.jpg)
![[who] LocationTech and the greater community](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-37-320.jpg)
![[who] synthesis](https://image.slidesharecdn.com/eg2014-ccri-141126121507-conversion-gate01/85/GeoMesa-Scalable-Geospatial-Analytics-38-320.jpg)
GeoMesa: Scalable Geospatial Analytics
- 1. GeoMesa: Scalable Geospatial Analytics Chris Eichelberger [email protected]
- 2. terms • GeoMesa: an open-source project organized under LocationTech • scalable: if you can continue to solve problems as N >> 1 with no more change than adding hardware and minor tweaks, you scale • geospatial: data that contain a geographic reference, a date/time, and zero or more additional attributes • analytics: formally, a logical decomposition via truth-preserving transformations; informally, any useful derivation (whether deductive or inductive)
- 3. outline • part 1: why? ( 3 minutes) • part 2: how? (10 minutes) • part 3: what? (10 minutes) • part 4: who? ( 2 minutes)
- 4. part 1: why?
- 5. [why] which X (points) are close to location Y? • hundreds: PostgreSQL and brute force – full table scan • hundreds of thousands: PostgreSQL and PostGIS – GeoTools API – GiST (think R-trees) • hundreds of millions: a funny thing happens as you collect much more data...
- 6. [why] dissolution of large-volume data
- 7. [why] perhaps SQL is the bottleneck? • NoSQL databases, such as Apache Accumulo • trade ACID for distributed processing, storage • but there’s no PostGIS for Accumulo, so how does the canonical diagram of an Accumulo (key, value) pair help us answer some simple questions...
- 8. [why] questions that ought to be easy for an index to answer • easy question: Which comes first, “Ontario” or “Quebec”?
- 9. [why] questions that ought to be easy for an index to answer • easy question: Which comes first, “Ontario” or “Quebec”? • similar question: Which comes first, or ?
- 10. [why] questions that ought to be easy for an index to answer • easy question: Which comes first, “Ontario” or “Quebec”? • similar question: Which comes first, or ? • simplify, and think only of representative cities, and think of them strictly as points
- 11. [why] geohashing
- 12. [why] geohashing
- 13. [why] geohashing City Coordinates (courtesy Wikipedia) Geohash Ottawa 45°25′15″N 75°41′24″W f244m Montréal 45°30′N 73°34′W f25dv Charlottesville (Virginia, USA) 38°1′48″N 78°28′44″W dqb0q ● Two unique orders: ○ Order by name: Charlottesville, Montréal, Ottawa ○ Order by longitude or latitude or geohash: Charlottesville, Ottawa, Montréal ● Lexicoding location -> geohash provides a deterministic, repeatable ordering ○ with this, we can index, store, and query points by lexicographic ranges
- 14. [why] build-versus-buy remorse • PostgreSQL+PostGIS has some nice functions – geometric predicates – secondary indexes – standard GeoTools API • some of our data are (multi) lines, (multi) polygons • time is often more than a secondary consideration • sometimes, analysis work needn’t be done on the same old client – distributed across the tablet servers? – using tools like Spark? – streaming?
- 15. [why] synthesis
- 16. part 2: how?
- 17. [how] GeoMesa features • GeoTools API • sharding distributes queries uniformly • flexible SFC can incorporate time • supports (multi) point, (multi) line, (multi) polygon geometries • secondary indexes and a multi-stage query planner • burgeoning raster support via WCS • GeoServer as a plugin-based GUI • WPS standards for computation (and function chaining)
- 19. [how] sharding
- 20. [how] space-filling curve progression %~#s%3#r%0,3#gh%yyyyMM#d::%~#s%3,2#gh::%~#s%5,2#gh%HHmm#d%id
- 21. [how] multi-step query planning
- 22. [how] multi-step query planning
- 24. [how] rasters + GeoWave integration
- 25. [how] supporting other frameworks
- 26. [how] GeoServer as a plug-in GUI
- 27. [how] Web Processing Service • WPS is another OGC standard • Think of it as an abstract function definition, mapping input types to output types, and defining the computation that occurs between the two. • WPS processes can be chained. • This provides for a natural extension mechanism to GeoMesa.
- 28. [how] synthesis Those are merely the highlights of some of GeoMesa’s current features… … so what?
- 29. part 3: what?
- 31. [what] queries that interpolate both position and time
- 34. [what] near-real-time streaming track analytics with web sockets
- 35. [what] track viewer utility
- 36. part 3: who?
- 37. [who] LocationTech and the greater community
- 38. [who] synthesis
- 39. questions For extended questions: [email protected] [email protected] [email protected] For additional reading: geomesa.org For code: github.com/locationtech/geomesa