Location analytics by Marc Planaguma at Big Data Spain 2014

Nov 25, 20140 likes870 views

http://www.bigdataspain.org/2014/conference/location-analytics While the implementation of analytic operations on distributed computing frameworks has been widely describing, enabling the computational core of a Big Data system with capabilities for supporting geospatial querying on data is yet a challenging issue. This session aims to target that specific aspect by reviewing how researchers at BDigital Technology Centre have designed and implemented a stack for advanced Machine Learning on Urban Data by providing a way to geoquery massive amounts of HDFS data from Spark processes without hindering the overall system performance.

SPATIAL DATA ANALYTICS
MARC PLANAGUMÀ
LEAD DATA ENGINEER AND RESEARCHER BDIGITAL

Ciudad 2020
Exploratory lines and further work
Joana Simoes, BDigital
27/05/2014
Scaling up a in a world of geolocated data
Marc Planagumà, BDigital Technology Centre

Location analytics by Marc Planaguma at Big Data Spain 2014

Spatial data analytics
Spatial analysis or spatial statistics includes any of
the formal techniques which studies entities using
their topological, geometric or geographic
properties

Spatial data analytics
Basic Spatial Types
Line Point Polygon
* plus others

Spatial data analytics
Operations
Contains Within

Spatial data analytics
Operations
Equals Disjoint

Spatial data analytics
Operations
Touches Overlaps
Crosses

Spatial data analytics
Operations
Distance

Spatial data analytics
Operations
Union Difference

Spatial data analytics
Operations
Intersection Symmetric Diff

Spatial data analytics
Complex operations
Buffering
Convex Hull

Spatial data analytics
Complex operations
Area

Spatial data analytics
Geospatial data mining techniques
Density-based clustering Hierarchy-based clustering

Spatial data analytics
Geospatial data mining techniques
Heatmaps

Spatial data analytics
Geospatial data mining techniques
Trajectory mining

Spatial data analytics
Geospatial data mining techniques
Geospatial querying
Spatial anomalies
Geographically weighted regressions
Self-organizing Maps
Agent-based modeling

Spatial data analytics
Our Datasets
Twitter (Point)
Meteo (Polygon)
Traffic status (Multi-Line-String)
Traffic incidence (Multi-Point)
Credit card ops (Multi-Point)
Demographic (Point & Polygon)
Shapefiles (Multi-Polygon)

Our Tools
Spatial data analytics
Queries
Clustering
Heat maps
Trajectory
Anomaly detection

Spatial data analytics
Old School Stack - Why Not?
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Spatial data analytics
Hadoop Stack - The new Classic
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Spatial data analytics
Complete Stack - Big Data GIS
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Do’s and Don’ts
Spatial data analytics
Geo Batch
Near Real-Time
Basic Geo Operations
Workflow Composition
Geo Streams
Real-Time
Advanced Geo Op.
Unattended Workflow

Moltes gràcies!
nom@bdigital.org
mplanaguma@bdigital.org
@djkram @bdigital

Rafael Gimenez – Scaling up in a world of geolocated data While the implementation of analytic operations on distributed computing frameworks has been widely describing, enabling the computational core of a Big Data system with capabilities for supporting geospatial querying on data is yet a challenging issue. This session aims to target that specific aspect by reviewing how researchers at BDigital Technology Centre have designed and implemented a stack for advanced Machine Learning on Urban Data by providing a way to geoquery massive amounts of HDFS data from Spark processes without hindering the overall system performance..The geospatial dimension of data is getting revealed as the most natural, powerful and intuitive way to explore the expanding world of data and services. The ability to rely on real-world axis such as places, people, events and things can provide better answers for everyday tasks for individuals, as well as a deep understanding for businesses and administrations.The Urban Data Analytics team at BDigital research efforts are focused on that scenario, with an offering built upon the ability to rapidly deploy pre-defined (but also arbitrary) analytic functions on geospatial time-series of data. Currently available developments already provide support for characterization, classification, clustering, anomaly detection and trajectory mining, while multivariate analytics and predictive functions (both on single and combined time-series) are targeted for the near future.In order to enable such analytic operations on geospatially enabled data, the underlying computational infrastructure must provide the distributed computational processes with a tool to support large-speed and highly dynamic geoquerying operations on massive amounts of data. The combination of end-to-end geoquerying components and GIS enhancements for HDFS data has been implemented and tested by BDigital as the most promising solution for such requirements.

The geography of geospatialTodd Barr

This document discusses the rise of geospatial data science and the differences between geospatial data science and traditional GIS. It notes that everyday 2.5 quintillion bytes of data are generated, 80% of which has a geospatial component. Geospatial data science bridges data science and GIS by treating location as core to the data. This field has grown due to increasing data availability, geospatial libraries, cloud computing, and the need to combine GIS and data science skills. The future of geospatial data science involves increased AI/ML use, more industry applications, and significant market and job growth.

Using Geospatial to Innovate in Last-Mile LogisticsCARTO

The document discusses how geospatial technologies like what3words and CARTO can help optimize last-mile logistics and delivery. what3words assigns a unique 3-word address to every 3m x 3m area in the world to simplify addressing. CARTO provides spatial data and analytics to optimize routing and delivery. Examples are given of companies using these technologies together to improve delivery by integrating 3-word addresses, optimizing routes using real-time traffic and weather data, and analyzing spatial data from CRM systems. This can lead to improvements like reduced costs, increased coverage, and fewer failed or late deliveries.

How to become a Spatial Data Scientist?CARTO

Google Analytics location data visualised with CARTO & BigQueryCARTO

The Role of Data Science in Real EstateCARTO

Are you a Data Scientist working in the Real Estate industry? Are you trying to build a Data Science team with expertise in spatial? We bring the London Real Estate Data Science community together to discuss use cases such as whitespace analysis, twin area analysis & indoor analytics - sharing best practices and experiences from both residential and commercial. Jaime Sanchez walked through a specific case of Spatial Data Science applied to Shared Workspace investment analysis, with an interactive component, before we break out into a discussion about the challenges and opportunities of building Data Science teams in the Real Estate sector. Geolytix joined the conversation to speak about location planning. As trusted advisors, they help their customers decide how many stores, who to acquire, where to open, which format and how to optimize home delivery and click & collect operations. Visit our website for more information: https://carto.com/

The Sum of our Parts: the Complete CARTO Journey [CARTO]CARTO

Why High-Resolution Spatial Data on Population MattersCARTO

The Role of Indoor Mapping in the “New Normal”CARTO

Tips and tricks for Working with Demographic Data [CARTOframes & Python]CARTO

Unlock the power of spatial analysis using CARTO and python [CARTOframes]CARTO

How to Use Spatial Data Science in your Site Planning Process? [CARTOframes] CARTO

Le rôle de l’intelligence géospatiale dans la reprise économiqueCARTO

Encuentro Aporta 2016 - Mesa 2 - Miguel AriasDatos.gob.es

This document discusses Carto's open source and open data approach to business decision making. It highlights that by 2020, 25 billion devices will generate location data, and that location unlocks insights from big data. Carto uses open source code and open data to democratize location intelligence and build communities. This allows for faster development at lower costs. The future is in services rather than code lines. Carto has over 250,000 users creating over 5,000 visualizations per month, and offers tools like Builder, Engine, and location data services to help organizations leverage location data.

CARTO en 5 Pasos: del Dato a la Toma de Decisiones [CARTO]CARTO

Tb geo dev_presentation_nov_5Todd Barr

This document discusses Todd Barr's background in GIS and spatial data consulting. It then provides information on R, including that it was developed in the 1990s based on S and S-Plus for statistical modeling. It discusses reasons to learn R, such as its community support, reproducibility, versatility, and ability to perform high-end analysis without economic barriers. Finally, it outlines R's use for spatial/GIS applications through libraries, interfaces with other software like ArcGIS and QGIS, visualization capabilities, and ability to create web-ready outputs.

Scaling Spatial Analytics with Google Cloud & CARTOCARTO

Think Spatial: Don't Ignore Location in your Models! [CARTOframes]CARTO

Embedding Location Intelligence in Web Apps that Enhance User Experience [Air...CARTO

This document discusses embedding location intelligence in web applications using CARTO VL and Airship. It introduces CARTO VL as a new map library for fast and flexible vector rendering in web apps. Airship is introduced as a new UI library for intuitive location intelligence apps. The document provides information on the advantages of CARTO VL and Airship, examples of code, and resources for learning more.

Market analysis through Consumer Behavior Pattern InsightsCARTO

Stepping up your Spatial Analysis with Mastercard Retail Location insights 3.0CARTO

The document discusses Mastercard Retail Location Insights 3.0, a spatial analysis tool from Mastercard and CARTO. It provides retailers and real estate professionals insights into questions about e-commerce trends, site consolidation, consumer habits, threats to commercial and residential real estate, and more. The new version includes additional industry and financial data, improved indexes to understand consumer behavior, and greater temporal data granularity. It allows users to analyze factors influencing store performance, understand customer bases and origins, create store forecasts, and provides workflows for business users and data scientists.

Understanding Retail Catchment Areas with Human Mobility DataCARTO

Applying Spatial Analysis to Real Estate Decision-MakingCARTO

Can Kanye West Save Gap? Real-Time Consumer Social Media Segmentation On CARTOCARTO

Kanye West's partnership with Gap led to an initial boost in sales of his clothing line released through Gap. However, the long term impact is unclear as only two products have been released after 18 months. Spatial analyzed customer segments near 95 Gap stores and found the segments most likely to follow Kanye on social media increased their visits to those Gap locations by 3% after the partnership announcement. This indicates Kanye's influence had a short term positive impact on Gap visits for those stores, but the partnership will need to release more products and invest strategically to fully realize its potential to boost Gap's brand over the long run.

Indoor Mapping: Why Precision MattersCARTO

Understanding short term rental & tourism trends through geospatial data [CAR...CARTO

CARTO BUILDER: from visualization to geospatial analysisJorge Sanz

The document discusses Carto Builder, a web-based drag-and-drop tool for geospatial analysis. It provides an overview of Carto's history and the design principles of Builder. Key features of the new Builder version include improved distribution of the sidebar, editing, and map panels, as well as new capabilities for publishing maps, advanced legends, live filtering widgets, and geospatial analyses. The goal is to make location intelligence intuitive and accessible to all users.

Introduction to GISAhmed Ali Rajpoot

Data warehouse modernization programme by TOBY WOOLFE at Big Data Spain 2014Big Data Spain

Intro to the Big Data Spain 2014 conferenceBig Data Spain

More Related Content

What's hot (20)

The Role of Indoor Mapping in the “New Normal”CARTO

Tips and tricks for Working with Demographic Data [CARTOframes & Python]CARTO

Unlock the power of spatial analysis using CARTO and python [CARTOframes]CARTO

How to Use Spatial Data Science in your Site Planning Process? [CARTOframes] CARTO

Le rôle de l’intelligence géospatiale dans la reprise économiqueCARTO

Encuentro Aporta 2016 - Mesa 2 - Miguel AriasDatos.gob.es

CARTO en 5 Pasos: del Dato a la Toma de Decisiones [CARTO]CARTO

Tb geo dev_presentation_nov_5Todd Barr

Scaling Spatial Analytics with Google Cloud & CARTOCARTO

Think Spatial: Don't Ignore Location in your Models! [CARTOframes]CARTO

Embedding Location Intelligence in Web Apps that Enhance User Experience [Air...CARTO

Market analysis through Consumer Behavior Pattern InsightsCARTO

Stepping up your Spatial Analysis with Mastercard Retail Location insights 3.0CARTO

Understanding Retail Catchment Areas with Human Mobility DataCARTO

Applying Spatial Analysis to Real Estate Decision-MakingCARTO

Can Kanye West Save Gap? Real-Time Consumer Social Media Segmentation On CARTOCARTO

Indoor Mapping: Why Precision MattersCARTO

Understanding short term rental & tourism trends through geospatial data [CAR...CARTO

CARTO BUILDER: from visualization to geospatial analysisJorge Sanz

Introduction to GISAhmed Ali Rajpoot

The Role of Indoor Mapping in the “New Normal”CARTO

Tips and tricks for Working with Demographic Data [CARTOframes & Python]CARTO

Unlock the power of spatial analysis using CARTO and python [CARTOframes]CARTO

How to Use Spatial Data Science in your Site Planning Process? [CARTOframes] CARTO

Le rôle de l’intelligence géospatiale dans la reprise économiqueCARTO

Encuentro Aporta 2016 - Mesa 2 - Miguel AriasDatos.gob.es

CARTO en 5 Pasos: del Dato a la Toma de Decisiones [CARTO]CARTO

Tb geo dev_presentation_nov_5Todd Barr

Scaling Spatial Analytics with Google Cloud & CARTOCARTO

Think Spatial: Don't Ignore Location in your Models! [CARTOframes]CARTO

Embedding Location Intelligence in Web Apps that Enhance User Experience [Air...CARTO

Market analysis through Consumer Behavior Pattern InsightsCARTO

Stepping up your Spatial Analysis with Mastercard Retail Location insights 3.0CARTO

Understanding Retail Catchment Areas with Human Mobility DataCARTO

Applying Spatial Analysis to Real Estate Decision-MakingCARTO

Can Kanye West Save Gap? Real-Time Consumer Social Media Segmentation On CARTOCARTO

Indoor Mapping: Why Precision MattersCARTO

Understanding short term rental & tourism trends through geospatial data [CAR...CARTO

CARTO BUILDER: from visualization to geospatial analysisJorge Sanz

Introduction to GISAhmed Ali Rajpoot

Viewers also liked (20)

Data warehouse modernization programme by TOBY WOOLFE at Big Data Spain 2014Big Data Spain

Intro to the Big Data Spain 2014 conferenceBig Data Spain

Big Data Web applications for Interactive Hadoop by ENRICO BERTI at Big Data...Big Data Spain

Dataflows: The abstraction that powers Big Data by Raul Castro Fernandez at ...Big Data Spain

http://www.bigdataspain.org/2014/conference/dataflows-the-abstraction-that-powers-the-big-data-technology Dataflows are an omnipresent abstraction across many big data technologies due to its suitability for representing programs in a way that is easy to parallelize. All dataflow models---such as those of Spark or MapReduce---are stateless, which facilitates achieving fault tolerance, a crucial property when running at large-scale. However, this stateless dataflow models have a negative impact on the programming models they expose, which need to adapt to match the stateless nature of the underlying platforms. With the “democratization of data”, different types of users with different skills want answers from their big datasets, but sometimes they lack the skills required to write programs adapted to these specific frameworks: A familiar programming model becomes crucial to open big data value to a broader set of users. Video https://www.youtube.com/watch?v=blY7EleXW6U

The top five questions to ask about NoSQL. JONATHAN ELLIS at Big Data Spain 2012Big Data Spain

Big Data the potential for data to improve service and business management by...Big Data Spain

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...Big Data Spain

http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks. Session presented at Big Data Spain 2014 Conference 18th Nov 2014 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014

Getting the best insights from your data using Apache Metamodel by Alberto Ro...Big Data Spain

We live in an age of an over-abundance of data stores choices and we also live in an age where the data has more value than ever.We think that data has no value until it’s shown and everybody is nowadays looking for simple ways to get insights from its data through self-explanatory visualizations. Would it be nice if we were able to create awesome graphs regardless the underlying technology that stores our data? Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-12.html

ToroDB: Scaling PostgreSQL like MongoDB by Álvaro Hernández at Big Data Spain...Big Data Spain

ToroDB is an open source document-oriented NoSQL database that uses PostgreSQL as its storage backend. It stores JSON documents by splitting them into subdocuments based on hierarchy levels and storing each subdocument type in a separate PostgreSQL table. This allows ToroDB to scale like MongoDB by querying "by structure" and only scanning relevant subsets of data. Version 0.4 of ToroDB adds support for replicating data from a MongoDB master database to a PostgreSQL database using MongoDB's replication protocol.

Convergent Replicated Data Types in Riak 2.0Big Data Spain

Talk by Gordon Guthrie, Senior Software Engineer at Basho Summary A review of the CAP Theorem and the difficulties of resolving conflicts in highly distributed systems. Covering the issues and various theories on how to resolve including the use CRDTs in Riak Details CRDTs are used to replicate data across multiple computers in a network, executing updates without the need for remote synchronisation. This leads to merge conflicts in systems using conventional eventual consistency technology, but CRDTs are designed such that conflicts are mathematically impossible. Under the constraints of the CAP theorem they provide the strongest consistency guarantees for available/partition-tolerant (AP) settings. The CRDT concept was first formally defined in 2007 by Marc Shapiro and Nuno Preguiça in terms of operation commutativity, and development was initially motivated by collaborative text editing. The concept of semilattice evolution of replicated states was first defined by Baquero and Moura in 1997, and development was initially motivated by mobile computing. The two concepts were later unified in 2011. Basho has worked with the EU and Marc Shapiro's team to push CRDTs into distributed systems. Riak v2.x is the first commercial product to include this functionality

CloudMC: A cloud computing map-reduce implementation for radiotherapy. RUBEN ...Big Data Spain

Big Data, analytics and 4th generation data warehousing by Martyn Jones at Bi...Big Data Spain

Martyn Jones presented on Big Data, analytics and 4th generation data warehousing. He discussed the importance of a comprehensive data supply framework to obtain, integrate and analyze data from various sources to provide strategic, tactical and operational decision making support. He described moving beyond traditional data warehousing to a 4th generation approach that leverages big data and analytics to gain insights and measure outcomes.

Essential ingredients for real time stream processing @Scale by Kartik pParam...Big Data Spain

This document discusses stream processing at scale. It begins with an introduction and agenda. It then discusses scenarios for stream processing like newsfeeds, cybersecurity, and IoT. It presents the canonical stream processing architecture with data buses, real-time and batch processing, and ingestion/serving tiers. The document dives into the essential ingredients for stream processing: scale, reprocessing, accuracy of results, and easy programmability. It provides examples and strategies for each of these essential ingredients to achieve efficient and accurate stream processing at large scales.

Analyzing organization e-mails in near real time using hadoop ecosystem tools...Big Data Spain

How to integrate Big Data onto an analytical portal, Big Data benchmarking fo...Big Data Spain

DKS EAP (Enterprise Analytical Portal) was first thought of as an integrated analytical portal; a conjunction of BI and BA products (ranging from advanced customer intelligence to marketing analytics) assembled under a unified interface, specifically designed to better assist business in their decision-making. We believe DKS EAP is of special interest for the Big Data community nowadays since it is currently leveraging technologies such as Hadoop, Cassandra, Spark and Storm to improve its analytic capabilities. Accordingly, the talk will briefly present DKS EAP as an integrated Big Data analytic environment, focusing on how the fast-growing needs for real-time, social media and geomarketing data analysis intensified its necessity to embrace Big Data technologies. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-8.html#spch9.1

Geospatial and bitemporal search in C* with pluggable Lucene index by Andrés ...Big Data Spain

Stratio presented its open source Lucene-based implementation of Cassandra’s secondary indexes at Cassandra Summit London 2014, which provided several search engine features. It used to be distributed as a fork of Apache Cassandra, which was a huge problem both for users and maintainers. Nowadays, due to some changes introduced at C* 2.1.6, we are proud to announce that it has become a plugin that can be attached to the official Apache Cassandra. With the plugin we have been able to provide C* with geospatial capabilities, making it possible to index geographical positions and perform bounding box and radial distance queries. This is achieved through Lucene’s geospatial module. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-9.html

IAd-learning: A new e-learning platform by José Antonio Omedes at Big Data Sp...Big Data Spain

Purpose of the talk: Describing the use of Machine Learning and Big Data Techniques to improve the performance of elearning students. Presenting an existing case of an elearning platform (iAdLearn¡ng) and the technology used behind the scenes, to make adaptive/high performance elearning a reality. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-17.html

Data Science in 2016: Moving up by Paco Nathan at Big Data Spain 2015Big Data Spain

This document discusses trends in data science in 2016, including how data science is moving into new use cases such as medicine, politics, government, and neuroscience. It also covers trends in hardware, generalized libraries, leveraging workflows, and frameworks that could enable a big leap ahead. The document discusses learning trends like MOOCs, inverted classrooms, collaborative learning, and how O'Reilly Media is embracing Jupyter notebooks. It also covers measuring distance between learners and subject communities, and the importance of both people and automation working together.

A new streaming computation engine for real-time analytics by Michael Barton ...Big Data Spain

Apache Spark has successfully built on Hadoop infrastructure to encompass real-time processing, moving from rigid Map-Reduce operations to general purpose functional operations distributed across a cluster of machines. However data storage has become a black box. The source data for a query has to be retrieved in full and sent through the analysis pipeline rather than processing the data where it is stored, as in traditional database systems. This introduces significant cost, both in network utilisation and in the time taken to produce a result. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-14.html

Processing large-scale graphs with Google(TM) Pregel by MICHAEL HACKSTEIN at...Big Data Spain