Building a Graph-based Analytics Platform
31 likes19,118 views
Meetup is a valuable source of data for understanding trends around products or brands. Meetup does not support an analytics package to track group statistics overtime unless you are an administrator of a group. There are no third-party tools or websites that analyze Meetup trends to understand how communities grow. In this talk I will present a graph-based analytics platform that uses the Meetup.com API to collect and analyze membership statistics over time. This talk will cover: How to poll and import periodic data from the Meetup.com API into Neo4j using Node.js. How to track meetup group growth over time using a Neo4j graph database using Node.js. How to apply tags to meetup groups and report combined growth of all groups over time. How to build an interactive documented analytics API to support applications using Node.js and Neo4j. How to build a business dashboard to visualize time-based statistics and reports using a Node.js based REST API that queries Neo4j.
![(graphs)-[:are]->(everywhere)
Building](https://image.slidesharecdn.com/buildingagraph-basedanalyticsplatform-140423000708-phpapp02/85/Building-a-Graph-based-Analytics-Platform-1-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Building a Graph-based Analytics Platform (20)
![[Webinar Deck] Google Data Studio for Mastering the Art of Data Visualizations](https://cdn.slidesharecdn.com/ss_thumbnails/webinardeckgoogledatastudioformasteringtheartofdatavisualizations-170830111718-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Kenny Bastani (13)
Ad
Recently uploaded (20)
Building a Graph-based Analytics Platform
- 2. a
- 3. graph-based
- 4. analytics
- 5. platform © All Rights Reserved 2014 | Neo Technology, Inc. @kennybastani Neo4j
- 6. Developer
- 7. Evangelist
- 8. Using Meetup as an example use case Meetup.com is a valuable source of data for understanding trends around products or brands. Understanding demand is key for delivering compelling content at meetups. It sounded like a great use case for Neo4j.
- 9. The Problem Track meetup group growth over time. Apply tags to meetup groups and report combined growth of all groups over time.
- 10. Questions
- 11. Question #1 Given a start date and an end date, what is the time series that plots the membership growth of a given meetup group?
- 12. Question #2 Given a start date, an end date, and a combination of tags, what is the time series that plots the combined membership growth of all meetup groups with those tags?
- 13. Question #3 How do you generate the JSON data of a time series for a basic JS line chart plugin?
- 14. The Goal
- 15. The GraphGist Project The GraphGist project is a way to quickly build a graph-based proof of concept on Neo4j. I started with a GraphGist. Neo4j for Graph Analytics: Meetup.com Example
- 16. Graph Data Model
- 17. How are groups connected?
- 18. How are locations connected?
- 19. How are tags/topics connected?
- 20. How are stats connected?
- 21. How are days connected?
- 22. How are weeks connected?
- 23. How are months connected?
- 24. How are years connected?
- 25. Tackling Time in Neo4j How do you implement a time series in Neo4j? For any node that represents a unit of time, use a timestamp. Traversals can be costly for selecting time series. Expose a REST API that takes a normal date format and then convert it to an integer that allows you to select a range of dates in your Neo4j Cypher query. For any node that represents a unit of time, use a timestamp. Traversals can be costly for selecting time series. Expose a REST API that takes a normal date format and then convert it to a Int32 that allows you to select a range of dates.
- 26. Scale it up! It started with a GraphGist and then I said “Why not?” let’s build something cool using Neo4j.
- 27. Challenges I decided to take my GraphGist and make a full platform. There were some challenges.
- 28. Challenge #1 How do I get historical Meetup group statistics for all groups?
- 29. Challenge #2 How do I handle the data import on a daily basis?
- 30. Challenge #3 What kind of reports do I want to create? What do I want to know about Meetup groups?
- 31. Challenge #4 How do I safely expose Neo4j to a client-side charting control?
- 32. Ask Questions I decided to start asking some questions about my data model.
- 33. What do I want to know? Assuming I had as much historical Meetup data as I pleased, what kind of questions would I want to ask about that data? How would I want to present it?
- 34. What’s the combined growth percent of Meetup groups having a certain topic? This chart plots a line chart of the time series for a meetup group topic on Meetup.com. Each group on Meetup.com has a set of topics associated with it. This chart is meant to show the percent growth month over month.
- 35. What’s the cumulative growth of Meetup groups with a specific topic? This chart plots a bar chart of the cumulative growth of a meetup group topic on Meetup.com. Using the time series data of monthly growth from the Meetup Tag Growth % chart, the growth percents over the period are aggregated into a sum for each topic. This chart shows total growth percentage over the period.
- 36. What’s the relative growth of Meetup groups with a topic for a date range? This chart plots an Donut Chart of the relative cumulative growth of a meetup group topic on Meetup.com. Using the data from Cumulative Meetup Growth, the percentage growth of each topic over the period is compared relative to one another as a ratio of 100.
- 37. How many groups does a topic have relative to others? This chart plots an Donut Chart of the number of groups in the region during the period for each topic. Each group is compared relative to one another as a ratio of 100.
- 38. What’s the growth percent of all groups for a topic in a location for a date range? This report is a simple table that shows the growth percent of all groups for a topic broke down by location. What do these high percentages tell us about Meetup? Within the last year there has been massive growth for meetup groups that are focused on NoSQL database technology. If I imported a different topic, not related to technology, what would the data show?
- 39. How do I give users a clean set of controls to filter and search?
- 40. Scaling it up Designing a graph-based analytics platform using Node.js and Neo4j
- 41. Architecture Front-end web-based dashboard in Node.js and bootstrap REST API via Neo4j Swagger in Node.js Data import services in Node.js Data storage in Neo4j graph database
- 42. Applications Analytics REST API (Node.js) Dashboard (Node.js) Analytics Data Import Scheduler (Node.js) Web Web Console
- 43. Neo4j (JVM) REST API (Node.js) Dashboard (Node.js) Import Scheduler (Node.js) Polls Meetup API Graph Data Storage Analytical Queries Presentation, Filtering FilterQuery Import Web App Web App Retrieves Report Data Visualizes Report Data
- 47. REST API The REST API is a fork of Neo4j Swagger. Swagger is a specification and complete framework implementation for describing, producing, consuming, and visualizing RESTful web services.
- 49. Swagger The REST API module of this project is based on a fork of Swagger.
- 50. The Neo4j Swagger Project The Swagger project was modified to use Neo4j as its data source. The REST API module of this project is extended from the Neo4j swagger project.
- 51. REST API Methods Get Weekly Growth Get Monthly Growth Get Monthly Growth By Tag Get Monthly Growth By Location Get Cities Get Countries Get Group Count By Tag
- 52. Get Weekly Growth Gets the weekly growth percent of meetup groups as a time series. Returns a set of data points containing the week of the year, the meetup group name, and membership count.
- 53. Get Monthly Growth Gets the monthly growth percent of meetup groups as a time series. Returns a set of data points containing the month of the year, the meetup group name, and membership count.
- 54. Get Monthly Growth By Tag Gets the monthly growth percent of meetup group tags as a time series. Returns a set of data points containing the month of the year, the meetup group tag name, and membership count.
- 55. Get Monthly Growth By Location Gets the monthly growth percent of meetup group locations and tags as a time series. Returns a set of data points containing the month of the year, the meetup group tag name, the city, and membership count.
- 56. Get Cities Gets a list of cities that meetup groups reside in. Returns a distinct list of cities for typeahead.
- 57. Get Countries Gets a list of countries that meetup groups reside in. Returns a distinct list of countries for typeahead.
- 58. Get Group Count By Tag Gets a count of groups by tag. Returns a list of tags and the number of groups per tag.
- 59. Analytics Dashboard The dashboard is a web application that uses client- side JavaScript to communicate with the Neo4j Swagger REST API to populate a series of interactive chart controls with data. This web application uses bootstrap for the front-end styles and highcharts.js for the charting controls.
- 61. Reports Meetup Tag Growth % Cumulative Meetup Growth Category Growth % Groups By Tag Meetup Tag Growth By Location
- 62. Meetup Tag Growth % https://github.com/kbastani/meetup-analytics/blob/master/docs/DOCS.md#meetup-tag-growth- This chart plots a line chart of the time series for a meetup group topic on Meetup.com. Each group on Meetup.com has a set of topics associated with it. This chart is meant to show the percent growth month over month.
- 63. Cumulative Meetup Growth https://github.com/kbastani/meetup-analytics/blob/master/docs/DOCS.md#cumulative-meetup-growth This chart plots a bar chart of the cumulative growth of a meetup group topic on Meetup.com. Using the time series data of monthly growth from the Meetup Tag Growth % chart, the growth percents over the period are aggregated into a sum for each topic. This chart shows total growth percentage over the period.
- 64. Category Growth % https://github.com/kbastani/meetup-analytics/blob/master/docs/DOCS.md#category-growth- This chart plots an Donut Chart of the relative cumulative growth of a meetup group topic on Meetup.com. Using the data from Cumulative Meetup Growth, the percentage growth of each topic over the period is compared relative to one another as a ratio of 100.
- 65. Groups By Tag https://github.com/kbastani/meetup-analytics/blob/master/docs/DOCS.md#group-count-by-tag This chart plots an Donut Chart of the number of groups in the region during the period for each topic. Each group is compared relative to one another as a ratio of 100.
- 66. Meetup Tag Growth By Location This report is a simple table that shows the growth percent of all groups for a topic broke down by location. What do these high percentages tell us about Meetup? Within the last year there has been massive growth for meetup groups that are focused on NoSQL database technology. If I imported a different topic, not related to technology, what would the data show?
- 67. Filter Search
- 71. © All Rights Reserved 2014 | Neo Technology, Inc. (Next
- 72. steps) “What are the next steps?”
- 73. © All Rights Reserved 2014 | Neo Technology, Inc. Get
- 74. in
- 75. touch Twitter: @kennybastani LinkedIn: /in/kennybastani Email: [email protected] “Get in touch with me about meetups and Neo4j community events happening around the world.” ! “I’ll now open up the floor to questions.”