MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB
3 likes1,689 views
The document discusses winning the Dreamforce hackathon using MongoDB, focusing on rapid prototyping and innovative applications like Droney Express. It emphasizes the importance of agility, team collaboration, and leveraging modern tools to manage data effectively, particularly in government and social activism contexts. Additionally, it critiques traditional ORM solutions, arguing for the superiority of MongoDB in handling complex object graphs and its suitability for IoT and GIS applications.
![if (! Links.find().count()) {
// init data
var nodeNames = [];
var data = ['Harry,Sally,1.2',
'Rubio,Mario,1.3',
'Obama,Barrack,0.2',
'Thompson,Fred,0.5',
'Peter,Johan,0.7'];
_.each(data, function (val) {
var comp = val.split(',');
Links.insert({ source: comp[0], target: comp[1], compute: comp[2] });
nodeNames.push(comp[0]);
nodeNames.push(comp[1]);
});
// remove duplicates
nodeNames = _.uniq(nodeNames);
_.each(nodeNames, function (name) { Things.insert({ _id: name }); });
}
Finesse by using new tools](https://image.slidesharecdn.com/tegbiz-mongodbdaysfinalpreso20151203finalsubmit-151217161003/85/MongoDB-Days-Silicon-Valley-Winning-the-Dreamforce-Hackathon-with-MongoDB-45-320.jpg)
![Droney Heartbeatmodeled from mongoDB University
Airborne MongoDB Prevent Election Of New Primary
Droney Flight Computer Always Have Priority Of 0
They Never Get Elected To Primary
[Priority, Optime, Connections]
Sync Using Oplog To HQ After Weeks In Field
Network Environment Spotty For MongoDB Servers](https://image.slidesharecdn.com/tegbiz-mongodbdaysfinalpreso20151203finalsubmit-151217161003/85/MongoDB-Days-Silicon-Valley-Winning-the-Dreamforce-Hackathon-with-MongoDB-62-320.jpg)
MongoDB Days Silicon Valley: Winning the Dreamforce Hackathon with MongoDB
- 1. MongoDB Days Silicon Valley Winning the Dreamforce Hackathon with MongoDB
- 2. db.mongoDaysSV.insert( { “init”: “Welcome Message”, “author”: “Greg Deeds”, “twitter”: “@TEGBIZ”, setInterval: 162000 } );
- 4. You want to change the game? You have 24 hours...
- 5. Two True Tales Of MongoDB MongoDB & Force // GR • API driven rapid prototyping • HEROKU CONTAINERIZATION • MASS IMPORT DATA SCIENCE DRONEY EXPRESS // MongoDB your range • Geospatial & Autonomous
- 7. Winning with MongoDB • Developing Rapid Prototypes • Winning Hackathons with MongoDB • Drones / MongoDB {Range Management}
- 8. MongoDB or stay in the RDBMS Matrix
- 9. Give Up The Old And Win • Traditional DevOps Practitioner moved to Micro-services {Microsoft .NET & SQL Server} {Oracle APEX-PL/SQL & Oracle Server} {Any Linux... MySQL/POSTGRES} • Teamwork to overworked evolutions • My SQL Dreams never materialized - {WHATS NEXT!} • AGILITY TRAIN YOUR MIND AS A LIFESTYLE
- 10. The Early Days
- 11. MY BACKGROUND • Started in 1979 {Ti99/Tape Deck} • Wrote first database on Commodore • BBS Author/ISP {One of the young ones} • Multimedia Producer // Software Engineer • Worked for corporations to startups { Military / Federal / Education } • The Writing is on the Wall - MongoDB Time!
- 15. Languages & Coding {Berkeley Logo}
- 16. Languages & Coding {Learning to Communicate}
- 17. Languages & Coding {Standing on the Shoulders of Giants to Communicate}
- 20. PROBLEM: The data is available but in various formats, not easy to use and is changing • Accessing your government should not be data science or outside of SalesForce.com • PLATFORM HEROKU - {the app processing grid}
- 21. Our target user ranges from the socially conscious individual to Publicly Traded Companies and Governments. Our comprehensive tool will allow… • Grassroots Organizations, Associations • Publicly Traded Companies • Lobbyists, PACs and Governments Manage information • Up-to-Date Congressional Legislator Directories • Congressional / State Legislation Tracking • Easy Meetings Management • Bill and Amendments Tracking • Committees and more We are most proud of the tool because it allows both individuals and organizations to understand the landscape of our Government, politics, and social activism.
- 22. MongoDB & Droney Express
- 23. ForceGR / SPEC • FORCE.COM APP Generator • APEX & PL/SQL Classes • Restful NODEJS HEROKU • Mongo for Import, Reduce and Production
- 28. Data Sets Matter • Variety Of Formats • YAML // XML // JSON // CSV // TAB • Normalized only for XML / DTD files. • Multiple websites crawled/parsed {Using RSYNC || PHP To Bulk Load Objects} • Aggregate Pipeline • Build Words / Terms from triples
- 29. • Semantic to understand {What you say} • Latent Semantic to understand {What you mean} • MongoDB // Provides both when used in pipeline with other word bag or terminology generators • The results are we find correlation in unnormalized data Data Miner Mongo DB
- 34. • Monitor the query in MongoDB reactively {Check to see how the indexes need to improve with a statistics collection of performance and counters for the API} • Compound indexes need to be built on geojson first then other fields when used • Consider using elastic search to boost the Hackathon Team to victory if free text is slow Use Conditional Indexes {add Agility_new ways working with this data?}
- 35. • Find source files over APIs // { to maintain high speed with no throttles. Hackathon bandwidth is varied sometimes when you need it. It gets worse with time } • Use Old School Shell Tools // { NodeJS is not ideal for all challenges } • PHP • PERL • PyThon • RSYNC MISSION: Data Capture
- 36. • The focus of TEG’s FORCEGR tool is to present information into SalesForce users Desktop • Then to make decisions about responses to these decisions • Further guiding the NodeJS learning engine APEX CLASSES / INTERACTIONS
- 37. question = (my_department, has, what) knowledge = ( (I, own, my_apartment), (my_department, has, my_computer), (my_department, has, my_vote), (my_department, is_in, Philadelphia) )
- 38. ‘$push’ ‘$pop’ ‘$unset’ ‘$set’ bye bye SQL... Sew the Dataset
- 39. The Team
- 40. • .NET/Oracle APEX/PLSQL is heavy Spiral DEVOPS • The Browser Wars- We Won {HTML5/Javascript Rocks} • SUBVERSION TO GIT DEVOPS TO NEW CHATOPS {Your Dated If You Can’t GIT With It…} • Shifts in web computer science • Multitouch surface the GPU and JavaScript are the next big shift Recognizing Shift
- 41. • Practice writing specifications for NOSQL • Use New tools to trap ideas • CHECK YOUR ROADMAP? { are you on course & time? } • Gain momentum by solving problems quicker • GIT more done { Practice like you play the game } • Organization to Reproduce Results {use automated ‘Agents’ when possible & makes sense} {containerize your app with docker or Heroku} Next Generation DevOps
- 42. question = (my_department, has, what) knowledge = ( (I, own, my_apartment), (my_department, has, my_computer), (my_department, has, my_vote), (my_department, is_in, Philadelphia) ) for each statement in knowledge { if ((statement.subject == question.subject or question.subject == what) { and (statement.predicate == question.predicate or question.predicate == what) and (statement.object == question.object or question.object == what)) call FoundAnswer(statement) } } Output: Answer: my_department has my_computer Answer: my_department has my_vote Pseudocode for Question- Answering Logic in NodeJS
- 44. • Ask yourself on each project freshly {What else is aggregation good for?} • HINT: You can use the $out pipeline operator and it will create or replace the collection it spits out, after the results are ready.... Aggregation Tips { Use the Pipeline, Luke! }
- 45. if (! Links.find().count()) { // init data var nodeNames = []; var data = ['Harry,Sally,1.2', 'Rubio,Mario,1.3', 'Obama,Barrack,0.2', 'Thompson,Fred,0.5', 'Peter,Johan,0.7']; _.each(data, function (val) { var comp = val.split(','); Links.insert({ source: comp[0], target: comp[1], compute: comp[2] }); nodeNames.push(comp[0]); nodeNames.push(comp[1]); }); // remove duplicates nodeNames = _.uniq(nodeNames); _.each(nodeNames, function (name) { Things.insert({ _id: name }); }); } Finesse by using new tools
- 46. VICTORY It never lasts long enough....
- 48. • HTML5 & WebGL ODM Tools • MongoDB Powering Geoserver Raster • Realtime 3-D Model Scenes In A Browser Hibernate ORM is Dead {long live MongoDB ODM}
- 49. Hibernate ORM VS MongoDB ODM • Hibernate Maps Java Objects To RDBMS And Is Proven Not Scalable • RDBMS load times for a single object graph (Hibernate/SQL Server) from 3 to 4 seconds • Relational model is not suited for complex object graphs. • Even with massive parallel processing is not economical Hibernate Search • Cannot Beat Elastic-search & MongoDB In Response Speed Business Opportunity
- 50. • ORM Hibernate JBOSS - Object/Relational Model {storage has reached the end of life period…} • MongoDB is the best object store on the market today • Hibernate is complex when the world needs BSON • MongoDB is Built For Replication • Hibernate Replacement Is An Opportunity. Market Need {MongoDB Sensor ODM}
- 51. Project Requirements • Open Source/No Internet/Limited WiFI • Runs on variety of low cost hardware • Auto-deploys Via Docker On IoT Appliance • Plugin To Existing GeoServer/ESRI/GRASS • Replace Hibernate With MongoDB Droney Express {Range management System}
- 52. • Interchange GeoJSON set of Shapes & Waypoints • RasberryPi Flight Computer running 2d Queries • Objects Linked To External GIS Engine (GRASS/ESRI) • AutoPilot Configuration In Flight/Over-The-Air • MongoDB Monitors Situational Awareness Of Missions • GIS Query Results Stream As BSON Objects What is a Mission?
- 53. Geographic Forensics {sensor data store/search}
- 55. The work of Droney Express • Planned Geofence Scanning • 2D/3D indexed sensor data • GIS & MongoDB Is Powerful • Integrated GIS Remote Monitoring • Connect Using DDP To Sync Collections Adhoc
- 57. Async Data Sync -> Use Oplog For Synchronization Serve GIS markup with pretty maps using Geoserver... MongoDB is perfect for Sensors // sorry Oracle
- 58. Flight Plan For Best Routes GIS Engines Accelerated By MonogDB Can Handle more ecological range management sensors faster coordinated ground and drone assets
- 59. GEOJSON / MONGODB 2d DRAINAGE MODELING
- 60. GIS ENGINE & PLUGIN Droney Express Flight Planner Least Cost Power & Route Calculator Export Tools For 3DR Ardupilot.
- 61. Replica Sets Are Ideal For Network Segregation HQ is Always Primary - Drones Always Secondary Force Constant State of Disaster Recovery - Dedup Drones Are Small Autonomous Collection Vehicles Eventual Consistency In Loosely Coupled Networks MongoDB Makes Life Easy Secondaries DeDup Secondaries To Update HQ GIS Backends For Mission Planning & Sensor Datasets
- 62. Droney Heartbeatmodeled from mongoDB University Airborne MongoDB Prevent Election Of New Primary Droney Flight Computer Always Have Priority Of 0 They Never Get Elected To Primary [Priority, Optime, Connections] Sync Using Oplog To HQ After Weeks In Field Network Environment Spotty For MongoDB Servers
- 63. Statement Replication ...Getting The Data To HQ Droney Acquires Forensic Sensor Data - ‘A Data Silo’ Facilitates Greater Independence Replicate Based Oplog For Forensic Time-coding Member Replica Sync Stored Records Using Oplog Use Oplog To Journal AutoPilot For Flight Review MongoDB Secondaries Apply Unique ID Vs Primary Keeps Mobile Workers From Overwriting Datasets
- 64. Cheap IoT Devices MongoDB 32bit Flight Computers & Range Sensors With Embedded MongoDB In Flight Computers Are Low Power/LINUX Threaded Sensors Streaming Data To/From MongoDB Autopilot Flight Recorder Is MongoDB/NodeJS IoT SD Cards - Ideal For Circular Buffers In Flight Circular Buffers Makes Room For New Documents Overwrite The Oldest Documents In A Collection
- 65. Final Thought.. NO CAMPING … KEEP CLIMBING! CONSIDER THE FUTURE OF YOUR TIME STUDY NEXT GENERATION DEVOPS SAY “WHY CANT WE FIRST” SHARPEN UP REGULARLY TEST YOUR SKILLS OR TEACH A SKILL
- 66. Greg Deeds Technology Exploration Group [email protected] www.TEGbiz.com @TEGbiz @DRONEYEXPRESS @3dVRviewers