MongoDB.local Austin 2018: PetroCloud: MongoDB for the Industrial IOT Ecosystem

Editor's Notes

• #3: Hi, my name is Luis Lobo Borobia, I have been working for PetroCloud since August 2013. PetroCloud is a Real-Time Oil Field Automation and Security Platform, focused primarily in Oil and Gas, and Electric utility verticals! There are other verticals in the works as we speak. It all started in 2012. Lance White had some oilfields he needed to monitor but their location were not convenient at all. He wanted to see the people working on it, as well as checking some values from the tank. He then contacted a friend, Martin Apesteguia, who was a co-worker from a previous company, and started the project. Later on, PetroCloud was born as a company. Martin then later on called me to join the project in August 2013. At that time PetroCloud had its very first customers, and we were a very small start-up, with 4 developers. Long time has passed (I know, just 5 years, but that is like... an eternity for Software Projects, right?) and now we are a solid company. We’ve had a big capital injection in December 2017, and we keep growing each month.
• #4: So, to give you some heads up, we will be talking about: Industrial IOT implementation example, where we were, where we are now Moving from MySQL to MongoDB and recently to MongoDB Atlas And about a new tool I developed for tracing MongoDB operations
• #5: Back in the days, we were working with Python inside our Router, PhP for our Web Application, and MySQL as our database engine. At that time, the current concepts of IOT, mesh networks, edge devices, were not something that everyone talked about. We new what we wanted to do, and tried to combine the right hardware elements with the right software components to get our job done. And we succeeded to a point, where we were happy and started having our first customers.
• #6: But we knew that the software stack that we had was not one that would scale or be flexible enough for us to grow to what we have envisioned. With our current stack, it was hard to connect to different devices and protocols We had too many languages for the few developers, and if we were to grow, we would prefer that any developer could work at any level of our software stack, at least be good at reading it. Our current application was a Static web application, server side rendered, not very flexible. We knew we had to re engineer and rethink our stack and by the end of 2013 we started the process
• #7: Key factors during our re engineering process # Integration Basically, for us to grow in this field, we need to *integrate to whatever device* or feature our customers need. That’s how we create new features: our customers drive our business. Be able to handle all the different protocols used by Industrial devices. # One language to rule them all Also, we were developing in four languages: Python, Javascript, PHP and SQL. We were really small, we wanted new talent. Our software stack was PHP based. Nothing wrong with that, I still love PHP!, but it was using Code Igniter. Yeah, I hear you. Not the best MVC based framework around, but it worked for us at the beginning. We wanted to focus in developing in one single language, so that anyone in the team could jump into Front-End development, Backend-API development, Device development. And that language was Javascript, and the data format was JSON. Our chosen solutions were and still are: MongoDB as our platform database, Node.js with Sails.js as our platform API and RCU software, and finally Angular JS (we are migrating to Angular) as our Frontend development framework.
• #8: Key factors (continued…) # People At the end, it's all about the people. Most of the developers we were in contact with at the time were all looking at Angular JS opportunities, Node.js was the new kid on the block. We wanted to be in a platform that allowed us to attract great developers, and to be appealing. And let's be honest, would you go back and develop apps in Code Igniter? ### Platform Finally, we wanted to build a platform, not just a "web page". And for that, we wanted to have an independant API, a SPA Responsive Web Application, and our RCU (Remote Computing Unit) software, and Video jobs processsing
• #9: At that time, I have had previous experience with some NoSQL databases, and I always liked these features about MongoDB: - High Availability is super easy with Replica Sets - Almost zero maintenance (compared to other database infrastructures) - JSON and Javascript friendly - Schema(less) flexibility
• #10: # Parallel System We had our PHP application, moved it into maintenance mode. That was the live App users had. Then we started developing in parallel the new Node.js based API, with a new front end built on top of Angular.js. We started creating first the new skeleton and then, in an iterative process, creating the different features we had in our legacy PHP application. One of the things that we implemented in this new API and Frontend was the extensive use of Websockets. When we matched the same features we had in our app, we rolled it over, and removed access to the old system. We kept it for a while, just in case there were any events that needed to be accessed but at some point we finally killed it. All new development was done in the new software stack.
• #11: After some (painful) months, we ended up migrating our system from PHP, Python, MySQL, to a platform 100% javascript based. Choosing a Node.js framework at that time was not that easy. I've always liked MVC frameworks. I was delighted working with Yii Framework, one of the best PHP frameworks I know. And I wanted something similar for our API. After some research, we chose Sails.js. It has an ORM, called Waterline, a nice structure to work on, and it let us focus on developing the software solution rather than thinking on "how to do the things right in Javascript". Waterline has a very nice adapter for MongoDB, and it really works great for us. We have been contributors for that project ever since, improving both Sails and their MongoDB adapters when possible/necessary. I can tell that these last years have been, technologically one of the most fun in my life. We have a solution that can drive tons of different disparate devices from different brands, and all integrated into a single Platform.
• #12: MongoDB has been such a great tool for us, we think of it as one of the most important parts of our solution. You know why? Because we almost never hear about it! In the last 4 years, the only time it's been down, was because of human error. Actually, because of one error. Actually because of me. Yeah. I messed up with the database, at the beginning. The good thing is that one thing I always pushed forward is having a great backup mechanism. Even paranoid. Since we were happy MongoDB Cloud users were were back in business really quick. Also, because of how our infrastructure is designed, we do at the edge event storage when the platform is unreachable.
• #13: MongoDB has been such a great tool for us, we think of it as one of the most important parts of our solution. You know why? Because we almost never hear about it! In the last 4 years, the only time it's been down, was because of human error. Actually, because of one error. Actually because of me. Yeah. I messed up with the database, at the beginning. The good thing is that one thing I always pushed forward is having a great backup mechanism. Even paranoid. Since we were happy MongoDB Cloud users were were back in business really quick. Also, because of how our infrastructure is designed, we do at the edge event storage when the platform is unreachable.
• #14: One of the things we wanted to have since our re-engineering process was to have a (near) real-time system. Today, if an event happens, you event panel populates with a very basic event that is augmented as the information about it increases. For example, we get a tank level high event. You can immediately see the data about it, but the video footage takes a little longer to process, so you first get the data, then the event is augmented with media information that comes from the cameras around the area where the tank is located. This is possible because our platform is getting the information from our remote RCU, to the platform, to the browser and jobs services that processes our snapshots and videos. All of this is built into our different device, API and front end components.
• #15: Because of the nature of our business, we have so many different devices reporting data, we really need a very flexible schema. MongoDB excels at it. We have a mixture between flexible and fixed schema. With Sails.js, specifically Waterline, you can design your Models to have a structure. And that is what we do. We have a specific Event structure. But we also have our device reported data into one attribute that can be anything. In yellow you can see attributes that are part of the schema of our event collection, and in bright green the ones that are not defined in our event collection schema. What it’s in data is flexible and it depends on the equipment type. From sensors readings: temperature, oil levels, audio noise decibels, to information about videos and snapshots. It can let us accomodate to any situation, as needed.
• #17: The second I heard of Change Streams I said that! Or something similar... I could not believe my eyes... That is something I would have liked to have so many years ago! We have close to a thousand RCUs, about 3.5K industrial devices/features managed, all sending information about what's relevant to them. We use websockets to keep all of this updated. But... I always wanted to have a tool during the development process that would allow me to see any change in the database as soon as it happens, and being able to filter it... It wasn't possible at the time... until now!
• #18: The second I heard of Change Streams I said that! Or something similar... I could not believe my eyes... That is something I would have liked to have so many years ago! We have close to a thousand RCUs, about 3.5K industrial devices/features managed, all sending information about what's relevant to them. We use websockets to keep all of this updated. But... I always wanted to have a tool during the development process that would allow me to see any change in the database as soon as it happens, and being able to filter it... It wasn't possible at the time... until now!
• #19: ## MongoDB Tracer Let me introduce a tiny super very alpha version of a tool I've been working on the last weeks. I had the idea of building it since the second I read about Change Streams, but never had the time/excuse to build it. So, thank you MongoDB .local, you were the perfect excuse. So here it is: a tool that will show you the documents as they come in right from the database, as a tracer tool, a real-time tool that is user friendly, with filters and all the whistles. Let me show how it works. [demo] (Open up webapp) First and foremost, remember, this is an alpha version, bear with me. Now, let's get into it: - we need to provide a connection URI to the database we want - Once you do that, you need to select a collection. - Let's add this 'event' collection. - As you can see... nothing happens... gosh... - But that is intended! We need to start feeding events now. - I have built this simple web poll, that will send events to the same database DA-Tracer is connected to. - I encourage you to go to https://poll.luislobo.xyz and answer the poll so that we can start seeing some results - And all of this is right from the database - It uses Sails.js, WebSockets, VUE - And the best of it… It’s open source under a MIT license.
• #23: ## MongoDB Tracer Let me introduce a tiny super very alpha version of a tool I've been working on the last weeks. I had the idea of building it since the second I read about Change Streams, but never had the time/excuse to build it. So, thank you MongoDB .local, you were the perfect excuse. So here it is: a tool that will show you the documents as they come in right from the database, as a tracer tool, a real-time tool that is user friendly, with filters and all the whistles. Let me show how it works. [demo] (Open up webapp) First and foremost, remember, this is an alpha version, bear with me. Now, let's get into it: - we need to provide a connection URI to the database we want - Once you do that, you need to select a collection. - Let's add this 'event' collection. - As you can see... nothing happens... gosh... - But that is intended! We need to start feeding events now. - I have built this simple web poll, that will send events to the same database DA-Tracer is connected to. - I encourage you to go to lobo.now.sh and answer the poll so that we can start seeing some results - And all of this is right from the database - It uses Sails.js, WebSockets, VUE - And the best of it... We are open sourcing it under a MIT license.