Migrating from RDBMS to MongoDB Atlas - Texas American Resources Company (TARC)
Download as PPTX, PDF0 likes356 views
Texas American Resources Company (TARC) is migrating from RDBMS to MongoDB Atlas to eliminate data silos caused by legacy systems and improve reporting and automation capabilities in oil and gas operations. The motivation for the migration includes the need for a flexible, scalable architecture with cleaner data and modern dashboarding for production reporting. Using MongoDB Atlas allows for significant maintenance time savings, enhanced security, and minimal code changes, facilitating a smoother transition.
Migrating from RDBMS to MongoDB Atlas - Texas American Resources Company (TARC)
- 1. Migrating from RDBMS to MongoDB Atlas Texas American Resources Company (TARC) Reggie Crawford [email protected]
- 2. Who Am I? • I’ve worked in IT for 21 years • Programming • Python, Perl, C/C++, Java, R, Scala • Systems Administration • Linux/Unix, AS/400, Windows • Database Administration • MongoDB, SQL Server, Oracle, MariaDB, PostgreSQL, Neo4j • Cloud Platforms • Atlas, AWS, Azure, Google AppEngine
- 3. What Does TARC Do? • Oil and Gas focused on Exploration and Production (Upstream)
- 4. Reasons for migration Data silos • Mostly caused by software choices • Accounting (UniData) • Production (UniData/Microsoft Access)* • Geology (SAP SQLAnywhere)* • Land (UniData) • Reservoir Engineering (Microsoft Access)* • Way too many Microsoft Excel spreadsheets Reporting (Focusing on Production) • No consistent data • Oil and Gas tools are very expensive, dated, and one size fits • A desire for dashboarding and automation
- 6. First Attempt - 2011 Web Interface PDF Document Excel File BOLO (UniData) Entrinsik Reporting Server UniQuery
- 7. Second Attempt - 2012 Productiondata Accountingdata Landdata Web Interface PDF Document Excel File BOLO (UniData) Entrinsik Reporting Server UniQuery Reservoirdata Acccess Database File SQL query
- 8. Third Attempt - 2012 Accountingdata Landdata Web Interface PDF Document Excel File BOLO (UniData) Entrinsik Reporting Server UniQuery Reservoirdata Acccess Database File SQL query ProductionData Production Database SQL Server
- 9. Current Attempt Production Data (UniData/Restful API/ SOAP) Accounting Data (UniData) Land Data (UniData/Restful API) Ascii Log Files ETL Data Sources Secondary Config Primary MongoDB
- 10. Why MongoDB? • No Joins!!! • JSON Documents • Easy to scale • Flexible schema • Spatially aware • No more lookup tables
- 11. Migration Planning Focus on production and production reporting Schema Design wells.json tanks.json meter.json gathering_system.json lease.json production.json runtickets.json gas_meter_readings.json economics.json Data Migration Python to the Rescue ETL all use PyMongo/.NET Drivers for IronPython Production Reporting and Economic Forecasting Dashboarding for Executive and managers Reservoir engineers can pull from onedata source to
- 12. Results of Migration • Cleaner and more accurate data • Better understanding of our core data model • A modern architecture • Automated production reporting • Reservoir Engineers can pull from on data source
- 13. Why MongoDB Atlas? • Huge time savings on maintenance • Follows the best practices for security • Save on purchasing SSL certs • Very minimal code rewrites, only needed to point to the Atlas database. • Scalability • Monitoring
- 15. References • MongoDB University • M034: New Features and Tools in MongoDB 3.4 • M102: MongoDB for DBAs • M123: Getting Started with MongoDB Atlas • UD032: Data Wrangling with MongoDB (Udacity)