![Enter MongoDB
person
address
cdc_person
cdc_address
[{
"id" : 6943896,
"lastname" : "Jansen",
"address" : {
"city" : "Amsterdam"
}
}, …]](https://image.slidesharecdn.com/pc2017-cdctothemax-171119094404/85/CDC-to-the-Max-29-320.jpg)
![Result
[{
"id" : 6943896,
"lastname" : "Jansen",
"address" : {
"city" : "Amsterdam"
}
}, …]](https://image.slidesharecdn.com/pc2017-cdctothemax-171119094404/85/CDC-to-the-Max-39-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to CDC to the Max! (20)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Ad
CDC to the Max!
- 1. CDC to the Max! CDC driving new technology EMEA PUG Challenge 2017 Bronco Oostermeyer
- 2. Who am I? Bronco Oostermeyer OpenEdge since 1996, everything around Progress and more NativeScript/Angular/Node.js/TypeScript Progress, UNIT4, WALVIS
- 3. Who are we? Flusso • Biggest OpenEdge consultancy firm in NL • Progress technologies • Open Source (Java & ServiceMix, etc), • Web Apps (Angular2, various app platforms) • OutSystems
- 4. Agenda • CDC • Use cases • High speed search engine
- 5. (CDC) To the Max! Since we didn’t qualify for the World Cup
- 6. The Challenge (for me) • A Use Case for CDC, new in 11.7 • “nice feature, but what to do with it?” • Come up with scenario which doesn’t suit a RDBMS • Challenge you to think differently about your application
- 7. Change Data Capture • It captures changes to data • ALL changes • Including originated from SQL • Transparent • Your current tables stay exact the way they are • CDC data in new tables, same database • Configurable rather than need for programming • NOT trigger based
- 8. Licensing • Standard in Enterprise DB advanced • Add-on for • Enterprise DB • Workgroup DB
- 9. Steps • Make dedicated storage area’s for CDC data & CDC index • Enable the CDC feature • Assign storage area’s • Create policies
- 10. Add area’s d "cdc_data":98,32;64 . d "cdc_index":99,32;8 . prostrct add <dbname> <st file>
- 12. proutil <dbname> -C enablecdc area <cdc_data_area> indexarea <cdc_index_area>
- 13. Policies
- 14. tracking changes After enabling CDC you get: and: • _cdc-field-policy • _cdc-table-policy
- 15. tracking changes– level 0 WARNING: Level 0 cannot be upgraded to higher level. • 4 levels • Just registers update event in _Cdc-Change-Tracking table:
- 16. tracking changes – level 1, 2 & 3
- 17. what will be tracked (and “name” as well)
- 18. levels explained • 1 – customer 1 has changed (w/ fieldlist) • 2 – customer 1 has changed, these are the new values • 3 – customer 1 has changed, these are the old & new values
- 20. CDC Summary • Capture all changes transparently • dedicated table for storing changes • a _change-sequence field to keep track off
- 21. CDC Use cases • Friendly alternative for auditing (sort of) • Off-loading changes to satellite systems • Search engine • Back-end for web • Data warehouses • High Availability/-Scalability • …
- 22. But why? • Assume an System of Record (f.e. an ERP) • You know all the users? • Do you still do when you want to go to the Web? • Do those complicated searches really perform? • Can you calculate the requirements for new hardware?
- 23. What if? • You can scale horizontally (dynamically) • In the cloud • You could deliver real 24/7 for web users • You can prevent your back office being locked out by web traffic • And still use your ERP as is?
- 24. Put it to use • OpenEdge database • 2 tables: person & address • 14.412.063 different person records (true names) • 256.221 unique last names • 1 address record per person • divided over 388 municipalities (cities) • based on true data on NL
- 25. Schema person id: int64 lastname: char address id: int64 person_id: int64 city: char For sake of this presentation: a very simple schema
- 26. db contents
- 27. the query approx.: (after clean db start ~1min)
- 28. RDBMS vs NoSQL • joined queries not fast (RDBMS) • stuck to db structure • NoSQL databases are very fast • but lack transactional capabilities (mostly) • Joins are not always supported • MongoDB, Redis, Cassandra, ElasticSearch • and CouchDB, Hbase, Accumulo, and … and … • https://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs- redis
- 29. Enter MongoDB person address cdc_person cdc_address [{ "id" : 6943896, "lastname" : "Jansen", "address" : { "city" : "Amsterdam" } }, …]
- 30. How keep you MongoDB up-to-date? dispatcher cdc_person cdc_address expressjsPOST http://localhost:4220/update { "changeseqid" : 136, "info" : { "id" : 32944039, "lastname" : "test_qwerty", "address" : { "city" : "Aalsmeer" } } } GET http://localhost:4220/changeseqid
- 31. cdc_person
- 32. What more? • The target system needs to keep track of processed changes • The target system needs to be able to report last change id • a search api
- 33. Technology Stack • OpenEdge Change Data Capture • Node.js (TypeScript) • express • mongodb (connector) • MongoDB • Angular (TypeScript) • Bootstrap
- 35. Demo
- 36. The code • https://github.com/bfv/pc2017 • OpenEdge dispatcher (/cdc) • httpclient • Node JS (/search) • Express, MongoDB • Angular (/webui) • bootstrap
- 37. Node /search code server.ts router.ts GET http://localhost:4220/search?lastname=Jansen&city=Amsterdam
- 38. connector code mongoconnector.tsdbconnector.ts { "lastname": "Jansen", "address.city": "Amsterdam" }
- 39. Result [{ "id" : 6943896, "lastname" : "Jansen", "address" : { "city" : "Amsterdam" } }, …]
- 40. But… • cdc data is in normal tables (delta.df might wipe data) • cdc configuration is dumped/loaded separately
- 41. Conclusion • CDC is pretty easy to configure • Resulting cdc table are readable • Use change_sequence to keep track • The possibilities are limited by your own fantasy • Highly scalable cloud systems can be fed by CDC • Enables new technology w/o starting all over • Last advise: Think big, start small
- 42. Questions?
- 45. follow us on:
Editor's Notes
- #20: Note here the _change-sequence field. This give the possibility to go through your changes in the correct order AND a means to keep track what changes have been off-loaded to another system and which one not.