Technical Challenges in Open Metadata
Download as PPTX, PDF0 likes89 views
Presented by: Mandy Chessell, IBM Presented at All Things Open 2020 Abstract: I am one of the leaders in the open metadata and governance initiative. This is seeking to develop standards and a reference implementation through and open source project called ODPi Egeria. Egeria enables organizations to manage data as an asset even when they use tools and platforms from multiple vendors. This type of problem is extremely complex and it needs the collaboration of multiple organizations to make it happen. In this talk I will go through the technical challenges we face and how they are being overcome.
Ad
More Related Content
What's hot (20)
Similar to Technical Challenges in Open Metadata (20)
Ad
More from All Things Open (20)
![How I Learned to Stop Worrying about my Infrastructure and Love [Open]Tofu](https://cdn.slidesharecdn.com/ss_thumbnails/ato-oct-2024-opentofu-241113165914-c664afc4-thumbnail.jpg?width=560&fit=bounds)
Ad
Recently uploaded (20)
Technical Challenges in Open Metadata
- 1. https://egeria.odpi.org/ Mandy Chessell FREng CEng FBCS IBM Distinguished Engineer and Master Inventor Lead of the ODPi Egeria project 20th October 2020 Technical Challenges in Open Metadata @MandyChessell
- 2. https://egeria.odpi.org/ In times of crisis … … organizations need to act quickly • locate the resources they have • understand their value, status and health • reposition, repurpose and retire them as needed 2
- 3. https://egeria.odpi.org/ But knowledge is dispersed • Most technologies retain knowledge of the resources that they are working with. • This knowledge is fragmented; locked behind proprietary formats into small nuggets of value. • Nothing binds this knowledge together. • Gathering the facts needed to act is slow and error prone. 3 Organizations need a solution that links this knowledge together so that they can cross between the organizational, professional and technological gaps in their digital landscape.
- 4. https://egeria.odpi.org/ But what if we linked everything together … 4
- 5. https://egeria.odpi.org/ Egeria Value Egeria enables the sharing of knowledge between people and process that use different tools and technologies Increasing agility, collaboration and the availability of relevant data when making decisions. Breaking down silos between traditional technologies such as data and applications; applications and infrastructure management; security and data. 5
- 6. https://egeria.odpi.org/ This is not a new problem … Over the years, our industry has gained plenty of experience discovering what does not work • Centralized metadata repository • “Just use our tools” – Single vendor lock in • Paper standards that each vendor implements • Bridges that copy metadata between tools • Bespoke point-to-point integrations between tools 6
- 7. https://egeria.odpi.org/ What is different about Egeria • Open source, multi-vendor and client collaboration • Peer-to-peer architecture; retaining the value of each technology • Self-configuring, distributed, real-time exchange of knowledge • Scales from Raspberry Pi to multi-instance HA cloud deployment • Multi-tenant • Instance based security • Customizable through connectors 7
- 8. https://egeria.odpi.org/ How do we work • Iteratively perform a series of practical experiments that ignore the boundaries of traditional approaches. • Holistic approach • Open, public collaboration • Developing interest and partnership as we go • Agile development with feedback loop • Enterprise grade software 8
- 9. https://egeria.odpi.org/ Achievements 9 • 500 linked open metadata types demonstrating how the knowledge from many tools can be linked together. • Open metadata repository interface proven for table, graph and hierarchical DB stores. • Enterprise queries and replication across heterogeneous technologies • Conformance test suite and mark • Automated configuration of data virtualization technology and security as new data sets are added to a data lake • Suite of persona-based labs and tutorial using Jupyter Notebooks. • Virtual graph of metadata maintained across distributed heterogenous metadata repositories. • Frameworks, APIs and connectors for minimizing integration cost for different types of technologies • Virtual repository explorer UI • Instance based security • Controlling visibility of assets through zones • Scalable, secure platform configurable and customizable through connectors • Purpose-based data access • Metadata versioning and provenance • Multi-tenant UI based on carbon • W3C semantic standards pattern for data model exchange • Automation of metadata acquisition through templates, daemons, discovery services and stewardship. • Classification of assets • Reference data management • Multi-technology collaboration and feedback • Multi-domain governance model • Digital service lifecycle, from business design, development, devOps and use. • Comprehensive open lineage services.
- 10. https://egeria.odpi.org/ How does ODPi Egeria work? 10 Open and Unified Metadata Development DevOps Data Science ODPi Egeria enables exchange of metadata between tools from different vendors
- 11. https://egeria.odpi.org/ What makes metadata integration hard? Metadata standards are many and each covers a small subset of the integration space Impedance mismatch in: Terminology Granularity Capability Availability Technology Low priority in offering managers eyes leads to lack of investment resulting in minimal capability 11
- 12. https://egeria.odpi.org/ What do you need to enable integration? Common language and data structures Integration points that match the needs of specific types of third-party technologies Reducing effort; increasing consistency of metadata exchanged Supporting both APIs and events/notifications Restart/resync – need APIs to compare states Ongoing operation – notifications help avoid polling More information – retrieved using APIs Hosting environment for connectors and related integration technologies 12
- 13. https://egeria.odpi.org/ External view of Egeria 13 Open Metadata and Governance BI Tools ETL and Data Movement Engines Analytics and AI Tools Virtualization Platform Governance tools Data Platforms Security Tools Dev and DevOps Tools
- 14. https://egeria.odpi.org/ Reality check - hybrid multi-cloud world Data Lake Mobile Apps Databases ApplicationsFiles Independent metadata Repository Linked metadata Repositories Business Partners Sharing data IoT devices and systems Applications New applications deployed to cloud
- 15. https://egeria.odpi.org/ With Egeria Data Lake Mobile Apps Databases ApplicationsFiles Independent metadata Repository Linked metadata Repositories Business Partners Sharing data IoT devices and systems Applications New applications deployed to cloud
- 16. https://egeria.odpi.org/ Egeria in operation Egeria’s platform (in blue) runs in each deployment environment (cloud or on premises) Egeria servers (orange circles) are deployed on the platform. Each are specialized to support the metadata needs of specific technologies Egeria manages the exchange of metadata between all parties.
- 17. https://egeria.odpi.org/ Open Connector Framework (OCF) Egeria’s open connector framework provides a consistent mechanism to plug integration logic into Egeria and to plug Egeria into third party technologies. 17
- 18. https://egeria.odpi.org/ Connected OMAG Servers 18 Can start at any level
- 20. https://egeria.odpi.org/ The OMAG Server Platform 20 OMAG Server Platform OMAG Server Platform OMAG Server Platform OMAG Server Platform Egeria Server 1 Egeria Server 2 Egeria Server 3 Kubernetes OMAG Server Platform Egeria Server 1 Egeria Server 2 Egeria Server 3 Multi-tenant OMAG Server Platform Egeria Server 1 Edge
- 21. https://egeria.odpi.org/ Example of a simple cohort 21 Cohort A Chief Data Office Data Lake Systems of Record
- 22. https://egeria.odpi.org/ Connecting to multiple cohorts 22 Cohort BCohort A Chief Data Office Data Lake Systems of Record Mobile Apps Data Lake Systems of Record Marketing
- 23. https://egeria.odpi.org/ Formation of a cohort 23 The first server to join the cohort issues a registration request and waits for others to join.
- 24. https://egeria.odpi.org/ Formation of a cohort 24 When another server joins the cohort they exchange registration information.
- 25. https://egeria.odpi.org/ Formation of a cohort 25 Once the registration is complete the cohort members can issue federated queries.
- 26. https://egeria.odpi.org/ Formation of a cohort 26 Metadata can also be replicated through the cohort to allow caching for availability and performance
- 27. https://egeria.odpi.org/ Instance representations in the metadata in the cohort 27
- 28. https://egeria.odpi.org/ Linking metadata across the cohort 28 Database Column Glossary Term OMAG Server 1 OMAG Server 2 Entity Entity
- 29. https://egeria.odpi.org/ Linking metadata across the cohort 29 Database Column Glossary Term Glossary Term Meaning OMAG Server 1 OMAG Server 2 Reference Copy Relationship
- 30. https://egeria.odpi.org/ Linking metadata across the cohort 30 Database Column Glossary Term OMAG Server 1 OMAG Server 3 OMAG Server 2 Database Column Glossary Term Meaning
- 31. https://egeria.odpi.org/ Linking metadata across the cohort 31 Database Column Glossary Term OMAG Server 1 OMAG Server 3 OMAG Server 2 Database Column Glossary Term Meaning Confidential
- 32. https://egeria.odpi.org/ Scope of metadata covered 32 Glossary Collaboration Governance Models and Reference Data Metadata Discovery Lineage Data Assets Base Types, Systems and Infrastructure
- 33. https://egeria.odpi.org/ Scope of metadata covered 33 Policy Metadata (Principles, Regulations, Standards, Approaches, Rule Specifications, Roles and Metrics) Governance Actions and Processes Augmentation MappingImplementation Business Objects and Relationships, Taxonomies and Ontologies Business Attributes Organization Teaming Metadata (people profiles, communities, projects, notebooks, …) Models and Schemas 4 3 1 5 Physical Asset Descriptions (Data stores, APIs, models and components) Asset Collections (Sets, Typed Sets, Type Organized Sets) Information Views Rights Management Reference Data Feedback Metadata (tags, comments, ratings, …) ClassificationSchemes Classification Strategy Subject Area Definition Campaigns and Projects Rollout 2 Discovery Metadata (profile data, technical classification, data classification, data quality assessment, …) Augmentation Instrument Association Information Process Instrumentation (design lineage) 6 7 ConnectorsBasic Types, Infrastructure and Systems Access 0
- 34. https://egeria.odpi.org/ Search Open Metadata Access Services Adding specialized integration APIs … 34 Open Metadata Repository Services Open metadata repository cohort
- 35. https://egeria.odpi.org/ Current Open Metadata Access Services (OMASs) 35 Project Management Community ProfileAsset Catalog Stewardship Action Information View Governance Program Data Process Subject Area Connected Asset Discovery EngineGovernance Engine Data Protection Software Developer Data Platform Asset Owner Digital Architecture Data Science DevOps Asset Consumer Data Infrastructure Data Privacy Asset Lineage
- 36. https://egeria.odpi.org/ Integration patterns Egeria supports different styles of integration since there is a huge variety in the capabilities of third-party technologies. • Integrated technology connects to access services • Other have connectors hosted in an integration daemon 36
- 37. https://egeria.odpi.org/ Example integrations for data managers (databases, file systems etc.) 37
- 38. https://egeria.odpi.org/ Governance Zones Metadata can become more valuable than data when it pulls together a complete view of the organization’s assets and activities. A governance zone defines a set of assets that are related in some way 38
- 39. https://egeria.odpi.org/ Zone Membership The zones that an asset belongs to determines its visibility to specific governance processes and API entry points. 39
- 40. https://egeria.odpi.org/ Use of governance zones Data sovereignty Adjustments for legal jurisdiction Asset visibility and access control Data access control Maintenance and backup processing Metering and billing Understanding dependencies 40
- 41. https://egeria.odpi.org/ Access services’ zone settings Default zones Zone membership for new assets (unless explicitly set) Null means not a member of any zone Supported zones Scope of assets that are visible through this OMAS Null means all assets Publish zones Zones to set in an asset when ready for broader consumption 41
- 42. https://egeria.odpi.org/ Metadata Security Metadata ranges from public information to highly sensitive information. The consequence of integrating metadata is that metadata security needs to be more granular – instance based. Picture is an example of the different levels of security challenge in an Egeria server. Security is controlled through a connector Asset security can change asset settings (eg zone membership) based on calling user. 42 Levels of challenge for an inbound request
- 43. https://egeria.odpi.org/ Metadata provenance If metadata is incorrect, it is important to know where it came from Every instance of open metadata has full provenance information 43
- 44. https://egeria.odpi.org/ Audit Log Framework (ALF) Egeria needs to be integratable with existing IT operations environment The ALF designed with automation and integration in mind Supplied Destinations Console SLF4J Event Topic File Directory 44
- 45. https://egeria.odpi.org/ User Interfaces - Repository explorer 45
- 46. https://egeria.odpi.org/ Why do vendors work with Egeria? Linking old platform with new platform Linking multiple versions of own product deployed across an enterprise Offload integration implementation costs Access to third party metadata: open source technology’s and other vendor’s metadata Governing third party technologies 46
- 47. https://egeria.odpi.org/ Development Status Development in plain sight Monthly releases What is ready, goes … 47
- 48. https://egeria.odpi.org/ Using ODPi Egeria … • Eases the cost of metadata integration through • Comprehensive standards and libraries. • Active vendor recruitment program. • Provides direct support to many governance roles, filling the gaps between function offered through commercial tools. • Provides best practices and content packs to accelerate an organization’s journey to becoming data driven. 48
- 49. https://egeria.odpi.org/ Links 49 Press Releases and Podcast Open source repositories • https://github.com/odpi/data-governance • https://github.com/odpi/egeria • https://www.linuxfoundation.org/press-release/2018/08/odpi-announces-egeria-for-open- sharing-exchange-and-governance-of-metadata/ • https://www.linuxfoundation.org/press-release/2019/02/odpi-announces-new-egeria- conformance-program-to-advance-open-metadata-exchange-between-vendor-tools/ • https://roaringelephant.org/2018/09/25/episode-107-open-metadata-and-governance- masterclass-with-mandy-chessell-part-1/ • https://roaringelephant.org/2018/10/09/episode-109-open-metadata-and-governance- • masterclass-with-mandy-chessell-part-2/ • https://youtu.be/ryd3KFWT1mc 49
Editor's Notes
- #14: If they should have, take steps to add " + "them to the authorized list of users. If this user should not have access, investigate where the request " + "came from to determine if the system is under attack, or it was a mistake.
- #28: Here is an example of a number of OMRS instance objects – there are two entities, that are connected by a relationship. Also, one of the entities has two classifications. All of the instances have attributes – some will be core attributes used for type or control information; others will be attributes that are specific to the instance type (known as type-defined attributes). You don’t need to remember this picture – we’ll stick a copy of it in the top corner so we can refer back to it…..