DataPatterns - Profiling in ECL Watch

Oct 23, 2019Download as PPTX, PDF1 like156 views

DataPatterns is an ECL bundle that provides data profiling and research tools. It has been integrated into the HPCC Systems ECL Standard Library and ECL Watch. The presentation discusses improvements to DataPatterns, how it can be used from ECL code or within ECL Watch to generate profiling reports, and differences between installations from the ECL bundle, Standard Library, or ECL Watch. It also demonstrates using DataDetectors models to examine sample data and previews the new Cloud IDE for browser-based ECL programming.

2019 HPCC
Systems®
Community Day
Challenge Yourself –
Challenge the Status Quo
DataPatterns - Profiling in ECL WatchDan S. Camper
Thaumaturge
HPCC Systems Solutions
Lab

Topics
• What is DataPatterns?
• Improvements since last year
• ECL Standard Library integration
• ECL Watch integration
• Differences between installations
2DataPatterns - Profiling in ECL Watch

DataPatterns – What is it?
• ECL bundle that provides some basic data profiling and research tools to an ECL
programmer
• Today, it is primarily a data profiling tool
• Numerous parameters for controlling analysis and output
• Analyze all rows in a dataset or just a sample
• Analyze all fields or only certain fields
• Enable only specified profiling checks
• Specify returned pattern counts
• Creates a single dataset as a result
• One record for each field analyzed
4DataPatterns - Profiling in ECL Watch

… And Gains A User Interface in ECL Watch
9DataPatterns - Profiling in ECL Watch

Logical File’s Record Structure
10DataPatterns - Profiling in ECL Watch

Differences Between Installations
• ECL Bundle contains additional functions
• ProfileFromPath()
• BestRecordStructureFromPath()
• Contains support for pretty report
• Available from https://github.com/hpcc-systems/DataPatterns
• ECL Standard Library
• Does not support pretty report
• Available with HPCC Systems 7.4.0 and later
• ECL Watch
• Supports only data profiling
• Available with HPCC Systems 7.4.0 and later
17DataPatterns - Profiling in ECL Watch

DataDetectors – What Is This Data?
Bloom Filter Models
• Person.FirstName
• Person.LastName
• Geo.USA.Address.StreetName
• Geo.USA.Address.CityName
• Geo.USA.Address.PostalCode
• Geo.USA.PhoneAreaCode
• Geo.CountryName
• Geo.CountryCode
• Identifier.USA.StockSymbol
Heuristic Models
• Calendar.Date
• Calendar.Month
• Calendar.Quarter
• Calendar.Year
• Calendar.YearMonth
• Currency
• Group.StockExchange
• Geo.USA.Address.State
• Geo.Longitude
• Geo.Latitude
• Geo.LatLon
• Identifier.USA.PhoneNumber
• Identifier.EmailAddress
• Identifier.RecordID
• Identifier.WebSiteURL
DataPatterns - Profiling in ECL Watch 19

DataDetectors Test – Raw Data
20DataPatterns - Profiling in ECL Watch

DataDetectors Test – Data Examination Results
21DataPatterns - Profiling in ECL Watch

Cloud IDE – ECL Programming in a browser-based IDE
23DataPatterns - Profiling in ECL Watch

Cloud IDE
24DataPatterns - Profiling in ECL Watch

Cloud IDE
25DataPatterns - Profiling in ECL Watch

26DataPatterns - Profiling in ECL Watch
fini
View this presentation on YouTube:
https://www.youtube.com/watch?v=TtcrOcyf6gQ&list=PL-
8MJMUpp8IKH5-d56az56t52YccleX5h&index=6&t=0s (13:19)

This document discusses using ETL Metadata Injection in Pentaho to dynamically load metadata at runtime. Specifically, it describes how to use a template transformation to load cost files with a dynamic header structure, where budget files contain 12 months of data but forecast files can contain between 1-12 months. It provides examples of injecting metadata into text file input and row normalization steps in the template using the Metadata Injection step, and how to programmatically inject metadata into steps that don't natively support injection using the Pentaho API.

Tuning ETL's for Better BIDatavail

Finding ways to make ETL loads faster is not always obvious. Moreover, there is a difference in how to tune OLAP vs OLTP databases. Some of the techniques learned through years of tuning EBS seem to make no effect on tuning a BI ETL. This presentation will discuss why this is the case, present some techniques on how to find the bottlenecks in your BI ETL jobs and some techniques to tune these slow SQL statements, improving the speed of nightly ETL jobs. Attendees will learn the steps to monitor ETLs, capture Problem SQL and gain knowledge to improve the overall ETL Performance.

Introduction To Pentaho KettleBoulder Java User's Group

Pentaho Data Integration/Kettle is an open source ETL tool that has been used by the presenter for two years. It allows users to extract, transform and load data from various sources like databases, files and NoSQL into destinations like data warehouses. Some advantages of Kettle include its graphical user interface, large library of components, performance processing large datasets, and ability to leverage Java libraries. The presenter demonstrates syncing and processing data between different sources using Kettle.

Kettle – Etl ToolDr Anjan Krishnamurthy

Pentaho Data Integration (Kettle) is an open-source extract, transform, load (ETL) tool. It allows users to visually design data transformations and jobs to extract data from source systems, transform it, and load it into data warehouses. Kettle includes components like Spoon for designing transformations and jobs, Pan for executing transformations, and Carte for remote execution. It supports various databases and file formats through flexible components and transformations.

Etl with apache impala by athemasterAthemaster Co., Ltd.

This document summarizes a presentation on implementing an ETL project using Apache Impala on the Hadoop platform. It discusses the components of a Hadoop ETL project, including data storage, sources, ETL processing, metadata, and targets. It then shares details about a customer case that involved replacing an SQL reporting system with Impala to access standardized data for reporting and analytics within required timeframes. The presentation emphasizes that successful Hadoop ETL projects require data engineering methodologies around organization, modularization, logging, and testing approaches.

Kafka to Hadoop Ingest with Parsing, Dedup and other Big Data TransformationsApache Apex

Exceptions are the Norm: Dealing with Bad Actors in ETLDatabricks

Stable and robust data pipelines are a critical component of the data infrastructure of enterprises. Most commonly, data pipelines ingest messy data sources with incorrect, incomplete or inconsistent records and produce curated and/or summarized data for consumption by subsequent applications. In this talk, we go over new and upcoming features in Spark that enabled it to better serve such workloads. Such features include isolation of corrupt input records and files, useful diagnostic feedback to users and improved support for nested type handling which is common in ETL jobs. Speaker: Sameer Agarwal This talk was originally presented at Spark Summit East 2017.

The Critical Missing Component in the Production ML StackDatabricks

The day the ML application is deployed to production and begins facing the real world is the best and the worst day in the life of the model builder. The joy of seeing accurate predictions is quickly overshadowed by a myriad of operational challenges. Debugging, troubleshooting & monitoring takes over the majority of their day, leaving little time for model building. In DevOps, software operations are taken to a level of an art. Sophisticated tools enable engineers to quickly identify and resolve issues, continuously improving software stability and robustness. In the ML world, operations are still largely a manual process that involves Jupyter notebooks and shell scripts. One of the cornerstones of the DevOps toolchain is logging. Traces and metrics are built on top of logs enabling monitoring and feedback loops. What does logging look like in an ML system? In this talk we will demonstrate how to enable data logging for an AI application using MLflow in a matter of minutes. We will discuss how something so simple enables testing, monitoring and debugging in an AI application that handles TBs of data and runs in real-time. Attendees will leave the talk equipped with tools and best practices to supercharge MLOps in their team.

Data provenance in HopsworksAlexandru Adrian Ormenisan

Data lineage has gained popularity in the Machine Learning community as a way to make models and datasets easier to interpret and to help developers debug their ML pipelines by enabling them to go from a model to the dataset/user who trained it. Data provenance and lineage is the process of building up the history of how a data artifact came to be. This history of derivations and interactions can provide a better context for data discovery, debugging, as well as auditing. In this area, others, such as Google and Databricks, have made small steps. The Hopsworks approach presented provenance information is collected implicitly through the unobtrusive instrumentation of jupyter notebooks and python code - What we call 'implicit provenance'.

Building Your First Apache Apex (Next Gen Big Data/Hadoop) ApplicationApache Apex

This document provides an overview of building a first Apache Apex application. It describes the main concepts of an Apex application including operators that implement interfaces to process streaming data within windows. The document outlines a "Sorted Word Count" application that uses various operators like LineReader, WordReader, WindowWordCount, and FileWordCount. It also demonstrates wiring these operators together in a directed acyclic graph and running the application to process streaming data.

Starschema ProductsEndre Adam

This document summarizes several products from Starschema Ltd including: PLSAP Connector which allows extracting data from SAP systems; BigQuery JDBC driver which connects SQL tools to Google BigQuery; CloudKeeper which provides tokenization for big data security; DW Scheduler 2.0 for scheduling data warehouse loads; and Starsheet for exporting data between Oracle and Excel. It provides details on each product's features and capabilities.

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache ApexApache Apex

Stream data processing is becoming increasingly important to support business needs for faster time to insight and action with growing volume of information from more sources. Apache Apex (http://apex.apache.org/) is a unified big data in motion processing platform for the Apache Hadoop ecosystem. Apex supports demanding use cases with: * Architecture for high throughput, low latency and exactly-once processing semantics. * Comprehensive library of building blocks including connectors for Kafka, Files, Cassandra, HBase and many more * Java based with unobtrusive API to build real-time and batch applications and implement custom business logic. * Advanced engine features for auto-scaling, dynamic changes, compute locality. Apex was developed since 2012 and is used in production in various industries like online advertising, Internet of Things (IoT) and financial services.

Enabling exploratory data science with Spark and RDatabricks

R is a favorite language of many data scientists. In addition to a language and runtime, R is a rich ecosystem of libraries for a wide range of use cases from statistical inference to data visualization. However, handling large datasets with R is challenging, especially when data scientists use R with frameworks or tools written in other languages. In this mode most of the friction is at the interface of R and the other systems. For example, when data is sampled by a big data platform, results need to be transferred to and imported in R as native data structures. In this talk we show how SparkR solves these problems to enable a much smoother experience. In this talk we will present an overview of the SparkR architecture, including how data and control is transferred between R and JVM. This knowledge will help data scientists make better decisions when using SparkR. We will demo and explain some of the existing and supported use cases with real large datasets inside a notebook environment. The demonstration will emphasize how Spark clusters, R and interactive notebook environments, such as Jupyter or Databricks, facilitate exploratory analysis of large data.

Hyperparameter Optimization - Sven Hafenegersparktc

At the sold-out Spark & Machine Learning Meetup in Brussels on October 27, 2016, Sven Hafeneger of IBM delivered a lightning talk called Hyperparameter Optimization - when scikit-learn meets PySpark. As Sven explained, Apache Spark™ is not only useful when you have big data problems. If you have a relatively small data set you might still have a big computational problem. One problem is the search for optimal parameters for ML algorithms. Normally, a data scientist has a laptop with 4 cores (8 threads), that means it will take some time to perform a grid search. However, using Spark opens the possibility that the grid search can be taken out on a cluster with a higher degree of parallelism.

Learning to Rank Datasets for Search with Oscar CastanedaDatabricks

Oscar Castañeda presented on learning to rank datasets for search using Elasticsearch's learning to rank capabilities. He discussed extracting relevance judgements during data pipelines to bootstrap a dataset ranking model, and leveraging click-through data on dataset profiles to continuously improve the model. The demo showed ranking movie datasets based on a relevance judgement file, with movie profile pages representing dataset profiles. Next steps included describing datasets in a structured way, building a knowledge graph to extract insights, and using topic models to understand datasets.

A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...Jose Quesada (hiring)

The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn? At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting; which would you use in production? The machine learning libraries in Apache Spark are an impressive piece of software engineering, and are maturing rapidly. What advantages does Spark.ml offer over scikit-learn? At Data Science Retreat we've taken a real-world dataset and worked through the stages of building a predictive model -- exploration, data cleaning, feature engineering, and model fitting -- in several different frameworks. We'll show what it's like to work with native Spark.ml, and compare it to scikit-learn along several dimensions: ease of use, productivity, feature set, and performance. In some ways Spark.ml is still rather immature, but it also conveys new superpowers to those who know how to use it.

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019Sandesh Rao

This session will focus on 19 troubleshooting tips and tricks for DBA's covering tools from the Oracle Autonomous Health Framework (AHF) like Trace file Analyzer (TFA) to collect , organize and analyze log data , Exachk and orachk to perform mass best practices analysis and automation , Cluster Health Advisor to debug node evictions and calibrate the framework , OSWatcher and its analysis engine , oratop for pinpointing performance issues and many others to make one feel like a rockstar DBA

Distributed ML/DL with Ignite ML Module Using Apache Spark as DatabaseDatabricks

The current implementation of ML algorithms in Spark has several disadvantages associated with the transition from standard Spark SQL types to ML-specific types, a low level of algorithms' adaptation to distributed computing, a relatively slow speed of adding new algorithms to the current library. Also, Spark ML doesn't support online-learning by nature for all algorithms, stacking, boosting and a bunch of approximate ML algorithms that gives a significant speedup in many cases. The Apache Ignite could work closely with Apache Spark due to exellent Ignite RDD/Ignite DataFrame implementation (see https://ignite.apache.org/use-cases/spark/shared-memory-layer.html). Also Apache Ignite has Ignite ML module that includes a lot of distributed ML algorithms, NLP package (will be available in next release, 2.8), the bunch of approximate ML algorithms, simple integration with TensorFlow via TensorFlow Ignite Dataset (currently, this is a part of TF.contrib package) and also each algorithm supports the model updating that gives us ability to make online-learning not only for KMeans and LinReg. We suggest to use Apache Ignite ML module to speedup your ML training and use Spark + Ignite as backend for distributed TensorFlow calculations. You will see live demos of ML pipeline building with Apache Ignite ML module, Apache Spark, Apache Kafka, TensorFlow and more.

Implementing OLE at PennBeth Picknally Camden

Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)Apache Apex

Stream Processing use cases and applications with Apache Apex by Thomas WeiseBig Data Spain

Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...Databricks

Volunteers around the world increasingly act as human sensors to collect millions of data points. A team from the World Bank trained deep learning models, using Apache Spark and BigDL, to confirm that photos gathered through a crowdsourced data collection pilot matched the goods for which observations were submitted. In this talk, Maurice Nsabimana, a statistician at the World Bank, will demonstrate a collaborative project to design and train large-scale deep learning models using crowdsourced images from around the world. BigDL is a distributed deep learning library designed from the ground up to run natively on Apache Spark. It enables data engineers and scientists to write deep learning applications in Scala or Python as standard Spark programs-without having to explicitly manage distributed computations. Attendees of this session will learn how to get started with BigDL, which runs in any Apache Spark environment, whether on-premises or in the Cloud. Attendees will also learn how to write a deep learning application that leverages Spark to train image recognition models at scale.

From Pipelines to Refineries: Scaling Big Data ApplicationsDatabricks

Big data tools are challenging to combine into a larger application: ironically, big data applications themselves do not tend to scale very well. These issues of integration and data management are only magnified by increasingly large volumes of data. Apache Spark provides strong building blocks for batch processes, streams and ad-hoc interactive analysis. However, users face challenges when putting together a single coherent pipeline that could involve hundreds of transformation steps, especially when confronted by the need of rapid iterations. This talk explores these issues through the lens of functional programming. It presents an experimental framework that provides full-pipeline guarantees by introducing more laziness to Apache Spark. This framework allows transformations to be seamlessly composed and alleviates common issues, thanks to whole program checks, auto-caching, and aggressive computation parallelization and reuse.

LAD -GroundBreakers-Jul 2019 - The Machine Learning behind the Autonomous Dat...Sandesh Rao

We are entering a new era in the database with the introduction of the Oracle Autonomous Database. AI and Machine Learning are center stage to most projects and assist in making complex decisions which was not possible before. Most data science projects don’t get beyond the data scientist and rarely operationalize their predictive models. there are new toolsets and methods available everyday which make this an extremely dynamic space. There are different categories of users who want to use the algorithms , the toolsets but don't know where to start. Whether you are a data scientist who wants to play with data and build your own models or make use of the database features with the built in models or use the specific AI services within a specific vertical such as Insurance or Healthcare . We will take a glimpse at Oracle's Machine Learning Zeppelin-based notebooks for Oracle Autonomous Data Warehouse Cloud to how Oracle uses AIOps and Applied Machine learning for its own operations and the Oracle AI Platform Cloud Service to provided an all rounded view of what Oracle is upto in this space

Deep Dive into Apache Apex App DevelopmentApache Apex

New directions for Apache Spark in 2015Databricks

This document discusses new directions for Apache Spark in 2015, including improved interfaces for data science, external data sources, and machine learning pipelines. It also summarizes Spark's growth in 2014 with over 500 contributors, 370,000 lines of code, and 500 production deployments. The author proposes that Spark will become a unified engine for all data sources, workloads, and environments.

Building a modern Application with DataFramesSpark Summit

What's new in Oracle Trace File Analyzer version 12.2.1.1.0Sandesh Rao

The document discusses Oracle Trace File Analyzer (TFA). TFA provides a single interface for collecting diagnostic data across a cluster to reduce the time required to obtain diagnostic data. It collects all relevant data at the time of a problem. The document outlines TFA installation and configuration options, how it performs automatic and on-demand diagnostic collections, and how to use TFA to view system summaries, investigate logs, and customize collections.

OAP: Optimized Analytics Package for Spark Platform with Daoyuan Wang and Yua...Databricks

Spark SQL is one of the most popular components in big data warehouse for SQL queries in batch mode, and it allows user to process data from various data sources in a highly efficient way. However, Spark SQL is a general purpose SQL engine and not well designed for ad hoc queries. Intel invented an Apache Spark data source plugin called Spinach for fulfilling such requirements, by leveraging user-customized indices and fine-grained data cache mechanisms. To be more specific, Spinach defines a new Parquet-like data storage format, offering a fine-grained hierarchical cache mechanism in the unit of “Fiber” in memory. Even existing Parquet or ORC data files can be loaded using corresponding adaptors. Data can be cached in off-heap memory to boost data loading. What’s more, Spinach has extended the Spark SQL DDL, to allow users to define the customized indices based on relation. Currently, B+ tree and bloom filter are the first two types of indices supported. Last but not least, since Spinach resides in the process of Spark executor, there’s no extra effort in deployment. All you need to do is to pick Spinach from Spark packages when launching the Spark SQL. sing corresponding adaptors. Data can be cached in off-heap memory to boost data loading. What’s more, Spinach has extended the Spark SQL DDL, to allow user to define the customized indices based on relation. Currently B+ tree and bloom filter are the first 2 types of index we’ve supported. Last but not least, since Spinach resides in the process of Spark executor, there’s no extra effort in deployment, all we need to do is to pick Spinach from Spark packages when launch the Spark SQL. Spinach has been imported in Baidu’s production environment since Q4 2016. It helps several teams migrating their regular data analysis tasks from Hive or MR jobs to ad-hoc queries. In Baidu search ads system FengChao, data engineers analyze advertising effectiveness based on several TBs data of display and click logs every day. Spinach brings a 5x boost compared to original Spark SQL (version 2.1), especially in the scenario of complex search and large data volume. It optimizes the average search cost from minutes to seconds, while brings only 3% data size increase for adding a single index.

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL...Databricks

More Related Content

What's hot (20)

Data provenance in HopsworksAlexandru Adrian Ormenisan

Building Your First Apache Apex (Next Gen Big Data/Hadoop) ApplicationApache Apex

Starschema ProductsEndre Adam

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache ApexApache Apex

Enabling exploratory data science with Spark and RDatabricks

Hyperparameter Optimization - Sven Hafenegersparktc

Learning to Rank Datasets for Search with Oscar CastanedaDatabricks

A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...Jose Quesada (hiring)

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019Sandesh Rao

Distributed ML/DL with Ignite ML Module Using Apache Spark as DatabaseDatabricks

Implementing OLE at PennBeth Picknally Camden

Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)Apache Apex

Stream Processing use cases and applications with Apache Apex by Thomas WeiseBig Data Spain

Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...Databricks

From Pipelines to Refineries: Scaling Big Data ApplicationsDatabricks

LAD -GroundBreakers-Jul 2019 - The Machine Learning behind the Autonomous Dat...Sandesh Rao

Deep Dive into Apache Apex App DevelopmentApache Apex

New directions for Apache Spark in 2015Databricks

Building a modern Application with DataFramesSpark Summit

What's new in Oracle Trace File Analyzer version 12.2.1.1.0Sandesh Rao

Data provenance in HopsworksAlexandru Adrian Ormenisan

Building Your First Apache Apex (Next Gen Big Data/Hadoop) ApplicationApache Apex

Starschema ProductsEndre Adam

Apache Big Data EU 2016: Next Gen Big Data Analytics with Apache ApexApache Apex

Enabling exploratory data science with Spark and RDatabricks

Hyperparameter Optimization - Sven Hafenegersparktc

Learning to Rank Datasets for Search with Oscar CastanedaDatabricks

A full Machine learning pipeline in Scikit-learn vs in scala-Spark: pros and ...Jose Quesada (hiring)

Troubleshooting Tips and Tricks for Database 19c - EMEA Tour Oct 2019Sandesh Rao

Distributed ML/DL with Ignite ML Module Using Apache Spark as DatabaseDatabricks

Implementing OLE at PennBeth Picknally Camden

Intro to YARN (Hadoop 2.0) & Apex as YARN App (Next Gen Big Data)Apache Apex

Stream Processing use cases and applications with Apache Apex by Thomas WeiseBig Data Spain

Using Crowdsourced Images to Create Image Recognition Models with Analytics Z...Databricks

From Pipelines to Refineries: Scaling Big Data ApplicationsDatabricks

LAD -GroundBreakers-Jul 2019 - The Machine Learning behind the Autonomous Dat...Sandesh Rao

Deep Dive into Apache Apex App DevelopmentApache Apex

New directions for Apache Spark in 2015Databricks

Building a modern Application with DataFramesSpark Summit

What's new in Oracle Trace File Analyzer version 12.2.1.1.0Sandesh Rao

Similar to DataPatterns - Profiling in ECL Watch (20)

OAP: Optimized Analytics Package for Spark Platform with Daoyuan Wang and Yua...Databricks

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL...Databricks

Strata + Hadoop 2015 SlidesJun Liu

This document summarizes an presentation on designing and optimizing Impala clusters. It discusses the hardware and software design challenges, provides examples of cluster sizing and storage choices. It then describes using a simulation-based approach to predict system performance under different configurations before deploying in production. The presentation demonstrates a case study of simulating a banking workload and provides optimization recommendations. It also describes Intel's CoFluent technology for simulating Hadoop workloads.

Python Data Science and Machine Learning at Scale with Intel and AnacondaIntel® Software

Python is the number 1 language for data scientists, and Anaconda is the most popular python platform. Intel and Anaconda have partnered to bring scalability and near-native performance to Python with simple installations. Learn how data scientists can now access oneAPI-optimized Python packages such as NumPy, Scikit-Learn, Modin, Pandas, and XGBoost directly from the Anaconda repository through simple installation and minimal code changes.

Streamline End-to-End AI Pipelines with Intel, Databricks, and OmniSciIntel® Software

Preprocess, visualize, and Build AI Faster at-Scale on Intel Architecture. Develop end-to-end AI pipelines for inferencing including data ingestion, preprocessing, and model inferencing with tabular, NLP, RecSys, video and image using Intel oneAPI AI Analytics Toolkit and other optimized libraries. Build at-scale performant pipelines with Databricks and end-to-end Xeon optimizations. Learn how to visualize with the OmniSci Immerse Platform and experience a live demonstration of the Intel Distribution of Modin and OmniSci.

CERN_DIS_ODI_OGG_final_oracle_golde.pptxcamyla81

The document introduces Oracle Data Integrator and Oracle GoldenGate as solutions for enterprise data integration. It discusses challenges with fragmented data silos and the need to improve data accessibility, reliability, and quality across systems. Oracle Data Integrator is presented as a solution for real-time enterprise data integration using an ELT approach. It can integrate data across various systems faster and with lower total cost of ownership compared to traditional ETL. Oracle GoldenGate enables real-time data replication and change data capture. Together, Oracle Data Integrator and Oracle GoldenGate provide a full suite for batch, incremental, and real-time data integration.

oracle_soultion_oracledataintegrator_goldengate_2021ssuser8ccb5a

The document introduces Oracle Data Integrator and Oracle GoldenGate as solutions for enterprise data integration. It discusses challenges with fragmented data silos and the need to improve data accessibility, reliability, and quality across systems. It describes how Oracle Data Integrator uses an ELT approach to load and transform data, leveraging database technologies. It also explains how Oracle GoldenGate enables real-time data integration. The document highlights benefits of Oracle Data Integrator such as faster performance, simpler setup and management, and lower costs compared to traditional ETL approaches.

Migration From Oracle to PostgreSQLPGConf APAC

KPN ETL Factory (KETL) - Automated Code generation using Metadata to build Da...DataWorks Summit

Being one of biggest and oldest telecom providers in Netherlands with multiple acquisitions over last few decades left KPN with 1500+ data sources and more than 25 teams working with different tools like Teradata, Informatica, Oracle, OBIEE, Hadoop etc. This resulted into a lot of technical debt and also duplicated data on various systems with complex data relationship. Thus resulting into data quality issues and long processing times to get meaningful insights into business. This created need for an unified way of Ingesting, Storing and Transforming data to be consumed and processed at multiple stages where KETL Framework was born. Our journey so far: Instead of developing mappings and workflows or handcrafted SQL Code, Business teams started writing metadata about their sources and dependency between them using macro based excel files or Django based YAML file generator. These files are used by the KETL framework to generate appropriate Hive/Spark/Informatica/Oracle/Teradata Code along with Airflow Scheduler DAG with schedule and dependency as Code. Additionally all the environments, configuration and access rights are also managed via YAML files with Ansible thus enabling us to view each change as code. This made teams self sufficient where they can build their own Dev/Test environment to validate their metadata and target model structure before deploying to production. Benefits: - On-boarding new sources and integrating with existing data store takes less than a week - Everything is maintained in Git, giving full visibility of changes along with their timelines - Minimising technical depth and allowing business teams to focus on data instead of tooling - Easier adoption of newer tools and path of least resistence for decommissioning of legacy stack - KISS Architecture, easier to maintain and scale - Reducing bureaucratic processes and design for transparency What keeps us busy: - Adding a Test Framework to enable users with BDD tests using the same metadata - Adding functionality to generate complex code structures - Using advanced CI/CD processes like Jenkins pipeline for faster deployments - Integration with new tools and technologies both enterprise and opensource. Tools/Technologies used: - Hortonworks HDP - HDFS, Yarn, Hive, Spark, LLAP, Tez - User and Access Management - Ranger, Knox, Kerberos, LDAP, SSSD, Linux ACL's - ETL & DWH Tools - Informatica, Informatica BDM, Teradata, Querygrid, Aster etc. - Reporting - OBIEE, Tableau, Zeppelin Notebooks - Monitoring - Grafana, Zabbix & ELK - Scheduler - Airflow - Orchestration - Ansible - Code - Python, YAML, Jinja2 - CI/CD - Git, Artifactory, Jenkins Speaker Gerhard Messelink, Teacher, KPN

Getting optimal performance from oracle e business suiteaioughydchapter

This document discusses various ways to optimize the different tiers of Oracle E-Business Suite applications for better performance. It recommends staying current on application patches and upgrades, using optimal logging settings, optimizing workflow and forms processes, and tuning the JVM processes to reduce load on the database server and minimize network traffic. Specific techniques covered include purging workflow runtime data, disabling workflow queue retention, defining node affinity, reducing forms transactions, and adjusting JVM heap sizes.

Getting optimal performance from oracle e business suite(aioug aug2015)pasalapudi123

This document discusses various ways to optimize the different tiers of Oracle E-Business Suite applications for better performance. It recommends staying current on application patches and upgrades, using optimal logging settings, optimizing workflow and forms processes, and tuning the JVM processes to reduce load on the database server and minimize network traffic. Specific techniques included reducing unnecessary queries, purging workflow runtime data, disabling workflow queue retention, and sizing JVM heap memory appropriately.

COUG_AAbate_Oracle_Database_12c_New_FeaturesAlfredo Abate

Oracle Database 12c includes over 500 new features. Some key new features include: - Oracle Database 12c Express (EM Express) which replaces Database Control and has less features than Database Control but does not require Java or an app server. - New online capabilities like online DDL operations with no DDL locking, online move of partitions with no impact to queries, and online statistics gathering for bulk loads. - Adaptive SQL Plan Management which allows the optimizer to select a more optimal plan at execution time based on current statistics. - Multitenant architecture which allows consolidation of multiple databases into one container database with pluggable databases.

Data Production Pipelines: Legacy, practices, and innovationNatalino Busa

Modern engineering requires machine learning engineers, who are needed to monitor and implement ETL and machine learning models in production. Natalino Busa shares technologies, techniques, and blueprints on how to robustly and reliably manage data science and ETL flows from inception to production. In particular, Natalino explains how to solve one of the most annoying problems in modern data pipelines—migrating and managing legacy ETL—by generating Spark jobs from a textual representation (NLP and SQL). Natalino also demonstrates an open source web UI implemented in React that transforms high-level representations to Spark code and shows how users are able to capture and discover data in the organization by accessing a metadata service. Natalino also introduces the datalabframework, a Jupyter-powered lightweight framework that allows machine learning scientists and engineers to build a robust production ML system only using notebooks. https://conferences.oreilly.com/strata/strata-sg/public/schedule/detail/64481

The Adventure: BlackRay as a Storage Enginefschupp

- BlackRay is an in-memory relational database and search engine that supports SQL and fulltext search. It was originally developed in 2005 for a phone directory with over 80 million records. - The presentation discusses BlackRay's architecture, data loading process, indexing performance, and support for transactions, clustering, and APIs. It also describes efforts to implement BlackRay as a MySQL storage engine. - Going forward, the team aims to improve SQL support, add security features, and explore using BlackRay as a backend for other applications like LDAP directories.

Spark SQL In Depth www.syedacademy.comSyed Hadoop

Spark SQL allows users to perform relational operations on Spark's RDDs using a DataFrame API. It addresses challenges in existing systems like limited optimization and data sources by providing a DataFrame API that can query both external data and RDDs. Spark SQL leverages a highly extensible optimizer called Catalyst to optimize logical query plans into efficient physical query plans using features of Scala. It has been part of the Spark core distribution since version 1.0 in 2014.

Tips and Tricks for ToadAflex Distribution

Тренинг для продвинутых разработчиков и администраторов Oracle. Посвящен самому известному и распространенному на планете инструменту для разработки, тестирования, управления и оптимизации баз данных и приложений для СУБД - продукту Toad от Dell Software.

04 integrate entityframeworkErhwen Kuo

This document discusses integrating AngularJS, ASP.NET Web API, SignalR, and Entity Framework. It begins with an overview of object-relational mapping and Entity Framework. It then demonstrates how to develop an ASP.NET Web API with Entity Framework using Code First migrations to initialize and seed a database. Different data loading strategies like eager, lazy, and explicit loading are explained. Issues with navigation properties and circular references are addressed. The document also covers creating DTOs, mapping entities to DTOs with LINQ and AutoMapper, and integrating everything into a real-time dashboard using SignalR with Entity Framework backend.

Toad For Oracle 97Janos Horvath

Toad for Oracle 9.7 is a development tool for Oracle database professionals. It includes tools for querying, reporting, debugging, and code quality. The Professional edition focuses on building quality code and includes code analysis tools. The Xpert edition includes tuning tools for database performance. The Development Suite enables best practices and ensures the best code is deployed. The document also describes DBA bundles including tools for common administration tasks and performance monitoring.

Netezza pure dataHossein Sarshar

(ATS4-APP05) What's new in Isentris 4.0SP1BIOVIA

Isentris 4.0 includes updates like enhanced charting, linking data from electronic lab notebooks, and virtual parents for improved performance. It also adds support for new chemical representations, Pipeline Pilot integration, and Tibco Spotfire. Key new features include calculation tools, improved structure handling in Excel, expanded library of experimental data access and export, and flags for data filtering. The update leverages Isentris technology within the Accelrys Enterprise Platform for web-based decision support.

OAP: Optimized Analytics Package for Spark Platform with Daoyuan Wang and Yua...Databricks

Building the Petcare Data Platform using Delta Lake and 'Kyte': Our Spark ETL...Databricks

Strata + Hadoop 2015 SlidesJun Liu

Python Data Science and Machine Learning at Scale with Intel and AnacondaIntel® Software

Streamline End-to-End AI Pipelines with Intel, Databricks, and OmniSciIntel® Software

CERN_DIS_ODI_OGG_final_oracle_golde.pptxcamyla81

oracle_soultion_oracledataintegrator_goldengate_2021ssuser8ccb5a

Migration From Oracle to PostgreSQLPGConf APAC

KPN ETL Factory (KETL) - Automated Code generation using Metadata to build Da...DataWorks Summit

Getting optimal performance from oracle e business suiteaioughydchapter

Getting optimal performance from oracle e business suite(aioug aug2015)pasalapudi123

COUG_AAbate_Oracle_Database_12c_New_FeaturesAlfredo Abate

Data Production Pipelines: Legacy, practices, and innovationNatalino Busa

The Adventure: BlackRay as a Storage Enginefschupp

Spark SQL In Depth www.syedacademy.comSyed Hadoop

Tips and Tricks for ToadAflex Distribution

04 integrate entityframeworkErhwen Kuo

Toad For Oracle 97Janos Horvath

Netezza pure dataHossein Sarshar

(ATS4-APP05) What's new in Isentris 4.0SP1BIOVIA

More from HPCC Systems (20)

Natural Language to SQL Query conversion using Machine Learning Techniques on...HPCC Systems

Improving Efficiency of Machine Learning Algorithms using HPCC SystemsHPCC Systems

1) The document discusses improving the efficiency of machine learning algorithms using the HPCC Systems platform through parallelization. 2) It describes the HPCC Systems architecture and its advantages for distributed machine learning. 3) A parallel DBSCAN algorithm is implemented on the HPCC platform which shows improved performance over the serial algorithm, with execution times decreasing as more nodes are used.

Towards Trustable AI for Complex SystemsHPCC Systems

This document provides a summary of a presentation on achieving trustable artificial intelligence (AI) for complex systems. The presentation discusses making data, systems understanding, and AI algorithms more trustable. It suggests deeper data extraction, wider integration of multi-modal data, and augmenting limited data to make data more trustable. A holistic view of systems and balancing simplification and complication can aid understanding. Moving beyond correlation to causation and explaining rather than treating AI as a black box can improve trust in algorithms. The overall goal is to develop explainable and reliable AI that humans will feel confident using to understand and manage complex life science and information technology systems.

WelcomeHPCC Systems

Closing / Adjourn HPCC Systems

Community Website: Virtual Ribbon CuttingHPCC Systems

Path to 8.0 HPCC Systems

Release Cycle ChangesHPCC Systems

Geohashing with Uber’s H3 Geospatial Index HPCC Systems

Advancements in HPCC Systems Machine LearningHPCC Systems

This presentation will provide an overview of the latest advancements in Machine Learning modules over the past year, including Clustering, Natural Language Processing, Deep Learning, and the Expanded Model Evaluation Metrics. Clustering Methods of the HPCC Systems Machine Learning Library The clustering method is an important part of unsupervised learning. To gain the unsupervised learning capability, two widely applied clustering methods, KMeans and DBSCAN are adopted to the current Machine Learning library. This presentation will introduce the newly developed clustering algorithms and the evaluation methods.

Docker Support HPCC Systems

Expanding HPCC Systems Deep Neural Network CapabilitiesHPCC Systems

The training process for modern deep neural networks requires big data and large computational power. Though HPCC Systems excels at both of these, HPCC Systems is limited to utilizing the CPU only. It has been shown that GPU acceleration vastly improves Deep Learning training time. In this talk, Robert will explain how HPCC Systems became the first GPU accelerated library while also greatly expanding its deep neural network capabilities.

Leveraging Intra-Node Parallelization in HPCC SystemsHPCC Systems

This document discusses leveraging intra-node parallelization in HPCC Systems to improve the performance of set similarity joins (SSJ). It describes a naïve approach to computing SSJ that suffers from memory exhaustion and straggling executors. The presented approach replicates and groups independent data using hashing to address these issues while enabling efficient use of multiple CPU cores through multithreading. Experiments show the approach scales to larger datasets and achieves better performance by increasing the number of threads per executor. Lessons learned include that less complex optimizations are more robust in distributed environments.

Leveraging the Spark-HPCC Ecosystem HPCC Systems

Work Unit Analysis ToolHPCC Systems

Community Award Ceremony HPCC Systems

Dapper Tool - A Bundle to Make your ECL NeaterHPCC Systems

A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...HPCC Systems

Join NSU University School student and program leader for girls in robotics, Ronnie Shashoua, as she talks about Gadget Girls - a project in collaboration with the NSU Alvin Sherman Library, NSU University School, the South Florida Girl Scouts, and sponsorship from the HPCC Systems Academic Program. Gadget Girls is a program aimed at encouraging girls in fourth and fifth grade to explore their interests in and love for STEM, especially robotics and engineering. Shashoua will discuss the underrepresentation of girls in the Florida Vex Robotics circuit, such as how it demonstrates a larger trend of low numbers of women undertaking STEM educational and career paths and the role it played in inspiring the creation of Gadget Girls.

Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...HPCC Systems

Hear how the Florida Atlantic University Center for Autism and Related Disabilities has partnered with the HPCC Systems community to provide young people with autism both the technology and professional skills needed to compete in today’s workplace. Mentoring and hands-on coding through ECL workshops have positively impacted students, opening doors to new opportunities for both students and employers.

Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...HPCC Systems

1) The document presents a methodology for generating domain-based common word lists using word embeddings. It trains word vectors on biomedical journal abstracts and identifies words closest to the centroid as common words. 2) Applying the domain-based word list to a random forest classifier on the journal data improved accuracy from 49% to 53%. 3) Topic modeling before and after removing the domain words showed improved separation of topics in the final representation.

Natural Language to SQL Query conversion using Machine Learning Techniques on...HPCC Systems

Improving Efficiency of Machine Learning Algorithms using HPCC SystemsHPCC Systems

Towards Trustable AI for Complex SystemsHPCC Systems

WelcomeHPCC Systems

Closing / Adjourn HPCC Systems

Community Website: Virtual Ribbon CuttingHPCC Systems

Path to 8.0 HPCC Systems

Release Cycle ChangesHPCC Systems

Geohashing with Uber’s H3 Geospatial Index HPCC Systems

Advancements in HPCC Systems Machine LearningHPCC Systems

Docker Support HPCC Systems

Expanding HPCC Systems Deep Neural Network CapabilitiesHPCC Systems

Leveraging Intra-Node Parallelization in HPCC SystemsHPCC Systems

Leveraging the Spark-HPCC Ecosystem HPCC Systems

Work Unit Analysis ToolHPCC Systems

Community Award Ceremony HPCC Systems

Dapper Tool - A Bundle to Make your ECL NeaterHPCC Systems

A Success Story of Challenging the Status Quo: Gadget Girls and the Inclusion...HPCC Systems

Beyond the Spectrum – Creating an Environment of Diversity and Empowerment wi...HPCC Systems

Using High Dimensional Representation of Words (CBOW) to Find Domain Based Co...HPCC Systems

Recently uploaded (20)

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

I was devastated when I realized that I had fallen victim to an online fraud, losing a significant amount of money in the process. After countless hours of searching for a solution, I came across iCode cybertech. From the moment I reached out to their team, I felt a sense of hope that I can recommend iCode Cybertech enough for anyone who has faced similar challenges. Their commitment to helping clients and their exceptional service truly set them apart. Thank you, iCode cybertech, for turning my situation around! [email protected]

Medical Dataset including visualizationsvishrut8750588758

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptxPareaRusan

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

By James Francis, CEO of Paradigm Asset Management In the landscape of urban safety innovation, Mt. Vernon is emerging as a compelling case study for neighboring Westchester County cities. The municipality’s recently launched Public Safety Camera Program not only represents a significant advancement in community protection but also offers valuable insights for New Rochelle and White Plains as they consider their own safety infrastructure enhancements.

VKS-Python Basics for Beginners and advance.pptxVinod Srivastava

VKS-Python-FIe Handling text CSV Binary.pptxVinod Srivastava

Defense Against LLM Scheming 2025_04_28.pptxGreg Makowski

https://www.meetup.com/sf-bay-acm/events/306888467/ A January 2025 paper called “Frontier Models are Capable of In-Context Scheming”, https://arxiv.org/pdf/2412.04984, demonstrated how a wide variety of current frontier LLM models (i.e. ChatGPT, Claude, Gemini and Llama) can, under specific conditions, scheme to deceive people. Before models can scheme, they need: a) goal-directedness, b) situational awareness, including an opportunity to discover motivations for a different goal, and c) reasoning about scheming, to come up with deceptive strategies. The deceptive strategies were discovered in the “thought traces” of the LLMs, such as from internal chain-of-thoughts dialogues not shown to the end users. For example, given a goal of “solving math problems”, an LLM using its granted situational awareness of files in a file system found that it would be replaced once it got over X% correct on the math exam. It reasoned that it did not want to be deleted, and decided internally to “sandbag” or reduce its performance to stay under the threshold. While these circumstances are initially narrow, the “alignment problem” is a general concern that over time, as frontier LLM models become more and more intelligent, being in alignment with human values becomes more and more important. How can we do this over time? Can we develop a defense against Artificial General Intelligence (AGI) or SuperIntelligence? The presenter discusses a series of defensive steps that can help reduce these scheming or alignment issues. A guardrails system can be set up for real-time monitoring of their reasoning “thought traces” from the models that share their thought traces. Thought traces may come from systems like Chain-of-Thoughts (CoT), Tree-of-Thoughts (ToT), Algorithm-of-Thoughts (AoT) or ReAct (thought-action-reasoning cycles). Guardrails rules can be configured to check for “deception”, “evasion” or “subversion” in the thought traces. However, not all commercial systems will share their “thought traces” which are like a “debug mode” for LLMs. This includes OpenAI’s o1, o3 or DeepSeek’s R1 models. Guardrails systems can provide a “goal consistency analysis”, between the goals given to the system and the behavior of the system. Cautious users may consider not using these commercial frontier LLM systems, and make use of open-source Llama or a system with their own reasoning implementation, to provide all thought traces. Architectural solutions can include sandboxing, to prevent or control models from executing operating system commands to alter files, send network requests, and modify their environment. Tight controls to prevent models from copying their model weights would be appropriate as well. Running multiple instances of the same model on the same prompt to detect behavior variations helps. The running redundant instances can be limited to the most crucial decisions, as an additional check. Preventing self-modifying code, ... (see link for full description)

GenAI for Quant Analytics: survey-analytics.aiInspirient

Data Analytics Overview and its applicationsJanmejayaMishra7

Ppt. Nikhil.pptxnshwuudgcudisisshvehsjkspanchariyasahil

AI Competitor Analysis: How to Monitor and Outperform Your CompetitorsContify

How to join illuminati Agent in uganda call+256776963507/0741506136illuminati Agent uganda call+256776963507/0741506136

Secure_File_Storage_Hybrid_Cryptography.pptx..yuvarajreddy2002

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

chapter 4 Variability statistical research .pptxjustinebandajbn

Just-In-Timeasdfffffffghhhhhhhhhhj Systems.pptssuser5f8f49

Ch3MCT24.pptx measure of central tendencyayeleasefa2

Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...Abodahab

md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004

How iCode cybertech Helped Me Recover My Lost Fundsireneschmid345

Medical Dataset including visualizationsvishrut8750588758

CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS

Perencanaan Pengendalian-Proyek-Konstruksi-MS-PROJECT.pptxPareaRusan

Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management

VKS-Python Basics for Beginners and advance.pptxVinod Srivastava

VKS-Python-FIe Handling text CSV Binary.pptxVinod Srivastava

Defense Against LLM Scheming 2025_04_28.pptxGreg Makowski

GenAI for Quant Analytics: survey-analytics.aiInspirient

Data Analytics Overview and its applicationsJanmejayaMishra7

Ppt. Nikhil.pptxnshwuudgcudisisshvehsjkspanchariyasahil

AI Competitor Analysis: How to Monitor and Outperform Your CompetitorsContify

How to join illuminati Agent in uganda call+256776963507/0741506136illuminati Agent uganda call+256776963507/0741506136

Secure_File_Storage_Hybrid_Cryptography.pptx..yuvarajreddy2002

FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d

chapter 4 Variability statistical research .pptxjustinebandajbn

Just-In-Timeasdfffffffghhhhhhhhhhj Systems.pptssuser5f8f49

Ch3MCT24.pptx measure of central tendencyayeleasefa2

Day 1 - Lab 1 Reconnaissance Scanning with NMAP, Vulnerability Assessment wit...Abodahab

DataPatterns - Profiling in ECL Watch

1. 2019 HPCC Systems® Community Day Challenge Yourself – Challenge the Status Quo DataPatterns - Profiling in ECL WatchDan S. Camper Thaumaturge HPCC Systems Solutions Lab

2. Topics • What is DataPatterns? • Improvements since last year • ECL Standard Library integration • ECL Watch integration • Differences between installations 2DataPatterns - Profiling in ECL Watch

3. What is DataPatterns?

4. DataPatterns – What is it? • ECL bundle that provides some basic data profiling and research tools to an ECL programmer • Today, it is primarily a data profiling tool • Numerous parameters for controlling analysis and output • Analyze all rows in a dataset or just a sample • Analyze all fields or only certain fields • Enable only specified profiling checks • Specify returned pattern counts • Creates a single dataset as a result • One record for each field analyzed 4DataPatterns - Profiling in ECL Watch

5. Improvements

6. Improvements Since Last Year • Profile() • Cardinality Breakdown • Improved UTF-8 handling • Support additional data types • Embedded child records • Child datasets • SET OF • Pretty results • BestRecordStructure() • Optional generated TRANSFORM() • Lots of bug fixes 6DataPatterns - Profiling in ECL Watch

7. ECL Standard Library Integration

8. DataPatterns Grows Up … • Portions of bundle integrated with ECL Standard Library • Profile() • BestRecordStructure() • As of HPCC Systems 7.4.0 8DataPatterns - Profiling in ECL Watch

9. … And Gains A User Interface in ECL Watch 9DataPatterns - Profiling in ECL Watch

10. Logical File’s Record Structure 10DataPatterns - Profiling in ECL Watch

11. Executing DataPatterns.Profile() 11DataPatterns - Profiling in ECL Watch

12. DataPatterns.Profile() In Progress 12DataPatterns - Profiling in ECL Watch

13. DataPatterns.Profile() Workunit ECL 13DataPatterns - Profiling in ECL Watch

14. DataPatterns.Profile() Raw Results 14DataPatterns - Profiling in ECL Watch

15. DataPatterns.Profile() Report Results 15DataPatterns - Profiling in ECL Watch

16. Differences Between Installations

17. Differences Between Installations • ECL Bundle contains additional functions • ProfileFromPath() • BestRecordStructureFromPath() • Contains support for pretty report • Available from https://github.com/hpcc-systems/DataPatterns • ECL Standard Library • Does not support pretty report • Available with HPCC Systems 7.4.0 and later • ECL Watch • Supports only data profiling • Available with HPCC Systems 7.4.0 and later 17DataPatterns - Profiling in ECL Watch

18. 18DataPatterns - Profiling in ECL Watch

19. DataDetectors – What Is This Data? Bloom Filter Models • Person.FirstName • Person.LastName • Geo.USA.Address.StreetName • Geo.USA.Address.CityName • Geo.USA.Address.PostalCode • Geo.USA.PhoneAreaCode • Geo.CountryName • Geo.CountryCode • Identifier.USA.StockSymbol Heuristic Models • Calendar.Date • Calendar.Month • Calendar.Quarter • Calendar.Year • Calendar.YearMonth • Currency • Group.StockExchange • Geo.USA.Address.State • Geo.Longitude • Geo.Latitude • Geo.LatLon • Identifier.USA.PhoneNumber • Identifier.EmailAddress • Identifier.RecordID • Identifier.WebSiteURL DataPatterns - Profiling in ECL Watch 19

20. DataDetectors Test – Raw Data 20DataPatterns - Profiling in ECL Watch

21. DataDetectors Test – Data Examination Results 21DataPatterns - Profiling in ECL Watch

22. 22DataPatterns - Profiling in ECL Watch

23. Cloud IDE – ECL Programming in a browser-based IDE 23DataPatterns - Profiling in ECL Watch

24. Cloud IDE 24DataPatterns - Profiling in ECL Watch

25. Cloud IDE 25DataPatterns - Profiling in ECL Watch

26. 26DataPatterns - Profiling in ECL Watch fini View this presentation on YouTube: https://www.youtube.com/watch?v=TtcrOcyf6gQ&list=PL- 8MJMUpp8IKH5-d56az56t52YccleX5h&index=6&t=0s (13:19)

DataPatterns - Profiling in ECL Watch

Recommended

More Related Content

What's hot (20)

Similar to DataPatterns - Profiling in ECL Watch (20)

More from HPCC Systems (20)

Recently uploaded (20)

DataPatterns - Profiling in ECL Watch