Poster Hadoop summit 2011: pig embedding in scripting languages

Jun 29, 2011Download as PPTX, PDF1 like948 views

Pig Scripting allows Pig to be used for iterative algorithms and user defined functions by embedding Pig within a scripting language. This allows transitive closure, an iterative process, to be computed with one file using Python functions as UDFs and Python variables in Pig scripts, rather than requiring seven files and Java UDFs as plain Pig would.

PIG ScriptingMaking Pig Turing-complete through embedding in a scripting languageJulien Le Dem - YahooOverviewPig execution model - Map/Reduce: a solution to process big data. - Pig: makes it easier to manipulate data on the grid. - Pig scripting: makes Pig easier with iterative algorithms and User Defined Functions.Jiras: PIG-1479, PIG-1794 (in 0.9), PIG-928 (in 0.8)Example: Transitive closure - Iterative process: requires a loop and a termination condition - Requires multiple join/group by: typical Pig usage - Requires User Defined FunctionsSolution 2: using Pig scriptingSolution 1: plain Pig1 file required:7 files required:UDFs take the elements of the tuple as parameters, not a tuple.The output schema is specified using a decorator (it can be a function if you need to manipulate the input schema)Modification flow:Modification flow:UDFs use standard Python constructs (tuple/list/dictionary) automatically converted to Pig.Python functions are automatically available as UDFsEmbedded Pig calls in PythonPython variables can be used in the pig scripts $n, $i, ...Unfortunately this solution does not fit here.It requires 7 artifacts: 3 Java UDFs: 3 classes must be compiled and packaged in a jar. The average UDF size is 50 to 100 lines of code.

This document discusses parallel programming using OpenMP and MPI. It begins by introducing OpenMP, describing it as an API for parallel programming using shared memory multiprocessing. Key aspects of OpenMP like compiler directives and the fork/join model are covered. The document then introduces MPI, describing it as a standard for message passing parallel programming on distributed memory systems. Basic MPI functions like Send, Recieve, Init and Finalize are defined. The document discusses strategies for parallelization including domain decomposition and task decomposition. It also provides examples of parallelizing loops and simple MPI and OpenMP code samples. In summary, the document provides an overview of OpenMP and MPI for parallel programming.

Python + Machine Learning Course, Session 2Amin Mesbahi

This document provides an overview and agenda for a Python fundamentals session, including discussions of Jupyter notebooks, magics, running scripts from Jupyter cells, and virtual environments. It also briefly introduces NumPy, describing NumPy arrays as the main data structure and noting NumPy is fast due to C library bindings. Key topics covered are Jupyter project overview, line and cell magics, running scripts from Jupyter using various interpreters, creating and activating virtual environments in virtualenv and Anaconda, and basic NumPy array types.

Pl/PythonCommand Prompt., Inc

Josh Williams You may be familiar with connecting to a database using Python, but did you know it's possible for your database itself to use Python? PostgreSQL has the ability to write functions and procedures in PL/Python, which can use Python to both work with data inside your database, and bring in information from the outside world. This tutorial-format demonstration is designed to show you how to get up and running on PL/Python, and some of the cool things you can accomplish using it.

Profiling in pythonJohn(Qiang) Zhang

This document provides an introduction to profiling in Python. It discusses three main profilers in the Python standard library: cProfile, profile, and hotshot. It also provides a quick example of how to profile an application using cProfile from the command line or by adding code to the application. The document concludes with a brief discussion of profiling in Cython and reviewing profile results using the pstats module.

Scientific PythonEueung Mulyana

This document provides an overview and introduction to key Python packages for scientific computing and data science. It discusses Jupyter notebooks for interactive coding and visualization, NumPy for N-dimensional arrays and math operations, SciPy for scientific computing functions, matplotlib for plotting, and pandas for working with labeled data structures. The document emphasizes that NumPy provides foundational N-dimensional arrays, SciPy builds on this with additional mathematical and scientific routines, and matplotlib and pandas complement these with visualization and labeled data functionality.

PyHEP 2019: Python 3.8Henry Schreiner

Introduction to ClimeMosky Liu

Some Rough Fibrous MaterialMurray Steele

Everything You Always Wanted to Know About Memory in Python - But Were Afraid...Piotr Przymus

Everything You Always Wanted to Know About Memory in Python But Were Afraid t...Piotr Przymus

Programming with Python - Adv.Mosky Liu

Boost.Python: C++ and Python IntegrationGlobalLogic Ukraine

오픈소스 라이브러리 개발기겨울 정

This document discusses best practices for Python package development including setup.py, testing, linting, documentation and continuous integration. It provides examples of using Pipenv, pytest, tox, coverage and Travis CI/Appveyor for testing and CI. Standard library modules like functools, itertools, pathlib and abc are demonstrated. API design, documentation, versioning and community guidelines are also covered.

What’s eating python performancePiotr Przymus

The Benefits of Type Hintsmasahitojp

The speaker discussed the benefits of type hints in Python. Type hints allow specifying the expected types of function parameters and return values, improving code readability, enabling code completion in editors, and allowing static type checking tools to analyze the code for type errors. The speaker demonstrated how to write type hints according to PEP 484 and PEP 526 standards and how to retrieve type information. Tools like Mypy were presented for doing static type analysis to catch errors. Using type hints and type checkers in continuous integration was recommended to catch errors early when collaborating on projects. The speaker concluded by explaining how using type hints made it easier for their team to port code from Python 2 to Python 3.

Using MPIKazuki Ohta

The document discusses using MPI (Message Passing Interface) for parallel programming on high performance computing systems, describing key MPI concepts like point-to-point communication, collective operations, and I/O functions. It also provides examples of how to implement simple MPI programs in C/C++/Fortran using libraries like MPICH2 and how MPI can scale to large clusters with millions of processes.

Open source projects with pythonroskakori

Embedding Pig in scripting languagesJulien Le Dem

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

- Arrow and Parquet are open source projects focused on column-oriented data formats for efficient in-memory (Arrow) and on-disk (Parquet) analytics. - They allow for interoperability across systems by eliminating the overhead of data serialization and enabling common data representations. - Column-oriented formats improve performance by reducing storage needs, enabling projection of only needed columns, and better utilization of CPU/memory through cache locality and vectorized processing.

Reducing the dimensionality of data with neural networksHakky St

(1) The document describes using neural networks called autoencoders to perform dimensionality reduction on data in a nonlinear way. Autoencoders use an encoder network to transform high-dimensional data into a low-dimensional code, and a decoder network to recover the data from the code. (2) The autoencoders are trained to minimize the discrepancy between the original and reconstructed data. Experiments on image and face datasets showed autoencoders outperforming principal components analysis at reconstructing the original data from the low-dimensional code. (3) Pretraining the autoencoder layers using restricted Boltzmann machines helps optimize the many weights in deep autoencoders and scale the approach to large datasets.

Low Latency Execution For Apache SparkJen Aman

This document describes Drizzle, a low latency execution engine for Apache Spark. It addresses the high overheads of Spark's centralized scheduling model by decoupling execution from scheduling through batch scheduling and pre-scheduling of shuffles. Microbenchmarks show Drizzle achieves milliseconds latency for iterative workloads compared to hundreds of milliseconds for Spark. End-to-end experiments show Drizzle improves latency for streaming and machine learning workloads like logistic regression. The authors are working on automatic batch tuning and an open source release of Drizzle.

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Parquet is a columnar storage format for Hadoop data. It was developed collaboratively by Twitter and Cloudera to address the need for efficient analytics on large datasets. Parquet provides more efficient compression and I/O compared to row-based formats by only reading and decompressing the columns needed by a query. It has been adopted by many companies for analytics workloads involving terabytes to petabytes of data. Parquet is language-independent and supports integration with frameworks like Hive, Pig, and Impala. It provides significant performance improvements and storage savings compared to traditional row-based formats.

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through enhancements like delta encoding, binary packing designed for CPU efficiency, and predicate pushdown using statistics. Benchmark results show Parquet provides much better compression and query performance than row-oriented formats on big data workloads. The project is developed as an open-source community with contributions from many organizations.

If you have your own Columnar format, stop now and use Parquet 😛Julien Le Dem

Lightning talk presented at HPTS 2015: http://hpts.ws/ Apache Parquet is the de facto standard columnar storage for big data. Open source and proprietary SQL engines already integrate with it as their users don’t want to load and duplicate their data in every tool. Users want an open, interoperable, efficient format to experiment with the many options they have. The format is defined by the open source community integrating feedback from many teams working on query engines (including but not limited to Impala, Drill, Hawq, SparkSQL, Presto, Hive, etc) or on infrastructure at scale (Twitter, Netflix, Stripe, Criteo, ...). Building on its initial success, the Parquet community is defining new features for the next iteration of the format. For example: improved metadata layout, type system completude or mergeable statistics used for planning.

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon

At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.

Multithreaded_Programming_in_Python.pdfgiridharsripathi

Multithreading allows programs to execute multiple tasks simultaneously by breaking tasks into independent threads that can run in parallel. In Python, the threading and _thread modules provide tools for multithreading. Threads share common memory and context switching between threads is faster than processes. The global interpreter lock in Python means only one thread can execute at a time, limiting true parallelism.

Multithreading by rjShree M.L.Kakadiya MCA mahila college, Amreli

The document discusses multithreaded programming in Python. It covers threads and the threading module, the difference between single-threaded and multithreaded execution, and provides an example of multithreading. Key points include that threads are lightweight processes that can run concurrently within a larger parent process and share resources like memory. The threading module provides higher-level thread management compared to the lower-level thread module.

Introduction to Chainer: A Flexible Framework for Deep LearningSeiya Tokui

concurrencyJonathan Wagoner

This document provides a brief overview of concurrency concepts in Python, focusing on threading. It defines processes and threads, explaining that threads run within a process and can share resources, while processes are independent and communicate via interprocess communication. It discusses why to use concurrency for performance, responsiveness, and non-blocking behavior. It also covers the Global Interpreter Lock in Python and its implications for threading, and provides examples of creating and running threads.

25 must know python for Interview by Tutort Academyyashikanigam1

**Master Python with Tutort Academy: The Ultimate Learning Experience** Python is one of the most powerful and versatile programming languages, widely used in **data science, web development, automation, artificial intelligence (AI), and machine learning (ML)**. At **Tutort Academy**, we offer **comprehensive Python courses** designed for beginners as well as working professionals looking to **upskill and advance their careers**. Whether you're a **software engineer, data analyst, or aspiring tech professional**, our courses will help you master Python with **real-world applications and industry-relevant projects**. ## **Why Choose Tutort Academy for Python Courses?** ### **1. Industry-Relevant Curriculum** Our Python courses are structured to provide in-depth knowledge, covering both **fundamentals and advanced concepts**. The curriculum includes: - **Python Basics:** Syntax, variables, loops, and conditional statements - **Object-Oriented Programming (OOP) in Python** - **Data Structures & Algorithms (DSA) with Python** - **Database Management (SQL & NoSQL) using Python** - **Web Development using Django and Flask** - **Automation and Scripting** - **Data Science, AI & Machine Learning with Python** - **System Design Concepts** (Ideal for professionals looking for a **system design course online**) ### **2. Hands-On Learning with Live Sessions** At Tutort Academy, we believe in **learning by doing**. Our courses feature **live interactive sessions, hands-on coding exercises, and real-world projects** to help you gain practical experience. Our **best DSA course online** integrates Python with **DSA and system design course** modules, ensuring that learners are well-equipped for technical interviews and job placements. ### **3. Expert Instructors & Personalized Mentorship** Our instructors are **industry experts with years of experience** in software development, AI, and system design. You will receive **personalized mentorship and career guidance**, helping you stay on the right track. ### **4. Flexible Learning Options for Professionals** We understand the challenges of balancing work and learning. That’s why our courses are tailored to be **flexible for working professionals**, offering: - **Weekend & evening live classes** - **Recorded sessions for self-paced learning** - **Doubt-clearing sessions with mentors** ### **5. Placement Assistance & Career Support** Our Python courses don’t just focus on technical skills; we also offer **resume building, mock interviews, and career counseling** to help you land your dream job. Many of our learners have successfully transitioned into roles in **top tech companies** after completing our **system design course online** and DSA modules. --- ## **Python Courses Offered at Tutort Academy** ### **1. Python for Beginners: From Zero to Hero** **Who is this for?** - Absolute beginners with no prior programming experience - Professionals switching careers to software development -

More Related Content

What's hot (9)

Everything You Always Wanted to Know About Memory in Python - But Were Afraid...Piotr Przymus

Everything You Always Wanted to Know About Memory in Python But Were Afraid t...Piotr Przymus

Programming with Python - Adv.Mosky Liu

Boost.Python: C++ and Python IntegrationGlobalLogic Ukraine

오픈소스 라이브러리 개발기겨울 정

What’s eating python performancePiotr Przymus

The Benefits of Type Hintsmasahitojp

Using MPIKazuki Ohta

Open source projects with pythonroskakori

Everything You Always Wanted to Know About Memory in Python - But Were Afraid...Piotr Przymus

Everything You Always Wanted to Know About Memory in Python But Were Afraid t...Piotr Przymus

Programming with Python - Adv.Mosky Liu

Boost.Python: C++ and Python IntegrationGlobalLogic Ukraine

오픈소스 라이브러리 개발기겨울 정

What’s eating python performancePiotr Przymus

The Benefits of Type Hintsmasahitojp

Using MPIKazuki Ohta

Open source projects with pythonroskakori

Viewers also liked (8)

Embedding Pig in scripting languagesJulien Le Dem

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

Reducing the dimensionality of data with neural networksHakky St

Low Latency Execution For Apache SparkJen Aman

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

If you have your own Columnar format, stop now and use Parquet 😛Julien Le Dem

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon

Embedding Pig in scripting languagesJulien Le Dem

Data Eng Conf NY Nov 2016 Parquet ArrowJulien Le Dem

Reducing the dimensionality of data with neural networksHakky St

Low Latency Execution For Apache SparkJen Aman

Parquet Strata/Hadoop World, New York 2013Julien Le Dem

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

If you have your own Columnar format, stop now and use Parquet 😛Julien Le Dem

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon

More from Julien Le Dem (19)

Data and AI summit: data pipelines observability with open lineageJulien Le Dem

Data pipelines observability: OpenLineage & MarquezJulien Le Dem

This document discusses OpenLineage and Marquez, which aim to provide standardized metadata and data lineage collection for data pipelines. OpenLineage defines an open standard for collecting metadata as data moves through pipelines, similar to metadata collected by EXIF for images. Marquez is an open source implementation of this standard, which can collect metadata from various data tools and store it in a graph database for querying lineage and understanding dependencies. This collected metadata helps with tasks like troubleshooting, impact analysis, and understanding how data flows through complex pipelines over time.

Open core summit: Observability for data pipelines with OpenLineageJulien Le Dem

This document discusses Open Lineage and the Marquez project for collecting metadata and data lineage information from data pipelines. It describes how Open Lineage defines a standard model and protocol for instrumentation to collect metadata on jobs, datasets, and runs in a consistent way. This metadata can then provide context on the data source, schema, owners, usage, and changes. The document outlines how Marquez implements the Open Lineage standard by defining entities, relationships, and facets to store this metadata and enable use cases like data governance, discovery, and debugging. It also positions Marquez as a centralized but modular framework to integrate various data platforms and extensions like Datakin's lineage analysis tools.

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

This document discusses principles for building a healthy data platform, including: 1. Establishing explicit contracts between teams to define dependencies and service level agreements. 2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest. 3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.

Data lineage and observability with Marquez - subsurface 2020Julien Le Dem

This document discusses Marquez, an open source metadata management system. It provides an overview of Marquez and how it can be used to track metadata in data pipelines. Specifically: - Marquez collects and stores metadata about data sources, datasets, jobs, and runs to provide data lineage and observability. - It has a modular framework to support data governance, data lineage, and data discovery. Metadata can be collected via REST APIs or language SDKs. - Marquez integrates with Apache Airflow to collect task-level metadata, dependencies between DAGs, and link tasks to code versions. This enables understanding of operational dependencies and troubleshooting. - The Marquez community aims to build an open

Strata NY 2018: The deconstructed databaseJulien Le Dem

The document discusses the evolution of big data systems from flat files in Hadoop to deconstructed databases. Originally, Hadoop used flat files with no schema and tied data storage to execution. This was flexible but inefficient. SQL databases provided schemas and separated storage from execution but were inflexible. Modern systems decompose databases into independent components like storage, processing, and machine learning that can be mixed and matched. Standards are emerging for columnar storage, querying, schemas, and data exchange. Future improvements include better interoperability, data abstractions, and governance.

From flat files to deconstructed databaseJulien Le Dem

From flat files to deconstructed databases: - Originally, Hadoop used flat files and MapReduce which was flexible but inefficient for queries. - The database world used SQL and relational models with optimizations but were inflexible. - Now components like storage, processing, and machine learning can be mixed and matched more efficiently with standards like Apache Calcite, Parquet, Avro and Arrow.

Strata NY 2017 Parquet Arrow roadmapJulien Le Dem

This document summarizes Apache Parquet and Apache Arrow, two open source projects for columnar data formats. It discusses how Parquet provides an on-disk columnar format for storage while Arrow provides an in-memory columnar format. The document outlines how Arrow builds on the success of Parquet by providing a common in-memory format that avoids serialization overhead and allows systems to share functionality. It provides examples of performance gains from the vertical integration of Parquet and Arrow.

The columnar roadmap: Apache Parquet and Apache ArrowJulien Le Dem

This document discusses Apache Parquet and Apache Arrow, open source projects for columnar data formats. Parquet is an on-disk columnar format that optimizes I/O performance through compression and projection pushdown. Arrow is an in-memory columnar format that maximizes CPU efficiency through vectorized processing and SIMD. It aims to serve as a standard in-memory format between systems. The document outlines how Arrow builds on Parquet's success and provides benefits like reduced serialization overhead and ability to share functionality through its ecosystem. It also describes how Parquet and Arrow representations are integrated through techniques like vectorized reading and predicate pushdown.

Improving Python and Spark Performance and Interoperability with Apache ArrowJulien Le Dem

This document discusses improving Python and Spark performance and interoperability with Apache Arrow. It begins with an overview of current limitations of PySpark UDFs, such as inefficient data movement and scalar computation. It then introduces Apache Arrow, an open source in-memory columnar data format, and how it can help by allowing more efficient data sharing and vectorized computation. The document shows how Arrow improved PySpark UDF performance by 53x through vectorization and reduced serialization. It outlines future plans to further optimize UDFs and integration with Spark and other projects.

Mule soft mar 2017 Parquet ArrowJulien Le Dem

This document discusses the future of column-oriented data processing with Apache Arrow and Apache Parquet. Arrow provides an open standard for in-memory columnar data, while Parquet provides an open standard for on-disk columnar data storage. Together they provide interoperability across systems and high performance by avoiding data copying and format conversions. The document outlines the goals and benefits of Arrow and Parquet, how they improve CPU and I/O efficiency, and examples of performance gains from integrating systems like Spark and Pandas with Arrow.

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

How to use Parquet as a basis for ETL and analyticsJulien Le Dem

Parquet is a columnar format designed to be extremely efficient and interoperable across the hadoop ecosystem. Its integration in most of the Hadoop processing frameworks (Impala, Hive, Pig, Cascading, Crunch, Scalding, Spark, …) and serialization models (Thrift, Avro, Protocol Buffers, …) makes it easy to use in existing ETL and processing pipelines, while giving flexibility of choice on the query engine (whether in Java or C++). In this talk, we will describe how one can us Parquet with a wide variety of data analysis tools like Spark, Impala, Pig, Hive, and Cascading to create powerful, efficient data analysis pipelines. Data management is simplified as the format is self describing and handles schema evolution. Support for nested structures enables more natural modeling of data for Hadoop compared to flat representations that create the need for often costly joins.

Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Julien Le Dem

Apache Parquet is an open-source columnar storage format for efficient data storage and analytics. It provides efficient compression and encoding techniques that enable fast scans and queries of large datasets. Parquet 2.0 improves on these efficiencies through techniques like delta encoding, dictionary encoding, run-length encoding and binary packing designed for CPU and cache optimizations. Benchmark results show Parquet provides much better compression and faster query performance than other formats like text, Avro and RCFile. The project is developed as an open source community with contributions from many organizations.

Parquet Hadoop Summit 2013Julien Le Dem

Parquet is a columnar storage format for Hadoop data. It was developed by Twitter and Cloudera to optimize storage and querying of large datasets. Parquet provides more efficient compression and I/O compared to traditional row-based formats by storing data by column. Early results show a 28% reduction in storage size and up to a 114% improvement in query performance versus the original Thrift format. Parquet supports complex nested schemas and can be used with Hadoop tools like Hive, Pig, and Impala.

Parquet Twitter Seattle open houseJulien Le Dem

This document discusses Parquet, an open-source columnar file format for Hadoop. It was created at Twitter to optimize their analytics infrastructure, which includes several large Hadoop clusters processing data from their 200M+ users. Parquet aims to improve on existing storage formats by organizing data column-wise for better compression and scanning capabilities. It uses a row group structure and supports efficient reads of individual columns. Initial results at Twitter found a 28% space savings over existing formats and scan performance improvements. The project is open source and aims to continue optimizing column storage and enabling new execution engines for Hadoop.

Parquet overviewJulien Le Dem

Parquet is a column-oriented storage format for Hadoop that supports efficient compression and encoding techniques. It uses a row group structure to store data in columns in a compressed and encoded column chunk format. The schema and metadata are stored in the file footer to allow for efficient reads and scans of selected columns. The format is designed to be extensible through pluggable components for schema conversion, record materialization, and encodings.

Data and AI summit: data pipelines observability with open lineageJulien Le Dem

Data pipelines observability: OpenLineage & MarquezJulien Le Dem

Open core summit: Observability for data pipelines with OpenLineageJulien Le Dem

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

Data lineage and observability with Marquez - subsurface 2020Julien Le Dem

Strata NY 2018: The deconstructed databaseJulien Le Dem

From flat files to deconstructed databaseJulien Le Dem

Strata NY 2017 Parquet Arrow roadmapJulien Le Dem

The columnar roadmap: Apache Parquet and Apache ArrowJulien Le Dem

Improving Python and Spark Performance and Interoperability with Apache ArrowJulien Le Dem

Mule soft mar 2017 Parquet ArrowJulien Le Dem

Strata NY 2016: The future of column-oriented data processing with Arrow and ...Julien Le Dem

Strata London 2016: The future of column oriented data processing with Arrow ...Julien Le Dem

Sql on everything with drillJulien Le Dem

How to use Parquet as a basis for ETL and analyticsJulien Le Dem

Efficient Data Storage for Analytics with Parquet 2.0 - Hadoop Summit 2014Julien Le Dem

Parquet Hadoop Summit 2013Julien Le Dem

Parquet Twitter Seattle open houseJulien Le Dem

Parquet overviewJulien Le Dem

Recently uploaded (20)

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

Most consumers believe they’re making informed decisions about their personal data—adjusting privacy settings, blocking trackers, and opting out where they can. However, our new research reveals that while awareness is high, taking meaningful action is still lacking. On the corporate side, many organizations report strong policies for managing third-party data and consumer consent yet fall short when it comes to consistency, accountability and transparency. This session will explore the research findings from TrustArc’s Privacy Pulse Survey, examining consumer attitudes toward personal data collection and practical suggestions for corporate practices around purchasing third-party data. Attendees will learn: - Consumer awareness around data brokers and what consumers are doing to limit data collection - How businesses assess third-party vendors and their consent management operations - Where business preparedness needs improvement - What these trends mean for the future of privacy governance and public trust This discussion is essential for privacy, risk, and compliance professionals who want to ground their strategies in current data and prepare for what’s next in the privacy landscape.

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

tecnologias de las primeras civilizaciones.pdffjgm517

Procurement Insights Cost To Value Guide.pptxJon Hansen

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

Spark is a powerhouse for large datasets, but when it comes to smaller data workloads, its overhead can sometimes slow things down. What if you could achieve high performance and efficiency without the need for Spark? At S&P Global Commodity Insights, having a complete view of global energy and commodities markets enables customers to make data-driven decisions with confidence and create long-term, sustainable value. 🌍 Explore delta-rs + CDC and how these open-source innovations power lightweight, high-performance data applications beyond Spark! 🚀

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, transcript, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Book industry standards are evolving rapidly. In the first part of this session, we’ll share an overview of key developments from 2024 and the early months of 2025. Then, BookNet’s resident standards expert, Tom Richardson, and CEO, Lauren Stewart, have a forward-looking conversation about what’s next. Link to recording, presentation slides, and accompanying resource: https://bnctechforum.ca/sessions/standardsgoals-for-2025-standards-certification-roundup/ Presented by BookNet Canada on May 6, 2025 with support from the Department of Canadian Heritage.

Splunk Security Update | Public Sector Summit Germany 2025Splunk

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-nomad-web-best-practices-und-verwaltung-von-multiuser-umgebungen/ HCL Nomad Web wird als die nächste Generation des HCL Notes-Clients gefeiert und bietet zahlreiche Vorteile, wie die Beseitigung des Bedarfs an Paketierung, Verteilung und Installation. Nomad Web-Client-Updates werden “automatisch” im Hintergrund installiert, was den administrativen Aufwand im Vergleich zu traditionellen HCL Notes-Clients erheblich reduziert. Allerdings stellt die Fehlerbehebung in Nomad Web im Vergleich zum Notes-Client einzigartige Herausforderungen dar. Begleiten Sie Christoph und Marc, während sie demonstrieren, wie der Fehlerbehebungsprozess in HCL Nomad Web vereinfacht werden kann, um eine reibungslose und effiziente Benutzererfahrung zu gewährleisten. In diesem Webinar werden wir effektive Strategien zur Diagnose und Lösung häufiger Probleme in HCL Nomad Web untersuchen, einschließlich - Zugriff auf die Konsole - Auffinden und Interpretieren von Protokolldateien - Zugriff auf den Datenordner im Cache des Browsers (unter Verwendung von OPFS) - Verständnis der Unterschiede zwischen Einzel- und Mehrbenutzerszenarien - Nutzung der Client Clocking-Funktion

Cyber Awareness overview for 2025 month of securityriccardosl1

How analogue intelligence complements AIPaul Rowe

Artificial Intelligence is providing benefits in many areas of work within the heritage sector, from image analysis, to ideas generation, and new research tools. However, it is more critical than ever for people, with analogue intelligence, to ensure the integrity and ethical use of AI. Including real people can improve the use of AI by identifying potential biases, cross-checking results, refining workflows, and providing contextual relevance to AI-driven results. News about the impact of AI often paints a rosy picture. In practice, there are many potential pitfalls. This presentation discusses these issues and looks at the role of analogue intelligence and analogue interfaces in providing the best results to our audiences. How do we deal with factually incorrect results? How do we get content generated that better reflects the diversity of our communities? What roles are there for physical, in-person experiences in the digital world?

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Hybrid Growth Mandate Model with TrsLabs Strategic Investments, Inorganic Growth, Business Model Pivoting are critical activities that business don't do/change everyday. In cases like this, it may benefit your business to choose a temporary external consultant. An unbiased plan driven by clearcut deliverables, market dynamics and without the influence of your internal office equations empower business leaders to make right choices. Getting things done within a budget within a timeframe is key to Growing Business - No matter whether you are a start-up or a big company Talk to us & Unlock the competitive advantage

Heap, Types of Heap, Insertion and DeletionJaydeep Kale

Linux Professional Institute LPIC-1 Exam.pdfRHCSA Guru

Role of Data Annotation Services in AI-Powered ManufacturingAndrew Leo

TrustArc Webinar: Consumer Expectations vs Corporate Realities on Data Broker...TrustArc

SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfPrecisely

tecnologias de las primeras civilizaciones.pdffjgm517

Procurement Insights Cost To Value Guide.pptxJon Hansen

Manifest Pre-Seed Update | A Humanoid OEM Deeptech In Francechb3

Electronic_Mail_Attacks-1-35.pdf by xploitniftliyevhuseyn

Massive Power Outage Hits Spain, Portugal, and France: Causes, Impact, and On...Aqusag Technologies

Drupalcamp Finland – Measuring Front-end Energy ConsumptionExove

Rusty Waters: Elevating Lakehouses Beyond Sparkcarlyakerly1

#StandardsGoals for 2025: Standards & certification roundup - Tech Forum 2025BookNet Canada

Transcript: #StandardsGoals for 2025: Standards & certification roundup - Tec...BookNet Canada

Splunk Security Update | Public Sector Summit Germany 2025Splunk

HCL Nomad Web – Best Practices und Verwaltung von Multiuser-Umgebungenpanagenda

Cyber Awareness overview for 2025 month of securityriccardosl1

How analogue intelligence complements AIPaul Rowe

The Evolution of Meme Coins A New Era for Digital Currency ppt.pdfAbi john

TrsLabs - Fintech Product & Business ConsultingTrs Labs

Poster Hadoop summit 2011: pig embedding in scripting languages

1. PIG ScriptingMaking Pig Turing-complete through embedding in a scripting languageJulien Le Dem - YahooOverviewPig execution model - Map/Reduce: a solution to process big data. - Pig: makes it easier to manipulate data on the grid. - Pig scripting: makes Pig easier with iterative algorithms and User Defined Functions.Jiras: PIG-1479, PIG-1794 (in 0.9), PIG-928 (in 0.8)Example: Transitive closure - Iterative process: requires a loop and a termination condition - Requires multiple join/group by: typical Pig usage - Requires User Defined FunctionsSolution 2: using Pig scriptingSolution 1: plain Pig1 file required:7 files required:UDFs take the elements of the tuple as parameters, not a tuple.The output schema is specified using a decorator (it can be a function if you need to manipulate the input schema)Modification flow:Modification flow:UDFs use standard Python constructs (tuple/list/dictionary) automatically converted to Pig.Python functions are automatically available as UDFsEmbedded Pig calls in PythonPython variables can be used in the pig scripts $n, $i, ...Unfortunately this solution does not fit here.It requires 7 artifacts: 3 Java UDFs: 3 classes must be compiled and packaged in a jar. The average UDF size is 50 to 100 lines of code.

2. 3 Pig scripts in 3 separate files: init, main loop content, finalization. The average Pig script size is 5 to 10 lines.

3. Main program: executes and coordinates the Pig scripts. It contains about 50 lines of code.This solution does fit on the poster.It requires 1 artifact, containing: 3 UDFs provided as functions: The average UDF size is 8 to 12 lines of code.

4. 3 Pig queries: init, main loop content, finalization. Each Pig query adds an average 5 to 10 lines.

5. Main function: executes and coordinates the Pig scripts. It adds about 10 lines to the script.Access to the output of pig scripts

Poster Hadoop summit 2011: pig embedding in scripting languages

Recommended

More Related Content

What's hot (9)

Viewers also liked (8)

Similar to Poster Hadoop summit 2011: pig embedding in scripting languages (20)

More from Julien Le Dem (19)

Recently uploaded (20)

Poster Hadoop summit 2011: pig embedding in scripting languages