Road to NODES - Handling Neo4j Data with Apache Hop

© 2022 Neo4j, Inc. All rights reserved.
1
Handling Neo4j Data with Apache Hop
Matt Casters
Neo4j Chief Solutions Architect
Apache Hop PMC

• Apache Hop introduction
• Setting up Hop
• The basics
• Neo4j functionality
• Advanced topics
• Q&A / Wrap-up
Topics

Apache Hop
“Facilitates all aspects of data and metadata
orchestration”
Typical use-cases:
● Data integration / Data orchestration / ETL
● Data migration
● Message processing
● Data synchronization
● IoT, Big Data, …

5
Bridging the gap…
Organizations Tech / Devs
Requirements

Concerns of organizations
6
• Setup costs
• Maintenance costs
• Running costs
• Time to market
• Resource availability & the bus factor
• DevOps
• Solution stability
• Investment protection

Concerns of developers
7
• Ability to succeed
• Have a fun development environment
• Ability to learn new things
• Work with new technology
• Use best development practices
• Life / Work balance
• …

Why Apache Hop?
● Lower development time and cost
● Lower maintenance time and cost
● Increase transparency
● Improve stability
● Make the learning curve steeper
● Protect against brain-drain
● …
https://xkcd.com/1319/

9
Why Apache Hop?
• A quickly diversifying technological data landscape
• → Makes it hard to manage complexity
• → Drives the need for rapid innovation
• → Makes it harder to support best dev practices
• Development done independent from a single large corporation
• By and for data orchestration professionals

10
Apache Hop
• Recursive acronym: Hop Orchestration Platform
• Orchestration:
◦ Data: pipelines and workflows
◦ Metadata: editing, handling, management,...
◦ Insights: data/execution lineage, logging, …
◦ Configurations: handling ecosystem complexity
• Platform:
◦ GUI, commands, server, scripts, docker, API, documentation,
community, ...

11
Background
• Community lead initiative
• Starting point was Kettle 8.2 + WebSpoon + patches + plugins + …
→ Representing 21 years of software development!
• New scalable GUI
• New architecture, metadata back-end
• Simplified toolset
• Code refactored, renamed, trimmed down, ...
• Extra plugins: Projects, Testing, Apache Beam, Debugging,
• …
• Years of work!

12
Apache Software Foundation
• Hop is a Top Level Project at the Apache Software Foundation
• Homepage: http://hop.apache.org
• Source: github.com/apache/hop/
• Building and IT on Apache Jenkins CI
• Released 2.1.0 : http://hop.apache.org/download
• Working on 2.2.0
• Fast growing and active community
• Check the website for regular updates & our Hot Hop Hangouts (3Hx)
⇒ 3H17 @ Thu 10 Nov 2022 7pm - 9pm CET

13
Key architecture features
• License: Apache Public License v2.0
• Metadata driven: no code generation
• Modular pluggable architecture: scale back to <30MB
• Fast startup, minimal overhead
• Apache Beam with support for Apache Spark, Apache Flink and GCP
Dataflow runners
• Version controlled documentation
• Ease of use: transparent naming and easy to use tools
• Integration test: critical components are tested daily with integration tests
• → runtime compatibility, stability, ...

14
Key GUI features
• Pluggable GUI features
• Scalable interface for high DPI displays or visually impaired
• Perspectives for easy fast context switching
• Designed for web browsers and mobile users
• → Single click mode for faster navigation
• Full support for 4 platforms: Windows, OSX, Linux & Web
• Support for “dark mode” themes on Linux and OSX

15
Key configuration features
• All GUI configuration options have a command line variant
• Single central system configuration JSON file
• Easy project and lifecycle environment configuration
• Configuration and metadata inheritance from other projects
• Standard docker containers
• Stateless server supporting multi-tenancy
• Version control friendly setup

16
Setting up Hop
With some live action

17
Download and install
• Visit hop.apache.org
• Download the latest version
• Use an up-to-date Java 11 Runtime Environment version:
◦ Windows
• Eclipse Adoptium OpenJDK build
• Microsoft build of OpenJDK
◦ Linux
• Just install OpenJDK 11
◦ OSX
• Azul build for x86 or ARM 64-bit

18
Configuration
Available configuration variables:
HOP_CONFIG_FOLDER Hop keeps a central system configuration file called
hop-config.json in this folder
HOP_AUDIT_FOLDER Usage history, logging and other auditing information is
stored here.
HOP_JAVA_HOME Points to the JRE you want to use
HOP_OPTIONS The Java options. For example you can change the
maximum memory consumption (2GB by default)
Give Hop tools 8GB or memory: -Xmx8g
HOP_SHARED_JDBC_FOLDER A folder where you can put your extra JDBC drivers

Tools
We’ll go over the tools in the Hop download:
● hop-conf : Hop Configuration
● hop-run : Run a pipeline or workflow
● hop-gui : Run the Hop graphical user interface.
● hop-search: Look for something in a project
● hop-server : Start a server to allow you to do remove execution
○ This is NOT a GUI, it’s just a server

20
Version control
Hop has built-in (basic) support for git integrated with the explorer perspective.

Check out a git repository
The samples from this workshop are stored here:
https://github.com/Neo4jSolutions/hop-nodes-2022

22
Projects
• Projects allow you to encapsulate all your work in a single folder
• Makes it easy to check in all your work into git
• Allows you to specify files in a relative way
◦ ${PROJECT_HOME}/files/input-file.csv
• Consider the following project options:

Create project
We’ll create a project using the git repository folder we checked out earlier:
hop-nodes-2022

24
Environments
Allow you to put environment specific variables separate from your project

Create environment
We’ll create a local development environment for our project:
hop-nodes-2022-dev-local

26
The basics
Pipelines, Workflows, Metadata, …

27
Concepts
Live explanation of Hop concepts and terminology using Hop GUI.
More information can be found online on the Hop website:
https://hop.apache.org/manual/latest/concepts.html
● Workflows and pipelines
● Actions and transforms
● Run configurations

28
Neo4j functionality
Actions, transforms & more

29
Live workshop topics
• Neo4j Connection
◦ How to set up for multiple environments
• Documentation: working with Neo4j data
• Northwind example
◦ Check Neo4j, Neo4j Constraint, Neo4j Output, Cypher,
◦ Graph Output with graph model
• Configure new Aura environment
◦ Run Northwind again
• Running on GCP Dataflow:
◦ BigQuery to Aura
◦ Snowflake to Aura
• Bulk loading basics

Beam vs Neo4j best practices
● Set transform copies to BEAM_BATCH
○ Set the flush interval and buffer size in the run configuration
● Match the parallelism to the number of CPUs that the Neo4j server has
○ Dataflow →Pick the right type and amount of worker node
○ Apache Spark / Flink: set the parallelism in the run configuration

31
Advanced topics
Execution information, docker, web services, hop-web, Apache Airflow k8s,

32
Execution information
Write information about the execution of workflows and pipelines to a location:
● A folder where execution information is written to JSON files
● A Hop server
● A Neo4j database where execution is written to a graph
⇒ live demo on the local dev environment and Northwind.

34
Best practices : Unit testing
Hop has built-in support for unit testing. It allows you to provide test data as
input to transforms and validates output of transforms.

35
Best practices : Integration testing
Integration testing is made easy with Hop. The project itself has daily
integration tests that are run:
https://ci-builds.apache.org/job/Hop/job/Hop-integration-tests/
It uses our standard Apache Hop docker container to execute. It uses other
containers for Postgres, Neo4j, Cassandra, ... to test functionality of those
plugins.

36
Logging & Reflection
• Capture logging and metrics from pipelines and workflows
◦ Every time you run a pipeline or workflow another one fires
• At the start
• At intervals during execution
• At the end of the execution
• Reflection: stream data to another pipeline in normalized format
◦ Only supported on “local” pipeline engines for now
• The Neo4j Logging perspective
◦ Install Neo4j or docker
◦ Configure a connection
◦ Set the NEO4J_LOGGING_CONNECTION variable

37
Web services
• Execute a pipeline
• Output results of a field as a result
◦ Supported only on a local pipeline engine
◦ But … for a Beam engine you can run a workflow
• Execute Beam pipeline
• Pick up results and output the results

38
Docker
• Great way to run your pipelines and workflows
• Starting point for customization
• Also capable of running hop-server
https://hub.docker.com/r/apache/hop
• Documentation:
https://hop.apache.org/tech-manual/latest/docker-container.html

39
Hop Web
• Run the Hop GUI in your browser!
• The exact same codebase is used
• Update the docker image:
docker pull apache/hop-web
• Run Hop Web Development in dark mode:
docker run
-p 8080:8080
-e HOP_WEB_THEME=dark
apache/hop-web:Development
• See also: the Hop Web developers guide

40
Apache Airflow
• The Hop community is working on specific Airflow Operators
• For now: use a BashOperator to execute
from datetime import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator
# Add a space after the shell filename, otherwise this does not work
#
create_command = "/home/matt/airflow/scripts/run-hop-pipeline.sh "
with DAG(dag_id='run_hop_pipeline'
, schedule_interval
="@daily",
start_date=datetime(2020, 1, 1), catchup=False) as dag:
bash_task = BashOperator(task_id
='bash_task', bash_command=create_command)

41
Apache Airflow
• This script runs a Beam sample pipeline using the Direct runner
#!/bin/bash
# Change to the Hop install folder
#
cd /opt/hop
# Set the configuration folder
#
export HOP_CONFIG_FOLDER=/opt/hop/config
# And the audit folder
#
export HOP_AUDIT_FOLDER=/opt/hop/audit
# Run a Beam pipeline from the samples with the Direct runner
#
sh hop-run.sh -j samples -f beam/pipelines/input-process-output.hpl -r Direct

Kubernetes
Run a pipeline on AWS EKS using the Apache Flink k8s operator.
→Automatically starts an Apache Flink cluster
→When the cluster is online, it starts an Apache Hop Pipeline

43
Apache Hop Roadmap
What’s next?

What’s next?
● Improvements & bug fixes
● Expand integration test suite
● A new welcome dialog with direct links and guided actions
● A new configuration perspective in the Hop GUI
● Give plugins access to the execution information
⇒ Build a Neo4j tab with paths to errors etc.
● Neo4j Cypher Builder transform
● ETA for 2.2.0 is mid-november (3-4 weeks from now)

45
Thank you! Any questions?
Join the Hop community: http://hop.apache.org

46
Join the Hop Community
• Website: hop.apache.org
• File a bug or feature request in JIRA
• Mattermost chat server
• The source code on github.
• Jenkins Hop builds
•
• Twitter: https://twitter.com/apachehop
• Youtube: https://youtube.com/apachehop
• Linkedin: https://www.linkedin.com/company/apachehop/
Welcome!

Road to NODES - Handling Neo4j Data with Apache Hop

Recommended

More Related Content

What's hot (20)

Similar to Road to NODES - Handling Neo4j Data with Apache Hop (20)

More from Neo4j (20)

Recently uploaded (20)

Road to NODES - Handling Neo4j Data with Apache Hop