Apache zeppelin, the missing component for the big data ecosystem
10 likes1,798 views
Apache Zeppelin is a web-based notebook that allows users to interact with data via interpreters like Spark, SQL, and Cassandra. It provides a GUI for data scientists to write code and visualizations in notebooks. Zeppelin has a modular architecture that allows new interpreters to be easily added. It also includes features like scheduling, sharing, and exporting of notebooks.
Apache zeppelin, the missing component for the big data ecosystem
- 1. @doanduyhai Apache Zeppelin, the missing GUI for your BigData eco-system DuyHai DOAN, Technical Advocate
- 2. @doanduyhai Who Am I ? Duy Hai DOAN Cassandra technical advocate • talks, meetups, confs • open-source devs (Achilles, …) • OSS Cassandra point of contact ☞ [email protected] ☞ @doanduyhai 2
- 3. @doanduyhai Datastax • Founded in April 2010 • We contribute a lot to Apache Cassandra™ • 400+ customers (25 of the Fortune 100), 400+ employees • Headquarter in San Francisco Bay area • EU headquarter in London, offices in France and Germany • Datastax Enterprise = OSS Cassandra + extra features 3
- 4. What is Apache Zeppelin ? Presentation Architecture
- 7. @doanduyhai Zeppelin Architecture Zeppelin Server Zeppelin Engine 7 R E S TWebSocket Spark Interpreter Group Spark SparkSQL Zeppelin Interpreter Factory Tajo Interpreter Flink Interpreter Cassandra Interpreter JVM JVM JVM JVM JVM
- 8. @doanduyhai What does Zeppelin provide ? Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON) 8
- 9. Zeppelin UI Layout Notebook Paragraph UI elements
- 11. Zeppelin Display System Raw, Table, HTML Available graphs View modes Dynamic form Iframe export
- 13. Interpreter system Core interpreters Third-parties interpreters Interpreters conf & usage
- 14. @doanduyhai Interpreter processing lifecycle ① Receive input commands/data • as raw text • from form data ② Process the input commands/data by the external back-end ③ Format the response using Zeppelin display system ④ Send response back to the Zeppelin engine 14
- 15. @doanduyhai Core interpreters • Spark (Spark core, SparkSQL/DataFrame, PySpark) • Spark core = default (or %spark) • SparkSQL = %sql • Shell (%sh) • Markdown (%md) • AngularJS (%angular) 15
- 16. @doanduyhai Third-parties interpreters • Hive • Phoenix • Tajo • Flink • Ignite • Lens • Cassandra • Geode • PostgreSQL • Kylin 16
- 17. @doanduyhai Interpreter conf & usage https://github.com/doanduyhai/incubator-zeppelin/tree/ZeppelinPresentation
- 18. Writing An Interpreter How To Simple interpreter example (AsciiDoc) Complex interpreter example (Cassandra)
- 19. @doanduyhai Steps to write your own interpreter • Create a class that extends Interpreter base class • Register it in a static block • Optionnally define default config params 19 static { Interpreter.register("MyInterpreterName", MyClassName.class.getName()); } static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder() .add("property1", "default value", "Description of property1").build()); }
- 20. @doanduyhai To register your interpreter as default • Edit the enum ZeppelinConfiguration.ConfVars • Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS 20
- 21. @doanduyhai To register your interpreter in config files • Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template • Add your interpreter FQCN in the property zeppelin.interpreters 21 <property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter, org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter, org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.shell.ShellInterpreter, org.apache.zeppelin.hive.HiveInterpreter,com.me.MyNewInterpreter </value> </property>
- 22. @doanduyhai Simple AsciiDoc Interpreter 22 Zeppelin Server AsciiDoc Interpreter JVMZeppelin Engine Raw Text Block Raw Text Block Converted To HTML HTML Output ① ② ③④ JVM
- 24. @doanduyhai Cassandra Interpreter Architecture 24 Cassandra Interpreter JVM Display Results as HTML ① ② ⑤ Zeppelin Server JVM Raw Text Block Raw Text Block Cassandra Cassandra Java Driver ③ Async CQL statements ④Render HTML ⑥
- 25. @doanduyhai Cassandra Interpreter Commands 25 Native CQL statements SELECT * FROM …; INSERT INTO …; … Schema commands DESCRIBE TABLE …; DESCRIBE KEYSPACE …; … Prepared statements Commands @prepare …; @bind …; @remove_prepared …; Help command HELP; Options Commands @consistency …; @retryPolicy …; @fetchSize …;
- 27. @doanduyhai Cassandra Online Interpreter Docs 27 • http://zeppelin.incubator.apache.org/docs/interpreter/cassandra.html
- 28. @doanduyhai Cassandra Interactive Help 28 • Type HELP; in the interpreter
- 30. @doanduyhai Roadmap & future • More graph options (Map viz ZEPPELIN-157) • Helium project, packaging Zeppelin view, logic (code) & resource into Applications • Interpreters packaging re-design • ship & compile core interpreters only • third-parties interpreters can be pulled from repository • which interpreter is core ? Who will maintain ? Community…. • Integrate security (Apache Shiro, pull request #53 by Hayssam Saleh) 30
- 31. @doanduyhai Roadmap & future • Out of incubation state to become 1st class Apache project 31
- 32. @doanduyhai Q & R ! "