HUG France Feb 2016 - Migration de données structurées entre Hadoop et RDBMS par Louis Rabiet (Squid Solution)
0 likes•1,769 views
The document discusses a method for migrating structured data between Hadoop and RDBMS using an analytics toolbox called Bouquet, which generates SQL via a REST API. It highlights the integration of various technologies including Spark, Kafka, and Avro schema for efficient data management and enrichment. Challenges in implementation, as well as future improvements for scalability and functionality, are also outlined.
![How to keep the data structured?
Use a schema registry (Avro in Kafka).
each schema has a corresponding kafka topic and a distinct hive table.
{
"type": "record",
"name": "ArtistGender",
"fields" : [
{"name": "count", "type": "long"},
{"name": "gender", "type": "String"]}
]
}](https://image.slidesharecdn.com/squidhugfebrary2016key-160219111806/85/HUG-France-Feb-2016-Migration-de-donnees-structurees-entre-Hadoop-et-RDBMS-par-Louis-Rabiet-Squid-Solution-13-320.jpg)
HUG France Feb 2016 - Migration de données structurées entre Hadoop et RDBMS par Louis Rabiet (Squid Solution)
- 1. February 16th 2016 [email protected] Migrating structured data between Hadoop and RDBMS
- 2. Who am I? • Full Stack engineer at Squid Solutions. • Specialised in Big data. • Fun fact: sleeping by myself in my tent on the top of the highest mountains of the world
- 3. What I do ? • Develop of an analytics toolbox. • No setup. No SQL. No compromise. • Generate SQL with a REST API. It is open source! https://github.com/openbouquet
- 4. Topic of today • You need Scalability? • You need a machine learning toolbox? Hadoop is the solution. •But you still need structured data? Our tool provide a solution. => We need both!
- 5. What does that mean? • Creation of dataset in Bouquet • Send the dataset to Spark • Enrich inside Spark • Re-injection in original database
- 6. How we do it? User input Relational DB SparkBouquet
- 8. How does it work? BouquetRelational DB Spark HDFS/ Tachyon Hive Metastore User select the data. Bouquet generate the corresponding SQL Code Kafka
- 9. How does it work? BouquetRelational DB Spark HDFS/ Tachyon Hive Metastore Data is read from the SQL database Kafka
- 10. How does it work? BouquetRelational DB Spark HDFS/ Tachyon Hive Metastore The BI tool creates an avro schema and send the data to Kafka Kafka
- 11. How does it work? BouquetRelational DB Spark Kafka HDFS/ Tachyon Hive Metastore Kafka Broker(s) receive the data
- 12. How does it work? BouquetRelational DB Spark HDFS/ Tachyon Hive Metastore Kafka The hive metastore is updated and the hdfs connectors writes into hdfs
- 13. How to keep the data structured? Use a schema registry (Avro in Kafka). each schema has a corresponding kafka topic and a distinct hive table. { "type": "record", "name": "ArtistGender", "fields" : [ {"name": "count", "type": "long"}, {"name": "gender", "type": "String"]} ] }
- 14. Challenges - Auto creation of topics/table in Hive for each datasets from Bouquet. - JDBC reads are too slow for something like Kafka. - Issue with types conversion: null is not supported for all cases for example (issue 272 on schema-registry). - Versions: Kafka 0.9.0, Tachyon 0.7.1, Spark 1.5.2 with HortonWorks 2.3.4 (Dec 2015) - Hive: Setting the warehouse directory. - In tachyon: Setting up hostname.
- 15. Tachyon? • Use it as in memory filesystem to replace HDFS. • Interact with Spark using the hdfs plugin. • Transparent from user point of view
- 16. Status Injection SQL -> Spark: OK Spark usage: OK Re-injection: In alpha stage.
- 17. Re-injection Two solutions: • Spark user notifies Bouquet that data has changed (using a custom function) • Bouquet pulls the data from spark
- 18. We use it for real! Collaborating with La Poste to be able to use Spark and the re-injection mechanism to use Bouquet and a geographical visualisation.
- 19. In the future • Notebook integration • We got a DSL for bouquet API, we may want to have built-in support spark. • Improve scalability (Bulk Unload and Kafka fine tuning)
- 21. DB HD Bouquet Architecture Bouquet Server SQL DATA JDBC Dynamic Caching & Indexing REST APIBusiness Modeling OAuth2 Generic Apps Multi-Tenant REDIS Elastic MongoDB JS/SDK Custom Apps