Building a Hadoop Cluster with Stacki
Download as PPTX, PDF1 like271 views
Stacki is a Linux server provisioning tool designed for rapid deployment to bare metal or virtual machines with no prerequisites. It facilitates the installation of the Hortonworks Data Platform (HDP) through a simple process of adding and configuring software pallets and managing cluster nodes. Additionally, it supports various functionalities, including partitioning, disk formatting, and Ambari appliance creation for efficient Hadoop deployment.
Building a Hadoop Cluster with Stacki
- 1. Stacki Lab Joe Kaiser Director of Open Source Engineering
- 2. Open Source Stack Installer Stacki is a very fast and ultra reliable Linux server provisioning tool … at scale. With zero prerequisites for taking systems from bare metal to a ping and prompt.
- 3. Stuff it does Installs to bare metal or VMs that look like bare metal Kickstart based Parallel formatting of disk Parallel Sharing of RPMs CentOS/RHEL Networking Multiple subnets, vlaning, bonding. Storage Controller config Partitioning Whatever you can do in Linux, you can do with Stacki only clustered
- 4. Stacki and Hortonworks Data Platform
- 5. Hortonworks – stacki-hdp-bridge pallet Add/enable/run Stacki creates ambari appliance Download software as pallets (isos) Add/enable HDP and Ambari Run gethdp script Or download (links in documentation) Assign a node to be Ambari deployment. Do partitions and preps backend nodes for Hadoop. Install all. Go to Ambari interface to deploy Hadoop. Current versions: (updated /export/HDP/hdp.cfg for new versions) distribution = 2.x os = centos7 ambari = 2.4.2.0 hdp = 2.5.3.0
- 6. Augment the “default” Box 1) Replace the “os” pallet with the “CentOS” and “CentOS- Updates” pallets 2) Add the Hortonworks pallets: “HDP”, “HDP-UTILS”, “Updates-ambari” or run /export/HDP/gethdp.py 3) Add a pallet to glue the two layers together: “stacki- hdp-bridge”
- 7. Specify One Host as the “Ambari” appliance “backend-0-0” will be automatically configured with Ambari Install the node and then point your web browser at it
- 8. Reinstall All Backend Nodes Wipe all hardware disk array configuration and rebuild all the LUNs: ◦ stack set host attr ambari backend attr=nukecontroller value=true Remove all partitions then repartition and reformat the disks: ◦ stack set host attr ambari backend attr=nukedisks value=true Instruct nodes to install on next PXE boot: ◦ stack set host boot ambari backend action=install
- 9. Try It Website www.stacki.com Vagrant tire kick https://github.com/rfkrocktk/vagrant-stacki Source Code and docs github.com/stackiq/stacki github.com/StackIQ/stacki-hdp-bridge Slack Channel (because everyone) Google Groups groups.google.com/forum/#!forum/stacki
- 10. Finis Thanks
Editor's Notes
- #3: Linux – Focused on RedHat-ish (Kickstart/Anaconda) Provisioning – Bare Metal (total stack control) Scale – solve 1000+ servers problem then scale down Ping and Prompt – Get machine up to known base OS fully configuration raid / disk / networking / ssh access on Nothing else … No agent left on the server