HadoopCon2015 Multi-Cluster Live Synchronization with Kerberos Federated Hadoop
9 likes1,305 views
In enterprise on-premises data center, we may have multiple Secured Hadoop clusters for different purpose. Sometimes, these Hadoop clusters might have different Hadoop distribution, Hadoop version, or even locat in different Data Center. To fulfill business requirement, data synchronize between these clusters could be an important mechanism. However, the story will be more complicated within the real world secured multi-cluster, compare to distcp between two same version and non-secured Hadoop clusters. We would like to go through our experience on enable live data synchronization for mutiple kerberos enabled Hadoop clusters. Which include the functionality verification, multi-cluster configurations and automation setup process, etc. After that, we would share the use cases among those kerberos federated Hadoop clusters. Finally, provide our common practice on multi-cluster data synchronization.
![Oops …
[root@tw-spnhadoop1 hadooppet]# hadoop distcp hdfs://cluster1/test hdfs://krb-1.spn.lab.trendnet.org:8020/test
15/01/22 15:11:44 INFO tools.DistCp: srcPaths=[hdfs://cluster1/test]
15/01/22 15:11:44 INFO tools.DistCp: destPath=hdfs://krb-1.spn.lab.trendnet.org:8020/test
15/01/22 15:11:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 381 for hdfs on ha-hdfs:cluster1
15/01/22 15:11:45 INFO security.TokenCache: Got dt for hdfs://cluster1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:cluster1, Ident:
(HDFS_DELEGATION_TOKEN token 381 for hdfs)
15/01/22 15:11:46 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw-
spnhadoop1.spn.tw.trendnet.org@ISPN.TRENDMICRO.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null):
org.apache.hadoop.ipc.RPC$VersionMismatch
15/01/22 15:11:46 INFO security.UserGroupInformation: Initiating logout for hdfs/tw-spnhadoop1.spn.tw.trendnet.org@ISPN.TRENDMICRO.COM
15/01/22 15:11:46 INFO security.UserGroupInformation: Initiating re-login for hdfs/tw-spnhadoop1.spn.tw.trendnet.org@ISPN.TRENDMICRO.COM
15/01/22 15:11:50 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw-
spnhadoop1.spn.tw.trendnet.org@ISPN.TRENDMICRO.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null):
org.apache.hadoop.ipc.RPC$VersionMismatch
15/01/22 15:11:50 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds
before.
15/01/22 15:11:53 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw-
spnhadoop1.spn.tw.trendnet.org@ISPN.TRENDMICRO.COM (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null):
org.apache.hadoop.ipc.RPC$VersionMismatch](https://image.slidesharecdn.com/hadoopcon2015-multi-clusterlivesynchronizationvpublic-150920065912-lva1-app6892/85/HadoopCon2015-Multi-Cluster-Live-Synchronization-with-Kerberos-Federated-Hadoop-15-320.jpg)
![Set different REAML in each cluster’s KDC
Cluster1 krb5.conf
[realms]
CLUSTER1.DOMAIN.COM = {
kdc = cluster1_kdc_master:88
kdc = cluster1_kdc_slave:88
admin_server = cluster1_kdc_master:749
}
[domain_realm]
cluster1.domain.com = CLUSTER1. DOMAIN.COM
.cluster1.domain.com = CLUSTER1. DOMAIN.COM
Cluster2 krb5.conf
[realms]
CLUSTER2.DOMAIN.COM = {
kdc = cluster2_kdc_master:88
kdc = cluster2_kdc_slave:88
admin_server = cluster2_kdc_master:749
}
[domain_realm]
cluster2.domain.com = CLUSTER2. DOMAIN.COM
.cluster2.domain.com = CLUSTER2. DOMAIN.COM](https://image.slidesharecdn.com/hadoopcon2015-multi-clusterlivesynchronizationvpublic-150920065912-lva1-app6892/85/HadoopCon2015-Multi-Cluster-Live-Synchronization-with-Kerberos-Federated-Hadoop-46-320.jpg)
![Add both cluster’s kerberos information to krb5.conf
Both Cluster1 and Cluster2 krb5.conf
[realms]
CLUSTER1.DOMAIN.COM = {
kdc = cluster1_kdc_master:88
kdc = cluster1_kdc_slave:88
admin_server = cluster1_kdc_master:749
}
CLUSTER2.DOMAIN.COM = {
kdc = cluster2_kdc_master:88
kdc = cluster2_kdc_slave:88
admin_server = cluster2_kdc_master:749
}
[domain_realm]
cluster1.domain.com = CLUSTER1. DOMAIN.COM
.cluster1.domain.com = CLUSTER1. DOMAIN.COM
cluster2.domain.com = CLUSTER2. DOMAIN.COM
.cluster2.domain.com = CLUSTER2. DOMAIN.COM](https://image.slidesharecdn.com/hadoopcon2015-multi-clusterlivesynchronizationvpublic-150920065912-lva1-app6892/85/HadoopCon2015-Multi-Cluster-Live-Synchronization-with-Kerberos-Federated-Hadoop-47-320.jpg)
s/^(.*)@CLUSTER.DOMAIN.COM$/$1/g
RULE:[2:$1@$0](^.*@CLUSTER.DOMAIN.
COM$)s/^(.*)@CLUSTER.DOMAIN.COM$/$1/g
DEFAULT
</value>
</property>
hdfs-site.xml
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
Verify the setting of rule
hadoop org.apache.hadoop.security.HadoopKerberosName mapred/machine.cluster.domain.com@CLUSTER.DOMAIN.COM
Name: mapred/machine.cluster.domain.com@CLUSTER.DOMAIN.COM to mapred](https://image.slidesharecdn.com/hadoopcon2015-multi-clusterlivesynchronizationvpublic-150920065912-lva1-app6892/85/HadoopCon2015-Multi-Cluster-Live-Synchronization-with-Kerberos-Federated-Hadoop-49-320.jpg)
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to HadoopCon2015 Multi-Cluster Live Synchronization with Kerberos Federated Hadoop (20)
Ad
Recently uploaded (20)
HadoopCon2015 Multi-Cluster Live Synchronization with Kerberos Federated Hadoop
- 1. Multi-Cluster Live Synchronization with Kerberos Federated Hadoop 張雅芳 Mammi Chang @ 2015 Taiwan HadoopCon
- 2. Who am I ? • Mammi Chang 張雅芳 • Sr. Engineer, SPN, Trend Micro • SPN Hadoop Cluster Administrator • DevOps on Hadoop ecosystem and AWS • 2014 HadoopCon Speaker
- 4. Original Data Center New Data Center TMH7TMH6 service Data SyncData Sync This is a story of move …
- 5. Production TMH6 TMH7 TMH6 TMH7 Original Data Center New Data Center Production Staging Staging Data Sync Data Sync Data Sync
- 6. Data SynchronizationData synchronization is the process of establishing consistency among data from a source to a target data storage and vice versa and the continuous harmonization of the data over time. - From wikipedia “Data synchronization”
- 7. One-way file synchronization Updated files copied from source to destination Two-way file synchronization Updated files are copied in both directories Dropbox, SafeSync, etc
- 8. Linux One-Way File Synchronization $ cp fileA fileB $ scp ./directory/my_file [email protected]:/home/mammi/ $ rsync -avP /source/data /destination/
- 9. Hadoop One-Way File Synchronization $ hadoop fs -cp /user/mammi/file1 /user/mammi/dir/ $ hadoop distcp hdfs://cluster1/file hdfs://cluster2/file
- 11. DistCp with the same Hadoop version is trivial.
- 12. Hadoop - 2.6 Cluster1 Hadoop – 2.6 Cluster2 $ hadoop distcp hdfs://cluster1_nn:8020/test hdfs://cluster2_nn:8020/test
- 13. Hadoop - 2.6 Cluster1 Hadoop – 2.6 Cluster2 $ hadoop distcp hdfs://cluster2_nn:8020/test hdfs://cluster1_nn:8020/test
- 14. DistCp with the same Hadoop version is trivial. different a little bit tricky
- 15. Oops … [root@tw-spnhadoop1 hadooppet]# hadoop distcp hdfs://cluster1/test hdfs://krb-1.spn.lab.trendnet.org:8020/test 15/01/22 15:11:44 INFO tools.DistCp: srcPaths=[hdfs://cluster1/test] 15/01/22 15:11:44 INFO tools.DistCp: destPath=hdfs://krb-1.spn.lab.trendnet.org:8020/test 15/01/22 15:11:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 381 for hdfs on ha-hdfs:cluster1 15/01/22 15:11:45 INFO security.TokenCache: Got dt for hdfs://cluster1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:cluster1, Ident: (HDFS_DELEGATION_TOKEN token 381 for hdfs) 15/01/22 15:11:46 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw- [email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null): org.apache.hadoop.ipc.RPC$VersionMismatch 15/01/22 15:11:46 INFO security.UserGroupInformation: Initiating logout for hdfs/[email protected] 15/01/22 15:11:46 INFO security.UserGroupInformation: Initiating re-login for hdfs/[email protected] 15/01/22 15:11:50 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw- [email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null): org.apache.hadoop.ipc.RPC$VersionMismatch 15/01/22 15:11:50 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before. 15/01/22 15:11:53 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs/tw- [email protected] (auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(null): org.apache.hadoop.ipc.RPC$VersionMismatch
- 16. Apache Hadoop – 2.0 Cluster1 Apache Hadoop – 2.6 Cluster2 $ hadoop distcp hftp://cluster1_nn:50070/test hdfs://cluster2_nn:8020/test HftpFileSystem is a read-only FileSystem, so DistCp must be run on the destination cluster
- 17. TMH6 Cluster1 TMH7 Cluster2 $ hadoop distcp ????://TMH6_NN:????/test hdfs://TMH7_NN:8020/test CDH Based Apache Based
- 18. TMH6 Cluster1 TMH7 Cluster2 $ hadoop distcp hftp://TMH6_NN:50070/test hdfs://TMH7_NN:8020/test CDH Based Apache Based Only support data sync from TMH6 to TMH7
- 19. DistCp with different Hadoop version is a little bit tricky plus kerberos security annoying !!
- 20. TMH6 Cluster1 TMH7 Cluster2 $ hadoop distcp ????://TMH6_NN:XXXX/test ????://TMH7_NN:XXXX/test
- 21. DistCp Data Copy Matrix: HDP1/HDP2 to HDP2 http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_system-admin- guide/content/distcp-table.html Webhdfs is a HTTP REST API supports the complete FileSystem interface for HDFS
- 22. DistCp Data Copy Matrix: TMH6/TMH7 to TMH6/TMH7 TMH6 TMH7 insecure secure hdfs hftp webhdfs 2
- 23. TMH6 Cluster1 TMH7 Cluster2 $ hadoop distcp webhdfs://TMH6_NN:8020/test webhdfs://TMH7_NN:8020/test
- 24. Hadoop Security with Kerberos Kerberos is a computer network authentication protocol which works on the basis of 'tickets' to allow nodes communicating over a non- secure network to prove their identity to one another in a secure manner - From wikipedia “Kerberos_(Protocol)”
- 25. REALM – CLUSTER.DOMAIN.COM Kerberos Negotiation KDC (Key Distributed Center) TGT (Ticket-Granting Ticket) KDC Client Hadoop Servers Msg3 : Authenticator, TGT Msg4 : client/server ticket Msg1 : client login KDC Msg2 : client TGT Msg5 : Authenticator, ticket Msg6 : time auth
- 26. REALM – CLUSTER2.DOMAIN.COM Kerberos Cross-Realm authenticate REALM – CLUSTER1.DOMAIN.COM KDC Client Hadoop Servers Msg3 : Authenticator, TGT Msg4 : client/server ticket Msg1 : client login KDC Msg2 : client TGT Msg5 : Authenticator, ticket Msg6 : time auth KDC
- 27. Kerberos Federation for Hadoop Kerberos Setting • Set different REALM in each cluster’s KDC • Add both cluster’s kerberos information to configs • Add federated kerberos principal to both KDC DB • Restart kerberos services Hadoop Setting • Add Hadoop configurations • Make sure both cluster nodes can recognize each other • Restart necessary Hadoop services
- 28. Multi-Cluster Kerberos Federation Cluster1 •Set different REALM in each cluster’s KDC •Add all other cluster’s kerberos information to configuration •Add all federated kerberos principal to KDC DB •Add Hadoop configurations •Make sure all cluster nodes can recognize each others •Restart necessary services Cluster2 •Set different REALM in each cluster’s KDC •Add all other cluster’s kerberos information to configuration •Add all federated kerberos principal to KDC DB •Add Hadoop configurations •Make sure all cluster nodes can recognize each others •Restart necessary services … •… Cluster N •Set different REALM in each cluster’s KDC •Add all other cluster’s kerberos information to configuration •Add all federated kerberos principal to KDC DB •Add Hadoop configurations •Make sure all cluster nodes can recognize each others •Restart necessary services
- 29. DistCp with different Hadoop version plus kerberos federation is annoying !! in cross DC multi-cluster not easy. Done!!
- 30. DistCp with different Hadoop version plus kerberos federation in cross DC mult-clusters is not easy at all.
- 31. Production TMH6 TMH7 TMH6 TMH7 Original Data Center New Data Center Production Staging Staging Two-way keberos federation link Data Sync Data Sync Data Sync
- 33. Issues • Computing resource • Zero-downtime • Schedule limitation • Network bandwidth
- 34. Computing Resource • Principle – Do not have production service impact when many DistCp jobs running • Strategy – Run distcp on Staging Env. Instand of Production Env.
- 35. Production TMH6 TMH7 TMH6 TMH7 Original Data Center New Data Center Production Staging Staging Two-way keberos federation link $ hadoop distcp webhdfs://TMH6_PROD_NN:8020/test webhdfs://TMH7_PROD_NN:8020/test Data Sync Data flow
- 36. Production TMH6 TMH7 TMH6 TMH7 Original Data Center New Data Center Production Staging Staging Two-way keberos federation link $ hadoop distcp webhdfs://TMH6_PROD_NN:8020/test webhdfs://TMH7_PROD_NN:8020/test Data Sync Data flow
- 37. Zero-downtime • Principle – Do not have Production Env. downtime • Strategy – Change KDC REALM in Staging only – Rolling restart services
- 38. Schedule Limitation • Principle – Provide minimum dataset that fulfill production services requirement • Strategy – Divide dataset into cold data and hot data – All necessary hot data need to be ready before service move to new DC
- 40. Automation is vital !!! • Automated CI tests on such complex and repeated tasks – save your life time – prevent plenty of human errors
- 41. Customization is necessary • Home made distcp running script with error handling • Setting permission by real case
- 42. Just try it • Survey is important but sometimes it cannot totally solve your problem
- 45. Kerberos Cross-Realm Federation • Set different REALM in each cluster’s KDC • Add both cluster’s kerberos information to configs • Add federated kerberos principal to both KDC DB • Add Hadoop configurations • Make sure both cluster nodes can recognize each other • Restart necessary services
- 46. Set different REAML in each cluster’s KDC Cluster1 krb5.conf [realms] CLUSTER1.DOMAIN.COM = { kdc = cluster1_kdc_master:88 kdc = cluster1_kdc_slave:88 admin_server = cluster1_kdc_master:749 } [domain_realm] cluster1.domain.com = CLUSTER1. DOMAIN.COM .cluster1.domain.com = CLUSTER1. DOMAIN.COM Cluster2 krb5.conf [realms] CLUSTER2.DOMAIN.COM = { kdc = cluster2_kdc_master:88 kdc = cluster2_kdc_slave:88 admin_server = cluster2_kdc_master:749 } [domain_realm] cluster2.domain.com = CLUSTER2. DOMAIN.COM .cluster2.domain.com = CLUSTER2. DOMAIN.COM
- 47. Add both cluster’s kerberos information to krb5.conf Both Cluster1 and Cluster2 krb5.conf [realms] CLUSTER1.DOMAIN.COM = { kdc = cluster1_kdc_master:88 kdc = cluster1_kdc_slave:88 admin_server = cluster1_kdc_master:749 } CLUSTER2.DOMAIN.COM = { kdc = cluster2_kdc_master:88 kdc = cluster2_kdc_slave:88 admin_server = cluster2_kdc_master:749 } [domain_realm] cluster1.domain.com = CLUSTER1. DOMAIN.COM .cluster1.domain.com = CLUSTER1. DOMAIN.COM cluster2.domain.com = CLUSTER2. DOMAIN.COM .cluster2.domain.com = CLUSTER2. DOMAIN.COM
- 48. Add federated kerberos principal to both KDC DB $ kadmin.local: addprinc -e "rc4-hmac:normal des3-hmac-sha1:normal" krbtgt/ [email protected] WARNING: no policy specified for krbtgt/CLUSTER1.DOMAIN.COM@ CLUSTER2.DOMAIN.COM; defaulting to no policy Enter password for principal "krbtgt/ [email protected] ": // 123456 Re-enter password for principal "krbtgt/[email protected]": // 123456 Principal "krbtgt/[email protected]" created. $ kadmin.local: addprinc -e "rc4-hmac:normal des3-hmac-sha1:normal" krbtgt/[email protected] WARNING: no policy specified for krbtgt/[email protected]; defaulting to no policy Enter password for principal "krbtgt/CLUSTER2.DOMAIN.COM @CLUSTER1.DOMAIN.COM ": // 654321 Re-enter password for principal "krbtgt/[email protected] ": // 654321 Principal "krbtgt/[email protected] " created. use the same password for a principal to make sure the encryption key is the same
- 49. Add Hadoop Configuration core-site.xml <property> <name>hadoop.security.auth_to_local</name> <value> RULE:[1:$1@$0](^.*@CLUSTER.DOMAIN. COM$)s/^(.*)@CLUSTER.DOMAIN.COM$/$1/g RULE:[2:$1@$0](^.*@CLUSTER.DOMAIN. COM$)s/^(.*)@CLUSTER.DOMAIN.COM$/$1/g DEFAULT </value> </property> hdfs-site.xml <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> Verify the setting of rule hadoop org.apache.hadoop.security.HadoopKerberosName mapred/[email protected] Name: mapred/[email protected] to mapred
- 50. Make sure both cluster nodes can recognize each other • /etc/hosts for both cluster1 and cluster2 nodes 10.1.145.1 machine1.cluster1.domain.com 10.1.145.2 machine2.cluster1.domain.com 10.1.145.3 machine3.cluster1.domain.com 10.1.144.1 machine1.cluster2.domain.com 10.1.144.2 machine2.cluster2.domain.com 10.1.144.3 machine3.cluster2.domain.com
- 51. Restart necessary services • KDC server – service krb5kdc restart – service kadmin restart • Namenodes, Datanodes – service hadoop-hdfs-namenode restart – servcie hadoop-hdfs-datanode restart