15 map reduce on azure
- 1. MapReduce on Azure UN EXEMPLE CONCRET
- 2. Se connecter sur notre instance ssh [email protected] git clone https://gitlab.com/pbury/m2isf_hadoop.git cd m2isf_hadoop sudo docker-compose –f docker-compose-local.yml build sudo docker-compose –f docker-compose-local.yml up sudo docker ps sudo docker exec –it resourcemanager bash Ca y est, on est dans le container !! exit 2
- 3. Sans Hadoop cat /usr/local/src/ascii_5000.txt | /usr/local/src/mapper.py | sort -k1,1 | /usr/local/src/reducer.py 3
- 4. Hadoop : on prépare le travail On copie la donnée hdfs dfs -mkdir /gutemberg hdfs dfs -put /usr/local/src/ascii_5000.txt /gutemberg On rentre dans le docker docker exec –it resourcemanager bash 4
- 5. On lance Map Reduce hadoop jar /opt/hadoop-2.7.1/share/hadoop/tools/lib/hadoop-streaming- 2.7.1.jar -file /usr/local/src/mapper.py -mapper /usr/local/src/mapper.py -file /usr/local/src/reducer.py -reducer /usr/local/src/reducer.py -input /gutemberg -output /gutemberg-output5 5
- 6. Résultats hdfs dfs -ls /gutemberg-output5/ Found 2 items -rw-r--r-- 3 root supergroup 0 2019-01-29 07:24 /gutemberg-output5/_SUCCESS -rw-r--r-- 3 root supergroup 340155 2019-01-29 07:24 /gutemberg-output5/part-00000 6
- 7. Résultats hdfs dfs –cat gutemberg-output5/part-00000 7
- 8. Références https://gitlab.com/pbury/m2isf_hadoop Créer une clef ssh pour azure : https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/virtual- machines/linux/ssh-from-windows.md Powershell : ssh-keygen 8