Hand-on Resources II: Extending SCMSWeb
Download as odp, pdf1 like2,411 views
The document describes the configuration of SCMSWeb, an open source monitoring system for cluster and grid resource usage. It includes details on probe configuration, catalog configuration, GeoMap configuration, and provides examples of using Python to parse and traverse the SCMSWeb XML output.
![Extending SCMSWeb 3 rd PRAGMA Institute Hand-on Resources II Sugree Phatanapherom [email_address]](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-1-320.jpg)
![probe.conf [html] xmlpath = /usr/share/scmsweb/incoming/cluster/pragma.xml textpath = /usr/share/scmsweb/grid/PRAGMA/pragma.txt htmlpath = /usr/share/scmsweb/grid/PRAGMA/pragma.html [probe] name = PRAGMA smtp = localhost #email = cindy.pragma@gmail.com timestamp_path = /var/tmp/scmsweb timeout = 40 probe0 = DNS probe1 = Authentication probe2 = Mds probe3 = JobRun probe4 = GridFTPTo probe5 = GridFTPFrom probe6 = WSGRAM probe7 = Iperf probe8 = SWCat [globus] basedn = mds-vo-name=npaci,o=sdsc mdshost = rocks-67.sdsc.edu [Gfarm] command = /usr/share/scmsweb/gfarm/probe_gfhost %(target)s [WSGRAM] command = /usr/share/scmsweb/wsgram/probe_wsgram %(target)s [time] rerun_false = 1 SWCat = 720 Iperf = 720 Mds = 720 GridFTPTo = 360 GridFTPFrom = 360 JobRun = 120 WSGRAM = 120](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-13-320.jpg)
![target.conf #[USM-CS] [aurora] target = aurora.usmgrid.myren.net.my iperf_args = -p 15001 email = zhengc@sdsc.edu,keat@webmail.cs.usm.my,khuen@cs.usm.my,hychan@cs.usm.my #[IOIT-HCM] [VENUS] target = venus.ioit-hcm.ac.vn iperf_args = -p 15001 email = zhengc@sdsc.edu,han@vast-hcm.ac.vn #[MIMOS] [MIMOS-Nucleus] target = nucleus.mygridusbio.net.my iperf_args = -p 5001 email = [email_address] ,jyluke@mimos.my #[SDSC_compute] [Rocks-52] target = rocks-52.sdsc.edu iperf_args = -p 15001 email = zhengc@sdsc.edu SWCat_target = rocks-52.sdsc.edu/jobmanager-fork #[BU] [popCluster] target = pop.cs.binghamton.edu GridFTPTo_probe = off GridFTPFrom_probe = off iperf_args = -p 15001 email = zhengc@sdsc.edu,john@jskro.org,kchiu@cs.binghamton.edu](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-14-320.jpg)
![swcatalog.conf [autodock4] url=http://autodock.scripps.edu/ [namd] url=http://www.ks.uiuc.edu/Research/namd/ [amber] url=http://amber.scripps.edu/ [ninf-g] url=http://ninf.apgrid.org/ [intel_c_compiler] url=http://www.intel.com/cd/software/products/asmo-na/eng/219768.htm [intel_fortran_compiler] url=http://www.intel.com/cd/software/products/asmo-na/eng/219768.htm [apbs] url=http://apbs.sourceforge.net/](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-16-320.jpg)
![0-init.sh #!/bin/sh for file in /etc/profile.d/*.sh; do [ -f "$file" ] && . "$file" done](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-17-320.jpg)
![amber.sh #!/bin/sh amber_version() { # bad guess? VERSION=`echo "$1" | grep -q 9` if [ X"$VERSION" = X ]; then VERSION="9" else VERSION="unknown" fi echo -n "$VERSION" } SANDER_PATH=`which sander 2> /dev/null` if [ X"$AMBERHOME" != X ]; then # how can we decide version of amber? VERSION=`amber_version "$AMBERHOME"/exe` echo "amber:$VERSION:$AMBERHOME/exe" elif [ X"$SANDER_PATH" != X ]; then VERSION=`amber_version "$SANDER_PATH"` echo "amber:$VERSION:$SANDER_PATH" fi](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-18-320.jpg)
![ninf-g.sh #!/bin/sh NG_VERSION=`ng_version 2> /dev/null` if [ X"$NG_VERSION" != X ]; then VERSION=`echo $NG_VERSION | sed "s/.*:[[:space:]]*\(.*\)/\1/g"` NG_PATH=`which ng_version` NG_PATH=`dirname "$NG_PATH"` echo "ninf-g:$VERSION:$NG_PATH" fi](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-19-320.jpg)
![Configuration [Site:Universiti Sains Malaysia] parent = Grid:PRAGMA UniqueID = USM_MY Latitude = 5.356667 Longitude = 100.300833 Location = Malaysia Description = USM MY SecurityContact = mailto: noreply@noreply.com SysAdminContact = mailto: noreply@noreply.com UserSupportContact = mailto: noreply@noreply.com [Site:MIMOS] parent = Grid:PRAGMA Latitude = 5.356667 Longitude = 100.300833 Location = Malaysia Description = MIMOS MY SecurityContact = mailto: noreply@noreply.com SysAdminContact = mailto: noreply@noreply.com UserSupportContact = mailto: noreply@noreply.com [Grid:PRAGMA] Name = PRAGMA Description = PRAGMA [Site:TNGC_TH] parent = Grid:PRAGMA Location = Bangkok, Thailand Latitude = 13.84544 Longitude = 100.567802 Description = sunyata.thaigrid.or.th SecurityContact = mailto: noreply@noreply.com SysAdminContact = mailto: noreply@noreply.com UserSupportContact = mailto: noreply@noreply.com](https://image.slidesharecdn.com/pragma15scmswebhandon-1224632586711879-8/85/Hand-on-Resources-II-Extending-SCMSWeb-21-320.jpg)
More Related Content
Similar to Hand-on Resources II: Extending SCMSWeb (20)
More from Sugree Phatanapherom (16)
Recently uploaded (20)
Hand-on Resources II: Extending SCMSWeb
- 1. Extending SCMSWeb 3 rd PRAGMA Institute Hand-on Resources II Sugree Phatanapherom [email_address]
- 2. SCMSWeb in Short Monitoring system for Cluster and Grid Usage statistics Service probes
- 5. Probe
- 6. Bandwidth
- 7. GeoMap
- 8. Catalog
- 9. Behind graphs, charts and tables Configuration APIs
- 10. Configurable Probe Catalog GeoMap
- 11. General Configuration /etc/sce/sce.conf /usr/share/scmsweb/
- 12. Probe Configuration /usr/share/scmsweb/grid/<name> probe.conf target.conf
- 13. probe.conf [html] xmlpath = /usr/share/scmsweb/incoming/cluster/pragma.xml textpath = /usr/share/scmsweb/grid/PRAGMA/pragma.txt htmlpath = /usr/share/scmsweb/grid/PRAGMA/pragma.html [probe] name = PRAGMA smtp = localhost #email = [email protected] timestamp_path = /var/tmp/scmsweb timeout = 40 probe0 = DNS probe1 = Authentication probe2 = Mds probe3 = JobRun probe4 = GridFTPTo probe5 = GridFTPFrom probe6 = WSGRAM probe7 = Iperf probe8 = SWCat [globus] basedn = mds-vo-name=npaci,o=sdsc mdshost = rocks-67.sdsc.edu [Gfarm] command = /usr/share/scmsweb/gfarm/probe_gfhost %(target)s [WSGRAM] command = /usr/share/scmsweb/wsgram/probe_wsgram %(target)s [time] rerun_false = 1 SWCat = 720 Iperf = 720 Mds = 720 GridFTPTo = 360 GridFTPFrom = 360 JobRun = 120 WSGRAM = 120
- 14. target.conf #[USM-CS] [aurora] target = aurora.usmgrid.myren.net.my iperf_args = -p 15001 email = [email protected],[email protected],[email protected],[email protected] #[IOIT-HCM] [VENUS] target = venus.ioit-hcm.ac.vn iperf_args = -p 15001 email = [email protected],[email protected] #[MIMOS] [MIMOS-Nucleus] target = nucleus.mygridusbio.net.my iperf_args = -p 5001 email = [email_address] ,[email protected] #[SDSC_compute] [Rocks-52] target = rocks-52.sdsc.edu iperf_args = -p 15001 email = [email protected] SWCat_target = rocks-52.sdsc.edu/jobmanager-fork #[BU] [popCluster] target = pop.cs.binghamton.edu GridFTPTo_probe = off GridFTPFrom_probe = off iperf_args = -p 15001 email = [email protected],[email protected],[email protected]
- 15. Catalog Configuration /usr/share/scmsweb/swcatalog.conf /usr/share/scmsweb/swcatalog/
- 16. swcatalog.conf [autodock4] url=http://autodock.scripps.edu/ [namd] url=http://www.ks.uiuc.edu/Research/namd/ [amber] url=http://amber.scripps.edu/ [ninf-g] url=http://ninf.apgrid.org/ [intel_c_compiler] url=http://www.intel.com/cd/software/products/asmo-na/eng/219768.htm [intel_fortran_compiler] url=http://www.intel.com/cd/software/products/asmo-na/eng/219768.htm [apbs] url=http://apbs.sourceforge.net/
- 17. 0-init.sh #!/bin/sh for file in /etc/profile.d/*.sh; do [ -f "$file" ] && . "$file" done
- 18. amber.sh #!/bin/sh amber_version() { # bad guess? VERSION=`echo "$1" | grep -q 9` if [ X"$VERSION" = X ]; then VERSION="9" else VERSION="unknown" fi echo -n "$VERSION" } SANDER_PATH=`which sander 2> /dev/null` if [ X"$AMBERHOME" != X ]; then # how can we decide version of amber? VERSION=`amber_version "$AMBERHOME"/exe` echo "amber:$VERSION:$AMBERHOME/exe" elif [ X"$SANDER_PATH" != X ]; then VERSION=`amber_version "$SANDER_PATH"` echo "amber:$VERSION:$SANDER_PATH" fi
- 19. ninf-g.sh #!/bin/sh NG_VERSION=`ng_version 2> /dev/null` if [ X"$NG_VERSION" != X ]; then VERSION=`echo $NG_VERSION | sed "s/.*:[[:space:]]*\(.*\)/\1/g"` NG_PATH=`which ng_version` NG_PATH=`dirname "$NG_PATH"` echo "ninf-g:$VERSION:$NG_PATH" fi
- 21. Configuration [Site:Universiti Sains Malaysia] parent = Grid:PRAGMA UniqueID = USM_MY Latitude = 5.356667 Longitude = 100.300833 Location = Malaysia Description = USM MY SecurityContact = mailto: [email protected] SysAdminContact = mailto: [email protected] UserSupportContact = mailto: [email protected] [Site:MIMOS] parent = Grid:PRAGMA Latitude = 5.356667 Longitude = 100.300833 Location = Malaysia Description = MIMOS MY SecurityContact = mailto: [email protected] SysAdminContact = mailto: [email protected] UserSupportContact = mailto: [email protected] [Grid:PRAGMA] Name = PRAGMA Description = PRAGMA [Site:TNGC_TH] parent = Grid:PRAGMA Location = Bangkok, Thailand Latitude = 13.84544 Longitude = 100.567802 Description = sunyata.thaigrid.or.th SecurityContact = mailto: [email protected] SysAdminContact = mailto: [email protected] UserSupportContact = mailto: [email protected]
- 22. Extensible Probe Catalog GeoMap http://goc.pragma-grid.net/cgi-bin/scmsweb/xml_display.cgi?grid=on http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=xml http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=kml
- 23. Basic Usage curl http://goc.pragma-grid.net/cgi-bin/scmsweb/xml_display.cgi?grid=on curl http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=xml curl http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=kml
- 24. For developers Your codes, your rules!
- 25. Try it and feel it! Host: ppkt-cluster.usm.my User: guest<1-20> Password: guest<1-20>
- 26. conf.py scmsweb_xml = 'http://goc.pragma-grid.net/cgi-bin/scmsweb/xml_display.cgi?grid=on' map_xml = 'http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=xml' map_kml = 'http://goc.pragma-grid.net/cgi-bin/scmsweb/glue.cgi?format=kml' nano conf.py cat conf.py python conf.py
- 27. fetch.py #!/usr/bin/python import conf from urllib import urlopen fi = urlopen(conf.scmsweb_xml) fo = open('scmsweb.xml', 'w') fo.write(fi.read()) fo.close() fi = urlopen(conf.map_xml) fo = open('map.xml', 'w') fo.write(fi.read()) fo.close() fi = urlopen(conf.map_kml) fo = open('map.kml', 'w') fo.write(fi.read()) fo.close() nano fetch.py python fetch.py ls -l
- 28. parse.py #!/usr/bin/python from xml.dom import minidom fi = open('scmsweb.xml') doc = minidom.parse(fi) print doc nano parse.py python parse.py
- 29. traverse.py #!/usr/bin/python from xml.dom import minidom fi = open('scmsweb.xml') doc = minidom.parse(fi) root = doc.documentElement for node in root.childNodes: if node.nodeType == doc.ELEMENT_NODE: print node.tagName, node.attributes.items()
- 30. x.py #!/usr/bin/python from xml.dom import minidom from xml import xpath fi = open('scmsweb.xml') doc = minidom.parse(fi) root = doc.documentElement nodes = xpath.Evaluate('cluster', root) for node in nodes: if node.nodeType == doc.ELEMENT_NODE: print node.tagName, node.attributes.items()
- 31. l.py #!/usr/bin/python from xml.dom import minidom from lxml import etree as ET fi = open('scmsweb.xml') root = ET.parse(fi) nodes = root.xpath('cluster') for node in nodes: print node.tag, node.attrib
- 32. Congratulation! You are python programmer now