Design of bare metal proxy compute node
3 likes4,460 views
This document proposes design changes to support bare metal proxy compute nodes in OpenStack Nova. It discusses adding a --baremetal_driver option to specify the underlying architecture or provisioning tool. A new ProxyConnection class would be added to handle bare metal domains. Key methods like create_domain(), activate_node(), get_console_output(), and update_available_resource() need to interface with bare metal specific functions, for example to get idle nodes, set images, boot systems, and retrieve hardware information. The document suggests separating proxy_baremetal.py into multiple files to encapsulate domain management and architecture specific logic.
![TODO [1] nova.conf
+ 2
Current Version Next Version
Proxy Compute Node Proxy Compute Node
LibvirtConnection LibvirtConnection
(libvirt_conn.py) proxy_baremetal (libvirt_conn.py)
--connection_type=libvirt specific model --connection_type=libvirt
--connection_type=gpu
--connection_type=gpu
ProxyConnection
(proxy_baremetal.py)
1 --connection_type=baremetal
--connection_type=tilera gPXE.py?
TileraConnection TileraConnection ….. ArmConnection
(tilera.py) (tilera.py) (arm.py)
--baremetal_driver=arm
HeckleConnection
2 --baremetal_driver=tilera
(heckle.py)
--baremetal_driver=heckle
Specific arch. or
1. Changing --connection_type: tilera baremetal Specific provisioning tool
2. Adding –baremetal_driver option in nova.conf: tilera/arm/heckle/…](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-2-320.jpg)
![TODO [2] proxy_baremetal.py overview
+ 3
Next Version
Proxy Compute Node
--connection_type=baremetal
def create_domain(..):
ProxyConnection …
(proxy_baremetal.py) load bare_metal as –baremetal_driver in nova.conf
baremetal_id = bare_metal.get_idle_node()
new_dom = {
class ProxyConnection ’baremetal_type': bare_metal,
def _get_connection(..): ’baremetal_id': baremetal_id,
// _conn <- baremetal_doms 'name': xml_dict['name'],
def destroy(..): 'mac_address': xml_dict['mac_address'],
// _conn.destroy_domain(..) 'ip_address': xml_dict['ip_address'],
// db.instance_set_state(..) 'dhcp_server': xml_dict['dhcp_server'],
'image_id': xml_dict['image_id'],
def reboot(..):
'kernel_id': xml_dict['kernel_id'],
// _conn.reboot_domain(..) 'ramdisk_id': xml_dict['ramdisk_id'],
// db.instance_set_state(..) 'status': power_state.NOSTATE}
def spawn(..): self.domains.append(new_dom)
// inject key into image self.change_domain_state(..)
// _conn.create_domain(..) bare_metal.set_fs(..)
// db.instance_set_state(..) bare_metal.activate_node(..)
def get_console_output(..): …
// console_log](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-3-320.jpg)
![TODO [3] proxy_baremetal.py: domain management
+ 4
• bare_metal.get_idle_node() is needed
baremetal_id - ID format is different system by system
- ID can be board_number or string name
- “baremetal_type” and “baremetal_id”
- return value: baremetal_id
• bare_metal.get_status() & set_status() are needed
- Status can be number(0/1) or string (None/ALLOCATED)
- argument and return value: 0/1 (conversion)
Tilera: <board_id> 0 ~ 9
Status field is added in __init__
[{'status': 0, 'kernel_id': '1896115634’, 'ramdisk_id': '', 'image_id': '1552326678', 'ip_address': '10.9.1.45’,
'board_id': 2, 'name': 'instance-00000046', 'vcpus': 1L, 'mac_address': '02:16:3e:1f:84:69', 'dhcp_server':
'10.9.1.1'}]
‘baremetal_type’:’tilera’, ‘baremetal_id’:2
Heckle: <Name> bb01~](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-4-320.jpg)
![TODO [4] proxy_baremetal.py: get_image & set_image
+ 5
• bare_metal.get_image(baremetal_id) is needed
set_fs - Case#1: passed image by euca-run-instances (every time)
- no needed
- Case#2: fake bare metal image / no passed image (manual)
- Actual images should be passed
- After getting images (path) from bare_metal,
vmlinux/initrd/disk should be copied to basepath/.
Then key should be injected to real disk image.
• bare_metal.set_image(baremetal_id) is needed
- In case that specific directory is used for images (/tftpboot/),
it should pass the key injected image into specific directory
Tilera: bpath/root -> /tftpboot/fs_x.tar.gz /tftpboot
[{'status': 0, 'kernel_id': '1896115634’, 'ramdisk_id': '', 'image_id': '1552326678', 'ip_address': '10.9.1.45’,
'board_id': 2, 'name': 'instance-00000046', 'vcpus': 1L, 'mac_address': '02:16:3e:1f:84:69', 'dhcp_server':
'10.9.1.1'}]
/bootconfig
Heckle: <Image> /node
/image
/hardware
/user
/powercontroller …](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-5-320.jpg)
![TODO [5] proxy_baremetal.py: activate_node
+ 6
• bare_metal.activate_node(baremetal_id, ip_addr, mac_addr)
Activate_node - In case of Tilera, the first/second boot use different vmlinux.
In case of Heckle, script is used for setting activation
-> bare_metal.boot_set() (Tilera:mboot <-> gPXE script)
- Tilera uses ePDU management script <-> IPMI
-> bare_metal.power_mgr()
-> bare_metal.check_activated()
- Some settings for network and services are needed
-> bare_metal.network_set()
• bare_metal.deactivate_node(baremetal_id) is needed [destroy]
Tilera: /tftpboot/vmalinux_x_1 and _2
Heckle: <vmlinux>](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-6-320.jpg)
![TODO [6] proxy_baremetal.py: get_console_output
+ 7
• bare_metal.get_console_output(instance) is needed
console_output - In case of Tilera, /proc/kmsg output is saved offline
and then reused whenever it is needed
b/c MDE 2.0 doesn’t support dmesg
- Most bare metal machines can get dmesg output
and then save before running OpenStack
Tilera:](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-7-320.jpg)
![TODO [7] proxy_baremetal.py: update_available_resource()
+ 8
• Needed host information
update_resource - get_vcpu_total() - get_memory_mb_total()
- get_local_gb_total() - get_vcpu_used()
- get_memory_mb_used() - get_local_gb_used()
- get_hypervisor_type() - get_hypervisor_version()
- get_cpu_info()
• bare_metal.get_hw_info(field) is needed
- In case of Tilera, all information is saved into tilera_boards file,
and each field is loaded from file when running OpenStack,
and then each specific field is retrieved through “field” arg.
- vcpu: # of bare_metal node, it should be changed acct. #(run)
- total/used: each information of each node
- Assumption: one proxy bare_metal for homogeneous machines
Tilera: tilera_boards
Heckle: Properties](https://image.slidesharecdn.com/proxybaremetalcomputev2-110524154208-phpapp01/85/Design-of-bare-metal-proxy-compute-node-8-320.jpg)
Design of bare metal proxy compute node
- 1. + Design of general bare_metal proxy compute node Mikyung Kang ([email protected]) Dong-In Kang ([email protected]) 1 May 20, 2011
- 2. TODO [1] nova.conf + 2 Current Version Next Version Proxy Compute Node Proxy Compute Node LibvirtConnection LibvirtConnection (libvirt_conn.py) proxy_baremetal (libvirt_conn.py) --connection_type=libvirt specific model --connection_type=libvirt --connection_type=gpu --connection_type=gpu ProxyConnection (proxy_baremetal.py) 1 --connection_type=baremetal --connection_type=tilera gPXE.py? TileraConnection TileraConnection ….. ArmConnection (tilera.py) (tilera.py) (arm.py) --baremetal_driver=arm HeckleConnection 2 --baremetal_driver=tilera (heckle.py) --baremetal_driver=heckle Specific arch. or 1. Changing --connection_type: tilera baremetal Specific provisioning tool 2. Adding –baremetal_driver option in nova.conf: tilera/arm/heckle/…
- 3. TODO [2] proxy_baremetal.py overview + 3 Next Version Proxy Compute Node --connection_type=baremetal def create_domain(..): ProxyConnection … (proxy_baremetal.py) load bare_metal as –baremetal_driver in nova.conf baremetal_id = bare_metal.get_idle_node() new_dom = { class ProxyConnection ’baremetal_type': bare_metal, def _get_connection(..): ’baremetal_id': baremetal_id, // _conn <- baremetal_doms 'name': xml_dict['name'], def destroy(..): 'mac_address': xml_dict['mac_address'], // _conn.destroy_domain(..) 'ip_address': xml_dict['ip_address'], // db.instance_set_state(..) 'dhcp_server': xml_dict['dhcp_server'], 'image_id': xml_dict['image_id'], def reboot(..): 'kernel_id': xml_dict['kernel_id'], // _conn.reboot_domain(..) 'ramdisk_id': xml_dict['ramdisk_id'], // db.instance_set_state(..) 'status': power_state.NOSTATE} def spawn(..): self.domains.append(new_dom) // inject key into image self.change_domain_state(..) // _conn.create_domain(..) bare_metal.set_fs(..) // db.instance_set_state(..) bare_metal.activate_node(..) def get_console_output(..): … // console_log
- 4. TODO [3] proxy_baremetal.py: domain management + 4 • bare_metal.get_idle_node() is needed baremetal_id - ID format is different system by system - ID can be board_number or string name - “baremetal_type” and “baremetal_id” - return value: baremetal_id • bare_metal.get_status() & set_status() are needed - Status can be number(0/1) or string (None/ALLOCATED) - argument and return value: 0/1 (conversion) Tilera: <board_id> 0 ~ 9 Status field is added in __init__ [{'status': 0, 'kernel_id': '1896115634’, 'ramdisk_id': '', 'image_id': '1552326678', 'ip_address': '10.9.1.45’, 'board_id': 2, 'name': 'instance-00000046', 'vcpus': 1L, 'mac_address': '02:16:3e:1f:84:69', 'dhcp_server': '10.9.1.1'}] ‘baremetal_type’:’tilera’, ‘baremetal_id’:2 Heckle: <Name> bb01~
- 5. TODO [4] proxy_baremetal.py: get_image & set_image + 5 • bare_metal.get_image(baremetal_id) is needed set_fs - Case#1: passed image by euca-run-instances (every time) - no needed - Case#2: fake bare metal image / no passed image (manual) - Actual images should be passed - After getting images (path) from bare_metal, vmlinux/initrd/disk should be copied to basepath/. Then key should be injected to real disk image. • bare_metal.set_image(baremetal_id) is needed - In case that specific directory is used for images (/tftpboot/), it should pass the key injected image into specific directory Tilera: bpath/root -> /tftpboot/fs_x.tar.gz /tftpboot [{'status': 0, 'kernel_id': '1896115634’, 'ramdisk_id': '', 'image_id': '1552326678', 'ip_address': '10.9.1.45’, 'board_id': 2, 'name': 'instance-00000046', 'vcpus': 1L, 'mac_address': '02:16:3e:1f:84:69', 'dhcp_server': '10.9.1.1'}] /bootconfig Heckle: <Image> /node /image /hardware /user /powercontroller …
- 6. TODO [5] proxy_baremetal.py: activate_node + 6 • bare_metal.activate_node(baremetal_id, ip_addr, mac_addr) Activate_node - In case of Tilera, the first/second boot use different vmlinux. In case of Heckle, script is used for setting activation -> bare_metal.boot_set() (Tilera:mboot <-> gPXE script) - Tilera uses ePDU management script <-> IPMI -> bare_metal.power_mgr() -> bare_metal.check_activated() - Some settings for network and services are needed -> bare_metal.network_set() • bare_metal.deactivate_node(baremetal_id) is needed [destroy] Tilera: /tftpboot/vmalinux_x_1 and _2 Heckle: <vmlinux>
- 7. TODO [6] proxy_baremetal.py: get_console_output + 7 • bare_metal.get_console_output(instance) is needed console_output - In case of Tilera, /proc/kmsg output is saved offline and then reused whenever it is needed b/c MDE 2.0 doesn’t support dmesg - Most bare metal machines can get dmesg output and then save before running OpenStack Tilera:
- 8. TODO [7] proxy_baremetal.py: update_available_resource() + 8 • Needed host information update_resource - get_vcpu_total() - get_memory_mb_total() - get_local_gb_total() - get_vcpu_used() - get_memory_mb_used() - get_local_gb_used() - get_hypervisor_type() - get_hypervisor_version() - get_cpu_info() • bare_metal.get_hw_info(field) is needed - In case of Tilera, all information is saved into tilera_boards file, and each field is loaded from file when running OpenStack, and then each specific field is retrieved through “field” arg. - vcpu: # of bare_metal node, it should be changed acct. #(run) - total/used: each information of each node - Assumption: one proxy bare_metal for homogeneous machines Tilera: tilera_boards Heckle: Properties
- 9. Overview + Should be separated into 2 or 3 files and each func. 9 should be added/modified! --connection_type=baremetal proxy_baremetal.py baremetal_dom.py arch.py (ex. tilera.py) __init__ __init__ class _nodes def set_status() Init_host() read_domain_from_file() def check_idle_node() _get_connection() reboot_domain() def get_idle_node() def find_ip_w_id() destroy() destroy_domain() def free_node() reboot() create_domain() def deactivate_node() def activate_node() spawn() change_domain_state() get_console_output() store_domain() def get_hw_info() def get_image() update_available_resource() find_domain() def set_image() … list_domains() def get_console_output() def check_activated() get_domain_info() … … --baremetal_driver=arch or tool (tilera/arm/heckle/…)