#Include os - From bootloader to REST API with the new C++

#include <os>
From bootloader to REST API with the new C++
1Alfred Bratterud, CppCon 2016
We’re making
this a reality

Let’s put some operating system
inside your ELF binary

• Coined in 2013 by Madhavapeddy, Mortier et. al.[1]
• Single purpose: lots of servers are anyway
• Library operating system: Link in what you need
• Turns a binary into a bootable disk image
• Mainly targeted at virtual machines and IoT
Unikernels 101

IncludeOS: A minimal, resource efficient implementation
from scratch in modern C++ [3].

• 300x smaller disk image
• 100x less memory usage by default
• That much reduction in attack surface
V.S.
0
75
150
225
300
Disk Memory

• IncludeOS is single threaded by default
• Faster, more efficient for single core VM’s:
• 5-20% less CPU cycles spent for the same binary  
and workload [3]
• IncludeOS boots in ~300ms. Ubuntu boots in minutes.
• Ubuntu is awesome. As a general purpose OS.
V.S.

You all know what virtual machines are.
…let’s go through it anyway

8
Popek / Goldberg 1974
Gerald J. Popek, Robert P. Goldberg CACM [4]
«A virtual machine is taken to be an
efficient, isolated duplicate of the real
machine»
Alfred Bratterud, CppCon 2016

All sensitive instructions have
to trap!
9
Hardware
Hypervisor VM1 VM2

10
Hardware
Hypervisor
VM1
add r1 r2
mul r1 r2
mov r1 0x88
VM2
to trap!

11
Hardware
Hypervisor VM1
VM2
mov ‘a’ r2
mul r1 r2
mul r2 8
out ‘!’ 0x87
to trap!

12
Hardware
Hypervisor VM1
out ‘!’ 0x87
TRAP!
to trap!

13
Hardware
Hypervisor
Oh, VM2 wants to write to
a device. let’s pretend.
VM1
out ‘!’ 0x87
TRAP!
to trap!

14
Hardware
Hypervisor VM1
out ‘!’ 0x87
in 0x87
…result as expected
to trap!

15
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
0 1 2 3 4 5 6 7 8 9
VMM
VM
If an address translation
results in a physical
address out of VM
bounds: TRAP!
Virtual memory

30 years later
Oh, good idea.

You run directly on hardware.
The VM-trap is just a glorified context switch.

Since we’re 100% self contained
we can run on Linux, Windows, Mac unchanged

It’s like a process.  
It just happens to need some operating system.

20
Bootable image "MyService.img"
Links into "MyService" ELF binary
Bootloader
Service.o
OS parts
from os.a
_start
IncludeOS
build-system overview
vmbuilder
Attach / prep
bootloader
OS classes
IRQ, PCI, Timers, Net
etc.
i.e. everything in ./src
part_1.o part_n.o
Builds into static library
Compile piecewise as usual
Stdlib
parts
os.a
Can become huge without affecting
VM size
Standard libraries
libc++.a, libc.a, libm.a etc.,
custom built for IncludeOS
Your C++ code
starts in service.cpp
A copy of ./seed,
+ all your service source ﬁles
• os.a: static library containing
all OS parts
• Drivers are now separated
• Linker pulls in what the
program needs
• OS library can bloat without
affecting image size
• vmbuild: Attach and modify
bootloader

Why C++?
(Yes, people really ask that)

C++ is the most evolved
“independent systems language”
Alfred Bratterud, CppCon 2016 22
• You need to write arbitrary bytes to arbitrary addresses
• You need access to the full instruction set
• You can’t write an OS in a 100% type safe language
• But new C++ gets you the best of both worlds

Hello World!
23
#include <os>
void Service::start(const std::string&) {
std::cout << "Hello world - OS included!n";
}
What about
“main”?
No parent
process - where
do you return?
“kernel” params
with multiboot
And where does stdout go?

Delegates everywhere!
A C++11 version of Don Clugstons  
“Fastest possible C++ Delegates” was found
… on Stack Exchange
http://codereview.stackexchange.com/questions/14730/impossibly-fast-delegate-in-c11

Delegates everywhere!
A C++11 version of Don Clugstons  
“Fastest possible C++ Delegates” was found
… on Stack Exchange
but we seriously owe
you a … beverage of
your choice
We’ll respect
your privacy
http://codereview.stackexchange.com/questions/14730/impossibly-fast-delegate-in-c11

Routing stdout anywhere
26
// VGA
OS::add_stdout(VGA::get_rsprint_handler());
// Custom
OS::add_stdout([] (const std::string& data) {
// Memory-only (webserver 'logger')
logger_->log(timestamp() + data);
});

What’s different  
about an OS  
for virtual machines?
In principle: nothing

CPU
VirtIO Net
LAPIC
PIT
Timer
Hardware overview
PCI BusAPIC
timer
IDTR
GDTR
IDE /
SCSI
VirtIO Block
VirtIO xyz
Serial
CR0-x
0 1 2 3 … n
ALL of memory
“Virtual“ devices
“Physical” interface
OS doesn’t know if
they’re physical or
virtual
PIC

How to include only the drivers you need -
without having to mention them in code?
29

Solution: link in
self registering drivers
30
// From IncludeOS v0.9, src/drivers/virtionet.cpp
// ...VirtioNet implementation
/** Register VirtioNet's driver factory at the PCI_manager */
struct Autoreg_virtionet {
Autoreg_virtionet() {
PCI_manager::register_driver<hw::Nic>(hw::PCI_Device::VENDOR_VIRTIO,
0x1000, &VirtioNet::new_instance);
}
} autoreg_virtionet;
1) Global constructor registers a
delegate to driver initialization.
2) OS probes PCI bus and applies any
registered driver
Service Makefile:
DRIVERS = virtionet …
Deployment tools /
sysadmins can apply
appropriate drivers

Kernel overview
31
Call any delegates
subscribing to IRQ’s,
then “hlt”
// 1. Initialize stack, heap, .bss etc.
// 2. Call global constructors
// 3. Initialize hardware
// 4. Call Service::start()
void OS::event_loop() { 
while (power_) {
IRQ_manager::get().notify();
}
// Cleanup
Service::stop();
// ACPI shutdown sequence
hw::ACPI::shutdown();
}
// From src/kernel/os.cpp
Note: All subscribed
IRQ’s are deferred

// Actual code from src/drivers/virtionet.cpp
auto del(delegate<void()>
::from<VirtioNet,
&VirtioNet::irq_handler>(this));
IRQ_manager::get().subscribe(10, del);
Subscribing to interrupts
32
Create delegate to
member function of
“this”
Subscribe to a given
IRQ number 
called by event loopFor most use cases don’t to worry
about IRQ (this is driver code).  
Abstractions coming up
!

• Each “box” is a class instance
• All connections are delegates
• The wiring happens in the “Inet”
wrapper class
• Rewiring during runtime is easy
• e.g. for activating and deactivating an IP-filter
• Can be done from the application
Wiring the IP stack
Prot.
Prot.
Prot.
Eth.Type
Eth.Type
Pong
Req.
Resp.
Ping 1
n 2^16
1
n
2^16
Downstream - Transmit
Upstream
TX RX
Ethernet
IP6IP4
ARP
TCPICMPUDP
Driver - VirtioNet
Device (Virtio)
53
DNS
reply
Eth.Type
33

• Each “box” is a class instance
• All connections are delegates
• The wiring happens in the “Inet”
wrapper class
• Rewiring during runtime is easy
• e.g. for activating and deactivating an IP-filter
• Can be done from the application
Wiring the IP stack
Prot.
Prot.
Prot.
Eth.Type
Eth.Type
Pong
Req.
Resp.
Ping 1
n 2^16
1
n
2^16
Downstream - Transmit
Upstream
TX RX
Ethernet
IP6IP4
ARP
TCPICMPUDP
Driver - VirtioNet
Device (Virtio)
53
DNS
reply
My IP-ﬁlter
34

• Single threaded by default: “no overhead principle”
• In a VM, threading is more expensive [5]
• 100% async: cheapest way to get performance
• No blocking POSIX calls (yet)
• You’ll get it. It’s your “constitutional right…”
Unikernels: 
The default should be minimal

Writing a TCP server
36
#include <os>
#include <net/inet4>
void Service::start(const std::string&) {
auto& server = net::Inet4::stack().tcp().bind(80);
server.on_connect([] (auto conn) {
conn->on_read(1024, [conn] (auto buf, size_t n) {
conn->write("My first unikernel!n"s);
});
});
}
Attach any delegate
to TCP events

• We don’t know
• If you get in, you’re in a hardware protected sandbox
• No shell or other “ad-hoc” code access
• Minimal attack surface: no tools left for the thieves
• Everything inside can be randomized
How do you break in?

“Leave no room for a lower-level language below C++
(except assembler)”
- Bjarne Stroustrup
You made it!

But do we need another language above C++?
The new C++ is awesome for high-level abstractions

Introducing “Acorn”
IncludeOS web server
with C++ framework for REST APIs

Static content
served from  
FAT file system
Replace with your
own content
Can serve from
memdisk inside ELF
or VirtioBlock device
Demo of POST’ing
data over REST

42
– Bjarne Stroustrup earlier today
“C++ is not the language for putting
up a simple web page”

How about a fancy web page?

Let’s do a dashboard with live CPU stats and live profiling.
It’s surprisingly easy with C++

10k http
requests
Mostly heap for the
application
TCP connection
status

Timer IRQ-based live
stack sampling
statman: Centrally
located stats for
everything, including
your service

Logs: stdout routed
to in-memory ring-
buffer

Individual REST
endpoints for each
component.  
100% done in
application.
The OS exposes
nothing  
except an API to
access STL data
structures from App

Routes!
RESTful endpoints for anything in IncludeOS

“Node/Express.js”-style routes
50
Router router;
// Assume data structure with "user"
// GET /api/users/:id where 'id' is a numeral
router.on_get("/api/users/:id(d+)", [users] (auto req, auto res) {
try {
// Try to retrieve param "id"
auto& params = req->params();
std::string id = params.get("id");
// Look for the user inside the bucket
auto& user = users->pick_up(std::stol(id));
} catch (const std::exception& e) {
res->send_code(http::Not_Found);
}
});
Regex-based mini-language for
expressing routes.
ported to C++ from express.js
The “:id” part is made
accessible to the app as key to
a value.
http://my_domain/api/users/42
matches regex and captures “id”, “42”

Default route: serve from FAT fs
51
Router router;
// Serve index.html from root URL
router.on_get("/", [](auto, auto res) {
// Stat disk for index.html
disk->fs().cstat("/public/index.html", [res] (auto err, const auto& entry) {
if(err)
res->send_code(http::Not_Found);
else
// Serve index.html
res->send_file({disk, entry});
}
});
});

Middleware
Applying a stack of functions to all requests

“Node/Express.js”-style middleware
53
Check for JSON
content type
Expect parsed JSON
server.use([](auto req, auto res, auto next) {
if(has_content_type_json(req)) {
auto json = std::make_shared<Json_doc>();
json->doc().Parse(req->get_body().c_str());
req->set_attribute(json);
}
(*next)();
});
router.on_post("/users", [](auto req, auto res) {
auto json = req->get_attribute<Json_doc>();
if(json) {
auto& doc = json->doc();
// Register user based on parsed JSON data
}
});
Parse and attach to
requestCall next middleware
(or exit to hang up)

Readymade middleware
54
// Director: Provide an auto-generated HTML directory listing
auto director = std::make_shared<middleware::Director>(disk, "/public/static");
server.use("/static", director);
// Cookie parser: parse all cookies on incoming requests
auto cookie_parser = std::make_shared<middleware::CookieParser>();
server.use(cookie_parser);
// Butler: Serve static files on root route with disk root on /public
auto opt = {"index.html", "index.htm"};
auto butler = std::make_shared<middleware::Butler>(disk, "/public", opt);
server.use(butler);

Performance:
Virtual machines aren’t faster than bare metal

The goal is to have thousands of self-
contained, secure micro services

• All services run in single core VM
• Apache / Node on Ubuntu 16.10
• “Acorn” on IncludeOS
• 11% faster than Apache
• 79% faster than Node.js
• 10k http req. with httperf
• Average over 66 samples
• Intel i7 desktop PC w. Ubuntu 16.04
“Acorn” preliminary
performance data
Req./sec.
0
2250
4500
6750
9000
Acorn Apache Node.js

Deploy in the cloud!

CppCon 2016
Deploy in OpenStack
59
$ make
$ glance --image-create --file <My_image.img> …
$ nova boot --image …
Get an IP,
Fire up a browser!
Literally three
commands
We’ll be adding support for
AWS, Google compute, 
Azure, you name it.

• 1.0 “Stable API” depends on your feedback!
• Production ready core by years end
• “Acorn”: open source alpha today (!)
• TLS, POSIX and multi-arch are priorities
• Tooling improvements:
• project unik for deployment / management
• Package manager (3rd party)
• CMake builds
• IncludeOS specific CLI tool
Project status

• Virtual machines aren’t heavy - OS’es are
• IncludeOS is faster, more efficient,  
more secure for single purpose
• C++ is great at both high- and low level.
• Definitely the best language for the job
Recap

• Pull Requests
• Ideas for cool use cases
• Expert advice!
Please contribute!
alfred@includeos.org
Demo!
Open Content session
showing the implementation
of an IRC server on
IncludeOS on Thursday
Fork: Chat:

Questions?

1. “Unikernels: Library operating systems for the cloud”,  
Anil Madhavapeddy, Richard Mortier et. al, ASPLOS 2013
2. “Unikernels: Rise of the Virtual Library Operating System”, 
Anil Madhavapeddy, Dave Scott, ACM Queue, 2013
3. “IncludeOS: A minimal, Resource efficient Unikernel for Cloud Systems, 
A. Bratterud, A. A. Walla et. al, IEEE CloudCom 2015
4. “Formal requirements for Virtualizable Third Generation Architectures”,
G. Popek, R.P. Goldberg, Communications of the ACM Vol. 17, 1974
5. “APLE: Addressing Lock Holder Preemption Problem with High
Efficiency”, J. Shan, X. Ding and N. Gehani, IEEE CloudCom 2015
References
64CppCon 2016

#Include os - From bootloader to REST API with the new C++

Recommended

More Related Content

What's hot (20)

Viewers also liked (7)

Similar to #Include os - From bootloader to REST API with the new C++ (20)

Recently uploaded (20)

#Include os - From bootloader to REST API with the new C++