The State of Lightweight Threads for the JVM

The State of Fibers for the JVM
Why all your JVM coroutines are cool, but broken,
and Project Loom is gonna fix that!
(Kilim threads, Quasar fibers, Kotlin coroutines, etc.)
and ...
Volkan Yazıcı
https://vlkan.com
@yazicivo
Lightweight Threads
Virtual
2018-03-08
2018-08-25
2019-12-10

Agenda
1) Motivation
2) History
3) Continuations
4) Processes, threads, and fibers
5) Project Loom
6) Structured concurrency
7) Scoped variables

Motivation: I/O
Quoting from Gor Nishanov's 2016 "LLVM Coroutines: Bringing resumable functions to LLVM" talk.

Without I/O, you don’t exist!
● println()
● file access
● network socket access
● database access
● etc.

In 1958, ...

The legend of Melvin Conway

Subroutine Coroutine⊂ Coroutine
“Coroutines” – Melvin Conway, 1958
“Generalization of subroutine” – Donald Knuth, 1968
subroutines coroutines
call
allocate frame,
pass params
allocate frame,
pass params
return
free frame,
return result
free frame,
return result
suspend No Yes
resume No Yes
Subroutine A Subroutine B
…
call B
…
B start
end
call B
B start
end
Subroutine A Coroutine C
…
call C
…
C start
end
suspend
suspend
resume C
resume C

Where did all the coroutines go?
Algol-60
… introduced code blocks and the begin and end pairs for
delimiting them. ALGOL 60 was the first language implementing
nested function definitions with lexical scope.
– Wikipedia “ALGOL 60”

One “continuation” to rule them all...

“Continuation Sandwich”
● The earliest description by Adriaan van Wijngaarden in 1964.
● “… a data structure that represents the computational process at a given point in the
process's execution; ...” – Wikipedia “Continuation”
● “Say you're in the kitchen in front of the refrigerator, thinking about a sandwich.
You take a continuation right there and stick it in your pocket. Then you get some
turkey and bread out of the refrigerator and make yourself a sandwich, which is now
sitting on the counter. You invoke the continuation in your pocket, and you find
yourself standing in front of the refrigerator again, thinking about a sandwich. But
fortunately, there's a sandwich on the counter, and all the materials used to make it
are gone. So you eat it.” – Luke Palmer, 2004

A glimpse of “continuation”
(define the-continuation #f)
(define (test)
(let ((i 0))
; call/cc calls its first function argument, passing
; a continuation variable representing this point in
; the program as the argument to that function.
(call/cc (lambda (k) (set! the-continuation k)))
; The next time the-continuation is called, we start here.
(set! i (+ i 1))
i))
> (test)
1
> (the-continuation)
2
3
> ; stores the current continuation (which will print 4 next) away
> (define another-continuation the-continuation)
> (test) ; resets the-continuation
1
2
> (another-continuation) ; uses the previously stored continuation
4
See Oleg Kiselyo’s An argument against call/cc and Undelimited continuations are not functions articles.

A glimpse of “delimited continuation”
; The reset delimits the continuation that shift captures (named by k in this example).
; The use of shift will bind k to the continuation (+ 1 []),
; where [] represents the part of the computation that is to be filled with a value.
(* 2 (reset (+ 1 (shift k (k 5)))))
(reset (* 2 (shift k (k (k 4)))))
; invokes (k 4) first (which returns 8),
; and then (k 8) (which returns 16).
; At this point, the shift expression has terminated,
; and the rest of the reset expression is discarded.
; Therefore, the final result is 16.
Unlike regular continuations, delimited continuations return a value, and thus may be
reused and composed.

What is so important about continuations?
Using continuations you can implement
● longjmp (C)
● exceptions (C++, Java, etc.)
● generators (Icon, Python, etc.)
● backtracking (Prolog, etc.)
● and… guess what else?

public SearchResponse search(SearchRequest request) {
// Check caches.
SearchResponse cachedResponse = cache.find(reqest);
if (cachedResponse != null) {
return cachedResponse;
}
// Check redirects.
SearchResponse redirectedResponse = redirectService.find(request);
if (redirectedResponse != null) {
return redirectedResponse;
}
// Perform plain search enriched with suggestions.
SearchRequest enrichedRequest = suggestionService.enrich(request);
return plainSearch(enrichedRequest);
}
public void search(
SearchRequest request,
Consumer<SearchResponse> callback) {
// Check caches.
return cache.find(request, cachedResponse -> {
callback.accept(cachedResponse);
} else {
// Check redirects.
redirectService.find(request, redirectedResponse -> {
return callback.accept(redirectedResponse);
} else {
suggestionService.enrich(request, enrichedRequest -> {
plainSearch(enrichedRequest, searchResponse -> {
callback.accept(searchResponse);
});
});
}
});
}
});
}
Callbacks (non-blocking)
public Mono<SearchResponse> search(SearchRequest request) {
return Flux
.concat(cache.find(request),
redirectService.find(request),
suggestionService
.find(request)
.flatMap(this::plainSearch))
.take(1)
.singleOrEmpty();
}
Reactor (non-blocking)
Blocking calls
How do we compose I/O?
(without changing neither the language, nor the VM byte code)

public SearchResponse search(SearchRequest request) {
// Check caches.
SearchResponse cachedResponse = cache.find(reqest);
return cachedResponse;
}
// Check redirects.
SearchResponse redirectedResponse = redirectService.find(request);
return redirectedResponse;
}
SearchRequest enrichedRequest = suggestionService.enrich(request);
return plainSearch(enrichedRequest);
}
executed
instructions
time
Blocking calls
thread
activity
running
blocked
Blocking what?

public Mono<SearchResponse> search(SearchRequest request) {
return Flux
.concat(cache.find(request),
redirectService.find(request),
suggestionService
.find(request)
.flatMap(this::plainSearch))
.take(1)
.singleOrEmpty();
}
Reactor (non-blocking)
callchain
How does asynchronous I/O get composed?

Why all this reactive hassle?
Thread-1 Thread-2 Thread-3 Thread-100...
Request-1 Request-2 Request-3 Request-100...
Thread-1
Request-1 Request-2 Request-3 Request-100...
Callchain-1
Callchain-2
Callchain-3
Callchain-100
...
Reactor(non-blocking)
Blockingcalls
If spawning and context
switching costs of threads
would be equal to the ones
in callchains, would you
still favor the latter?

Process-vs-thread
Data1
Code1
(single-threaded)
OS Process
Registers1
Stack1
CPU1
CPU2
CPUK
...
OS Scheduler
Files1
Data2
Code2
Registers2
Stack2
Files2
Registers1
Stack1
Data
Code
CPU1
CPU2
CPUK
...
OS Scheduler
Registers2
Stack2
Registers3
Stack3
...
RegistersN
StackN
(multi-threaded)
OS Process
Files
Both processes and threads denote a
continuation: a sequence of instructions
that can be suspended and resumed.
process = continuation + scheduler
thread = continuation + schedulerCheaper!

What is a fiber?
OS Process
CPU1
CPU2
CPUK
...
OS Scheduler
Registers1
Stack1
Data
Code
Registers2
Stack2
Registers3
Stack3
RegistersN
StackN
Files
...
Both processes and threads denote a
continuation: a sequence of instructions
that can be suspended and resumed.
process = continuation + scheduler
thread = continuation + schedulerCheaper!
fiber = continuation + schedulerCheaper!
kernel
user
...
User-level scheduler
M321
kernel both share memory
M >> N

● Yes, it can, but it (natively) doesn’t.
● Lisp(call/cc)
, BEAM(Erlang VM)
, Haskell, Go, JavaScript(async, await)
(natively) do.
● Quasar, Kilim, etc. provides continuations and fibers for JVM.
● What if someone calls Thread.sleep()?
● What if a coroutine calls a non-coroutine method?
What if that non-coroutine method is blocking?
● What about Kotlin coroutines?
Does JVM support fibers?
in essence, “continuations”

What about actors(Erlang, Akka)
/ channels(Go, Kotlin)
?
m1
m2
m3...mN
fiber { BlockingQueue<V>(bufferSize) } = Channel<V>(bufferSize)

A glimpse of Quasar & Kilim
Quasar
Kilim

So what is the problem?
Asynchronous task composition
Reactive Streams FibersCallbacks
● Language-agnostic
● Difficult to write
(callbackhell.com)
● Difficult to debug & profile
● Misses batteries
● Viral
● Language-agnostic
● Includes batteries
● Widely supported
● Need to learn a new language
● Difficult to write & get right
● Difficult to debug & profile
● Difficult to optimize (by compiler)
● Viral
● Just works
● Language-specific
● Requires explicit marking
● Misses batteries
Reflective calls are always considered suspendable. This is because the target
method is computed at runtime, so there’s no general way of telling if it’s going to call a suspendable method
or not before execution.
Java 8 lambdas too are always considered suspendable. This is because they can’t
declare checked exceptions, they are ultimately linked (via invokedynamic) to synthethic static methods that
can’t be annotated and it is difficult to tell at instrumentation time if lambdas implement a suspendable
interface.
Quasar will reject with an error any attempt to mark special methods
(that is, constructors and class initializers) as suspendable. This is because
suspending in an initializer could expose objects or classes before they’re fully initialized and this is an error-
prone, difficult-to-troubleshoot situation that can always (and must) be avoided.

http://cr.openjdk.java.net/~rpressler/loom/Loom-Proposal.html
Project Loom proposal
by Ron Pressler
The proposal

● Quasar: lightweight threads (fibers) for the JVM
● Comsat: fiber-aware impl’s of servlets, JAX-RS/Spring REST services, HTTP clients and JDBC
● SpaceBase: in-memory spatial and geo-spatial database
● Galaxy: distributed in-memory data grid that horizontally scales
In 2012, founded Parallel Universe with the following F/OSS product line:
Who is Ron Pressler anyway?
http://www.paralleluniverse.co/
http://openjdk.java.net/census#loom
Lead at Project Loom
(since 2017)

● One of Java's most important contributions when it was first released, over twenty
years ago, was the easy access to threads and synchronization primitives.
● … today's requirements is that the software unit of concurrency offered by the
runtime — the thread — cannot match the scale of the domain's unit of
concurrency, …
● … asynchronous APIs ... were created
● not because they are easier to write and to understand
● for they are actually harder
● not because they are easier to debug or profile
● they are harder (they don't even produce meaningful stacktraces)
● not because they compose better than synchronous APIs
● they compose less elegantly
● not because they fit better with the rest of
the language or integrate well with existing code
● they are a much worse fit
● but just because the implementation of the software unit of concurrency in
Java — the thread — is insufficient from a footprint and performance
perspective.
Proposal highlights

The key project deliverable
See Ron Pressler's detailed post in the loom-dev mailing list.
Lightweight threads
Asymmetric
one-shot (non-reentrant)
stackful
multi-prompt
delimited
continuations.

Are we there yet?
● OIO rewrite
● Continuations
● Strand → Fiber → Lightweight thread → Virtual thread
● Structured concurrency
● Scoped variables
A word on “legacy”...

... ... ... ... ...
The curse of control flow constructs
sequence goto if loop call
Quoting from Martin Sustrik's "Structured Concurrency" talk in FOSDEM'19.

Is GOTO harmful?
(So are threads!)
Quoting from Nathaniel J. Smith's "Notes on structured concurrency" blog post.

What does structured concurrency look like?
Quoting from Nathaniel J. Smith's "Notes on structured concurrency" blog post.
with open("my-file") as file_handle:
...
... ... ...

A glimpse of Trio
import trio
async def child1():
print(" child1: started! sleeping now...")
await trio.sleep(1)
print(" child1: exiting!")
async def child2():
print(" child2: started! sleeping now...")
await trio.sleep(1)
print(" child2: exiting!")
async def parent():
print("parent: started!")
async with trio.open_nursery() as nursery:
print("parent: spawning child1...")
nursery.start_soon(child1)
print("parent: spawning child2...")
nursery.start_soon(child2)
print("parent: waiting for children to finish...")
# -- we exit the nursery block here --
print("parent: all done!")
trio.run(parent)
parent: started!
parent: spawning child1...
parent: spawning child2...
parent: waiting for children to finish...
child2: started! sleeping now...
child1: started! sleeping now...
[... 1 second passes ...]
child1: exiting!
child2: exiting!
parent: all done!
A word on cancellation...

public class ScopeDemo {
private static final Logger LOGGER = LoggerFactory.getLogger(ScopeDemo.class);
private final List<Runnable> tasks = new ArrayList<>();
private final ThreadLocal<StringBuilder> stringBuilderRef =
ThreadLocal.withInitial(StringBuilder::new);
public void addTask(Runnable task) {
tasks.add(task);
}
public static void main(String[] args) throws IOException {
try (InputStream inputStream = new FileInputStream("/etc/passwd")) {
int firstByte = inputStream.read();
{
int randomByte = (int) Math.abs(Math.random() * 0xFF);
firstByte += randomByte;
}
}
}
}
Don’t we already have scoped variables?
What is the scope of a ThreadLocal?

> x=1
> function g() {
echo $x;
x=2;
}
> function f() {
local x=3;
g;
}
> f
> echo $x
Static-vs-Dynamic scoping
What do these two statements output?

> color=null
> function terrier(nursery) {
echo “Terrier sees $color.”;
}
> function dog(nursery) {
echo “Dog sees $color.”
color=”black-and-white”;
nursery.schedule(terrier)
}
> function cat(nursery) {
echo “Cat sees $color.”
}
> with nursery {
color=”colorful”
nursery.schedule(dog)
nursery.schedule(cat)
}
Dynamic scoping and structured concurrency
What does this statement output?

Conclusions
● Loom will radically change I/O composition in JVM.
● Structured concurrency and scoped variables will supplement that ease.

Thank you!
(Questions?)
Volkan Yazıcı
https://vlkan.com
@yazicivo

The State of Lightweight Threads for the JVM

Recommended

More Related Content

What's hot (20)

Similar to The State of Lightweight Threads for the JVM (20)

Recently uploaded (20)

The State of Lightweight Threads for the JVM