Reactive integrations with Akka Streams
3 likes1,907 views
This document summarizes a presentation about Akka streams, which is a toolkit for building highly concurrent, distributed, and resilient message-driven applications on the JVM. It provides asynchronous back pressured stream processing using sources, sinks, and flows. Key features include actors for concurrency, clustering for location transparency and resilience, and integration with technologies like Kafka, Cassandra and HTTP. The document outlines how Akka streams work, how to write stream applications, and how streams can be used for scenarios like HTTP requests/responses and streaming data. It encourages contributions to Akka and discusses next steps like improved remoting and more stream connectors.
![Streaming in Akka HTTP
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
HttpServer as a:
Flow[HttpRequest, HttpResponse]](https://image.slidesharecdn.com/2016-reactivesummit-reactiveintegrationswithakkastreams-170118094531/85/Reactive-integrations-with-Akka-Streams-36-320.jpg)
![Streaming in Akka HTTP
HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html](https://image.slidesharecdn.com/2016-reactivesummit-reactiveintegrationswithakkastreams-170118094531/85/Reactive-integrations-with-Akka-Streams-37-320.jpg)
![Streaming in Akka HTTP
HttpServer as a:
Flow[HttpRequest, HttpResponse]
HTTP Entity as a:
Source[ByteString, _]
Websocket connection as a:
Flow[ws.Message, ws.Message]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala
“Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html](https://image.slidesharecdn.com/2016-reactivesummit-reactiveintegrationswithakkastreams-170118094531/85/Reactive-integrations-with-Akka-Streams-38-320.jpg)
![Streaming from Akka HTTP (Java)
public static void main(String[] args) {
final ActorSystem system = ActorSystem.create();
final Materializer materializer = ActorMaterializer.create(system);
final Http http = Http.get(system);
final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));
final Route tweetsRoute =
path("tweets", () ->
completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json())
);
final Flow<HttpRequest, HttpResponse, NotUsed> handler =
tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler,
ConnectHttp.toHost("localhost", 8080),
materializer
);
System.out.println("Running at http://localhost:8080");
}](https://image.slidesharecdn.com/2016-reactivesummit-reactiveintegrationswithakkastreams-170118094531/85/Reactive-integrations-with-Akka-Streams-43-320.jpg)
![Streaming from Akka HTTP (Java)
public static void main(String[] args) {
final ActorSystem system = ActorSystem.create();
final Materializer materializer = ActorMaterializer.create(system);
final Http http = Http.get(system);
final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));
final Route tweetsRoute =
path("tweets", () ->
completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json())
);
final Flow<HttpRequest, HttpResponse, NotUsed> handler =
tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler,
ConnectHttp.toHost("localhost", 8080),
materializer
);
System.out.println("Running at http://localhost:8080");
}](https://image.slidesharecdn.com/2016-reactivesummit-reactiveintegrationswithakkastreams-170118094531/85/Reactive-integrations-with-Akka-Streams-44-320.jpg)
Reactive integrations with Akka Streams
- 1. akka streams Reactive Integrations with that just work™ Johan Andrén Konrad Malawski
- 2. Johan Andrén Akka Team Stockholm Scala User Group
- 3. Konrad `ktoso` Malawski Akka Team, Reactive Streams TCK, Persistence, HTTP
- 4. Make building powerful concurrent & distributed applications simple. Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
- 5. Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community Complete Java & Scala APIs for all features What’s in the toolkit?
- 8. akka streams Asynchronous back pressured stream processing Source Sink Flow
- 9. akka streams Asynchronous back pressured stream processing Source Sink (possible) asynchronous boundaries Flow
- 10. akka streams Asynchronous back pressured stream processing Source Sink 10 msg/s 1 msg/s OutOfMemoryError!! Flow
- 11. akka streams Asynchronous back pressured stream processing Source Sink 10 msg/s 1 msg/s hand me 3 morehand me 3 more 1 msg/s Flow
- 12. akka streams Not only linear streams Source SinkFlow Source Sink Flow Flow
- 13. Reactive Streams Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols http://www.reactive-streams.org
- 14. Part of JDK 9 java.util.concurrent.Flow
- 15. Reactive Streams RS Library A RS library B async boundary
- 16. Reactive Streams RS Library A RS library B async boundary Make building powerful concurrent & distributed applications simple.
- 17. The API Akka Streams Complete and awesome Java and Scala APIs (Just like everything in Akka)
- 18. Akka Streams in 20 seconds: Source<Integer, NotUsed> source = null; Flow<Integer, String, NotUsed> flow = Flow.<Integer>create().map((Integer n) -> n.toString()); Sink<String, CompletionStage<Done>> sink = Sink.foreach(str -> System.out.println(str)); RunnableGraph<NotUsed> runnable = source.via(flow).to(sink); runnable.run(materializer);
- 19. Akka Streams in 20 seconds: CompletionStage<String> firstString = Source.single(1) .map(n -> n.toString()) .runWith(Sink.head(), materializer);
- 20. Source.single(1).map(i -> i.toString).runWith(Sink.head()) // types: _ Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>> Akka Streams in 20 seconds:
- 21. Source.single(1).map(i -> i.toString).runWith(Sink.head()) // types: _ Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>> Akka Streams in 20 seconds:
- 22. Materialization Gears from GeeCON.org,(it’s an awesome conf)
- 23. What is “materialization” really?
- 24. What is “materialization” really?
- 25. What is “materialization” really?
- 26. What is “materialization” really?
- 27. What is “materialization” really? Check out the “Implementing an akka-streams materializer for big data” talk later today.
- 28. AlpakkaA community for Streams connectors http://blog.akka.io/integrations/2016/08/23/intro-alpakka
- 29. Alpakka – a community for Stream connectors Threading & Concurrency in Akka Streams Explained (part I) Mastering GraphStages (part I, Introduction) Akka Streams Integration, codename Alpakka A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II) Writing Akka Streams Connectors for existing APIs Flow control at the boundary of Akka Streams and a data provider Akka Streams Kafka 0.11
- 30. Alpakka – a community for Stream connectors Existing examples: MQTT AMQP Streaming HTTP Streaming TCP Streaming FileIO Cassandra Queries “Reactive Kafka” (akka-stream-kafka) S3, SQS & other Amazon APIs Streaming JSON Streaming XML …
- 31. Alpakka – a community for Stream connectors Demo
- 32. Alpakka – a community for Stream connectors Demo
- 33. Akka Streams & HTTP streams & HTTP
- 34. A core feature not obvious to the untrained eye…! Akka Streams / HTTP Quiz time! TCP is a ______ protocol?
- 35. A core feature not obvious to the untrained eye…! Akka Streams / HTTP Quiz time! TCP is a STREAMING protocol!
- 36. Streaming in Akka HTTP http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html HttpServer as a: Flow[HttpRequest, HttpResponse]
- 37. Streaming in Akka HTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
- 38. Streaming in Akka HTTP HttpServer as a: Flow[HttpRequest, HttpResponse] HTTP Entity as a: Source[ByteString, _] Websocket connection as a: Flow[ws.Message, ws.Message] http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming” http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
- 39. It’s turtles buffers all the way down! xkcd.com
- 40. Streaming from Akka HTTP
- 41. Streaming from Akka HTTP
- 42. Streaming from Akka HTTP No demand from TCP = No demand upstream = Source won’t generate tweets => Bounded memory stream processing! Demo
- 43. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
- 44. Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system); final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world")); final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) ); final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer); http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080"); }
- 45. Streaming from Akka HTTP (Scala) object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer() implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet) def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080"); }
- 46. Next steps for Akka Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron). More integrations for Akka Streams stages, project Alpakka. Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski Akka Typed progressing again, likely towards 3.0. Akka HTTP 2.0 Proof of Concept in progress. Collaboration with Reactive Sockets
- 47. Ready to adopt on prod?
- 48. Totally, go for it.
- 49. Akka <3 contributions Easy to contribute tickets: https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22 Akka Stream Contrib https://github.com/akka/akka-stream-contrib Mailing list: https://groups.google.com/group/akka-user Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka
- 50. Reactive Platform Reactive Platform Reactive Platform
- 51. Further reading: Reactive Streams: reactive-streams.org Akka documentation: akka.io/docs Free O’Reilly report – very out soon. Example Sources: ktoso/akka-streams-alpakka-talk-demos-2016 Get involved: sources: github.com/akka/akka mailing list: akka-user @ google groups gitter channel: https://gitter.im/akka/akka Contact: Konrad [email protected] Malawski http://kto.so / @ktosopl