Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

1
Taro L. Saito
Treasure Data
October 17th, 2020
Scala Matsuri 2020
Scala For Everything
From Frontend to Backend Applications
Scala無双: フロントエンドからバックエンドも全てScalaでなぎ倒せ

Copyright 1995-2020 Treasure Data. All rights reserved.
About Me: Taro L. Saito
2
● Treasure Data
○ Principal Software Engineer
○ Living in US for 5 years
● Ph.D. of Computer Science
○ Database systems, genome science.
● OSS:
○ Airframe, snappy-java (used in
Parquet, Spark, etc.), sbt-sonatype,
sbt-pack, etc.
● Book:
自己紹介

Today’s Topic
● There are a lot of libraries and frameworks for backend and frontend applications
● Airframe
○ An OSS library collection that enables Scala-oriented development
● Goal: Forget everything other than Scala
3
Scala無双：Scala以外を全て駆逐する 
Airframe
Scala.js Web Browsers
JSON

Airframe: Core Scala Modules of Treasure Data
● Airframe (GitHub: wvlet/airframe)
○ 20+ Common Utilities for Scala and Scala.js (Scala Matsuri 2019)
■ Logging, Dependency Injection (DI), JSON/MessagePack parser, etc.
○ Airframe RPC, Airframe Rx (Scala Matsuri 2020)
○ AirSpec
■ Testing framework for Scala and Scala.js (ScalaDays. Seattle, May 2021)
4
Knowledge
Experiences
Design Decisions
Products
24/7 Services
Business Values
Programming OSS Outcome
Airframe
AirframeはScala.jsでも利用することを前提に開発されてきたOSSツール群

Treasure Data: A Ready-to-Use Cloud Data Platform
5
Logs
Device
Data
Batch
Data
PlazmaDB
Table Schema
Data Collection Cloud Storage Distributed Data Processing
Jobs
Job Management
SQL Editor
Scheduler
Workflows
Machine
Learning
Treasure Data OSS
Third Party OSS
Data
トレジャーデータはすぐに使えるクラウドデータプラットフォーム

Scala.js Application: Presto Service Monitor
● Presto Usage: More than 1,000,000 queries / day (2020), 4 regions
6
Scala.jsで開発されたアプリケーションの例

Server APIs and Library for Spark
Airframe DI
DataFrame MPC1
airframe-codec
airframe-msgpack
Plazma Public API
airframe-http
airframe-finagle
Airframe DI
Airframe RPC
airframe-fluentd
Master Worker
DesignSparkContext
TDSparkContext TDSparkService
MPC1 Reader/Writer IO Manager
Airframe DI
airframe-http
airframe-config
airframe-launcher
airframe-jmx
airframe-metrics
airframe-control
airframe-metrics
td-spark.jarairframe-log
airframe-log
airframe-codec
airframe-json
Airframe
7
Sparkをサポートするために、ScalaとAirframeで作られたサービスが使われている 
● MPC1: MessagePack Columnar Format
● See also td-spark internals (Spark Meetup Tokyo
2020)

Scala is Versatile
● Backend
○ Can leverage JVM ecosystem (e.g.,
netty, gRPC, etc.)
● Frontend
○ Scala.js:
■ 1.0.0 is released in 2020
● The current version is 1.3.0
■ Compile Scala code into
Javascripts
○ Out of scope of this talk
■ iOS/iPhone native applications
● How can we bridge server (Scala JVM)
and client (Scala.js) applications?
Scalaはバックエンド、フロントエンド開発に使える言語に。では両者をどう繋ぐ？ 
Scala.js
Web Browsers
Serve-Client
Interaction
8

Bridging Scala and Scala.js with RPC
Program Function
Call
Return
Local Function Call
(ideal)
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
ローカル関数呼び出しと違い、実際にはネットワーク経由でのRPCが必要になる 
Remote Procedure Call
(reality)
Network
Client
Call
Return
Server
Call
Return
9
Scala.js
Scala.js

Existing Approaches for Implementing RPC
● REST
○ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.)
○ REST web frameworks for Scala
■ Play framework, akka-http, Finatra, Finch, skinny-framework, etc.
● gRPC
○ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language
○ Generate server and client code stubs from proto files
○ Scala wrappers:
■ ScalaPB, muScala, akka-grpc, etc.
RPCを実装するには、RESTやgRPCなどが使われている 
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return
10

API Design War
https://cloud.google.com/blog/products/api-management/
understanding-grpc-openapi-and-rest-and-when-to-use-th
em
11
● Which API design should we choose?
● REST API
○ Designing good APIs over HTTP
requires experiences and signiﬁcant
amount of time
● gRPC
○ Requires special software like code
generators for using the ecosystem
around ProtocolBuﬀers
● And also, there are too many choices for
web frameworks
しかし、APIデザイン、フレームワーク選びでの争いは付きない

Why Not Using Scala for RPC Interface?
● Scala is a functional and statically-typed object-oriented programming language
○ Scala functions = RPC methods
○ Scala objects (statically typed) = RPC request/response types
● Scala is a perfect ﬁt for RPC!
12
関数型で静的型付きオブジェクト指向言語のScalaはRPCインターフェースに適している

Scala-First Approach: Airframe RPC
RPC InterfaceScala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
13
Airframe RPCはScalaをサーバー・クライアント共通インターフェースとして使う新手法 
RPC Call RPC Call
Implements
● Forget about REST or ProtocolBuﬀers!
● Use Scala as a uniﬁed interface between server and clients Airframe

Airframe RPC Backend Is Pluggable
RPC Interface RPC Web Server
Generates API Documentation
14
Airframe RPCのServer, Clientは差し替え可能 
Airframe
RPC Call RPC Call
Implements
● Backend server
○ Finagle (HTTP/1), gRPC (HTTP/2)
● Backend client
○ Finagle, gRPC, okhttp, Java URLConnection client etc.
RPC Clients

Building Blocks of RPC Frameworks
● 1. Message serializer and deserializer
● 2. Network data format
○ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC)
● 3. RPC interface language
○ REST API, Protobuf, or Scala
● 4. HTTP client and server implementation
○ Code generator
15
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Return
Server
Call
Return
RPCフレームワークを作るのに必要な構成要素

MessagePack-Based Object Serialization
● MessagePack
○ A compact binary format compatible
with JSON
Object Object
Pack Unpack
PackUnpack
Server SideClient Side
16
Scala.js
JSON
Airframe
MessagePackを使ってクライアントとサーバー間通信を行う

airframe-codec: Pack/Unpack Interface
● Pack
○ Converting Data to MessagePack
● Unpack
○ Converting MessagePack to Data
Input Output
Pack Unpack
PackUnpack
17
Airframe
MessageCodec MessageCodec
MessagePackを通したデータ変換インターフェースをScalaに適用

Pre-deﬁned Codecs in airframe-codec
● Primitive Codecs
○ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec
○ FloatCodec, DoubleCodec
○ StringCodec
○ BooleanCodec
○ TimeStampCodec
● Collection Codec
○ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec, etc.
○ OptionCodec
● Java-speciﬁc Codec
○ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec
○ JDBCResultSetCodec
● etc.
18
Scalaで必要なほぼ全てのデータ型へのマッピングをサポート 
Airframe

Serializing Complex Objects
Pack
Unpack
IntCodec
StringCodec
DoubleCodec
MessageCodec.of[A]
19
オブジェクトの型に合わせてCodecを合成できる 
Serialize
Deserialize
JSON

Deﬁning RPC Interface with Scala
● Add ＠RPC annotation to a trait (= RPC interface)
● Deﬁne model classes (= data structures) using case classes
● Use the same interface and model classes between servers and clients
RPC InterfaceをScalaで記述する 
20

Serializing Function Call Request
● Function call arguments
○ A sequence of argument name -> value, ….
● Example:
○ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”)
○ Serialize this Map into MessagePack with airframe-codec
● HTTP Mapping
○ HTTP method: POST
○ Path: /(package name)/hello
■ e.g., /hello.api.v1/hello
○ Content body: serialized function call argument data
21
関数呼び出しの引数もシリアライズする

Implementing RPC Servers
● Extending the RPC interface trait
Serverの実装はRPC traitの継承 
22

● Finagle: Twitter’s HTTP server written in Scala
○ HTTP/1 based. Accessible from web browsers running Scala.js code
airframe-http: Starting A Finagle Backend RPC Server
23
Airframe
Scala.js
airframe-rx-html
airframe-http: FinagleバックエンドのHTTPサーバーを起動する 
airframe-http-finagle

airframe-http: gRPC Backend
● Suited to internal microservices
● HTTP/2 based protocol
○ Multiplexing multiple RPC
requests within a single
connection
● No .proto ﬁle is required
24
Airframe
airframe-http: gRPCバックエンドサーバーを起動する。ProtoBuf定義は必要ない

Advanced Topic: Extending gRPC
● gRPC is data-format agonistic
framework
○ To support Scala interface and
MessagePack, we have two
extension points in grpc-java.
● MethodDescriptor
○ Deﬁne gRPC endpoints
corresponding to Scala
functions
○ Register marshallers
● Request/ResponseMarshaller
○ Deﬁne how to encode/decode
RPC request/response data
25
Airframe RPCはgrpc-javaを拡張してMessagePackデータの受け渡しをしている

RPC Performance Comparison (Greeter Service)
● Airframe RPC
○ serde: MessagePack (airframe-codec) <->
Scala case classes
○ Finagle (HTTP1) or gRPC (HTTP2)
● ScalaPB
○ serde: Protobuf <-> Scala case classes
○ gRPC (HTTP2)
● grpc-java
○ serde: Protobuf <-> Java classes
○ gRPC (HTTP2)
● Notes
○ Using Finagle with HTTP2 had almost no
beneﬁt over HTTP1
■ Multiplexing RPC requests over
HTTP2 is the key performance factor
○ Overhead of ScalaPB
■ Scala Future (10% overhead)
■ Mapping Protobuf to Scala case
classes (10% overhead)
26
gRPCはリクエスト多重送信により高速。MessagePackによるオーバーヘッドは少ない 
HTTP/1 HTTP/2 (gRPC)

sbt-airframe: Generating RPC Clients
● sbt-airframe plugin
○ Read RPC interface classes
○ Generate HTTP client code for accessing the RPC server
○ OpenAPI schema generation support
sbt-airframe
Code
Generation
RPC Client
Scala.js
27
Scala.js Client
HTTP/gRPC Client
Open API Spec
Cross-Language
RPC Client
RPCクライアントを生成するsbt-airframeプラグイン

sbt-airframe: Generating Open API Schema
● RPC Interface -> Open API schema YAML ﬁle
● Generating RPC Clients & Swagger Documentation
28
ScalaのインターフェースからOpenAPIのYAMLを生成する 
RPC Interface
API Documentation
Open API Spec
(YAML)
Cross-Language
RPC Client
Generate
sbt-airframe
API Documentation

Making An RPC Call
RPCを実行する。サーバー、クライアント共にScalaで動いている 
Program Function
Serialize
Deserialize
Request Data
Deserialize
Response Data
Serialize
Client
Call
Response
Server
Call
Response
AirframeScala.js
29

Scala.js Programming Tips
● Scala.js: Compiling Scala -> JavaScript
● Use Pure Scala Code
○ [NG] Java code
■ Dependent libraries also need to be pure Scala
● Airframe is carefully designed to minimize non-Scala dependencies
○ [NG] Annotation, runtime-reﬂection
○ [OK] Scala Macros (compile-time meta-programming)
● RPC
○ Asynchronous calls using Scala Future
○ Await(Future) is not supported
■ Because JS is a single-threaded model.
■ Need to properly chain Future operators with map, ﬂatMap, rescue, etc.
30
Scala.jsでのプログラミングのコツ 
Scala.js

Debugging Scala.js Code with airframe-log
● Showing logs at the developer console of the browser
31
airframe-logとブラウザのコンソールでScala.jsコードのデバッグ

Rendering DOM with Scala.js
● DOM element:
○ <div class=”container”> …. </div>
○ scalajs-dom library has functions necessary for generating DOMs
● Previous approach: Converting XML literal into DOM elements
○ monadic-html, Binding.scala
○ But XML literal will be deprecated in Dotty (Scala 3.0)
32
Scala.jsでのDOMレンダリング手法には様々な手法がある

Rendering DOM
● Current Best Practice
○ Using Scala functions for
building DOM elements
● Option 1: Separation between
DOM elements and attributes
○ ScalaTags, Slinky
○ Too many parentheses!
● Option 2: No separation
between DOM elements and
attributes
○ scalajs-react, Airframe Rx
○ More Scala-friendly syntax
33
DOMのelement, attributeを区別しない方が関数型に適している 
ScalaTags
Airframe Rx

Complex DOM Rendering
● Airframe Rx
● RxElement interface
○ def render(): RxElement
○ Support nesting with apply(...)
■ Embedding Scala functions
and collections
34
airframe-rx-html: Scalaの関数をネストしてDOMをレンダリングする

Airframe Rx
● Rx.variable
○ Re-render DOM if the variable is updated
● Gathering best practices from existing libraries
○ Type-safe DOM elements and attributes
■ ScalaTags, scalajs-react
○ Reactive streaming
■ Monix
■ Scala Collection
■ Scala.rx
○ Cancelable Interface
■ Cleanup DOM and event handlers when
updating DOM
■ Monix, monadic-html
35
既存技術の良いところ取りをしたAirframe Rx

Reactive Stream Interface: Rx[X]
● Rx[X]
○ ReactiveX stream implementation
○ e.g., Monix, Scala.Rx
● Reactive DOM Rendering
○ Rewriting DOM interactively upon upstream data
change
● Reactive Operators
○ map, ﬂatMap, ﬁlter, join, etc.
○ Almost the same with Scala collection APIs
36
Reactive stream interface Rx[X]でインタラクティブなDOMを作成

gRPC Streaming with Airframe Rx
● Interactive communication between server and clients with reactive stream
interface: Rx[A]
37
gRPC StreamingもRxインターフェースでサポートできる

Airframe Enables Scala-Oriented Development
38
RPC InterfaceScala.js Client RPC Web Server
Generates
Scala.js
Web Application
API Documentation
RPC Call RPC Call
Implements
Airframe
AirframeでScala中心のアプリケーション開発を可能に 
● Only need to learn how to write Scala for kick-starting client/server application
development

Summary
● Scala is a powerful language that can consolidate existing technologies
● Minimizing the learning cost of application development with Scala-First
approach
○ Airframe OSS (GitHub: wvlet/airframe) is designed for this purpose
39
まとめ：既存技術の長所を兼ね備えることで、Scala無双を可能にするAirframe 
Airframe
Scala.js Web Browsers
JSON

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020

Recommended

More Related Content

What's hot (20)

Similar to Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020 (20)

More from Taro L. Saito (18)

Recently uploaded (20)

Scala for Everything: From Frontend to Backend Applications - Scala Matsuri 2020