How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019

How to grow GraphQL and
remove SQLAlchemy and REST
API from a high-load Python
project
Oleksandr Tarasenko
EVO.company / prom.ua

Some of prom.ua numbers
● RPS 2500-3500
● total sites over 300 000
● products ~ 150 million
● pages ~ 400 million
2

What we had
● monolithic WSGI app (11 years)
● mako templates
● monolithic database
● monolithic Nginx
● poor REST API for all sub-services
● slow Delivery and Deployment
● new features were hard to develop
3

http://graphql.org/users/
5

http://graphql.org/users/ 6

Key History Notes
● 2008 - Start. SQLAlchemy
● 2014 - SQLConstruct
● 2014 - ElasticSearch ecosystem
● 2016 - GraphQL
● 2017 - Microservices way
● 2018 - Python 3.7 + async
7

GraphQL concept in prom.ua
● new client-proposal paradigm
● good for read-only requests and data
● two-level (x-level) graph:
○ low-level graph for database mapping (data
loading)
○ high-level graph for business logic
● auto documentation and easy testing with
graphiql tool
● data validation 8

9
Step 1. Mobile App API with GraphQL
Step 2. Separating Frontend from Backend
Step 3. Graph services as Proxy via different Graph APIs
Step 4. Replace SQLAlchemy models logic via Graph
Step 5. Mutations in Graph API
Step 6. A brave new world with GraphQL

Why we choose GraphQL
● good for read-only requests and data
● no mutations needed
● hard to update REST API versions (v1, v2,...vn)
● auto generated documentation
● new client-proposal paradigm
● difficult routes
10

2016
● hiku, https://hiku.readthedocs.io/
● two-level graph
● mobile API with DSL-QL (EDN)
11

from hiku.graph import Graph, Node, Link, Field, ...
from hiku.sources import sqlalchemy as sa
from hiku.types import Boolean, Integer, String, TypeRef, ...
product_query = sa.FieldsQuery('db.session', Product.__table__)
low_level_graph = Graph([
Node('Product', [
Field('id', Integer, product_query),
Field('name', String, product_query),
Field('price', None, product_query),
...
])
12

from hiku.graph import Graph, Node, Link, Field
from hiku.types import Boolean, Integer, String, TypeRef
Node('Product', [
...
])
13

from hiku.graph import Graph, Node, Link, Field
from hiku.types import Boolean, Integer, String, TypeRef
Node('Product', [
...
])
14

15

2017
● hiku upgrade to native GraphQL
● new site 2nd level graph
● reimplementing mobile sites
● SSR Node.js Apollo Client
● success story with metrics
16

Node.js implementation
● Only for the first request (SSR)
● Good for React prerender
● Routing
● Two-step implementation
19

Success metrics
23
from 20.5% to 14.5%

Success metrics
24
from 57.5% to 49%

Some of the numbers of the new scheme
● node.js workers 3-6 per front
● React render 5 ms
● prom.ua workers over 750
● number of requests split to node/python
26

27
Step 3. Graph services as Proxy via
different Graph APIs

Early 2018
● new order and shopping cart graphs
● new user cabinet
● Node.js and apollo for stitching two
graphs
● REST API under GraphQL
28

GraphQL schema stitching
29https://labs.getninjas.com.br/sharing-data-in-a-microservices-architecture-using-graphql-97db59357602

How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019

31
Step 4. Replace SQLAlchemy models
logic via Graph

Middle 2018
● Hiku upgrade to aliases and new data types
● Use SSR + GraphQL for portal mobile version
● Graph for replace SQLAlchemy models logic
and queries
● Rewriting sites SQL queries to GraphQL
● Remove Models from BL
● https://hiku.readthedocs.io/
32

Node('Product', [
...
])
33

Node('Product', [
...
])
34

product(ids: [1234, 1235]) {
id
name
price
}
"""
SELECT id, name, price FROM product
WHERE id IN (:id1, :id2, ..., :idN)
"""
35

product(ids: [1234, 1235]) {
id
name
price
}
"""
SELECT id, name, price FROM product
"""
36

Node('Product', [
...
Link(
'discount',
Optional[TypeRef['Discount']],
product_to_discount_query,
requires='id',
),
]),
Node('Discount', [
Field('id', Integer, discount_query),
Field('amount', Integer, discount_query),
])
])
37

Node('Product', [
...
Link(
'discount',
requires='id',
),
]),
Node('Discount', [
])
])
38

Node('Product', [
...
Link(
'discount',
requires='id',
),
]),
Node('Discount', [
])
])
39

40
discount_query = sa.FieldsQuery('db.session', Discount.__table__)
"""
SELECT id, name, price, price_currency_id FROM product
"""
product_to_discount_query = sa.LinkQuery(
'db.session',
from_column=Discount.product_id,
to_column=Discount.product_id,
)
"""
SELECT product_id FROM discount
WHERE product_id IN (:id1, :id2, ..., :idN)
"""

41
discount_query = sa.FieldsQuery('db.session', Discount.__table__)
"""
SELECT id, name, price, price_currency_id FROM product
"""
product_to_discount_query = sa.LinkQuery(
'db.session',
from_column=Discount.product_id,
to_column=Discount.product_id,
)
"""
SELECT product_id FROM discount
WHERE product_id IN (:id1, :id2, ..., :idN)
"""

42
product_sg = SubGraph(low_level_graph, 'Product')
high_level_graph = Graph([
Node('Product', [
Field('id', String, product_sg),
Field('name', String, product_sg.compile(S.this.name)),
Field('priceText', String, product_sg.compile(
get_price_text(S.this)
)),
Field('hasDiscount', Boolean, product_sg.compile(
is_discount_available(S.this, S.this.discount)
)),
Field('discountedPriceText', String, product_sg.compile(
get_discounted_price_text(S.this, S.this.discount)
)),
]),
])

43
Node('Product', [
)),
)),
)),
]),
])

44
Node('Product', [
)),
)),
)),
]),
])

45
@define(Record[{
'price': Float,
'price_currency_id': Integer,
'currency_settings': Any
}])
def get_price_text(product):
product_price_text = format_currency_data(
product_price,
product['price_currency_id'],
)
...
//..
return product_price_text.formatted_number

Key History Notes
● 2008 - Start. SqlAlchemy
● 2014 - SQL Construct
● 2014 - ElasticSearch ecosystem
● 2016 - GraphQL
● 2017 - Microservices way
● 2018 - Python 3.7 + async
46

47
Python 3.7 + Async + GraphQL

48
def link_to_some_product(opts):
product = db.session.query(
Product.id
).filter(
Product.id == opts['id'],
Product.status_on_display(),
).first()
if product is not None:
return product.id
else:
return Nothing

49
async def link_to_some_product(opts):
expr = select([Product.id]).where(
and_(
Product.status_on_display()
)
)
async with async_engine() as query_ctx:
product_id = await query_ctx.scalar(expr)
return product_id or Nothing

50
async def link_to_some_product(opts):
expr = select([Product.id]).where(
and_(
Product.status_on_display()
)
)
async with async_engine() as query_ctx:
product_id = await query_ctx.scalar(expr)
return product_id or Nothing

51
product_query = sa.FieldsQuery(
'db.session', Product.__table__)
product_query = asyncsa.FieldsQuery(
'db.session_async', Product.table)

52
product_query = sa.FieldsQuery(
'db.session', Product.__table__)
product_query = asyncsa.FieldsQuery(
'db.session_async', Product.__table__)

Mobile API average across all queries:
383 ms -> 323 ms 15%
Catalog Graph API average across all queries:
82 ms -> 62 ms 25%
Site Graph Api average across all queries
121 ms -> 108 ms 11%
Async + GraphQL results

54
Example with aliases
def get_price_lists_data():
query = build([
Q.priceLists[
Q.id,
Q.name,
Q.file_id << Q.priceFileId,
Q.date_posted << Q.datePosted,
Q.url << Q.fileUrl,
]
])
graph_data = execute(query)
return graph_data.priceLists

55
Example with aliases
def get_price_lists_data():
query = build([
Q.priceLists[
Q.id,
Q.name,
Q.file_id << Q.priceFileId,
Q.date_posted << Q.datePosted,
Q.url << Q.fileUrl,
]
])
graph_data = execute(query)
return graph_data.priceLists

A few facts of prom.ua graphs
● number of graphs: ~ 20
● number of fields: ~ 2000
● number of links: ~ 300
● number of nodes: ~ 250
● single entry point /graphql
● refactoring and new vision for code
● better monitoring
● easy to test API with graphiql
56
Hiku

57

End of 2018 - now
● Hiku upgrade to mutations
● New 2nd level graph for opinions and
mobile cabinet API
● Read and write with graph
58

mutation_graph = Graph(repr_graph.nodes + [
Root([
Link(
'addNewOpinion',
TypeRef['NewCommentResult'],
add_new_comment_for_opinion,
options=[
Option('opinion_id', Integer),
Option('comment', String),
], requires=None,
),
]),
])

def add_new_comment_for_opinion(ctx, options):
opinion_id = options['opinion_id']
comment = options['comment']
user = ctx['user']
opinion = Opinion.get(opinion_id)
...
form = OpinionCommentForm()
if not form.validate():
return NewCommentResult(id=None, errors=form.validate_resp())
comment = opinion.new_comment(...)
comment.message = clear_text(form.data['comment'])
comment.author_user_id = user.id
db.session.commit()
return NewCommentResult(id=comment.id, errors=[])

65

New client-proposal paradigm
via GraphQL

How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019

Recommended

More Related Content

What's hot (8)

Similar to How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019 (20)

Recently uploaded (20)

How to grow GraphQL and remove SQLAlchemy and REST API from a high-load Python project - Pycon Belarus 2019