ГАННА КАПЛУН «noSQL vs SQL: порівняння використання реляційних та нереляційних СУБД в комерційних програмах» Kyiv QADay 2021

noSQL vs SQL
Comparative usage of noSQL and SQL DBMS in enterprise applications

Contents
noSQL vs SQL
MongoDB in production
MongoDB + Oracle in production
Measuring performance
Querying: examples

Intro
A few words about me
• My name is Hanna Kaplun, I’m a QA Lead at Intellias
• Got more than 10 years of experience in testing
• and a few as a support specialist, technical writer, Oracle DB
developer, BA
• Here’s my LinkedIn
• Yes, testing is my conscious choice 
• Held a few trainings, meetups etc. for various audiences
• Wrote a number of articles about testing
• and data science using R
• Got myself a husband and two kids, 6 yrs and 1 yr
correspondingly 

SQL vs noSQL: main differences
• They are relational, i.e. they can be normalized to store the data
efficiently taking less space
• There is a pre-defined schema, the structure where the data is
stored (DDL portion of SQL)
• Data structures to store data are tables with fixed columns
• Vertical scaling (buy a larger server)
• Multi-record transactions are supported
SQL noSQL
• They are non-relational, therefore no storage optimization
• The schema is dynamic as the data are unstructured
• Data structures may vary
– Document (JSON, i.e. trees)
– Key-value
– Wide-column (tables with dynamic columns)
– Graph (nodes and edges, dict of lists as a rule)
• Horizontal scaling (buy more servers)
• Multi-record transactions are usually not supported

SQL vs noSQL: advantages and disadvantages
• Advantages
– Normalization is available to optimize data
storage
– Transactions to support ACID
– ANY data can be stored this way
• Disadvantages
– Joins and lots of complicated query stuff
– May be slow
– Additional interfaces to communicate with
DBMS from a programming language
SQL noSQL
• Advantages
– Fast
– Easy to understand and query as the data are
stored naturally and dynamically, no need to
restructure them
– Easy to query from a programming language
• Disadvantages
– Possible data duplication
– No transactions, so implementing ACID
principles may be a challenge (usually it is done
via a programming language, not via a DBMS)

SQL vs noSQL: DBMS examples
• Oracle
• MySQL
• MS SQL Server
• PostgreSQL
• SQLite
SQL noSQL
• Document-based
– MongoDB
– CouchDB
• Key-value based
– DynamoDB
– Redis
• Wide-column based:
– Cassandra
– HBase
• Graph based:
– Neo4j
– Amazon
Neptune

The application
• The general idea of the application was to sign the documents with electronic
signatures and store the signature and the information about the document
but not the document itself
• The electronic signatures have two- or three-level of verification, depending on
the authority issuing the certificate to use for this purpose
• The meta-data about the document may or may not contain some information
though there are the mandatory fields like document title, size, format
• The non-mandatory fields may be the following: category, date of verification,
author information (name, surname, the company, job title etc.)

The application: conclusions
• The data to be stored has tree structure (the certificate issuing authorities,
documents and their authors)
• The data is dynamic (some of the values may or may not be present, as well as
the nodes in the tree)
• Initial DBMS proposal: Oracle (the team is familiar with it; any data can be
stored in SQL format)
• Alternative DBMS proposal: MongoDB (dynamic schema, tree-like data, JSONs
as a format to store it in by default)

The application: DBMS choice
• Advantages
– The team knows it, we have a DBA and Oracle DB devs on the
team
– Everything can be stored in SQL, no matter what the new data
we need to store, it will be possible
• Disadvantages
– The DML queries became complicated almost from scratch
when the DB schema was normalized (to BCNF – Boyce-Codd
normal form, 3.5NF), it appeared that we needed to write e.g.
recursive queries to fetch the info we needed from tree-like
data
– Vertical scaling was not available while horizontal was
(customer’s request)
Oracle MongoDB
• Advantages
– Backend devs could just use it, no need for a DBA or a
separate DB dev team
– Querying appeared to be easy and fast
– We could use MongoDB sharding for horizontal scaling
• Disadvantages
– Significant data duplication, so sharding has been on demand
almost immediately after we deployed to production
– Further performance optimization was either exponential or
index-based

MongoDB + Oracle in
production

The application: extension
• The application we have deployed in previous section had to be expanded: we
were now supposed to store the documents themselves and monitor user
activity so that the potential data fraud or data loss or other types of attacks
focused on the data could be detected
• Therefore, it was necessary to store the documents in addition to their
metadata (quite a challenge as the formats could vary from .txt to .jpg or .pdf)
• We also needed to add extensive logging and add a complicated real-time log
analysis system to detect the possible threats and inform the relevant
stakeholders about them

The application extension: conclusions
• There is no need to query the documents, just to store them and make sure
they are intact
• The logs and the relevant business logic raise a few challenges
– The number of records is increasing alarmingly fast as well as the file size to store these along with the increasing
number of users (approximately 1 000 000 records per day)
– The log records are to be analyzed in the real time, so performance of this business logic implementation is of
essence
• The alerting mechanism should be fast not only from performance point of
view but also reach the responsible person as quickly as possible

The application extension: solutions
• We decided to store the documents in Oracle (allowing full-functional
transactions, batch processing and regular data backups)
• The main data architecture challenge of storing the data in relational DBMS and
its meta-data in non-relational DBMS is to ensure the proper correspondence
between the two (we wrote a correspondence layer for that purpose in a
backend programming language)
• The logs were stored in MongoDB despite their pre-defined structure (log
message type, log message, date and time) – the reason for that were relatively
simple queries including date-time and message type and further text analysis
implemented as part of the backend source code (i.e. the performance was of
essence and good enough for the project needs)

Measuring performance: a few hints
• In Oracle something like profiling can be used,
which was the case for us – a few queries of
interest were selected, and we have measured
the time it took to get the view with the results
on different DB states (empty, average data
amount, maximum foreseeable data amount)
• Useful links
– Measuring DB performance for Oracle
– Oracle DB performance tuning guide
Oracle MongoDB
• MongoDB has a DB profiler and a diagnostic log.
It has explain() method to see the indexes
applied and the execution stats for the query of
interest (which is exactly what we have used)
• Useful links
– Evaluating operations
– Performance in MongoDB
– Monitoring

Querying: the differences
• INSERT INTO users (user_id, age, status)
VALUES ('bcd001', 45, 'A’)
• SELECT * FROM users
• UPDATE users SET status = 'C' WHERE age
> 25
• db.start_transaction()
cursor.execute(orderInsert, orderData)
cursor.execute(stockUpdate, stockData)
db.commit()
Oracle MongoDB
• db.users.insert({ user_id: 'bcd001', age: 45,
status: 'A’ })
• db.users.find()
• db.users.update( { age: { $gt: 25 } }, { $set:
{ status: 'C' } }, { multi: true } )
• s.starttransaction() orders.insertone(order,
session=s) stock.updateone(item,
stockUpdate, session=s)
s.committransaction()

Querying Stackoverflow: the schema
• StackExchange Data Explorer

Thinking JSON: Post to User
{"id": 3048642,
"creationDate": "2010-06-15 20:07:34",
"Body": "<p>You can try something like that :</p><..."}
...,
"Author": {"id": 12164792,
"creationDate": "2019-10-04 13:40:14",
"DisplayName": "q8cworu271",
...}
}

Thinking JSON: User to Post
{"id": 12164792,
"creationDate": "2019-10-04 13:40:14",
"DisplayName": "q8cworu271",
...,
"posts": [{"id": 3048642,
"creationDate": "2010-06-15 20:07:34",
"Body": "<p>You can try something like that :.
…},
...]
}

Querying Stackoverflow: the examples
SQL MongoDB
• select top 10 * from posts
• select * from users where DisplayName =
'q8cworu271’
• select * from users where DisplayName =
'q8cworu271' AND Reputation = 1
• select top 10 * from users order by
CreationDate
• db.posts.find().limit(10)
• db.users.find( { “DisplayName":
"q8cworu271" } )
• db.users.find( { $and: [ {“DisplayName":
"q8cworu271"}, { “Reputation”: 1 } ] } )
• db.users.find().sort( { “CreationDate": 1 }
).limit(10)

ГАННА КАПЛУН «noSQL vs SQL: порівняння використання реляційних та нереляційних СУБД в комерційних програмах» Kyiv QADay 2021

Recommended

More Related Content

What's hot (20)

Similar to ГАННА КАПЛУН «noSQL vs SQL: порівняння використання реляційних та нереляційних СУБД в комерційних програмах» Kyiv QADay 2021 (20)

More from QADay (20)

Recently uploaded (20)

ГАННА КАПЛУН «noSQL vs SQL: порівняння використання реляційних та нереляційних СУБД в комерційних програмах» Kyiv QADay 2021