SlideShare a Scribd company logo
Atomicity in RedisAtomicity in Redis
Thomas Hunter IIThomas Hunter II
RoadmapRoadmap
1.
2.
3.
4.
The Basics
Pipelining
Transactions
Lua
Scripting
The BasicsThe Basics
What is Atomicity?What is Atomicity?
Series of database operations
Guaranteed to all run or none run
Prevents operations from running partially
The effects of all operations are immediately visible
I.e. another client cannot see partial state
Also referred to as a Transaction
How do these tools work?How do these tools work?
Redis is Mostly Single-Threaded
Except for things like Background IO
Node.js is Mostly Single-Threaded
Except for IO and Node v10 Worker Threads
Single Client & Server is simple
Things get complicated with multiple clients
EveryEvery SingleSingle Command is AtomicCommand is Atomic
MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Scenario: Two clients want to increment counter
MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #1 reads value of counter
MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #2 reads value of counter
MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #1 sets value of counter to 1
MultipleMultiple Commands aren't AtomicCommands aren't Atomic
Client #2 sets value of counter to 1
Atomic, Multi-Operation CommandsAtomic, Multi-Operation Commands
Common use-cases have single-command variants
INCR key # GET key ; ~value++ ; SET key ~value
SETNX key value # !EXISTS key ; SET key value
LPUSHX key value # EXISTS key ; LPUSH key value
RPOPLPUSH src dest # RPOP src ; LPUSH dest ~value
GETSET key value # GET key ; SET key value
INCRINCR is an Atomic Incrementis an Atomic Increment
Scenario: Two clients want to increment counter
INCRINCR is an Atomic Incrementis an Atomic Increment
Client #1 atomically increments value of counter
INCRINCR is an Atomic Incrementis an Atomic Increment
Client #2 atomically increments value of counter
PipeliningPipelining
PipeliningPipelining
Ensures commands are run in order per-connection
Sends a batch of commands separated by newlines
Commands are sent in the same message
The Node.js redis module usually does this anyway
Pipelining: Example CodePipelining: Example Code
redis.batch()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
ZRANGEBYSCORE jobs 0 1000rnZREMRANGEBYSCORE jobs 0 1000
Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
It looks atomic
Prevents command interleaving on one connection
A subset of commands can fail
Other client pipelines can interleave commands
Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
const redis = require('redis').createClient();
// run 10 instances of this process in parallel
let keys = [];
for (let i = 0; i < 100000; i++) {
keys.push(i);
}
shuffle(keys);
let pipeline = redis.batch();
for (let key of keys) {
pipeline = pipeline.hsetnx('pipeline', `k${key}`, process.pid);
}
pipeline.exec(() => redis.quit());
Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry
> HGETALL pipeline
...
k46468: 25198
k67664: 25197
k62167: 25197
k5933: 25202
k19146: 25202
k202: 25196
k47418: 25198
k88650: 25202
...
Pipelining: What's it for?Pipelining: What's it for?
Reducing network latency
Send several commands in one message
Receive several responses in one message
echo "PINGrnPINGrnPINGrn" | nc localhost 6379
+PONG
+PONG
+PONG
TransactionsTransactions
MULTI: True AtomicityMULTI: True Atomicity
Atomic regardless of other clients / connections
Client sends MULTI , more commands, EXEC
Other clients can still run commands
Queued commands are run sequentially
Any failures and the entire transaction fails
MULTI: Example CodeMULTI: Example Code
redis.multi()
.zrangebyscore('jobs', 0, now) // get jobs
.zremrangebyscore('jobs', 0, now) // delete jobs
.exec((error, data) => {
let jobList = data[0];
console.log('jobs', jobList); // perform work
});
MULTI
ZRANGEBYSCORE jobs 0 1553099335332
ZREMRANGEBYSCORE jobs 0 1553099335332
EXEC
MULTI Drawback: No command chainingMULTI Drawback: No command chaining
Can't use command result as argument
E.g., cannot pop from list, assign to new key
Lua ScriptingLua Scripting
Lua: The Ultimate in AtomicityLua: The Ultimate in Atomicity
There's a simpler, less-efficient EVAL command
Send the entire script every time
Like sending a normal SQL query
Or use SCRIPT LOAD ahead of time
Then use EVALSHA to run code via resulting hash
Like executing a SQL Stored Procedure
Declare key names as arguments for sharding
Lua: Server-Side LogicLua: Server-Side Logic
Output of one command can be piped into another
Other processing can happen, too
Lua: Game Lobby Example CodeLua: Game Lobby Example Code
-- add-user.lua: add user to lobby, start game if 4 players
local lobby = KEYS[1] -- Set
local game = KEYS[2] -- Hash
local user_id = ARGV[1] -- String
redis.call('SADD', lobby, user_id)
if redis.call('SCARD', lobby) == 4 then
local members = table.concat(redis.call('SMEMBERS',lobby),",")
redis.call('DEL', lobby) -- empty lobby
local game_id = redis.sha1hex(members)
redis.call('HSET', game, game_id, members)
return {game_id, members}
end
return nil
Lua: Game Lobby Example CodeLua: Game Lobby Example Code
const redis = require('redis').createClient();
const rEval = require('util').promisify(redis.eval).bind(redis);
const script = require('fs').readFileSync('./add-user.lua');
const LOBBY = 'lobby-elo-1500', GAME = 'game-hash';
(async () => {
await rEval(script, 2, LOBBY, GAME, 'alice');
await rEval(script, 2, LOBBY, GAME, 'bob');
await rEval(script, 2, LOBBY, GAME, 'cindy');
const [gid,plyrs] = await rEval(script, 2, LOBBY, GAME,'tom');
console.log('GAME ID', gid, 'PLAYERS', plyrs.split(','));
})();
Lua: DrawbacksLua: Drawbacks
Another language to maintain
Simple language, easy syntax
Increases overhead on Redis server
An infinite loop could lock up server
Need to load scripts before using to be efficient
It's idempotent; load scripts when app starts
RecapRecap
Executing singular commands are atomic
Executing multiple commands are not atomic
Pipelining is not atomic, but it's fast
MULTI is atomic, but you can't chain results
Lua scripts are atomic and chainable
Intrinsic: Node.js Security PoliciesIntrinsic: Node.js Security Policies
const REDIS = 'redis://redishost:6379/1';
routes.allRoutes(policy => {
policy.redis.allowConnect(REDIS);
});
routes.get('/users/*', policy => {
policy.redis.allowCommandKey(REDIS, 'GET', 'user-*');
});
routes.post('/server/stats', policy => {
policy.redis.allowInfoSection(REDIS, 'memory');
});
FinFin
Follow me:
About Intrinsic:
This presentation:
@tlhunter
intrinsic.com
bit.ly/redis-atomicity
@tlhunter@tlhunter

More Related Content

What's hot (20)

PDF
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
PPTX
Introduction to Apache ZooKeeper
Saurav Haloi
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
PDF
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
PDF
High Availability PostgreSQL with Zalando Patroni
Zalando Technology
 
PDF
Traffic Control with Envoy Proxy
Mark McBride
 
KEY
Introduction to memcached
Jurriaan Persyn
 
PPT
Introduction to redis
Tanu Siwag
 
PPTX
Introduction to Redis
TO THE NEW | Technology
 
PPTX
Caching solutions with Redis
George Platon
 
PDF
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
SANG WON PARK
 
PDF
Massive service basic
DaeMyung Kang
 
PDF
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
PDF
Introduction to redis - version 2
Dvir Volk
 
PDF
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
PDF
Inside MongoDB: the Internals of an Open-Source Database
Mike Dirolf
 
PDF
Redis cluster
iammutex
 
PDF
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
PPTX
Kafka 101
Clement Demonchy
 
PDF
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Altinity Ltd
 
ClickHouse Data Warehouse 101: The First Billion Rows, by Alexander Zaitsev a...
Altinity Ltd
 
Introduction to Apache ZooKeeper
Saurav Haloi
 
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
High Availability PostgreSQL with Zalando Patroni
Zalando Technology
 
Traffic Control with Envoy Proxy
Mark McBride
 
Introduction to memcached
Jurriaan Persyn
 
Introduction to redis
Tanu Siwag
 
Introduction to Redis
TO THE NEW | Technology
 
Caching solutions with Redis
George Platon
 
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안
SANG WON PARK
 
Massive service basic
DaeMyung Kang
 
MySQL Database Architectures - MySQL InnoDB ClusterSet 2021-11
Kenny Gryp
 
Introduction to redis - version 2
Dvir Volk
 
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
 
Inside MongoDB: the Internals of an Open-Source Database
Mike Dirolf
 
Redis cluster
iammutex
 
ClickHouse Deep Dive, by Aleksei Milovidov
Altinity Ltd
 
Kafka 101
Clement Demonchy
 
Introduction to the Mysteries of ClickHouse Replication, By Robert Hodges and...
Altinity Ltd
 

Similar to Atomicity In Redis: Thomas Hunter (20)

PDF
Tuga IT 2017 - Redis
Nuno Caneco
 
PDF
Redis is not just a cache, Andrew Lavers, ConFoo Montreal 2020
Andrew Lavers
 
PDF
Redis Lua Scripts
Itamar Haber
 
PDF
Everything you always wanted to know about Redis but were afraid to ask
Carlos Abalde
 
PPTX
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
PDF
RedisConf18 - Writing modular & encapsulated Redis code
Redis Labs
 
PPTX
Redis
Rajesh Kumar
 
PDF
An Introduction to Redis for Developers.pdf
Stephen Lorello
 
PPT
Redis And python at pycon_2011
sunilar0ra
 
PDF
Introduction to Redis - LA Hacker News
Michael Parker
 
PPT
Python redis talk
Josiah Carlson
 
PDF
A Brief Introduction to Redis
Charles Anderson
 
PPTX
Get more than a cache back! - ConFoo Montreal
Maarten Balliauw
 
PDF
Fun with Ruby and Redis
javier ramirez
 
PDF
quickguide-einnovator-9-redis
jorgesimao71
 
KEY
Redis - N✮SQL Berlin
mattmatt
 
PDF
quickguide-einnovator-10-redis-admin
jorgesimao71
 
PDF
Redispresentation apac2012
Ankur Gupta
 
PDF
Introduction to Redis
Saeid Zebardast
 
PDF
mar07-redis.pdf
AnisSalhi3
 
Tuga IT 2017 - Redis
Nuno Caneco
 
Redis is not just a cache, Andrew Lavers, ConFoo Montreal 2020
Andrew Lavers
 
Redis Lua Scripts
Itamar Haber
 
Everything you always wanted to know about Redis but were afraid to ask
Carlos Abalde
 
Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...
Redis Labs
 
RedisConf18 - Writing modular & encapsulated Redis code
Redis Labs
 
An Introduction to Redis for Developers.pdf
Stephen Lorello
 
Redis And python at pycon_2011
sunilar0ra
 
Introduction to Redis - LA Hacker News
Michael Parker
 
Python redis talk
Josiah Carlson
 
A Brief Introduction to Redis
Charles Anderson
 
Get more than a cache back! - ConFoo Montreal
Maarten Balliauw
 
Fun with Ruby and Redis
javier ramirez
 
quickguide-einnovator-9-redis
jorgesimao71
 
Redis - N✮SQL Berlin
mattmatt
 
quickguide-einnovator-10-redis-admin
jorgesimao71
 
Redispresentation apac2012
Ankur Gupta
 
Introduction to Redis
Saeid Zebardast
 
mar07-redis.pdf
AnisSalhi3
 
Ad

More from Redis Labs (20)

PPTX
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
PPTX
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
PPTX
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
PPTX
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
PPTX
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
PPTX
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
PPTX
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
PPTX
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
PPTX
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
PPTX
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
PPTX
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
PPTX
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
PPTX
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
PPTX
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
PPTX
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
PDF
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
PPTX
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
PPTX
Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...
Redis Labs
 
Redis Day Bangalore 2020 - Session state caching with redis
Redis Labs
 
Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020
Redis Labs
 
The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...
Redis Labs
 
SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020
Redis Labs
 
Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...
Redis Labs
 
Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle
Redis Labs
 
Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020
Redis Labs
 
Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020
Redis Labs
 
Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...
Redis Labs
 
JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...
Redis Labs
 
Highly Available Persistent Session Management Service by Mohamed Elmergawi o...
Redis Labs
 
Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...
Redis Labs
 
RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020
Redis Labs
 
RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020
Redis Labs
 
Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...
Redis Labs
 
Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...
Redis Labs
 
Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...
Redis Labs
 
Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...
Redis Labs
 
Ad

Recently uploaded (20)

PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PDF
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
PDF
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
PPTX
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PDF
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
PDF
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
PDF
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
PDF
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PDF
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
PDF
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
PDF
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
Log-Based Anomaly Detection: Enhancing System Reliability with Machine Learning
Mohammed BEKKOUCHE
 
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
"Autonomy of LLM Agents: Current State and Future Prospects", Oles` Petriv
Fwdays
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
How Startups Are Growing Faster with App Developers in Australia.pdf
India App Developer
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Empower Inclusion Through Accessible Java Applications
Ana-Maria Mihalceanu
 
Using FME to Develop Self-Service CAD Applications for a Major UK Police Force
Safe Software
 
HCIP-Data Center Facility Deployment V2.0 Training Material (Without Remarks ...
mcastillo49
 
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
"AI Transformation: Directions and Challenges", Pavlo Shaternik
Fwdays
 
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 

Atomicity In Redis: Thomas Hunter

  • 1. Atomicity in RedisAtomicity in Redis Thomas Hunter IIThomas Hunter II
  • 4. What is Atomicity?What is Atomicity? Series of database operations Guaranteed to all run or none run Prevents operations from running partially The effects of all operations are immediately visible I.e. another client cannot see partial state Also referred to as a Transaction
  • 5. How do these tools work?How do these tools work? Redis is Mostly Single-Threaded Except for things like Background IO Node.js is Mostly Single-Threaded Except for IO and Node v10 Worker Threads Single Client & Server is simple Things get complicated with multiple clients
  • 6. EveryEvery SingleSingle Command is AtomicCommand is Atomic
  • 7. MultipleMultiple Commands aren't AtomicCommands aren't Atomic Scenario: Two clients want to increment counter
  • 8. MultipleMultiple Commands aren't AtomicCommands aren't Atomic Client #1 reads value of counter
  • 9. MultipleMultiple Commands aren't AtomicCommands aren't Atomic Client #2 reads value of counter
  • 10. MultipleMultiple Commands aren't AtomicCommands aren't Atomic Client #1 sets value of counter to 1
  • 11. MultipleMultiple Commands aren't AtomicCommands aren't Atomic Client #2 sets value of counter to 1
  • 12. Atomic, Multi-Operation CommandsAtomic, Multi-Operation Commands Common use-cases have single-command variants INCR key # GET key ; ~value++ ; SET key ~value SETNX key value # !EXISTS key ; SET key value LPUSHX key value # EXISTS key ; LPUSH key value RPOPLPUSH src dest # RPOP src ; LPUSH dest ~value GETSET key value # GET key ; SET key value
  • 13. INCRINCR is an Atomic Incrementis an Atomic Increment Scenario: Two clients want to increment counter
  • 14. INCRINCR is an Atomic Incrementis an Atomic Increment Client #1 atomically increments value of counter
  • 15. INCRINCR is an Atomic Incrementis an Atomic Increment Client #2 atomically increments value of counter
  • 17. PipeliningPipelining Ensures commands are run in order per-connection Sends a batch of commands separated by newlines Commands are sent in the same message The Node.js redis module usually does this anyway
  • 18. Pipelining: Example CodePipelining: Example Code redis.batch() .zrangebyscore('jobs', 0, now) // get jobs .zremrangebyscore('jobs', 0, now) // delete jobs .exec((error, data) => { let jobList = data[0]; console.log('jobs', jobList); // perform work }); ZRANGEBYSCORE jobs 0 1000rnZREMRANGEBYSCORE jobs 0 1000
  • 19. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry It looks atomic Prevents command interleaving on one connection A subset of commands can fail Other client pipelines can interleave commands
  • 20. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry const redis = require('redis').createClient(); // run 10 instances of this process in parallel let keys = []; for (let i = 0; i < 100000; i++) { keys.push(i); } shuffle(keys); let pipeline = redis.batch(); for (let key of keys) { pipeline = pipeline.hsetnx('pipeline', `k${key}`, process.pid); } pipeline.exec(() => redis.quit());
  • 21. Pipelining: Not Atomic, SorryPipelining: Not Atomic, Sorry > HGETALL pipeline ... k46468: 25198 k67664: 25197 k62167: 25197 k5933: 25202 k19146: 25202 k202: 25196 k47418: 25198 k88650: 25202 ...
  • 22. Pipelining: What's it for?Pipelining: What's it for? Reducing network latency Send several commands in one message Receive several responses in one message echo "PINGrnPINGrnPINGrn" | nc localhost 6379 +PONG +PONG +PONG
  • 24. MULTI: True AtomicityMULTI: True Atomicity Atomic regardless of other clients / connections Client sends MULTI , more commands, EXEC Other clients can still run commands Queued commands are run sequentially Any failures and the entire transaction fails
  • 25. MULTI: Example CodeMULTI: Example Code redis.multi() .zrangebyscore('jobs', 0, now) // get jobs .zremrangebyscore('jobs', 0, now) // delete jobs .exec((error, data) => { let jobList = data[0]; console.log('jobs', jobList); // perform work }); MULTI ZRANGEBYSCORE jobs 0 1553099335332 ZREMRANGEBYSCORE jobs 0 1553099335332 EXEC
  • 26. MULTI Drawback: No command chainingMULTI Drawback: No command chaining Can't use command result as argument E.g., cannot pop from list, assign to new key
  • 28. Lua: The Ultimate in AtomicityLua: The Ultimate in Atomicity There's a simpler, less-efficient EVAL command Send the entire script every time Like sending a normal SQL query Or use SCRIPT LOAD ahead of time Then use EVALSHA to run code via resulting hash Like executing a SQL Stored Procedure Declare key names as arguments for sharding
  • 29. Lua: Server-Side LogicLua: Server-Side Logic Output of one command can be piped into another Other processing can happen, too
  • 30. Lua: Game Lobby Example CodeLua: Game Lobby Example Code -- add-user.lua: add user to lobby, start game if 4 players local lobby = KEYS[1] -- Set local game = KEYS[2] -- Hash local user_id = ARGV[1] -- String redis.call('SADD', lobby, user_id) if redis.call('SCARD', lobby) == 4 then local members = table.concat(redis.call('SMEMBERS',lobby),",") redis.call('DEL', lobby) -- empty lobby local game_id = redis.sha1hex(members) redis.call('HSET', game, game_id, members) return {game_id, members} end return nil
  • 31. Lua: Game Lobby Example CodeLua: Game Lobby Example Code const redis = require('redis').createClient(); const rEval = require('util').promisify(redis.eval).bind(redis); const script = require('fs').readFileSync('./add-user.lua'); const LOBBY = 'lobby-elo-1500', GAME = 'game-hash'; (async () => { await rEval(script, 2, LOBBY, GAME, 'alice'); await rEval(script, 2, LOBBY, GAME, 'bob'); await rEval(script, 2, LOBBY, GAME, 'cindy'); const [gid,plyrs] = await rEval(script, 2, LOBBY, GAME,'tom'); console.log('GAME ID', gid, 'PLAYERS', plyrs.split(',')); })();
  • 32. Lua: DrawbacksLua: Drawbacks Another language to maintain Simple language, easy syntax Increases overhead on Redis server An infinite loop could lock up server Need to load scripts before using to be efficient It's idempotent; load scripts when app starts
  • 33. RecapRecap Executing singular commands are atomic Executing multiple commands are not atomic Pipelining is not atomic, but it's fast MULTI is atomic, but you can't chain results Lua scripts are atomic and chainable
  • 34. Intrinsic: Node.js Security PoliciesIntrinsic: Node.js Security Policies const REDIS = 'redis://redishost:6379/1'; routes.allRoutes(policy => { policy.redis.allowConnect(REDIS); }); routes.get('/users/*', policy => { policy.redis.allowCommandKey(REDIS, 'GET', 'user-*'); }); routes.post('/server/stats', policy => { policy.redis.allowInfoSection(REDIS, 'memory'); });
  • 35. FinFin Follow me: About Intrinsic: This presentation: @tlhunter intrinsic.com bit.ly/redis-atomicity @tlhunter@tlhunter