SlideShare a Scribd company logo

Implementing Server Side Data Synchronization for Mobile Apps

Michele Orselli
Michele Orselli

The document describes an architecture and implementation for server-side data synchronization for mobile apps. It discusses syncing scenarios, challenges with the existing solution, and the new architecture and implementation. The key aspects covered are using GUIDs for unique identifiers, suggesting a "from" timestamp for incremental syncing, transferring record states instead of operations, and algorithms for resolving conflicts including for hierarchical data using a sort by hierarchy and updating ids.

Software
1 of 75
Downloaded 347 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
Implementing Server Side Data Synchronization for Mobile Apps
Michele Orselli CTO@Ideato ! _orso_ ! micheleorselli ! mo@ideato.it
Agenda scenario design choices implementation alternative approaches
Implementing Server Side Data Synchronization for Mobile Apps
Implementing Server Side Data Synchronization for Mobile Apps
Sync scenario A B C
Sync scenario A B C A B C A B C
Dealing with conflicts A1 A2 ?
Scenario Brownfield project ! several mobile apps for tracking user generated data (calendar, notes, bio data) ! iOS & Android ! ~10 K users steadily growing at 1.2 K/month
Scenario MongoDB ! Legacy App based on codeigniter ! Existing RPC-wannabe-REST API for data sync
Scenario get updates: ! POST /m/<app>/get/<user_id>/<res>/<updated_from> ! ! ! send updates: ! POST /m/<app>/update/<user_id>/<res_id>/<dev_id>/<res> ! !
api
Scenario ! ! 6 different resources, 12 calls per sync ! apps sync by polling every 30 sec ! every call sync little data ! !
Challenge ! ! rebuild sync API for old apps + 2 incoming ! allow image synchronization ! more efficient than previous API ! !
Implementing Server Side Data Synchronization for Mobile Apps
Existing Solutions Tstamps, Vector clocks, CRDTs syncML, syncano Algorithms Protocols/API Azure Data sync Platform couchDB, riak Storage
Not Invented Here? Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels ! J. Atwood
Architecture ! ! 2 different mobile platforms several teams with different skill level ! changing storage wasn’t an option forcing a particular technology client side wasn’t an option
Architecture c1 server c2 c3 sync logic conflicts resolution thin clients
Implementation ! ! In the sync domain all resources are the same ! For every app one endpoint for getting new data one endpoint for pushing changes one endpoint for uploading images
Get changes ! Get all changes (1st sync): ! GET /apps/{app}/users/{user_id}/changes ! Get latest changes: ! GET /apps/{app}/users/{user_id}/changes?from={from}
Get changes ! Get all changes (1st sync): ! GET /apps/{app}/users/{user_id}/changes ! Get latest changes: ! GET /apps/{app}/users/{user_id}/changes?from={from} timestamp?
Server suggest the sync time timestamp are inaccurate (skew and developer errors) ! server suggests the “from” parameter to be used in the next request GET /changes c1 server { ‘next’ : 123456, ‘data’: […] }
Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: […] } c1 server
Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: […] } c1 server GET /changes?from=12345 { ‘next’ : 45678, ‘data’: […] }
what to transfer operations: {‘op’: ’add’, id: ‘1’, ’data’:[…]} {‘op’: ’update’, id: ‘1’, ’data’:[…]} {‘op’: ’delete’, id: ‘1’} {‘op’: ’add’, id: ‘2’, ’data’:[…]} ! ! states: {id: ‘1’, ’data’:[…]} {id: 2’, ’data’:[…]} {id: ‘3’, ’data’:[…]}
what to transfer ! we chosen to transfer states {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true} {id: 2’, ‘type’: ‘note’} {id: ‘3’, ‘type’: ‘note’} ! ! ps: soft delete all the things!
unique identifiers How do we generate an unique id in a distributed system? ! UUID: several implementations (RFC 4122) ! Local Ids/Global Id: server generates GUIDs clients use local ids to manage their records GET /changes c1 server {‘data’:{’guid’: ‘58f0bdd7-1481’}}
unique identifiers POST /merge { ‘data’: [ {’lid’: ‘1’, …}, {‘lid’: ‘2’, …} ] } c1 server { ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, …}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, …} ] }
conflict resolution algorithm (plain data) ! server handles conflicts resolution mobile generated data are “temporary” until sync to server ! conflict resolution: domain indipendent: last-write wins domain dipendent: use domain knowledge to resolve
conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! }
conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! }
conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } no conflict
conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } remote wins
conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } server wins
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server {‘ok’ : { ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
conflict resolution algorithm (hierarchical data) ! How to manage hierarchical data? ! ! { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, … } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
conflict resolution algorithm (hierarchical data) ! How to manage hierarchical data? 1) sync root record 2) update ids 3) sync child records ! ! { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, … } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! parent records first ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! … no conflict
conflict resolution algorithm (hierarchical data) ! ! ! …! ! ! ! ! ! if ($newRootRecord->updated > $s->updated) {! ! ! ! ! ! ! update($s, $newRecord);! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! ! send($newRootRecord);! ! ! ! continue;! ! ! } else {! ! ! ! updateRecordIds($s, $data);! ! ! ! updateRemote($newRecord, $s);! ! ! }! ! ! } else {! ! ! sync($data);! ! }! ! }! remote wins
conflict resolution algorithm (hierarchical data) ! ! ! …! ! ! ! ! ! if ($newRootRecord->updated > $s->updated) {! ! ! ! ! ! ! update($s, $newRecord);! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! ! send($newRootRecord);! ! ! ! continue;! ! ! } else {! ! ! ! updateRecordIds($s, $data);! server wins ! ! ! updateRemote($newRecord, $s);! ! ! }! ! ! } else {! ! ! sync($data);! ! }! ! }!
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } {‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data?
enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data? 1) relax constraints
enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data? 1) relax constraints 2) integrate constraints in sync algorithm
! ! from findByGuid to findSimilar ! first lookup by GUID then by domain rules ! “two measures are similar if are referred to the same date” ! ! ! ! enforcing domain constraints
enforcing domain constraints c1 server
enforcing domain constraints { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server { ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
dealing with binary data ! Binary data uploaded via custom endpoint ! Sync data remain small ! Uploads can be resumed
dealing with binary data ! Two steps* 1) data are synched to server 2) related images are uploaded ! * this means record without file for a given time
dealing with binary data POST /merge { ‘lid’ : 1, ‘type’ : ‘baby’, ‘image’ : ‘myimage.jpg’ } { ‘lid’ : 1, ‘guid’ : ‘ac435-f8345’ } c1 server POST /upload/ac435-f8345/image
What we learned ! Implementing this stuff is tricky ! Explore existing solution if you can ! Understanding the domain is important
vector clocks
CRDT ! Conflict-free Replicated Data Types (CRDTs) ! Constraining the types of operations in order to: - ensure convergence of changes to shared data by uncoordinated, concurrent actors - eliminate network failure modes as a source of error
Math!!! CRDT ! Bounded-join semilattices - join operation defining a least upper bound - partially order set - always increasing
Gateways handles sync Data flows through channels - partition data set - authorization - limit the data ! Use revision trees Couchbase Mobile
Riak Distributed DB Eventually/Strong Consistency ! Data Types ! Configurable conflic resolution - db level for built-in data types - application level for custom data
! That’s all folks! Questions? ! Please leave feedback! https://joind.in/11797 !
Links Vector Clocks http://basho.com/why-vector-clocks-are-easy/ http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks http://basho.com/why-vector-clocks-are-hard/ ! CRDTs http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems. html http://www.infoq.com/presentations/problems-distributed-systems https://www.youtube.com/watch?v=qyVNG7fnubQ ! Riak http://docs.basho.com/riak/latest/dev/using/conflict-resolution/ ! Couchbase Sync Gateway http://docs.couchbase.com/sync-gateway/ http://www.infoq.com/presentations/sync-mobile-data ! API http://developers.amiando.com/index.php/REST_API_DataSync https://login.syncano.com/docs/rest/index.html
Credits phones https://www.flickr.com/photos/15216811@N06/14504964841 wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpg darth http://www.listal.com/viewimage/3825918h blueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpg! building: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpg! brownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpg! no connection: https://www.flickr.com/photos/77018488@N03/9004800239! no internet con https://www.flickr.com/photos/roland/9681237793! vector clocks: http://en.wikipedia.org/wiki/Vector_clock! crdts: http://www.infoq.com/presentations/problems-distributed-systems

More Related Content

What's hot (20)

PDF
JS Fest 2019. Anjana Vakil. Serverless Bebop
JSFestUA
 
PDF
React lecture
Christoffer Noring
 
PPTX
Parse Advanced
Tushar Acharya
 
PDF
My Top 5 APEX JavaScript API's
Roel Hartman
 
PDF
Use Kotlin scripts and Clova SDK to build your Clova extension
LINE Corporation
 
PPTX
Creating sub zero dashboard plugin for apex with google
Roel Hartman
 
PPTX
Rxjs swetugg
Christoffer Noring
 
PPTX
Rxjs ngvikings
Christoffer Noring
 
PPTX
Using Change Streams to Keep Up with Your Data
MongoDB
 
PDF
Integrating React.js with PHP projects
Ignacio Martín
 
PDF
Min-Maxing Software Costs - Laracon EU 2015
Konstantin Kudryashov
 
PDF
Designing The Right Schema To Power Heap (PGConf Silicon Valley 2016)
Dan Robinson
 
PDF
Akka: Actor Design & Communication Technics
Alex Fruzenshtein
 
PDF
Controller specs
Alexander Miller
 
PDF
Goal Based Data Production with Sim Simeonov
Databricks
 
PDF
Inversion Of Control
Chad Hietala
 
PDF
Tweaking the interactive grid
Roel Hartman
 
KEY
Zend framework service
Michelangelo van Dam
 
PPTX
201410 2 fiware-orion-contextbroker
FIWARE
 
PDF
Automation in angular js
Marcin Wosinek
 
JS Fest 2019. Anjana Vakil. Serverless Bebop
JSFestUA
 
React lecture
Christoffer Noring
 
Parse Advanced
Tushar Acharya
 
My Top 5 APEX JavaScript API's
Roel Hartman
 
Use Kotlin scripts and Clova SDK to build your Clova extension
LINE Corporation
 
Creating sub zero dashboard plugin for apex with google
Roel Hartman
 
Rxjs swetugg
Christoffer Noring
 
Rxjs ngvikings
Christoffer Noring
 
Using Change Streams to Keep Up with Your Data
MongoDB
 
Integrating React.js with PHP projects
Ignacio Martín
 
Min-Maxing Software Costs - Laracon EU 2015
Konstantin Kudryashov
 
Designing The Right Schema To Power Heap (PGConf Silicon Valley 2016)
Dan Robinson
 
Akka: Actor Design & Communication Technics
Alex Fruzenshtein
 
Controller specs
Alexander Miller
 
Goal Based Data Production with Sim Simeonov
Databricks
 
Inversion Of Control
Chad Hietala
 
Tweaking the interactive grid
Roel Hartman
 
Zend framework service
Michelangelo van Dam
 
201410 2 fiware-orion-contextbroker
FIWARE
 
Automation in angular js
Marcin Wosinek
 

Viewers also liked (14)

PDF
Data Synchronization Patterns in Mobile Application Design
Eric Maxwell
 
PPTX
AppSync.org: open-source patterns and code for data synchronization in mobile...
Niko Nelissen
 
PPTX
Offline first: application data and synchronization
EatDog
 
PDF
Sync is hard: building offline-first Android apps from the ground up
droidcon Dubai
 
ODP
Offline First Applications
GeekNightHyderabad
 
PDF
FITC presents: Mobile & offline data synchronization in Angular JS
FITC
 
PPS
Doom3
DOINA __COME HERE ONLY IF YOU LIKE THE POWERPOINT PROGRAM, IF YOU LIKE TO DREAM and IF YOU LIKE A GOOD WORK! THIS LINK HAS 270 PRESENTATION EASY TO FIND IN "VIEW ALL"
 
PPT
Shadow Techniques for Real-Time and Interactive Applications
stefan_b
 
PPT
Threading Game Engines: QUAKE 4 & Enemy Territory QUAKE Wars
psteinb
 
PDF
Beyond GPS - Neogeograpy Data Collection
Andrew Turner
 
PPT
Hageman id software project
Adam Hageman
 
PDF
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
FrenchWeb.fr
 
PDF
BSPTreesGameEngines-2
Jason Calvert
 
PDF
BSPTreesGameEngines-1
Jason Calvert
 
Data Synchronization Patterns in Mobile Application Design
Eric Maxwell
 
AppSync.org: open-source patterns and code for data synchronization in mobile...
Niko Nelissen
 
Offline first: application data and synchronization
EatDog
 
Sync is hard: building offline-first Android apps from the ground up
droidcon Dubai
 
Offline First Applications
GeekNightHyderabad
 
FITC presents: Mobile & offline data synchronization in Angular JS
FITC
 
Doom3
DOINA __COME HERE ONLY IF YOU LIKE THE POWERPOINT PROGRAM, IF YOU LIKE TO DREAM and IF YOU LIKE A GOOD WORK! THIS LINK HAS 270 PRESENTATION EASY TO FIND IN "VIEW ALL"
 
Shadow Techniques for Real-Time and Interactive Applications
stefan_b
 
Threading Game Engines: QUAKE 4 & Enemy Territory QUAKE Wars
psteinb
 
Beyond GPS - Neogeograpy Data Collection
Andrew Turner
 
Hageman id software project
Adam Hageman
 
[SEN#7] Le Top 100 des entreprises qui recrutent dans le numérique
FrenchWeb.fr
 
BSPTreesGameEngines-2
Jason Calvert
 
BSPTreesGameEngines-1
Jason Calvert
 
Ad

Similar to Implementing Server Side Data Synchronization for Mobile Apps (20)

PPTX
When the connection fails
Barbara Fusinska
 
PPTX
Rich collaborative data structures for everyone
Bartosz Sypytkowski
 
PDF
Grokking #9: Building a real-time and offline editing service with Couchbase
Oliver N
 
PPTX
Offline db
Ahmed Kamel Taha
 
PPTX
SYNCING IN JAVASCRIPT: MULTI-CLIENT COLLABORATION THROUGH DATA SHARING (Steve...
Future Insights
 
PPTX
Major ppt
Mohit Sandal
 
PPTX
Mobile app master class - Session 2.pptx
Hathimary A
 
PPTX
Application architecture for cloud
Marco Parenzan
 
PDF
Real-time collaboration in distributed systems for JavaScript developers. 
Dawid Urbański
 
KEY
Event Driven Architecture
andreaskallberg
 
PDF
Grokking TechTalk 9 - Building a realtime & offline editing service from scra...
Grokking VN
 
PDF
Event Driven-Architecture from a Scalability perspective
Jonas Bonér
 
PDF
Schema Evolution Patterns - Texas Scalability Summit 2019
Alex Rasmussen
 
PDF
Data synchronization
lmrei
 
PDF
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn
LinkedIn
 
PDF
synquery platform
EastCloud
 
PDF
CTO Leadership Series: Schema Evolution Patterns
BrittanyShear
 
PDF
CTO Leadership Series: Schema Evolution Patterns
Aggregage
 
PDF
The Syncables Framework
Manas Tungare
 
PDF
Storing 16 Bytes at Scale
fabxc
 
When the connection fails
Barbara Fusinska
 
Rich collaborative data structures for everyone
Bartosz Sypytkowski
 
Grokking #9: Building a real-time and offline editing service with Couchbase
Oliver N
 
Offline db
Ahmed Kamel Taha
 
SYNCING IN JAVASCRIPT: MULTI-CLIENT COLLABORATION THROUGH DATA SHARING (Steve...
Future Insights
 
Major ppt
Mohit Sandal
 
Mobile app master class - Session 2.pptx
Hathimary A
 
Application architecture for cloud
Marco Parenzan
 
Real-time collaboration in distributed systems for JavaScript developers. 
Dawid Urbański
 
Event Driven Architecture
andreaskallberg
 
Grokking TechTalk 9 - Building a realtime & offline editing service from scra...
Grokking VN
 
Event Driven-Architecture from a Scalability perspective
Jonas Bonér
 
Schema Evolution Patterns - Texas Scalability Summit 2019
Alex Rasmussen
 
Data synchronization
lmrei
 
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn
LinkedIn
 
synquery platform
EastCloud
 
CTO Leadership Series: Schema Evolution Patterns
BrittanyShear
 
CTO Leadership Series: Schema Evolution Patterns
Aggregage
 
The Syncables Framework
Manas Tungare
 
Storing 16 Bytes at Scale
fabxc
 
Ad

More from Michele Orselli (20)

PDF
Tackling Tech Debt with Rector
Michele Orselli
 
PDF
Comunicare, condividere e mantenere decisioni architetturali nei team di svil...
Michele Orselli
 
PDF
A dive into Symfony 4
Michele Orselli
 
PDF
A recommendation engine for your applications codemotion ams
Michele Orselli
 
PDF
A recommendation engine for your applications phpday
Michele Orselli
 
PDF
Hopping in clouds - phpuk 17
Michele Orselli
 
PDF
A recommendation engine for your php application
Michele Orselli
 
PDF
Symfony e micro (non così tanto) services
Michele Orselli
 
PDF
Hopping in clouds: a tale of migration from one cloud provider to another
Michele Orselli
 
PDF
Vagrant for real (codemotion rome 2016)
Michele Orselli
 
PDF
Vagrant for real codemotion (moar tips! ;-))
Michele Orselli
 
PDF
Migrare a Symfony 3
Michele Orselli
 
PDF
Vagrant for real
Michele Orselli
 
PDF
Continuous, continuous, continuous
Michele Orselli
 
PDF
Deploy a PHP App on Google App Engine
Michele Orselli
 
PDF
Deploy a php app on Google App Engine
Michele Orselli
 
PDF
Sf2 wtf
Michele Orselli
 
PDF
Manage a project portfolio
Michele Orselli
 
PDF
Developing sustainable php projects
Michele Orselli
 
PDF
Zend Framework 2 per chi viene da Symfony2
Michele Orselli
 
Tackling Tech Debt with Rector
Michele Orselli
 
Comunicare, condividere e mantenere decisioni architetturali nei team di svil...
Michele Orselli
 
A dive into Symfony 4
Michele Orselli
 
A recommendation engine for your applications codemotion ams
Michele Orselli
 
A recommendation engine for your applications phpday
Michele Orselli
 
Hopping in clouds - phpuk 17
Michele Orselli
 
A recommendation engine for your php application
Michele Orselli
 
Symfony e micro (non così tanto) services
Michele Orselli
 
Hopping in clouds: a tale of migration from one cloud provider to another
Michele Orselli
 
Vagrant for real (codemotion rome 2016)
Michele Orselli
 
Vagrant for real codemotion (moar tips! ;-))
Michele Orselli
 
Migrare a Symfony 3
Michele Orselli
 
Vagrant for real
Michele Orselli
 
Continuous, continuous, continuous
Michele Orselli
 
Deploy a PHP App on Google App Engine
Michele Orselli
 
Deploy a php app on Google App Engine
Michele Orselli
 
Sf2 wtf
Michele Orselli
 
Manage a project portfolio
Michele Orselli
 
Developing sustainable php projects
Michele Orselli
 
Zend Framework 2 per chi viene da Symfony2
Michele Orselli
 

Recently uploaded (20)

PDF
Alur Perkembangan Software dan Jaringan Komputer
ssuser754303
 
PPTX
Automatic_Iperf_Log_Result_Excel_visual_v2.pptx
Chen-Chih Lee
 
PPTX
ManageIQ - Sprint 264 Review - Slide Deck
ManageIQ
 
PPTX
Android Notifications-A Guide to User-Facing Alerts in Android .pptx
Nabin Dhakal
 
PDF
Laboratory Workflows Digitalized and live in 90 days with Scifeon´s SAPPA P...
info969686
 
PDF
Building scalbale cloud native apps with .NET 8
GillesMathieu10
 
PDF
IObit Uninstaller Pro 14.3.1.8 Crack for Windows Latest
utfefguu
 
PDF
AI Software Development Process, Strategies and Challenges
Net-Craft.com
 
PDF
WholeClear Split vCard Software for Split large vCard file
markwillsonmw004
 
PDF
Writing Maintainable Playwright Tests with Ease
Shubham Joshi
 
PPTX
ERP - FICO Presentation BY BSL BOKARO STEEL LIMITED.pptx
ravisranjan
 
PDF
The Rise of Sustainable Mobile App Solutions by New York Development Firms
ostechnologies16
 
PPTX
CONCEPT OF PROGRAMMING in language .pptx
tamim41
 
PPTX
NeuroStrata: Harnessing Neuro-Symbolic Paradigms for Improved Testability and...
Ivan Ruchkin
 
PPTX
Avast Premium Security crack 25.5.6162 + License Key 2025
HyperPc soft
 
PDF
Automated Test Case Repair Using Language Models
Lionel Briand
 
PDF
Dealing with JSON in the relational world
Andres Almiray
 
PDF
IDM Crack with Internet Download Manager 6.42 Build 41
utfefguu
 
PDF
TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural N...
Lionel Briand
 
PDF
How DeepSeek Beats ChatGPT: Cost Comparison and Key Differences
sumitpurohit810
 
Alur Perkembangan Software dan Jaringan Komputer
ssuser754303
 
Automatic_Iperf_Log_Result_Excel_visual_v2.pptx
Chen-Chih Lee
 
ManageIQ - Sprint 264 Review - Slide Deck
ManageIQ
 
Android Notifications-A Guide to User-Facing Alerts in Android .pptx
Nabin Dhakal
 
Laboratory Workflows Digitalized and live in 90 days with Scifeon´s SAPPA P...
info969686
 
Building scalbale cloud native apps with .NET 8
GillesMathieu10
 
IObit Uninstaller Pro 14.3.1.8 Crack for Windows Latest
utfefguu
 
AI Software Development Process, Strategies and Challenges
Net-Craft.com
 
WholeClear Split vCard Software for Split large vCard file
markwillsonmw004
 
Writing Maintainable Playwright Tests with Ease
Shubham Joshi
 
ERP - FICO Presentation BY BSL BOKARO STEEL LIMITED.pptx
ravisranjan
 
The Rise of Sustainable Mobile App Solutions by New York Development Firms
ostechnologies16
 
CONCEPT OF PROGRAMMING in language .pptx
tamim41
 
NeuroStrata: Harnessing Neuro-Symbolic Paradigms for Improved Testability and...
Ivan Ruchkin
 
Avast Premium Security crack 25.5.6162 + License Key 2025
HyperPc soft
 
Automated Test Case Repair Using Language Models
Lionel Briand
 
Dealing with JSON in the relational world
Andres Almiray
 
IDM Crack with Internet Download Manager 6.42 Build 41
utfefguu
 
TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural N...
Lionel Briand
 
How DeepSeek Beats ChatGPT: Cost Comparison and Key Differences
sumitpurohit810
 

Implementing Server Side Data Synchronization for Mobile Apps

  • 1. Implementing Server Side Data Synchronization for Mobile Apps
  • 2. Michele Orselli CTO@Ideato ! _orso_ ! micheleorselli ! [email protected]
  • 3. Agenda scenario design choices implementation alternative approaches
  • 7. Sync scenario A B C A B C A B C
  • 9. Scenario Brownfield project ! several mobile apps for tracking user generated data (calendar, notes, bio data) ! iOS & Android ! ~10 K users steadily growing at 1.2 K/month
  • 10. Scenario MongoDB ! Legacy App based on codeigniter ! Existing RPC-wannabe-REST API for data sync
  • 11. Scenario get updates: ! POST /m/<app>/get/<user_id>/<res>/<updated_from> ! ! ! send updates: ! POST /m/<app>/update/<user_id>/<res_id>/<dev_id>/<res> ! !
  • 12. api
  • 13. Scenario ! ! 6 different resources, 12 calls per sync ! apps sync by polling every 30 sec ! every call sync little data ! !
  • 14. Challenge ! ! rebuild sync API for old apps + 2 incoming ! allow image synchronization ! more efficient than previous API ! !
  • 16. Existing Solutions Tstamps, Vector clocks, CRDTs syncML, syncano Algorithms Protocols/API Azure Data sync Platform couchDB, riak Storage
  • 17. Not Invented Here? Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels ! J. Atwood
  • 18. Architecture ! ! 2 different mobile platforms several teams with different skill level ! changing storage wasn’t an option forcing a particular technology client side wasn’t an option
  • 19. Architecture c1 server c2 c3 sync logic conflicts resolution thin clients
  • 20. Implementation ! ! In the sync domain all resources are the same ! For every app one endpoint for getting new data one endpoint for pushing changes one endpoint for uploading images
  • 21. Get changes ! Get all changes (1st sync): ! GET /apps/{app}/users/{user_id}/changes ! Get latest changes: ! GET /apps/{app}/users/{user_id}/changes?from={from}
  • 22. Get changes ! Get all changes (1st sync): ! GET /apps/{app}/users/{user_id}/changes ! Get latest changes: ! GET /apps/{app}/users/{user_id}/changes?from={from} timestamp?
  • 23. Server suggest the sync time timestamp are inaccurate (skew and developer errors) ! server suggests the “from” parameter to be used in the next request GET /changes c1 server { ‘next’ : 123456, ‘data’: […] }
  • 24. Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: […] } c1 server
  • 25. Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: […] } c1 server GET /changes?from=12345 { ‘next’ : 45678, ‘data’: […] }
  • 26. what to transfer operations: {‘op’: ’add’, id: ‘1’, ’data’:[…]} {‘op’: ’update’, id: ‘1’, ’data’:[…]} {‘op’: ’delete’, id: ‘1’} {‘op’: ’add’, id: ‘2’, ’data’:[…]} ! ! states: {id: ‘1’, ’data’:[…]} {id: 2’, ’data’:[…]} {id: ‘3’, ’data’:[…]}
  • 27. what to transfer ! we chosen to transfer states {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true} {id: 2’, ‘type’: ‘note’} {id: ‘3’, ‘type’: ‘note’} ! ! ps: soft delete all the things!
  • 28. unique identifiers How do we generate an unique id in a distributed system? ! UUID: several implementations (RFC 4122) ! Local Ids/Global Id: server generates GUIDs clients use local ids to manage their records GET /changes c1 server {‘data’:{’guid’: ‘58f0bdd7-1481’}}
  • 29. unique identifiers POST /merge { ‘data’: [ {’lid’: ‘1’, …}, {‘lid’: ‘2’, …} ] } c1 server { ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, …}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, …} ] }
  • 30. conflict resolution algorithm (plain data) ! server handles conflicts resolution mobile generated data are “temporary” until sync to server ! conflict resolution: domain indipendent: last-write wins domain dipendent: use domain knowledge to resolve
  • 31. conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! }
  • 32. conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! }
  • 33. conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } no conflict
  • 34. conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } remote wins
  • 35. conflict resolution algorithm (plain data) function sync($data) {! ! ! foreach ($data as $newRecord) {! ! ! ! $s = findByGuid($newRecord->getGuid());! ! ! ! ! if (!$s) {! ! ! ! add($newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! ! if ($newRecord->updated > $s->updated) {! ! ! ! update($s, $newRecord);! ! ! ! send($newRecord);! ! ! ! continue;! ! ! }! ! ! ! ! ! updateRemote($newRecord, $s);! } server wins
  • 36. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } server
  • 37. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 38. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 39. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 40. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } c1 server
  • 41. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server {‘ok’ : { ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
  • 42. conflict resolution algorithm (hierarchical data) ! How to manage hierarchical data? ! ! { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, … } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
  • 43. conflict resolution algorithm (hierarchical data) ! How to manage hierarchical data? 1) sync root record 2) update ids 3) sync child records ! ! { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, … } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
  • 44. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
  • 45. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! parent records first ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
  • 46. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! …
  • 47. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) {! ! ! sortByHierarchy($data);! ! ! foreach ($data as $newRootRecord) {! ! ! ! ! ! $s = findByGuid($newRootRecord->getGuid());! ! ! ! ! ! if($newRecord->isRoot()) {! ! ! ! ! if (!$s) {! ! ! ! ! add($newRootRecord);! ! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! send($newRootRecord);! ! ! ! ! continue;! ! ! ! }! ! ! ! ! ! ! ! … no conflict
  • 48. conflict resolution algorithm (hierarchical data) ! ! ! …! ! ! ! ! ! if ($newRootRecord->updated > $s->updated) {! ! ! ! ! ! ! update($s, $newRecord);! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! ! send($newRootRecord);! ! ! ! continue;! ! ! } else {! ! ! ! updateRecordIds($s, $data);! ! ! ! updateRemote($newRecord, $s);! ! ! }! ! ! } else {! ! ! sync($data);! ! }! ! }! remote wins
  • 49. conflict resolution algorithm (hierarchical data) ! ! ! …! ! ! ! ! ! if ($newRootRecord->updated > $s->updated) {! ! ! ! ! ! ! update($s, $newRecord);! ! ! ! updateRecordIds($newRootRecord, $data);! ! ! ! ! ! send($newRootRecord);! ! ! ! continue;! ! ! } else {! ! ! ! updateRecordIds($s, $data);! server wins ! ! ! updateRemote($newRecord, $s);! ! ! }! ! ! } else {! ! ! sync($data);! ! }! ! }!
  • 50. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server
  • 51. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
  • 52. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
  • 53. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } {‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
  • 54. enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data?
  • 55. enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data? 1) relax constraints
  • 56. enforcing domain constraints ! ! e.g. “only one temperature can be registered in a given day” ! how to we enforce domain constraints on data? 1) relax constraints 2) integrate constraints in sync algorithm
  • 57. ! ! from findByGuid to findSimilar ! first lookup by GUID then by domain rules ! “two measures are similar if are referred to the same date” ! ! ! ! enforcing domain constraints
  • 59. enforcing domain constraints { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
  • 60. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
  • 61. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
  • 62. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
  • 63. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server { ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
  • 64. dealing with binary data ! Binary data uploaded via custom endpoint ! Sync data remain small ! Uploads can be resumed
  • 65. dealing with binary data ! Two steps* 1) data are synched to server 2) related images are uploaded ! * this means record without file for a given time
  • 66. dealing with binary data POST /merge { ‘lid’ : 1, ‘type’ : ‘baby’, ‘image’ : ‘myimage.jpg’ } { ‘lid’ : 1, ‘guid’ : ‘ac435-f8345’ } c1 server POST /upload/ac435-f8345/image
  • 67. What we learned ! Implementing this stuff is tricky ! Explore existing solution if you can ! Understanding the domain is important
  • 69. CRDT ! Conflict-free Replicated Data Types (CRDTs) ! Constraining the types of operations in order to: - ensure convergence of changes to shared data by uncoordinated, concurrent actors - eliminate network failure modes as a source of error
  • 70. Math!!! CRDT ! Bounded-join semilattices - join operation defining a least upper bound - partially order set - always increasing
  • 71. Gateways handles sync Data flows through channels - partition data set - authorization - limit the data ! Use revision trees Couchbase Mobile
  • 72. Riak Distributed DB Eventually/Strong Consistency ! Data Types ! Configurable conflic resolution - db level for built-in data types - application level for custom data
  • 73. ! That’s all folks! Questions? ! Please leave feedback! https://joind.in/11797 !
  • 74. Links Vector Clocks http://basho.com/why-vector-clocks-are-easy/ http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks http://basho.com/why-vector-clocks-are-hard/ ! CRDTs http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems. html http://www.infoq.com/presentations/problems-distributed-systems https://www.youtube.com/watch?v=qyVNG7fnubQ ! Riak http://docs.basho.com/riak/latest/dev/using/conflict-resolution/ ! Couchbase Sync Gateway http://docs.couchbase.com/sync-gateway/ http://www.infoq.com/presentations/sync-mobile-data ! API http://developers.amiando.com/index.php/REST_API_DataSync https://login.syncano.com/docs/rest/index.html
  • 75. Credits phones https://www.flickr.com/photos/15216811@N06/14504964841 wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpg darth http://www.listal.com/viewimage/3825918h blueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpg! building: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpg! brownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpg! no connection: https://www.flickr.com/photos/77018488@N03/9004800239! no internet con https://www.flickr.com/photos/roland/9681237793! vector clocks: http://en.wikipedia.org/wiki/Vector_clock! crdts: http://www.infoq.com/presentations/problems-distributed-systems