OWD - Push Notification Server Architecture [DEVCON1_2012]
20 likes7,858 views
The document discusses the architecture and development of a push notification server by Telefonica Digital, aimed at creating a mobile network-friendly, easy-to-use, and scalable system. Key technologies highlighted include Node.js for server-side operations, MongoDB for data storage, and RabbitMQ for messaging. The proposed solution prioritizes reducing network resource usage and battery consumption by leveraging an innovative notification mechanism without the need for persistent TCP connections.
OWD - Push Notification Server Architecture [DEVCON1_2012]
- 1. Push Notification Server Architecture talk Telefonica Digital 28th November 2012
- 3. 01 Push Notification Server Who are we? • Fernando Rodríguez Sela [email protected] @mangiacaprini • Guillermo López Leal [email protected] @willyaranda OWD 3 Telefonica Digital
- 4. 01 Push Notification Server The project • Objective: Develop a PUSH Notification Server Mobile network friendly No developer registration Easy to use Scalable Based on web technologies http://www.lacofa.es/index.php/general/notification-server https://github.com/telefonicaid/notification_server OWD 4 Telefonica Digital
- 5. 02 Technologies used OWD 5 Telefonica Digital
- 6. 02 Node.JS Javascript on the server … awesome ! • Server side platform based on the V8 JS engine (Chrome) • Event-driven based • Non-blocking I/O model • Very fast and low fingerprint • High performance with heavy load • http://nodejs.org OWD 6 Telefonica Digital
- 7. 02 MongoDB NO-SQL database • Document oriented storage • JSON style documents (BSON) • High performance • Scalability (replication & autosharding) • http://mongodb.org OWD 7 Telefonica Digital
- 8. 02 RabbitMQ Asynchronous messaging queue system • Robust and fast messaging • Small software 12.000 lines of erlang code Erlang is highly concurrent and used for real time HA software • Multiple delivery schemes. We use publish/subscribe with round-robin ActiveMQ don't support round-robin out of the box • High performance Outstands ActiveMQ performance in persistent mode • Scalability (Clustering mode) Far better than ActiveMQ options (master-slave) • http://rabbitmq.com OWD 8 Telefonica Digital
- 9. 03 Mobile network issues OWD 9 Telefonica Digital
- 10. 03 The problem Current problem – a bit of history • Designed without considering the mobile network • Push servers can't connect directly to the device Long polling Keep alives • Consequences Signalling storms Draining user's batteries Low QoS • Network stress • Real case: WhatsApp & Telefonica november 2012 OWD 10 Telefonica Digital
- 11. 03 The problem Current problem – Radio states & Why signalling storms RRC Idle • Battery consume 1 relative unit Cell_PCH • Battery consume <2 relative units URA_PCH • Battery consume ≤ Cell_PCH Cell_FACH • Battery consume 40 relative units Cell_DCH • Battery consume 100 relative units Each keep-alive message force move the handset from the IDLE state to a shared channel OWD 11 Telefonica Digital
- 12. 04 State of the art OWD 12 Telefonica Digital
- 13. 04 Current Situation Cellular PS network App Server 1 Internet 1 Polling App 2 App Server N OS Two different behaviours: Polling: 1. Polling applications check periodically if there is new information in their App Servers. 2. Most of the requests are not delivering data, creating network congestion: both RRC and TCP connections open/close in core. Keep-alives: 1. The application keeps an permanently open TCP connection with their server, by sending keep-alives. 2. The server can reach the application trough this TCP connection. OWD 13 Telefonica Digital
- 14. 04 Current Situation OWD 14 Telefonica Digital
- 15. 04 Current Situation OWD 15 Telefonica Digital
- 16. 04 Solutions: WAP Push Cellular PS network Application server Internet Encoded WAP push (SMS) WAP Push to e.g. WAP Push Proxy +44802123456 Gateway @ppg.genie.co.uk WAP Push Initiator Server Cellular CS network • The push message starts at the WAP Push Initiator and it is sent to the Push Proxy Gateway specified in the destination address of the push message. All addresses are relative to the WAP push proxy gateway (PPG). The PPG sends the SMS messages through CS to the phone identified using its MSISDN. Then, the Mobile opens the PS connection. • Two variants: • Service Indication: On receiving a WAP Push the handset will automatically give the user the option to access the WAP content. • Service Load: Directly open de browser to display the WAP content without user interaction. OWD 16 Telefonica Digital
- 17. 04 Solutions: Apple Push Notification Service (APNS) & Google Cloud Messaging (GCM) Application Server Cellular PS network 5 Internet 1 3 2 NS Server 4 6 1. The application registers for push notifications. The OS asks NS Server for a token. 2. The application receives the device token. 3. The app sends the token to the Application Server. 4. The mobile keeps 1 single permanently open TCP connection with the NS server, by sending keep-alives. 5. The Application Server sends the push notifications to the NS Server with their token. 6. The NS sends the push notification to the app through the open TCP connection. OWD 17 Telefonica Digital
- 18. 05 Our solution OWD 18 Telefonica Digital
- 19. OWD 19 Telefonica Digital
- 20. 05 Firefox OS Push Notification Service Application Server Cellular PS network Internet 1 4 3 2 Push Notification Server 5 http://push.telefonica.es 1. The mobile application registers for push notifications in the Push Notification Server. The Push Notification Server assigns a token to the device or application, and stores the private IP of the mobile associated to that token. 2. The mobile app receives the “URL to notify me” containing the Push Notification Server address and the token (e.g.: http://push.telefonica.es/token=123456789) 3. The mobile app sends the “URL to notify me” to its Application Server. 4. The Application Server sends the push notifications to that URL (the Push Notification Server with their token). 5. The Push Notification Server sends the push notification to the mobile app. As there is not an open TCP connection (but the device has a private IP), the network will page the mobile to “find it” and forward the packet to it. OWD 20 Telefonica Digital
- 21. 05 Firefox OS Push Notification Service • No need to keep any open TCP connection. => No keep-alives • Open and standard solution •W3C, OMA •Other interested carriers • No registration needed • No quotas • No fees • Secure OWD 21 Telefonica Digital
- 22. 05 The problem Proposed solution • Objective: Avoid Keep-Alive and open connections Reduce required network resources Reduce battery consume (increasing the radio IDLE status) • How to achieve it? Notification server should have “direct vision” with the handset • Avoid maintaining open sockets – WakeUps sent via WAP-PUSH, UDP or TCP packets • The device listens for those notifications – Then, retrieve the notification OWD 22 Telefonica Digital
- 23. 05 Interested carriers We are not alone OWD 23 Telefonica Digital
- 24. 06 Proposed protocol OWD 24 Telefonica Digital
- 25. 06 Proposed protocol Actors • WebApp (WA) – Web • Notification Server (NS) – Application which receives Server dedicated to manage asynchronous notifications all PUSH notifications and responsible to deliver to the correct UA/WA. • User Agent (UA) – Web • AppServer (AS) – Application rendering engine which offers Server – This is the server a simple API to the WA to deployed by the WA receive notifications. Also the developer UA shall maintain a communication channel with the notification server. OWD 25 Telefonica Digital
- 26. 06 Proposed protocol Boxes OWD 26 Telefonica Digital
- 27. 06 Proposed protocol APIs • (1) API defined by W3C. Used • (3) Private Protocol designed by the application to register it by the application developer. into the NS and receives Should be used to inform the notifications over the AS about the publicURL given publicURL given by the NS to the WA by the NS (2) API used by the UA to • (4) REST API used to sent register a communication notifications from the AS to channel with the NS in order the WA to receives notifications. This channel should be different based on the situation: WiFi or external networks (WebSocket) Same TCP Network (UDP) ... OWD 27 Telefonica Digital
- 28. 06 Proposed protocol Tokens and how we identify each actor • NS uses tokens to identify UAs and WAs UAToken → Cryptographic token (AES) WAToken → Identify an application installation Public Key (PBK) → Identify each application & used for signing AppToken → SHA256(WAToken + PBK) OWD 28 Telefonica Digital
- 29. 06 Proposed protocol Kind of deliveries • To one user and one device The WAToken given is unique and secret • To one user and multiple devices The WAToken given is shared between user devices and secret • To all the users of concrete application The same WAToken is used in all the instances (or group of them) Obviously, not a secret OWD 29 Telefonica Digital
- 30. 06 Proposed protocol How it works 1) Get a valid UAToken through any secure mechanism 2) On UA starts 1) Open a WebSocket with the configured NS 2) Register itself sending his UAToken and some network parameters 3) Register applications by sending WAToken + PBK 1) Receive a notification URL for each registered application 4) Each app. Send his own notification URL to their AS OWD 30 Telefonica Digital
- 31. 06 Proposed protocol How it works 1) AS POST a notification to the public notification URL 2) The Notification server: 1) Verify signature 2) Delivery: 1) If open WebSocket, just send the notification to the recipients 2) Else: 1) If UDP: send a wakeup packet 1) The UA connects to the NS and retrieves the notification. OWD 31 Telefonica Digital
- 32. 07 Our architecture OWD 32 Telefonica Digital
- 33. 07 Architecture Refreshing diagram OWD 33 Telefonica Digital
- 34. 07 Internal Architecture Multiple Servers Message Queue User Agent Web Socket Application Server User Agent UDP NO-SQL MongoDB OWD 34 Telefonica Digital
- 35. 07 Inside NS Registration Message Queue 1 User Agent Web Socket Application 2 Server User Agent UDP NO-SQL MongoDB OWD 35 Telefonica Digital
- 36. 07 Inside NS Send a notification Message 4 Queue 6 User Agent 4 3 Web Socket Application 1 5 Server User Agent UDP 2 5 NO-SQL MongoDB OWD 36 Telefonica Digital
- 37. 08 Security OWD 37 Telefonica Digital
- 38. 08 Security & Privacy PvK / PbK – Signed • Check signature for each notification Avoid malicious senders Verify the origin Private Key (PvK) on AS / Public Key (PbK) on WA On the WA registration process, WA provides the WAToken and a PbK OWD 38 Telefonica Digital
- 39. 08 Security & Privacy publicURL • A publicURL ↔ WAToken & PbK PublicURL = https://server.domain.com/notify/APPToken OWD 39 Telefonica Digital
- 40. 08 Security & Privacy Possible attacks • An evil AS wants to send notifications Need to know the Private Key • An evil WA wants to receive notifications from another WA Need to know the WAToken which SHALL be a secret • An evil device wants to register as another one Need to know the UAToken (managed by the OS) • Try DoS, flood messages, … → abuse controls on server side OWD 40 Telefonica Digital
- 41. 03 Security & Privacy Security always should be improved ! We're open to suggestions OWD 41 Telefonica Digital
- 42. 09 Advantages for developers OWD 42 Telefonica Digital
- 43. 09 We need to give some sweets to the developers The way to spread the use OWD 43 Telefonica Digital
- 44. 09 We need to give some sweets to the developers The way to spread the use • Easy to use API Based on Web technologies && standarization in progress • Reduce developer deployment costs • More efficient use of networks && battery • No registration process needed and no subscriptions • Bigger payloads and more messages per app OWD 44 Telefonica Digital
- 45. 10 Issues found during Deployment OWD 45 Telefonica Digital
- 46. 10 Issues found Theory vs practice • RabbitMQ Use persistence only when you need to • We prefer using MongoDB Chose carefully your configuration (publish-subscribe, round robin, priority) • MongoDB: Use sharding and think for a good sharding key Delete your SQL knowledge and start from zero • Self-contained documents • Node.JS: collapses (100% CPU) when reaches ulimit • A lot of connections == a lot of file descriptors • Raise ulimit and file descriptors (inodes) make tests (Travis-ci) Explore variations for your algorithms • Delete vs de-reference OWD 46 Telefonica Digital
- 47. 11 Standarization works OWD 47 Telefonica Digital
- 48. 11 Standarization Convert it in the standard solution for pushing • W3C interested • OMA interested • A lot of carriers want to push this solution OWD 48 Telefonica Digital
- 49. 11 Standarization Convert it in the standard solution for pushing • W3C WebApps Working Group: Specifying Push device API. Telefónica co- editing the Push API draft specification together with AT&T. Recently progressed to FWPD (First Public working Draft), meaning it has been accepted as a working group draft specification • OMA AOI (Always online Infrastructure): Defining a Push framework, with same objectives as the Firefox OS one. Telefonica has successfully contributed to align OMA AOI requirements with Firefox OS push framework. Recently started the discussion on the best architecture to fulfill the requirements, to which Telefonica is also contributing. OWD 49 Telefonica Digital
- 50. 11 Proposed protocol Standarization http://dvcs.w3.org/hg/push/raw-file/default/index.html OWD 50 Telefonica Digital
- 51. 12 Next steps What for V2? OWD 51 Telefonica Digital
- 52. 12 The future (IPv6) App Server N Internet Every UE will have a public IP, but the operator’s firewall will block connections from the applications servers. It will be necessary to have rules in the firewall or proxy to accept some incoming connections. That rules should keep in mind what applications the UE has. ¿Should present solution be compatible with IPv6? OWD 52 Telefonica Digital
- 53. 12 Some improvement ideas • Support priority Enqueue low priority messages • TTL • Backup PINGs when the device establish any other connection • On open connections send messages as keep-alive response • Support WAP PUSH for wake-up (require carrier integration) • Abuse control • Carriers with multiple private sub-networks • More control about sent notifications Is the UA connected? (presence) Is the notification delivered? OWD 53 Telefonica Digital
- 54. 13 Questions? OWD 54 Telefonica Digital
- 55. OWD 55 Telefonica Digital