optiq_technical_note_mdg_snapshot_20170131
optiq_technical_note_mdg_snapshot_20170131
SUMMARY
The Optiq MDG Snapshot service is a recovery functionality provided by the exchange in order to allow
customers to recover from packet loss or in case of an intraday/late start. Providing an image of the
market data and the last Market Data Sequence Number used to build the Snapshot image allows
customers to synchronize with the real time data and continue/start processing messages in real time.
Snapshot messages will be provided on dedicated Snapshot channels, alongside the real-time channels
and the messages will be compressed using LZ4.
A snapshot contains an image of all the market data message types for all instruments that are part
of the real-time channel and the messages are sent with the Rebroadcast indicator field set to 1.
The snapshots are sent out periodically and the minimum time interval between 2 snapshots is
2 seconds.
1. THE SNAPSHOT SEQUENCE EXPLAINED
A snapshot sequence contains multiple channels that when combined provide the necessary
information for our customers to align with the real time MDG feed. The following snapshot channels
are part of a sequence:
BBO Channel
Both “Start Of Snapshot” and “End Of Snapshot” messages contain the last “Market Data Sequence
Number” of the last real-time message taken into account by the snapshot. Both these messages are
not compressed.
In order to use the Snapshot, the first step is to queue messages on the real-time channel from the
moment you start receiving the snapshot data.
Clients can use the Start Of Snapshot or End Of Snapshot messages to retrieve the last “Market Data
Sequence Number” used to build the snapshot that is being sent out and can use it to synchronize with
the real time channel by discarding from the real-time queued data those messages with a “Market
Data Sequence Number” equal or smaller than the last “Market Data Sequence Number” received in
the Start or End Of Snapshot messages and continue in real time channel processing the next Market
Data Sequence Numbers.
It is important to note that since the Market Data Sequence Number does not necessarily increment
by 1, within a real time channel, the sequence number in the start or end snapshot messages
might belong to another channel, and was in fact not actually lost. As a reminder, the only criteria to
determine packet loss is by using the Packet Sequence Number which increments monotonically with 1
and is unique per channel ID.
2. SNAPSHOT SEQUENCE FOR CASH:
Start Of Snapshot
Market Status Change
End Of Snapshot
Start Of Snapshot
Market Status Change
Start Of Snapshot
Timetable
Full Trade Information messages
Price Update
Real Time Index (for BdL)
Index Summary (for BdL)
Statistics
Exchange Announcement
End Of Snapshot
Cash is composed of: Bourse de Luxembourg (BdL), Equities, Funds and Fixed Income.
1 Reference Data represents: all instruments characteristics, scheduled phases and market administration
messages.
2 This message will not provide: New Bid (3)/New Offer (4), Updated Bid (5) /Updated Offer (6), New Bid With
Liquidity Provider (58)/New Offer With Liquidity Provider (59), Updated Bid With Liquidity Provider(60)/ Updated
Offer With Liquidity Provider (61), New Bid RLP (Retail Liquidity Provider) (16)/ New Offer RLP (Retail Liquidity
Provider) (17), Updated Bid RLP Retail Liquidity Provider) (18)/ Updated Offer RLP (Retail Liquidity Provider) (19),
New Bid SI (20)/ New Offer SI (21) and Updated Bid SI (22)/ Updated Offer SI (23).
3 This message will provide only: New Bid RLP (Retail Liquidity Provider) (16)/ New Offer RLP (Retail Liquidity
Provider) (17), Updated Bid RLP Retail Liquidity Provider) (18)/ Updated Offer RLP (Retail Liquidity Provider) (19)
or Clear-Book (254).
4 This message will provide only: New Bid SI (20)/ New Offer SI (21), Updated Bid SI (22)/ Updated Offer SI (23), SI
Trade (47) or Clear-Book (254).
SNAPSHOT SEQUENCE FOR DERIVATIVES:
Start Of Snapshot
Market Status Change
End Of Snapshot
Start Of Snapshot
Market Status Change
2 Seconds
End Of Snapshot
Start Of Snapshot
Intraday Outright or Strategy Standing data
Full Trade Information messages
Price Update
Statistics
Exchange Announcement
End Of Snapshot
Derivatives is composed of: Options, Futures, Warrants & Certificates.
1 Reference Data represents: all instruments characteristics, scheduled phases and market administration
messages.
2 This message will not provide: New Bid (3)/New Offer (4), Updated Bid (5) /Updated Offer (6), New Bid With
Liquidity Provider (58)/New Offer With Liquidity Provider (59), Updated Bid With Liquidity Provider(60)/ Updated
Offer With Liquidity Provider (61), New Bid RLP (Retail Liquidity Provider) (16)/ New Offer RLP (Retail Liquidity
Provider) (17), Updated Bid RLP Retail Liquidity Provider) (18)/ Updated Offer RLP (Retail Liquidity Provider)
(19), New Bid SI (20)/ New Offer SI (21) and Updated Bid SI (22)/ Updated Offer SI (23).
3. SNAPSHOT SEQUENCE FOR INDICES:
Indices
Start Of Snapshot
Real Time Index
2 Seconds Index Summary
Statistics
End Of Snapshot
The Packet Sequence Number (PSN) is part of the packet header and should be used for UDP gap
detection and packet ordering. Each channel has its own PSN sequence.
Aggregators are MDG internal components that are dealing with a set of channels. The Market Data
Sequence Numbers are managed at the aggregator level. Each one of them has its own sequence,
starting from 0 and incrementing by step of 1 along the day. Since clients may listen to only a subset
of the channels managed by one aggregator, they won’t see all the Market Data Sequence Numbers
in the messages they get from the channels they listen to. Therefore on one channel the Market
Data Sequence Numbers will increment all along the day but not necessarily by step of 1.
Packet Sequence Numbers are used for detecting packet loss. The Snapshot Channels have their own
PSNs that are not the same as the PSNs on the Real Time Channels.
Line A Line B
PSN PSN
101 101
102 102
103 103
104 104
Dropped packet can
Gap detected 105 be recovered from
Line B channel
106 106
107 107
Using this method, a lost packet can be recovered from the second line. In case of packet loss on both
lines, then the snapshot mechanism should be used.
UDP packets can potentially arrive unordered and potentially sent twice. As such, systems should be
able to reorder the packets and detect duplicate packets.
The snapshot feed is also carried over line A and B, therefore line arbitrage is also possible on the
snapshot multicast feeds.
6. SNAPSHOT APPLIED
Customers that connect intraday (late start) have only one viable way to use the snapshot:
“Hop On” (connect) to the Snapshot Channel corresponding with the Real Time Channel the Packet loss
occurred, while starting to queue the Real Time messages.
The customer clears the order books for all instruments on that channel.
The Start Of Snapshot (2101) message indicates the start of a Snapshot Sequence.
The image provided in this sequence contains everything to rebuild the order book up to the last
Market Data Sequence Number that is in the end snapshot messages.
Clients then discard the real-time queued messages with a “Market Data Sequence Number” less
than or equal to the last “Market Data Sequence Number” of the snapshot.
Customers that experience packet loss (however unlikely (on both A and B feed) can use the same
preferred method:
“Hop On” (connect) to the Snapshot Channel corresponding with the Real Time Channel the Packet loss
occurred, while starting to queue the Real Time messages.
The customer clears the order books for all instruments on that channel.
The Start Of Snapshot (2101) message indicates the start of a Snapshot Sequence.
The image provided in this sequence contains everything to rebuild the order book up to the last
Market Data Sequence Number that is in the end snapshot messages.
Clients then discard the real-time queued messages with a “Market Data Sequence Number” less
than or equal to the last “Market Data Sequence Number” of the snapshot.
Use the snapshot to correct those instruments impacted - message type by message type. This method
is not supported by Euronext and there is no documentation available on this topic. The reason behind
this is that the snapshot provides an image and does not provide all messages sent within a certain
frame. Please note, this follows not the same logic as the XDP’s Refresh Service. This is why members
are discouraged to build a solution that only correct the impacted parts.
Regardless if one has a 10 Gb, 1Gb or 100Mb connection clients use the same dedicated Snapshot data.
Summarized:
7. Q&A ON THE SNAPSHOT SERVICE
Q: What is the Snapshot Service?
Snapshot is a service provided by the exchange in order to provide at a given time of the day an
image of the market data. The Snapshot Service allows to resynchronize with the real-time data
and provides key market data depending on the nature and content of the corresponding real-
time feed.
Q: What are the bandwidth needs for subscribing to the Snapshot Channel?
Everyone connects to the same snapshot channel (for a subset of realtimes channels) regardless
their connection bandwidth, the service is developed in such a way all connection types are
supported and use the same data broadcast.
Q: Will every snapshot contain all the data since start of the day ?
No. The snapshot will only provide an image of all the messages needed to build the book as it is
at the time of the snapshot sending.
Q: Why do I use the Packet Sequence Numbers for detecting potential data gaps? Can’t I just look at
the Market Data Sequence Numbers?
No, you can’t as the logic behind the MDSNs is such that missing MDSN are not per definition
identifying a gap. PSN gaps, however is an indicator of potential missing data and an action is
required.
Q: Does a Snapshot image contain all needed referential data to rebuild the order book?
Snapshot provides only Intraday standing data, for the full list of instruments (full standing data)
you should use the File Service.
Q: Does a member need a single snapshot channel for the entire Cash market for example or one
snapshot channel per Cash Asset Class?
The repartition is as following: Per Asset Class – Per Sub Asset – Per Partition – Per channel
Type.
Jack Cohecha
Tel: +33 (1) 70 48 25 40
[email protected]
Disclaimers
This document contains information which is confidential and of value to Euronext. The information and materials contained in this document are provided ‘as is’ and Euronext
does not warrant the accuracy, adequacy or completeness and expressly disclaims liability for any errors or omissions or changes enabled to be made for any reason included
correction, update and upgrade purpose. This document contains links to certain Internet Websites developed, sponsored or maintained by third parties unaffiliated with
Euronext. The content you view therein is not provided or controlled by Euronext. Euronext is not responsible for that content, nor has it developed, checked for accuracy or
otherwise reviewed the content or privacy policy of any such third party Website. This document is not intended to impose any legal obligation on Euronext. This document
and any contents thereof, as well as any prior or subsequent information exchanged with Euronext in relation to the subject matter of this document, are confidential and are
for the sole attention of the intended recipient. Except as described below, all proprietary rights and interest in or connected with this publication shall vest in Euronext. No
part of it may be redistributed or reproduced without the prior written permission of Euronext. Portions of this publication may contain materials or information copyrighted,
trademarked or otherwise owned by a third party. No permission to use these third party materials should be inferred from this publication. Implementation of the project may
be subject to regulatory approval. Euronext refers to Euronext N.V. and its affiliates. Information regarding trademarks and intellectual property rights of Euronext is located at
https://www.euronext.com/terms-use.
01/17