Deduplicating and analysing time-series data with Apache Beam and QuestDB
0 likes50 views
Time series data pipelines tend to prioritise speed and freshness over completeness and integrity. In such scenarios, it is very common to ingest duplicate data, which may be fine for many analytical use cases, but is very inconvenient for others. There are many open source databases built specifically for the speed and query semantics of time series, and most of them lack automatic deduplication of events in near real-time. One such database is QuestDB, which requires a manual batch process to deduplicate ingested data. In this talk, we will see how we can successfully use Apache Beam to deduplicate streaming time series, which can then be analysed by a time series database.
![BEAM SUMMIT NYC 2023
#
The Python QuestDB Sink
https://github.com/javier/questdb-beam/tree/main/python
pcoll | WriteToQuestDB(table,
symbols=[list_of_symbols],
columns=[list_of_columns],
host=host,
port=port,
batch_size=optionalSizeOfBatch,
tls=optionalBoolean,
auth=optionalAuthDict)](https://image.slidesharecdn.com/deduplicatingandanalysingtime-seriesdatawithapachebeamandquestdb-230619004945-20218bd9/85/Deduplicating-and-analysing-time-series-data-with-Apache-Beam-and-QuestDB-23-320.jpg)
Ad
More Related Content
Similar to Deduplicating and analysing time-series data with Apache Beam and QuestDB (20)
More from javier ramirez (20)
Ad
Recently uploaded (20)
Ad
Deduplicating and analysing time-series data with Apache Beam and QuestDB
- 2. NYC 2023 Javier Ramírez QuestDB @supercoco9 / @[email protected] Deduplicating And Analysing Time-Series Data With Apache Beam And QuestDB
- 3. About me: I like databases & open source 2022- today. Developer relations at an open source database vendor ● QuestDB, PostgreSQL, MongoDB, Timescale, InfluxDB, Apache Flink 2019-2022. Data & Analytics specialist at a cloud provider ● Amazon Aurora, Neptune, Athena, Timestream, DynamoDB, DocumentDB, Kinesis Data Streams, Kinesis Data Analytics, Redshift, ElastiCache for Redis, QLDB, ElasticSearch, OpenSearch, Cassandra, Spark… 2013-2018. Data Engineer/Big Data & Analytics consultant ● PostgreSQL, Redis, Neo4j, Google BigQuery, BigTable, Google Cloud Spanner, Apache Spark, Apache BEAM, Apache Flink, HBase, MongoDB, Presto 2006-2012 - Web developer ● MySQL, Redis, PostgreSQL, Sqlite, ElasticSearch late nineties to 2005. Desktop/CGI/Servlets/ EJBs/CORBA ● MS Access, MySQL, Oracle, Sybase, Informix As a student/hobbyist (late eighties - early nineties) ● Amsbase, DBase III, DBase IV, Foxpro, Microsoft Works, Informix The pre-SQL years The licensed SQL period The libre and open SQL revolution / The NoSQL rise The hadoop dark ages / The python hegemony/ The cloud database big migrations The streaming era/ The database as a service singularity The SQL revenge/ the realtime database/the embedded database
- 4. BEAM SUMMIT NYC 2023 # Agenda ● The problem of data duplication ● The problem of data duplication ● The problem of data duplication ● The problem of data duplication ● Behold: a dashboard! ● The many challenges of time-series data ● QuestDB to the rescue ● Down the rabbit hole of writing a custom BEAM Sink ○ Finding several needles on a documentation haystack ○ When I sadly discovered Python streaming support is meh ○ The unsung hero saves the day (again): implementing the Sink in Java
- 5. BEAM SUMMIT NYC 2023 # Duplication WHY
- 6. BEAM SUMMIT NYC 2023 # Duplication HOW
- 7. BEAM SUMMIT NYC 2023 # Duplication WHAT
- 8. BEAM SUMMIT NYC 2023 # My lazy approach to choosing a database If you can use only one database for everything, go with PostgreSQL* * Or any other major and well supported RDBMS
- 9. BEAM SUMMIT NYC 2023 # Imagine… a factory floor with 500 machines, or a fleet with 500 vehicles, or 50 trains, with 10 cars each, or 500 users with a mobile phone, or 500 financial instruments generating tick data …sending data every second
- 10. BEAM SUMMIT NYC 2023 # A conventional database’s nightmare 43,200,000 rows a day……. 302,400,000 rows a week…. 1,314,144,000 rows a month
- 11. BEAM SUMMIT NYC 2023 # Timestamps are hard
- 12. BEAM SUMMIT NYC 2023 # Time-series analytics in a nutshell Working with timestamped data in a database is tricky* * specially working with analytics of data changing over time or at a high rate
- 14. BEAM SUMMIT NYC 2023 # We’d like to be known for ● Performance ○ Better performance with smaller machines ● Developer Experience ● Proudly Open Source (Apache 2.0)
- 15. NYC 2023 A quick overview of some interesting queries
- 16. BEAM SUMMIT NYC 2023 # Try it live on https://demo.questdb.io WHERE … TIME RANGE SELECT * from trips WHERE pickup_datetime in '2018'; SELECT * from trips WHERE pickup_datetime in '2018-06'; SELECT * from trips WHERE pickup_datetime in '2018-06-21T23:59'; SELECT * from trips WHERE pickup_datetime in '2018;2M' LIMIT -10; SELECT * from trips WHERE pickup_datetime in '2018;10s' LIMIT -10; SELECT * from trips WHERE pickup_datetime in '2018;-3d' LIMIT -10; SELECT * from trips WHERE pickup_datetime in '2018-06-21T23:59:58;4s;1d;7' SELECT * from trips WHERE pickup_datetime in '2018-06-21T23:59:58;4s;-1d;7'
- 17. BEAM SUMMIT NYC 2023 # Try it live on https://demo.questdb.io SAMPLE BY Aggregates data in homogeneous time chunks SELECT timestamp, sum(price * amount) / sum(amount) AS vwap_price, sum(amount) AS volume FROM trades WHERE symbol = 'BTC-USD' AND timestamp > dateadd('d', -1, now()) SAMPLE BY 15m ALIGN TO CALENDAR; SELECT timestamp, min(tempF), max(tempF), avg(tempF) FROM weather SAMPLE BY 1M;
- 18. BEAM SUMMIT NYC 2023 # Try it live on https://demo.questdb.io SAMPLE BY … FILL Can fill missing time chunks using different strategies (NULL, constant, LINEAR, PREVious value) SELECT timestamp, sum(price * amount) / sum(amount) AS vwap_price, sum(amount) AS volume FROM trades WHERE symbol = 'BTC-USD' AND timestamp > dateadd('d', -1, now()) SAMPLE BY 1s FILL(NULL) ALIGN TO CALENDAR;
- 19. BEAM SUMMIT NYC 2023 # Try it live on https://demo.questdb.io LATEST ON … PARTITION BY … Retrieves the latest entry by timestamp for a given key or combination of keys, for scenarios where multiple time series are stored in the same table. SELECT * FROM trades WHERE symbol in ('BTC-USD', 'ETH-USD') LATEST ON timestamp PARTITION BY symbol, side;
- 20. BEAM SUMMIT NYC 2023 # Try it live on https://demo.questdb.io ASOF JOIN / LT JOIN SPLICE JOIN ASOF JOIN joins two different time-series measured. For each row in the first time-series, the ASOF JOIN takes from the second time-series a timestamp that meets both of the following criteria: ● The timestamp is the closest to the first timestamp. ● The timestamp is strictly prior or equal to the first timestamp. WITH trips2018 AS ( SELECT * from trips WHERE pickup_datetime in '2016' ) SELECT pickup_datetime, fare_amount, tempF, windDir FROM trips2018 ASOF JOIN weather;
- 21. BEAM SUMMIT NYC 2023 # Building a Sink connector QuestDB cannot do in-stream deduplications. Apache BEAM can help
- 22. BEAM SUMMIT NYC 2023 # The Python QuestDB Sink ● WriteToQuestDB(PTransform) class ○ Receives the args you need to pass to the sink ○ Implements the expand method, which receives the PCollection then invokes ParDo to _WriteTOQuestDBFn ○ ● _WriteToQuestDBFn(DoFn) class ○ Instantiates _QuestDBSink on start_bundle ○ Flushes/releases _QuestDBSink on finish_bundle ○ Implements display_data to show info on the UI ○ Calls to _QuestDBSink.write on the process method ○ ● _QuestDBSink class ○ Deals with the QuestDB connection itself
- 23. BEAM SUMMIT NYC 2023 # The Python QuestDB Sink https://github.com/javier/questdb-beam/tree/main/python pcoll | WriteToQuestDB(table, symbols=[list_of_symbols], columns=[list_of_columns], host=host, port=port, batch_size=optionalSizeOfBatch, tls=optionalBoolean, auth=optionalAuthDict)
- 25. BEAM SUMMIT NYC 2023 # The Java QuestDB Sink ● QuestDbIO.Write class, extends PTransform ○ Receives the args you need to pass to the sink ○ Uses @AutoValue to generate classes “magically” ○ Implements the expand method, which receives the PCollection then invokes ParDo to QuestDbIO.Write.WriteFn (with optional deduplication) ○ Implements populateDisplayData ● QuestDbIO.Write.WriteFn class, extends DoFn ○ Instantiates QuestDBSender on start_bundle ○ Flushes/closes QuestDBSender on finish_bundle ○ Parses/sends the QuestDbRow to QuestDB on the process method
- 26. BEAM SUMMIT NYC 2023 # Where the magic happens https://github.com/javier/questdb-beam/blob/main/java/src/main/java/org/apache/beam/sdk/io/questdb/QuestDbIO.java keydAndWindowed = (PCollection) input.apply(WithKeys.of(new SerializableFunction<QuestDbRow, String>() { @Override public String apply(QuestDbRow r) { return String.valueOf(r.hashCode()); } })); PCollection windowedItems = (PCollection) keydAndWindowed.apply( Window. <KV<String, String>>into( Sessions. withGapDuration( Duration.standardSeconds(deduplicationDurationMillis()) ) ) ); PCollection<QuestDbRow> uniqueRows = (PCollection<QuestDbRow>) ((PCollection) keydAndWindowed.apply( Deduplicate.keyedValues() ) ).apply(Values.create());
- 27. BEAM SUMMIT NYC 2023 # The Java QuestDB Sink https://github.com/javier/questdb-beam/tree/main/java // pcoll needs to be of type QuestDbRow pcoll.apply(ParDo.of(new LineToMapFn())); parsedLines.apply(QuestDbIO.write() .withUri("your-instance-host.questdb.com:YOUR_PORT") .withTable("beam_demo") .withDeduplicationEnabled(true) .withDeduplicationByValue(false) .withDeduplicationDurationMillis(5L) .withSSLEnabled(true) .withAuthEnabled(true) .withAuthUser("admin") .withAuthToken("verySecretToken")