Test strategies for data processing pipelines, v2.0
4 likes2,719 views
This talk will present recommended patterns and corresponding anti-patterns for testing data processing pipelines. We will suggest technology and architecture to improve testability, both for batch and streaming processing pipelines. We will primarily focus on testing for the purpose of development productivity and product iteration speed, but briefly also cover data quality testing.
![www.mapflat.com
18
Putting batch tests together
class CleanUserTest extends FlatSpec {
val inFile = (tmpDir / "test_user.json")
val outFile = (tmpDir / "test_output.json")
def writeInput(users: Seq[User]) = inFile.overwrite(users.map(u => Json.toJson(u).toString).mkString("n"))
def readOutput() = outFile.contentAsString.split("n").map(line => Json.fromJson[User](Json.parse(line))).toSeq
def runJob(input: Seq[User]): Seq[User] = {
writeInput(input)
val args = Seq(
"--user", inFile.path.toString,
"--output", outFile.path.toString)
CleanUserJob.main(args) // Works for some processing frameworks, e.g. Spark
readOutput()
}
"Clean users" should "remain untouched" {
val output = runJob(Seq(TestInput.userTemplate))
assert(output === Seq(TestInput.userTemplate))
}
}](https://image.slidesharecdn.com/teststrategiesfordataprocessingpipelinesv2-171023203400/85/Test-strategies-for-data-processing-pipelines-v2-0-18-320.jpg)
assert(lens.get(output.head) === "SE")
}
}
● Avoid full record comparisons
○ Except for a few tests
● Examine only fields relevant to test
○ Or tests break for unrelated changes
● Lenses are your friend
○ JSON: JsonPath (Java), Play (Scala), ...
○ Case classes: Monocle, Shapeless,
Scalaz, Sauron, Quicklens](https://image.slidesharecdn.com/teststrategiesfordataprocessingpipelinesv2-171023203400/85/Test-strategies-for-data-processing-pipelines-v2-0-19-320.jpg)
![www.mapflat.com
class CleanUserTest extends FlatSpec {
def runJob(input: Seq[User]): Seq[User] = {
writeInput(input)
val args = Seq("--user", inFile.path.toString,
"--output", outFile.path.toString)
CleanUserJob.main(args)
// Ugly way to expose counters
val counters = Map(
"invalid" -> CleanUserJob.invalidCount,
"upper-cased" -> CleanUserJob.upperCased)
(readOutput(), counters)
}
"Lower case country" should "translate to upper case" {
val input = Seq(userTemplate.copy(country = "se"))
val (output, counters) = runJob(input)
assert(output.size === 1)
assert(output.head.country === "SE")
assert(counters.get("upper-cased") === 1)
assert((counters - "upper-cased").filter(_._2 != 0)
shouldBe empty)
}
}
20
Inspecting counters
● Counters (accumulators in Spark) are
critical for monitoring and quality
● Test that expected counters are bumped
○ But no other counters](https://image.slidesharecdn.com/teststrategiesfordataprocessingpipelinesv2-171023203400/85/Test-strategies-for-data-processing-pipelines-v2-0-20-320.jpg)
![www.mapflat.com
class CleanUserTest extends FlatSpec {
def validateInvariants(
input: Seq[User],
output: Seq[User],
counters: Map[String, Int]) = {
output.foreach(recordInvariant)
// Dataset invariants
assert(input.size === output.size)
assert(input.size should be >= counters["upper-cased"])
}
def recordInvariant(u: User) =
assert(u.country.size === 2)
def runJob(input: Seq[User]): Seq[User] = {
// Same as before
...
validateInvariants(input, output, counters)
(output, counters)
}
// Test case is the same
}
21
Invariants
● Some things are true
○ For every record
○ For every job invocation
● Not necessarily in production
○ Reuse invariant predicates as quality
probes](https://image.slidesharecdn.com/teststrategiesfordataprocessingpipelinesv2-171023203400/85/Test-strategies-for-data-processing-pipelines-v2-0-21-320.jpg)
![[Meetup] a successful migration from elastic search to clickhouse](https://cdn.slidesharecdn.com/ss_thumbnails/meetupasuccessfulmigrationfromelasticsearchtoclickhouse-191004114403-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Viewers also liked (9)
Ad
Similar to Test strategies for data processing pipelines, v2.0 (20)
Ad
More from Lars Albertsson (20)
Recently uploaded (20)
Test strategies for data processing pipelines, v2.0
- 1. www.mapflat.com Test strategies for data processing pipelines 1 Lars Albertsson, independent consultant (Mapflat)
- 2. www.mapflat.com Who's talking ● Swedish Institute of Computer. Science. (test & debug tools) ● Sun Microsystems (large machine verification) ● Google (Hangouts, productivity) ● Recorded Future (NLP startup) (data integrations) ● Cinnober Financial Tech. (trading systems) ● Spotify (data processing & modelling, productivity) ● Schibsted Products & Tech (data processing & modelling) ● Mapflat (independent data engineering consultant) 2
- 3. www.mapflat.com Agenda ● Data applications from a test perspective ● Testing batch processing products ● Testing stream processing product ● Data quality testing Main focus is functional, regression testing Prerequisites: Backend dev testing, basic data experience, reading Scala 3
- 4. www.mapflat.com Data lake Batch pipeline anatomy 4 Cluster storage Unified log Ingress ETL Egress DB Service DatasetJob Pipeline Service Export Business intelligence DB DB Import
- 5. www.mapflat.com Workflow orchestrator ● Dataset “build tool” ● Run job instance when ○ input is available ○ output missing ○ resources are available ● Backfill for previous failures ● DSL describes DAG ○ Includes ingress & egress Luigi / Airflow 5 DB Orchestrator
- 6. www.mapflat.com 6 Stream pipeline anatomy ● Unified log - bus of all business events ● Pub/sub with history ○ Kafka ○ AWS Kinesis, Google Pub/Sub, Azure Event Hub ● Decoupled producers/consumers ○ In source/deployment ○ In space ○ In time ● Publish results to log ● Recovery from link failures ● Replay on job bug fix Job Ads Search Feed App App App StreamStream Stream Stream Stream Stream Job Job Stream Stream Stream Job Job Data lake Business intelligence Job
- 7. www.mapflat.com Online vs offline 7 Online Offline ETLCold store AI feature DatasetJob Pipeline Service Service Online
- 8. www.mapflat.com Online failure vs Offline failure 8 10000s of customers, imprecise feedback Need low probability => Proactive prevention => Low ROI 10s of employees, precise feedback Ok with medium probability => Reactive repair => High ROI Risk = probability * impact
- 9. www.mapflat.com Value of testing 9 For data-centric (batch) applications, in this order: ● Productivity ○ Move fast without breaking things ● Fast experimentation ○ 10% good ideas, 90% neutral or bad ● Data quality ○ Challenging, more important than ● Technical quality ○ Technical failure => ■ Operations hassle. ■ Stale data. Often acceptable. ■ No customers were harmed. Usually. Significant harm to external customer is rare enough to be reactive - data-driven by bug frequency.
- 10. www.mapflat.com Test concepts 10 Test harness Test fixture System under test (SUT) 3rd party component (e.g. DB) 3rd party component 3rd party component Test input Test oracle Test framework (e.g. JUnit, Scalatest) Seam IDEs Build tools
- 11. www.mapflat.com 11 Data pipeline properties ● Output = function(input, code) ○ No external factors => deterministic ○ Easy to craft input, even in large tests ○ Perfect for test! ● Pipeline and job endpoints are stable ○ Correspond to business value ● Internal abstractions are volatile ○ Reslicing in different dimensions is common q
- 12. www.mapflat.com 12 Potential test scopes ● Unit/component ● Single job ● Multiple jobs ● Pipeline, including service ● Full system, including client Choose stable interfaces Each scope has a cost Job Service App Stream Stream Job Stream Job
- 13. www.mapflat.com 13 Recommended scopes ● Single job ● Multiple jobs ● Pipeline, including service Job Service App Stream Stream Job Stream Job
- 14. www.mapflat.com 14 Scopes to avoid ● Unit/Component ○ Few stable interfaces ○ Not necessary ○ Avoid mocks, DI rituals ● Full system, including client ○ Client automation fragile “Focus on functional system tests, complement with smaller where you cannot get coverage.” - Henrik Kniberg Job Service App Stream Stream Job Stream Job
- 15. www.mapflat.com Testing single batch job 15 Job Standard Scalatest harness file://test_input/ file://test_output/ 1. Generate input 2. Run in local mode 3. Verify output f() p() Runs well in CI / from IDE
- 16. www.mapflat.com Testing batch pipelines - two options 16 Standard Scalatest harness file://test_input/ file://test_output/ 1. Generate input 2. Run custom multi-job Test job with sequence of jobs 3. Verify output f() p() A: Customised workflow manager setup + Runs in CI + Runs in IDE + Quick setup - Multi-job maintenance p() + Tests workflow logic + More authentic - Workflow mgr setup for testability - Difficult to debug - Dataset handling with Python f() B: ● Both can be extended with ingress (Kafka), egress DBs
- 17. www.mapflat.com 17 Test data Test input data is code ● Tied to production code ● Should live in version control ● Duplication to eliminate ● Generation gives more power ○ Randomness can be desirable ○ Larger scale ○ Statistical distribution ● Maintainable > git add src/resources/test-data.json userInputFile.overwrite(Json.toJson( userTemplate.copy( age = 27, product = "premium", country = "NO")) .toString) Prod data Privacy!
- 18. www.mapflat.com 18 Putting batch tests together class CleanUserTest extends FlatSpec { val inFile = (tmpDir / "test_user.json") val outFile = (tmpDir / "test_output.json") def writeInput(users: Seq[User]) = inFile.overwrite(users.map(u => Json.toJson(u).toString).mkString("n")) def readOutput() = outFile.contentAsString.split("n").map(line => Json.fromJson[User](Json.parse(line))).toSeq def runJob(input: Seq[User]): Seq[User] = { writeInput(input) val args = Seq( "--user", inFile.path.toString, "--output", outFile.path.toString) CleanUserJob.main(args) // Works for some processing frameworks, e.g. Spark readOutput() } "Clean users" should "remain untouched" { val output = runJob(Seq(TestInput.userTemplate)) assert(output === Seq(TestInput.userTemplate)) } }
- 19. www.mapflat.com 19 Test oracle class CleanUserTest extends FlatSpec { // ... "Lower case country" should "translate to upper case" { val input = Seq(userTemplate.copy(country = "se")) val output = runJob(input) assert(output.size === 1) assert(output.head.country === "SE") val lens = GenLens[User](_.country) assert(lens.get(output.head) === "SE") } } ● Avoid full record comparisons ○ Except for a few tests ● Examine only fields relevant to test ○ Or tests break for unrelated changes ● Lenses are your friend ○ JSON: JsonPath (Java), Play (Scala), ... ○ Case classes: Monocle, Shapeless, Scalaz, Sauron, Quicklens
- 20. www.mapflat.com class CleanUserTest extends FlatSpec { def runJob(input: Seq[User]): Seq[User] = { writeInput(input) val args = Seq("--user", inFile.path.toString, "--output", outFile.path.toString) CleanUserJob.main(args) // Ugly way to expose counters val counters = Map( "invalid" -> CleanUserJob.invalidCount, "upper-cased" -> CleanUserJob.upperCased) (readOutput(), counters) } "Lower case country" should "translate to upper case" { val input = Seq(userTemplate.copy(country = "se")) val (output, counters) = runJob(input) assert(output.size === 1) assert(output.head.country === "SE") assert(counters.get("upper-cased") === 1) assert((counters - "upper-cased").filter(_._2 != 0) shouldBe empty) } } 20 Inspecting counters ● Counters (accumulators in Spark) are critical for monitoring and quality ● Test that expected counters are bumped ○ But no other counters
- 21. www.mapflat.com class CleanUserTest extends FlatSpec { def validateInvariants( input: Seq[User], output: Seq[User], counters: Map[String, Int]) = { output.foreach(recordInvariant) // Dataset invariants assert(input.size === output.size) assert(input.size should be >= counters["upper-cased"]) } def recordInvariant(u: User) = assert(u.country.size === 2) def runJob(input: Seq[User]): Seq[User] = { // Same as before ... validateInvariants(input, output, counters) (output, counters) } // Test case is the same } 21 Invariants ● Some things are true ○ For every record ○ For every job invocation ● Not necessarily in production ○ Reuse invariant predicates as quality probes
- 22. www.mapflat.com 22 Streaming SUT, example harness Scalatest Spark Streaming jobs IDE, CI, debug integration DB Topic Kafka Test input Test oracle Docker IDE / Gradle Polling Service JVM monolith for test
- 23. www.mapflat.com 23 Test lifecycle 1. Initiate from IDE / build system 2. Start fixture containers 3. Await fixture ready 4. Allocate test case resources 5. Start jobs 6. Push input data 7. While (!done && !timeout) { pollDatabase() sleep(1ms) } 8. While (moreTests) { Goto 4 } 9. Tear down fixture 8 3 6 1 2 4,9 5 7
- 24. www.mapflat.com 24 Testing for absence 1. Send test input 2. Send dummy input 3. Await effect of dummy input 4. Verify test output absence Assumes in-order semantics 1,2 3, 4
- 25. www.mapflat.com Testing in the process 1. Integration tests for happy paths. 2. Likely odd cases, e.g. ○ Empty inputs ○ Missing matching records, e.g. in joins ○ Odd encodings 3. Dark corners ○ Motivated for externally generated input 25 ● Minimal test cases triggering desired paths ● On production failure: ○ Debug in production ○ Create minimal test case ○ Add to regression suite ● Cannot trigger bug in test? ○ Consider new scope? Significantly different from old scopes. ○ Staging pipelines? ○ Automatic code inspection?
- 26. www.mapflat.com Quality testing variants ● Functional regression ○ Binary, key to productivity ● Golden set ○ Extreme inputs => obvious output ○ No regressions tolerated ● (Saved) production data input ○ Individual regressions ok ○ Weighted sum must not decline ○ Beware of privacy 26
- 27. www.mapflat.com 27 Quality metrics ● Processing tool (Spark/Hadoop) counters ○ Odd code path => bump counter ● Dedicated quality assessment pipelines ● Workflow quality predicate for consumption ○ Depends on consumer use case Hadoop / Spark counters DB Quality assessment job Tiny quality metadataset
- 28. www.mapflat.com 28 Quality testing in the process ● Binary self-contained ○ Validate in CI ● Relative vs history ○ E.g. large drops ○ Precondition for consuming dataset? ● Push aggregates to DB ○ Standard ops: monitor, alert DB ∆? Code ∆!
- 29. www.mapflat.com 29 Data pipeline = yet another program Don’t veer from best practices ● Regression testing ● Design: Separation of concerns, modularity, etc ● Process: CI/CD, code review, static analysis tools ● Avoid anti-patterns: Global state, hard-coding location, duplication, ... In data engineering, slipping is in the culture... :-( ● Mix in solid backend engineers ● Document “golden path”
- 30. www.mapflat.com Test code = not yet another program ● Shared between data engineers & QA engineers ○ Best result with mutual respect and collaboration ● Readability >> abstractions ○ Create (Scala, Python) DSLs ● Some software bad practices are benign ○ Duplication ○ Inconsistencies ○ Lazy error handling 30
- 31. www.mapflat.com Honey traps The cloud ● PaaS components do not work locally ○ Cloud providers should provide fake implementations ○ Exceptions: Kubernetes, Cloud SQL, Relational Database Service, (S3) ● Integrate PaaS service as fixture component is challenging ○ Distribute access tokens, etc ○ Pay $ or $$$. Much better with per-second billing. 31 Vendor batch test frameworks ● Spark, Scalding, Crunch variants ● Seam == internal data structure ○ Omits I/O - common bug source ● Vendor lock-in - when switching batch framework: ○ Need tests for protection ○ Test rewrite is unnecessary burden Test input Test output Job
- 32. www.mapflat.com Top anti-patterns 1. Test as afterthought or in production. Data processing applications are suited for test! 2. Developer testing requires cluster 3. Static test input in version control 4. Exact expected output test oracle 5. Unit testing volatile interfaces 6. Using mocks & dependency injection 7. Tool-specific test framework - vendor lock-in 8. Using wall clock time 9. Embedded fixture components, e.g. in-memory Kafka/Cassandra/RDBMS 10. Performance testing (low ROI for offline) 11. Data quality not measured, not monitored 32
- 33. www.mapflat.com Thank you. Questions? Credits: Øyvind Løkling, Schibsted Media Group Images: Tracey Saxby, Integration and Application Network, University of Maryland Center for Environmental Science (ian.umces.edu/imagelibrary/). 33
- 35. www.mapflat.com Deployment, example 35 Hg/git repo Luigi DSL, jars, config my-pipe-7.tar.gz HDFS Luigi daemon > pip install my-pipe-7.tar.gz Worker Worker Worker Worker Worker Worker Worker Worker Redundant cron schedule, higher frequency + backfill (Luigi range tools) * 10 * * * bin/my_pipe_daily --backfill 14 All that a pipeline needs, installed atomically
- 36. www.mapflat.com Continuous deployment, example 36 ● Poll and pull latest on worker nodes ○ virtualenv package/version ■ No need to sync environment & versions ○ Cron package/latest/bin/* ■ Old versions run pipelines to completion, then exit Hg/git repo Luigi DSL, jars, config my-pipe-7.tar.gz HDFS my_cd.py hdfs://pipelines/ Worker > virtualenv my_pipe/7 > pip install my-pipe-7.tar.gz * 10 * * * my_pipe/7/bin/*