Ad
More Related Content
What's hot (13)
Viewers also liked (14)
Ad
Similar to IBM Streams V4.1 and Incremental Checkpointing (20)
Ad
More from lisanl (18)
Recently uploaded (20)
IBM Streams V4.1 and Incremental Checkpointing
- 1. © 2015 IBM Corporation Incremental Checkpointing IBM Streams Version 4.1 Fang Zheng Streams Development [email protected]
- 2. 2 © 2015 IBM Corporation Agenda Introduction to Incremental Checkpointing How It Works How to Use It Case Study: VWAP Application
- 3. 3 © 2015 IBM Corporation Important Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR • ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE. IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
- 4. 4 © 2015 IBM Corporation Introduction to Incremental Checkpointing Since Streams V4, an operator can implement the StateHandler interface to checkpoint and reset its state • StateHandler::checkpoint(Checkpoint & ckpt) is periodically called by Streams Runtime to checkpoint operator state • StateHandler::reset(Checkpoint & ckpt) is called upon operator restart to restore operator state By convention, each checkpoint contains the full operator state When operator state is large, the time and space costs of checkpointing can be high This can in turn cause negative impact on tuple processing performance Large checkpoints can also cause resource contention on checkpointing data store Incremental checkpointing is a known technique to reduce checkpointing cost • Save full operator state in a “Base” checkpoint • Track changes made to operator state during normal processing • For each subsequent checkpoint, only save the changed portion of state since previous checkpoint to form a “Delta” checkpoint
- 5. 5 © 2015 IBM Corporation How Incremental Checkpointing Works Why incremental checkpointing? • If the size of a “Delta” checkpoint is smaller than a full-state checkpoint, then the time and size of checkpoint is reduced Incremental checkpointing is applicable to many streaming applications Some operator state data is intensively read but rarely updated A large sliding window which moves slowly Only a small number of “hot” keys are updated in a large key-value map Such opportunity exists in many analytics (Aggregate, Join, Sort, Dedup, TopK, …) What’s in Streams V4.1 • Low-overhead change tracking and incremental checkpointing for commonly-used data structures (in particular, all configurations of SPL windows) • Automatically and dynamically adjust incremental checkpointing behavior to i) improve overall application performance and ii) bound worst-case restoration cost
- 6. 6 © 2015 IBM Corporation How Incremental Checkpointing Works (Cont.) An example: Incrementally checkpointing a queue A B C D E F G H I J B C D E F G H I J K D E F G H I J K L M Insert K Delete A Insert L Delete B Insert M Delete C A,B,C,D,E,F,G,H,I,J Take Checkpoint #1 Take Checkpoint #2 K 1 L M 2 Take Checkpoint #3 Base checkpoint Delta checkpoint Delta checkpoint new item K number of deletions: 1 new item L, M number of deletions: 2 Queue
- 7. 7 © 2015 IBM Corporation How Incremental Checkpointing Works (Cont.) An example: Restore the queue from a Delta checkpoint A B C D E F G H I J B C D E F G H I J K D E F G H I J K L M Step 2: retrieve Checkpoint #2, and re-apply changes in Checkpoint #2 Insert K Delete A Step 3: retrieve Checkpoint #3, and re-apply changes in Checkpoint #3 Insert L Insert M Delete B Delete C A,B,C,D,E,F,G,H,I,J Step 1: Retrieve Checkpoint #1, De-serialize the queue from it K 1 L M 2 new item K number of deletions: 1 new item L, M number of deletions: 2 Queue
- 8. 8 © 2015 IBM Corporation How Incremental Checkpointing Works (Cont.) An example: an unordered map K0 V0 Insert (K10, V10) Update (K1) <- V1’ Delete(K2, V2) Update (K1) <- V1’’ (K0,V0), (K1,V1), …, (K9,V9) Take Checkpoint #1 Map …… K1 V1 K2 V2 K9 V9 K0 V0 …… K1 V1’’ K9 V9 K10 V10 Base checkpoint (K1,V1’’), (K10,V10) K2 Delta checkpointTake Checkpoint #2 Updated/inserted key-values Plus deleted keys
- 9. 9 © 2015 IBM Corporation How Incremental Checkpointing Works (Cont.) Issues with incremental checkpointing • The overhead of change tracking must be paid off by savings in checkpointing cost • What if operator state is changed completely between two checkpoints? • The restoration cost is increased with incremental checkpointing Our solution: dynamically adapt incremental checkpointing behavior • The Streams Runtime continuously assesses whether incremental checkpointing is beneficial, and adjusts the number of consecutive Delta checkpoints to take • The worst-case restoration cost is bounded by capping number of Delta checkpoints • Automatically turn off incremental checkpointing when it’s not beneficial • Such adaptiveness saves end user the burden of manual configuration and tuning BASE …DELTA TIMECHECKPOINT 1 CHECKPOINT 2 CHECKPOINT 3 CHECKPOINT 4 CHECKPOINT D+1 CHECKPOINT D+2… DELTA DELTA DELTA BASE D: Number of consecutive Delta checkpoints DELTA …
- 10. 10 © 2015 IBM Corporation How to Use Incremental Checkpointing? For C++ operator developer • The SPL Runtime provides two container classes with built-in change tracking and incremental checkpointing capabilities: • SPL::IncrDeque<T> for double-ended queue • SPL::IncrUnorderedMap<K, V> for unordered map • Operator code can enable incremental checkpointing for an SPL Window by instantiate the window with incremental deque and map: • Two CodeGen APIs are extended to help generating C++ codes for SPL Window: // an SPL window without incremental checkpointing capability SPL::Window<T, G, D=std::deque<T>, S=std::tr1::unordered_map<G,D> > myWindow; // an SPL window with incremental checkpointing capability SPL::Window<T, G, D=SPL::IncrDeque<T>, S=SPL::IncrUnorderedMap<G,D> > myWindow; SPL::CodeGen::getWindowCppType() SPL::CodeGen::getWindowEventCppType()
- 11. 11 © 2015 IBM Corporation How to Use Incremental Checkpointing? (Cont.) For C++ operator developer • Example: Instantiate SPL Window with incremental data types # Code taken from Aggregate_h.cgt # Check if the operator is configured to do checkpointing # (in a Consistent Region or with config checkpoint clause configured) my $isInConsistentRegion = $model->getContext()->getOptionalContext("ConsistentRegion"); my $ckptKind = $model->getContext()->getCheckpointingKind(); if ($isInConsistentRegion || $ckptKind ne "none") { # Instantiate window with deque and unordered map which are of incremental checkpointing capability $windowCppType = ($partitionByParam) ? SPL::CodeGen::getWindowCppType($window, $windowTupleType, 'PartitionByType', 'SPL::IncrDeque', 'SPL::IncrUnorderedMap') : SPL::CodeGen::getWindowCppType($window, $windowTupleType, '', 'SPL::IncrDeque'); $windowEventCppType = ($partitionByParam) ? SPL::CodeGen::getWindowEventCppType($window, $windowTupleType, 'PartitionByType','SPL::IncrDeque','SPL::IncrUnorderedMap') : SPL::CodeGen::getWindowEventCppType($window, $windowTupleType, '', 'SPL::IncrDeque'); }
- 12. 12 © 2015 IBM Corporation How to Use Incremental Checkpointing? (Cont.) For SPL programmer • The Join, Aggregate, and Sort operators instantiate their SPL windows with incremental checkpointing if the operators are configured to do checkpointing (in Consistent Region or with “config checkpoint” clause) • No extra configuration is needed in SPL code @consistent(trigger=periodic, period4.0) stream<SourceFormat> Message = CustomSource() { …. } // Aggregate operator is in Consistent Region stream< Message > VWAPAggregator0_0_1 as O = Aggregate(TradeFilter as I) { window I : sliding, count(2000), count(1), partitioned; param partitionBy : ric; output O : svwap = Sum(myvwap), svolume = Sum(volume); } // Aggregate operator is in an Autonomous region and with config checkpoint clause stream< Message > VWAPAggregator0_0_1 as O = Aggregate(Message as I) { window I : sliding, count(2000), count(1), partitioned; param partitionBy : ric; output O : svwap = Sum(myvwap), svolume = Sum(volume); config checkpoint : periodic(10.0); }
- 13. 13 © 2015 IBM Corporation Case Study: VWAP Application VWAP Application • Ingest financial transaction data streams • Contains an Aggregate operator • Run VWAP within a consistent region, varying the checkpointing period • Run VWAP with Streams V4 which always checkpoints full operator state vs. Streams V4.1 which does incremental checkpointing
- 14. 14 © 2015 IBM Corporation Questions?