Pipeline oriented programming
7 likes13,917 views
Pipeline-oriented programming emphasizes a one-directional flow of data, encouraging composability and adherence to design principles, making maintenance and testing easier. It promotes reusable, understandable, and extendable components that fit well with modern architectures. Examples in C# and F# demonstrate its implementation in practical scenarios like LINQ operations and web APIs.
![choose [
]
Picks first component
that succeeds
Define a new component](https://image.slidesharecdn.com/pipelineorientedprogramming-211021063722/85/Pipeline-oriented-programming-83-320.jpg)
![choose [
GET >=> route "/hello" >=> OK "Hello"
GET >=> route "/goodbye" >=> OK "Goodbye"
]
Pick first path
that succeeds](https://image.slidesharecdn.com/pipelineorientedprogramming-211021063722/85/Pipeline-oriented-programming-84-320.jpg)
![choose [
GET >=> route "/hello" >=> OK "Hello"
GET >=> route "/goodbye" >=> OK "Goodbye"
POST >=> route "/bad" >=> BAD_REQUEST
]
All the benefits of pipeline-oriented programming](https://image.slidesharecdn.com/pipelineorientedprogramming-211021063722/85/Pipeline-oriented-programming-85-320.jpg)
![choose [
GET >=> route "/hello" >=> OK "Hello"
GET >=> route "/goodbye" >=> OK "Goodbye"
POST
>=> route "/user"
>=> mustBeLoggedIn UNAUTHORIZED
>=> requiresRole "Admin"
// etc
]
All the benefits of pipeline-oriented programming](https://image.slidesharecdn.com/pipelineorientedprogramming-211021063722/85/Pipeline-oriented-programming-86-320.jpg)
![choose [
GET >=> route "/hello" >=> OK "Hello"
GET >=> route "/goodbye" >=> OK "Goodbye"
POST
>=> route "/user"
>=> mustBeLoggedIn UNAUTHORIZED
>=> requiresRole "Admin"
// etc
]
All the benefits of pipeline-oriented programming](https://image.slidesharecdn.com/pipelineorientedprogramming-211021063722/85/Pipeline-oriented-programming-87-320.jpg)
Pipeline oriented programming
- 1. Pipeline Oriented Programming (DotNext 2021) @ScottWlaschin fsharpforfunandprofit.com/pipeline Code examples will be in C# and F#
- 3. Object-oriented Arrows go in all directions
- 4. Pipeline-oriented All arrows go left to right
- 5. A pipeline-oriented web backend One directional flow, even with branching
- 6. What are the benefits of a pipeline-oriented approach?
- 7. Benefits • Pipelines encourage composability • Pipelines follow good design principles • Pipelines are easier to maintain • Pipelines make testing easier • Pipelines fit well with modern architectures – E.g. Onion/Clean/Hexagonal, etc
- 8. Benefit 1: Pipelines encourage composability Composable == “Like Lego”
- 9. Connect two pieces together and get another "piece" that can still be connected You don't need to create a special adapter to make connections.
- 10. Component Component Component Pipelines composition Pipelines encourage you to design so that: • Any pieces can be connected • You don’t need special adapters If you do this, you get nice composable components
- 11. Bigger Component Sub- Component Sub- Component Sub- Component Sub-components composed into a new bigger unit
- 12. Benefit 2: Pipelines follow good design principles Many design patterns work naturally with a pipeline-oriented approach
- 13. Single responsibility principle Component Can't get more single responsibility than this!
- 14. Open/Closed principle You could be able to add new functionality (“open for extension”) without changing existing code (“closed for modification”) New Behavior Component Component Component Extension Not changed
- 17. Decorated Component Decorator Pattern Decorator Decorator "decorated" component works just like the original one
- 18. Benefit 3: Pipelines are easier to maintain
- 19. This is easier to maintain This is harder to maintain
- 20. Object-oriented: Real-world dependency graph Multiple circular dependencies Arrows go in all directions
- 21. Pipeline-oriented: Real-world dependency graph All arrows go left to right Same functionality as the OO example
- 22. More benefits: • Pipelines make testing easier • Pipelines fit well with modern architectures
- 24. var count = 0; foreach (var i in list) { var j = i + 2; if (j > 3) { count++; } } return count; Here is a traditional for- loop in C#
- 25. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Here is the same code using LINQ in C#
- 26. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Composability The LINQ components have been designed to fit together in many different ways You could write your code this way too!
- 27. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Single responsibility Each LINQ component does one thing only. Easier to understand and test
- 28. return list .Select(x => x + 2) .Where(x => x > 3) .Where(x => x < 10) .Count(); Benefit: open for extension It is easy to add new steps in the pipeline without touching anything else
- 29. list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#
- 30. list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#
- 31. Immutability and pipelines If you have immutable data you will probably need a pipeline
- 32. // Person is immutable var p = new Person("Scott","[email protected]",21); var p2 = p.WithName("Tom"); var p3 = p2.WithEmail("[email protected]"); var p4 = p3.WithAge(42); When each call returns a new object, it gets repetitive
- 33. // Person is immutable var p = new Person("Scott","[email protected]",21); p .WithName("Tom") .WithEmail("[email protected]") .WithAge(42); Pipelines make it look nicer
- 34. Pipes vs. Extension methods Pipes are more general
- 35. int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; }
- 36. int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; } int NestedCalls() { return Double(Square(Add1(5))); } How can I make this look nicer? How about a pipeline?
- 37. return 5 .Add1 .Square .Double; But this doesn't work, because these are not extension methods.
- 38. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); } Introducing "Pipe"
- 39. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); }
- 40. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); }
- 41. return 5 .Pipe(Add1) .Pipe(Square) .Pipe(Double); And now we can create a pipeline out of any* existing functions
- 42. int Add(int i, int j) { return i + j; } int Times(int i, int j) { return i * j; } A two parameter function
- 43. public static TOut Pipe<TIn, TParam, TOut> (this TIn input, Func<TIn, TParam, TOut> fn, TParam p1) { return fn(input, p1); }
- 44. int Add(int i, int j) { return i + j; } int Times(int i, int j) { return i * j; } public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2); } We can now create a pipeline out of any existing functions with parameters
- 45. public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2); } Why bother? Because now we get all the benefits of a pipeline
- 46. public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2) .Pipe(Add, 42) .Pipe(Square); } Why bother? Because now we get all the benefits of a pipeline, such as adding things to the pipeline easily (diffs look nicer too!)
- 47. let add x y = x + y let times x y = x * y 5 |> add 1 |> times 2 And here's what the same code looks like in F# F# uses pipelines everywhere!
- 48. Pipes vs. Extension methods, Part 2 Extension methods cannot be parameters
- 49. int StrategyPattern(... list, ... injectedFn) { return list .Select(x => x + 2) .injectedFn .Where(x => x > 3) .Count(); } We cant use a function parameter as an extension method My "strategy"
- 50. int StrategyPattern(... list, ... injectedFn) { return list .Select(x => x + 2) .Pipe(injectedFn) .Where(x => x > 3) .Count(); } But we can use a function parameter in a pipeline!
- 51. Pipelines in practice Part 3:
- 52. Three demonstrations • Pipeline-oriented Roman Numerals • Pipeline-oriented FizzBuzz • Pipeline-oriented web API
- 53. Roman Numerals
- 54. To Roman Numerals Task: convert an integer to roman numerals • 5 => "V" • 12 => "XII" • 107 => "CVII"
- 56. To Roman Numerals • Use the "tally" approach – Start with N copies of "I" – Replace five "I"s with a "V" – Replace two "V"s with a "X" – Replace five "X"s with a "L" – Replace two "L"s with a "C" – etc
- 57. To Roman Numerals number etc Replicate "I" Replace_IIIII_V Replace_VV_X Replace_XXXXX_L
- 58. string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example
- 59. string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example .Replace("LL", "C") .Replace("VIIII", "IX") .Replace("IIII", "IV") .Replace("LXXXX", "XC") .Replace("XXXX", "XL"); } Extend functionality without touching existing code
- 60. let toRomanNumerals n = String.replicate n "I" |> replace "IIIII" "V" |> replace "VV" "X" |> replace "XXXXX" "L" |> replace "LL" "C" // special cases |> replace "VIIII" "IX" |> replace "IIII" "IV" |> replace "LXXXX" "XC" |> replace "XXXX" "XL" F# example
- 61. FizzBuzz
- 62. FizzBuzz definition • Write a program that prints the numbers from 1 to N • But: – For multiples of three print "Fizz" instead – For multiples of five print "Buzz" instead – For multiples of both three and five print "FizzBuzz" instead.
- 63. for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example
- 64. for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example
- 65. Pipeline implementation number Handle 15 case
- 66. Pipeline implementation Handle 3 case number Handle 15 case
- 67. Pipeline implementation Handle 3 case Handle 5 case number Handle 15 case
- 68. Pipeline implementation Handle 3 case Handle 5 case number Answer Handle 15 case Final step
- 69. number Handle case Processed (e.g. "Fizz", "Buzz") Unprocessed (e.g. 2, 7, 13) record FizzBuzzData(string Output, int Number);
- 70. record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); }
- 71. record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); } Extension method
- 72. static string FizzBuzzPipeline(int i) { return new FizzBuzzData("", i) .Handle(15, "FizzBuzz") .Handle(3, "Fizz") .Handle(5, "Buzz") .FinalStep(); } static void FizzBuzz() { var words = Enumerable.Range(1, 30) .Select(FizzBuzzPipeline); Console.WriteLine(string.Join(",", words)); }
- 73. Demo: FizzBuzz in F# Extensions, Decorations and Parallelization
- 74. A pipeline-oriented web API
- 75. HttpContext Web Component Not Handled Handled
- 77. GET Define a component matches verb doesn't match verb HttpContext
- 78. route "/hello" Define a component matches route doesn't match route HttpContext
- 79. setStatusCode 200 Define a component Always succeeds HttpContext
- 80. GET route "/hello" >=> A special pipe for web components setStatusCode 200 >=>
- 81. Are you wondering how do we compose these?
- 82. Easy!
- 83. choose [ ] Picks first component that succeeds Define a new component
- 84. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" ] Pick first path that succeeds
- 85. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/bad" >=> BAD_REQUEST ] All the benefits of pipeline-oriented programming
- 86. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming
- 87. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming
- 88. Demo: A pipeline oriented web app For more on the web framework I'm using, search the internet for "F# Giraffe"
- 90. Benefit 4: Pipelines encourage good architecture
- 91. The "onion" architecture Core domain is pure, and all I/O is at the edges Core domain
- 92. Domain Logic I/O I/O The "onion" architecture Core domain is pure, and all I/O is at the edges
- 93. In a well-designed pipeline, all I/O is at the edges. Easy to enforce this with a pipeline-oriented approach
- 94. Database Database Load data Process it Save it
- 95. I/O in the middle of a pipeline Keep them separate
- 96. Benefit 5: Easier to test
- 98. This makes testing easy!
- 99. In Conclusion
- 100. Why bother? • Reusable components • Understandable – data flows in one direction • Extendable – add new parts without touching old code • Testable – parts can be tested in isolation • A different way of thinking – it's good for your brain to learn new things!
- 101. Databases Object-oriented programmng Pipeline-oriented programming Domain-driven design Does pipeline-oriented programming work for every situation? No! But it should be part of your toolkit
- 102. Pipeline Oriented Programming – Slides and video will be posted at • fsharpforfunandprofit.com/pipeline Related talks – "The Power of Composition" • fsharpforfunandprofit.com/composition – “Railway Oriented Programming" • fsharpforfunandprofit.com/rop Thanks! Twitter: @ScottWlaschin