Building and Incredible Machine with Pipelines and Generators in PHP (IPC Berlin)

Building an Incredible Machine
With Generators and Pipelines in PHP
by Dan Leech
@dantleech

/**
* @Revs(1000)
* @Iterations(10)
*/
class HashingBench
{
public function benchMd5()
{
md5('hello world');
}
public function benchSha1()
{
sha1('hello world');
}
}

Interpolated Execution
The results are now consistently inconsistent
PHPBench will report a high standard deviation, and you will
know to start again.

Instant Comparison
By interpolating tests we can provide faster
feedback to the user

Can I do this with Generators?

<?php
$iterator = new ArrayIterator([
‘key1’ => ‘value1’,
‘key2’ => ‘value2’,
]);
foreach ($iterator as $key => $value) {
echo “{$key} {$value}”;
}

AppendIterator
ArrayIterator
CachingIterator
CallbackFilterIterator
DirectoryIterator
EmptyIterator
FilesystemIterator
FilterIterator
AppendIterator
ArrayIterator
CachingIterator
CallbackFilterIterator
DirectoryIterator
EmptyIterator
FilesystemIterator
FilterIterator
GlobIterator
InfiniteIterator
IteratorIterator
LimitIterator
MultipleIterator

← Pseudo-type accepting arrays and Traversable (7.1)
← Cannot implement

<?php
function dump_many(iterable $iterable)
{
foreach ($iterable as $value) {
var_dump($value);
}
}
dump_many(['one', 'two', 'three']);
dump_many(new ArrayObject(['one', 'two', 'three']));
// string(3) "one"
// string(3) "two"
// string(5) "three"
// string(3) "one"
// string(3) "two"
// string(5) "three"

class MyAwesomeIterator implements Iterator {
private $myArray;
public function __construct(array $myArray) {
$this->myArray = $myArray;
}
function rewind() {
return reset($this->myArray);
}
function current() {
return current($this->myArray);
}
function key() {
return key($this->myArray);
}
function next() {
return next($this->myArray);
}
function valid() {
return key($this->myArray) !== null;
}

Generators are easy Iterators
That you cannot rewind
But that's OK, because they do cool stuff

final class Generator implements Iterator {
function rewind() {}
function valid() {}
function current() {}
function key() {}
function next() {}
function send($value) {}
function throw(Exception $exception) {}
function getReturn() {}
}

function valid() {}
function key() {}
function next() {}
}
Iterator

function valid() {}
function key() {}
function next() {}
}
Generator

function numbers(): Generator
{
yield "1";
yield "2";
yield "3";
}
foreach (numbers() as $number) {
echo $number;
}
// 123

<?php
function lines($fileName)
{
$h = fopen($fileName, 'r');
$lines = [];
while (false !== $line = fgets($h)) {
$lines[] = $line;
}
fclose($h);
return $lines;
}
foreach (lines() as $line) {
echo $line;
}
Example with standard array

<?php
function lines($fileName)
{
$h = fopen($fileName, 'r');
while (false !== $line = fgets($h)) {
yield $line;
}
fclose($h);
}
foreach (lines() as $line) {
echo $line;
// do something else
Example with generator

Generators make functions interruptible

send()
function myGenerator(): Generator
{
$data = yield;
yield $data;
}
echo myGenerator()->send('Goodbye');
// Goodbye
Fast forwards to first yield returns
the sent value

send()
<?php
function echoGenerator(): Generator
{
$data = yield;
$data = yield $data;
}
$generator = echoGenerator();
var_dump($generator->send('Hello'));
var_dump($generator->send('Goodbye'));
var_dump($generator->send('Adios'));

send()
string(5) "Hello"
NULL
NULL

throw()
{
try {
yield;
} catch (MyException $e) {
echo 'Hello!';
}
}
echo myGenerator()->throw(new MyException());
Exception thrown here

invalid generator
{
try {
yield;
} catch (MyException $e) {
echo 'Hello!';
}
}
$generator = myGenerator();
$generator->next();
echo myGenerator()->throw(new MyException());
Exception thrown here

getReturn
<?php
function hello(): Generator
{
yield 'hello';
yield 'goodbye';
return 'ciao';
}
$generator = hello();
echo $generator->getReturn();

PHP Fatal error: Uncaught Exception: Cannot get
return value of a generator that hasn't returned

getReturn
<?php
function hello(): Generator
{
yield 'hello';
yield 'goodbye';
return 'ciao';
}
$generator = hello();
$generator->next()
$generator->next();
echo $generator->getReturn();
// ciao

Generators will not return the return value
Unless you call getReturn()
After the generator has returned

Async Operations
function requestUrl($url): Generator {
$process = new Process("curl -I $url");
$process->start();
while ($process->isRunning()) {
echo $url . "n";
yield;
}
yield $process->getOutput();
}
WARNING: This code is terrible, please do not use it

Async Operations
$generators = [
requestUrl('https://inviqa.com'),
requestUrl('https://bbc.co.uk'),
];
while ($generators) {
foreach ($generators as $i => $gen) {
$result = $gen->current()
if ($result !== false) {
var_dump($result); // success!!!
}
$gen->next();
if ($gen->valid() === false) {
unset($generators[$i]);
}
}
WARNING: This code is terrible, please do not use it

https://www.inviqa.com
http://www.bbc.com

string(344) "HTTP/1.1 301 Moved Permanently
Date: Tue, 20 Feb 2018 21:21:23 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Tue, 20 Feb 2018 22:21:23 GMT
Location: http://inviqa.com/
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-
cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 3f0483ebdb3a2d6b-TXL
string(612) "HTTP/1.1 200 OK
Server: Apache
Content-Type: text/html
Expires: Tue, 20 Feb 2018 21:21:32 GMT
Content-Language: en
Etag: "25e4bf9a0519a562c8fb0a6379044365"
X-PAL-Host: pal143.back.live.telhc.local:80
Content-Length: 209756
Date: Tue, 20 Feb 2018 21:21:23 GMT
Connection: keep-alive
Set-Cookie: BBC-
UID=055ac8cca981056306da9a3821c959a1b1d799b4875494361aa037e2fe86c6080curl/7.55.1;
expires=Sat, 19-Feb-22 21:21:23 GMT; path=/; domain=.bbc.com
Cache-Control: private, max-age=60
X-Cache-Action: HIT
X-C"...

AMP
Amp is a non-blocking concurrency
framework for PHP. It provides an event
loop, promises and streams as a base for
asynchronous programming.
Promises in combination with generators
are used to build coroutines, which allow
writing asynchronous code just like
synchronous code, without any callbacks.
https://github.com/amphp/amp

<?php
function natural_numbers($number = 1): Generator
{
while (true) {
yield $number++;
}
}
foreach (numbers() as $number) {
echo $number . ' ';
}
// 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74
75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
108 109 110 111 112 113 114 115 116 117 118 ...

The number of iterations is no longer the concern
of the generator

Recursion
{
$count = yield;
while (true) {
$count = $count * 2;
$count = yield $count;
}
}
$generator = myGenerator();
$count = 1;
while (true) {
$count = $generator->send($count);
}

A pipeline is a sequence of stages.
Each stage has a specific job to do.

1)Make each program do one thing well. To do a new job, build afresh
rather than complicate old programs by adding new "features".
2)Expect the output of every program to become the input to another, as yet
unknown, program. Don't clutter output with extraneous information. Avoid
stringently columnar or binary input formats. Don't insist on interactive
input.
3)Design and build software, even operating systems, to be tried early,
ideally within weeks. Don't hesitate to throw away the clumsy parts and
rebuild them.
4)Use tools in preference to unskilled help to lighten a programming task,
even if you have to detour to build the tools and expect to throw some of
them out after you've finished using them.
Unix Philosophy

$ echo '<?php echo "hello";' | php
hello

This is a simple phraze this is
Spell Checker
$ echo 'This is a a simple phraze this is'
Command
Output

This
is
a
simple
phraze
this
is
Spell Checker
$ echo 'This is a simple phraze this is' |
words
Command
Output

this
is
a
simple
phraze
this
is
Spell Checker
$ echo 'This is a simple phraze' |
words |
lowercase
Command
Output

a
is
is
phraze
simple
this
this
Spell Checker
words |
lowercase |
sort
Command
Output

a
is
phraze
simple
this
Spell Checker
words |
lowercase |
sort |
unique
Command
Output

phraze
Spell Checker
words |
lowercase |
sort |
unique |
mismatch dictionary.txt
Command
Output

That wasn't GNU
Stage GNU
words fmt -1
lowercase tr '[:upper:]' '[:lower:]'
sort sort
unique uniq
mismatch dictionary.txt comm -23 - dictionary.txt

Working together + Specialisation

Input
Callable
Stage
Output
Stage
Stage

Input
Callable
Callable
Output
Callable
Callable function ($value) { return $value * 2; }
10
20
function ($value) { return $value +1; }
21
function ($value) { return $value * 2; }
42

$pipeline = (new Pipeline)
->pipe(function ($value) {
return $value * 2;
})
return $value +1;
})
return $value * 2;
})
;
$pipeline->process(10);

class TimesTwoStage {
public function __invoke($value) {
return $value * 2;
}
}
class AddOneStage {
public function __invoke($value) {
return $value + 1;
}
}

$pipeline = (new Pipeline)
->pipe(new TimesTwoStage())
->pipe(new AddOneStage())
->pipe(new TimesTwoStage());
$pipeline->process(10);

function Stage1(): Generator
{
$payload = yield;
while (true) {
$data = yield data;
}
}

Blog Posts
●
https://nikic.github.io/2012/12/22/Cooperative-multitasking-using-coroutines-in-PHP.html
●
https://blog.ircmaxell.com/2012/07/what-generators-can-do-for-you.html
● https://speakerdeck.com/chrispitt/cooperative-multitasking-with-generators
– https://www.youtube.com/watch?v=cY8FUhZvK7w
●
https://medium.com/@aaronweatherall/the-pipeline-pattern-for-fun-and-profit-9b5f43a98130
●
https://www.cise.ufl.edu/research/ParallelPatterns/PatternLanguage/AlgorithmStructure/Pipeline.htm
● https://markbakeruk.net/2016/01/19/a-functional-guide-to-cat-herding-with-php-generators/
● http://sergeyzhuk.me/2018/02/15/amp-coroutines/
●
https://medium.com/ifixit-engineering/functional-programming-with-php-generators-837a6c91b0e3
● https://www.entropywins.wtf/blog/2017/10/16/introduction-to-iterators-and-generators-in-php/
●
http://blog.kevingomez.fr/2016/01/20/use-cases-for-php-generators/

Videos
● Cooperative multitasking with Generators:
https://www.youtube.com/watch?v=cY8FUhZvK7w
● Pushing the limits of PHP with React:
https://www.youtube.com/watch?
v=fQxxm4vD8Ok&feature=youtu.be
● Herding Cats: https://vimeo.com/189755262
● What the heck is the event loop anyway?:
https://www.youtube.com/watch?v=8aGhZQkoFbQ

Use Generators NOW!
● Importing Data
● Data providers in PHPUnit (and PHPBench!)

Before
/**
* @dataProvider provideMultiplies
*/
function testMultiplies($v, $m, $expected) {
$result = $this->multiplier->multiply($v, $m);
$this->assertEquals($expected, $result);
}
public function provideMultiplies()
{
return [
'one times one' => [ 1, 1, 1 ],
'two times two' => [ 2, 2, 4 ],
];
}

After
/**
* @dataProvider provideMultiplies
*/
function testMultiplies($v, $m, $expected) {
$result = $this->multiplier->multiply($v, $m);
$this->assertEquals($expected, $result);
}
public function provideMultiplies()
{
yield 'one times one' => [ 1, 1, 1 ];
yield 'two times two' => [ 2, 2, 4 ];
}

● Saved 2 lines of code
● Looks nicer
● Not actually lazy ... but it could be!

Building and Incredible Machine with Pipelines and Generators in PHP (IPC Berlin)

More Related Content

What's hot (20)

Similar to Building and Incredible Machine with Pipelines and Generators in PHP (IPC Berlin) (20)

More from dantleech (7)

Recently uploaded (20)

Building and Incredible Machine with Pipelines and Generators in PHP (IPC Berlin)

Editor's Notes