Filesystems Lisbon 2018

Olá SymfonyConOlá SymfonyCon
Lisboa!Lisboa!

File Storage forFile Storage for
ModernModern PHPPHP ApplicationsApplications

Hi, I'm Frank!Hi, I'm Frank!
@frankdejonge@frankdejonge

Free-lance DeveloperFree-lance Developer
++
Free-software DeveloperFree-software Developer

I18n Routing in 4.1I18n Routing in 4.1
Routing Speedup in 3.2Routing Speedup in 3.2

EventSauce.ioEventSauce.io
Event-sourcing LibraryEvent-sourcing Library

FlysystemFlysystem
Filesystem AbstractionFilesystem Abstraction

50.000.000 installs!50.000.000 installs!
FlysystemFlysystem

The things I've seen...The things I've seen...The things I've seen...The things I've seen...The things I've seen...

File storage forFile storage for modernmodern
PHP applications.PHP applications.

It depends!It depends!
(and that's the problem)(and that's the problem)

1. Filesystems
2. Modern Apps & FS
3. Choosing an FS
4. Storage in Code

How not to fail atHow not to fail at
ﬁlesystemsﬁlesystems..

What is aWhat is a
ﬁlesystem?ﬁlesystem?

FILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHEREFILESYSTEMS EVERYWHERE

Common ProblemsCommon Problems
and/or Symptomsand/or Symptoms

Unstructured FilesystemUnstructured Filesystem
The "/uploads/" directory.The "/uploads/" directory.

Are these ﬁles used?Are these ﬁles used?

Can I delete these ﬁles?Can I delete these ﬁles?

Where did these ﬁlesWhere did these ﬁles
come from?come from?

Can we changeCan we change
ﬁlesystems?ﬁlesystems?

But is it fine?But is it fine?But is it fine?But is it fine?But is it fine?

Assumptions
Creating a feature

Assumptions
Creating a feature
Experience with tools
Feature created

Assumptions
Creating a feature
Feature created
Initial usage
Put in production

Assumptions
Creating a feature
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption

Assumptions
Creating a feature
Feature created
Initial usage
Put in production
Actual Usage
Feature Adoption
Experience
Actual Problems

What is aWhat is a
ﬁlesystem, actually?ﬁlesystem, actually?

A ﬁlesystem is a
system that controls
how data is stored
and retrieved.

A system is a
collection of parts,
conventions, and
interactions, forming
a complex whole.

So, what is aSo, what is a
ﬁlesystem?ﬁlesystem?

A ﬁlesystem is a system
that provides a collection
of parts, conventions, and
interactions, forming a
complex whole that
controls how data is
stored and retrieved.

Something you use toSomething you use to
store and retrieve data.store and retrieve data.

src/
Filesystem.php
AdapterInterface.php
tests/
Filesystem.php
AdapterInterface.php
src/Filesystem.php
src/AdapterInterface.php
tests/Filesystem.php
tests/AdapterInterface.php
Structure types:Structure types:
Nested or Linear

Nested filesystemsNested filesystems
Are very common.
Your desktop has it.
Have directories
which have files and more directories.

Linear filesystemLinear filesystem
Are like key/value stores
only have files
require a different approach.
are very "cloud"

Filesystem OfferingFilesystem Offering

FS = NASFS = NAS
AWS Volume |AWS Volume | Digital Ocean BlobDigital Ocean Blob
StorageStorage | Gluster FS || Gluster FS | CephCeph

FS + CDNFS + CDN
AWS S3 |AWS S3 | Digital Ocean SpacesDigital Ocean Spaces
Rackspace Cloud Files |Rackspace Cloud Files | GoogleGoogle
Cloud StorageCloud Storage

FS + SharingFS + Sharing
Dropbox |Dropbox | Box.comBox.com || Google DriveGoogle Drive
Microsoft OneDriveMicrosoft OneDrive

FS + Not Really a FSFS + Not Really a FS
WebDAV |WebDAV | MongoDB GridFSMongoDB GridFS
PostgreSQL Large Object StoragePostgreSQL Large Object Storage

What isWhat is NOTNOT
a ﬁlesystem?a ﬁlesystem?

Content Distribution NetworkContent Distribution Network
Streaming ServerStreaming Server
File Sharing ServiceFile Sharing Service
Media ServerMedia Server
File ManagerFile Manager
Web ServerWeb Server
DatabaseDatabase

Ryan Singer (2010)Ryan Singer (2010)
So much complexity
in software comes
from trying to make
one thing do two
things.

Frank de Jonge (right now)Frank de Jonge (right now)
Stop trying to make a
ﬁlesystem do 100
things.

Create
Read
Update
Delete
Inspect
List

Filesystems and PHP.Filesystems and PHP.

$result = file_put_contents($location, $contents);
$contents = file_get_contents($location);
unlink($location);
$timestamp = mtime($location);
$mimetype = (new Finfo(FILEINFO_MIME_TYPE))->file($location);

glob($location . '/*'); // ?
scandir($location); // ?
readdir(opendir($location));
// Or...
array_map('normalize_fileinfo',
iterator_to_array(new DirectoryIterator($location)));
// Or...
array_map('normalize_fileinfo',
iterator_to_array(new RecursiveIteratorIterator(
new RecursiveDirectoryIterator($location))));

$resource = fopen($src, 'r+');
$handle = fopen($dest, 'w+');
while ( ! feof($resource)) {
$chunk = fread($resource, 1024);
fwrite($handle, $chunk, 1024);
}
$result = fclose($handle);
fclose($resource);

$resource = fopen($src, 'r+');
if ($resource === false) handle_error(); //
$handle = fopen($dest, 'w+');
if ($handle === false) handle_error(); //
while ( ! feof($resource)) {
$chunk = fread($resource, 1024);
if ($chunk === false) handle_error(); //
$writeResult = fwrite($handle, $chunk, 1024);
if ($writeResult === false) handle_error(); //
}
$result = fclose($handle);
if ($result === false) handle_error(); //
$closeResult = fclose($resource);
if ($closeResult === false) handle_error(); //

/**
* Why not just use .... ?
*/
copy($src, $dest);

use Flysystemuse Flysystem
for ﬁle storagefor ﬁle storage

use LeagueFlysystemAdapterLocal;
$adapter = new Local(__DIR__.'/path/to/dir');
$filesystem = new Filesystem($adapter);
$filesystem->write($destination, $contents);
$filesystem->read($location);
$filesystem->update($destination, $contents);
$filesystem->listContents($location, $recursive);
$filesystem->delete($location);
$filesystem->getSize($location);

use LeagueFlysystemAwsS3v2AwsS3Adapter;
$adapter = new AwsS3Adapter($client, 'bucket-name');

use LeagueFlysystemAdapterFtp;
$adapter = new Ftp($ftpSettings);

$resource = $remoteFs->readStream('some/path.mp4');
$localFs->writeStream('another/path.mp4', $resource);
fclose($resource);

$resource = $remoteFs->readStream('some/path.mp4');
$localFs->writeStream('another/path.mp4', $resource);
fclose($resource);
// do stuff
$resource = $localFs->readStream('another/path.mp4');
$remoteFs->writeStream('some/path.mp4', $resource);
fclose($resource);

$resource = $azure->readStream('some/path.mp4');
$aws->writeStream('another/path.mp4', $resource);
fclose($resource);

domain/concern.txt
domain/concern.txt
/conﬁguration/
/new/path/

// Configuration
$adapter = new Local('/configuration/');
// Usage
$filesystem->writeStream('financial-reports/latest.csv', $resource);

// Configuration
$adapter = new Local('/new/path/');
// Usage

use LeagueFlysystemAwsS3v2AwsS3Adapter;
// Configuration
$adapter = new AwsS3Adapter($client, 'bucket-name');
// Usage

Flysystem =Flysystem =
No Vendor Lock-inNo Vendor Lock-in
Isolation +Isolation +
Encapsulation +Encapsulation +
Predictability +Predictability +
PortabilityPortability

Don't use Flysystem for everything.Don't use Flysystem for everything.

Choosing anotherChoosing another
FilesystemFilesystem

Using the localUsing the local
ﬁlesystem isﬁlesystem is

Why?Why?
We want to scale (horizontally).
... so, we want stateless apps.
We don't have to serve ﬁles ourselves.
Share ﬁles across multiple application.

We're not askingWe're not asking
the rightthe right questionsquestions..

On Premise?On Premise?On Premise?On Premise?On Premise?

"Cloud""Cloud""Cloud""Cloud""Cloud"
Still a Computer™Still a Computer™Still a Computer™Still a Computer™Still a Computer™

Who will doWho will do
maintenance?maintenance?

How will weHow will we
use the ﬁles?use the ﬁles?

Why are weWhy are we
going to store it?going to store it?

How long are weHow long are we
going to keep the ﬁles?going to keep the ﬁles?

How big willHow big will
the ﬁles be?the ﬁles be?

Needs > CapabilitiesNeeds > Capabilities

NeedNeed
Something you requireSomething you require

CapabilityCapability
What something can doWhat something can do

- Agile Manifesto- Agile Manifesto
Working Software
over Comprehensive
Documentation

ADRADR
Architecture Decision RecordArchitecture Decision Record

ContextContext
The team uses AWS / AzureThe team uses AWS / Azure

ContextContext
We expect massive storage growthWe expect massive storage growth

RequirementsRequirements
Data must be stored on premiseData must be stored on premise

RequirementsRequirements
Data must be stored in the EUData must be stored in the EU

DecisionDecision
We'll use AWS S3 / Azure BlobWe'll use AWS S3 / Azure Blob
StorageStorage

ConsequencesConsequences
Operations of ﬁles will requireOperations of ﬁles will require
download and re-upload.download and re-upload.

Share ResponsibilitiesShare Responsibilities

Create Use-Case
Driven Storage
Models

interface ProfilePictureStorage {
/**
* @param resource $profilePicture
*/
public function store(User $user, $profilePicture);
}

interface FinancialReportStorage {
/**
* @param resource $reportFile
*/
public function store(Organization $organization, $reportFile);
}

class FilesystemFinancialReportStorage
implements FinancialReportStorage {
private $filesystem;
public function __construct(FilesystemInterface $filesystem) {
$this->filesystem = $filesystem;
}
public function store(Organization $organization, $reportFile)
{
$path = sprintf('%s/%s/financial-report.csv',
$organization->id()->toString(),
date('Y/m/d'));
$this->filesystem->writeStream($path, $reportFile);
}
}

Isolate your use-case.Isolate your use-case.

Document experiencesDocument experiences
When stuff goesWhen stuff goes

(Blameless) Post-Mortem(Blameless) Post-Mortem

Storage was unavailableStorage was unavailable
Full diskFull disk
Deleted old filesDeleted old files
manuallymanually
Automated cleanupAutomated cleanup
routinesroutines
# What happened# What happened
# What caused it# What caused it
# What fixed it# What fixed it
# How to prevent# How to prevent

Documents requirementsDocuments requirements

Document decisionsDocument decisions
(using ADR)(using ADR)

CreateCreate speciﬁcspeciﬁc storagestorage
classes per use-caseclasses per use-case

Use generic solutions forUse generic solutions for
generic problemsgeneric problems

Use special solutions forUse special solutions for
special problemsspecial problems

Gluster FSGluster FS
on Kuberneteson Kubernetes

Persistent VolumePersistent Volume
on Gluster FSon Gluster FS

AA
ServiceService
with PersistentVolumewith PersistentVolume
on Gluster FSon Gluster FS

So what do I do?So what do I do?

Thank you for listening!Thank you for listening!
@frankdejonge@frankdejonge

Filesystems Lisbon 2018

Recommended

More Related Content

What's hot (18)

Similar to Filesystems Lisbon 2018 (20)

Recently uploaded (20)

Filesystems Lisbon 2018