PyConFR 2014 - DEPOT, Story of a file.write() gone wrong
2 likes20,857 views
DEPOT ( http://depot.readthedocs.org/en/latest/ ) is a file storage framework born from the experience on a project that saved a lot of files on disk, until the day it went online and the customer system engineering team diceded to switch to Heroku, which doesn't support storing files on disk. Like SQLAlchemy makes possible to switch your storage on the fly without touching code, DEPOT aims at making so possible for files and even use multiple different storages together.
![Easy to Extend
● Custom attachments can be easily created
UploadedFileField(upload_type=UploadedImageWithMaxSize)
● Filters can be applied to attachments
UploadedFileField(filters=[WithThumbnailFilter()])
● Multiple filters can be applied (rescale
image and create thumbnails)](https://image.slidesharecdn.com/pyconfr2014-depotstoryofafile-141025075509-conversion-gate02/85/PyConFR-2014-DEPOT-Story-of-a-file-write-gone-wrong-19-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to PyConFR 2014 - DEPOT, Story of a file.write() gone wrong (20)
Ad
More from Alessandro Molina (15)
Ad
Recently uploaded (20)
PyConFR 2014 - DEPOT, Story of a file.write() gone wrong
- 1. DEPOT - STORY OF A FILE.WRITE() GONE WRONG Alessandro Molina @__amol__ [email protected]
- 2. Who am I ● CTO @ AXANT.it, mostly Python company (with some iOS and Android) ● TurboGears2 dev team member ● Contributions to web world python libraries ○ MING MongoDB ODM ○ ToscaWidgets2 ○ Formencode
- 3. Background ● Everything starts from a project which was just a POT with budget constraint. ● Obviously it became the final product. ● It saved and updated a lot of files, mostly images.
- 4. Technologies. ● Short on budget: cloud storage was not an available choice ● Short on time: developers choose to just store everything on disk and rely on nginx to serve them in a good enough manner
- 5. The Technical Consultant ● Customer had a technical leader that enforced deployment decisions. ● Customer decided production environment three days before the “go live” ● Due to limited budget he decided they were not going to rent a server.
- 6. The product owner choice
- 7. Murphy Law ● They went for Heroku free plan as PaaS ● Heroku doesn’t support storing files on disk ● The whole software did store files on disk
- 8. Ooops
- 9. Panic ● The day before launch, team rewrote 30% of the software to switch saving files from disk to GridFS (app was mongodb based) ● It was an huge hack based on monkeypatching the attachment classes ● It went online with practically no testing on the field.
- 10. The day after ● After emergency has been solved it was clear that we needed a better way to handle such issues. ● We decided to create a tool to solve the issue independently from the web development framekwork in use
- 12. DEPOT Requirements ● Permit choosing between multiple storage engines just by changing a configuration file ● Permit switching storage engine at runtime without breaking past files ● Permit to concurrently use multiple storages
- 13. Multiple Storages ● One “default” storage, any other storage can be promoted to default, anytime. ● When uploading a file it goes to the default storage unless otherwise specified. ● Each storage has a name, files can be uniquely identified among storages by storage_name/fileid.
- 14. DepotManager ● The DepotManager is the single interface to DEPOT. ● It tracks the active storages, the default one, and the WSGI middleware. ● To work on a storage just get it from the DepotManager.
- 15. Easy to Use ● Simple things should be simple from depot.manager import DepotManager # Configure a *default* depot to store files on MongoDB GridFS DepotManager.configure('default', { 'depot.backend': 'depot.io.gridfs.GridFSStorage', 'depot.mongouri': 'mongodb://localhost/db' }) depot = DepotManager.get() # Save the file and get the fileid fileid = depot.create(open('/tmp/file.png')) # Get the file back stored_file = depot.get(fileid) print stored_file.filename print stored_file.content_type
- 16. With Batteries ● Complex things should be straightforward from depot.fields.sqlalchemy import UploadedFileField from depot.fields.specialized.image import UploadedImageWithThumb class Document(Base): __tablename__ = 'document' uid = Column(Integer, autoincrement=True, primary_key=True) name = Column(Unicode(16), unique=True) # photo field will automatically generate thumbnail photo = Column(UploadedFileField(upload_type=UploadedImageWithThumb)) # Store documents with attached files, the source can be a file or bytes doc = Document(name=u'Foo', content=b'TEXT CONTENT STORED AS FILE', photo=open('/tmp/file.png'))
- 17. No environment constraints ● Attachment field for SQLAlchemy ● Attachment field for MongoDB ● Bultin support for S3, LocalFiles and GridFS ● Delivery of files happens on a plain WSGI middleware compatible with any web framework.
- 18. Copes with Database ● Transactions rollback should delete newly uploaded files and recover the previous ones. ● Deleting an item deletes attached files (unless rollback happens)
- 19. Easy to Extend ● Custom attachments can be easily created UploadedFileField(upload_type=UploadedImageWithMaxSize) ● Filters can be applied to attachments UploadedFileField(filters=[WithThumbnailFilter()]) ● Multiple filters can be applied (rescale image and create thumbnails)
- 20. Custom Attachments ● Attachment Classes are in charge of storing the actually uploaded file ● They can change the file before it’s uploaded. ● They can add additional data and even behaviours to the file.
- 21. Filters ● Each attachment can have multiple filters ● They run after upload, so they can add metadata or generate new files but not replace the original one. ● They can store additional metadata with the file, but not behaviours (methods).
- 23. Web Application Friendly ● Need to serve stored files? Just mount DepotManager.make_middleware around your app and start serving them. ● If files are stored on a backend that supports HTTP, the user will be permanently redirected there by the middleware instead of serving files itself.
- 24. Made for the Web ● Storage backends can provide public url for any CDN ● File information common in HTTP are provided as properties out of the box ○ content_type ○ last_modified ○ content_length ○ filename
- 25. Feel free to try it! ● Python 2.6, 2.7, 3.2, 3.3 and 3.4 ● pip install filedepot ● Fully Documented https://depot. readthedocs.org ● Tested with 100% coverage https://travis-ci.org/amol-/depot
- 26. Questions?