Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery

Editor's Notes

• #2: Hi, good morning. Welcome to my talk.
• #4: My name is Tatsuhiko Miyagawa.
• #5: I work for a company called Six Apart in San Francisco. It’s a company behind many blogging software like Movable Type, Vox and TypePad.
• #6: I mostly work on TypePad and recently we shipped new services like TypePad Connect and TypePad Profile. I did most of the backend code for those products.
• #7: This is my CPAN/PAUSE ID and you might have already known my name by:
• #8: the 154 modules I wrote and uploaded on CPAN. Some of these might look familiar to you.
• #9: miyagawa is also my twitter ID, and I’ll post links to the slides of this talk to Twitter later. Since this is 20 minute talk and there are many things to talk about, it’ll be pretty fast presentation, you don’t need to take notes or remember URLs and what not. Just look at the slides later online.
• #10: Let me give you a quick intro of the application called “Remedie” I’ve been working on recently like half an year. I’ll use this application as an example app today.
• #12: So, long story short, I wanted to build some Desktop applications, with nice UI rather than Command Line interface.
• #13: There might be many options to build a desktop GUI application. Some MFC/.NET microsoft windows thing. That’s not something you want to learn in 2009.
• #14: wxWidgets. I think this is a cross platform library to build a GUI application that runs on Windows, Mac and Linux and there’s a perl binding. Padre, PerlIDE actually uses this to build the Perl editor. Might not be a bad choice if you want to build an editor.
• #15: Objective C and Cocoa API is the best way to build some beautiful looking applications that run on Mac OS X. I guess it’s worth investing your time learning these technologies since you can leverage that to build iPhone applications for instance. And there’re also scripting language glues for Python and Ruby.
• #16: Adobe AIR would probably be the best choice if you already know some ActionScript and want to build a nice looking GUI app that runs on major platforms.
• #17: So, all of these options have pros and cons. Some are platform specific and some things need a completely different paradigm and I would need a lot of time learning those stuff. I’m a Perl hacker and do web stuff every day. I know a lot of Perl, and little bit of JavaScript.
• #19: Why not just build a web application and ship it as a desktop application.
• #20: Remember when Steve Jobs announced their initial developer SDK for iPhone in 2007. It was Web 2.0 Ajax application that has an access to iPhone services via sandboxed JavaScript APIs.
• #21: Palm Pre has this operating system called WebOS and its SDK called Mojo SDK allows you to write an application with HTML, CSS and JavaScript.
• #22: And there’s also a project called PhoneGap that allows you to do the same thing against iPhone, Android and BlackBerry: PhoneGap gives you an abstract wrapper API to access phone device functionalities through JavaScript so you can write an application using HTML, CSS and JavaScript for these devices. Luke used this framework to write his YAPC schedule app for Android.
• #23: And now HTML5 is coming on its way, with lots of features and APIs to access stuff like Geo location, native video embeds and local storage access.
• #24: It’s coming, but it hasn’t fully come yet. It’s definitely the future, but not available right now as its final form, in 2009. So what can fill this “gap”, like Phone Gap does?
• #25: It’s micro web app, or desktop web applications.
• #26: So let’s get back to this Remedie example and give you a quick demo. This is a pluggable, perl-based media center application that allows you to subscribe to virtually ANY video sites with RSS feeds, or even without RSS feeds if you write plugins. And you can watch videos or photos inline or using external player like QuickTime with single, pretty user interface.
• #30: Amazing? Cool. Let’s talk about how I built this application using which tools and modules. That’s probably what you’re interested in.
• #33: HTTP::Engine is based on Catalyst::Engine. It’s an extraction from Catalyst’s HTTP request and response handling code so that you can use it outside Catalyst to write a web application that runs under CGI, FastCGI or standalone server or mod_perl without changing anything. You can also use this to build your own micro web framework.
• #36: This is the callback function you need to bind the request. It takes an HTTP::Engine::Request object, which has a nice set of APIs to access request headers, cookies and parameters and all of that, and your callback is supposed to return the Response object with headers and body. That’s it.
• #37: It already has many interfaces supported, mostly ported over from Catalyst. Standalone, POE for event loop asynchronous process, FastCGI and mod_perl. Very nice.
• #41: So here, we’re making a desktop application. We don’t need a separate web server to serve millions of page views in a minute, so no need for FastCGI or mod_perl thing, right? Let’s just use ServerSimple, or POE if you want non-blocking processing.
• #42: And here are the things your callback handler is supposed to do. First of all, you need to serve static files to build the user interface. That will be HTML, CSS and JavaScript files.
• #45: HTTP::Engine has a extension framework called Middleware and they probably have static file serving code, so I’d probably switch to that instead of doing this file manipulation directly, which is kind of insecure.
• #46: The other thing your controller is supposed to is to implement Ajax backend actions with REST-JSON.
• #48: Note that this is a simplified code. And yes this is a little insecure.
• #49: And as I said, the construction to map HTTP request to internal API call is kind of crufty and insecure.
• #50: There are lots of solution to this, notably Path::Router, Dispatcher and HTTP::Dispatcher. They’re from best practical and Infinity Interactive, so pick whichever company you like. JSORB is also from infinity interactive and does more robust and kind of complicated REST-JSON mappings to call methods in the backend from JS client.
• #51: And lastly those REST APIs are vulnerable to Cross Site Request Forgery. If a hacker knows which port you’re running your application on, they can just link or send form POST to your endpoint to do whatever they want, using your identity.
• #52: Some authentication might help, but cookies or basic auth don’t because it’s CSRF. In Remedie we add some special header to the normal POST request, in jQuery client, so CSRF attacks will be just ignored. Switching to JSONRPC instead of normal GET/POST could solve that problem too.
• #53: And Bonjour. This is something I implemented last week and NOT the feature of HTTP::Engine yet, but when Remedie server launches it also publishes its hostname and port and machine name etc. over multicast DNS (aka Bonjour).
• #54: Will be useful for auto-discovery by clients, or share the list of channels with someone in your local networks.
• #65: Lots of people would have opinions which SQL database to use, for your web applications.
• #66: Usually you would pick one of MySQL or PostgreSQL, to build some scalable web sites with complex business logic to run against data stored on SQL server.
• #67: SQLite is a flat file based, portable SQL engine. It doesn’t implement all of ANSI SQL features, but very fast and useful for embedded software. There will be concurrency problems like locking if you have multiple processes opening the same SQLite database, but
• #68: So it’s best for desktop applications, because you’ll only have one process to open the same database, even if it’s a micro desktop web application that we’re talking about now.
• #69: Actually, software like Firefox, OS X’s Mail.app and iCal all use SQLite as its database.
• #70: Another reason that it’s good for desktop apps is the end-users don’t want to and need to run SQL server processes like MySQL or postgreSQL.
• #71: Also, if your use SQLite as a backend for your desktop app, backing up is easy. And you can synchronize your database between two machines like Home and Work using tools like Dropbox. Very nice.
• #72: Let’s talk about DB schema.
• #73: Actually let’s NOT talk about DB schema. You probably don’t need a DBA to design Desktop application. It depends on the size of data set your app is trying to handle, but most of the time, just use as simple schema as possible, to make it flexible and easily extensible.
• #74: Remedie’s schema is pretty simple. It has just a few index columns, like id and names, and then column called ‘props’ where we store all the properties as a JSON encoded object, or key-value pairs.
• #77: Key value pair is hot.
• #78: There are many document database, or key-value database systems like Couch, Mongo or TokyoTyrant and they actually need servers again, like MySQL or postgres.
• #79: So the idea is the same: we use SQLite to store arbitrary key-value pairs with some indexes.
• #80: How do we abstract those Database access. We need ORM. Another fight here which ORM to use.
• #81: Well, you can use anything you like. I don’t have strong opinion about this right now.
• #82: You can use DBIC or RDBO.
• #83: Remedie uses RDBO right now but there’s no special reason for that. I just wanted some excuse to try something different because I always use Catalyst, Data::ObjectDriver, YUI and Template Toolkit at work and for this desktop app I wanted to try the whole new stack to build a micro web app.
• #84: KiokuDB is one of the hottest technologies in the Perl these days, and there were lots of talks mentioning about it in this YAPC.
• #85: KiokuDB can actually do with its DBI backend to save key-value JSPON data in the SQLite database. So i think Yuval and maybe Stevan are interested in porting my Remedie RDBO code over to Kioku over the next few days.
• #87: So for this desktop web app thing, I would’t introduce anything special from the normal jQuery usage for your web applications. Just use HTML and CSS to design the UI.
• #88: You better not write HTML by hand, because your app is truly dynamic. You better manipulate the DOM object and use jQuery’s Ajax method to do Ajax JSON calls to the backend.
• #90: There are couple of templating plugins for jQuery but I chose jQuery.flydom. which is kind of similar to Template::Declare in Perl.
• #92: jQuery.hotkeys is a plugin for jQuery to enable hotkeys or keyboard shortcuts, like vi-style navigation, or using arrow keys to control video playback.
• #93: The usage is pretty simple. It overrides the original ‘keydown’ event so that you can specify which keyboard that it listens to, and then assign a callback to the pressed key.
• #94: jQuery.contextMenu is a plugin to enable some right-click or control-click menu so your web application actually looks like a desktop application.
• #96: jQuery corners is a plugin to do round corners, pretty important for Web 2.0 sites, and for desktop applications, to implement something like this:
• #97: this badge-like overlay is implemented using some hover divs with rounded corners.
• #98: jQuery has this event system that you can bind callbacks to browser events, like mouse over, or DOM ready, or link clicked, but you can also use the event system to trigger your app specific custom events and bind callbacks to them. I heavily use this system to make my JS like a real GUI application, because with GUI application most of them are coded in Event Loops.
• #103: jQuery UI is an extension to jQuery that implements some UI stuff.
• #104: Currently, we only use jQuery UI to implement the drag & drop stuff that you saw in the demo. Maybe I could use other features of jQuery UI to do some more fancy stuff.
• #105: There are many other jQuery plugins that I won’t talk about today but you might be interested in, to implement some desktop-app like features.
• #107: That’s it for jQuery. Let’s get back to this Desktop app thing again.
• #108: This desktop web app sounds sexy, but it’s actually just Client-Server model, right?
• #109: So we have 2 “apps” basically. One is the client and the other is the server. Let’s make the client as a real application. In this case client is a browser.
• #110: There’re tools to make Site-Specific Browser, or SSB in short.
• #111: Fluid and Prism. Fluid is Safari and Prism is Firefox.
• #112: This is how Fluid apps would look like. They’re just another browser (Safari) process that are specific to the site, like Github or Gmail or Flickr. You can use Fluid to build a site specific browser, to your application running on localhost:9999 or whatever port.
• #113: The users of your desktop app will be able to customize the frontend UI using Userscripts, like Greasemonkey for Prism and GreaseKit for Fluid. Users can also skin their app using Userstyles. So you don’t need to implement those plugin mechanisms or anything like that, to allow users to customize their app. SSB can do this already.
• #114: Fluid also allows you to extend your app and look more like a desktop app, so your app notification can be hooked to Growl, instead of jQuery.jGrowl that I mentioned, and has an access to Dock menu, like this.
• #116: Remedie’s RPC is implemented as an JSON REST API, so client and server are basically decoupled. There’s no tight connection between front-end app and backend micro web server APIs.
• #117: So theoretically, you can have more client applications other than normal browsers like Safari or Firefox, that talk to the same backend server over API. And you have more “views” to the application.
• #118: For example, iPhone.
• #123: Finally, we can actually make the Server as an app as well.
• #124: by packaging the server process as a bundle.
• #125: I used local::lib the excellent CPAN module to have application specific library path and install all dependencies locally inside the app bundle.
• #126: You can also use best practical’s shipwright to do the same thing, with slightly different approach.
• #127: And for Mac OS X, I used a tool called Platypus, to bundle those scripts, libraries and dependent CPAN modules in a .app bundle. it runs everywhere under Leopard, no CPAN installation required.