FleetDB: A Schema-Free Database in Clojure
1 like721 views
FleetDB is a schema-free database built in Clojure that aims to optimize for agile development. It implements a declarative query planner and executor to operate over database representations as Clojure data structures. The core database functions are wrapped by additional layers that provide identity, durability, and a JSON client API. The source code is relatively small at around 1300 lines thanks to Clojure's powerful data structures and functional programming model.
![Client Quickstart
(use ’fleetdb.client)
(def client (connect))
(client ["ping"])
=> "pong"](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-6-320.jpg)
![Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-7-320.jpg)
![Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])
=> 1](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-8-320.jpg)
![Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-9-320.jpg)
![Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])
=> [{"id" 1 "owner" "Eve" "credits" 100}]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-10-320.jpg)
![Select with Conditions
(client
["select" "accounts"
{"where" [">=" "credits" 150]}])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-11-320.jpg)
![Select with Order and Limit
(client
["select" "accounts"
{"order" ["credits" "asc"] "limit" 2}])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-12-320.jpg)
![Update
(client
["update" "accounts" {"credits" 105}
{"where" ["=" "owner" "Eve"]}])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-13-320.jpg)
![Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-14-320.jpg)
![Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["filter" ["=" "owner" "Eve"]
["collection-scan" "accounts"]]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-15-320.jpg)
![Create Index
(client
["create-index" "accounts" "owner"])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-16-320.jpg)
![Explain (II)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["index-lookup" ["accounts" "owner" "Eve"]]]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-17-320.jpg)
![Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-19-320.jpg)
![Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-20-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-22-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-23-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-24-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-38-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-39-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-40-320.jpg)
![fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-43-320.jpg)
![fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))](https://image.slidesharecdn.com/talk-100123191406-phpapp02/85/FleetDB-A-Schema-Free-Database-in-Clojure-44-320.jpg)
FleetDB: A Schema-Free Database in Clojure
- 1. FleetDB A Schema-Free Database in Clojure Mark McGranaghan January 23, 2010
- 3. Motivation
- 4. Motivation optimize for agile development
- 5. Walkthrough Client Quickstart Basic Queries Concurrency Tools
- 6. Client Quickstart (use ’fleetdb.client) (def client (connect)) (client ["ping"]) => "pong"
- 7. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}])
- 8. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}]) => 1
- 9. Select (client ["select" "accounts" {"where" ["=" "id" 1]}])
- 10. Select (client ["select" "accounts" {"where" ["=" "id" 1]}]) => [{"id" 1 "owner" "Eve" "credits" 100}]
- 11. Select with Conditions (client ["select" "accounts" {"where" [">=" "credits" 150]}])
- 12. Select with Order and Limit (client ["select" "accounts" {"order" ["credits" "asc"] "limit" 2}])
- 13. Update (client ["update" "accounts" {"credits" 105} {"where" ["=" "owner" "Eve"]}])
- 14. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]])
- 15. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["filter" ["=" "owner" "Eve"] ["collection-scan" "accounts"]]
- 16. Create Index (client ["create-index" "accounts" "owner"])
- 17. Explain (II) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["index-lookup" ["accounts" "owner" "Eve"]]]
- 18. Multi-Write
- 19. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]])
- 20. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]]) => [result-1 result-2 ...]
- 21. Checked-Write
- 22. Checked-Write (client ["checked-write" read-query expected-read-result write-query])
- 23. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result]
- 24. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result] => [false actual-read-result]
- 25. Implementation Background Organization Key ideas
- 26. Background http://clojure.org/data_structures http://clojure.org/sequences http://clojure.org/state
- 27. Organization
- 28. Organization fleetdb.core: pure functions
- 29. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability
- 30. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability fleetdb.server: network interface
- 31. fleetdb.core Pure functions
- 32. fleetdb.core Pure functions Databases as Clojure data structures
- 33. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result
- 34. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result ‘Write’: db + query → result + new db
- 36. fleetdb.core Query Planner Implements declarative queries
- 37. fleetdb.core Query Planner Implements declarative queries Database + query → plan
- 38. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}]
- 39. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
- 40. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] ["index-lookup" ["accounts" "owner" "Eve"]]
- 42. fleetdb.core Query Executor Database + plan → result
- 43. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
- 44. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] (filter (fn [r] (= (r "owner") "Eve")) (vals (:rmap (db "accounts"))))
- 45. fleetdb.embedded Wraps fleetdb.core
- 46. fleetdb.embedded Wraps fleetdb.core Adds identity and durability
- 47. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms
- 48. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms Append-only log
- 50. fleetdb.embedded Read Path Dereference database atom
- 51. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core
- 52. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core Return result
- 54. fleetdb.embedded Write Path Enter fair lock
- 55. fleetdb.embedded Write Path Enter fair lock Dereference database atom
- 56. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core
- 57. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log
- 58. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value
- 59. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value Return result
- 60. fleetdb.server Wraps fleetdb.embedded
- 61. fleetdb.server Wraps fleetdb.embedded Adds JSON client API
- 63. Aside: Source Size Lines of Code core 630 embedded 130 server 120 lint 280 utilities 140 total 1300
- 64. Takeaways
- 65. Takeaways Clojure’s data structures are awesome
- 66. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software
- 67. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software Try FleetDB!
- 68. Learn More http://FleetDB.org http://github.com/mmcgrana