FleetDB A Schema-Free Database in Clojure
2 likes1,164 views
FleetDB is a schema-free database built in Clojure that aims to optimize for agile development. It implements databases as Clojure data structures and uses pure functions to handle reads and writes. The core library contains query planning and execution logic. FleetDB adds durability by storing databases in atoms and appending queries to a log. It also includes an embedded server that exposes a JSON client API. At around 1300 lines of code, it leverages Clojure's data structures to provide a full-featured but compact database system.
![Client Quickstart
(use ’fleetdb.client)
(def client (connect))
(client ["ping"])
=> "pong"](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-6-320.jpg)
![Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-7-320.jpg)
![Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])
=> 1](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-8-320.jpg)
![Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-9-320.jpg)
![Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])
=> [{"id" 1 "owner" "Eve" "credits" 100}]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-10-320.jpg)
![Select with Conditions
(client
["select" "accounts"
{"where" [">=" "credits" 150]}])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-11-320.jpg)
![Select with Order and Limit
(client
["select" "accounts"
{"order" ["credits" "asc"] "limit" 2}])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-12-320.jpg)
![Update
(client
["update" "accounts" {"credits" 105}
{"where" ["=" "owner" "Eve"]}])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-13-320.jpg)
![Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-14-320.jpg)
![Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["filter" ["=" "owner" "Eve"]
["collection-scan" "accounts"]]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-15-320.jpg)
![Create Index
(client
["create-index" "accounts" "owner"])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-16-320.jpg)
![Explain (II)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["index-lookup" ["accounts" "owner" "Eve"]]]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-17-320.jpg)
![Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-19-320.jpg)
![Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-20-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-22-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-23-320.jpg)
![Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-24-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-38-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-39-320.jpg)
![fleetdb.core Query Planner
Implements declarative queries
Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-40-320.jpg)
![fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-43-320.jpg)
![fleetdb.core Query Executor
Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))](https://image.slidesharecdn.com/fleetdb20100107baclojureusergroup-100131101353-phpapp01/85/FleetDB-A-Schema-Free-Database-in-Clojure-44-320.jpg)
FleetDB A Schema-Free Database in Clojure
- 1. FleetDB A Schema-Free Database in Clojure Mark McGranaghan January 8, 2010
- 3. Motivation
- 4. Motivation optimize for agile development
- 5. Walkthrough Client Quickstart Basic Queries Concurrency Tools
- 6. Client Quickstart (use ’fleetdb.client) (def client (connect)) (client ["ping"]) => "pong"
- 7. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}])
- 8. Insert (client ["insert" "accounts" {"id" 1 "owner" "Eve" "credits" 100}]) => 1
- 9. Select (client ["select" "accounts" {"where" ["=" "id" 1]}])
- 10. Select (client ["select" "accounts" {"where" ["=" "id" 1]}]) => [{"id" 1 "owner" "Eve" "credits" 100}]
- 11. Select with Conditions (client ["select" "accounts" {"where" [">=" "credits" 150]}])
- 12. Select with Order and Limit (client ["select" "accounts" {"order" ["credits" "asc"] "limit" 2}])
- 13. Update (client ["update" "accounts" {"credits" 105} {"where" ["=" "owner" "Eve"]}])
- 14. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]])
- 15. Explain (I) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["filter" ["=" "owner" "Eve"] ["collection-scan" "accounts"]]
- 16. Create Index (client ["create-index" "accounts" "owner"])
- 17. Explain (II) (client ["explain" ["select" "accounts" {"where" ["=" "owner" "Eve"]}]]) => ["index-lookup" ["accounts" "owner" "Eve"]]]
- 18. Multi-Write
- 19. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]])
- 20. Multi-Write (client ["multi-write" [write-query-1 write-query-2 ...]]) => [result-1 result-2 ...]
- 21. Checked-Write
- 22. Checked-Write (client ["checked-write" read-query expected-read-result write-query])
- 23. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result]
- 24. Checked-Write (client ["checked-write" read-query expected-read-result write-query]) => [true write-result] => [false actual-read-result]
- 25. Implementation Background Organization Key ideas
- 26. Background http://clojure.org/data_structures http://clojure.org/sequences http://clojure.org/state
- 27. Organization
- 28. Organization fleetdb.core: pure functions
- 29. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability
- 30. Organization fleetdb.core: pure functions fleetdb.embedded: identity and durability fleetdb.server: network interface
- 31. fleetdb.core Pure functions
- 32. fleetdb.core Pure functions Databases as Clojure data structures
- 33. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result
- 34. fleetdb.core Pure functions Databases as Clojure data structures Read: db + query → result ‘Write’: db + query → result + new db
- 36. fleetdb.core Query Planner Implements declarative queries
- 37. fleetdb.core Query Planner Implements declarative queries Database + query → plan
- 38. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}]
- 39. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
- 40. fleetdb.core Query Planner Implements declarative queries Database + query → plan ["select" "accounts" {"where" ["=" "owner" "Eve"]}] ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] ["index-lookup" ["accounts" "owner" "Eve"]]
- 42. fleetdb.core Query Executor Database + plan → result
- 43. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]]
- 44. fleetdb.core Query Executor Database + plan → result ["filter" ["=" "owner" "Eve"] ["record-scan" "accounts"]] (filter (fn [r] (= (r "owner") "Eve")) (vals (:rmap (db "accounts"))))
- 45. fleetdb.embedded Wraps fleetdb.core
- 46. fleetdb.embedded Wraps fleetdb.core Adds identity and durability
- 47. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms
- 48. fleetdb.embedded Wraps fleetdb.core Adds identity and durability Databases in atoms Append-only log
- 50. fleetdb.embedded Read Path Dereference database atom
- 51. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core
- 52. fleetdb.embedded Read Path Dereference database atom Pass to fleetdb.core Return result
- 54. fleetdb.embedded Write Path Enter fair lock
- 55. fleetdb.embedded Write Path Enter fair lock Dereference database atom
- 56. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core
- 57. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log
- 58. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value
- 59. fleetdb.embedded Write Path Enter fair lock Dereference database atom Pass to fleetdb.core Append query to log Swap in new database value Return result
- 60. fleetdb.server Wraps fleetdb.embedded
- 61. fleetdb.server Wraps fleetdb.embedded Adds JSON client API
- 63. Aside: Source Size Lines of Code core 630 embedded 130 server 120 lint 280 utilities 140 total 1300
- 64. Takeaways
- 65. Takeaways Clojure’s data structures are awesome
- 66. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software
- 67. Takeaways Clojure’s data structures are awesome Clojure is viable for infrastructure software Try FleetDB!
- 68. Thanks for listening! Questions?