MySQLConf2009: Taking ActiveRecord to the Next Level

Taking ActiveRecord to the Next Level Blythe Dunham [email_address] http://snowgiraffe.com

Goal Leverage advanced MySQL functionality with ActiveRecord

Disclaimer!!!! Premature Optimization

ActiveRecord 101 What's going on under the covers?

ActiveRecord 101 Database Table ff Active Record Model CREATE TABLE ùsers` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(255) default NULL, `password` varchar(255) default NULL, èmail` varchar(255) default NULL, `created_at` datetime default NULL, ùpdated_at` datetime default NULL, PRIMARY KEY (ìd`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 class User < ActiveRecord::Base end

ActiveRecord 101 with Animals! Database Table ff Active Record Model CREATE TABLE ànimals` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(255) default NULL, `password` varchar(255) default NULL, èmail` varchar(255) default NULL, `species_id` int(11) default NULL, `created_at` datetime default NULL, ùpdated_at` datetime default NULL, PRIMARY KEY (ìd`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 class Animal < ActiveRecord::Base end

Creating a Record animal = Animal.new animal.name = 'Jerry Giraffe' animal.password = 'jerry' animal.save! INSERT INTO ànimals` (`name`, ùpdated_at`, `species_id`, `password`, èmail`, `created_at`) VALUES('Jerry Giraffe', '2009-03-15 00:48:28', NULL, 'jerry', NULL, '2009-03-15 00:48:28')

Updating a Record animal.name = 'Jerry G' animal.save! UPDATE ànimals` SET ùpdated_at` = '2009-03-15 03:01:06', `name` = 'Jerry G' WHERE ìd` = 1

Finding a Record jerry = Animal.find :first, :conditions => ['name = ?', 'Jerry G'] SELECT * FROM `animals` WHERE (name = 'Jerry G') LIMIT 1 #shortcut Animal.find_by_name 'Jerry G'

Representing Relationships Species name Animal name email password fav_beer updated_at created_at species_id

Representing Relationships(DDL) CREATE TABLE ànimals` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(35) NOT NULL, èmail` varchar(40) default NULL, `fav_beer` enum('Guiness','Pabst','Redhook','Chimay') default 'Pabst', `created_at` datetime default NULL, ùpdated_at` datetime default NULL, `password` varchar(25) character set latin1 collate latin1_bin NOT NULL, PRIMARY KEY (ìd`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 CREATE TABLE `species` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(255), PRIMARY KEY (ìd`), ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8

Representing Relationships (AR) class Animal < ActiveRecord::Base belongs_to :species end class Species < ActiveRecord::Base has_many :animals end

Representing Relationships (AR) jerry.species SELECT * FROM `species` WHERE (`species`.ìd` = 1) species.animals SELECT * FROM ànimals` WHERE (ànimals`.species_id = 1)

Representing Relationships (AR) giraffe = Species.find_by_name 'giraffe' giraffe.animals << jerry SELECT * FROM `species` WHERE (`species`.`name` = 'giraffe' ) LIMIT 1 UPDATE ànimals` SET `species_id` = 1, ùpdated_at` = '2009-03-19 23:15:54' WHERE ìd` = 7

Migration Set limits Set default values Identify NOT NULL columns Use enumerated columns Custom DDL Add (unique) indices Foreign Keys are great Primary Key Modifications

Migration 101 ruby script/generate scaffold Animal name:string password:string email:string fav_beer:string class CreateAnimals < ActiveRecord::Migration def self.up create_table :animals do |t| t.string :name t.string :password t.string :email t.string :fav_beer t.timestamps end end def self.down drop_table :animals end end

Set Limits Default String is VARCHAR(255) create_table :animals do |t| t.string :name, :limit => 35 t.string :password, :limit => 25 t.string :email, :limit => 40 t.string :fav_beer, :limit => 40 t.timestamps end

Numeric Type Limits t.integer :mysmallint, :limit => 2 "Smart types" determines numeric type for MySQL 8 bytes bigint 5 to 8 4 bytes int(11) 4, nil, 11 3 bytes mediumint 3 2 bytes smallint 2 1 byte tinyint 1 Column Size Numeric Type :limit

Set columns to NOT NULL create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string :email, :limit => 40 t.string :fav_beer, :limit => 40 t.timestamps end

Set default values create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string :email, :limit => 40, :default => nil t.string :fav_beer, :limit => 40 :default => 'Pabst' t.timestamps end

Remove unneeded columns create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string :email, :limit => 40, :default => nil t.string :fav_beer, :limit => 40 :default => 'Pabst' t.timestamps end

Enumerated Column Plugin create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string :email, :limit => 40, :default => nil t.enum :fav_beer, :default => 'Pabst' :limit => %w(Chimay Pabst Redhook) t.timestamps end

Think about the table parameters create_table :animals, :options => 'ENGINE=MyISAM' do |t| t.string :name, :limit => 35, :null => false t.string :password, :limit => 25, :null => false t.string :email, :limit => 40, :default => nil t.enum :fav_beer, :default => nil :limit => %w(Chimay Pabst Redhook) t.timestamps end

Custom DDL create_table :animals do |t| t.string :name, :limit => 35, :null => false t.string :email, :limit => 40, :default => nil t.enum :fav_beer, :default => nil :limit => %w(Chimay Pabst Redhook) t.timestamps end #case sensitive password (encrypted) execute "ALTER TABLE `animals` ADD `password` varchar(25) character set latin1 collate latin1_bin NOT NULL"

Create (Unique) Indices create_table :species do |t| t.string :name, :null => false, :limit => 25 end add_index :species, :name, :unique => true, :name => 'uk_species_name'

ActiveRecord Uniqueness class Species < ActiveRecord::Base validates_uniqueness_of :name end Doesn't Guaranty Data Integrity!

I Heart Foreign Keys Referential Integrity

The AR Way: Foreign Keys class Species < ActiveRecord::Base has_many :animals, :dependent => :nullify end

The Rails Way: Foreign Keys class Species < ActiveRecord::Base has_many :animals, :dependent => :nullify end Dependent Value SQL Equivalent: :nullify => ON DELETE SET NULL :delete_all => ON DELETE CASCADE :destroy => No SQL equivalent. Every association is instantiated and and callbacks are executed before destruction

Redhills Foreign Key Migration Plugin to the rescue! add_column :animals, :species_id, :integer, :references => :species, :name => 'fk_animal_species', :on_delete => :set_null, :on_update => :cascade ALTER TABLE `animals` ADD `species_id` int(11); ALTER TABLE animals ADD CONSTRAINT fk_animal_species FOREIGN KEY (species_id) REFERENCES species (id) ON UPDATE CASCADE ON DELETE SET NULL;

Primary Keys CREATE TABLE ànimals` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(35) NOT NULL, èmail` varchar(40) default NULL, `fav_beer` enum('Guiness','Pabst','Redhook','Chimay') default 'Pabst', `species_id` int(11) default NULL, `created_at` datetime default NULL, ùpdated_at` datetime default NULL, `password` varchar(25) character set latin1 collate latin1_bin NOT NULL, PRIMARY KEY (ìd`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 CREATE TABLE `species` ( ìd` int(11) NOT NULL auto_increment, `name` varchar(255), PRIMARY KEY (ìd`), ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8

Modify the Rails Primary Key Change type with change_column MySQL Migration Optimization Plugin create_table :animals, :primary_key => "special_key", :primary_column => { :type => :integer, :limit => 2, :precision => :unsigned, :scale => 3 } do |t| end CREATE TABLE `animals` (`special_key` smallint(3) UNSIGNED NOT NULL auto_increment PRIMARY KEY) ENGINE=InnoDB;

Advanced ActiveRecord Insert and update options Import bulk data Finder Options Tools: plugin AR-Extensions

ActiveRecord on Steroids: ar-extensions plugin Additional Create and Update options save(options={}) save!(options={}) create(args, options={}) create!(args, options={}) Options :ignore :on_duplicate_key_update :keywords :reload :pre_sql :post_sql

:ignore Standard ActiveRecord: Create the record if it does not already exist unless Animal.find_by_name('Jerry G') Animal.create!(:name => 'Jerry G', :password => 'jerry') end

:ignore Ignore duplicates! One query, less code, fewer queries! Animal.create!({:name => 'Jerry G', :password => 'jerry'}, {:ignore => true})

:on_duplicate_key_update Update the record if it exists, if not create a new one. A lot of code to upsert and performs two SQL queries! jerry = Animal.find_by_name 'Jerry G' jerry ||= Animal.new(:name => 'Jerry G') jerry.password = 'frenchfry' jerry.save!

:on_duplicate_key_update jerry = Animal.new :name => 'Jerry G', :password => 'frenchfry' jerry.save! :on_duplicate_key_update => [:password, :updated_at] INSERT INTO animals (`name`, ùpdated_at`, `species_id`, `password`,èmail`, `created_at`) VALUES('Jerry G', '2009-03-15 06:17:51', NULL, 'frenchfry', NULL, '2009-03-15 06:17:51') ON DUPLICATE KEY UPDATE ànimals`.`password`=VALUES(`password`), ànimals`.ùpdated_at`=VALUES(ùpdated_at`)

Reloading the instance AR Data can become inconsistent with DB after an IGNORE, UPDATE, or ON DUPLICATE KEY UPDATE reload executes more queries For UPDATE the duplicate is automatically reloaded jerry.email = 'jerry@snowgiraffe.com' jerry.save! :on_duplicate_key_update => [:password, :updated_at], :reload => true, :duplicate_columns => [:name]

More Customization jerry.save(:keywords => 'LOW_PRIORITY', :pre_sql => '/*Pre comment*/', :post_sql => "/*#{__FILE__} #{__LINE__}*/") /*Pre comment*/ UPDATE LOW_PRIORITY ànimals` SET `created_at` = '2009-03-15 06:13:48', `species_id` = NULL, èmail` = NULL, `password` = 'frenchfry', ùpdated_at` = '2009-03-15 06:45:38', `name` = 'Jerry G' WHERE ìd` = 7 /*animal_controller.rb 147 */

Import (Bulk Insert) Instead of one-by-one, insert a ton of records fast

Import (Bulk Insert) Standard way: Insert each animal one by one Animal.create!(:name => 'dolly dolphin', :password => 'dolly') Animal.create!(:name => 'big birdie', :password => 'birdie') and so on….

Fast Import: One INSERT animals = [ Animal.new(:name => 'dolly dolphin', :password => 'dolly'), Animal.new(:name => 'big birdie', :password => 'birdie')] Animal.import animals INSERT INTO ànimals` (ìd`,`name`,èmail`,`fav_beer`,`created_at`,ùpdated_at`,`password`) VALUES (NULL,'dolly dolphin',NULL,'Pabst', '2009-03-20 00:17:15','2009-03-20 00:17:15','dolly'), (NULL,'big birdie',NULL,'Pabst', '2009-03-20 00:17:15','2009-03-20 00:17:15','birdie') ON DUPLICATE KEY UPDATE ànimals`.ùpdated_at`=VALUES(ùpdated_at`)

Fastest Import: fewer columns columns = [ :name, :password ] values = [['dolly dolphin', 'dolly'], ['big birdie', 'birdie']] options = { :validate => false, :timestamps => false } Animal.import columns, values, options INSERT INTO `animals` (`name`,`password`) VALUES ('dolly dolphin','dolly'),('big birdie','birdie')

Insert Select Standard: Query and Insert one by one Species.find(:all).each do |s| SpeciesZoo.create!(:species_id => s.id, :zoo_id => zoo.id, :extra_info => 'awesome') end Executes a query for each species INSERT INTO `species_zoos` (`zoo_id`, ìd`, `species_id`, èxtra_info`) VALUES (1, 3, 3, 'awesome') INSERT INTO `species_zoos` (`zoo_id`, ìd`, `species_id`, èxtra_info`) VALUES (1, 3, 2 , 'awesome') And so on…

Insert Select Import SpeciesZoo.insert_select( :from => :species, :select => ['species.id, ?', zoo], :into => [:species_id, :zoo_id]) One INSERT statement INSERT INTO `species_zoos` ( `species_id`, `zoo_id` ) SELECT species.id, 1 FROM `species`

Temporary Tables Not so good for slave replication Can be used as a sandbox then imported into a real table with ar-extensions gem Animal. create_temporary_table do |t| t.create!(:name => 'giraffe', :password => 'goo') Animal. insert_select ( :from => t, :select => [:name, :password, :fav_beer], :into => [:name, :password, :fav_beer], :on_duplicate_key_update => [:password, :fav_beer]) end

Customizing Find Additional finder options :keywords :pre_sql :post_sql :index_hint

Customizing Find Animal.find(:all, :conditions => ['name = ?', 'Jerry G'], :keywords => 'HIGH_PRIORITY', :pre_sql => '/*Pre comment*/', :post_sql => 'FOR UPDATE /*After the fact*/', :index_hint => 'USE INDEX (uk_animal_name)' ) /*Pre comment*/ SELECT HIGH_PRIORITY * FROM `animals` USE INDEX (uk_animal_name) WHERE (name = 'Jerry G') FOR UPDATE /*After the fact*/

Need more? Get dirty with find_by_sql sql = Animal.send :finder_sql_to_string, :conditions => ['name = ?', 'Jerry G'] sql.gsub! /WHERE/, 'where /* Dirty hand */' Animal.find_by_sql sql

More: find_union & count_union Animal.find_union( {:conditions => ['animals.name like ?', 'Jerry%']}, {:conditions => ['species.name = ?', 'giraffe'], :include => :species} ) (SELECT ànimals`.* FROM ànimals` WHERE (animals.name = 'Jerry G')) UNION (SELECT ànimals`.* FROM ànimals` LEFT OUTER JOIN `species` ON `species`.id = ànimals`.species_id WHERE (species.name = 'giraffe'))

Finder Issues: Speed and Memory paginate - less loaded into memory :select option - Data is retrieved faster when fewer columns are selected

Paginated Finders Rails 2.3.2 includes :batch_size option Animal.find_each(:batch_size => 2) do |animal| #do something end Will Paginate Plugin page = 1 begin animals = Animal.paginate :per_page => 2, :page => page animals.each{|animal| …do something… } end while (page = animals.next_page) Paginating Find Plugin

:select Data is retrieved faster when fewer columns are selected Animal.find :first, :select => 'name'

:include hates :select But :select is ignored with eager loading (:include)! Animal.find :first, :select => 'animals.name, species.name', :include => :species, :conditions => ['species.name like ?', 'giraffe'] SELECT ànimals`.ìd` AS t0_r0, ànimals`.`name` AS t0_r1, ànimals`.èmail` AS t0_r2, ànimals`.`fav_beer` AS t0_r3, ànimals`.`created_at` AS t0_r4, ànimals`.ùpdated_at` AS t0_r5, ànimals`.`password` AS t0_r6, ànimals`.`species_id` AS t0_r7, `species`.ìd` AS t1_r0, `species`.`name` AS t1_r1 FROM ànimals` LEFT OUTER JOIN `species` ON `species`.id = ànimals`.species_id WHERE (species.name like 'giraffe') LIMIT 1

Alternatives to Eager Loading Eager loading for sparse :include can be time consuming Use :join instead of :include Eager Load Plugin Rails 2.3.2 Query Cache helps ActiveRecordContext Plugin PiggyBack Plugin

:join instead of :include Eager loading is slow in Rails and can be slow on the database. Use an (inner) :join instead of :include animal = Animal.find :first, :select => 'animals.name, species.name as spec_name', :joins => :species, :conditions => ['species.name like ?', 'giraffe'] animal.spec_name == 'giraffe'

Force it with Eager loading plugins Eager loading is slow in Rails and can be slow on the database. eload -select plugin

Help from Rails 2 Query Cache Animals of the same species are only loaded once ActiveRecord::Base.cache { Animal.find(:all).each {|a| a.species } } Animal Load (1.8ms) SELECT * FROM ànimals` Species Load (0.3ms) SELECT * FROM `species` WHERE (`species`.ìd` = 2) CACHE (0.0ms) SELECT * FROM `species` WHERE (`species`.ìd` = 2) Species Load (0.3ms) SELECT * FROM `species` WHERE (`species`.ìd` = 1) CACHE (0.0ms) SELECT * FROM `species` WHERE (`species`.ìd` = 1)

ActiveRecordContext Plugin ActiveRecord::Base.with_context do animals = Animal.find :all Species.prefetch animals animals.each {|a| a.species } end Animal Load (0.4ms) SELECT * FROM `animals` [Context] Storing Animal records: 1, 2, 3, 4, 5, 6, and 7 Species Load (0.4ms) SELECT * FROM `species` WHERE (`species`.`id` IN( 2,1 )) [Context] Storing Species records: 2 and 1 [Context] Found Species #2 [Context] Found Species #2 [Context] Found Species #1 [Context] Found Species #1

Piggyback Plugin Delegate records with :has_one and :belongs_to associations Great for accessing extension tables with TEXT or BLOB

Piggyback Plugin Uses joins to delegate records from :has_one and :belongs_to associations class Animal < ActiveRecord::Base belongs_to :species piggy_back :species_name, :from => :species, :attributes => [:name] end animal = Animal.find :first, :piggy => :species_name animal.species_name == 'giraffe' SELECT animals.*, species.name AS species_name FROM `animals` LEFT JOIN species ON species.id=animals.species_id LIMIT 1

Avoiding Deadlock Deadlock Retry plugin - retries query up to 3 times Batch Operations (AR-Extension plugin) Animal.delete_all(['name like ?','giraffe%'], :batch_size => 50)

Reading from a Replica Masochism plugin

Export Data ar-fixtures: Export entire table to yaml ar-extensions: export to csv ar-dumper: export (paginated) to yaml, xml or csv

Cache! Show me the money! Query Cache Memcache Static Record Cache plugin

ActiveRecord plugins and gems AR-Extensions http://www. continuousthinking .com/tags/ arext Piggy Back http:// railsexpress .de/ svn / plugins /piggy_back/trunk/README Eager Loading http://www. snowgiraffe .com/tech/?p=329 Active Record Context http:// svn .techno-weenie.net/projects/ plugins /active_record_context/ Will Paginate http:// github .com/ mislav /will_paginate/tree/master Deadlock Retry http:// agilewebdevelopment .com/ plugins /deadlock_retry Paginating Find http://www. railslodge .com/ plugins /287-paginating-find Static Record Cache http:// github .com/ blythedunham /static_record_cache/tree/master

Other Useful Plugins Migration Plugins Redhills Consulting Foreign Key Migrations http:// agilewebdevelopment .com/ plugins /foreign_key_migrations Enum Column http:// enum -column. rubyforge .org/ MySQL Migration Optimizer http://github.com/blythedunham/mysql_migration_optimizer/tree/master Replica Plugins Masochism http://github.com/technoweenie/masochism/tree/master Active Delegate http://www. robbyonrails .com/articles/2007/10/05/multiple-database-connections-in-ruby-on-rails Export Plugins Ar-fixtures http://github.com/topfunky/ar_fixtures/tree/master Ar-dumper http://github.com/blythedunham/ar_dumper/tree/master

Questions? Thanks for attending! I hope you are on the next level. Slides available at: http://snowgiraffe.com

Pics Biz card: http://www.twozdai.com/ Toolbox: http://www.flickr.com/photos/29802022@N07/2971774302/ Cash: http://www. flickr .com/photos/ gnerk /2466566500/ Emoo cow: http://blog.petegraham.co.uk/2007/05/15/animals-with-emo-haircuts/ Giraffe in Car: http://www. norrishallstudio .com/assets/ img /products/ switchplates /Giraffe_in_Car_SO.jpg Leafcutters: http://www.makingthelink.co.uk/Leafcutter%20Ants%20Corcovado.jpg Puppy: http://www. flickr .com/photos/ todorrovic /2287792473/ Dump truck: http://farm3.static.flickr.com/2065/2197925262_bd2726c3fa.jpg?v=0 Ignore sign: http://www.flickr.com/photos/alexsuarez/2504638107/ Tigers: http://www.flickr.com/photos/sharynmorrow/19981568/ Single Leafcutter: http://www.flickr.com/photos/mattblucas/2176783448/ Giraffe with tongue: http://www.flickr.com/photos/ucumari/2570608134/ Giraffe in a can: http://www.flickr.com/photos/10159247@N04/2877489356/ Beaver: http://www.flickr.com/photos/krrrista/2286455954/ Mini Pig: http://www.richardaustinimages.com/ Giraffe herd: http://www.flickr.com/photos/acastellano/2260928018/ Giraffe skin: http://www.flickr.com/photos/tomorrowstand/1806095442 Dirty hands: http://www.flickr.com/photos/dragonhide/2372544373/ Blythe twins: http://www.flickr.com/photos/petitewanderlings/434252916/ Dead lock: http://www.flickr.com/photos/fpsurgeon/2453544236/ Foreign Keys: http://www.flickr.com/photos/zeeny79/347753999/ Gorilla: http://www.flickr.com/photos/shuttershrink/425613091/ Speed of light: http://www.flickr.com/photos/laserstars/908946494/ Fat Giraffe: http://funnies.com/funny-picture-fat-giraffe.htm

MySQLConf2009: Taking ActiveRecord to the Next Level

More Related Content

What's hot (20)

Similar to MySQLConf2009: Taking ActiveRecord to the Next Level (20)

Recently uploaded (20)

MySQLConf2009: Taking ActiveRecord to the Next Level