![DBDeploy Go to directory with SQL files: “ 1 create_customer_table.sql” “ 2 add_customer_id_constraint.sql” Run “ant” Output: gen-and-exec-delta-script: [dbdeploy] dbdeploy v2.11 [dbdeploy] Reading change scripts from directory C:\Projects\dbdeploy-demo\sql\deltas... [dbdeploy] Changes currently applied to database: [dbdeploy] 1, 2 [dbdeploy] Scripts available: [dbdeploy] 1, 2, 3, 4 [dbdeploy] To be applied: [dbdeploy] 3, 4 [sql] Executing file: C:\Projects\dbdeploy-demo\build_output\db-deltas.sql [sql] 8 of 8 SQL statements executed successfully](https://image.slidesharecdn.com/evolutionarydbdevelopment-100831015401-phpapp02/85/Evolutionary-db-development-32-320.jpg)
Evolutionary db development
- 2. Software Development Process is Changing © ThoughtWorks 2009 Fix / Integrate $ Test Code Design Specifications Use Cases / Functional Specs Requirements Gathering Project Plan/Estimation $ $ $ $ Level Stories Vision & High- Iteration Release
- 4. Team Collaboration DB Apps BackEnd Traditional Team In Agile team, DBA Role!= Person knowledge of the functionality Acknowledge team in interaction DEVS I’m responsible for Web design DBA I’m responsible for Database Domain expert I’m responsible for backend apps DBA Our responsible for business value Our responsible for business value It’s our responsibility DEVS Domain Expert Agile Team
- 5. DBA sit close to all the roles Help make a decision Educating Developers write better SQL How to make the DBA redundant Best Practices
- 6. Refactoring “ A disciplined way to make change to your source code to improve its design, making it easier to work with” Martin Folwer
- 7. Database Refactoring Before: After: Behavioral semantics Change structure A small change to Database schema that improves its design while retaining both its behavioral and informational semantics Data Migration Verify data quality Informational semantics Customer … Balance Account AccountID(PK) Customer … Account AccountID(PK) Balance
- 8. Structural Data Quality Referential Integrity Architectural Method Non-Refactoring Transformation What will be changed in DB Refactoring? Small Steps Frequent Changes Test First Principles
- 9. Test your Database Schema Test the way Apps uses schema Validate your data migration Test external program code Check data quality … Test-Driven is also important for DB design TEST Fail Fast Fail Often
- 10. Not trying to "Get it Right up Front“ Build the simplest thing that can possibly work Treat changes as database refactoring ... Every DAY functionality added in increments Evolutionary Modeling Design
- 11. A Story
- 12. Background CRM system Transactional operations Reporting/ Statistics functions for manager Based on Legacy system 24*7 , a very busy system Client’s expectation Improvement of Usability and performance Legacy Database Sql server 2005 200G Team Distributed Agile team. (Beijing + Hongkong)
- 13. Refactoring database schema Introduce transition period for SAFTY
- 14. Never get rid of old schema immediately Data sync in real time
- 15. What we did in this project Legacy DB NEW DB FlatFile Triger SystemNameID Newvalue OldValue columnName Read Data Near real Time Read Data systemName SystemNameID(PK) SystemLogin … systemName_log SystemNameID(PK) OperDatetime Account AccountID(PK) AccountName SystemNameID Isdelete
- 16. Onetime Migration Need to figure out logical mapping Problem: There are over 400 procs, over 1000 tables, over 150 views Data Concept/logics changed How we resolve: Communication is very important Split a function into small parts, more smaller more easy more safe Person Organisation Person User contact
- 17. The process of database refactoring
- 18. Verify data quality by domain logics after one time migration 1.One user Should not have 2 office telephone Select user From user a left join telephone b On a.userid=b.userid Group by a.user Having count(*)>=2 2. A telephone should be assigned to a user Select telephone From telephone Where userid is null 3. A telehpone shoulde not be assigned to 2 or more user Select officeTele From telephone a left join user Group by officeTele Having count(userid)>=2 … adding more verification sql User UserID(PK) FirstName LastName … Telephone TeleID(PK) OfficeTele HomeTele UserID(FK)
- 19. Sanity Check Brainstorming Feedback from CI/Production Don’t make an issue to happen for two times
- 20. Put test scripts into CI Put verification scripts into CI Run verification scripts in production often, be aware of production data
- 21. Our workflow Production App Pre-Production App Highly iterative Development System and Acceptance Testing Operations and support Devs’ App DB App Bug Reports Bug Reports Bug Reports Project-level Integration Testing Frequent Deployment Frequent Deployment Highly Controled Deployment CVS Tests SanityCheck Brainstorming Production Users QA Devs Devs
- 22. The strategy for migration performance Online system don’t allow that migration spend too long time Migration Update contactNote a Set ContactNote=c1,ContactId=c2 From ContactInfo where a.messid=b.Messid Too long ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId
- 23. Delcare row_num number:=0; Begin for c_CN in (select MessId,C1,C2 from contactInfo a left join CI_Mig_log b on a.Messid=b.messid where flag=0) update contactNote set contactNote=c_CN.c1, contactId=c_CN.c2 where Messid=c_CN.Messid update CI_MIG_log f set f.flag=1 where messid=c_Cn.MessId row_num:=row_num+1; if mod(row_num,1000)=0 then commit; end if; End loop; Cancel it if there is no enough resource for migration Migration ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId CI_Mig_log MessID(PK) Flag
- 24. Versioning Database All database schemas can be thought of as DB refactoring As updates are applied to a database, the changes will be recorded in a table similar to the following: Change Date 1_Create_Customer_Table.sql 4-15-07 2_Add_e-mail_address_column.sql 4-17-07 3_Add_fax_number_column.sql 4-18-07 4_Add_transaction_table.sql 4-21-07 5_Add_transaction_status_column.sql 4-24-07 6_Add_customer-transaction_view.sql 4-27-07
- 25. Put them under configuration management control CREATE TABLE money ( eek NUMBER ); //Test for… DB DDL Insert into AA(mydata) Values(11); Meta Data Delete from .. DML Create index optimization Merge into Data Migration Tests Installation scripts
- 26. Configuration of DB project Database project under version control Tiny db backup Deltascripts of dbdeploy For defining db objects which depends on schema For data onetime migration For data sync For checking dirty data Tool exclusively for database project other scripts…
- 27. Database Deployment An automated process is needed to make the process of upgrading out-of-date databases
- 28. Our Practices Nothing is used only once Automate tasks such as Physical table deployment Usage statistics Schema verification Data migration verification Introduce tools ,like Ant, dbdeploy
- 29. Management DB deploymnet DB Deploy - http://dbdeploy.com/
- 30. DBDeploy http://dbdeploy.com/documentation/getting-started/rules-for-using-dbdeploy/ N aming convention for delta scripts : NUMBER COMMENT.SQL e.g. 1_Create_Customer_Table.sql … Undo section – marked by comments: CREATE TABLE FOO ( FOO_ID INTEGER NOT NULL, FOO_VALUE VARCHAR(30) ); ALTER TABLE FOO ADD CONSTRAINT PK_FOO PRIMARY KEY (FOO_ID); --//@UNDO DROP TABLE FOO;
- 31. <target name="gen-and-exec-delta-script"> <dbdeploy driver="oracle.jdbc.OracleDriver" url="jdbc:oracle:thin:@localhost:1521:XE" userid="dylan" password="nalyd" dir="./sql/deltas/" outputfile="./build_output/db-deltas-hsql.sql" dbms="ora"/> <sql driver=" oracle.jdbc.OracleDriver" url=" jdbc:oracle:thin:@localhost:1521:XE " userid="dylan" password="nalyd" src="./build_output/db-deltas.sql" onerror="abort"/> </target> Ant
- 32. DBDeploy Go to directory with SQL files: “ 1 create_customer_table.sql” “ 2 add_customer_id_constraint.sql” Run “ant” Output: gen-and-exec-delta-script: [dbdeploy] dbdeploy v2.11 [dbdeploy] Reading change scripts from directory C:\Projects\dbdeploy-demo\sql\deltas... [dbdeploy] Changes currently applied to database: [dbdeploy] 1, 2 [dbdeploy] Scripts available: [dbdeploy] 1, 2, 3, 4 [dbdeploy] To be applied: [dbdeploy] 3, 4 [sql] Executing file: C:\Projects\dbdeploy-demo\build_output\db-deltas.sql [sql] 8 of 8 SQL statements executed successfully
- 33. Automate Tasks CreateNewTestDB upgradeDB <target name= “-parseDbScxripts” >…</target> … <target name="-upgradeDB" depends="-parseDbScripts, -dbdeploy, -runDeltaScript, -dropDbLogic, -createDbLogic" description="Upgrade specified database to latest version" /> <target name="rebuildDB" depends=" -parseDbScripts, -dropDb, -createDb, -initialiseDb, -dbdeploy, -runDeltaScript, -createDbLogic" description="drop, recreate and initialise the Connect database" />
- 34. Batch file: library\nant\nant.exe –buildfile:evovle.build –D:rebuildDB -logfile:build.txt Shared to devs Do what we want by just One Command
- 35. Reference: Evolutionary Database Design http://martinfowler.com/articles/evodb.html Refactoring Databases: Evolutionary Database Design
- 36. Q&A
Editor's Notes
- #3: Changing from waterfall to Agile At an early stage identify requirements , reconcile , design , begin coding minimize changes due extensive preliminary work
- #4: automation
- #9: explain
- #37: Classify and Prioritize data Client list the most important data Try to figure out Data pattern( Data cleaning) Only working on meaningful data Verify data by system logics ( data referencing, records number…) Check it randomly