SlideShare a Scribd company logo
Evolutionary Database Development
Software Development Process is Changing © ThoughtWorks 2009 Fix / Integrate $ Test Code Design Specifications Use Cases /  Functional Specs Requirements Gathering Project Plan/Estimation $ $ $ $ Level Stories Vision & High- Iteration Release
 
Team Collaboration DB Apps BackEnd Traditional Team In Agile team, DBA  Role!= Person knowledge of the functionality Acknowledge team in interaction DEVS I’m responsible for Web design  DBA I’m responsible for Database Domain expert I’m responsible for backend apps DBA Our responsible for business value Our responsible for business value It’s our responsibility DEVS Domain  Expert Agile Team
DBA sit close to all the roles Help make a decision Educating Developers write better SQL How to make the DBA redundant Best Practices
Refactoring “ A disciplined way to make change to your source code to improve its design, making it easier to work with” Martin Folwer
Database Refactoring Before: After: Behavioral  semantics   Change structure A small change to Database schema that improves its design while retaining both its  behavioral  and  informational  semantics Data Migration Verify data quality Informational  semantics   Customer … Balance Account AccountID(PK) Customer … Account AccountID(PK) Balance
Structural Data Quality Referential Integrity Architectural Method Non-Refactoring Transformation What will be changed in DB Refactoring? Small Steps Frequent Changes Test First Principles
Test your Database Schema Test the way Apps uses schema Validate your data migration Test external program code Check data quality … Test-Driven is also important for DB design TEST Fail Fast Fail Often
Not trying to "Get it Right up Front“ Build the simplest thing that can possibly work  Treat changes as database refactoring ... Every DAY functionality added in increments Evolutionary Modeling Design
A Story
Background CRM system Transactional operations  Reporting/ Statistics functions for manager Based on Legacy system  24*7 , a very busy system Client’s expectation Improvement of Usability and performance Legacy Database Sql server 2005 200G Team Distributed Agile team. (Beijing + Hongkong)
Refactoring database schema Introduce transition period for SAFTY
Never get rid of old schema immediately Data sync in real time
What we did in this project Legacy DB NEW DB FlatFile Triger SystemNameID Newvalue OldValue columnName Read Data Near real Time Read Data systemName SystemNameID(PK) SystemLogin … systemName_log SystemNameID(PK) OperDatetime Account AccountID(PK) AccountName SystemNameID Isdelete
Onetime Migration Need to figure out logical mapping Problem: There are over 400 procs, over 1000 tables, over 150 views Data Concept/logics changed How we resolve: Communication is very important Split a function into small parts, more smaller more easy more safe Person Organisation Person User contact
  The process of database refactoring
Verify data quality by domain logics after one time migration 1.One user Should not have 2 office telephone Select user From user a left join telephone b On a.userid=b.userid Group by a.user Having count(*)>=2 2. A telephone should be assigned to a user Select telephone From telephone  Where userid is null 3. A telehpone shoulde not be assigned to 2 or more user Select officeTele From telephone a left join user Group by officeTele Having count(userid)>=2 …  adding more verification sql User UserID(PK) FirstName LastName … Telephone TeleID(PK) OfficeTele HomeTele UserID(FK)
Sanity Check Brainstorming Feedback from CI/Production Don’t make an issue to happen for two times
Put test scripts into CI Put verification scripts into CI  Run verification scripts in production often, be aware of production data
Our workflow Production App Pre-Production App Highly iterative Development System and Acceptance Testing Operations and support Devs’ App DB App Bug Reports Bug Reports Bug Reports Project-level Integration Testing Frequent Deployment Frequent Deployment Highly Controled Deployment CVS Tests SanityCheck Brainstorming Production Users QA Devs Devs
The strategy for migration performance Online system don’t allow that migration spend too long time Migration Update contactNote a Set ContactNote=c1,ContactId=c2 From ContactInfo  where a.messid=b.Messid Too long  ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId
Delcare row_num number:=0; Begin  for c_CN in (select MessId,C1,C2 from contactInfo a left join CI_Mig_log b on a.Messid=b.messid where flag=0) update contactNote set contactNote=c_CN.c1, contactId=c_CN.c2 where Messid=c_CN.Messid update CI_MIG_log f set f.flag=1 where messid=c_Cn.MessId row_num:=row_num+1; if mod(row_num,1000)=0 then commit; end if; End loop; Cancel it if there is no enough resource for migration Migration ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId CI_Mig_log MessID(PK) Flag
Versioning Database All database schemas can be thought of as DB refactoring As updates are applied to a database, the changes will be recorded in a table similar to the following:  Change Date 1_Create_Customer_Table.sql 4-15-07 2_Add_e-mail_address_column.sql 4-17-07 3_Add_fax_number_column.sql 4-18-07 4_Add_transaction_table.sql 4-21-07 5_Add_transaction_status_column.sql 4-24-07 6_Add_customer-transaction_view.sql 4-27-07
Put them under configuration management control CREATE TABLE money ( eek NUMBER ); //Test for… DB DDL Insert into AA(mydata) Values(11); Meta Data Delete from .. DML Create index optimization Merge into Data Migration Tests Installation scripts
Configuration of DB project Database project under version control  Tiny db backup Deltascripts of dbdeploy For defining db objects which depends on schema For data onetime migration For data sync For checking dirty data Tool exclusively for database project other scripts…
Database Deployment An automated process  is needed to make the process of upgrading out-of-date databases
Our Practices Nothing is used only once Automate tasks such as Physical table deployment Usage statistics Schema verification Data migration verification Introduce tools ,like Ant, dbdeploy
Management DB deploymnet DB Deploy -  http://dbdeploy.com/
DBDeploy http://dbdeploy.com/documentation/getting-started/rules-for-using-dbdeploy/   N aming convention for delta scripts : NUMBER COMMENT.SQL e.g.  1_Create_Customer_Table.sql … Undo section – marked by comments: CREATE TABLE FOO (  FOO_ID INTEGER NOT NULL,   FOO_VALUE VARCHAR(30) );  ALTER TABLE FOO ADD CONSTRAINT PK_FOO PRIMARY KEY (FOO_ID); --//@UNDO  DROP TABLE FOO;
<target name=&quot;gen-and-exec-delta-script&quot;> <dbdeploy driver=&quot;oracle.jdbc.OracleDriver&quot; url=&quot;jdbc:oracle:thin:@localhost:1521:XE&quot; userid=&quot;dylan&quot; password=&quot;nalyd&quot; dir=&quot;./sql/deltas/&quot; outputfile=&quot;./build_output/db-deltas-hsql.sql&quot; dbms=&quot;ora&quot;/> <sql driver=&quot; oracle.jdbc.OracleDriver&quot; url=&quot; jdbc:oracle:thin:@localhost:1521:XE &quot; userid=&quot;dylan&quot; password=&quot;nalyd&quot; src=&quot;./build_output/db-deltas.sql&quot; onerror=&quot;abort&quot;/> </target> Ant
DBDeploy Go to directory with SQL files: “ 1 create_customer_table.sql” “ 2 add_customer_id_constraint.sql” Run “ant” Output: gen-and-exec-delta-script: [dbdeploy] dbdeploy v2.11 [dbdeploy] Reading change scripts from directory C:\Projects\dbdeploy-demo\sql\deltas... [dbdeploy] Changes currently applied to database: [dbdeploy] 1, 2 [dbdeploy] Scripts available: [dbdeploy] 1, 2, 3, 4 [dbdeploy] To be applied: [dbdeploy] 3, 4 [sql] Executing file: C:\Projects\dbdeploy-demo\build_output\db-deltas.sql [sql] 8 of 8 SQL statements executed successfully
Automate Tasks  CreateNewTestDB upgradeDB <target name= “-parseDbScxripts” >…</target> … <target name=&quot;-upgradeDB&quot; depends=&quot;-parseDbScripts, -dbdeploy,  -runDeltaScript, -dropDbLogic, -createDbLogic&quot;  description=&quot;Upgrade specified database to latest version&quot; /> <target name=&quot;rebuildDB&quot; depends=&quot; -parseDbScripts, -dropDb,  -createDb, -initialiseDb, -dbdeploy, -runDeltaScript, -createDbLogic&quot;  description=&quot;drop, recreate and initialise the Connect database&quot; />
Batch file: library\nant\nant.exe –buildfile:evovle.build –D:rebuildDB -logfile:build.txt Shared to devs Do what we want by just One Command
Reference: Evolutionary Database Design http://martinfowler.com/articles/evodb.html Refactoring Databases: Evolutionary Database Design
Q&A

More Related Content

What's hot (14)

DOC
Informatica 5+years of experince
Dharma Rao
 
PPTX
EMC Documenutm xCP 2.2 vs 1.x
Haytham Ghandour
 
PPT
Open ERP Version 7 Functional & Technical Overview
Pragmatic Techsoft
 
PDF
EMC Documentum - xCP 2.x Updating Java Services
Haytham Ghandour
 
PDF
Bapi jco[1]
mateenjambagi
 
PPT
Chapter09
sasa_eldoby
 
PPTX
Workflow functional concept on openerp7
Aziza Mohamed
 
PDF
EMC Documentum - xCP 2.x Installation and Deployment
Haytham Ghandour
 
DOCX
Sap ps module tutorial
achyuth10
 
DOCX
Swarn Singh_CV_SSE
Swarn Singh
 
PDF
Foundations of programming
Stefan von Niederhäusern
 
PPT
New Enhancements + Upgrade Path to Oracle EBS R12.1.3
iWare Logic Technologies Pvt. Ltd.
 
PDF
Advanced Visualforce Webinar
Salesforce Developers
 
DOCX
Pratk kambe rac
Pratik Kambe
 
Informatica 5+years of experince
Dharma Rao
 
EMC Documenutm xCP 2.2 vs 1.x
Haytham Ghandour
 
Open ERP Version 7 Functional & Technical Overview
Pragmatic Techsoft
 
EMC Documentum - xCP 2.x Updating Java Services
Haytham Ghandour
 
Bapi jco[1]
mateenjambagi
 
Chapter09
sasa_eldoby
 
Workflow functional concept on openerp7
Aziza Mohamed
 
EMC Documentum - xCP 2.x Installation and Deployment
Haytham Ghandour
 
Sap ps module tutorial
achyuth10
 
Swarn Singh_CV_SSE
Swarn Singh
 
Foundations of programming
Stefan von Niederhäusern
 
New Enhancements + Upgrade Path to Oracle EBS R12.1.3
iWare Logic Technologies Pvt. Ltd.
 
Advanced Visualforce Webinar
Salesforce Developers
 
Pratk kambe rac
Pratik Kambe
 

Viewers also liked (18)

PDF
Liberty Grant.Collection
Liberty Grant
 
PDF
Улучшение бизнес-процессов
Liberty Grant
 
PPT
西藏10日游
Open Party
 
PDF
Презентация лучше, чем документ
Liberty Grant
 
PDF
Как защитить IT-бюджеты перед бизнесом
Liberty Grant
 
PDF
Liberty Grant Frameworks
Liberty Grant
 
PDF
О компании Liberty Grant
Liberty Grant
 
PDF
Liberty grant.Tariffbenchmarking
Liberty Grant
 
PDF
Очереди
Liberty Grant
 
PDF
Дэшборд для мониторинга платежей
Liberty Grant
 
PDF
Revista dh-unjfsc
Atilio Omar Violeta Tovar
 
PPTX
夸父通讯中间件
Open Party
 
PPT
淘宝广告技术部开发流程和Scrum实践
Open Party
 
PPT
Hs java open_party
Open Party
 
PDF
Система онлайн-мониторинга Liberty Grant
Liberty Grant
 
PDF
LibertyGrant.InetMobileBanking
Liberty Grant
 
PDF
Мониторинг розничной сети банка
Liberty Grant
 
PDF
Liberty grant.Paymentsoptimization
Liberty Grant
 
Liberty Grant.Collection
Liberty Grant
 
Улучшение бизнес-процессов
Liberty Grant
 
西藏10日游
Open Party
 
Презентация лучше, чем документ
Liberty Grant
 
Как защитить IT-бюджеты перед бизнесом
Liberty Grant
 
Liberty Grant Frameworks
Liberty Grant
 
О компании Liberty Grant
Liberty Grant
 
Liberty grant.Tariffbenchmarking
Liberty Grant
 
Очереди
Liberty Grant
 
Дэшборд для мониторинга платежей
Liberty Grant
 
Revista dh-unjfsc
Atilio Omar Violeta Tovar
 
夸父通讯中间件
Open Party
 
淘宝广告技术部开发流程和Scrum实践
Open Party
 
Hs java open_party
Open Party
 
Система онлайн-мониторинга Liberty Grant
Liberty Grant
 
LibertyGrant.InetMobileBanking
Liberty Grant
 
Мониторинг розничной сети банка
Liberty Grant
 
Liberty grant.Paymentsoptimization
Liberty Grant
 
Ad

Similar to Evolutionary db development (20)

PPT
Evolutionary Database Design
Andrei Solntsev
 
PPTX
Evolutionary database design
Salehein Syed
 
PDF
Database Schema Evolution
Lars Thorup
 
PPTX
Evolutionary database design
David Price
 
PDF
Are we there Yet?? (The long journey of Migrating from close source to opens...
Marco Tusa
 
PDF
Database Comparison & Synch | Change Manager Success Story
Embarcadero Technologies
 
PDF
Delivering changes for applications and databases
Eduardo Piairo
 
PDF
KYSUC - Keep Your Schema Under Control
Coimbra JUG
 
PDF
javazone 2014
Rikard Thulin
 
PDF
Gaelyk - Web Apps In Practically No Time
Saltmarch Media
 
PPTX
Bringing DevOps to the Database
Michaela Murray
 
PDF
L Migrator Pres
Brian Loomis
 
ODP
Handling Database Deployments
Mike Willbanks
 
PPT
Database Refactoring Sreeni Ananthakrishna 2006 Nov
melbournepatterns
 
PDF
Delivering Changes for Applications and Databases
Miguel Alho
 
DOC
Navendu_Resume
Navendu Baghel
 
KEY
Agile Database Development with Liquibase
Tim Berglund
 
KEY
Database Refactoring With Liquibase
IASA
 
PPTX
DevOps 101 for data professionals
Alex Yates
 
DOC
Vijay_Kr_Singh_Oracle_SQL_PLSQL_Developer
Vijay Kumar Singh
 
Evolutionary Database Design
Andrei Solntsev
 
Evolutionary database design
Salehein Syed
 
Database Schema Evolution
Lars Thorup
 
Evolutionary database design
David Price
 
Are we there Yet?? (The long journey of Migrating from close source to opens...
Marco Tusa
 
Database Comparison & Synch | Change Manager Success Story
Embarcadero Technologies
 
Delivering changes for applications and databases
Eduardo Piairo
 
KYSUC - Keep Your Schema Under Control
Coimbra JUG
 
javazone 2014
Rikard Thulin
 
Gaelyk - Web Apps In Practically No Time
Saltmarch Media
 
Bringing DevOps to the Database
Michaela Murray
 
L Migrator Pres
Brian Loomis
 
Handling Database Deployments
Mike Willbanks
 
Database Refactoring Sreeni Ananthakrishna 2006 Nov
melbournepatterns
 
Delivering Changes for Applications and Databases
Miguel Alho
 
Navendu_Resume
Navendu Baghel
 
Agile Database Development with Liquibase
Tim Berglund
 
Database Refactoring With Liquibase
IASA
 
DevOps 101 for data professionals
Alex Yates
 
Vijay_Kr_Singh_Oracle_SQL_PLSQL_Developer
Vijay Kumar Singh
 
Ad

More from Open Party (17)

PDF
Sunshine library introduction
Open Party
 
PPT
食品安全与生态农业──小毛驴市民农园项目介绍
Open Party
 
PDF
Cs open-party
Open Party
 
KEY
网站优化实践
Open Party
 
PPTX
Introduction to scientific visualization
Open Party
 
PPTX
Applying BDD in refactoring
Open Party
 
PDF
移动广告不是网盟
Open Party
 
PPTX
Android 开源社区,10年后的再思考
Open Party
 
PPTX
企业创业融资之路
Open Party
 
PPTX
Java mobile 移动应用开发
Open Party
 
PPT
如何做演讲
Open Party
 
KEY
爬虫点滴
Open Party
 
PPT
Positive psychology
Open Party
 
PPT
价值驱动的组织转型-王晓明
Open Party
 
PPTX
对云计算的理解
Open Party
 
PPTX
Web前端标准在各浏览器中的实现差异
Open Party
 
KEY
Douban pulse
Open Party
 
Sunshine library introduction
Open Party
 
食品安全与生态农业──小毛驴市民农园项目介绍
Open Party
 
Cs open-party
Open Party
 
网站优化实践
Open Party
 
Introduction to scientific visualization
Open Party
 
Applying BDD in refactoring
Open Party
 
移动广告不是网盟
Open Party
 
Android 开源社区,10年后的再思考
Open Party
 
企业创业融资之路
Open Party
 
Java mobile 移动应用开发
Open Party
 
如何做演讲
Open Party
 
爬虫点滴
Open Party
 
Positive psychology
Open Party
 
价值驱动的组织转型-王晓明
Open Party
 
对云计算的理解
Open Party
 
Web前端标准在各浏览器中的实现差异
Open Party
 
Douban pulse
Open Party
 

Evolutionary db development

  • 2. Software Development Process is Changing © ThoughtWorks 2009 Fix / Integrate $ Test Code Design Specifications Use Cases / Functional Specs Requirements Gathering Project Plan/Estimation $ $ $ $ Level Stories Vision & High- Iteration Release
  • 3.  
  • 4. Team Collaboration DB Apps BackEnd Traditional Team In Agile team, DBA Role!= Person knowledge of the functionality Acknowledge team in interaction DEVS I’m responsible for Web design DBA I’m responsible for Database Domain expert I’m responsible for backend apps DBA Our responsible for business value Our responsible for business value It’s our responsibility DEVS Domain Expert Agile Team
  • 5. DBA sit close to all the roles Help make a decision Educating Developers write better SQL How to make the DBA redundant Best Practices
  • 6. Refactoring “ A disciplined way to make change to your source code to improve its design, making it easier to work with” Martin Folwer
  • 7. Database Refactoring Before: After: Behavioral semantics Change structure A small change to Database schema that improves its design while retaining both its behavioral and informational semantics Data Migration Verify data quality Informational semantics Customer … Balance Account AccountID(PK) Customer … Account AccountID(PK) Balance
  • 8. Structural Data Quality Referential Integrity Architectural Method Non-Refactoring Transformation What will be changed in DB Refactoring? Small Steps Frequent Changes Test First Principles
  • 9. Test your Database Schema Test the way Apps uses schema Validate your data migration Test external program code Check data quality … Test-Driven is also important for DB design TEST Fail Fast Fail Often
  • 10. Not trying to &quot;Get it Right up Front“ Build the simplest thing that can possibly work Treat changes as database refactoring ... Every DAY functionality added in increments Evolutionary Modeling Design
  • 12. Background CRM system Transactional operations Reporting/ Statistics functions for manager Based on Legacy system 24*7 , a very busy system Client’s expectation Improvement of Usability and performance Legacy Database Sql server 2005 200G Team Distributed Agile team. (Beijing + Hongkong)
  • 13. Refactoring database schema Introduce transition period for SAFTY
  • 14. Never get rid of old schema immediately Data sync in real time
  • 15. What we did in this project Legacy DB NEW DB FlatFile Triger SystemNameID Newvalue OldValue columnName Read Data Near real Time Read Data systemName SystemNameID(PK) SystemLogin … systemName_log SystemNameID(PK) OperDatetime Account AccountID(PK) AccountName SystemNameID Isdelete
  • 16. Onetime Migration Need to figure out logical mapping Problem: There are over 400 procs, over 1000 tables, over 150 views Data Concept/logics changed How we resolve: Communication is very important Split a function into small parts, more smaller more easy more safe Person Organisation Person User contact
  • 17.   The process of database refactoring
  • 18. Verify data quality by domain logics after one time migration 1.One user Should not have 2 office telephone Select user From user a left join telephone b On a.userid=b.userid Group by a.user Having count(*)>=2 2. A telephone should be assigned to a user Select telephone From telephone Where userid is null 3. A telehpone shoulde not be assigned to 2 or more user Select officeTele From telephone a left join user Group by officeTele Having count(userid)>=2 … adding more verification sql User UserID(PK) FirstName LastName … Telephone TeleID(PK) OfficeTele HomeTele UserID(FK)
  • 19. Sanity Check Brainstorming Feedback from CI/Production Don’t make an issue to happen for two times
  • 20. Put test scripts into CI Put verification scripts into CI Run verification scripts in production often, be aware of production data
  • 21. Our workflow Production App Pre-Production App Highly iterative Development System and Acceptance Testing Operations and support Devs’ App DB App Bug Reports Bug Reports Bug Reports Project-level Integration Testing Frequent Deployment Frequent Deployment Highly Controled Deployment CVS Tests SanityCheck Brainstorming Production Users QA Devs Devs
  • 22. The strategy for migration performance Online system don’t allow that migration spend too long time Migration Update contactNote a Set ContactNote=c1,ContactId=c2 From ContactInfo where a.messid=b.Messid Too long ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId
  • 23. Delcare row_num number:=0; Begin for c_CN in (select MessId,C1,C2 from contactInfo a left join CI_Mig_log b on a.Messid=b.messid where flag=0) update contactNote set contactNote=c_CN.c1, contactId=c_CN.c2 where Messid=c_CN.Messid update CI_MIG_log f set f.flag=1 where messid=c_Cn.MessId row_num:=row_num+1; if mod(row_num,1000)=0 then commit; end if; End loop; Cancel it if there is no enough resource for migration Migration ContactInfo Messid(pk) C1 C2 … ContactNote ContactNoteId(pk) ContactNote ContactId MessId CI_Mig_log MessID(PK) Flag
  • 24. Versioning Database All database schemas can be thought of as DB refactoring As updates are applied to a database, the changes will be recorded in a table similar to the following: Change Date 1_Create_Customer_Table.sql 4-15-07 2_Add_e-mail_address_column.sql 4-17-07 3_Add_fax_number_column.sql 4-18-07 4_Add_transaction_table.sql 4-21-07 5_Add_transaction_status_column.sql 4-24-07 6_Add_customer-transaction_view.sql 4-27-07
  • 25. Put them under configuration management control CREATE TABLE money ( eek NUMBER ); //Test for… DB DDL Insert into AA(mydata) Values(11); Meta Data Delete from .. DML Create index optimization Merge into Data Migration Tests Installation scripts
  • 26. Configuration of DB project Database project under version control Tiny db backup Deltascripts of dbdeploy For defining db objects which depends on schema For data onetime migration For data sync For checking dirty data Tool exclusively for database project other scripts…
  • 27. Database Deployment An automated process is needed to make the process of upgrading out-of-date databases
  • 28. Our Practices Nothing is used only once Automate tasks such as Physical table deployment Usage statistics Schema verification Data migration verification Introduce tools ,like Ant, dbdeploy
  • 29. Management DB deploymnet DB Deploy - http://dbdeploy.com/
  • 30. DBDeploy http://dbdeploy.com/documentation/getting-started/rules-for-using-dbdeploy/ N aming convention for delta scripts : NUMBER COMMENT.SQL e.g. 1_Create_Customer_Table.sql … Undo section – marked by comments: CREATE TABLE FOO ( FOO_ID INTEGER NOT NULL, FOO_VALUE VARCHAR(30) ); ALTER TABLE FOO ADD CONSTRAINT PK_FOO PRIMARY KEY (FOO_ID); --//@UNDO DROP TABLE FOO;
  • 31. <target name=&quot;gen-and-exec-delta-script&quot;> <dbdeploy driver=&quot;oracle.jdbc.OracleDriver&quot; url=&quot;jdbc:oracle:thin:@localhost:1521:XE&quot; userid=&quot;dylan&quot; password=&quot;nalyd&quot; dir=&quot;./sql/deltas/&quot; outputfile=&quot;./build_output/db-deltas-hsql.sql&quot; dbms=&quot;ora&quot;/> <sql driver=&quot; oracle.jdbc.OracleDriver&quot; url=&quot; jdbc:oracle:thin:@localhost:1521:XE &quot; userid=&quot;dylan&quot; password=&quot;nalyd&quot; src=&quot;./build_output/db-deltas.sql&quot; onerror=&quot;abort&quot;/> </target> Ant
  • 32. DBDeploy Go to directory with SQL files: “ 1 create_customer_table.sql” “ 2 add_customer_id_constraint.sql” Run “ant” Output: gen-and-exec-delta-script: [dbdeploy] dbdeploy v2.11 [dbdeploy] Reading change scripts from directory C:\Projects\dbdeploy-demo\sql\deltas... [dbdeploy] Changes currently applied to database: [dbdeploy] 1, 2 [dbdeploy] Scripts available: [dbdeploy] 1, 2, 3, 4 [dbdeploy] To be applied: [dbdeploy] 3, 4 [sql] Executing file: C:\Projects\dbdeploy-demo\build_output\db-deltas.sql [sql] 8 of 8 SQL statements executed successfully
  • 33. Automate Tasks CreateNewTestDB upgradeDB <target name= “-parseDbScxripts” >…</target> … <target name=&quot;-upgradeDB&quot; depends=&quot;-parseDbScripts, -dbdeploy, -runDeltaScript, -dropDbLogic, -createDbLogic&quot; description=&quot;Upgrade specified database to latest version&quot; /> <target name=&quot;rebuildDB&quot; depends=&quot; -parseDbScripts, -dropDb, -createDb, -initialiseDb, -dbdeploy, -runDeltaScript, -createDbLogic&quot; description=&quot;drop, recreate and initialise the Connect database&quot; />
  • 34. Batch file: library\nant\nant.exe –buildfile:evovle.build –D:rebuildDB -logfile:build.txt Shared to devs Do what we want by just One Command
  • 35. Reference: Evolutionary Database Design http://martinfowler.com/articles/evodb.html Refactoring Databases: Evolutionary Database Design
  • 36. Q&A

Editor's Notes

  • #3: Changing from waterfall to Agile At an early stage identify requirements , reconcile , design , begin coding minimize changes due extensive preliminary work
  • #4: automation
  • #9: explain
  • #37: Classify and Prioritize data Client list the most important data Try to figure out Data pattern( Data cleaning) Only working on meaningful data Verify data by system logics ( data referencing, records number…) Check it randomly