KYSUC - Keep Your Schema Under Control
0 likes5,749 views
Abstract: A apresentação centra-se na temática de ter forma de controlar, versionar e actualizar toda a parte de Base de Dados de um projecto. Estamos a falar, desde a produção de modelos ER, a versionamento de scripts, passando pelo deploy dos mesmos e terminado na documentação. A apresentação conta ainda com uma breve demonstração do uso da ferramenta Flyway para versionar e controlar a execução de scripts nos diversos ambientes de um projecto. Sobre o Nuno Alves: Chamo-me Nuno Alves nascido em Coimbra, Portugal e vivi maioritariamente em Leiria. Licenciado em Engenharia Informática na ESTG-IPLeiria (Escola Superior de Tecnologia e Gestão) onde o gosto por dados e bases de dados se começou a desenvolver. Daí, profissionalmente a minha área de actuação ser em torno de bases de dados e infra-estruturas. Tenho cerca de 10 anos de experiência repartidos pelas áreas Financeira, Seguros, Governo, Militar em tecnologias que vão desde Oracle, PostgreSQL, MSSQLServer a DB2.
Ad
More Related Content
What's hot (20)
Viewers also liked (20)
Ad
Similar to KYSUC - Keep Your Schema Under Control (20)
Ad
Recently uploaded (20)
KYSUC - Keep Your Schema Under Control
- 1. KYSUC Keep Your Schema Under Control 04.FEV.2016 Nuno Alexandre Alves
- 2. Speaker • Nuno Alves • Born at Coimbra / Lives in Leiria • Degree in Informatics Engineering (ESTG-IPLeiria) • Some experience with databases • Nowadays working in Critical Software, SA [email protected] nuno.alexandre.alves
- 3. Disclaimer This presentation is: • From my personal experience • My own suggestions This presentation is NOT to: • Pointing fingers • Enumerate errors • This is better than that • Do not use this
- 4. Just another day in “paradise” • How do you develop your scripts ? • Do you have a ER model of your Database(s) ? • Do you have your Database documented ? • How do you control your scripts versions ? • Do you have your test data under scripts ? • How do you test your scripts ? • How do you deploy your application and database to a specific version ?
- 5. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
- 6. Goals for Today • Updated ER model • Scripts versioned • Easy Deploy • Easy Documentation • Schema Under Control !
- 7. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
- 8. ER Model (1/4) Start New Project (from requirements) • Build your Conceptual Model • First Physical Model • First Scripts (version 1.0.0) • Print (paper not PDF) your Physical Model and put it in a Wall next to the team (old-school but it works) Assumptions: • High Level Architecture designed • RBDMS almost decided (for the first scripts)
- 9. ER Model (2/4) Start New Project (already in PROD) / Maintenance • Ask for actual documentation (get lucky ) • DDL of actual production database • Import DDL in your development servers • Reverse Engineering • Build Conceptual Model (from Physical) if necessary • Script Physical Model with actual application version (ex: v3.5.2 – it’s your baseline)
- 10. ER Model (3/4) Development / Maintenance • Keep your database changes updated in your ER Model • Use Reverse Engineering to do checkpoints (compare)
- 11. ER Model (4/4) Tools • Use a Case Tool • PowerDesigner • Enterprise Architect • …
- 12. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
- 13. Scripts (1/10) Scripts Development (projects reality) • Manually • I don’t have scripts. My ORM tool does it for me !?! • Database is something to store some data on it. I don’t care.
- 14. Scripts (2/10) Scripts Development (projects reality) • WTF ?! • Your medical records relies on a database system • Your bank account relies on a database system • Your salary relies on a database system • … • Your life relies on too many database systems • Take your own conclusions
- 15. Scripts (3/10) Then someday • How do I execute the database scripts ? Order By ? • Which development database is updated ? • How do you develop scripts ? • One script vs multiple scripts ? • Full Scripts vs Partial Scripts ? • It’s your ORM your Database developer ? • Naming Conventions for objects (where is the guide?) • I have a app log with this error “constraint SYS_098374 violated”. What is that constraint ? • The column CGTRTHY stores what ?
- 16. Scripts (4/10) Database Scripts Under Control • Why ? • Is your source code under control ? • Does your source code have versions ? • Database scripts = source code ? • I think Yes ! • Table definition • Table initial data • Function / Stored Procedure / Package
- 17. Scripts (5/10) Decide your Database Storage Strategy • Check production for rolling projects • Decide according to hosting supplier or client needs • Tablespaces • Data files • File groups My Suggestion (minimum) • One Tablespace for Data • One Tablespace for Indexes • One Tablespace for LOBs
- 18. Scripts (6/10) Decide your Database / Schema(s) architecture • One schema, multiple users with diff grants • Multiple schemas, one/multiple users My Suggestion • Keep it simple • One schema/database, multiple users (grants/views) • FKs between multiple schemas not supported in all RDBMS • Distributed Transactions (do you really want ?!) • Multiple queries (joins in app server) • Are you creating your own RDBMS ?!
- 19. Scripts (7/10) Decide your Database Naming Conventions • Easy identify a database object • Name every object in database • Don’t let default names under RDBMS
- 20. Scripts (8/10) Script like source code • Use SQL formatting (for human readable) • Comment your SQL • Define objects language: PT; EN; FR … (don’t mix) Script for documentation • Comment every object (mandatory: wait and see) • Keep documentation in database dictionary • Useful for Reverse Engineering • Useful for Automatic Documentation • COMMENT ON TABLE • COMMENT ON COLUMN TABLE_NAME.COLUMN_NAME
- 21. Scripts (9/10) Tools • Manually (Notepad, VI, … is your friend!) • Liquibase • mybatis • MIGRATEdb • migrate4j • Flyway • …
- 22. Scripts (10/10) Scripts Under Control ?
- 23. Flyway (1/6) Overview • Scripts: SQL, Java • RDBMS Agnostic (JDBC driver) • Inside your source code • Placeholders (variables) • Multiple Execution Modes • Java API • Maven / Ant • Command line • Ex: V2_0_0_1_0001__script_name.sql
- 24. Flyway (2/6) Decide your Script Naming Convention • V<JIRA_FIX_VERSION>_1_<4_digits>__<JIRA_KEY>.sql • <4_digitos> • Execution order
- 25. Flyway (3/6) Put some code in your Script • Add an header to your SQL file • Add JIRA_KEY or JIRA URL as a comment • Commit
- 26. Flyway (4/6) Decide your Execution Mode • Command line (java -jar …) • Maven/ANT • configure inside your IDE or in CLI • Use Maven profiles is necessary (useful in multiple databases/environments)
- 27. Flyway (5/6) Flyway in action • flyway:info • Check schema version already installed • You need flyway:baseline for the first time
- 28. Flyway (6/6) Flyway in action • flyway:migration • Script execution • With/without target release
- 29. Flyway In Action
- 30. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
- 31. Deploy (1/7) Deploy Database Scripts • Do you test your scripts ? (how many times) • Do you re-create your schema from scratch ? • Do you have automatic deployment for your app ? • And your database ? • Do you have night builds ? • They include database too ? and automated tests ? • Do you simulate your client releases in development environments ? • Do you have confidence in your own scripts ?
- 32. Deploy (2/7) Deploy Database (full / upgrade) • Use your Continuous Integration • Jenkins • Team City • Bamboo • Other …
- 33. Deploy (3/7) Jenkins + Flyway = easy database deploy • Why Jenkins ? • Easy to configure • Parametrized builds • Many plugins (CVS, SVN, Maven, JIRA, Email, …) • Scheduler • Live Build Logs / Console output directly in browser • Why not ?!
- 34. Deploy (4/7) Jenkins + Flyway = easy database deploy • Configure a Jenkins Job • Checkout script from VCS • Put some parameters • if necessary for customs build • for multiple versions • for multiple databases • Execute Flyway • via Maven • via custom Shell Script / Windows batch
- 35. Jenkins + Flyway In Action
- 36. Deploy (5/7) Continuous Monitor your database • Oracle Enterprise Manager (Oracle) • pgwatch (PostgreSQL) • db2top (DB2) • SQL Server Management Studio (SQLServer) • MySQL Enterprise Monitor (MySQL) • … • Check administrator views (available in every RDBMS)
- 37. Deploy (6/7) Continuous Monitor your database • Check some random SQL Statements • CPU • I/O • Get query plan for these statements • Help development team write better SQL • Create indexes according to business needs • Monitor sessions • Check Connection Pooling parameters • Min / Max Pool sizes (warn: multiple Connections Pools)
- 38. Deploy (7/7) Help Development Team to help yourself • Write SQL Statements to log (debug mode) • Ask for log4jdbc / P6Spy / … (with parameters) • Write query times to log • Analyze top time consuming queries • Analyze top running queries • Ask for application times vs database times (in log) • Most of times the application consumes more time (yes it’s true) • Try to analyze how many queries are made for a business operation • Check the number of columns retrieved in every query (it’s really necessary 80 columns ?)
- 39. Agenda 1. Goals 2. Diagrams (ER models) 3. Scripts 4. Deploy 5. Documentation
- 40. Documentation (1/3) How do you produce your database documentation ? • Manually • I don’t have database documentation • “My client doesn’t need this documentation” • And your team ? • How do you maintain a project without documentation? • Most (~All) of your production tickets are resolved by SQL statements ! • Client business data is stored in “your” database (really?!)
- 41. Documentation (2/3) Automatic Documentation • Use RDBMS data dictionary • information_schema.tables, … • user_tables, … • syscat.tables, … • Remember COMMENT ON … from scripts ? • “Export” to something useful • Word Document/PDF ? • Who reads a 800 pages document ?
- 42. Documentation (3/3) Automatic Documentation • Perl script (reads data dictionary, outputs One HTML file) • Send to your client as a deliverable • Publish in your wiki • Open locally in your browser • Produce custom CSS: 30m - 1h • Update Documentation: 1 minute • Remember COMMENT ON
- 43. Summary (from initial goals) (1/2) ER Model • Use a Case Tool • Keep your model updated Scripts versioned • Use Flyway / Liquibase / Other • Script everything (data tests included) • Review your scripts • Naming Conventions • Comments
- 44. Summary (from initial goals) (2/2) Easy Deploy • Use Jenkins / Team City / Bamboo / Other • Keep your database under Continuous Integration • Schedule your database builds • Parametrize your database deploy • Monitor your database close • Release Matrix isn’t nice to have (is mandatory) Easy Documentation • Comment your objects in data dictionary
- 45. Main Goal Updated ER Model + Scripts versioned + Easy Deploy + Easy Documentation = KYSUC
- 46. Extra Credits Define / ask for backup/restore plan • Test your backups (restore is your friend) • Check backup/restore times • Check database growth Compare data • Compare your production data with development data: reference data • Try to equal PKs IDs across environments (for referencial data)
- 47. That’s It! Thank you for your time.