DSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade
0 likes112 views
Presentation by Jan Verkade, Deltares, at the Delft-FEWS International User Days 2020, during Delft Software Days - Edition 2020. Monday, 2 November 2020.
Ad
More Related Content
What's hot (20)
Similar to DSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade (20)
Ad
More from Deltares (20)
Ad
Recently uploaded (20)
![Minitab 22 Full Crack Plus Product Key Free Download [Latest] 2025](https://cdn.slidesharecdn.com/ss_thumbnails/avtpastoralvisits20252026-250407114051-0c8f0fbf-250407123507-bf8f5b04-250426175635-00bc6197-thumbnail.jpg?width=560&fit=bounds)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
DSD-INT 2020 Scripting a Delft-FEWS configuration - Verkade
- 1. Scripting a Delft-FEWS configuration “Scripting is the automated creation of parts of a Delft-FEWS configuration, based on metadata.” • Gauge locations • Basin models • Rating curves • Thresholds • … • Workflow files • Map layer files • Topology files • ModuleDataSetFiles • …
- 3. Scripting a Delft-FEWS configuration What? Why? How?
- 4. 17:05h
- 5. Dr Jan Verkade November 2, 2020 Scripting a Delft-FEWS configuration
- 6. “Scripting a Delft-FEWS configuration 1. Why at all relevant? 2. What do we mean by “scripting”? 3. Why script? 4. Approaches − what do we (not) script? − where do we script from? − what tools do we use? 5. Example: the Deltares global fluvial flow forecasting system 6. Where/ how to start?
- 7. Some notes • Scripting is very much ‘a field in progress’ • Not all questions have been asked • No definitive answers have been given • Slowly, some ‘best practices’are forming This Photo by Unknown Author is licensed under CC BY-NC
- 8. Why spend time talking about scripting? • Delft-FEWS developments in place that allow for scripting to be done − templating of General Adapters, Transformations − storage of metadata in .csv, .dbf/.shp files − increased use of ‘functions’ in location set definition, threshold definitions, etc. • Solves potential problems − efficient configuration building − separation of responsibilities: metadata management v. Delft-FEWS config management − near immediate uptake of metadata changes in forecasting applications − availability of configuration skills • Growing number of applications where scripting is used to build configurations
- 9. What do we mean by “scripting a Delft-FEWS config”? “Scripting is the automated creation of parts of a Delft-FEWS configuration, based on metadata.” • Gauge locations • Basin models • Rating curves • Thresholds • … • Workflow files • Map layer files • Topology files • ModuleDataSetFiles • …
- 10. Why script? 1. Ensure compliance between metadata and configurations 2. Quickly include metadata changes in a forecasting application 3. Impose uniformity on configurations 4. Cost efficiency 5. Allow for separation of tasks: management of metadata v management of configurations 6. … This Photo by Unknown Author is licensed under CC BY-SA-NC
- 11. What do we (not) script? Do script: • config elements that are subject to frequent updates • elements that are repeated throughout the configuration Do not script: • elements that are highly custom (explorer.xml, global.properties, …) • elements that appear few times in your configuration only • elements that don’t change very often → this varies from application to application
- 12. Approaches: where to script from? • file based storage (e.g., .xlsx or .csv files) • local database (e.g., MS Access Database) • database on a server (e.g., SQL) • … Considerations include… • What’s currently available? • Who manages the metadata v. who builds the configuration • Degree of referential data integrity required • Costs • … This Photo by Unknown Author is licensed under CC BY-SA
- 13. Approaches: tools for scripting • Any scripting language. So far we’ve seen scripts in Visual Basic, Python and R. • Export of data to .csv files is almost trivial • Helpful if there is a good xml library available − Matlab: XML toolbox − Python: xml.etree.ElementTree library − R: xml2 package
- 14. 28 november 2019 Some examples
- 15. Scripting “light”: Velika Morava map layer files
- 16. IMFS: England’s new flood forecasting system EA database GUI Python Ingest Telemetry Thresholds Models Delft-FEWS configuration
- 17. IMRD: the metadata repository underlying IMFS Database and GUI developed by JBA. Additional information from [email protected]
- 18. Database and GUI developed by JBA. Additional information from [email protected]
- 19. Database and GUI developed by JBA. Additional information from [email protected]
- 20. Scripts • Python based; written by Martijn Kwant • Produce over 5,100 configuration files − ~2,100 workflow files − ~1,800 ModuleDataSetFiles − ~ 80 mapLayerFiles jointly containing ~380,000 lines of metadata • In future, outputs will automatically be included in the operational client-server system (!)
- 22. gloffis: the Deltares global fluvial flow forecasting system • patchwork of multiple models that can jointly cover the earth’s land • firmly based on wflow models, made possible by ‘three clicks to a model’ principle • models vary in − coverage (and may, and indeed do, overlap) − temporal and spatial resolution − model type (sbm, hbv, …) − required wflow.exe version (2019.1, 2020.1 …) • workflows vary in − model that is called − weather forecast product that is used (currently, choice between various DWD ICON products: regional/global, ensemble/deterministic) → this is where the scripting rationale is: NWPforecasts, wflow models and workflows
- 23. gloffis MS Access Database Python • RegionConfigFiles • WorkflowFiles • DisplayConfigFiles (plus some)
- 24. gloffis metadata repository: MS Access database • Enforces referential data integrity • Allows for user-friendly access and editing of data using forms • Not centrally hosted → pros and cons
- 28. Changes that are made, are conveniently highlightedby version control software
- 29. video
- 30. Scripting: where and how to start? • Does it make sense for me to invest in ‘scripting’? • How ‘dynamic’ is my configuration? Where, in the config, are the dynamics? • How is my metadata organized? • Do I have expertise in scripting available? • … → feel free to ask for help! This Photo by Unknown Author is licensed under CC BY-SA
- 31. Whom to talk to about scripting • First and foremost: your ‘account manager’ • Any of below Deltares staff members… Marc van Dijk [email protected] Jan Verkade [email protected] Bart van Osnabrugge – de Vries [email protected] Martijn Kwant [email protected]
- 32. Jan Verkade [email protected], +31 6 5161 6107 Questions to the audience Have you considered moving to a ‘scripted configuration’? If so, what were/are your considerations?