將 Open Data 放上 Open Source Platforms: 開源資料入口平台 CKAN 開發經驗分享

將 Open Data 放上 Open Source Platforms
開源資料入口平台 CKAN 開發經驗分享
@ FOSS and Project Collaboration (Spring 2015)
This work is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Taiwan License.
Presenter: 李承錱 Cheng-Jen Lee (Sol)
Email: cjlee AT iis.sinica.edu.tw

2
About Me
●
Sol, @u10313335
●
Institute of Information Science, Academia Sinica
●
https://about.me/SolLee
●
Python / R / Java
●
Focused Areas
– CMS
– Data Repository
– Open Data
– *nix System Administration

3
Agenda
●
Open Data and Open Data Portals
●
About CKAN
●
CKAN and 5 Open Data★
●
Experiences
●
Contribution: What and How?

4
Open Data and Open Data Portals
●
Open Data
– The idea that certain data should be freely available to
everyone to use and republish as they wish, without
restrictions from copyright, patents or other
mechanisms of control1
.
●
Open Data Portals
– Facilitate access to and re-use of public sector
information2
.
– “Infrastruction” of open data
1. Wikipeida: open data https://en.wikipedia.org/wiki/Open_data
2. Open Data Portals - Digital Agenda for Europe
http://ec.europa.eu/digital-agenda/en/open-data-portals

6
CKAN
●
The Comprehensive Knowledge Archive
Network
●
A powerful data management system
– Publishing
– Sharing
– Finding
– Using Data

8
The Most Popular Platform for
Open Data
116 instances
around the world
in March 2015
http://ckan.org/instances

9
The Most Popular Platform for
Open Data
●
Widely used in government data portal
– In EU member states, 30% open data portals adopted
CKAN (OpenDataMonitor1
, March 2015)
●
Workflow support for publishing data
●
Data Visualization
●
100+ Extensions
●
Powerful APIs
●
Open-sourced (AGPLv3)
1. http://www.opendatamonitor.eu

13
European Union
PUBLICDATA.EU

14
Tainan City
DATA.TAINAN.GOV.TW

15
Nantou County
DATA.NANTOU.GOV.TW

16
Hsinchu City
OPENDATA.HCCG.GOV.TW

18
台江內海研究資料集
TAIJIANG.TW

20
Publish Datasets
① Add Dataset Information

21
Publish Datasets
② Add Data under the Dataset

25
Data Preview and Visualization
recline_view (csv, xls)
Grid

26
Graph

27
Lat/Long fields

28
wms_preview

29
geojson_preview

30
●
Docs: recline_view, text_view, json_view, pdf_view,
webpage_view, officedocs_view...
●
Pics: image_view
●
And more!

31
Authorization
organization
http://opendata.hccg.gov.tw/organization

32
Data Exchange
Harvest and Federation

33
CKAN and 5 Open Data★ 1
1. Tim Berners-Lee, “Linked Data”
http://www.w3.org/DesignIssues/LinkedData.html

34
●
★ Make your stuff available on the Web (whatever
format) under an open license
Customizable licenses

35
●
★★ Make it available as structured data (e.g., Excel
instead of image scan of a table)
★★★ Use non-proprietary formats (e.g., CSV instead of
Excel)
– Upload any data format
– Data API
●
Get records from
structured data
Data API

36
●
★★★★ Use open standards from W3C (RDF and SPARQL) to
identify things, so that people can point at your stuff
●
★★★★★ Link your data to other data to provide context
– Built-in RDF exporting capabilities
– Expose or consume metadata from other catalogs using RDF
(DCAT) docs1
●
ckanext-qa2
: Check the openess of datasets or resources
1. Supported by ckanext-dcat extension
2. https://github.com/ckan/ckanext-qa

39
Installation
●
Official Documents:
– http://docs.ckan.org/en/latest/
●
Installation Notes (In Chinese):
– https://ckan-docs-tw.readthedocs.org/

40
Customizations for Taijiang.tw
●
Custom Metadata
●
Data Visualization
●
Custom filters
●
Harvest
●
Localization
●
Source Code Released under AGPLv3 (On GitHub: u10313335)
– ckanext-taijiang
– ckanext-spatial
– taijiang-ckan-translations
– taijiang-bulk-uploader

41
Custom Metadata
●
Extension ckanext-scheming1
– Configure and share CKAN schemas using a JSON
schema description.
– Custom template snippets for editing and display
fields.
Template Name Function
text.html a simple text field for free-form text
large_text.html a larger text field
date.html a date widget
markdown.html a markdown field
select.html a select box
multiple_choice.html a group of checkboxes
repeating.html a repeating fields
1. https://github.com/open-data/ckanext-scheming, only for CKAN 2.3+

42
Custom Metadata – Example
{
"field_name": "data_type",
"label": {"en": "Data Type", "zh_TW": " 資料類型 "},
"preset": "select",
"form_attrs": {"data-module": "autocomplete"},
"choices": [{"value": "statistics", "label": Statistics"}]
}
{
"field_name": "ref",
"preset": "repeating_text",
"label": {"en": "Reference", "zh_TW": " 參考來源 "},
"form_blanks": 3
}
select
repeating_text

43
Validator and Converter
●
Ensure data quality

44
●
Validator
– Validate user inputs
– Ex. json_validator
def json_validator(value, context):
if value == '':
return value
try:
json.loads(value)
except ValueError:
raise Invalid('Invalid JSON')
return value

45
●
Converter
– Convert data to storage
– Ex. duplicate_validator
def duplicate_validator(key, data, errors,
context):
if errors[key]:
return
value = json.loads(data[key])
unduplicated = list(set(value))
data[key] = json.dumps(unduplicated)

46
Data Visualization
●
There is no viewer for some GIS formats
– WMTS services
– ESRI Shapefile (*.shp and *.dbf)
●
Do It Ourselves!
– wmts_view
– shp_view

47
Write a CKAN Plugin
●
PyUtilib Component Architecture (PCA)
●
Inherits from
– ckan.plugins.SingletonPlugin
●
Implements
– one (or several) ckan.plugins.* interfaces

48
To Build a "viewer"
●
We need more…
– View template (Jinja template engine)
– JavaScript module
●
Ex. Shapefile preview includes shp2geojson.js1
.
1. http://gipong.github.io/shp2geojson.js/ (Released under MIT license)

49
Example: Plugin for SHP Preview
from ckan import plugins as p
class SHPView(p.SingletonPlugin):
p.implements(p.IResourceView,
inherit=True)
def info(self):
return {'name': shp_view',
'title': 'shp',
'icon': 'map-marker',
'iframed': True,
'default_title': 'SHP',
}
def can_view(self, data_dict):
resource = data_dict['resource']
format_lower = resource['format'].lower()
if format_lower in self.SHP:
return self.same_domain or
self.proxy_is_enabled
return False
def view_template(self, context, data_dict):
<div data-module="shppreview" id="data-
preview" data-module-
map_config="{{ h.dump_json(map_config) }}"
></div>
// shapefile preview module
ckan.module('shppreview', function (jQuery, _)
{
Return {
initialize: function () {
…
}
showPreview: function (url, data) {
…
}
}
}
Python Plugin View Template
(shp.html)
JS Module (shp_view.js)

50
Result
http://taijiang.tw/dataset/tainangis-wmts
wmts_view

51
Result
shp_view QGIS
http://taijiang.tw/dataset/proj4-29
shp_view

52
Custom Filters
●
Find Datasets by
– Time period
– Self-defined categories
●
A New Plugin
– For Time Search
●
Implement IPackageController.before_search
– For Self-defined Categories
●
Implement IPackageController.before_index and
Ifacets.dataset_facets
– Both needs new definitions in solr schema

53
Example: Plugin for Time Search
from ckan import plugins as p
class TaijiangDatasets(p.SingletonPlugin):
p.implements(p.IPackageController, inherit=True)
p.implements(p.IFacets)
def before_search(self, search_params):
…
begin = parse_date(search_params['extras']
['ext_begin_date'])
end = parse_date(search_params['extras']['ext_end_date'])
...
query = ("(start_time: [* TO {0}Z] AND end_time: [{0}Z TO
*]) OR (start_time: [{0}Z TO {1}Z] AND end_time: [{0}Z TO *])")
query = query.format(begin.isoformat(), end.isoformat())
search_params['q'] = query
return search_params
def dataset_facets(self, facets_dict, package_type):
facets_dict['date_facet'] = p.toolkit._('Date of Dataset')
return facets_dict
<dynamicField
name="*_time"
type="date"
indexed="true"
stored="true"
multiValued="false"/>
Python Plugin Solr Schema

55
Harvest
●
ckanext-harvest
– Remote harvesting extension
– https://github.com/okfn/ckanext-harvest
●
Source Type
– CKAN
– CSW* (Catalog Service for the Web)
– WAF* (Web Accessible Folder)
– Custom (csv/xls/website… etc.)
*Provided by ckanext-spatial

57
Harvest
Background Process
●
Manually
– (pyenv) $ paster --plugin=ckanext-harvest harvester
gather_consumer/fetch_consumer/run -c
/etc/ckan/default/production.ini
●
Automatically
– Supervisor (for gather & fetch consumer)
– Cron (for run)

58
Harvest
The Harvesting Interface
from base import HarvesterBase
class SRDAHarvester(HarvesterBase):
def _set_config(self,config_str):
def info(self):
...
def gather_stage(self, harvest_job):
…
def fetch_stage(self, harvest_object):
...
def import_stage(self, harvest_object):
...
See http://goo.gl/ZMnND7 for
details.

59
Localization
●
Translation for UI
– Gettext Style i18n
– Babel (*.po & *.mo)
●
In Python
p.toolkit._('String')
●
In Jinja Template
{{ _('String') }}
●
Transifex
Open Knowledge / CKAN
– Jed (For JavaScript Modules)
●
_('String')_

60
Localization
●
Translation for Extensions
– opendatatrentino/ckan-custom-translations (GitHub)
●
Translation for Metadata
– Defined in JSON Schema
– "label": {"en": "Data Type", "zh_TW": " 資料類型 "}

61
Localization
●
Chinese Search
– Solr + mmseg4j1
(A Java Tokenizer)
– Maximum Matching Algorithm2
(By Dr. Chih-Hao Tsai)
– Copy to Solr folder and modify Solr schema
– Ref: http://is.gd/2Vpzgb
1. https://github.com/chenlb/mmseg4j-solr (Released under Apache 2.0
license)
2. http://technology.chtsai.org/mmseg/

62
Contribution: What and How?

63
What to Contribute?
●
CKAN Core Features
– Time and spatial search for private datasets
– Publish datasets as a catalogue service Ex. CSW
– Web interface for bulk uploads
– A simplified deployment process
– Issues on GitHub: https://github.com/ckan/ckan/issues
– More ideas:
https://github.com/ckan/ideas-and-roadmap

64
What to Contribute?
●
i18n
– Non-ascii Filename
– Translate JS Modules (Ex. Recline.js)
– UI Translation (Transifex)

65
What to Contribute?
●
More Functions for Using Data in Web Browser
– Audios & Videos playback (Ex. Integrates plyr.io)
– Link to third party services1
, like Shiny2
(R-based) or
Ipython Notebook (Python-based)
1. http://www.data.gov/meta/open-apps/
2. https://github.com/ckan/ideas-and-roadmap/issues/35

66
What to Contribute?
●
Rebuild data.g0v.tw with CKAN?
● data.g0v.tw ( 零時資料中心 )
– Built with DKAN (A CKAN clone for Drupal)
●
Problems of DKAN
– Development is much slower than CKAN
– Lack of features introduced in latter versions of CKAN
●
Ex. Multiple persistent views of data (In CKAN 2.3)
– Most gov sites in TW use (or will use) CKAN instead of
DKAN

67
How to Contribute?
●
CKAN Core: ckan/ckan (GitHub)
●
Most plugins are also available on GitHub
– http://extensions.ckan.org/
●
Development Discussions (Mailing List)
– https://lists.okfn.org/mailman/listinfo/ckan-dev
●
Contributing Guide
– http://docs.ckan.org/en/latest/contributing/index.html

68
Thanks for your attention!
Any Q?
Email: cjlee AT iis.sinica.edu.tw
Profile: http://about.me/sollee
Google Groups: CKAN Taiwan Interest Group

將 Open Data 放上 Open Source Platforms: 開源資料入口平台 CKAN 開發經驗分享

Recommended

More Related Content

What's hot (20)

Viewers also liked (9)

Similar to 將 Open Data 放上 Open Source Platforms: 開源資料入口平台 CKAN 開發經驗分享 (20)

More from Chengjen Lee (8)

Recently uploaded (20)

將 Open Data 放上 Open Source Platforms: 開源資料入口平台 CKAN 開發經驗分享

Editor's Notes