Scribd
Upload
22 views

Wis2box Automating Data Ingestion (1)

Uploaded by

manga
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
22 views

Wis2box Automating Data Ingestion (1)

Uploaded by

manga
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 6

Automating data ingestion into wis2box

Automating data ingestion

The WIS2 Node serves as a gateway between National MET systems and the WIS 2.0 Network
How data is collected at the source and sent into the WIS2 Node is not governed by the WIS2 standard

National Sector International Sector


Field
Global Discovery
Catalogue
Manual

TP
observer

HT
Data collection
AWS data MQTT
logger
and processing WIS2-node Global Broker
system data
HT consumer
TP

Server, Cloud Global Cache


or Satellite Climate Data
External Management
provider System
wis2box data ingestion
National Sector
storage
data uploaded
incoming public
Dataset driven workflow triggered when wis2box-management
listen to publish to
incoming- public- receives a notification from the storage service
bucket bucket
wis2box-management
Files are matches with a dataset if:
apply data-mappings
create WIS2-notifications
• File-path contains the metadata-id for the dataset, or
create WCMP2-records • File-path contains the topic for the data

An automated process should be configured to send data into the incoming bucket
from the data source
Preferred solution depends on local data processing system in National Sector
Data ingestion script
National Sector
storage Data can be uploaded directly into storage endpoint
data uploaded
public
• For example: crontab using a Python script
incoming
listen to publish to
incoming Python example to upload data using the MinIO client module:
public
bucket bucket import glob
wis2box-management import sys
apply data mappings
from minio import Minio
create WIS2 -notifications
create WCMP2 records
filepath = '/home/wis2box-user/local-data/mydata.bin'
# path should match the metadata or the topic in the data mappings
minio_path = 'urn:wmo:md:it-roma_met_centre:surface-weather-
observations'

endpoint = 'http://localhost:9000'
WIS2BOX_STORAGE_USERNAME = 'wis2box'
WIS2BOX_STORAGE_PASSWORD = 'XXXXXXXXX'

client = Minio(
endpoint=endpoint,
access_key=WIS2BOX_STORAGE_USERNAME,
secret_key=WIS2BOX_STORAGE_PASSWORD,
secure=False)

filename = filepath.split('/')[-1]
wis2box-ftp
National Sector
storage
‘wis2box-ftp’: optional service to aid ingestion over FTP
dat
au
plo incoming public • copies any file received to preconfigured storage endpoint,
ade
d
listen to publish to preserving the full filepath to match with dataset
Optional ingestion tool:
wis2box-ftp
incoming
bucket
public
bucket
https://github.com/wmo-im/wis2box-ftp
wis2box-management
apply data mappings
create WIS2 notifications
create WCMP2 records

Example of wis2box-ftp environment variables:


MYHOSTNAME=hostname.domain.tld

FTP_USER=wis2box
FTP_PASS=wis2box123
FTP_HOST=${MYHOSTNAME}

WIS2BOX_STORAGE_ENDPOINT=http://$
{MYHOSTNAME}:9000
WIS2BOX_STORAGE_USERNAME=wis2box
Thank you
Merci
谢谢

You might also like