See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/308802095

Downloading species occurrence data using the GBIF web-service API

Presentation · September 2016

DOI: 10.13140/RG.2.2.19572.35202

CITATIONS READS
0 1,697

1 author:

Dag Terje Filip Endresen

University of Oslo
62 PUBLICATIONS   256 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Nordic Crop Wild Relatives in iNaturalist View project

BioDATA – Biodiversity data management skills for students View project

All content following this page was uploaded by Dag Terje Filip Endresen on 04 October 2016.

The user has requested enhancement of the downloaded file.

NINA, Trondheim, 15th September 2016

GBIF data use

Dag Endresen
GBIF Norway
UiO Natural History Museum in Oslo
University of Oslo

Thursday, September 15th, 2016

Slides: CC-BY-4.0, GBIF.no

 Status
14th September 2016

GBIF enables free and open access to

biodiversity data online.

We are an interna2onal government-ini2ated and -funded ini2a2ve focused on

making biodiversity data available to all and anyone, for scien2fic research,
conserva2on and sustainable development.
GBIF provides a data discovery system
that is dependent on resolvable stable iden;fiers for efficient
func;onality

global registry data portal

3
 GBIF and GEO
Intergovernmental group on earth observations

GEO BON
Biodiversity observa2on network

Data Integration & Interoperability

GBIF provides the infrastructure delivering species occurrence
data in GEO.
GBIF BY THE NUMBERS

649,054,525
species occurrence records

32,440
datasets

813
data-publishing institutions

http://www.gbif.org | 06 JUN 2016

GBIF BY THE NUMBERS: MAY 2016

+3,818,408
species occurrence records

+4,267
datasets

+4
data-publishing institutions

http://www.gbif.org | 6 JUN 2016

data mobilizatio
DATA PUBLISHED THROUGH GBIF.ORG
Occurrence records (millions)
700

650

600

550

500

450

400

350

300

250

200

150

100

http://www.gbif.org | 6 JUN 2016

 Asia (lack of data)

Africa (lack of data)

participatio
MAP OF GBIF COUNTRY PARTICIPANTS

Asia (lack of data)

Africa (lack of data)

August 2016
data publishin
DATA—BY GBIF PARTICIPANT Status May 2016

Other
United States

So
ut
hA
fri
Cos ca
taR
ica

Austra
li a

Denmark Norway
Netherlands

Belgium

Germany

Spain
Norway
Norway

Number of new records published—Top 10 participant Countries Total number of records published—Top 10 Participant Countries
(1 to 31 May 2016) (as of 31 May 2016)

1. United States 3,348,499 6. Belgium 1,620423 1. United States 271,901,500 6. Netherlands 24,241,092

2. Denmark 2,972,094 7. Netherlands 1,094,804 2. Sweden 53,776,182 7. Norway 23,811,863

3. Germany 2,868,240 8. Australia 859,896 3. United Kingdom 49,786,646 8. Germany 22,151,479

4. Norway 2,322,797 9. Costa Rica 810,035 4. France 39,896,982 9. Finland 16,612,735

5. Spain 2,238,363 10. South Africa 436,236 5. Australia 37,489,401 10. Spain 13,630,866

NOTE: Datasets are assigned to countries according to the location of the publishing institution,
including aggregated datasets with contributors from many other countries. http://www.gbif.org | 09 JUN 2016
use of gbif.or
DATA DOWNLOAD REQUESTS, BY COUNTRY
1 January – 31 May 2016

Total of
37,552 requests
From 5,131 users in
127 countries, islands
and territories

1. United States 7128 6. Colombia 2235

2. Mexico 5526 7. Italy 1319

3. Brazil 3079 8. China 1263

4. United Kingdom 2670 9. France 949

5. Spain 2478 10. Australia 858

Norwegian scien2sts generally use Artskart…

Requests for download do not necessarily result in data actually being downloaded. Based on country indicated by user login | 06 JUN 2016
data us
CITATIONS IN PEER-REVIEWED RESEARCH
Annual number of peer-reviewed publications using GBIF-mediated data

9 JUN 2016
research us
USE CITATIONS, BY COUNTRY OF AUTHORS
May 2016
1. United States 15 7. Australia 4
2. Germany 9 7. Brazil 4
3. China 5 9. Canada 3
3. France 5 9. Netherlands 3
3. Spain 5 9. South Africa 3
3. United Kingdom 5

Number of research publications in May 2016 citing use of

GBIF-mediated data, ranked by country according to affiliation of author.
Top 11 countries shown.

Total 2016
1. United States 49 5. Brazil 14
2. Germany 22 5. United Kingdom 14
3. France 18 8. Australia 11
4. China 17 8. Spain 11
5. Mexico 14 10. Canada 10

Number of research publications in 2016 citing use of GBIF-mediated data,

Norway ranked by country according to affiliation of author. Top 10 countries shown.

10 JUN 2016
research us
RESEARCH EXAMPLES (FOR NORWAY)
•  Araújo R, Assis J, Aguillar R, Airoldi L, Bárbara I, Bartsch I, Bekkby T et al. (2016)
Status, trends and drivers of kelp forests in Europe: an expert assessment.
Biodiversity and Conservation 25(7) 1319-1348.

•  Jb N (2016)
Some interesting lichenized fungi from old Fraxinus excelsior and Ulmus glabra in
Norway, including four new country records. Graphis Scripta 28(1-2) 17-21.

•  Newbold T, Hudson L, Hill S, Contu S, Gray C, Scharlemann J, Sheil D et al.

(2016)
Global patterns of terrestrial assemblage turnover within and among land uses.
Ecography.

hGp://www.gbif.org/country/NO/publica2ons
A complete archive of research citing use of GBIF can be accessed at http://www.mendeley.com/groups/1068301/gbif-public-library
10 JUN 2016
GBIF portal:

22,0 million occurrences with loca2ons in Norway.

Published from 31 countries worldwide.

Updated 5 September 2016

GBIF portal:

21,5 million occurrences from Norwegian ins2tu2ons.

Coverage 219 countries worldwide.

Updated 5 September 2016

 STATUS FOR NORDIC GBIF NODES (DATA HOSTED BY…)

hGp://www.gbif.org/country/NO
Danmark Finland

Norway Sweden

Sept 2016 Datasets Occurrences

Denmark 66 + 1 10 905 213
Finland 54 3 611 729
Iceland 4 458 705
Norway 112 + 2 21 684 727
Iceland
Sweden 42 53 787 704
Download data
GBIF DATA PORTAL
SPECIES SEARCH
 Portal API
webservices
GBIF DATA PORTAL API

An interface to access
data published through
the GBIF network using
web services.

PORTAL API
GBIF Data Portal API:
h9p://api.gbif.org/v1/ (+parameters)

Summary and informa2on:
hGp://www.gbif.org/developer/summary

The RESTful API take search parameters as
key=value pairs and respond with json content
type.

RESTful query format
JSON response type
GBIF API SECTIONS

•  Registry
informa2on about the datasets, organiza2ons (e.g. data
publishers), networks and the means to access them (technical
endpoints)
•  Species
informa2on about species and higher taxa, and u2lity services for
interpre2ng names and looking up the iden2fiers (access to all
published checklists in the GBIF checklist bank)
•  Occurrence
occurrence informa2on crawled and indexed by GBIF and search
services to do real 2me paged search and asynchronous
download services to do large batch downloads
•  Maps
simple services to show the maps of GBIF mobilized content
API EXAMPLE : DATASET

Search for datasets by publishing country:

http://api.gbif.org/v1/dataset/search?publishingCountry=NO

Dataset information (UiO NHM Lichens):

http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-
ed1015b26252

Contacts for a dataset:

http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-
ed1015b26252/contact

Dataset endpoint (get the download URL):

http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670-
ed1015b26252/endpoint

http://www.gbif.org/developer/registry
API EXAMPLE : SPECIES
List all name usages (across all checklists):
http://api.gbif.org/v1/species?name=Beta%20vulgaris

Name usage across checklists (Beta vulgaris, 5383920):

http://api.gbif.org/v1/species/5383920/related

Name parsed into epithets and author etc.:

http://api.gbif.org/v1/parser/name?name=Abies%20alba
%20Mill.%20sec.%20Markus%20D.
{"scientificName": "Abies alba Mill. sec. Markus D.",
"type": "SCINAME",
"genusOrAbove": "Abies",
"specificEpithet": "alba",
"authorsParsed": true,
"authorship": "Mill.",
"sensu": "sec. Markus D.",
"canonicalName": "Abies alba",
"canonicalNameWithMarker": "Abies alba",
"canonicalNameComplete": "Abies alba Mill."
}

http://www.gbif.org/developer/species
API EXAMPLE : OCCURRENCE

List occurrences of Beta vulgaris:

http://api.gbif.org/v1/species/match?name=Beta+vulgaris => taxonKey
http://api.gbif.org/v1/occurrence/search?taxonKey=5383920

List occurrences from Norway (of Beta vulgaris):

http://api.gbif.org/v1/occurrence/search?publishingCountry=NO
http://api.gbif.org/v1/occurrence/search?publishingCountry=NO&taxonKey=5383920

Information about a single occurrence record:

http://api.gbif.org/v1/occurrence/1040970640
http://api.gbif.org/v1/occurrence/1040970640/fragment
http://api.gbif.org/v1/occurrence/1040970640/verbatim

List occurrence counts for datasets of country (or taxon):

http://api.gbif.org/v1/occurrence/counts/datasets?country=NO

http://www.gbif.org/developer/occurrence
API EXAMPLE : DOWNLOAD DATA
Lookup speciesKey (1) and download occurrences (2):

http://api.gbif.org/v1/species/match?
verbose=false&kingdom=Plantae&name=Beta+vulgaris
=> usageKey/speciesKey = 5383920

http://api.gbif.org/v1/occurrence/search?
taxonKey=5383920 [&limit=1000&offset=0]
=> notice: count = 25 513
=> then: page through results…
(using offset & limit)

http://api.gbif.org/v1/occurrence/download/request
[POST] => downloadKey (see next slide)
API EXAMPLE : ASYNCHRONOUS (1)
Request asynchronous download:

$ curl -i --user yourGbifUserName:yourGbifPassord -H

"Content-Type: application/json" -H "Accept: application/json"
-X POST -d @filter.json http://api.gbif.org/v1//occurrence/
download/request >> log.txt

Search parameters in a json text file: filter.json (in current

directory or located in a “PATH-directory”):
{
"creator":”yourGbifUserName",
"notification_address": [“[email protected]"],
"predicate":
{
"type":"and",
"predicates":
[{"type":"equals","key":"HAS_COORDINATE","value":"false"},
{"type":"equals","key":"TAXON_KEY","value":"5383920"}]
}
}
DOWNLOADS ARE AVAILABLE IN THE
PORTAL (FROM YOUR USER PROFILE)
API EXAMPLE : ASYNCHRONOUS (2A)

Request asynchronous download:

function gbifapi {
curl -i –user yourGbifUserName:yourGbifPassword -H "Content-Type:
application/json" -H "Accept: application/json" -X POST -d "{\"creator\":
\”yourGbifUserName\", \"notification_address\": [\”[email protected]\"],
\"predicate\": {\"type\":\"and\", \"predicates\": [{\"type\":\"equals\",\"key\":
\"HAS_COORDINATE\",\"value\":\"true\"}, {\"type\":\"equals\", \"key\":
\"TAXON_KEY\", \"value\":\"$1\"}] }}" http://api.gbif.org/v1/occurrence/
download/request >> log.txt
echo -e "\r\n$1 $2\r\n\r\n----------------\r\n\r\n" >> log.txt
}

$ gbifapi 4140730 "Aciachne acicularis"

$ gbifapi 4140704 "Aciachne flagellifera"
$ gbifapi 5289784 "Aegilops comosa”
…
API EXAMPLE : ASYNCHRONOUS (2B)

(…clean log.txt with the downloadKeys using regular

expressions…)

function gbifwget {
echo -e "\n\n----------------\n$1 $2 $3\n" >> log_wget.txt
wget http://api.gbif.org/v1/occurrence/download/request/$1.zip 2>&1 | tee /
dev/tty >> log_wget.txt
mv $1.zip ./dwca/$2.zip 2>&1 | tee /dev/tty >> log_wget.txt
}

$ gbifwget 0006050-141024112412452 4140730 "Aciachne acicularis"

$ gbifwget 0006053-141024112412452 4140704 "Aciachne flagellifera"
$ gbifwget 0006056-141024112412452 5289784 "Aegilops comosa"
…

(work in progress…)
 Slide by Daniel Amariles, 2013

MAPPING API V1.0

You can easily overlay GBIF content on
your own maps.
http://www.gbif.org/developer/maps
 Slide by Daniel Amariles, 2013

MAPPING API V1.0

This service is intended for use with commonly used map clients such
as the Google Maps API, Leaflet JS library or Modest maps JS library.




hGp://leafletjs.com/




hGp://modestmaps.com/

These libraries allow the GBIF layers to be visualized with other
content, such as those coming from Web Map Service (WMS)
providers. It should be noted that the mapping API is not a WMS
service, nor does it support WFS capabili2es.
USEFUL TOOLS (JSON & REST)

REST client …
JSON client/parser …
JSONView (Firefox, Chrome, …)
http://jsonview.com/
Display formatted JSON in browser

R CRAN : jsonlite
http://cran.r-project.org/web/packages/jsonlite/
E.g. read json into a dataframe [link]

OpenRefine
http://openrefine.org/
ROPENSCI : RGBIF
library(rgbif)
key <- name_backbone(name='Hepatica nobilis', kingdom=‘Plantae')$speciesKey
sp <- occ_search(taxonKey=key, return='data', hasCoordinate=TRUE, limit=1000)
gbifmap(sp)
R CRAN

rOpenSci provides programmatic access to scientific data

with R (rgbif, taxize, EML, geonames, …).

https://github.com/ropensci
http://ropensci.org/packages/

http://ropensci.org/tutorials/rgbif_tutorial.html
http://ropensci.org/tutorials/taxize_tutorial.html
RASTER : WORLDCLIM, BIOCLIM LAYERS
# using GBIF data (bv) from the previous slide…
library(raster)
xy <- cbind('lon'=bv$decimalLongitude, 'lat'=bv$decimalLatitude);
env <- getData('worldclim', var='bio', res=10) # bioclim (pkg raster)
plot(env, 1) # plot the first bioclim layer
points(xy[,'lon'], xy[,'lat'], col='red') # plot points
bio <- extract(env, xy); # extract environment to points (pkg raster)
bv_bio <- cbind(bv, bio); # column-bind GBIF-data and bioclim
ROPENSCI : RWBCLIMATE
library(rWBclimate, ggplot2)
country_dat <- get_historical_temp(c("NOR", "SWE", "DNK", "FIN"), "year")
ggplot(country_dat, aes(x = year, y = data, group = locator)) +
theme_bw(base_size=18) + geom_point() + geom_path() +
labs(y="Average annual temperature of Nordic countries", x="Year") +
stat_smooth(se = F, colour = "black") +
facet_wrap(~locator, scale = "free")
RESOLVE TAXONOMIC NAMES
library(taxize) # rOpenSci Taxize
gnr <- gnr_resolve(names = "Beta vuulgariss") # Misspelled name
gnr$results # display suggested names
submitted_name matched_name data_source_title score
1 Beta vuulgariss Beta vulgaris L. Catalogue of Life 0.75
2 Beta vuulgariss Beta vulgaris L. ITIS 0.75
3 Beta vuulgariss Beta vulgaris NCBI 0.75
4 Beta vuulgariss Beta vulgaris var.-gr. crassa Alef. GRIN Taxonomy for Plants 0.75

specieslist <- c("Beta vulgaris", "Phleum pratensis", "Nicotiana glauca")

classification(specieslist, db = 'itis') # lookup higher taxonomy

db = ’i2s'
db = ’col'

Global Names Resolver: http://resolver.globalnames.org/

rOpenSci Taxize: http://ropensci.org/tutorials/taxize_tutorial.html
ROPENSCI : EML

library(EML, rfigshare)

description <- "My dataset published in GBIF"

eml_write(dat = dat, meta, title = "My Dataset",

description = description, creator = "Your Name
<[email protected]>", file = "dataset.xml")

eml_publish("dataset.xml", description = description,

categories = "Ecology", tags = "biodiversity", destination
= "figshare", visibility = "public")

meta <- eml_read("eml_example.xml")

GBIF API SUPPORT

Subscribe to the mailing-list for help and

information messages:
[email protected]
 Node team at NHM, University of Oslo
Dag Endresen, Node manager
Chris2an Svindseth, Database manager

Fridtjof Mehlum, Research director
Einar Timdal, Associate professor
Geir Søli, Associate professor
Vidar Bakken, Consultant

Artsdatabanken, Trondheim
Wouter Koch
Nils Valland

NTNU University Museum
Anders Finstad, GBIF Science commiKee

Research Council of Norway
Per Backe-Hansen, Head of delega;on

Contact us at: [email protected]

View publication stats