Introduction to data.table in R
2 likes1,807 views
Presentation given at the November meeting of the Sheffield R Users Group on the data.table package in R.
![Examples
WHO TB data:
D <- read.csv('TB_burden_countries_2014-09-10.csv')
names(D)[1:10]
## [1] "country" "iso2" "iso3"
## [5] "g_whoregion" "year" "e_pop_num"
## [9] "e_prev_100k_lo" "e_prev_100k_hi"](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-6-320.jpg)
![Examples
WHO TB data:
head(D[,c(1,6,8)])
## country year e_prev_100k
## 1 Afghanistan 1990 327
## 2 Afghanistan 1991 359
## 3 Afghanistan 1992 387
## 4 Afghanistan 1993 412
## 5 Afghanistan 1994 431
## 6 Afghanistan 1995 447](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-7-320.jpg)
![Examples
Mean TB in Afghanistan
mean(D[D$country=='Afghanistan','e_prev_100k'])
## [1] 397.6087
As data.table:
library(data.table)
E <- as.data.table(D) #convert
E[country=='Afghanistan',mean(e_prev_100k)]
## [1] 397.6087](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-8-320.jpg)
![Examples
dataframe multi-column access:
D[D$country=='Afghanistan',
c('e_prev_100k','e_prev_100k_lo',
'e_prev_100k_hi')]
data.table multi-column means, renamed:
E[country=='Afghanistan',
list(mid=mean(e_prev_100k),
lo=mean(e_prev_100k_lo),
hi=mean(e_prev_100k_hi))]
## mid lo hi
## 1: 397.6087 187.913 684.7391](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-9-320.jpg)
![Examples
Means for each country? data.table solution:
E[,list(mid=mean(e_prev_100k)),by=country]
## country mid
## 1: Afghanistan 397.60870
## 2: Albania 29.52174
## 3: Algeria 133.95652
## 4: American Samoa 15.09130
## 5: Andorra 30.71304
## ---
## 215: Wallis and Futuna Islands 117.86957
## 216: West Bank and Gaza Strip 11.14783
## 217: Yemen 180.30435
## 218: Zambia 501.39130
## 219: Zimbabwe 386.30435](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-10-320.jpg)
![Examples
A more complicated example:
E[,
list(lo=mean(e_prev_100k_lo),
hi=mean(e_prev_100k_hi)),
by=list(country,
century=factor(year<2000)
)]](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-11-320.jpg)
![Examples
eo <- E[,plot(sort(e_prev_100k))]
0 1000 2000 3000 4000 5000
050010001500
Index
sort(e_prev_100k)
(1-
line combination with aggregations](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-13-320.jpg)
![Fast insertion
A new column can be inserted by:
E[,country_t := paste0(country,year)]
head(E[,country_t])
## [1] "Afghanistan1990" "Afghanistan1991" "Afghanistan1992
## [5] "Afghanistan1994" "Afghanistan1995"](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-14-320.jpg)
![Keys: fast row retrieval
Need to pre-compute (setkey line)
setkey(E,country) #must be sorted
E['Afghanistan',e_inc_100k]
## country e_inc_100k
## 1: Afghanistan 189
## 2: Afghanistan 189
## 3: Afghanistan 189
## 4: Afghanistan 189
## 5: Afghanistan 189
## 6: Afghanistan 189
## 7: Afghanistan 189
## 8: Afghanistan 189
## 9: Afghanistan 189
## 10: Afghanistan 189
## 11: Afghanistan 189
## 12: Afghanistan 189](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-15-320.jpg)
![Gotchas: column access
E[,1]
## [1] 1
E[,1,with=FALSE]
## country
## 1: Afghanistan
## 2: Afghanistan
## 3: Afghanistan
## 4: Afghanistan
## 5: Afghanistan
## ---
## 4899: Zimbabwe
## 4900: Zimbabwe
## 4901: Zimbabwe
## 4902: Zimbabwe
## 4903: Zimbabwe](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-16-320.jpg)
![Gotchas: copying
E2 <- E
E[,foo:='bar']
head(E2[,foo])
## [1] "bar" "bar" "bar" "bar" "bar" "bar"](https://image.slidesharecdn.com/previewofthedata-150120101311-conversion-gate01/85/Introduction-to-data-table-in-R-17-320.jpg)
![Introduction to Pandas and Time Series Analysis [PyCon DE]](https://cdn.slidesharecdn.com/ss_thumbnails/introductiontopandasandtimeseriesanalysispyconde-170617163724-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Viewers also liked (11)
Similar to Introduction to data.table in R (20)
More from Paul Richards (7)
Recently uploaded (20)
Introduction to data.table in R
- 1. data.table talk January 21, 2015
- 2. The data.table package author: Pete Dodd date: 4 November, 2014
- 3. dataframes in R What is a dataframe? default R objects for holding data can mix numeric, and text data ordered/unordered factors many statistical functions require dataframe inputs
- 4. dataframes in R Problems: print! slow searching verbose syntax no built-in methods for aggregation Which is most annoying depends on who you are. . .
- 5. Constructing data.tables myDT <- data.table( number=1:3, letter=c('a','b','c') ) # like data.frame constructor myDT2 <- as.data.frame(myDF) #conversion The data.table class inherits dataframe, so data.tables (mostly) can be used exactly like dataframes, and should not break existing code.
- 6. Examples WHO TB data: D <- read.csv('TB_burden_countries_2014-09-10.csv') names(D)[1:10] ## [1] "country" "iso2" "iso3" ## [5] "g_whoregion" "year" "e_pop_num" ## [9] "e_prev_100k_lo" "e_prev_100k_hi"
- 7. Examples WHO TB data: head(D[,c(1,6,8)]) ## country year e_prev_100k ## 1 Afghanistan 1990 327 ## 2 Afghanistan 1991 359 ## 3 Afghanistan 1992 387 ## 4 Afghanistan 1993 412 ## 5 Afghanistan 1994 431 ## 6 Afghanistan 1995 447
- 8. Examples Mean TB in Afghanistan mean(D[D$country=='Afghanistan','e_prev_100k']) ## [1] 397.6087 As data.table: library(data.table) E <- as.data.table(D) #convert E[country=='Afghanistan',mean(e_prev_100k)] ## [1] 397.6087
- 9. Examples dataframe multi-column access: D[D$country=='Afghanistan', c('e_prev_100k','e_prev_100k_lo', 'e_prev_100k_hi')] data.table multi-column means, renamed: E[country=='Afghanistan', list(mid=mean(e_prev_100k), lo=mean(e_prev_100k_lo), hi=mean(e_prev_100k_hi))] ## mid lo hi ## 1: 397.6087 187.913 684.7391
- 10. Examples Means for each country? data.table solution: E[,list(mid=mean(e_prev_100k)),by=country] ## country mid ## 1: Afghanistan 397.60870 ## 2: Albania 29.52174 ## 3: Algeria 133.95652 ## 4: American Samoa 15.09130 ## 5: Andorra 30.71304 ## --- ## 215: Wallis and Futuna Islands 117.86957 ## 216: West Bank and Gaza Strip 11.14783 ## 217: Yemen 180.30435 ## 218: Zambia 501.39130 ## 219: Zimbabwe 386.30435
- 11. Examples A more complicated example: E[, list(lo=mean(e_prev_100k_lo), hi=mean(e_prev_100k_hi)), by=list(country, century=factor(year<2000) )]
- 12. Examples Output: ## country century lo hi ## 1: Afghanistan TRUE 189.20000 749.80000 ## 2: Afghanistan FALSE 186.92308 634.69231 ## 3: Albania TRUE 13.20000 65.40000 ## 4: Albania FALSE 10.59231 47.53846 ## 5: Algeria TRUE 49.40000 212.80000 ## --- ## 427: Yemen FALSE 62.69231 218.38462 ## 428: Zambia TRUE 291.60000 1024.90000 ## 429: Zambia FALSE 197.00000 733.76923 ## 430: Zimbabwe TRUE 14.81000 1074.60000 ## 431: Zimbabwe FALSE 56.07692 1219.61538
- 13. Examples eo <- E[,plot(sort(e_prev_100k))] 0 1000 2000 3000 4000 5000 050010001500 Index sort(e_prev_100k) (1- line combination with aggregations
- 14. Fast insertion A new column can be inserted by: E[,country_t := paste0(country,year)] head(E[,country_t]) ## [1] "Afghanistan1990" "Afghanistan1991" "Afghanistan1992 ## [5] "Afghanistan1994" "Afghanistan1995"
- 15. Keys: fast row retrieval Need to pre-compute (setkey line) setkey(E,country) #must be sorted E['Afghanistan',e_inc_100k] ## country e_inc_100k ## 1: Afghanistan 189 ## 2: Afghanistan 189 ## 3: Afghanistan 189 ## 4: Afghanistan 189 ## 5: Afghanistan 189 ## 6: Afghanistan 189 ## 7: Afghanistan 189 ## 8: Afghanistan 189 ## 9: Afghanistan 189 ## 10: Afghanistan 189 ## 11: Afghanistan 189 ## 12: Afghanistan 189
- 16. Gotchas: column access E[,1] ## [1] 1 E[,1,with=FALSE] ## country ## 1: Afghanistan ## 2: Afghanistan ## 3: Afghanistan ## 4: Afghanistan ## 5: Afghanistan ## --- ## 4899: Zimbabwe ## 4900: Zimbabwe ## 4901: Zimbabwe ## 4902: Zimbabwe ## 4903: Zimbabwe
- 17. Gotchas: copying E2 <- E E[,foo:='bar'] head(E2[,foo]) ## [1] "bar" "bar" "bar" "bar" "bar" "bar"
- 18. Gotchas: copying This is because copying is by reference. Use: E2 <- copy(E) instead.
- 19. Summary more compact faster (sometimes lots) less memory great for aggregation/exploratory data crunching But: - a few traps for the unwary Good package vignettes & FAQ,
- 20. Related aggregate in base R plyr: use of ddply sqldf: good if you know SQL RSQLlite: ditto other: - RODBC etc: talk to databases - dplyr: nascent, by Hadley, internal & external