How to deal with nested lists in R?
1 like103 views
This document discusses how to deal with nested lists in R using the purrr, furrr, and future packages. It summarizes working with nested list data from a JSON response, including adding custom IDs to nested data frames and parallelizing the process using future_map to speed it up. Anonymous functions are also discussed as they are used with the apply functions in purrr, and examples are provided of their syntax.
![Examples
11
# anonymous function syntax in purrr
your.list = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377] , 233,
377, 610, 987, 1597
purrr::map(your.list, ~.x * 0.75)
purrr::map(your.list, function(single.element){
single.element * 0.75
})](https://image.slidesharecdn.com/kopiawhyrautomatyczniezapisany1-210517122149/85/How-to-deal-with-nested-lists-in-R-11-320.jpg)
More Related Content
What's hot (19)
Similar to How to deal with nested lists in R? (20)
More from Sotrender (19)
Recently uploaded (20)
How to deal with nested lists in R?
- 1. How to deal with nested lists in R? Using the purrr, furrr and future packages in practice. Lidia Kołakowska • Data Scientist, Sotrender Why R? 2019 Conference • Warsaw, 29.09.2019 }
- 2. Data set ◉ Downloaded data on social issues, elections or politics ads from Facebook Ad Library API ◉ JSON-formatted response ◉ Facebook allows publish articles or research about or related to the use of the Ad Library API e.g. political advertising analysis 2
- 3. 3 Example nested list loaded from JSON
- 4. Nested lists ◉ Elements of list e.g. demographic_distribution are data frames – we cannot easily flatten them into a data frame ◉ Custom ids are needed for elements in nested list elements to connect them to the parent elements 4
- 5. How to deal with data in nested list? 5
- 6. Key packages 6 purrr furrr future /HenrikBengtsson/future /tidyverse/purrr /DavisVaughan/furrr
- 7. Environment setup 7 # load necessary packages library(dplyr) library(jsonlite) library(purrr) library(fs) library(furrr) library(future) # define project path and data directory proj.dir = "/home/lkolakowska/path/to/your/project/” data.dir = proj.dir %>% paste0("data/")
- 8. Enviroment 8
- 9. Loading data into R 9 # 1. load data from json format data.microtargeting = data.dir %>% dir_ls %>% map(~dir_ls(.x, regexp = "json")) %>% unlist() %>% map(fromJSON) %>% map("data")
- 10. What is anonymous functions? ◉ They are used in apply() functions family ◉ In purrr packages they are user-specified, one-sided formula ◉ They have no identity and no name ◉ They will not live in the global environment 10
- 11. Examples 11 # anonymous function syntax in purrr your.list = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377] , 233, 377, 610, 987, 1597 purrr::map(your.list, ~.x * 0.75) purrr::map(your.list, function(single.element){ single.element * 0.75 })
- 12. “ „When it’s not worth the effort to give it a name" 12 When should you use anonymous functions? Hadley Wickham
- 13. Anonymous functions in pipe 13 # 2. add custom ids to nested data frames data.microtargeting = data.microtargeting %>% purrr::compact() %>% map(function(df) { custom_ad_id = seq( 1, nrow(df), by = 1) df = df %>% mutate(ad_id = paste0(page_id, "_", custom_ad_id)) return(df) })
- 14. The process takes too long… 14
- 15. Let’s parallaize software! ◉ Speed up processing ◉ Decrease memory footprint ◉ Avoid data transfer 15
- 16. 16 mclapply
- 17. Parallization using pipe in R 17 # 3. parallaize process plan(multicore= 6) data.microtargeting = data.microtargeting %>% purrr::compact() %>% future_map(function(df) { custom_ad_id = seq( 1, nrow(df), by = 1) df = df %>% mutate(ad_id = paste0(page_id, "_", custom_ad_id)) return(df) }, .progress = TRUE)
- 18. Summary ◉ Working with nested lists can be very efficient ◉ First perform operations on the individual elements of the list, only then combine them into one large data frame ◉ Speed up your work by performing a parallaized process 18
- 19. Resources that made it easier to create this presentation ☺ ◉ H. Bengtsson, „Future: Parallel & Distributed Processing in R for Everyone”, eRum 2018, Budapest ◉ H. Bengtsson, „Future: Friendly Parallel Processing in R for Everyone”, SatRday Paris 2019 ◉ M. Jones, „Quick Intro to Parallel Computing in R”, 2017 ◉ L. Singham, „Anonymous Functions in R and Python”, 2017 ◉ H.Wickham, „Advanced R” - Functional programming ◉ Cool but useless, „Anonymous Functions in R - Part 1”, 2019 19
- 20. 20 I am Lidia Kołakowska You can find me at ◉ /in/lidia-kolakowska/ ◉ /lidkol Thanks!