Scribd
Upload
35 views

Reading in Larger Datasets With Read - Table Reading in Larger Datasets With Read - Table

The document provides tips for reading in larger datasets into R. It recommends checking the memory required, setting comment.char = "" if there are no comments, using colClasses to specify column types to speed up reading, and setting nrows to estimate the number of lines to help with memory usage. It also stresses the importance of knowing your system specifications like available memory and other applications when working with large datasets in R.

Uploaded by

RustEd
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
35 views

Reading in Larger Datasets With Read - Table Reading in Larger Datasets With Read - Table

The document provides tips for reading in larger datasets into R. It recommends checking the memory required, setting comment.char = "" if there are no comments, using colClasses to specify column types to speed up reading, and setting nrows to estimate the number of lines to help with memory usage. It also stresses the importance of knowing your system specifications like available memory and other applications when working with large datasets in R.

Uploaded by

RustEd
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Reading in Larger Datasets with read.

table
With much larger datasets, doing the following things will make your life easier and will prevent R
from choking.
Read the help page for read.table, which contains many hints
Make a rough calculation of the memory required to store your dataset. If the dataset is larger
than the amount of RAM on your computer, you can probably stop right here.
Set comment.char = "" if there are no commented lines in your file.

6/9

Reading in Larger Datasets with read.table


Use the colClasses argument. Specifying this option instead of using the default can make
read.table run MUCH faster, often twice as fast. In order to use this option, you have to know the
class of each column in your data frame. If all of the columns are numeric, for example, then
you can just set colClasses = "numeric". A quick an dirty way to figure out the classes of
each column is the following:

initial <- read.table("datatable.txt", nrows = 100)


classes <- sapply(initial, class)
tabAll <- read.table("datatable.txt",
colClasses = classes)

Set nrows. This doesnt make R run faster but it helps with memory usage. A mild overestimate
is okay. You can use the Unix tool wc to calculate the number of lines in a file.

7/9

Know Thy System


In general, when using R with larger datasets, its useful to know a few things about your system.
How much memory is available?
What other applications are in use?
Are there other users logged into the same system?
What operating system?
Is the OS 32 or 64 bit?

8/9

Calculating Memory Requirements


I have a data frame with 1,500,000 rows and 120 columns, all of which are numeric data. Roughly,
how much memory is required to store this data frame?
1,500,000 120 8 bytes/numeric
= 1440000000 bytes
= 1440000000 / 220 bytes/MB
= 1,373.29 MB
= 1.34 GB

9/9

You might also like