Map a function over a file by chunks

A wrapper around the core iotools functions to easily apply a function over chunks of a large file. Results can be either written to a file or returned as an internal list.

chunk.map(input, output = NULL, formatter = .default.formatter,
          FUN, key.sep = NULL, max.line = 65536L,
          max.size = 33554432L, output.sep = ",", output.nsep = "\t",
          output.keys = FALSE, skip = 0L, ...)

Arguments

input

an input connection or character vector describing a local file.

output

an optional output connection or character vector describing a local file. If NULL, the results are returned internally as a list.

formatter

a function that takes raw input and produces the input given to FUN

FUN

a user provided function to map over the chunks. The result of FUN is either wrapper in a list item, when output is NULL, or written to the output file using as.output

key.sep

optional key separator given to chunk.reader

max.line

maximum number of lines given to chunk.reader

max.size

maximum size of a block as given to read.chunk

output.sep

single character giving the field separator in the output.

output.nsep

single character giving the key separator in the output.

output.keys

logical. Whether as.output should interpret row names as keys.

skip

integer giving the number of lines to strip off the input before reading. Useful when the input contains a row a column headers

...

additional parameters to pass to FUN

Value

A list of results when output is NULL; otherwise no output is returned.

Author

Taylor Arnold