Re: Strategey Reading VERY large files

I found fscan to be considerably faster for reading text coded numbers than
dlmread, especially if you read line by line.

I hope you are using a 64-bit system, because 32 bit matlab and windoze can
only handle 2 GB of memory max.

Best wishes,

Pete

"MAK" <mak2000@xxxxxxxxxxxxxxxxxxxx> wrote in message
news:ef3177e.1@xxxxxxxxxxxxxxxxxxx
One possible solution is use dlmread in a loop, each iteration of
loop would read set of lines from file, process them and then go to
next iteration...
-----------
Syntax of dlmread
M = dlmread('filename', delimiter, R, C) reads numeric data from the
ASCII-delimited file filename, using the specified delimiter. The
values R and C specify the row and column where the upper left corner
of the data lies in the file. R and C are zero based, so that R=0,
C=0 specifies the first value in the file, which is the upper left
corner

<http://www.geocities.com/mak2000sw/>

Jan wrote:


Hey

I have to read very large text, dat files and perform computations.
These files are like 1 TB of data.

What is the best strategy to read this, as the physical memory of
normal computer platforms are not obviously big enough!

I have been thinking of reading line by line and performing the
computation line by line and then save the data. and go on to the
next line. But this is also a problem as i donnot understand how to
implement this. I mean i need for some calculations several
different
columns from the textfile, and the locations of these columns is
contained in the header row.

any suggestions


.

Relevant Pages

  • Re: Strategey Reading VERY large files
    ... loop would read set of lines from file, process them and then go to ... Syntax of dlmread ... values R and C specify the row and column where the upper left corner ...
    (comp.soft-sys.matlab)
  • RE: Reading Audio CD Text Data
    ... You should specify 2048 bytes when reading in cooked ... Reading Audio CD Text Data ... using IOCTL_CDROM_MEDIA_REMOVAL then I read the TOC from the cd using ...
    (microsoft.public.development.device.drivers)
  • Re: [RFC/RFT][PATCH -mm] swsusp: userland interface
    ... > Looks mostly okay, few nits... ... >> +The device can be open either for reading or for writing. ... please refer to the source code. ... > We should specify that userland suspend/resume utilities should lock ...
    (Linux-Kernel)
  • Re: [RFC/RFT][PATCH -mm] swsusp: userland interface
    ... Looks mostly okay, few nits... ... > +The device can be open either for reading or for writing. ... please refer to the source code. ... We should specify that userland suspend/resume utilities should lock ...
    (Linux-Kernel)
  • Re: FILE_SHARE_READ and FILE_SHARE_WRITE
    ... > I have an application that needs to be able to open any file for reading. ... If I specify FILE_SHARE_WRITE, ... needed to account for my open mode. ... This will permit you to open any file ...
    (microsoft.public.win32.programmer.kernel)