Re: loading data from ascii file

Felix <feboll@xxxxxx> writes:

> Hi!
> I have to read data from an ascii file which is several GB large.
> lines represent individual datapoints. I would like to read only some
> lines, the linenumbers are stored in a matlab vector. is there any
> possibilty to this without loading the whole array (which is too)
> big.

Random access in an ascii file is very difficult, if not impossible,
as the program can never know how many linefeeds exist between any two
byte offsets, without reading them all.

I would suggest converting the whole mess to a binary file, ONCE.
Seeking around a binary file is straightforward, and faster too.

I would use a block-processing approach to converting the file. Read
one line, or a bunch of lines (100? 1000?) at once, using fscanf or
fgetl. Convert, then fwrite the output to your output file. That
way, only a small amount of the file is in memory at any time.

Then, for random access, fseek to the byte offset you need. (row
number x row size x element size).



--
Peter Boettcher <boettcher@xxxxxxxxxx>
MIT Lincoln Laboratory
MATLAB FAQ: http://www.mit.edu/~pwb/cssm/
.

Relevant Pages

  • Coversion of Binary file to ASCII file using C.
    ... I am able to open a binary file for reading but can someone tell me as ... how to go about converting a Binary file to ASCII file using C. ...
    (comp.lang.c)
  • Re: Old DOS flatfile wont import to Access 2002
    ... I can open the ASCII file using WORDPAD, ... and the DOS EDIT window and it looks fine. ... Even if the flatfile were storing records in some proprietary format, ... again converting it to ASCII/TXT format. ...
    (comp.databases.ms-access)
  • Re: Words default formatting for .TXT files
    ... a synonym for ASCII file, a file in which characters are represented by ... Contrast with a binary file, ... is no one-to-one mapping between bytes and characters. ... binary files to preserve the formatting. ...
    (microsoft.public.word.conversions)
  • Re: binary file
    ... I have used the profile module to measure some thing as the next ... Others how can I read the binary file to am ascii file? ... Kent ...
    (comp.lang.python)
  • Re: .pst upgrade from Outlook 97 to Outlook 2007
    ... "converting" an ascii file to Unicode. ... Outlook 2003 vis-a-vis Outlook 97, I seem to have gotten the ...
    (microsoft.public.outlook)
Loading