Re: Out of memory when reading a large file
- From: "Rune Allnor" <allnor@xxxxxxxxxxxx>
- Date: 11 Jul 2006 19:56:09 -0700
yusuf wrote:
Hi,
I'm a newbie to matlab. I have to do the following task:
- read a very large file ~300MB which contains 32 bit binary integers.
I would ideally like to read the data in an array and then be able to
work with the array values. What is happening right now is that I try
to read the file (using fopen/fread), but it gives me out of memory
errors.
Since I need to run functions on the whole data set, is there a way to
read the file into an array, or two arrays?
In order to do what you want, you need enough RAM on your computer.
To get all the data into memory at once, requires 512 MB RAM.
To do work while all the data reside in memory, requires 512 MB more.
So unless your computer has at least 1 GB of RAM, you will have to
chop up the data into smaller chunks. Even if you have enough RAM,
you may want to chop the data up anyway, to ease data handling
and reduce cache misses.
Here's one way to do it:
fid = fopen(filename);
[D,M] = fread(fid,[1000,500],'int32');
I am assuming now that the data is a N x 500 matrix, where you
choose to read 1000 rows (2MB) at a time. You can either check
the number of read elements, M, or the End of File flag, feof, to find
out when you have read all data in the file.
Rune
.
- References:
- Out of memory when reading a large file
- From: yusuf
- Out of memory when reading a large file
- Prev by Date: Re: Simulink Square root
- Next by Date: Access Current Simulation Time in Level-2 M-File
- Previous by thread: Re: Out of memory when reading a large file
- Next by thread: accessing array without using FOR loop
- Index(es):
Relevant Pages
|
