Re: memmapfile


In light of the other tests posted, it would seem that when you call mexCallMATLAB with an input to memmapobj.data it creates a copy of the data. This is not the case for an ordinary array in matlab [Isn't it true that whenever you enter anything at the matlab command prompt that matlab invokes mexCallMATLAB ?]

I have been attempting to access a memory mapped array not by explicitly passing it to my mex function, but by using the API to access variables in the workspace. My biggest concern (that applies to your example) is that if I do in fact pass the memmapobj.data as a prhs it will invoke a copy (Peter, how can you be sure that this does not occur in your example?)

I understand what you are saying about the underlying mechanism of memory mapped files.What I understand of memory mapping is this (and please correct me if I'm wrong): At the program level, it should look like any other memory resource; At the OS level, when the program tries to access a memory location, a check is performed to see whether this data is in memory (the current page) and if not, it loads it into memory from the disk (into a new page).

What I am seeking is a way to get the same program level access to the memory-mapped file in matlab, withiin mex. This should be possible, because matlab itself invokes this resource (via mex or its C-based engine). It is only matlab's implementation of the memory map file class that doesn't play nice with mex functions like mxGet*() (where *=Field, Property,Data,Pr... whatever the "data" really is in "memmapobj.data" I don't know and that's the root of the problem)

The thing is, I actually need to access the memory mapped file in mex. If I remap it in the mex file, this mapping would occur whenever the mex function is called (and this is very inefficient-- the memory mapped file object should be persistent, so it must exist in the matlab environment and not in mex). The only other alternative is to just do everything in C++ and pass the end result to a file-- this would be fine for calculations on the entire data set, but doesn't make sense for calculations on small slices (this is why matlab<->mex is so ideal). So in the middle range where you will call a mex function quite frequently, yet it will only operate on a (small or medium) slice of your data, the only efficient option is to try to get this pointer.

Are there any other ideas out there? What is the official "mathworks" word on this?

once again, thank you everyone.

Dylan Walker


Peter Boettcher <boettcher@xxxxxxxxxx> wrote in message <muy8wp2hygv.fsf@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx>...
"Dylan Walker" <blah@xxxxxxxxx> writes:

Once again, thank you both for your prompt responses. You are both
very helpful. I guess I didn't explain entirely what I am trying to
do. I am not trying to alter the data whatsoever [that is a ball of
wax I'd rather not deal with :)], I just want the ability to read the
data and input it into whatever functionality I implement in a mex
file. It is true that I could simply use C functionality to memory
map the file in parallel (i.e., a C memory mapped file in the mex
function and a matlab memory mapped file in matlab). I would in
principle need both as I need to access the data in an interpretive
setting, while loops in matlab are too slow to effiiciently perform
some operations over the entire data set. But memory mapping the file
twice is wasteful and I thought there might be a way to access the
data that is already mapped in matlab.

I have not been able to get an mxArray * to the data array that would
allow me to obtain a (e.g., double *) to the actual data. I have tried
using mxGetField(memmapobj,"data"), mxGetProperty(memmapobj,"data),
etc. and this just crashes. Perhaps I am not clear on what you
suggested before about using mexCallMatlab.

You mention that I can use the GET or SUBSTRUCT method with
mexCallMatlab to give me mxArray * of the class specified in format.
This seems like exactly what I am trying to do; so I would be very
much obliged if you could describe this method in a little more
detail. To access some portion of the data within matlab, I can call
memmapobj.data(i1:i2) , however "data" is not a proper array in the
usual sense and the traditional mxGet... functions seem to fail. I
think you are saying that the mexCallMatlab can be used as a way
around this, because the GET method in matlab is already "overloaded"
to deal with the memorymappedobj. But can I return the general
pointer to the data from mexCallMatlab (rather than just some slice of
the data that I call? Sorry if I am being unclear.

Memory mapping is an OS-level function. Once a file is memory-mapped, a
pointer to that data exists, and the user application (MATLAB) sees the
entire mapped region as valid memory that can be accessed normally.
Once the application tries to access a page of that memory region that
has not yet been read from disk, the OS transparently takes care of
faulting in that page. As far as I know, no object/subscripting
trickery is needed to access data. It's just a regular MATLAB variable
that has a Pr pointer that points to a memory-mapped region of virtual
memory.

So treat the Data element of the memmapfile object like any other MATLAB
array. That fact that the Pr pointer points to a memory mapped file
instead of a malloc-allocated region on the heap is completely
irrelevant. Just pass the Data element directly to the MEX file and be
happy.

Test code:

fid = fopen('testfile.dat', 'wb');
fwrite(fid, [17 18 19], 'double');
fclose(fid);

m = memmapfile('testfile.dat', 'Format', 'double');
s = memmaptest(m.Data);

clear m


----------------------------------------------------------------------------
memmaptest.c
----------------------------------------------------------------------------
#include "mex.h"

void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[] )
{
double *x;

plhs[0] = mxCreateDoubleMatrix(1,1,mxREAL);
x = mxGetPr(plhs[0]);
*x = *mxGetPr(prhs[0]);
}
----------------------------------------------------------------------------



-Peter
.

Loading