Re: database questions
- From: TideMan <mulgor@xxxxxxxxx>
- Date: Tue, 23 Jun 2009 22:29:21 -0700 (PDT)
On Jun 24, 4:39 pm, "Tom" <tdd214donts...@xxxxxxxxx> wrote:
Couple questions, one closely related to Matlab, and another that is not so much, but that I'm hoping someone might have some insight into anyway.
First, I have a program that does an optimization over and over and over again with different conditions. I store the results (the objective function and independent variable values, as well as the initial guess variables) from each trial, and that is a lot of data. I started putting my data in a database instead of an .mat file, since the .mat files were getting large enough to cause memory errors (even though the size of the .mat files was significantly smaller than the amount of RAM I have). I'm rewriting my program to use the database toolbox to read and write data from a database. I suspect the best strategy is to not read and write from the database every trial, but to load a bunch of records (say 1000) into a local variable, run those 1000 optimizations, then update the database with those records, and repeat with new records. I know very little about databases, so just
wanted to do a sanity check. The program would be somewhat easier to write loading one record at a time, but am I correct that that would be significantly slower than my described alternative? The database is local (on my workstation).
Second, I'm currently using an Access database (as it is the only database program that I have and know how to use well). After some amount of using that Access database, the .mdb file gets huge. I can compact and repair the database, but then sometimes it seems to just get slow or corrupted, and can't be repaired. Someone mentioned this might be due to the fact that Access has horrible databases. The said using something like mySQL would be much more stable. Any insight? I really don't think my databases are that huge... They are several million rows, and a bit larger than 1 GB at present, probably to grow to 10 GB or so. I mean, that's large, but I'm sure there are a lot of databases that are used by people who know what they are doing that are much larger and do not have these stability issues. What do those people use?
Thx,
Tom
I use netCDF for this sort of thing (Google it).
There's a bit of a learning curve, but it's very easy to retrieve data
and to update the database once you've learnt how.
And in contrast to a .mat file, you can load just the data you want to
load, not the whole database.
It was designed for huge databases - global weather forecasting models
that update several times a day.
.
- References:
- database questions
- From: Tom
- database questions
- Prev by Date: Re: Avoiding a for loop
- Next by Date: Re: small matrix question
- Previous by thread: database questions
- Next by thread: Visual basic example "bouncing ball " not works
- Index(es):
Relevant Pages
|
