Re: Perl hashing speedup ?

Hi,

Randy wrote:
Is it necessary to load all the data in memory at once? What operations
do you need to perform on the data?

Yes, For me it is necessary to load all data. I need to access this
data in indexed fashion. That is the reason why I am using hash.

I also tried storing data in DBM file using MLDBM package but it is
taking too much time, so I tried constructing hash in memory to see how
much time it takes. The stats mentioned in my first mail are for in
memory construction of this hash.

Thanks,
Sujit



Randy W. Sims wrote:
sujit wrote:
Hi,

This is regarding perl hashing. I am creating a 3 level hash (actually
2 such hashes at a time) with approx. 2.6 million entries.

Regarding length of each key;

Level 1 key length - Approx 40 chars
Level 2 Key length - Approx 12 chars
Level 3 key length - Approx 40 chars

Now this hash creation process itself is taking around 3.5 to 4 hrs for
me.

Is it necessary to load all the data in memory at once? What operations
do you need to perform on the data?

Randy.
.

Relevant Pages

  • RE: Hash table & threading
    ... Hash Tables use more memory, but in almost every case a Hash Table will have ... an array is when the keys are all consecutive integers. ... Another way to lower memory usage is to set the load factor of your hash ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Calculating CRC32 for uploaded files
    ... so you don't have to load the entire file ... > into memory to calculating the hash. ...
    (comp.lang.php)
  • Re: How big can an array be ?
    ... > Using linux I was able to load a hash, where the process was taking up ... > imagine that on some OS's you might not be able to occupy more that ... I was told that I ran out of memory much earlier. ...
    (comp.lang.perl.misc)
  • Re: Calculating CRC32 for uploaded files
    ... > supported by must systems (and free source code is available) ... so you don't have to load the entire file into ... memory to calculating the hash. ...
    (comp.lang.php)
  • Re: hash table size
    ... talking about chess programming ideas and what they do rather than actually ... you can clear out the memory. ... Just a couple cycles per hash check. ... The amount of physical memory each user gets is likely to be somewhat ...
    (rec.games.chess.computer)
Loading