Re: heavy loop functions slow
- From: Jano Svitok <jan.svitok@xxxxxxxxx>
- Date: Tue, 8 Apr 2008 02:43:36 -0500
On Tue, Apr 8, 2008 at 7:17 AM, Michael Linfield <globyy3000@xxxxxxxxxxx> wrote:
Alright so I was playing with my large amounts of data and ran into yet
another problem with shoving it into a loop that requires a substantial
amount of memory.
dataArray = []
output = arrayOut.to_s.chop!.split(",")
set arrayOut to nil if you don't need it any more.
output1 = output[0..356130]
output2 = output[356131..712260]
output3 = output[712261..1068390]
output4 = output[1068391..1424521]
You dont need output here, set it to nil to allow for garbage collection
count = 0
output1.each do |out|
out = out.to_i
push = hashRange[out]
dataArray << push
count+=1
puts "#{push} - #{count}" #Testing purposes
end
1. you can convert the output to numbers in one pass, though use
benchmark to see the actual gain:
output = arrayOut.to_s.chop!.split(",").map {|out| out.to_i }
2. if you are looking for numbers only, you can do something like
output = []
arrayOut.to_s.chop!.scan(/\d+/) {|out| output << out.to_i }
(you can count the items, and switch to output2 when output1 has
enough, thus 1. creating smaller arrays, 2. doing two things in one
step.)
3. even in this case, you still have both the original arrayOut and
the long string (.to_s) in memory.
It might be faster, if you could iterate through the array without
creating the intermediate string. The question is 1. will it help? 2.
Is it worth it?
.
- References:
- heavy loop functions slow
- From: Michael Linfield
- heavy loop functions slow
- Prev by Date: Re: Quicken copy time when processing large files
- Next by Date: Re: Is possible to define various methods at same time?
- Previous by thread: Re: heavy loop functions slow
- Next by thread: Re: heavy loop functions slow
- Index(es):
Relevant Pages
|
