Re: Best way to download >1GB files

Is there some reason to not use wget or curl? Those are both written
already. What are you hoping to do with the files you download?

-Bryan

On Dec 31, 2007, at 2:04 PM, thefed wrote:

What is the best way to download files from the internet (HTTP)
that are greater than 1GB?

Here's the story in whole....
I was trying to use Ruby Net::HTTP to manage a download from
wikipedia... Specifically all current versions of the english
one... But anyways, as I was downloading it, I got a memory error
as I ran out of RAM.

My current code:
open(@opts[:out], "w") do |f|
http = Net::HTTP.new(@url.host, @url.port)
c = http.start do |http|
a = Net::HTTP::Get.new(@url.page)
http.request(a)
end
f.write(c.body)
end

I was hoping there'd be some method that I can attach a block to,
so that for each byte it will call the block.

Is there some way to write the bytes to the file as they come in,
not at the end?

Thanks,
---------------------------------------------------------------|
~Ari
"I don't suffer from insanity. I enjoy every minute of it"
--1337est man alive






.

Relevant Pages

  • pssstt
    ... Post keys again and you won't be around here ... The number 1 reason for having moderators for forums is to be ... potentual pirate! ... The tracking cookie must be enabled before you can download or post ...
    (rec.video.satellite.dbs)
  • Pssstt
    ... Post keys again and you won't be around here ... The number 1 reason for having moderators for forums is to be ... potentual pirate! ... The tracking cookie must be enabled before you can download or post ...
    (rec.video.satellite.tvro)
  • Re: [SLE] Need advice
    ... download it using curl or wget. ... In Mozilla and Firefox, both these ...
    (SuSE)
  • Re: CPAN install issues
    ... If I manually download the source files using ... wget and decompress, the process work. ... Does curl have a dot-file or environment variable that will tell it to auto- ...
    (comp.lang.perl.modules)
  • Re: vss2005 web project
    ... I see a lot of reasons for them not to be in the VSS database binary or not. ... they are just there for customers to download perhaps they are .msi ... these are things that I create and can think of no sane reason why I ... There really needs to be a method to exclude files from the source safe ...
    (microsoft.public.vstudio.sourcesafe)