Re: Efficient file downloading


On 22 Feb 2008, at 01:54, Kyle Hunter wrote:

Hello,

I'm using open-uri to download files using a buffer. It seems very
inefficient in terms of resource usage (CPU is ~10-20% in usage).

If possible, I'd like some suggestions for downloading a file which
names the outputted file the same as the URL, and does not actually
write if the file comes out to a 404 (or some other exception hits).

Current code:
BUFFER_SIZE=4096

Try making that a lot lot bigger.


def download(url)
from = open(url)
if (buffer = from.read(BUFFER_SIZE))
puts "Downloading #{url}"
File.open(url.split('/').last, 'wb') do |file|
begin
file.write(buffer)
end while (buffer = from.read(BUFFER_SIZE))
end
end
end
--
Posted via http://www.ruby-forum.com/.



.

Relevant Pages

  • RE: download hangs with no error on DSL under Windows 2000 pro
    ... try downloading another browser to verify that's not ... > a laptop with Windows XP home. ... > of this card explicitely mentions that this is the size of its buffer. ...
    (microsoft.public.win2000.networking)
  • Efficient file downloading
    ... I'm using open-uri to download files using a buffer. ... inefficient in terms of resource usage (CPU is ~10-20% in usage). ... I'd like some suggestions for downloading a file which ...
    (comp.lang.ruby)
  • Re: Efficient file downloading
    ... I'm using open-uri to download files using a buffer. ... I'd like some suggestions for downloading a file which ... $ snoopy http://en.wikipedia.org/wiki/Main_Page ...
    (comp.lang.ruby)
  • Re: Finding it impossible to cancel a web download
    ... the PDA is downloading from, and when it has broken out of its read loop ... > I'm using the HttpWebRequest class to download a file from the Internet. ... So if I cancel reading from the response stream before Ive read ... > internal buffer. ...
    (microsoft.public.dotnet.framework.compactframework)
Loading