Re: Looking for web crawler written in Ruby


Kev wrote:
> Hi, I am looking for web crawler(spider) written in Ruby. I googled but
> couldn't find much information. Can anyone give me some links?
>

http://www.jamesbritt.com/articles/RubyAndVbaForWebDictionarySearches.html
http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf
http://www.acc.umu.se/~r2d2/programming/ruby/webfetcher/

http://snippets.textdrive.com/posts/show/74
(look at cached page)

http://rubyforge.org/projects/wee/
look at Mechanize

.

Relevant Pages

  • Re: Waiter, theres a noob in my soup!
    ... pipe their output to a ruby program so that I could do most of the work ... in ruby instead of perl. ... Mechanize handles great, and the Cingular has a weird multi-step login ...
    (comp.lang.ruby)
  • Re: Directory and file listing
    ... increment in ruby? ... again any coments or help would be apreciated ... Prev by Date: ...
    (comp.lang.ruby)
  • Re: super-newbee Ruby regex help?
    ... I had to install the 'mechanize' gem, ... (don't forget to escape /in the RE, else ruby will think that it is ... proxies = # array where the proxies will be ... Have a good time with Ruby! ...
    (comp.lang.ruby)
  • Re: scrubyt anyone?
    ... I'm unable to make scrubyt's 6 tests pass on any version of Ruby. ... to scrubyt on the Google Groups list, ... $ gem list hpricot ... $ gem list mechanize ...
    (comp.lang.ruby)
  • Join an ArrayList
    ... compiler complained about converting from an object to a string array. ... Is there another, i.e. shorter, way to do this? ... Kev ... Prev by Date: ...
    (microsoft.public.dotnet.languages.csharp)