Re: stock symbol listing

billb <sevenoutpinball@xxxxxxxxx> wrote in message
news:a4b5e$4351b21d$42f83980$2654@xxxxxxxxxxxxxxxx
> "Bill Reid" <hormelfree@xxxxxxxxxxxxxxxx> wrote in message
> news:qdR3f.140839$qY1.71983@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > liberated <liberated@xxxxxxxxxx> wrote in message
> > news:hXB3f.23113$0c.5179@xxxxxxxxxxxxxxxxxxxxxxx
> > > Does anybody know where I can get a file that I can import that
> contains,
> > oh
> > > let's say, ALL STOCK SYMBOLS that last price was <$10? or <$5?
> > >
> > > Tons of sights have searches, listings, but cannot export any of the
> > lists.
> > > Doesn't matter if it has anything else in it, I'll clean it up.
> > >
> > Ever think about copying and pasting the text from these lists? and
then
> parsing out the relevant stuff to a file?
> >
> > Or,
> > a little more high-tech, writing a script to retreive the lists from the
> > URL
>
> how does one do that? any samples?
>
Gee, as recently as August of this year some dude posted such a
script right here:

Newsgroups: misc.invest.stocks
From: Paul <none@none>
Date: Wed, 03 Aug 2005 10:01:44 +1000
Local: Tues, Aug 2 2005 5:01 pm
Subject: Re: Downloading EOD data 4 free

shane-dated-1125558744.d1c...@xxxxx wrote:
>>>anyone know where I can download EOD data for all stocks traded on the
major
>>>U.S. exchanges (NYSE, AMEX, NASDAQ) for FREE?

>>>I don't mind paying for a program to do that. I'm just hoping not to
>>>continue paying a monthly fee to get EOD data. I use metastock but can
>>>probably convert the data to metastock format if needed.

....

>>finance.yahoo.com

....

> I don't see how one could download the data for the entire market from
yahoo
> finance. They do seem to provide a csv download feature but you need to
> provide a list of symbols. That might make for a really long url. Do
they
> provide an actual eod data download I'm not seeing?

....

I download 6313 symbols evry day - it comes down in CSV files 100 at a
time. I then update the info from the history for stocks that have "N/A"
in the EOD data. The whole process - including appending to the
individual files for each stock - takes 3 or 4 minutes.

You need a bit of programming ( perl ) to get the files down and handle
stripping the data but I have been using it for 2 years now and have
sorted out most of the bugs - the one that gives me the most problem is
the changing tickers !!

As far as long URLs - comouters don't care - give them the list and they
sort it out for you.

For info the perl script is shown below - there's lots of other stuff to
go with it :-

#!/usr/bin/perl

# path to the lynx browser
$lynxCmd = "/usr/bin/lynx -source";

# path to the Stock Quotes Grabber
$progDir = "/home/mas/getUSData";

# the file where the results are stored
$data = "/home/getUSData/results";

# first part of the Yahoo URL
$quotesUrl = "http://finance.yahoo.com/d/quotes.csv?";;

# path to the ticker file
$symbolFile = "$progDir/tickerTemp.dat";

# add the list of symbols to the URL
$symbols = "";
open(FILE, "< $symbolFile") || die "Error opening the symbols file!";
while(<FILE>){

chop($_);
@fields = split(/[ ]+/, $_);
foreach $field (@fields){
$symbols .= $field;
$symbols .= "+";
}

}

chop($symbols);

# append the arguments to the Yahoo URL
$quotesUrl .= "s=$symbols&f=sl1d1t1c1ohgv&e=.csv";
# use lynx to retrieve the data and append it to the results file.
system("$lynxCmd \"$quotesUrl\" >> $data 2>/dev/null");

close(FILE);
exit 0;

Note that $symbols gets expanded to a list of 100 tickers. The original
script ( which is now unrecognisable ) was posted by an unknown person
several years ago ( he however, has my thanks ).

---end of archived post

Also note that there's a free program (PERL script) called "QuoteMonster"
that downloads quotes from Yahoo!(TM) and other not-quite-as-free
products available to "scrape" real-time data and such-like...


Note that the above poster is actually downloading CSV files from
Yahoo!(TM), not HTML, and using the darling little text-only
browser "Lynx" to do the "heavy" lifting of establishing a connection
to a particular URL.

Another way of doing this is to partially re-invent the Lynx
wheel by programmatically connecting to a Windows "socket" as
a client of the Internet port (I am presuming you are on a Winbloze
crap box, but the basic concept applies to UNIX et. al.), send out
your HTTP URL requests that way, and grab and parse the
socket input buffer text. You should have everything you need
to do this in the Windows API; I have it in my crappy $100
C++ development platform.

As far as parsing the HTML is concerned, the original poster
could be looking (programmatically) for a text string called
"Next 20" or something like that associated with a tag with the
URL for the next 20 stocks between $5 and $10. If you
actually need the URL there it is; if not, it's at least possibly a
good point to stop stripping out the HTML tags for your text-only
output list. Put the whole thing in a loop that continues to send out
a request for the "Next 20" URL until you reach EOB without finding
the text.

There are some extra-curricular hassles associated with this type of
activity, such as the web site possibly detecting that you are not an actual
human downloading all this data and cutting you off (their data
providers frown on just giving out massive amounts for free), and
periodically they will "update" their web page/site formats and
screw the whole thing up.

For example, here's a little snippet from my Yahoo!(TM) data
scraping code that does a two-step to accomodate them changing
their date format (the code is for the point that I've retrieved the
date for stock quote from the HTML in the some type of
format like "Aug 22 1980" or "22 Aug 1980" and I convert it to my
CSV "database" format of "19800822"):

/* get the date string for every line and convert */
/* 12/1/02 - Yahoo bastards apparently changed the date format from
DD-MMM-YY to MMM-DD-YY, so new code checks for alpha at
position 0 to switch between old date handling code and new */
if(isalpha(in_strings[istr_num][0]))
sscanf(in_strings[istr_num],"%[A-Za-z]-%d",&date_string,&day);
else sscanf(in_strings[istr_num],"%d-%[A-Za-z]",&day,&date_string);
month=change_month_abrevs(date_string);
sprintf(date_string,"%d%02d%02d",year,month,day);

---
William Ernest Reid



.