I would use curl. If there is a login/password, you can then use curl to POST to a URL, create and save a cookie, then grab the page you want to look at. If I recall correctly, the man page is fairly well documented. <br><br>
Good luck!<br><br><div><span class="gmail_quote">On 11/10/07, <b class="gmail_sendername"><a href="mailto:bb.odenthal@gmail.com">bb.odenthal@gmail.com</a></b> <<a href="mailto:bb.odenthal@gmail.com">bb.odenthal@gmail.com
</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">John,<br><br>I may be over simplifying this but a web "search" is usually just a POST or GET method action on an HTML form. If you can take a packet trace of the transaction (assuming it's not SSL) then it's easy to discover the URL format and method for the search. A simple "lynx -dump" of that URL using "watch" every 120 seconds could be helpful (Assuming that a text only version of the web page would be of any use to you):
<br><br>#watch -n 120 "lynx -dump <a href="http://foo.com/search?bar=san_gabriel_valley">http://foo.com/search?bar=san_gabriel_valley</a>"<br><br>If the site requires more interaction than that (login, password, click on a few links, fill out a form) or requires cookies then I suggest using a Perl script. Maybe WWW::Mechanize for some simple HTML form automation.
<br><br>**I'm putting on my Nomex jacket**<br><br>Or...just spend $30 on <a href="http://www.newdigitalsoft.com/airobot/">http://www.newdigitalsoft.com/airobot/</a> or similar and use a windows box? It IS an option.
<br><br>-bb<br>-----Original Message-----<br>From: <a href="mailto:juanslayton@dslextreme.com">juanslayton@dslextreme.com</a><br><br>Date: Sat, 10 Nov 2007 21:37:04<br><a href="mailto:To:sgvlug@sgvlug.net">To:sgvlug@sgvlug.net
</a><br>Subject: [SGVLUG] Polling Web Sites<br><br><br><br> Got a little project here that I could use some help on. El Monte<br>City School District uses a program called Aesop to post daily<br>openings for substitute teachers. All I have to do is go to their
<br>web site and click on the search button and I can see who has<br>currently called in to be absent. Trouble is, if someone calls in<br>sick just after I've checked, I won't find out about it until the<br>next time I check. And I have better things to do than sit and click
<br>on the search button all evening.<br> So I began to figure out ways to poll that site automatically. The<br>current approach works like this: A timing program (written in C)<br>runs in the background on a virtual terminal and produces a negative
<br>pulse on data line 1 of the parallel port every few minutes. I 'hot<br>wired' the left click switch (high, pull-down side) of a USB mouse to<br>that data line (through a diode to protect the port in case someone
<br>physically clicks the mouse). By leaving the cursor on the search<br>button, the background program electronically clicks that button<br>every few minutes. All I have to do as I go about my business is<br>glance at the screen every now and then to see if anything new has
<br>come up.<br> But this is over-complicated. There ought to be a simple way to poll<br>that page programatically without messing with the hardware. Say, by<br>using the usb event mechanisms? Like as not somebody somewhere has
<br>already written code to do it. I'd appreciate anyone who could point<br>me in the right direction.<br><br>John<br><br>***************************************************************************************<br>If the mind is not constrained by walls and fences, where is the need for
<br>Windows and Gates?<br></blockquote></div><br><br clear="all"><br>