It just keeps coming at ya

Re: It just keeps coming at ya

PostBy: brckwlt On: Wed Apr 15, 2009 3:39 pm

Richard S. wrote:FYI you can see what a website is blocking from indexing by bots by looking at the robots.txt file, for example:

One thing to be aware of is you can serve dynamic robots.tx files based on the user agent string/IP etc so what is served to you might not be the same thing served to google. Useful if for example you wanted serve a different file to Google and another to suspected scraper bots.

That is all very confusing to me. Ill need to hire someone to show me how to use it
Stove/Furnace Make: Axeman-Anderson
Stove/Furnace Model: Rebuilt 1953 AA-130

Re: It just keeps coming at ya

PostBy: tvb On: Thu Jun 11, 2009 7:20 am

Well, I guess this report (commissioned by the Bushco admin btw but not released until April of this year) isn't so far-fetched.
Stove/Furnace Make: Alaska
Stove/Furnace Model: Channing III