It just keeps coming at ya

Re: It just keeps coming at ya

PostBy: brckwlt On: Wed Apr 15, 2009 3:39 pm

Richard S. wrote:FYI you can see what a website is blocking from indexing by bots by looking at the robots.txt file, for example:

http://nepacrossroads.com/robots.txt

One thing to be aware of is you can serve dynamic robots.tx files based on the user agent string/IP etc so what is served to you might not be the same thing served to google. Useful if for example you wanted serve a different file to Google and another to suspected scraper bots.



That is all very confusing to me. Ill need to hire someone to show me how to use it
brckwlt
 
Stove/Furnace Make: Axeman-Anderson
Stove/Furnace Model: Rebuilt 1953 AA-130

Re: It just keeps coming at ya

PostBy: tvb On: Thu Jun 11, 2009 7:20 am

Well, I guess this report (commissioned by the Bushco admin btw but not released until April of this year) isn't so far-fetched.
tvb
 
Stove/Furnace Make: Alaska
Stove/Furnace Model: Channing III