Google Hacks - Fascinating:Robots.txt

RSS Reading at work as usual, and I came across this. It really caught my attention and stuck out as something I think I’ll be doing regularly now.

Go to Google and search "robots.txt" at any domain that you want (sites:domain). Robots.txt is a file on a web server that defines what you’re not allowed to see. Too bad you still have access to the file to read it. It’s very interesting to try this one.. robots.txt site:whitehouse.gov.

Read this at Digg: Google Screenshots. At the bottom of the article is the stuff about robots.txt, but all of it is interesting. Best thing is, the site linked to Ultimate Online Resource for Google Hacking! The site is currently down due to very high traffic volume, try back later.