Tagged and Bagged: website
If you have a website and you want to control the amount of unwanted bandwidth usage or simply don’t want some of your content indexed by the search engines or image scrapers, then Khalid’s article on the invesp blog is a valuable resource. It presents the necessary elements of a robots.txt file in the clearest manner that I’ve seen yet. Its a good read for any webmaster.
Good find.
Actually I am looking for an easy to understand .htaccess tutorial, particularly with relation to mod_rewrite. I’ve seen plenty, but that’s far more complex than robots.txt.
hari, I’ve not really had to check out .htaccess too much, as the scripts that I’ve needed either have been automatic through cPanel (like hotlink protection and redirecting the WRONG http://www. to @), or I’ve gotten the script from a site as a drop in. Other wise, I could be your guru
There might be something in the Apache docs about it (but it might not be too readable). You’ve got me interested in the subject, now.
Using mod_rewrite, you can achieve some nifty effects, including creating subdomains out of subdirectories on your server. I’m trying to look that up.
This is a great post. It never occurred to me to block feeds like the RSS. Consider them blocked now. :p
Hari, that wouldn’t be a simple tutorial, there’s a lot that can be achieved with .htaccess so a tutorial on each action that can be done with .htaccess would be a little better.