This can be useful on development sites where you have many IIS sites running but don’t want search engines to index them without creating robots.txt on every site.

To do this we’ll have to add a X-Robots-Tag HTTP Response header which is some search engines including Google as detailed here: http://googleblog.blogspot.com/2007/07/robots-exclusion-protocol-now-with-even.html

Here are the steps:

1. Open Internet Information Services (IIS) Manager
2. Click the server instance name from the left menu
3. Open HTTP Response Headers
4. Click Add… from the action menu
5. In the Name input, type: X-Robots-Tag and in the Value input, type: noindex