There are times when websites have different domain names but point to the same code base. The most common is a CMS system that hosts multiple sites.
As the robots.txt file sits in the root of a website it will apply to all domains running from that codebase.
A easy way around this issue is to use the free ISAPI rewrite DLL from Helicontech for IIS.
First create 2 files in the root of your site
robots.block.txt – to block acess
robots.allow.txt – to allow access
Once you install the DLL and go into the editor just add the following
Depending on the domain name of the visitor, this will either read in the content of the block file or the allow one into a robots.txt file.
Note: Your files need to have a .txt extension and the robots.txt file should not actually exist otherwise.