| http://www.w3.org/ns/prov#value | - Robot.txt is a file which you can generate by your self and upload it to your root folder to tell the bots what should they crawl and what not? if you want to disallow something to crawl them simply put disallow and rite the url name or directory name to be restricted by the robots.
|
