| http://www.w3.org/ns/prov#value | - This file informs the web crawler which directories should and should not be indexed, but does not enact any actual access restrictions. + +Technically, a web crawler is a specific type of bot, or software agent. + +See [[bot]] and [[intelligent agent]]s.
|
