Robot text is important because it regulates the access the search engine bots is granted to specific directories, folders and files. Do not be surprised when robots visit a sensitive part of your web site and the pages will turn up in the search results.

We deny access to the bots to certain files, which should not be presented to the Internet audience at all, by telling the robots that this section is disallowed and they therefore will not index it.

Additionally it's a perfect tool to exclude duplicate content, which is in some cases necessary when dynamic generated content is replaced with static content for the search engines, for indexing purpose.

Duplicate content of a web site can create big issues it can lead to a really poor ranking.