Robots.txt is a file that contains instructions on how to crawl a website. This is also called as robots exclusion protocol, and this standard is used by sites to tell the bots which part of their website needs indexing. Also, we can specify which areas you don’t want to get processed by these crawlers, such areas restricted content. Bots like malware detectors, email harvesters don’t follow this standard and will scan for weaknesses in your securities, and there is a considerable probability that they will begin examining your site from the areas you don’t want to be indexed.