Need Better Explained about the Robots.txt

Updated on May 16, 2019 in Google
2 on May 7, 2019

Anyone give better explain for about robots.txt . and why the mobile responsive test says “blocked by robots.txt”.

  • Liked by
  • Chartheepan
1 on May 10, 2019

Hello ,

Actually, the robots.txt file is an important and standard protocol used by websites to communicate with the web crawlers and other robots.which means to command Crawl bots which pages need to crawl/allow and which page don’t need to allow, that means, which need to disallow. That will instruct to the search engine to leave out some pages (private, security related, etc.) while crawling. 

This is actually the meaning of robots.txt. If you need any further clarifications, please let me know!

on May 16, 2019

 Thank you for replay to me. Its really helpful for me.  

Show more replies
  • Liked by
Loading more replies