In this video we will take a look at robots.txt Tester in Google Webmaster Tools.
What is robots.txt Tester?
Robots.txt is a text file that you can use to communicate with web robots how they should behave on your site.
- You can use it to grant access to certain robots and not others.
- You can grant certain sections of your site.
- You can tell where your Sitemaps are located so these robots can find them easily.
Google Webmaster Tools’ robots.txt tester can help us see if the text file is created correctly.
If you changed your robots.txt recently, you can also notify Google using this tool to ensure Google is aware of the latest version.