What is robots? What is the Difference between robots.txt and robots meta?
What is robots (robots.txt)?
Robots.txt is a protocol between a website and search engine spiders. When a search engine spider visits a site, it first checks whether a robots.txt file exists in the root directory of the site. If it finds the file, the spider will determine the scope of its access according to the content of this file. If the file does not exist, the search spider will crawl the site by following links. It is important to note that the txt file must be placed in the root directory of the site, and the file name must be in lowercase.
Robots.txt is a plain - text file. In this file, website administrators can declare the parts of the website that they do not want search engines to access or specify the content that search engines are only allowed to index. Therefore, the optimization of the robots.txt file will directly affect how search engines index the site.
You only need to write a robots.txt file when you want to prohibit the crawling of certain content. There can be some server - setting issues: when the robots.txt file does not exist, the server may return a 200 status code and some error messages instead of a 404 page. This is because the search spider misinterprets the information in the robots.txt file.
What is robots meta (robots meta tag)?
The robots meta tag is mainly targeted at individual pages. Like other META tags (such as those indicating the language used, page description, keywords, etc.), the robots meta tag is also placed in the <head> section of a page. It is specifically used to tell search engine spiders how to crawl the content of that page.
-
The "follow" instruction means that search spiders can continue to crawl the links on that page.
-
The default value of the robots meta tag is "index" and "follow".
-
The "index" instruction tells search spiders to crawl the page.
-
There is no case - sensitivity in the robots meta tag. "name = 'robots'" means it applies to all search engines. You can also specify a particular search engine, for example, "name = 'baiduspider'". The content part has four directive options: index, noindex, follow, and nofollow, and these directives are separated by commas.
The Difference between robots.txt and robots meta
Currently, all search engines support and comply with the rules of robots.txt. However, fewer search engines support the robots meta tag. So, you can use them according to your specific needs.
【版权与免责声明】如发现内容存在版权问题，烦请提供相关信息发邮件至
，我们将及时沟通进行删除处理。
本站内容除了
谷歌外链,友情链接 （
https://www.abcdlink.com ）特别标记的原创外，其它均为网友转载内容，涉及言论、版权与本站无关。
