.Google's John Mueller responded to a question about why Google indexes web pages that are actually refused coming from creeping through robots.txt as well as why the it is actually risk-free to dismiss the associated Look Console reports about those crawls.Robot Visitor Traffic To Question Specification URLs.The person inquiring the question recorded that bots were producing hyperlinks to non-existent concern criterion Links (? q= xyz) to pages with noindex meta tags that are actually also obstructed in robots.txt. What motivated the inquiry is actually that Google is crawling the web links to those pages, acquiring shut out by robots.txt (without seeing a noindex robotics meta tag) then obtaining reported in Google Explore Console as "Indexed, though shut out by robots.txt.".The person asked the following question:." But listed below's the significant question: why would certainly Google.com mark webpages when they can't even see the web content? What's the perk because?".Google.com's John Mueller affirmed that if they can not crawl the webpage they can not find the noindex meta tag. He likewise helps make an intriguing reference of the website: hunt driver, urging to ignore the outcomes due to the fact that the "common" users will not observe those outcomes.He created:." Yes, you are actually appropriate: if we can not crawl the page, we can not find the noindex. That mentioned, if our team can not crawl the web pages, then there is actually certainly not a great deal for our team to index. Thus while you could view several of those pages with a targeted website:- query, the average user will not observe them, so I would not fuss over it. Noindex is likewise great (without robots.txt disallow), it only indicates the URLs will definitely end up being actually crawled (and also wind up in the Explore Console document for crawled/not catalogued-- neither of these statuses result in concerns to the remainder of the web site). The fundamental part is actually that you don't create them crawlable + indexable.".Takeaways:.1. Mueller's response affirms the restrictions being used the Web site: search accelerated search driver for analysis factors. Some of those explanations is actually because it is actually certainly not connected to the routine search index, it's a distinct thing completely.Google.com's John Mueller talked about the web site search driver in 2021:." The quick solution is actually that a website: concern is actually not implied to become full, neither made use of for diagnostics objectives.An internet site query is actually a details type of hunt that confines the end results to a specific web site. It's generally only words web site, a digestive tract, and afterwards the site's domain.This question limits the end results to a specific internet site. It is actually certainly not suggested to be a complete selection of all the web pages coming from that website.".2. Noindex tag without using a robots.txt is actually alright for these sort of circumstances where a crawler is linking to non-existent web pages that are actually receiving found by Googlebot.3. Links along with the noindex tag are going to create a "crawled/not listed" item in Browse Console and also those won't possess an adverse result on the remainder of the web site.Review the question as well as respond to on LinkedIn:.Why will Google.com index pages when they can not also view the content?Featured Graphic through Shutterstock/Krakenimages. com.