Do we really need to create robot.txt for our Google Blogspot ?
What is Robot.txt anyway ? according google Robot.txt is a text that allow us to specify how would you like to be crawled, and within this robot.txt we can include some specific to direct search engine crawler to let them know which part they can or can not access. Why we need to direct crawler when they crawling our website ? well for any reason, some time we don't want they access a certain area in our website, that's why we need to create a certain robot.txt code using ASCII text code.
But right know I don't want to discus about how to create robot.txt code, because what bother me right know is a necessity of this robot.txt for our web site. See I have been search on the internet recently just try to find the most suitable and SEO friendly robot header tag setting fo my Blogspot, but what I have found is not as I expected, most the site suggesting the same exact setting, as if is copy paste or something, and what most disturbing me is that most is them don't include the reason why they chose the setting, for newbie like me is quite confusing to understand it, and anyway I can't just follow without knowing the reason.
Finally I have find an answer from Google webmaster it self :
When Google boot visit a Website we first ask for permission to crawl by attempting to retrieve the robot.txt file, a website without a robot.txt file, robot meta tag or X-robot tag HTTP header will generally be crawled and indexed normally.
Based on information above, I think there is no need for us (especially if we are still a newbie) to mess with robot.txt or robot header tag, because it has no great impact for our SEO campaign.
high traffic to our blog is not decided by robot code, so my suggestion is just leave it that why as the default setting are, because if you don't know what you are doing, it can result your blog can be ignored by Google search engine.
Once again I repeat, we don't need to create robot.txt text for our blog, unless if know exactly what we are doing, but if you happen already set you robot.txt by your self and anything went wrong, I think you don't need to panic because Google today have generally very flexible web crawler and will not easily be swayed, just go to your Google webmaster to fix this problem or simply getting back your robot.txt setting to default.
No comments:
Post a Comment