Uncrawled URLs in search results

kery | Oct 09, 2010 | Comments 7

Advertisement


Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google’s search results.

Related items

Filed Under: seo world news and articles

Tags: • • •

About the Author:

RSSComments (7)

Leave a Reply | Trackback URL

  1. mkarakas0690 says:
    October 9, 2010 at 4:02 am

    that tip is so good for me.. thanx matt!.. :)

  2. MichaelDadona says:
    October 9, 2010 at 4:49 am

    I think better maintain that way, Matt. As many bloggers can ‘ferry’ that anchor texts like what you said. Especially, your example for NISSAN. Many small entrepreneurs (vendors) related to the industry able to get benefits from it. Like mine, Nissan Impul.

  3. kevinargh says:
    October 9, 2010 at 5:30 am

    Just make sure you remove the rule from robots.txt first or Google will never see the noindex meta tag on the page.

  4. allison30dc says:
    October 9, 2010 at 6:23 am

    Cool, that explains a lot, including what happened with those Google local listings that appeared to have been crawled that were robots.txt’d

  5. danielgayle says:
    October 9, 2010 at 7:17 am

    This was actually pretty interesting. I didn’t know that the meta tag “noindex” would actually totally dump it from the index. Very cool.

  6. gbmodern says:
    October 9, 2010 at 7:28 am

    Wow, I never knew that! I though robots.txt actually blocked Google from listing the site.

  7. BKPrecisionVideos says:
    October 9, 2010 at 7:44 am

    Very useful information. Thanks Matt.

Leave a Reply