+ Reply to Thread
Results 1 to 7 of 7

Thread: Googlebot and Slurp seem to be permanently on this site

  1. #1
    Administrator
    Join Date
    Jan 2010
    Location
    Essex, UK
    Posts
    6,599
    Blog Entries
    28
    Thanks
    2,651
    Thanked 1,694 Times in 992 Posts
    Rep Power
    86

    Googlebot and Slurp seem to be permanently on this site

    These two bots seem to have taken up permanent residence if you look at the who's online page.

    As someone who doesn't normally buy/run forums I don't know if it's normal. Is it? Or have I screwed up with the URIs and the bots are being served an infinite number of duplicate content pages?

  2. #2
    Member
    Join Date
    Jan 2010
    Posts
    63
    Thanks
    17
    Thanked 13 Times in 8 Posts
    Rep Power
    2
    This is pretty normal dont fret, once these leave MSN bot and a few others will take residence

  3. #3
    Administrator
    Join Date
    Jan 2010
    Location
    Essex, UK
    Posts
    6,599
    Blog Entries
    28
    Thanks
    2,651
    Thanked 1,694 Times in 992 Posts
    Rep Power
    86
    Thanks, mpcovcd.

  4. #4
    Senior Member
    Join Date
    Jan 2010
    Posts
    470
    Thanks
    86
    Thanked 122 Times in 73 Posts
    Rep Power
    9
    Might be a good idea to block certain forums from bots, like where members discuss sites currently for sale.

  5. #5
    Administrator
    Join Date
    Jan 2010
    Location
    Essex, UK
    Posts
    6,599
    Blog Entries
    28
    Thanks
    2,651
    Thanked 1,694 Times in 992 Posts
    Rep Power
    86
    You mean the Buy / Sell / Trade / Employ ? I'm hoping that eventually members will start using that to post their own offerings whether they're looking to sell a site or employ a PHP programmer. Wouldn't they want it to be crawled?

  6. #6
    Senior Member
    Join Date
    Jan 2010
    Posts
    470
    Thanks
    86
    Thanked 122 Times in 73 Posts
    Rep Power
    9
    yeah that would be the one I guess. I agree if people list sites for sale they definitely would want them to be indexed. Maybe you can do another closed forum then where we could discuss currently available for sale sites. E.g. if someone wants some help with dd on a purchase, or other similar things where people would share details they usually don't want others to see freely.. Just an idea, up to you of course

  7. #7
    Administrator
    Join Date
    Jan 2010
    Location
    Essex, UK
    Posts
    6,599
    Blog Entries
    28
    Thanks
    2,651
    Thanked 1,694 Times in 992 Posts
    Rep Power
    86
    I'm here only to click the right buttons and press the right keys. You are going to decide how this forum develops. Over to the others - your thoughts?

+ Reply to Thread

Similar Threads

  1. Yahoo Site Explorer To Close ... permanently
    By Clinton in forum General & Miscellaneous
    Replies: 13
    Last Post: November 23rd, 2011, 06:12 PM
  2. Replies: 24
    Last Post: June 15th, 2011, 10:54 AM
  3. Help me with a Googlebot problem, guys
    By Clinton in forum Website 101
    Replies: 3
    Last Post: January 15th, 2010, 05:27 AM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts