Results 1 to 2 of 2

Thread: Webmasters conceren - Robot.txt Good or Bad!

  1. #1
    Join Date
    Jul 2002
    Posts
    170
    Rep Power
    0

    Default Webmasters conceren - Robot.txt Good or Bad!

    I found an article - what do you people think?

    ----------------------------------

    Robot.txt Good or Bad!
    This article will help you to decide for yourself once you have read it. It shows you how to create a robot.txt file.

    Basically, robots.txt is a plain text file which is placed in a server's root directory it includes information on whether search engine robots should index the site or parts of the site. The file (line begins with '#'), then 'User-agent' lines.
    Usually, the User-agent line is simply a wildcard, to exclude all robots, like so :

    # robots.txt for http://yoursite.com
    User-agent: *

    although you can write seperate agent/disallow sections for different robots.
    Next comes the Disallow section. this is read by the robot and from there, it determines what's off-limits when it comes to indexing your site.

    # robots.txt for http://yoursite.com
    User-agent: *
    Disallow: /administration/ # nothing under /administration/ should be spidered Disallow: /temp/ # these are temporary files
    Disallow: /active.asp # active content here, no point spidering it

    Disallowing pages deep into your structure can be good for your users, they won't find themselves halfway through the site with no idea how to get out. Then again, the more search engine entries you have, the better, right? It's up to you to decide what should or shouldn't be excluded.

    OK so robot.txt is good for your users and to tell the search engines which pages to list but. Here is the bad part not all robots or bots are good some will ignore the robot.txt file and just index all pages it comes across. So some of your admin pages could get displayed somewhere.
    Also now you know about robot.txt it needs to be in the root directory what's to stop someone going around and typing http://yoursite.com/robot.txt this would display your robot.txt file!

    That would be like going into a bar and leaving you wallet on the counter and going to the restroom. What's the chances of it still being there when you get back :-\

    So now you know a bit about robot.txt it up to you to decide.

    -----------------------------------


    Should we stop using robots.txt & instead place the robots meta field in pages?

  2. #2
    Join Date
    Jul 2002
    Posts
    5,446
    Rep Power
    10

    Default Re:Webmasters conceren - Robot.txt Good or Bad!

    I'm not all too familiar with robots.txt files, actually it's the first that I've heard of it :-[. IMHO I think the negatives outweigh the positives of robot.txt so I'll do without it. There must be a more secure way of accomplishing the same objectives.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •