# For further information, refer to : # http://www.archive.org/about/exclude.php # http://www.global-positioning.com/robots_text_file/index.html # http://www.webtoolcentral.com/webmaster/tools/robots_txt_file_generator # http://pageresource.com/zine/robotstxt.htm # This section disallows all User-agent: * Disallow: /secure/ Disallow: /cgi-bin/ Disallow: /sorder/ Disallow: /analyser.zip Disallow: /ezkit.zip Disallow: /nhl.zip Disallow: /nhl.tar.gz Disallow: /sigdemo.zip Disallow: /sigimag.zip Disallow: /wgenpp.zip Disallow: /siglibdllz.zip Disallow: /siglibz.zip Disallow: /siglibz.zip Disallow: /dllsiglibz.zip Disallow: /siglibdllz.zip Disallow: /siglib_linux.tar.gz Disallow: /siglib.nzf Disallow: /siglib.z Disallow: /secure Disallow: /sorder Disallow: /keyfiles Disallow: /eval/siglib_eval.zip Disallow: /files/siglibz.zip Disallow: /files/siglibbinz.zip Disallow: /files/siglib_linux.tar.gz # This section disallows web archive (www.archive.com) User-agent: ia_archiver Disallow: /secure/ Disallow: /cgi-bin/ Disallow: /sorder/ Disallow: /analyser.zip Disallow: /ezkit.zip Disallow: /nhl.zip Disallow: /nhl.tar.gz Disallow: /sigdemo.zip Disallow: /sigimag.zip Disallow: /wgenpp.zip Disallow: /siglibdllz.zip Disallow: /siglibz.zip Disallow: /siglibz.zip Disallow: /dllsiglibz.zip Disallow: /siglibdllz.zip Disallow: /siglib_linux.tar.gz Disallow: /siglib.nzf Disallow: /siglib.z Disallow: /secure Disallow: /sorder Disallow: /keyfiles Disallow: /eval/siglib_eval.zip Disallow: /files/siglibz.zip Disallow: /files/siglibbinz.zip Disallow: /files/siglib_linux.tar.gz