User-agent: *
Allow: *
User-agent: googlebot 
Disallow:/urllist.txt.gz
User-agent: yahoo-slurp 
Disallow:/sitemap.xml.gz