We should switch robots.txt to allow more things, a good template to use would be Mozilla's bugzilla robots.txt: https://bugzilla.mozilla.org/robots.txt
yep this apparently is quite common, web-crawling of bugzilla can be pretty heavy so mozilla set up a default that banned pretty much everything. i'm not so bothered so have set it to "Allow /"
(In reply to Jacob Lifshay from comment #0) > a good template to use > would be Mozilla's bugzilla robots.txt: > https://bugzilla.mozilla.org/robots.txt just copied it entirely, just... because :)
(In reply to Luke Kenneth Casson Leighton from comment #2) > (In reply to Jacob Lifshay from comment #0) > > a good template to use > > would be Mozilla's bugzilla robots.txt: > > https://bugzilla.mozilla.org/robots.txt > > just copied it entirely, just... because :) Thanks, sounds good to me! if you have more time, it would be nice to also fix #149