[Dirvish] [administrivia] taming web spiders, particularly Baidu?

[Dale Amon]

On Mon, Jul 23, 2012 at 11:31:41AM -0700, Keith Lofstrom wrote:
> Is there any way to tell the search spiders to visit once a day
> or once a week, rather than four times per hour?  Or send them
> "recent changes" lists instead of them repeatedly downloading the
> same files?  Any other ideas for calming down the web crawlers?

I don't know of a good answer but I could suggest a hack. Keep
two copies of your robots.txt file and use a cron job to swap them
out for an hour or two in the middle of the night or wherever
your slowest traffic period falls.