[Dirvish] [administrivia] taming web spiders, particularly Baidu?

Dale Amon amon at vnl.com
Mon Jul 23 18:55:02 UTC 2012

On Mon, Jul 23, 2012 at 11:31:41AM -0700, Keith Lofstrom wrote:
> Is there any way to tell the search spiders to visit once a day
> or once a week, rather than four times per hour?  Or send them
> "recent changes" lists instead of them repeatedly downloading the
> same files?  Any other ideas for calming down the web crawlers?

I don't know of a good answer but I could suggest a hack. Keep
two copies of your robots.txt file and use a cron job to swap them
out for an hour or two in the middle of the night or wherever
your slowest traffic period falls.

More information about the Dirvish mailing list