[Dirvish] [administrivia] taming web spiders, particularly Baidu?
amon at vnl.com
Mon Jul 23 18:55:02 UTC 2012
On Mon, Jul 23, 2012 at 11:31:41AM -0700, Keith Lofstrom wrote:
> Is there any way to tell the search spiders to visit once a day
> or once a week, rather than four times per hour? Or send them
> "recent changes" lists instead of them repeatedly downloading the
> same files? Any other ideas for calming down the web crawlers?
I don't know of a good answer but I could suggest a hack. Keep
two copies of your robots.txt file and use a cron job to swap them
out for an hour or two in the middle of the night or wherever
your slowest traffic period falls.
More information about the Dirvish