[Dirvish] [administrivia] taming web spiders, particularly Baidu?
hopkins.jenny at gmail.com
Mon Jul 23 20:28:47 UTC 2012
On 23 July 2012 20:04, James Stanley <james at incoherency.co.uk> wrote:
> Hi Keith,
> I've not noticed any problems with the dirvish site (though I don't
> visit it particularly often), but you may find that you have better luck
> using robots.txt to prevent all robots from indexing the large videos,
> but still allow them to index text content. Something like:
> User-agent: *
> Disallow: /directory-full-of-videos
> Good luck!
This worked for me when I had a similar problem but with trac.
More information about the Dirvish