[Dirvish] [administrivia] taming web spiders, particularly Baidu?

Jenny Hopkins hopkins.jenny at gmail.com
Mon Jul 23 20:28:47 UTC 2012

On 23 July 2012 20:04, James Stanley <james at incoherency.co.uk> wrote:
> Hi Keith,
> I've not noticed any problems with the dirvish site (though I don't
> visit it particularly often), but you may find that you have better luck
> using robots.txt to prevent all robots from indexing the large videos,
> but still allow them to index text content. Something like:
>   User-agent: *
>   Disallow: /directory-full-of-videos
> Good luck!

This worked for me when I had a similar problem but with trac.


More information about the Dirvish mailing list