[Dirvish] [administrivia] taming web spiders, particularly Baidu?

[Jenny Hopkins]

On 23 July 2012 20:04, James Stanley <james at incoherency.co.uk> wrote:
> Hi Keith,
>
> I've not noticed any problems with the dirvish site (though I don't
> visit it particularly often), but you may find that you have better luck
> using robots.txt to prevent all robots from indexing the large videos,
> but still allow them to index text content. Something like:
>
>   User-agent: *
>   Disallow: /directory-full-of-videos
>
> Good luck!
>

This worked for me when I had a similar problem but with trac.

Jenny