User-Agent: *
Allow: /

User-Agent: Mozilla/5.0 (compatible; Tubenowbot/1.0; http://tubenow.net/link-checker-bot/)
Allow: /

User-Agent: MJ12bot
Crawl-Delay:   5