rajwap.pro/robots.txt:
User-Agent: *
Disallow: /*.php*
User-agent: 008
Disallow: /
User-Agent: BLEXBot
Disallow: /
User-Agent: AhrefsBot
Disallow: /
User-Agent: Vagabondo
Disallow: /
User-Agent: SEOkicks-Robot
Disallow: /
User-Agent: ia_archiver
Disallow: /
User-Agent: archive.org_bot
Disallow: /
User-Agent: special_archiver
Disallow: /
User-Agent: MJ12bot
Disallow: /
User-Agent: special_archiver
Disallow: /
User-Agent: heritrix
Disallow: /
User-Agent: netEstate NE Crawler
Disallow: /
User-Agent: sistrix
Disallow: /
User-Agent: WBSearchBot
Disallow: /
User-Agent: QuerySeekerSpider
Disallow: /
User-agent: proximic
Disallow: /
User-agent: SiteExplorer
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: Baiduspider
Crawl-delay: 30
User-agent: MSNBot
Crawl-delay: 7
User-agent: Bingbot
Crawl-delay: 7
в яндексе
Прочее:
Ping/Trace,
История в Веб-архиве,
История владельцев,
Валидность HTML,
Валидность CSS,
Проверка на вирусы.
Запрашиваемая страница недоступна!
|