www.uzelok.ru/robots.txt:
User-Agent: *
Disallow: /search.php
Disallow: /forum/
Disallow: /forum/search.php
Disallow: /forum/memberlist.php
Disallow: /forum/profile.php
Disallow: /forum/posting.php
Disallow: /forum/rlink/
Disallow: /shem.php
Disallow: /spici/
Disallow: /kruchok/
Disallow: /baby/
Disallow: /shapki/
Disallow: /zakaz/
Disallow: /myauthor/
Disallow: /otherauthor/
Disallow: /makrame/
Disallow: /gall/
Disallow: /clickban.php
Disallow: /catalogue.php
Disallow: /images/
Disallow: /im/
Crawl-delay: 5
Host: www.uzelok.ru
User-agent: Bingbot
Crawl-delay: 5
User-agent: Mail.RU_Bot
Crawl-delay: 5
User-agent: Pinterestbot
Crawl-delay: 5
User-agent: VelenPublicWebCrawler
Disallow: /
User-agent: VelenPublicWebCrawler/1.0
Disallow: /
User-agent: AhrefsBot/6.1
Disallow: /
User-agent: AhrefsBot/5.1
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: wotbox
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: BUbiNG
Disallow: /
User-agent: Vedma
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: Riddler
Disallow: /
в яндексе
Прочее:
Ping/Trace,
История в Веб-архиве,
История владельцев,
Валидность HTML,
Валидность CSS,
Проверка на вирусы.
Запрашиваемая страница недоступна!
|