nmtp.info/robots.txt:
User-Agent: *
Crawl-delay: 60
Sitemap: http://nmtp.info/sitemap.xml
Disallow: /auth/
Disallow: /awstats/
Disallow: /bitrix/
Disallow: /content/
Disallow: /search/
Disallow: /upload/
Disallow: /auth.php
Disallow: *.jpg
Disallow: *.png
Disallow: *.gif
Disallow: *.doc
Disallow: *.docx
Disallow: *.pdf
Disallow: *.xls
Disallow: *.xlsx
Disallow: *.zip
Disallow: /*PAGE_NAME=search
Disallow: /*PAGE_NAME=user_post
Disallow: /*PAGE_NAME=detail_slide_show
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*register=yes
Disallow: /*forgot_password=yes
Disallow: /*change_password=yes
Disallow: /*login=yes
Disallow: /*logout=yes
Disallow: /*auth=yes
Disallow: /*action=ADD_TO_COMPARE_LIST
Disallow: /*action=DELETE_FROM_COMPARE_LIST
Disallow: /*action=ADD2BASKET
Disallow: /*action=BUY
Disallow: /*print_course=Y
Disallow: /*bitrix_*=
Disallow: /*backurl=*
Disallow: /*BACKURL=*
Disallow: /*back_url=*
Disallow: /*BACK_URL=*
Disallow: /*back_url_admin=*
Disallow: /*index.php$
Disallow: /bitrix/rk.php
в яндексе
Прочее:
Ping/Trace,
История в Веб-архиве,
История владельцев,
Валидность HTML,
Валидность CSS,
Проверка на вирусы.
Запрашиваемая страница недоступна!
|