User agent поисковых роботов Яндекса, Google, Rambler, Yahoo, MSN, Alexa

��  �� �� (�� . crawler) - ��, �� , �� , �� . �� . �� , �� , �� , � �� . �� .

�� -�� (User-agent) �� . �� �� User-agent �� , �� , �� , �� .

��, �� , �� AdSense.

User-agent: *
Disallow: /
User-agent: Mediapartners-Google
Disallow:

� �� �� .

�� (Yandex)

(�� )

User-agent �� : Yandex

- Yandex/1.01.001 (compatible; Win16; I) � �� .
- Yandex/1.01.001 (compatible; Win16; P) � �� .
- Yandex/1.01.001 (compatible; Win16; H) � ��, �� .
- Yandex/1.02.000 (compatible; Win16; F) � ��, �� (favicons)
- Yandex/1.03.003 (compatible; Win16; D) � ��, �� URL�
- Yandex/1.03.000 (compatible; Win16; M) � ��, ��
- YaDirectBot/1.0 (compatible; Win16; I) � ��, �� , ��
- YandexBlog/0.99.101 (compatible; DOS3.30,B) � ��, �� xml-�� .
- YandexSomething/1.0 � ��, �� -��.
- Bond, James Bond (version 0.07) � �� . �� . Referer �� . �� . ��, �� .�.

IP-��

IP-��, � �� , ��, � �� . �� .

�� ��-�� (�� “��“) , �� , �� .

- Yandex/2.01.000 (compatible; Win16; Dyatel; C) � �� .��. �� , �� . �� , �� .
- Yandex/2.01.000 (compatible; Win16; Dyatel; Z) � �� .��. �� .
- Yandex/2.01.000 (compatible; Win16; Dyatel; D) � �� .��. �� . �� .
- Yandex/2.01.000 (compatible; Win16; Dyatel; N) � �� .��. �� -��, �� , �� , �� .

�� Google (��)

User-agent �� : Googlebot

- Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) - �� Google, �� HTTP 1.1.
- Mediapartners-Google - �� AdSens
- Googlebot-Image (Google) Googlebot-Image/1.0 - ��
- Google Search Appliance (Google) gsa-crawler - �� Search Appliance

�� Rambler (��)

User-agent �� : StackRambler

�� , Ip-�� 81.19.64.0 - 81.19.66.255

- StackRambler/2.0 (MSIE incompatible)
- StackRambler/2.0

�� Yahoo

User-agent �� Yahoo: Slurp �� Yahoo! Slurp

- Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) - �� .
- Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) - �� Yahoo 3-�� , �� .

�� MSN

User-agent �� MSN: MSNBot

- msnbot/1.1 (+http://search.msn.com/msnbot.htm) - �� MSN.
- msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) - �� .
- msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) - �� .
- msnbot-news (+http://search.msn.com/msnbot.htm) - �� .