关于百度robots文件设置的探讨

User-agent: BaiduspiderDisallow: /baiduDisallow: /s?User-agent: GooglebotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: MSNBotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Baiduspider-imageDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: YoudaoBotDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou web spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou inst spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou spider2Disallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou blogDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou News SpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Sogou Orion spiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: JikeSpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: SosospiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: PangusoSpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: Yahoo! Slurp ChinaDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: EasouSpiderDisallow: /baiduDisallow: /s?Disallow: /shifen/Disallow: /homepage/Disallow: /cproUser-agent: *Disallow: /上面是百度的robots文件你看他为什么要这成这样呢？ 我们仔细看下上面都是一个一个的禁止部分目录，然后呢突然下面又来一个所有都禁止不知道百度这是玩的什么啊！

主要规范每一个蜘蛛 最后是规范杂七杂八的不知名的蜘蛛

应该最后一个是全集。特别是包含了上面没提到的360.
我印象中，似乎最后一个在之前是没有的，估计最近刚加上的。