SemrushBot、MJ12bot等垃圾蜘蛛经常占用服务器资源，用了很多方法都不奏效。这些垃圾蜘蛛根本不遵循robots协议，有点流氓的感觉。
这一次用了一种新的方法，彻底屏蔽了他们。
通过宝塔面板网站设置粘贴有代码：
#禁止垃圾蜘蛛抓取
if ($http_user_agent ~* (SemrushBot|python|Linespider|crawler|DingTalkBot|simplecrawler|ZoominfoBot|zoombot|Neevabot|coccocbot|Facebot|YandexBot|Adsbot|DotBot|Applebot|DataForSeoBot|MJ12bot|BLEXBot|trendictionbot0|trendictionbot|AhrefsBot|hubspot|opensiteexplorer|leiki|webmeup)) { return 444;
}
把这组代码粘贴保存即可。
经过几天的测试观察，这些蜘蛛完全屏蔽了。
声明：本站标注原创发布或特殊说明内容。任何个人或组织，在未征得本站同意时，禁止复制、盗用、采集、发布本站内容到任何网站及平台。未标注原创内容来源于公开的网络或由用户分享，如若本站内容侵犯了原著者的合法权益，可联系我们进行处理。
All content on this website are from the network, if inadvertently violated your copyright (or interest), please email letter to tell us will be removed as soon as possible
All content on this website are from the network, if inadvertently violated your copyright (or interest), please email letter to tell us will be removed as soon as possible
评论(0)