Thread: Chinese Spiders
View Single Post
Old 05-06-2021, 04:52 PM   #25
T3rminator
FF.Com.Au Hardcore
 
T3rminator's Avatar
 
Join Date: Jul 2005
Location: Melbourne
Posts: 6,813
Default Re: Chinese Spiders

Have a look at these two references in case it happens again. The first link seems to recommend some plugins and tools which might help. The second link refers to Robot.txt, pretty standard, guessing you've already done it, relies on the crawlers to obey the rules, but obviously not mandatory.

https://codewithhugo.com/importance-crawler-bot-block/​

https://help.dreamhost.com/hc/en-us/...s-and-crawlers

Interesting, Huawei has been diversifying from their mobile and network equipment business, looks like they have now started their own search engine which would explain the crawlers.
T3rminator is offline   Reply With Quote