Welcome to the Australian Ford Forums forum.

You are currently viewing our boards as a guest which gives you limited access to view most discussions and inserts advertising. By joining our free community you will have access to post topics, communicate privately with other members, respond to polls, upload content and access many other special features without post based advertising banners. Registration is simple and absolutely free so please, join our community today!

If you have any problems with the registration process or your account login, please contact us.

Please Note: All new registrations go through a manual approval queue to keep spammers out. This is checked twice each day so there will be a delay before your registration is activated.

Go Back   Australian Ford Forums > General Topics > Ford Forums Central > Site Support

Site Support If something isn't working or you have a suggestion ( a nice one !! ) let us know here.

Reply
 
Thread Tools Display Modes
Old 03-06-2021, 12:24 PM   #1
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Chinese Spiders

Good afternoon.

No, I'm not posting about some exotic 8-legged creature but about the web crawlers that have been hammering this site for the last 5 days or so.

These crawlers are ignoring the normal directives that robots are supposed to follow and thus we have several hundred of them crawling the site constantly.

The impact is that the server CPU's get maxxed out at 100% and overall performance slows to a (no pun intended) crawl.

They can normally be kept at bay by blocking them at the server level but for some reason this doesn't appear to be stopping this lot of Huawei crawlers.

There have been problems reported around the world from them so we aren't the only ones being impacted as Huawei try to get their search engine up to the standard of Google as part of their ongoing war with each other.

We are working with out ISP to find a solution.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
Old 03-06-2021, 12:36 PM   #2
Citroënbender
DIY Tragic
 
Citroënbender's Avatar
 
Join Date: Apr 2018
Location: Sydney, more than not. I hate it.
Posts: 20,674
Chairman's Award: Chairman's Award - Issue reason: Your outstanding contributions to this community have not gone unnoticed. IN my view you are a worthy recipient of the (rarely used) Chairman's Award. 
Default Re: Chinese Spiders

Thank you. I’d assumed the issue ran downstream and was just my problem.
Citroënbender is online now   Reply With Quote Multi-Quote with this Post
2 users like this post:
Old 03-06-2021, 12:39 PM   #3
lumen8
FF.Com.Au Hardcore
 
lumen8's Avatar
 
Join Date: Oct 2020
Posts: 670
Default Re: Chinese Spiders

How about blocking large swathes of chinese IP's in cpanel, crude but could be a stopgap fix. I doubt youd exclude any real members with this approach.

I realise you may already have done this, thought id mention it.
lumen8 is offline   Reply With Quote Multi-Quote with this Post
This user likes this post:
Old 03-06-2021, 12:49 PM   #4
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Quote:
Originally Posted by lumen8 View Post
How about blocking large swathes of chinese IP's in cpanel, crude but could be a stopgap fix. I doubt youd exclude any real members with this approach.

I realise you may already have done this, thought id mention it.
Already done that!
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
5 users like this post:
Old 03-06-2021, 01:09 PM   #5
aussiblue
FG XR6 Ute & Sedan
Donating Member3
 
Join Date: Oct 2006
Location: Bibra Lake WA
Posts: 22,300
Technical Contributor: For members who share their technical expertise. - Issue reason: Has been floating around the oze tech section for a long time and is always there to give advice when people have an issue. 
Default Re: Chinese Spiders

I was going to suggest "nuke em till they glow and then shoot 'em in the dark"


But they are the ones with all the nukes and they just want to be loved it seems https://www.bbc.com/news/world-asia-china-57327177

Quote:
China's president has said he wants the country to "expand its circle of friends" by revamping its image.

Xi Jinping told senior Communist Party officials it was important to present an image of a "credible, loveable and respectable China", according to a report by state-run news agency Xinhua.
__________________
regards Blue
aussiblue is offline   Reply With Quote Multi-Quote with this Post
Old 03-06-2021, 01:21 PM   #6
aussiblue
FG XR6 Ute & Sedan
Donating Member3
 
Join Date: Oct 2006
Location: Bibra Lake WA
Posts: 22,300
Technical Contributor: For members who share their technical expertise. - Issue reason: Has been floating around the oze tech section for a long time and is always there to give advice when people have an issue. 
Default Re: Chinese Spiders

Any of the comments on this blog helpful https://www.johnlarge.co.uk/blocking...scrapers-bots/ ?
__________________
regards Blue
aussiblue is offline   Reply With Quote Multi-Quote with this Post
Old 03-06-2021, 02:08 PM   #7
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Quote:
Originally Posted by aussiblue View Post
Any of the comments on this blog helpful https://www.johnlarge.co.uk/blocking...scrapers-bots/ ?
I had read that but we aren't seeing quite the same thing. I've moved the directives into the base apache config file to see if that has any impact.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
2 users like this post:
Old 03-06-2021, 02:18 PM   #8
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

It 'might' be resolved but there are quite a few of the spiders still connected so we'll need to wait until their sessions expire before knowing for sure.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
5 users like this post:
Old 03-06-2021, 02:51 PM   #9
Beastie
The Terrain Tamer
Donating Member3
 
Beastie's Avatar
 
Join Date: May 2013
Posts: 35,941
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Catering services for a bunch of layabouts and for being an all-round good guy whose sense of community goes above and beyond. 
Default Re: Chinese Spiders

Maybe closing the Covid Thread might have an impact...
__________________
Current Ride : A Ford owned D3...
Beastie is offline   Reply With Quote Multi-Quote with this Post
Old 03-06-2021, 03:27 PM   #10
Bossxr8
Peter Car
 
Bossxr8's Avatar
 
Join Date: Dec 2004
Location: geelong
Posts: 23,192
Default Re: Chinese Spiders

Have you tried aeroguard?
Bossxr8 is offline   Reply With Quote Multi-Quote with this Post
2 users like this post:
Old 03-06-2021, 03:58 PM   #11
lumen8
FF.Com.Au Hardcore
 
lumen8's Avatar
 
Join Date: Oct 2020
Posts: 670
Default Re: Chinese Spiders

Quote:
Originally Posted by Beastie View Post
Maybe closing the Covid Thread might have an impact...
Housing bubble thread too.


Maybe theyre here to try pick the bottom, get back into it and buy the dip.



lumen8 is offline   Reply With Quote Multi-Quote with this Post
This user likes this post:
Old 03-06-2021, 04:09 PM   #12
T3rminator
FF.Com.Au Hardcore
 
T3rminator's Avatar
 
Join Date: Jul 2005
Location: Melbourne
Posts: 6,800
Default Re: Chinese Spiders

Is it the sheer number of crawlers or are the crawlers doing weird stuff? Asked some of the folks at work, if the former, they were pointing as some form of DDOS protection....but I believe that is big $$$.
T3rminator is offline   Reply With Quote Multi-Quote with this Post
This user likes this post:
Old 03-06-2021, 04:57 PM   #13
Tickford.
🚫⏰4️⃣🐃💩
 
Tickford.'s Avatar
 
Join Date: Sep 2012
Posts: 1,905
Default Re: Chinese Spiders

I think I've found what these "Chinese Spiders" are trying to attack...

https://fordforums.com.au/showthread.php?t=11484389


Tickford. is offline   Reply With Quote Multi-Quote with this Post
Old 03-06-2021, 05:59 PM   #14
wodahs
FF.Com.Au Hardcore
 
wodahs's Avatar
 
Join Date: Nov 2005
Location: perth
Posts: 4,359
Default Re: Chinese Spiders

your kidding right

we all know what thread their here (like every one else) for !!
https://fordforums.com.au/showthread.php?t=11483655

__________________
yes still (as money n time permit) doing the

rebuilding the zh fairlane with a clevo 400m 4v heads injected whipple blown with aode 4 speed trans to a 9" ....... we'll get there eventually

just remember don't be afraid to try something new.
Remember, amateurs built the Ark...Professionals built the Titanic!

I have taken up meditation... at least it's better than sitting around doing nothing !!
wodahs is offline   Reply With Quote Multi-Quote with this Post
4 users like this post:
Old 03-06-2021, 10:22 PM   #15
Pis-ton broke
Banned
 
Pis-ton broke's Avatar
 
Join Date: Jan 2009
Posts: 1,621
Default Re: Chinese Spiders

there jealous because they don't have 351's and xa to xc coupes, oh and not forgetting xy gtho phase 3's.
Pis-ton broke is offline   Reply With Quote Multi-Quote with this Post
Old 04-06-2021, 08:36 AM   #16
FoxtrotGolfXray 5.0
Donating Member
Donating Member3
 
FoxtrotGolfXray 5.0's Avatar
 
Join Date: Mar 2007
Location: Heading thru Hell (Corner)
Posts: 7,918
Technical Contributor: For members who share their technical expertise. - Issue reason: Willingly providing technical info and documents, despite glitches. 
Default Re: Chinese Spiders

I thought they were all just salivating over the delicious delicacies in this thread
__________________
Labels are for jars, not for people.

Life is a journey, not a destination.

~~~~~~~~~~~~~~

Daily: 2013 FGII EcoLPi in Winter White
Play: 2015 FG X XR8 in Emperor Show' N Shine thread

Gone, but not forgotten: 2015 SZII petrol Titanium Territory in Emperor
FoxtrotGolfXray 5.0 is offline   Reply With Quote Multi-Quote with this Post
Old 04-06-2021, 08:50 AM   #17
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Quote:
Originally Posted by T3rminator View Post
Is it the sheer number of crawlers or are the crawlers doing weird stuff? Asked some of the folks at work, if the former, they were pointing as some form of DDOS protection....but I believe that is big $$$.
It's mostly the number but also the speed they are crawling. They are doing what they normally do when they are here but the general rule is that the number of spiders is kept reasonably small so that the impact isn't great. For example, we generally have about 30 Google or MSN spiders at any one time but we had more like 150 of these Huawei ones and they were working 3x as fast as the Google or MSN ones.

The (useless) answer from our ISP was to install a Fortinet Firewall in front of our server for an extra US$50 / month but I've managed to block a number of them and I can still use the Linux iptables firewall if I need to.

To give you an idea of the impact (and why it is hurting us) here are some raw numbers....

This time of year our data usage averages 15 Gb / day but since May 28th the average has been 141 Gb / day.

Likewise, we would normally serve ~500k pages a day with 700k server hits but the averages since 28th May have been 3.3M pages and 3.5M hits.

It is improving gradually and there are currently no Huawei spiders active but there are some other Chinese ones like Baidu that I still need to knock on the head.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
Old 04-06-2021, 02:28 PM   #18
mad2
Regular...with metamusal
Donating Member2
 
Join Date: Oct 2009
Location: Geeeloong
Posts: 5,841
Default Re: Chinese Spiders

is that what happened earlier?
mad2 is offline   Reply With Quote Multi-Quote with this Post
2 users like this post:
Old 04-06-2021, 02:57 PM   #19
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

No - that was me being stupid and restarting mySQL without confirming that the 'forum' user had access to the database.

I'm going to have to take it down again to fix that problem as the temporary workaround in place isn't very good.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
4 users like this post:
Old 04-06-2021, 02:58 PM   #20
FTE217
T3/Sprint8
Donating Member2
 
FTE217's Avatar
 
Join Date: Jan 2005
Location: Australia
Posts: 15,871
Default Re: Chinese Spiders

Quote:
Originally Posted by mad2 View Post
is that what happened earlier?
expect so for couldn't load page up some 10mins ago......edit, ah ok it was Russell stand corrected.

and people made noise about our stance re Huawei.
Glad the Poms went back on this as well, they quoted in the end it would run their IT period in time, more so CCP could just shut down/control whenever they wish. Baidu friggin their own google.
All power to you Russell.
__________________
Tickfords T3/TS50 '02
Sprint8 manual Sept 24 '16
Daily Macan GTS
"Don't believe everything you read on the internet. Abraham Lincoln"
FTE217 is offline   Reply With Quote Multi-Quote with this Post
2 users like this post:
Old 04-06-2021, 03:05 PM   #21
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Just as an FYI, I am going to shut down the server at 3:15 for the 5 minutes it will take to fix the database issue.

Apologies but it is necessary.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
6 users like this post:
Old 04-06-2021, 03:35 PM   #22
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Took more like 12 minutes but it's resolved now.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
Old 04-06-2021, 05:44 PM   #23
Tonz
FF.Com.Au Hardcore
 
Join Date: Jun 2011
Location: Pt Lincoln far side South Oz
Posts: 5,607
Default Re: Chinese Spiders

__________________
Dont p i s s off older people. At our age the term Life in Prison is not a deterrent
Tonz is offline   Reply With Quote Multi-Quote with this Post
Old 04-06-2021, 08:32 PM   #24
MITCHAY
FF.Com.Au Hardcore
 
Join Date: Apr 2005
Location: Canberra
Posts: 13,297
Default Re: Chinese Spiders

Thanks Russ. I thought something was not quite right.

I'm not sure what model we use here but it could be quite a difference in cost with that sort of demand.

I know that Google collect all sort of **** here but as I'm given to understand there is a lot of censorship over there so not sure sure what they are trying to achieve except to compile a hit list
MITCHAY is offline   Reply With Quote Multi-Quote with this Post
Old 05-06-2021, 03:52 PM   #25
T3rminator
FF.Com.Au Hardcore
 
T3rminator's Avatar
 
Join Date: Jul 2005
Location: Melbourne
Posts: 6,800
Default Re: Chinese Spiders

Have a look at these two references in case it happens again. The first link seems to recommend some plugins and tools which might help. The second link refers to Robot.txt, pretty standard, guessing you've already done it, relies on the crawlers to obey the rules, but obviously not mandatory.

https://codewithhugo.com/importance-crawler-bot-block/​

https://help.dreamhost.com/hc/en-us/...s-and-crawlers

Interesting, Huawei has been diversifying from their mobile and network equipment business, looks like they have now started their own search engine which would explain the crawlers.
T3rminator is offline   Reply With Quote Multi-Quote with this Post
Old 05-06-2021, 05:56 PM   #26
T3rminator
FF.Com.Au Hardcore
 
T3rminator's Avatar
 
Join Date: Jul 2005
Location: Melbourne
Posts: 6,800
Default Re: Chinese Spiders

Sorry, first link got screwed.

https://codewithhugo.com/importance-crawler-bot-block/

Here are some helpful tips to prevent malicious bots from attacking your website:

Quote:
Adding an additional application plugin, such as the Wordfence security plugin for WordPress prevents such attacks.

It’s also advisable to set access rules to get rid of malicious requests. You can disallow a specific search engine by inputting the name of the search engine on the disallow user agent of your robots.txt file.

You can disallow other search engines from crawling your website except for Googlebot by only allowing Googlebot as your user agent in your robots.txt file.

Note: A robots.txt refers to a file that’s associated with your website. It’s used to ask web crawlers to crawl or not to crawl some parts or web pages of your website. In short, a robots.txt file specifies which of your web pages should be crawled by web crawlers or spiders.
T3rminator is offline   Reply With Quote Multi-Quote with this Post
Old 05-06-2021, 11:40 PM   #27
russellw
Chairman & Administrator
Donating Member3
 
russellw's Avatar
 
Join Date: Dec 2004
Location: 1975
Posts: 106,469
Community Builder: In recognition of those who have helped build the AFF community. - Issue reason: Raptor: For Continued, and prolonged service to the wider Ford Community 
Default Re: Chinese Spiders

Yes, we already had robots.txt in use but they ignore it so they have been blocked at the firewall level to keep them out. I've also set up a honey pot and fail2ban to keep hyperactive ones at bay in the future.
__________________

__________________________________________________

Observatio Facta Rotae


russellw is offline   Reply With Quote Multi-Quote with this Post
2 users like this post:
Reply

Thread Tools
Display Modes

Forum Jump


All times are GMT +11. The time now is 04:50 AM.


Powered by vBulletin® Version 3.8.5
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Other than what is legally copyrighted by the respective owners, this site is copyright www.fordforums.com.au
Positive SSL