Fuck Semrush

— Cunts.

Published on 08-10-2023 by emily

Hello everyone! In this post, I express my frustration toward web crawlers. Lots of swearing inbound!

Exhibit A: "Semrush"

You know what, let's just see how many times Semrush have hit my subdomains, shall we? As of writing this, it's only 10 in the morning.

┌─[donut][~]
└─> grep -i "semrush" /var/log/apache2/other_vhosts_access.log | wc -l
2098

Oh... Well, perhaps this is just a very small amount when compared to other traffic, right? Let's see.

┌─[donut][~]
└─> grep -iv "semrush" /var/log/apache2/other_vhosts_access.log | wc -l
433

...Yep, that's right. Semrush makes up 80% of all traffic on my subdomains so far today. Need I say more?

Exhibit B: "Palo Alto Networks"

162.216.150.161 - - [08/Oct/2023:07:08:12 +0000] "GET / HTTP/1.1" 200 13548 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: [email protected]"

Well, that's a new strategy. Advertising your service in access.log.

Anyway, what is this bullshit? I'm not one of your "customers". God, this sounds so condescending. Like I'm just one of their "customers" and this is my "customer's presence on the Internet". Not just a presence on the internet, no no, a CUSTOMER presence. God dammit.

At least they're not as horrible as Semrush. For example, they don't give a shit about my subdomains, and only crawl my main page. They make up less than half of a percent of traffic on there. :)

Honourable Mentions: Script Kiddies

For this section, I want you to do nothing but point and laugh.

37.221.92.222 - - [08/Oct/2023:00:07:43 +0000] "GET ../../proc/ HTTP" 400 392 "-" "-"
134.122.38.188 - - [08/Oct/2023:00:43:57 +0000] "POST /wsman HTTP/1.1" 404 6451 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
138.68.130.67 - - [08/Oct/2023:00:43:59 +0000] "GET /sslvpnLogin.html HTTP/1.1" 404 6423 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
64.227.30.198 - - [08/Oct/2023:00:49:13 +0000] "Gh0st\xad" 400 392 "-" "-"
172.70.46.127 - - [08/Oct/2023:02:33:19 +0000] "GET /alfadheat.php HTTP/1.1" 404 6452 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"
172.71.94.8 - - [08/Oct/2023:02:33:35 +0000] "GET /M1.php HTTP/1.1" 404 6458 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"

That's just a small part of the logs. I don't want to include all of it, because that would be boring.

I do want to mention 172.70.46.127 and 172.71.94.8's poor spelling abilities. "Mozlila", "Bulid", "Moblie"... Really? Perhaps you should pay more attention to school instead of trying to l33t h4xx0r websites.

Closing

Well, that's it for today. Thanks for reading this blog post. I appreciate it <3.

--emily