The web is broken, IMHO
So there is a (IMHO) shady market out there that gives app developers on iOS, Android, MacOS and Windows money for including a library that sells users network bandwidth. Infatica [1] is just one example, there are many more.
I am 99% sure that these companies cause what effectively are DDoS attacks that many webmasters have to deal with since months. This business model should simply not exist. Apple, Microsoft and Google should act.
1/8
[1] https://infatica.io/sdk-monetization/
What these companies then sell to *their* customers is network access through the devices/PCs that have an app with this SDK installed. They are proud to tell you how you can funnel your (AI) web scraping etc through millions of rotating, residential and mobile IP addresses. Exactly the pattern we see hitting our servers.
2/8
Now, again, this company is just one of many selling similar services. And they all promise that they carefully check what commands their customers send to the (IMHO) infected apps on your phone and PC. Yeah, I am sure they "do no evil". And when they do, they can claim it's not their problem because they are merely the proxy. Again, IMHO, a shady business model.
3/8
But this explains the explosion of bot traffic that really cripples a lot of smaller services (like my forgejo instance, that I had to make non-public).
So if you include such an SDK in your app to make some money — you are part of the problem and I think you should be punished for that. You are delivering malware to your users, making them botnet members.
Unfortunately it is next to impossible for normal users to detect the inclusion of such shady SDKs and the network traffic they cause.
4/8
I already blogged about this at https://jan.wildeboer.net/2025/02/Blocking-Stealthy-Botnets/
I might rewrite that blog post to make the problem clearer. And to explain why I am now of the opinion that *every* form of web-scraping should be considered abusive. If you think your web-scraping is acceptable behaviour, you can thank these shady companies and the "AI" hype for moving you to the bad corner.
TL;DR certain companies recruit app developers to create botnets. Botnets are malware. Period.
The web is broken, IMHO.
5/8
Addendum: Trend Micro did some research on these companies back in 2023 and it confirms my suspicions. And I guess with AI scraping this kind of business is booming. For the paranoid:
„There are malicious actors who repacked freeware and shareware written by other people to conduct drive-by downloads of the Infatica peer-to-business (P2B) service“
6/8
Addendum 2: If you want to feel really dirty, go to https://proxyway.com/reviews?e-filter-da2a7bc-reviews_categories=proxy-providers for a collection of reviews on these services. It's a huge market and I am 100% convinced that "AI" web scraping is currently the biggest "growth" driver for these companies.
And when I see that quite some of them rely on injecting SDKs into 3rd party apps to "extend" their "Reach", I would call these "residential proxy providers" malware/botnets. But that's just my personal opinion. I am sure they are all legit.
7/8
If you've made it to this final post of this thread — thank you for your time and interest! I hope it helps you understand why web crawlers have become a real problem and how this is more and more an attack on the foundation of the Web as it was intended to be. This "residential proxy" business is just one part of this. And we webmasters/admins can only try to block. It is getting more and more difficult to keep up with these waves. Thanks "AI"!
I will convert this thread to a blog post.
8/8
Done: This thread is now a blog post at https://jan.wildeboer.net/2025/04/Web-is-Broken-Botnet-Part-2/
