Numerous crawlers aggressively visit web server by sending many simultaneous requests to a web server in a small time frame. We think that it is not the right thing to do. Right now, the SiteCozy broken link scanner visits one resource at a time.
There are several reasons why we think that offering a very fast broken link checker is not the right thing to do for several reasons:
- If there is already a significant amount of web traffic to a web server, it is unwanted to stress the web server. I especially think about web servers with little resources.
- If firewall rules are like rate limiter are in place, they may refuse some requests by “thinking” that the web server is under a Denial-of-service attack. Some legitimate requests from regular visitors can be refused by the web server.
- We don’t want the Sitecozy broken link scanner application to be used as a denial-of-service
attacker against a web server. Imagine that the SiteCozy broken link scanner would send as many simultaneous requests as it could. An individual could install the app on several machines and perform a DDOS attack. We already saw that a crawler can take down a web server - If we stress servers from main shared web hosting providers, they will block our user agent.