• 0 Posts
  • 5 Comments
Joined 3 years ago
cake
Cake day: June 30th, 2023

help-circle
  • My hypothesis is that those scrappers run on botnets hosted on infected servers, so they want to get the info as fast as possible, since you don’t know how long it will last before they are detected and removed.

    Another option is that so many servers are scrapping the same web that one server may send only a request every a few seconds, but the whole fleet could be sending thousands of requests per second, some big websites can handle the load, but most small to medium size sites get overwhelmed.




  • One thing to note, CORS only makes sense if your API uses cookies for authentication, most APIs use custom headers, the Authentication header, or even url tokens, they don’t rely on cookies, so most of the time, APIs don’t care about CORS. People keep blindly repeating that accepting all origins “*”, is bad for security, but the situations where this is relevant are really uncommon.