High Memory Usage with ScrapydWeb

[vikhilrestaverse]

High Memory Usage with ScrapydWeb

I've observed an issue where ScrapydWeb causes excessive memory usage when running alongside Scrapyd. On an EC2 instance with 8GB of RAM and 2 vCPUs, Scrapyd alone uses around 2GB of memory, which is expected. However, when ScrapydWeb is started, the memory usage spikes to 100%, leading to the instance becoming unresponsive and eventually stopping.

Expected behavior
ScrapydWeb should operate without causing such a drastic increase in memory usage, allowing the system to remain stable.

Actual behaviour
When ScrapydWeb is running, the memory usage increases significantly, leading to the system becoming unresponsive.

Environment:
Instance: AWS EC2, 8GB RAM, 2 vCPUs
Scrapy Version: 2.11.2
Scrapyd Version: 1.4.3
ScrapydWeb Version: 1.5.0
Operating System: Linux Arm

Additional Information:
This issue occurs consistently with ScrapydWeb.
The instance is running minimal other processes, so the memory spike is attributable to ScrapydWeb.

*Request for Assistance:
Any guidance on resolving this memory usage issue or optimizing ScrapydWeb would be greatly appreciated.

[vikhilrestaverse]

@my8100 Can you please giude on this ?