Caution
This project is intended for educational purposes only. It serves as a learning resource for understanding how to develop a Django REST Framework API for scraping cryptocurrency data. The project utilizes Celery for asynchronous task management and Selenium for dynamic web scraping.
This repository contains a Django REST Framework API for scraping cryptocurrency data from various sources. It utilizes Celery for asynchronous task management, allowing parallel scraping of multiple cryptocurrencies. The API enables users to initiate scraping tasks, monitor their progress, and retrieve the scraped data.
- Scraping Initiation: Initiate scraping tasks for multiple cryptocurrencies simultaneously.
- Asynchronous Processing: Utilize Celery for asynchronous execution of scraping jobs, ensuring scalability and responsiveness.
- Real-time Monitoring: Track the progress of scraping tasks and retrieve the current status.
- Flexible Scraping: Utilize Selenium for dynamic web scraping, enabling extraction of data from JavaScript-rendered pages.
Endpoint: POST /api/taskmanager/start_scraping
Endpoint: GET /api/taskmanager/scraping_status/<job_id>
-
Clone the repository:
git clone https://github.com/yourusername/cryptocurrency-scraping-api.git
-
Install dependencies:
pip install -r requirements.txt
-
Run migrations:
python manage.py migrate
-
Start the Celery worker:
celery -A yourprojectname worker --loglevel=info
-
Start the Django development server:
python manage.py runserver
Contributions are welcome! Please open an issue or submit a pull request to propose changes or enhancements to the project.
Feel free to customize the content and styling further according to your preferences and project specifics.