Drama Harvester is a dynamic tool designed to interface seamlessly with rDrama's public APIs, capturing and caching content like comments and posts in Redis for efficient access and processing. Perfect for developers seeking to integrate or analyze rDrama data, it offers automatic and incremental fetching to ensure up-to-date content retrieval without redundancy. With its focus on performance and scalability, Drama Harvester serves as a robust foundation for applications requiring real-time or near-real-time access to rDrama content.

Go to file

j 657b11e36b Updated exponential fallback to retry 10 times for up to 68.2 minutes		2024-03-23 13:54:03 -04:00
src	Updated exponential fallback to retry 10 times for up to 68.2 minutes	2024-03-23 13:54:03 -04:00
tests/api	Initial commit	2024-03-19 01:16:12 -04:00
.gitignore	Initial commit	2024-03-19 01:16:12 -04:00
README.MD	updated repo link in readme	2024-03-22 22:02:52 -04:00
package-lock.json	tree shaking	2024-03-22 01:25:26 -04:00
package.json	tree shaking	2024-03-22 01:25:26 -04:00
tsconfig.json	Initial commit	2024-03-19 01:16:12 -04:00
tsconfig.tests.json	Initial commit	2024-03-19 01:16:12 -04:00

README.MD

Drama Harvester

Introduction

Drama Harvester is an innovative tool crafted for seamless interaction with rDrama's public APIs. It efficiently retrieves a variety of content, including comments and posts, and publishes them to Redis. This mechanism ensures not only the effective caching of data but also serves as a critical infrastructure for applications that necessitate immediate and streamlined access to the dynamic world of rDrama content.

Features

Efficient Data Retrieval: Fetches comments and posts directly from rDrama's APIs, optimized for minimal bandwidth usage.
Redis Integration: Utilizes Redis for caching content, significantly enhancing data retrieval speed and reducing API call redundancy.
Incremental Fetching: Employs smart fetching strategies to avoid duplicate data retrieval, ensuring that only new or updated content is processed.
Scheduled Fetching: Automatically executes data retrieval operations at configured intervals, enabling up-to-date synchronization with rDrama content.
Graceful Shutdown: Implements robust error handling and graceful shutdown processes, ensuring data integrity and application stability.
Scalable Architecture: Designed with scalability in mind, allowing for easy horizontal scaling to accommodate growing data volumes.

Getting Started

Prerequisites

Node.js (version 14 or later recommended)
Redis server

Installation

Clone the repository:

git clone https://fsdfsd.net/J/DramaHarvester.git

Navigate to the project directory:
```
cd DramaHarvester
```
Install the required dependencies:
```
npm install
```

Configure environment variables by creating a .env file in the project root with the following content:

REDIS_HOST=localhost
REDIS_PORT=6379
#REDIS_PASSWORD=optional
RDRAMA_API_KEY=your_rdrama_api_key_here

Usage

To start Drama Harvester, run the following command in the project directory:

npm start

Drama Harvester will begin fetching content from rDrama and publishing it to Redis at regular intervals (configurable in the source code).

Contributing

Contributions to Drama Harvester are warmly welcomed. Whether it's bug reports, feature requests, or contributions to code, we appreciate your effort to make Drama Harvester better.

To contribute:

Fork the repository.
Create a new branch for your feature (git checkout -b feature/AmazingFeature).
Commit your changes (git commit -m 'Add some AmazingFeature').
Push to the branch (git push origin feature/AmazingFeature).
Open a Pull Request.

License

Distributed under the MIT License. See LICENSE for more information.