# DramaBox Database Scraping System

## Overview

This system allows you to scrape DramaBox API data and store it in your local database with optional S3 video storage. This solves the problem of frequent API changes and provides better performance and reliability.

## Features

- **Stable Data**: No more API breaking changes
- **Better Performance**: Local database queries vs external API calls
- **Video Storage**: Download videos to S3 for complete independence
- **Smart Fallback**: Falls back to live API when scraped data is unavailable
- **Rate Limiting**: Respects DramaBox servers with configurable delays
- **Incremental Scraping**: Skip existing dramas unless forced

## Setup

### 1. Database Migration

```bash
php artisan migrate
```

This creates two tables:
- `scraped_dramas` - Drama metadata
- `scraped_episodes` - Episode data and video references

### 2. S3 Configuration (Optional)

Add to your `.env` file:

```env
# Basic S3 Configuration
AWS_ACCESS_KEY_ID=your_key
AWS_SECRET_ACCESS_KEY=your_secret
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=your-bucket-name

# Scraping Configuration
DRAMA_SCRAPING_ENABLED=true
DRAMA_SCRAPING_VIDEO_DOWNLOAD=false  # Set to true to download videos
DRAMA_SCRAPING_RATE_LIMIT=500       # Delay between requests (ms)
DRAMA_SCRAPING_BATCH_SIZE=50        # Dramas per run
DRAMA_SCRAPING_MAX_VIDEO_SIZE=500   # Max video size (MB)
DRAMA_USE_DATABASE_FIRST=true       # Use scraped data before API
```

## Usage

### Basic Scraping

```bash
# Scrape 100 dramas in English
php artisan dramabox:scrape-database --language=en --limit=100

# Scrape with video downloads to S3
php artisan dramabox:scrape-database --language=en --limit=50 --videos

# Force re-scrape existing dramas
php artisan dramabox:scrape-database --language=en --limit=25 --force

# Dry run to see what would be scraped
php artisan dramabox:scrape-database --language=en --limit=10 --dry-run
```

### Incremental Scraping (Recommended)

Set up a scheduled job to scrape new content regularly:

```php
// In app/Console/Kernel.php
protected function schedule(Schedule $schedule)
{
    // Scrape new dramas daily
    $schedule->command('dramabox:scrape-database --language=en --limit=50')
             ->daily()
             ->withoutOverlapping();
             
    // Scrape multiple languages
    $languages = ['en', 'es', 'pt'];
    foreach ($languages as $lang) {
        $schedule->command("dramabox:scrape-database --language={$lang} --limit=30")
                 ->daily()
                 ->withoutOverlapping();
    }
}
```

### Testing

```bash
# Test scraped database functionality
php artisan dramabox:test-scraped --language=en

# Test specific drama
php artisan dramabox:test-scraped --language=en --book-id=123456

# Compare scraped vs live API data
php artisan dramabox:test-scraped --language=en --book-id=123456
```

## Integration with Existing Code

### Option 1: EpisodeDataService Integration

Update your `EpisodeDataService` to use scraped data first:

```php
// In EpisodeDataService.php
use App\Services\ScrapedDatabaseService;

public function getEpisodeDetails($bookId, $episodeIndex, $lang = 'en')
{
    // Try scraped database first
    if (config('drama.scraping.use_database_first')) {
        $scrapedService = new ScrapedDatabaseService($lang);
        $scrapedData = $scrapedService->getEpisodeData($bookId, $episodeIndex);
        
        if ($scrapedData) {
            return $scrapedData;
        }
    }
    
    // Fallback to existing logic
    return $this->fetchFromExistingServices($bookId, $episodeIndex, $lang);
}
```

### Option 2: Direct Usage in Livewire Components

```php
// In HomePage.php or other components
use App\Services\ScrapedDatabaseService;

public function mount($lang = 'en')
{
    $service = new ScrapedDatabaseService($lang);
    
    // Set DramaBoxService as fallback
    $service->setFallbackService(new DramaBoxService($lang));
    
    // Get data (will use scraped DB first, then fallback)
    $popularData = $service->getPopularDramas(20);
    $this->dramas = $popularData['results'];
}
```

## Architecture

### Database Schema

**scraped_dramas**
- `book_id` - Drama identifier
- `language` - Language code
- `book_name` - Drama title
- `cover` - Cover image URL
- `introduction` - Description
- `author` - Author/creator
- `chapter_count` - Total episodes
- `tags` - JSON array of tags
- `scraped_at` - When scraped

**scraped_episodes**
- `drama_id` - Foreign key to scraped_dramas
- `chapter_id` - Episode identifier
- `chapter_index` - Episode number (0-based)
- `chapter_name` - Episode title
- `video_s3_path` - S3 path if downloaded
- `video_qualities` - JSON of available qualities
- `scraped_at` - When scraped

### Service Flow

1. **ScrapedDatabaseService** checks local database
2. If data exists, return it immediately
3. If not, fall back to live API (DramaBoxService)
4. Optionally cache API results back to database

## Performance Benefits

- **Database queries**: ~1-5ms vs API calls ~100-500ms
- **No rate limits**: Query your own DB as fast as needed
- **Offline capability**: Works even if DramaBox API is down
- **Better search**: Full-text search in your database
- **Analytics**: Track user behavior on your own data

## Storage Estimates

- **Metadata only**: ~1KB per drama + 0.5KB per episode
- **With video files**: ~50-200MB per episode (depends on quality)
- **1000 dramas with 20 episodes each**: 
  - Metadata: ~11MB
  - Videos: ~1-4TB (if downloading)

## Monitoring

Monitor scraping progress with:

```bash
# Check statistics
php artisan dramabox:test-scraped --language=en

# View logs
tail -f storage/logs/laravel.log | grep -i "scraping\|scraped"
```

## Troubleshooting

### Common Issues

1. **S3 Upload Failures**: Check AWS credentials and bucket permissions
2. **Rate Limiting**: Increase `rate_limit_ms` in config
3. **Memory Issues**: Reduce `batch_size` for large videos
4. **API Changes**: Scraping command will log specific endpoint failures

### Recovery Strategies

1. **API Endpoint Fails**: Update scraping command to handle new format
2. **Video URLs Change**: Re-scrape with `--force` flag
3. **Database Corruption**: Drop tables and re-migrate, then re-scrape

## Security Considerations

- Store S3 credentials securely
- Respect DramaBox's rate limits and terms of service
- Consider legal implications of video downloading
- Monitor storage costs and implement cleanup policies

## Next Steps

1. **Start with metadata**: Run without `--videos` flag first
2. **Test integration**: Use test commands to verify data quality
3. **Monitor performance**: Compare response times before/after
4. **Scale gradually**: Start with small batches, increase as needed
5. **Implement cleanup**: Add commands to remove old data