YouTube Scraper Architecture
YouTube Data API v3 channel and video retrieval with quota-aware batch processing
Internal Components
Service Configuration
Replicas1
Daily Quota10,000 units
Batch Size50 channels/call
APIYouTube Data API v3
Database Outputs
youtube_profilesChannel data + videosyoutube_profile_linksJunction tableyoutube_profile_snapshotsTrend trackingPipeline Position
Upstream
Website Scraper (YouTube URLs)
Downstream
Database (social_profiles)
Component Breakdown
URL Resolver
Handles /channel/, /@handle, /c/, /user/ URL patterns. Resolves custom handles to channel IDs.
Batch Fetcher
Groups up to 50 channels per API call (1 quota unit = 50 channels). Theoretical max: 500K channels/day.
Video Fetcher
Fetches up to 50 latest videos per channel. Video merging preserves historical data.
Quota Manager
Tracks 10,000 daily units (resets midnight PT). Auto-refresh mode when queue empty.