Social Media Collectors Architecture
9 platform scrapers - 8 via Brightdata API + YouTube Data API v3
Platform Overview
Pipeline Position
Upstream
Website Scraper (social URLs)
Downstream
Email Finder (names for patterns)
Database Outputs
social_profilesPlatform-specific profile data stored per platform with junction tables linking to employees/companies
Common Pattern
APIBrightdata + YouTube
Platforms9
Poll Interval60-300s
Batch Size10 links/cycle
Shared Architecture (Brightdata Scrapers)
URL QueueBrightdata APIPoll SnapshotResponse ParserDatabase
Platform Architectures
LinkedIn People
Employee profiles - experience, education, skills
LinkedIn Company
Company pages - size, industry, specialties
Professional profiles - DACH region focus
Business and personal profiles with posts
Business page collection
X (Twitter)
Profile and tweet collection
TikTok
Business account profiles
Business pin profiles
YouTube
YouTube Data API v3 - channel & video data