LinkedIn People Collector Architecture
Employee profile collection via Brightdata API - experience, education, skills, recommendations
Internal Components
Service Configuration
Replicas1
Poll Interval300s
Batch Size10 links
URL Pattern
/in/*Database Outputs
linkedin_people_profilesFull profile datalinkedin_people_profile_linksJunction tablesocial_linksVerification statusPipeline Position
Upstream
Website Scraper (LinkedIn URLs)
Downstream
Email Finder (names for patterns)
Component Breakdown
URL Queue
Reads employee LinkedIn URLs (/in/ pattern) from social_links where verified=FALSE.
Brightdata API Client
POSTs URL array to Brightdata, receives snapshot_id, polls progress endpoint (5s intervals, 120s max).
Response Parser
Normalizes to schema: name, position, company, education, experience, certifications, languages, publications.
Database Writer
UPSERT on linkedin_id. Stores JSONB arrays for experience, education, activity, recommendations.