Data Engineer

Role Summary:

As a Senior Data & Pipeline Engineer, you will serve as the founding technical owner of the ingestion architecture and data pipeline infrastructure. This role sits directly at the intersection of high-scale web scraping and pragmatic LLM orchestration. You will own the full data lifecycle—from building resilient scraping loops to constructing automated LLM evaluation frameworks that transform raw, unstructured public data into production-ready relational schemas.

Responsibilities:

• Own Ingestion Architecture: Serve as the founding owner of the data ingestion pipeline, transitioning the platform from single-source ingestion to a massively scalable, distributed processing architecture.
• Build Resilient Scraping Systems: Design, deploy, and scale advanced web-scraping loops capable of bypassing dynamic captcha-evasion patterns to extract data from obscure financial repositories and regulatory filings.
• Optimize Pipeline Performance: Deploy modular pipeline services to drastically reduce data latency down to 5-minute processing windows.
• Orchestrate LLM Ingestion: Implement high-confidence, low-cost structured data extraction loops using LLM APIs, and construct an automated LLM evaluation framework to safely benchmark and swap underlying multi-model inference frameworks.
• Database & Schema Design: Transform unstructured data into clear, structured, and production-ready relational schemas inside Supabase (PostgreSQL).
• Drive Engineering Excellence: Maintain strict data lineage, manage background worker queues, and lay the structural foundation for future AI-agentic query interfaces.

Desired Skills:

• English B2 or above
• 5+ years of professional experience as Data Engineer or similar positions. 
• 5+ years of professional experience with Python (advanced): Python scripting applied directly to multi-tier ETL pipelines and automation.
• Proven track record building large-scale scraping systems using Scrapy, Playwright, or Selenium.
• Strong cloud engineering depth deploying data infrastructure on GCP (Cloud Run, BigQuery) or closely related AWS equivalents using Docker.
• Solid command of PostgreSQL / Supabase systems, including advanced indexing and schema design.
• Pragmatic LLM Orchestration: Practical experience using OpenAI API / Multi-Model Frameworks for high-confidence structured data extraction and parsing.
• Experience managing background worker queues and pipeline orchestrators like Apache Airflow.

Who You Are:

• Extreme individual autonomy: Capable of shipping clean, production-grade logic from abstract specs.
• Strong architectural pragmatism: Balancing API calling costs against prompt extraction accuracy.
• Highly iterative execution focus: Aligning smoothly with rapid 2-week sprint cadences.
• Clear, concise technical communication
• You help your team and peers align to the company vision and mission
• You consistently leave code and projects better than you found them