Data Engineer

Summary of the Role

We are looking for a Data Engineer to own the technical implementation of data ingestion pipelines and production data marts. You will ingest messy external data (EDI files, flat files, vendor extracts and REST APIs), turn mapping specifications into tested dbt transformations, build and schedule orchestration jobs in Python and create Snowflake staging objects. In this role, work is done only when the output reconciles 100% back to the raw source, with full documentation and automated tests.

Desired Skills

• Full-Time availability
• Advanced English (C1) or above
• 5+ years of professional experience in data engineering, analytics engineering or software engineering with a strong data focus
• Advanced SQL: High-level SQL in a cloud data warehouse (Snowflake strongly preferred), including CTEs, window functions, deduplication, conditional logic, stored procedures and query tuning.
• Production dbt: Proven track record building, testing, documenting and maintaining complex dbt projects in production.
• Python for Data Pipelines: Working proficiency in Python for file manipulation, API data extraction and workflow orchestration.
• Workflow Orchestration: Hands-on experience with modern DAG orchestrators (Dagster preferred, or Airflow / Prefect).
• Messy External Ingestion: Demonstrated experience ingesting inconsistent external data, such as unpredictable file deliveries, malformed layouts and vendor schema changes.
• AI-Assisted Development: Daily, practical use of AI development tools (e.g., Claude, Claude Code) to speed up writing, testing and debugging.
• Timezone Overlap: Availability for substantial daily overlap with US Central Time (CST).

Responsibilities

• Ingestion & Pipeline Architecture: Design and maintain resilient ingestion workflows for diverse external sources (EDI feeds, flat files, third-party vendor extracts, secure email drops and REST APIs), handling dropouts, late records and sudden schema changes gracefully.
• Snowflake Engineering: Build the Snowflake objects that support ingestion, including stored procedures, views, staging tables and incremental batch loading.
• Orchestration: Author and schedule Python-based Dagster jobs, configuring dependencies, retries, alerts and message-based integration queues (topics/subscriptions).
• dbt Development & Onboarding: Translate functional mapping specs into modular dbt models across claims, disputes and invoice data marts, writing tests and documentation for every transformation and pushing back when specs conflict with the raw data.
• Data Quality & Reconciliation: Own pipelines end-to-end: profile source data, build automated quality checks (freshness, null rates, duplicates, row counts) and provide before-and-after reconciliation evidence before every production release.
• AI-Assisted Engineering: Use AI tools daily to write and refactor SQL and Python, scaffold dbt tests, draft runbooks and support code reviews, always checking AI output against real data.

Nice-to-Haves

• Healthcare Data: Parsing healthcare EDI formats (X12 837 claims and 835 remittances), and experience with healthcare revenue cycle management or No Surprises Act / IDR data.
• Observability: Building automated data observability and pipeline alerting systems.
• Async Python: Concurrent, rate-limited API consumers and batch workers with asyncio, aiohttp or HTTPX.
• Zero-to-One Platforms: Architecting production data platforms from scratch, including event-driven intake, dead-letter queues, backfills and idempotent upserts.
• Cloud Ecosystems: Azure cloud messaging services; GCP (GCS, Pub/Sub, Cloud Tasks, BigQuery, Dataform); or AWS (S3, IAM, Lambda, ECS/Glue, Step Functions).
• Data Architecture Patterns: Medallion layers (raw, standardized, curated), Delta Sharing, Apache Iceberg or Delta Lake.
• PostgreSQL: Schema design, index optimization, incremental loading and data contract enforcement.
• Web Scraping at Scale: Scrapy, Playwright, Selenium or Beautiful Soup, with concurrency, proxy rotation and anti-blocking strategies.
• LLM & AI Data Work: LLM-driven structured extraction from unstructured text, entity resolution and golden records, semantic layers or retrieval-ready pipelines; agentic coding tools (Cursor, GitHub Copilot) embedded in code review and CI/CD.
• DevOps & Tooling: Azure DevOps or GitHub CI/CD, Docker, Terraform / AWS CDK, Jira Service Management, Confluence and Power BI for output validation.
• Other Exposure: Node.js / TypeScript as a secondary language; energy, utilities or carbon domain data.

Who You Are

• Reconciliation-Obsessed: You take pride in matching numbers down to the last record and never let dropped data slide.
• Constructive Collaborator: You push back proactively on analysts and vendors when a mapping spec is ambiguous or contradicts the raw source data.
• Root-Cause Fixer: You dig into pipeline anomalies and permanently fix the underlying schema or validation issue instead of applying temporary patches.