We’re looking for a Mid Data Engineer to take ownership of our data lake — the system of record for millions of financial events every day, including bets, wallet transactions, and live odds, serving 12M+ active users. You’ll be responsible for designing and maintaining reliable data pipelines and defining how data is ingested, stored, retained, reconciled, and governed across AWS S3 and Snowflake/Databricks. Your work will ensure that Analytics, Finance, and Regulatory teams have access to accurate, consistent, and fully traceable data — with zero drift from source systems.
Location: Kraków, Poland. Hybrid — 2 days per week from the office.
What you will do: Own the lakehouse architecture: bronze/silver/gold layers, Iceberg/Delta tables, schema evolution. Land operational data via CDC streaming (Kafka, Debezium), handling late and duplicate events. Design data layout for speed and cost: partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake. Own retention and archival: storage tiering, regulatory retention, immutability, GDPR deletion. Guarantee correctness: freshness SLAs, drift detection, reconciliation against the source wallet and ledger systems. Own governance: catalog and lineage, row/column access control, PII masking, encryption, audit trails. Monitor ingestion health, data anomalies, and cloud storage/compute spend.
Requirements: Must-have: 2 years of hands-on in a production lakehouse environment. Lakehouse architecture — bronze/silver/gold layering, an open table format (Iceberg, Delta, or Hudi), schema evolution. Data layout & query optimization at TB+ scale — partitioning, compaction, file sizing, query performance on Trino/Athena/Snowflake. Cloud lakehouse/DWH in production — Snowflake, Databricks, or BigQuery. CDC & streaming ingestion — Kafka + Debezium or equivalent; late, duplicate, and out-of-order events. Strong SQL and data modeling — enough relational grounding to reason about the OLTP systems you capture from. Critical for financial ledgers. Correctness — freshness SLAs, drift detection, reconciliation against source wallet/ledger systems. Governance — catalogs, lineage, row/column access control, PII masking, retention, GDPR deletion. Cloud object storage — S3 or GCS, plus storage tiering and archival. Python and an orchestrator — Airflow or Dagster, as tools.
#J-18808-Ljbffr
Middle Data Engineer/ Data Lakehouse in Warsaw – pracodawca: CommIT
Commit to global technologii usługowej, która dynamicznie rozwija swoją obecność w regionie EMEA, oferując wyjątkowe możliwości rozwoju zawodowego w zespole GCP Cloud Account Executive. Pracując w innowacyjnym i przedsiębiorczym środowisku, będziesz miał szansę na realny wpływ na rozwój działu chmurowego oraz na budowanie długotrwałych relacji z klientami, co sprzyja nie tylko osobistemu rozwojowi, ale także przyczynia się do sukcesu całej organizacji. Oferujemy elastyczne warunki pracy, wsparcie w rozwoju kariery oraz dostęp do najnowszych technologii, co czyni nas doskonałym pracodawcą dla ambitnych profesjonalistów.
