Back
Senior Data Engineer – Data Engineer Latin America
About Distillery Distillery is a software development partner that helps organizations design, build, and scale modern technology solutions. Leading companies work with us to accelerate digital initiatives, solve complex technical challenges, and bring innovative products to market faster. Working as an extension of our clients' teams, we deliver lasting business value. Distillery is committed to diversity and inclusion. We embrace a dynamic and inclusive culture where you have the opportunity to continuously learn and grow.
About the Position We are seeking a hands‑on Data Engineer to join the data engineering team of a US‑based technology client as the dedicated engineer for a set of platform migrations. In this role, you will move Java and Spark pipelines from Cloudera to AWS EMR, upgrade the remaining Cloudera workloads to the latest CDP version, and migrate orchestration from Airflow 2 to Airflow 3. Working with a small, experienced team that uses AI‑assisted development tools day to day, you will execute the technical direction set by the client's staff engineers, making sure each pipeline is migrated, tested, and running reliably in production. This is a full‑time, long‑term position with immediate start, requiring at least 5 hours of overlap with US Pacific Time.
Responsibilities
Migrate Spark batch and MapReduce pipelines written in Java from Cloudera to AWS EMR, adapting code, configuration, and business logic as needed.
Upgrade HBase‑dependent pipelines to the newest Cloudera CDP version, including Spark and Java version upgrades.
Validate each migrated pipeline end to end, covering data parity, performance, and failure handling.
Update Airflow DAGs and support the migration of orchestration from Airflow 2 to Airflow 3, ideally in the same pass as each pipeline migration.
Support a small number of streaming pipelines through the same migrations.
Build new ETL pipelines based on stakeholder requests.
Troubleshoot and resolve production issues in existing pipelines.
Document migration steps and findings so the rest of the team can reuse them.
Requirements
4+ years of experience in data engineering or backend software engineering, building and running data pipelines in production.
Strong Java skills, with solid backend engineering fundamentals (build tools, dependency management, testing, and debugging).
Hands‑on experience with Apache Spark, ideally Spark batch jobs written in Java.
Working knowledge of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive, or HBase.
Experience scheduling and maintaining pipelines with Apache Airflow.
Experience upgrading or migrating pipelines between platforms or framework versions, and validating that results remain unchanged.
Ability to work independently, raise risks early, and keep work moving without close supervision.
Professional English, both spoken and written.
Nice to Have
Experience with AWS EMR, S3, and related AWS data services.
Cloudera CDH or CDP administration or upgrade experience.
Experience migrating from Airflow 2 to Airflow 3.
Experience with Spark Structured Streaming or other streaming frameworks.
Familiarity with DynamoDB, Redis, or other alternatives to HBase.
Day‑to‑day use of AI coding tools such as Cursor or Claude.
Senior Data Engineer – Data Engineer in Katowice – pracodawca: Distillery
Distillery to doskonały pracodawca, który oferuje elastyczne godziny pracy oraz możliwość pracy zdalnej, co pozwala na dostosowanie harmonogramu do indywidualnych potrzeb. Nasza kultura opiera się na różnorodności i inkluzji, a także na ciągłym rozwoju zawodowym, co sprawia, że każdy pracownik ma szansę na rozwój swoich umiejętności w dynamicznym i innowacyjnym środowisku. Dodatkowo, oferujemy konkurencyjne wynagrodzenie oraz kompleksowy pakiet benefitów, co czyni nas atrakcyjnym miejscem pracy dla utalentowanych specjalistów.
