About The Role In this role, you will help transform a data platform that currently relies on Spark and Python, orchestrated with Airflow DAGs and stored in Apache Iceberg, into a more modular target architecture centered on dbt and Lakekeeper.
Responsibilities
Support the migration of the Agora data platform to a cloud-first architecture
Develop and maintain data transformation logic using dbt as the central business logic layer
Refactor and consolidate logic currently distributed across Spark, Python, and Airflow into a more standardized dbt-based approach
Work with Apache Iceberg as the core storage layer in both current and target architecture
Contribute to the implementation of an open and modular platform built around Lakekeeper
Support data ingestion workflows across Starburst, Microsoft Fabric, and Snowflake
Help build and maintain scheduling and orchestration processes using dbt and/or Microsoft Fabric
Enable and improve data quality, lineage, documentation, and CI/CD practices across the platform
Collaborate on the transition of the semantic layer from Dremio to Starburst
Support semantic modeling for analytics using Iceberg views stored in Lakekeeper
Work closely with cross-functional stakeholders to reduce operational complexity and improve platform maintainability
Contribute to the development of a flexible data foundation prepared for future analytics and AI-driven use cases
Requirements
Strong experience in data engineering or analytics engineering
Hands-on experience with dbt
Solid knowledge of SQL and SQL-based transformation design
Experience with modern data platform and warehouse/lakehouse concepts
Practical experience with Apache Iceberg
Experience with at least some of the following technologies: Spark, Python, Airflow, Starburst, Microsoft Fabric, Snowflake
Understanding of data modeling, semantic layers, and analytics-focused data design
Experience with CI/CD, testing, and data quality practices in data platforms
Familiarity with lineage and technical documentation standards
Ability to work in an open, modular architecture with separated storage and compute principles
Understanding of cloud-first platform migration challenges and best practices
Strong collaboration and communication skills
Nice to have
Experience with Lakekeeper
Experience migrating from legacy or fragmented data transformation patterns to dbt-centered architectures
Familiarity with Dremio and migration to Starburst
Experience with ODBC/JDBC-based BI connectivity
Exposure to AI-ready data platform design principles
SoftServe is an equal opportunity employer. Qualified applicants will receive consideration regardless of race, color, ancestry, ethnicity, national origin, religion, sex, sexual orientation, gender identity or expression, age, citizenship, disability, health condition, marital or family status, veteran status, or any other characteristic protected by applicable law.
Lead DBT Engineer in Katowice – pracodawca: SoftServe Inc.
SoftServe Inc. oferuje elastyczne opcje pracy oraz możliwość współpracy z zespołem ekspertów w dziedzinie AI i Data Science. Pracownicy mają szansę na rozwój zawodowy poprzez mentoring oraz udział w innowacyjnych projektach związanych z generatywną AI i systemami wieloagentowymi. Firma stawia na budowanie silnych relacji z klientami, co sprzyja dynamicznemu rozwojowi kariery.
