Senior Data Engineer

We are looking for a Senior Data Engineer/Architect with extensive hands-on experience in the Databricks ecosystem and exceptional communication skills to join our flagship project: a cutting-edge Data Platform for the life sciences industry. This platform supports industry leaders such as Pfizer, Moderna, and Novartis in developing innovative RNA-based solutions, leveraging data-driven research, cloud computing, and advanced AI capabilities. This role offers a unique opportunity for an experienced data engineer to take the next step in their career, playing a key technical leadership role in the rapidly evolving world of AI-driven solutions

Responsibility

• Design, develop, and optimize data pipelines and workflows within the Databricks platform.
  

• Take part and lead architecture discussions with engineering, product managers and data scientists to implement advanced analytics solutions that drive business insights.
  

• Build, optimize, and fine-tune Databricks workflows to improve performance and reliability.
  

• Work closely with data scientists and analysts to ensure the quality of data.
  

• Ensure the integrity, accuracy, and security of data across all processing stages.
  

• Implement data ingestion from various sources into Databricks, ensuring data quality and reliability.
  

• Participate in daily Scrum ceremonies and collaborate with team members in the US and Europe with required online presence from 9 AM to 5 PM EST
  

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
  

• 10+ years of experience in the software development industry, preferably in data engineering, data warehousing or data analytics companies and teams.
  

• 5+ year of experience with the DataBricks ecosystem.
  

• Expert level of Python and Typescript.
  

• Expert level of understanding and hands-on experience with Lake House architecture.
  

• Expert level of experience with Spark/Glue and Delta tables/Iceberg.
  

• Experienced in designing and implementing complex, scalable data pipelines/ETL processes using Databricks.
  

• Skilled in cloud-based data storage and processing technologies, particularly AWS services such as S3, Step Functions, Lambda, and Airflow.
  

• Familiar with CI/CD practices, version control (Git), automated testing, and Agile environments.
  

• Experience with the Agile development process in a distributed engineering team.
  

• Ability to articulate ideas clearly, present findings persuasively, and build rapport with clients and team members.
  

• Experience working in US-led high-tech companies and startups.
  

Nice to Have

• DataBricks certifications
  

• AWS or Azure DevOps or SA certifications
  

• Knowledge of basic DevOps and MLOps principles
  

• Experience in working with Data Scientists and ML Developers
  

• Experience in management and lead developer roles in technology services companies