Lead Data Engineer - Python
7+ years
Full-time (40h)
Finance
Full Remote
Python
AWS
SQL
NoSQL
Apache Spark
Requirements
Must-haves
- 7+ years of experience in data engineering
- Leadership experience
- AWS experience
- Apache Spark experience
- Experience building scalable data pipelines and ETL processes from scratch
- Strong proficiency with Python
- Proficiency with SQL and relational database technologies
- Expertise in distributed systems and data processing frameworks
- Expertise in data lake and cloud computing platforms
- Deep knowledge of data modeling and data warehousing concepts
- Familiarity with data governance, access controls, security, and compliance principles
- Ability to optimize data pipelines for performance, scalability, and reliability
- Strong problem-solving and analytical skills for complex data engineering challenges
- Excellent communication skills in both spoken and written English
- Bachelor's Degree in Computer Engineering, Computer Science, or equivalent
Nice-to-haves
- Experience with real-time data processing and streaming frameworks
- Knowledge of modern data lake house technologies
- Experience with Docker and Kubernetes
- Experience with Terraform, Airflow, Pandas, PySpark
- Experience with NoSQL databases
- Experience with data visualization tools and data exploration techniques
- Actively participating in the data engineering community
(e.g. making contributions to open-source data engineering projects)
What you will work on
- Develop effective data pipelines and ETL processes for data lake integration
- Optimize data infrastructure considering data volume, velocity, and variety
- Ensure data architecture performance, reliability, and scalability
- Implement data governance practices to ensure data quality, integrity, and security
- Adopt optimal engineering practices, methodologies, procedures, and technologies
- Work with cross-functional teams to develop data solutions that meet business needs
- Stay updated with data engineering trends
- Share insights with the team and organization
- Effectively communicate technical concepts, solutions, and recommendations to both technical and non-technical stakeholders