Data Engineer (AI-driven Data Platform)

Overview

About The Role: Build and improve large-scale data platforms that support an AI-powered SaaS product. Work with tens of terabytes of enterprise data used by real business users. Design data foundations for future AI / data science use cases. Work on a proven product with real users and real impact.

Responsibilities

• Design, build, and improve big data pipelines for a SaaS + database product
• Develop and maintain data pipelines for: Web data crawling; Data cleansing and enrichment; Build data quality monitoring and validation systems; Improve scalability, performance, and cost efficiency of data infrastructure; Design data platforms and data quality foundations with future analytics and AI usage in mind

Tech Stack

Data Ingestion

• Python (Scrapy)
• AWS Fargate (Amazon ECS)
• Amazon Managed Workflows for Apache Airflow

Data Storage

• A...