Az álláshirdetés lejárt.
Responsibilites
- Architect, build, and maintain production-grade ELT/ETL data pipelines that stream and batch high-volume, complex datasets into Google BigQuery.
- Manage, monitor, and upgrade the Apache Airflow or Cloud Composer platform.
- Write clean, modular, and maintainable Python DAGs, implement custom operators and hooks, and optimize cluster performance.
- Maintain and optimize core Google Cloud Platform data services, including BigQuery, Cloud Storage, Cloud Functions or Cloud Run, Pub/Sub, and IAM.
- Implement advanced BigQuery performance tuning, including partitioning, clustering, slot management, and query cost control.
- Develop high-performance Python scripts, API connectors, and automation tools for data extraction, schema validation, and third-party data integrations.
- Enforce data quality frameworks, monitoring, and automated alerting across ingestion pipelines to minimize data loss and latency.
- Maintain infrastructure-as-code and CI/CD deployment pipelines for data workloads using Git.
- Assist in architectural reviews and mentor junior engineers on data warehousing best practices.
Requirements
- Software Engineering Experience: 4–7+ years of professional software development experience, with at least 3+ years focused heavily on data warehousing, cloud data pipelines, and data platform operations.
- Advanced Python Mastery: Production-grade Python development skills for data processing, custom pipeline tooling, and API integrations.
- Airflow Platform Operations: Hands-on experience with Apache Airflow (Cloud Composer preferred), including DAG architecture, troubleshooting, environment configuration, and platform version upgrades.
- GCP Ecosystem Expertise: Deep experience with the Google Cloud analytics suite—specifically BigQuery, Cloud Storage (GCS), Cloud Composer, Pub/Sub, and Cloud Functions/Run.
- BigQuery Optimization: Advanced understanding of BigQuery architecture, DDL/DML, partitioning, clustering, slot allocation, and query cost-control strategies.
- DevOps & Version Control: Proficient with Git, CI/CD pipelines, and modern software development workflows.
Nice to have
- GCP Certification: Google Cloud Certified Professional Data Engineer (highly preferred).
- High-Volume Data Experience: Experience working with multi-terabyte or billion-row datasets, unstructured web-scraped data, or streaming architectures.
- Security & Compliance: Familiarity with government compliance frameworks, IAM policy design, and data encryption standards.