Data Engineer
APN Consulting, Inc. is a progressive IT staffing and services company offering innovative business solutions to improve client business outcomes. We focus on high impact technology solutions in ServiceNow, Fullstack, Cloud & Data, and AI / ML. Due to our globally expanding service offerings we are seeking top-talent to join our teams and grow with us. Direct Client Requirement Job Title - Data Engineer Location - Remote Employment Type - Full- time Experience - 6+ years in data engineering, ETL/ELT, analytics engineering, data warehouse development, or big data pipelines; minimum 4 years on GCP cloud data platforms preferred. Common Job description - Strong SQL and Python. - Data modelling, warehousing concepts, partitions, clustering, indexing, optimisation. - ETL/ELT pipeline design and production support. - Batch and streaming pipeline fundamentals. - Orchestration using Airflow, Composer, Control-M, ADF, dbt, or equivalent. - Big data processing using Spark, PySpark, Beam, Dataflow, Dataproc, Databricks, EMR, or equivalent. - Cloud storage and data lake concepts. - Data quality, reconciliation, schema validation, lineage, monitoring, and alerting. - CI/CD for data pipelines and code versioning. - Ability to explain scale scenarios, not just tool definitions. Detailed Job description Generic Skills (Must Have) - Python and advanced SQL. - Data quality checks, reconciliation, alerting, retries, and backfills. - Production pipeline monitoring and troubleshooting. GCP Skills (Must Have) - Mandatory: BigQuery: datasets, tables, views, partitioning, clustering, SQL optimisation, cost awareness. - Option 1: Dataflow / Apache Beam or equivalent streaming/batch framework. - Option 2: Dataproc / PySpark or strong Spark experience. - Pub/Sub or equivalent event streaming. - Cloud Composer / Airflow DAG development. Nice to have (Trainable) - Dataform, dbt, CI/CD for data. - BigQuery ML. - Dataflow templates. - Looker / LookML basics. - Terraform for data infrastructure. We are committed to fostering a diverse, inclusive, and equitable workplace where individuals from all backgrounds feel valued and empowered to contribute their unique perspectives. We strongly encourage applications from candidates of all genders, races, ethnicities, abilities, and experiences to join our team and help us build a culture of belonging.