[Remote] Platform Data Engineering Lead
Note: The job is a remote job and is open to candidates in USA. Boston Insights is an innovative startup creating competitive advantage for pharmaceutical companies by unlocking clinical supply chain data and enabling end-to-end visibility. The Platform Data Engineering Lead will lead the design, development, governance, automation, and maintenance of scalable multi-cloud data platforms and pipelines, including Microsoft Fabric AI agents, to support pharmaceutical supply chain analytics and optimization.
Responsibilities
- Lead the integration strategy for ingesting and harmonizing data from external sources, including AWS-based data lakes/warehouses (such as S3, Redshift) and SAP systems
- Design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake / MS Fabric OneLake
- Design reusable metadata-driven ingestion framework for rapid on-boarding of new datasets
- Hands-on experience in metadata driven pipelines with event-based ingestion
- Automate ETL processes for data extraction, transformation, and loading across hybrid and multi-cloud environments
- Build and maintain real-time and batch data integration workflows between Azure, AWS, and on-premises sources
- Design and implement data lake and data warehouse solutions on Azure platform
- Establish data governance frameworks and ensure data quality across all pipelines
- Implement security best practices for handling sensitive pharmaceutical data
- Create and maintain data documentation and lineage tracking
- Define and enforce data governance frameworks: data cataloging, lineage, quality, privacy, and compliance
- Implement robust data validation, cleansing, and monitoring systems to ensure accuracy and reliability
- Support security standards through effective data management practices
- Design and develop Microsoft Fabric-based AI agents leveraging OneLake, Fabric Data Agents, Semantic Models, Lakehouses, and enterprise data governance capabilities to support metadata discovery, lineage interpretation, policy validation, anomaly triage, and data stewardship workflows
- Develop data automation tools and reusable components using Python, PySpark (and other relevant frameworks/languages)
- Enable end-to-end process automation for data ingestion, processing, and reporting
- Implement CI/CD processes for data solutions, including testing, monitoring, and alerting
- Collaborate with data scientists and analysts to support advanced analytics
- Build data models that enable risk assessment and supply chain optimization
- Develop APIs and data services to support front-end applications
- Create monitoring and alerting systems for data pipeline health
- Build agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows
- Partner with supply chain, analytics, and business stakeholders to understand business requirements and translate them into scalable technical solutions
- Collaborate with SAP functional and technical teams to optimize data extraction and synchronization
Skills
- • 5+ years of professional data engineering experience
- • Data Integration: Proven track record integrating data from AWS services (S3, Redshift, Glue, etc.) into Azure or other cloud environments
- • Azure Data Services: Expert-level knowledge of Azure Data Factory, Azure Synapse Analytics, Data Bricks, Azure Data Lake Storage, Azure SQL Database, and Apache Spark
- • Database Technologies: Strong knowledge of both relational (SQL Server, PostgreSQL) and NoSQL (Cosmos DB) databases
- • Programming Languages: Proficiency in Python, SQL, PySpark, and PowerShell for data automation and wrangling
- Hands-on experience developing AI agents using Microsoft Fabric infrastructure and services, including Fabric Data Agents, Lakehouses, Semantic Models, OneLake, Notebooks, and Data Factory
- • Building agents that support data cataloging, metadata enrichment, lineage interpretation, and data stewardship workflows
- • Designing agent workflows, retrieval patterns, governance controls, and security guardrails for enterprise pharmaceutical and supply chain data
- • Integrating Fabric-based agents with data platforms, APIs, notebooks, pipelines, and automation frameworks to streamline data quality reviews, compliance checks, and anomaly detection
- • Evaluating agent responses for accuracy, traceability, governance compliance, security, and business relevance
- • Version Control: Proficiency with Git and Azure DevOps
- • Hands-on experience with SAP data models and integrating SAP data with Azure data lake
- • Strong problem-solving and analytical thinking abilities
- • Excellent communication skills with ability to explain technical concepts to non-technical stakeholders
- • Experience with Agile development methodologies
- • Attention to detail and commitment to data quality
- • Experience with API-based data integration for cloud and enterprise applications
- • Experience with Infrastructure as Code (ARM templates, Terraform)
- • Familiarity with data quality tools, metadata management, and automated data lineage tracking
- • Knowledge of containerization (Docker, Kubernetes) for data automation workflows
- • Knowledge of machine learning pipelines and MLOps practices
- • Experience with data visualization tool, Power BI
Benefits
- Professional development training and certifications.
Company Overview