How to Apply:

Please submit your application to [email protected]

Job Title: Senior Data Engineer

Location: Bangalore, India

Department: IT

Position Summary:

Penn Engineering is seeking a skilled and motivated Senior Data Engineer who designs, builds and operates core components of the PEM Data Lake and leads delivery of complex data pipelines and platform improvements. This is a technical leadership role and will own end-to-end solutions from source ingestion (including CDC from JDE) through the refinement layers to governed delivery in Qlik and to partners set engineering standards, mentor Data Engineers and partner with stakeholders to translate business needs into scalable, cost-optimized architectures. The role mirrors the profile of our most effective platform engineers, who combine deep AWS and SQL/Python expertise with strong ownership and the ability to lead others.

Key Responsibility

• Design and implement scalable ingestion and transformation architectures across the PEM Data Lake including the Raw and Refined1–Refined4 layers and cross-system models (e.g., SalesShipments-ThruMargin).
• Lead data integration initiatives — Qlik Data Integration with an open lakehouse target, change data capture (CDC) from JDE (DB2/400), and AWS DMS for CEBOS and other systems — to move data faster and more reliably.
• Own orchestration and reliability: build robust orchestration pipelines with structured entry/exit points, automated rollback, failover and disaster-recovery mechanisms to drive down report-generation failure rates.
• Optimize cost and performance across S3, Glue, Athena, Redshift and the DuckDB/dbt transformation layer; evaluate and pilot new technologies (e.g., S3 Tables, SageMaker Lakehouse Catalog, Iceberg/Athena endpoints for partner data sharing).
• Drive backend storage decisions for high-volume machine/log data — weighing TimescaleDB, S3 Tables + Athena/Iceberg and hybrid approaches against volume, growth, retention, latency and cross-source join needs — and document the rationale in architecture decision records (ADRs).
• Lead the IIoT analytics stack where assigned — reducing machine-to-visualization latency, TimescaleDB for near real-time, and dbt/Bitbucket pipelines feeding Grafana.
• Build and operate the manufacturing data pipeline — edge capture from PLCs via KEPServerEX, Mosquitto MQTT and Node-RED payload refinement at the edge gateway, and ingestion through AWS IoT Core into TimescaleDB — to deliver OEE and machine-performance analytics across PEM's North America and Europe plants.
• Implement and enforce data governance: zero-trust / least-privilege access, attribute-based access controls, AWS DataZone and Lake Formation for fine-grained access, and Qlik Section Access.
• Establish engineering standards and reusable patterns; perform code and design reviews; and run regular knowledge-sharing sessions (demos, guild meetings, write-ups) that keep the team aligned on tools, techniques and best practices.
• Mentor and technically guide Data Engineers; provide estimates, scope work, and manage delivery timelines for data projects.
• Partner with data owners, analysts, data scientists and marketing/operations stakeholders to gather requirements and deliver measurable business value.
• Prepare manufacturing time-series datasets for AI/ML use cases — predictive maintenance, anomaly detection and OEE forecasting — and partner with data scientists to operationalize models on MES West machine data.
• Evaluate and integrate AI tooling for manufacturing data — including MCP servers (e.g., SQL Server and PostgreSQL) for agentic, natural-language access to MES data — and prepare governed time-series datasets that data scientists and AI tools can query reliably for analytics and predictive use cases.
• Champion AI-embedded development workflows (e.g., Kiro, Amazon Q, agentic tooling), establishing effective practices and feedback loops for the team.

Requirements:

• Bachelor’s degree in computer science, Engineering, Information Systems or a related technical field.
• 5–8 years of data engineering experience, including hands-on delivery of cloud data platforms (data lake / data warehouse / lakehouse).
• Advanced SQL and strong Python/PySpark; proven ability to build performant, production-grade pipelines.
• Deep, hands-on AWS data experience — S3, Glue, Athena, Redshift, Lambda, plus SNS/SQS/EventBridge and Lake Formation.
• Strong data modeling expertise — dimensional and SCD modeling, facts/dimensions, and designing analytics-ready (AI-ready) data models.
• Experience with ELT/ETL tooling (dbt and/or Informatica/IICS/SSIS/Fivetran) and modern transformation engines (DuckDB or similar).
• Experience with workflow orchestration (Apache Airflow, AWS EventBridge, or similar) and CI/CD using Git/Bitbucket pipelines.
• Demonstrated ownership of end-to-end solutions and the ability to mentor engineers and set technical standards.

Preferred Qualifications:

• Experience with change data capture (CDC), AWS DMS, or Qlik Talend / Qlik Data Integration.
• Experience with Qlik Sense / Qlik Cloud data architecture (QVD strategy, Section Access, data reduction).
• Experience with data governance and access frameworks — AWS DataZone, Lake Formation, zero-trust / attribute-based access control.
• Familiarity with identity and access management (IAM) — AWS IAM, role-based and attribute-based access control, SSO / identity federation (e.g., Azure AD, Okta, AWS IAM Identity Center).
• Manufacturing / ERP domain experience (JDE, SAP, IQMS, CEBOS) and exposure to IIoT / time-series data (TimescaleDB, MQTT, Node-RED).
• Multi-cloud familiarity (GCP / Azure) and cloud cost-optimization experience.
• Experience with DevOps practices — infrastructure-as-code (Terraform / CloudFormation), container orchestration (Docker / ECS / EKS), and CI/CD pipeline automation.
• Pre-sales / RFP / POC and project estimation experience.
• Relevant certifications (e.g., AWS Solutions Architect Professional, AWS/Azure/GCP Data Engineer).

Key Technologies:

AWS S3, Glue, Athena, Redshift, Lambda, SNS/SQS, EventBridge, Lake Formation, DataZone · Qlik Cloud / Qlik Data Integration / Qlik Talend · dbt · DuckDB · Python / PySpark / SQL · CDC / AWS DMS · Iceberg · TimescaleDB / Grafana · KEPServerEX / MQTT (Mosquitto) / Node-RED · Git / Bitbucket · JDE (DB2/400), Oracle, SQL Server.