Data Engineer

Position Responsibilities

· Work in a dedicated data reporting and analytics team building a world-class data platform to produce data-driven insights for Transact Campus and our clients

· Analyze, interpret and orchestrate complex data across disparate sources comprising unstructured, semi-structured and structured datasets in streaming and batch modes

· Design and develop real-time data pipelines using the latest Databricks and Delta Lake Azure cloud technologies

· Collaborate with data consumers (reporting, analysis, or data science) to provide metrics that meet their needs

· Contribute to standards for data producers streaming data into the Lakehouse

· Test commercial software products using both manual and automated testing processes

· Support the application lifecycle during QA, UAT, and post-release phases

· Comply with and contribute to consistent development guidelines (coding, change control, build, versioning)

· Participate in peer code reviews

Required Skills

· Bachelor’s degree in Computer Science, IT or related field OR equivalent related work experience – preferably with a focus on Data Analytics

· 5+ years of experience in software enterprise-level Data Engineering

· Expertise in big data workloads

· Experience with data lakes and scale-out processing

· Relational database design and best practices

· Hands-on experience designing and developing Spark data pipelines

· Strong SQL and Python skills

· Understanding of cloud-based big data workloads

· Experience with ETL/ELT patterns, preferably using Databricks jobs

· Excellent technical documentation skills

· Experience with data lakes (HDFS, Azure Data Lake or AWS S3)

· Experience with source code management systems such as Git/TFS/SVN

· Experience working in Agile teams (Scrum, XP, Kanban)

· Ability to present ideas and insights to business stakeholders

· Fluency in written and spoken English

Preferred Skills

· Good understanding of Azure Data Services (Azure Databricks, Azure Data Factory, Azure Data Lake Gen 2)

· Experience with Databricks Delta Lake, Delta Sharing, and Delta Live Tables

· Experience with Spark Structured Streaming

· Experience with NoSQL databases

· Experience with Infrastructure as Code technologies such as Terraform or ARM

· Experience in Data Science and ML methodologies

· Experience with Azure services for streaming data (EventHub, EventGrid)

· Understanding of Data strategy including Data Governance and Data managemen