Data Engineer

Job Title: Data Engineer Company: National Heavy Vehicle Regulator Job Category: Machine Learning Roles & Responsibilities: About the role As a Data Engineer, you’ll bring a well\-rounded understanding of data\-driven practices to support senior team members and stakeholders in enhancing regulatory processes. From uncovering insights through data profiling and analytics to building robust models and engineering solutions, you’ll play a key role in how data is governed, interpreted, and transformed into actionable intelligence. Key responsibilities of this role are: * Collaborate within a high\-performing Scrum team to deliver BAU work and Project Roadmap milestones that support the Data Science and Analytics Strategy. This includes interpreting requirements, estimating effort, and raising risks and impediments early. * Interpret business requirements, estimate workloads, prioritise tasks, and manage time in the delivery of quality solutions to meet deadlines. * Design, build, and maintain scalable data pipelines in Databricks using Python, PySpark, and Spark SQL, primarily within our metadata\-driven ingestion and transformation framework. * Develop and operate Databricks notebooks and jobs on classic compute. Optimize performance and cost through cluster configuration, cluster policies, and Spark and Delta Lake tuning. * Extend and improve the framework's configuration, patterns, and reusable components so that new data sources can be onboarded quickly and consistently. * Contribute to the platform's evolution toward Lakeflow declarative pipelines and modern Databricks capabilities. * Deliver changes through modern CI/CD practices, including source control, automated testing, peer code review, and CAB\-approved releases. * Support, maintain, and upgrade existing workflows, deployment pipelines, and environments across the full application lifecycle. * Implement and manage data security with Unity Catalog, including catalog, schema, and table permissions, row filters, column masks, service principals, and secrets management. * Identify, classify, and protect sensitive and personal (PII) data. Apply tagging, masking, and access controls in line with privacy obligations and organizational policy. * Participate in a rotating incident management roster. This includes troubleshooting and resolving issues, performing root cause analyses, configuring monitoring and alerting, and writing troubleshooting guides. * Proactively remediate and document operational incidents, project risks, and impediments where possible or otherwise surface these items as appropriate to project managers and senior team members. * Salary package of $135,995\.90* \+ superannuation* * Monthly RDO* and paid leave over Christmas closure. * Hybrid working environment – up to 2 days from home. * Up to15\.25% superannuation through a co\-contribution scheme * Leave loading 17\.5% * Wellbeing programs – discounted fitness through Fitness Passport, Discount health insurance, flu vaccine, financial programs including discounted banking products, finance information sessions. * Salary packaging options including novated lease. * Discounted IT products including Microsoft and free family password security plan. * Ongoing training and professional development support * Professional membership Please Note: We will not require any support from Recruitment Agencies at this time and kindly request no contact is made. Only direct applications will be considered. Work Location: Brisbane, Australia Availability: Full time

You'll have at least four years of data engineering experience, including significant hands\-on work with Databricks or equivalent cloud data platform. A bachelor's degree in a relevant field is desirable. You'll also be experienced with Agile Scrum, the SDLC, and data lifecycle processes. * Experience with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Lakeflow Connect, Auto Loader, and data quality expectations. * Relevant certifications including Professional or Associate Databricks Data Engineer, and Azure Certification. * Experience with data visualization tools such as Microsoft PowerBI, Tableau. * Familiarity with supporting machine learning workloads, such as MLflow. We are committed to being an inclusive, diverse and flexible workplace where differences are valued. We welcome applications from people of all backgrounds, experience and abilities. Key Skills: Machine Learning, Python Salary: AUD 135996 - 135996 per yearly