Senior Data Engineer (Healthier Futures)

Job overview

The post holder will focus on data engineering–led transformation and development work required to support delivery of the Trust’s Healthier Futures programme for the new digital hospital. This includes developing and maintaining high quality, reliable and scalable data solutions to support population health management, service transformation and improved patient outcomes. The role will ensure data from multiple clinical and operational systems is effectively integrated, modelled and made available to support insight-driven decision making.

Working within the Data Warehouse team, the post holder will design, build and optimise robust data pipelines, data models and reporting foundations, initially within the Trust’s current SQL-based data warehouse environment, while supporting the transition towards a more modern, scalable data platform, including use of SQL and established data warehousing tools, and evolving cloud-based data engineering approaches over time.

They will take lead responsibility for one or more key data domains aligned to Healthier Futures priorities, acting as a subject matter expert in data engineering, performance optimisation and data quality. The post holder will work closely with analysts, clinicians and operational leads to deliver well-governed, trusted data products aligned to agreed data architecture standards, while supporting best practice and continuous improvement across the team.

Main duties of the job

Design, build and optimise data models and ETL/data pipelines using expert T‑SQL skills, ensuring pipelines are performant, scalable and resilient.

Optimise queries and ETL performance through indexing, partitioning and other SQL Server tuning techniques, and use a metadata‑driven ETL framework to automate and standardise data flows.

Validate ETL processes after system upgrades to ensure data continuity and integrity, maintaining test suites and validation scripts.

Design and maintain dimensional models (star/snowflake schemas) to support efficient and reusable analytics, and perform data profiling to identify data quality issues, anomalies and lineage.

Liaise with service users and BI teams to understand reporting requirements and translate them into data models and data solutions.

Support advanced analytics by preparing high‑quality datasets and enabling analytical workflows where appropriate.

Promote and support automation (e.g. CI/CD pipelines), APIs and modern data platform techniques, and contribute to DevOps practices including version control and release management.

Assist with DBA‑related activities such as backup and recovery, index and statistics maintenance, capacity planning, and participate in patching and upgrades of SQL Server instances.

Detailed job description and main responsibilities

Role Duties & Responsibilities

• Support the maintenance of the data warehouse, monitor ETL and SQL Server Agent jobs, troubleshoot issues, and tune performance.
• Participate in out-of-hours support for monitoring critical processes, respond to alerts, and restore services.
• Work collaboratively across the Data Engineering team to design, build, and optimise data models and ETL/data pipelines, applying expert T-SQL skills.  Ensure pipelines are performant, scalable, and resilient.
• Optimise queries and ETL performance through indexing, partitioning, and other SQL Server tuning techniques.
• Use a metadata-driven ETL framework to automate and standardise data flows.
• Validate ETL processes after system upgrades to ensure data continuity and integrity; maintain test suites and validation scripts.
• Design and maintain dimensional models (star/snowflake schemas) to support efficient and reusable analytics.
• Perform data profiling to identify quality issues, anomalies, and lineage
• Liaise with service users and BI teams to understand reporting requirements and translate them into data models.
• Support advanced analytics by preparing high-quality datasets, building pipelines for predictive modelling, and enabling Python/R-based analytical workflows.
• Promote and support the adoption of automation (e.g., CI/CD pipelines), APIs, and modern data platform techniques.
• Contribute to DevOps practices: use DevOps tickets, repositories, version control, and branching strategies.
• Assist with DBA tasks such as backup/recovery, index/statistics maintenance, and capacity planning; ensure database health, security, and compliance.
• Participate in patching, upgrades, and version control of SQL Server instances.

Governance

• Ensure all development work complies with agreed processes, including peer review, documentation, change management, and proof-of-concepts.
• Maintain detailed documentation: data lineage, source-to-target mappings, metadata repository, standard operating procedures, and runbooks.
• Uphold data governance, architecture standards, and NHS data requirements.
• Share knowledge and good practice to reduce single points of failure and promote consistency
• Document incidents and contribute to post-incident reviews and continuous improvement.

Managerial / Leadership

• Deputise for the Data Warehouse Manager including supervising other members of the team, leading daily standups, allocating and monitoring tasks.
• Lead technical delivery for a specific source system or domain, taking ownership of its data pipeline architecture.
• Promote coding standards, continuous improvement, and a safe, compliant working environment.
• Ensure a healthy, safe and secure working environment, ensuring compliance with legal and regulatory requirements, maintaining accurate documentation and reporting any concerns.

Education /Learning

• Take responsibility for own learning and development by recognising and taking advantage of all opportunities to learn, including full participation in KSF/appraisal, supervision, action learning and by maintaining a professional/personal portfolio of learning.
• Support the training of colleagues by sharing knowledge in SQL Server, data modelling.
• Maintain awareness of advancing technologies, specialist work areas and wider NHS data initiatives.

Communications

• Communicate effectively with internal and external stakeholders, including engineers, BI developers, analysts, IT teams, clinical and operational managers, and external organisations.
• Engage in technical discussions (ETL performance, schema design, database optimisation) and translate complex information for non-technical audiences.
• Translate business requirements into technical data solutions and provide regular updates on pipeline status, risks, and improvements.
• Produce written technical reports to support informed decision-making.
• Communicate sensitive or complex information clearly, including data load delays affecting dashboards or statutory submissions.