Senior Data Engineer

Job overview

This is an exciting opportunity to join a leading health data engineering team, based at GSTT, but operating across the wider region as part of the AI Centre for Value-Based Healthcare.

We are looking for a motivated individual with excellent data engineering skills, with enthusiasm and passion to learn.

The successful applicant will use a professional tool stack (including Snowflake, SQL, dbt, Python, and Natural Language Processing/Language AI tools) for building and orchestrating data pipelines, from source systems to analyst ready marts which will be used in clinical analytics, research, population health, and to support Real-World Data functions.

The postholder will work closely with a range of key stakeholders across GSTT and other NHS Trusts (including Lewisham and Greenwich, King’s College Hospital, and others).

Essential Criteria

- Proficient in SQL for data engineering and analytics

- Experience of using dbt

- Experience of EHR data and its meaning in clinical contexts

- Experience with best development practices, CI, and tools such as git

Desirable Criteria

- Experience in Snowflake or other cloud platforms

- Proficiency in Python

- Experience of NLP / LLM usage in production settings

- Experience in data pipeline orchestration

Main duties of the job

In this role, you will be expected to develop modern data pipelines built to production standards (using Snowflake / SQL / dbt / Python) to harmonise data from highly complex transactional data, including from Cerner and Epic systems. You will be responsible for owning pipeline functions, and regularly engaging with data product end-users to update and improve functionality. Understanding of clinical data quality, best practices in testing and monitoring, and ability to build visualisation applications, are all highly desirable.

Your work will be the foundation for a wide variety of valuable use-cases, from clinical care to research, and from audit to innovation and AI projects.

You will have the opportunity to deploy Large Language Model and NLP tooling orchestration frameworks as part of standard data transformation pipelines, for extracting information from unstructured free text.

You will be supported in continuing your personal development within a friendly and highly expert team, particular in areas such as data orchestration, cloud infrastructure, and continuous integration/deployment. There will also be opportunity to mentor and supervise more junior technical staff.

Detailed job description and main responsibilities

• Developing modern data pipelines built to production standards, that harmonises data from highly complex transactional data, including from Cerner and Epic systems
• Designing and building data products that serve frequent end-user needs across clinical analytics, research, and population health
• Develop pipelines that drive value for a wide range of end-user stakeholders, including clinicians, researchers, delivery teams, and population health analysts
• To use expertise in data modelling and health data to build deep insights into health data quality, including implementing tests and real-time monitoring, and producing impactful visualisations and reports
• Lead collaborations with multi-disciplinary teams to discover and extract data from new sources
• Co-ordinate and support local analysts to ensure alignment on projects and timelines
• Develop, deploy, and maintain Large Language Model and NLP tooling orchestration frameworks as part of standard data transformation pipelines, for extracting information from unstructured free text
• Contribute to a culture of shared learning, and support capability building around engineering and analytics within the team
• To use understanding of health data to act as subject matter expert for other teams
• Help to scale exemplar product solutions across other NHS Trusts
• Conduct stakeholder presentations and help to produce materials that support engagement across the Trust and the wider region