Partnering with London Borough of Newham to Develop an Enterprise Cloud Data Platform on Microsoft Azure

Description

Pre-market engagement In February 2024, we conducted pre-market engagements with suppliers recommended by Microsoft to gain insights into how other local authorities have modernised their data architecture and analytical reporting. Which phase the project is in Not started Existing team There is currently a data team consisting of four members: the Head of Data, two Data Analysts and one Data Engineer. Additionally, there are BI and data analysts across various service areas within the Council. At present, our IT and data infrastructure are managed as a shared service between Newham and another council. However, both boroughs are in the process of separating their ICT departments and disaggregating shared infrastructure. In this context, you will be expected to work closely with an interim Managed Service Provider to ensure that the appropriate services are configured within the Azure tenant to support the development of the new data platform. Address where the work will be done Much of the work can be delivered remotely. Where there is a requirement to visit the Local Authority office this will be at Newham Dockside, 1000 Dockside Rd, London E16 2QU. Working arrangements The supplier staff will keep Newham Council Data team working hours in order to maximise collaboration, efficiencies and knowledge transfer. Remote working, communicating with Newham Council staff via Microsoft teams meetings, calls and messages. No scope for expenses to be provided as part of this project. Provide more information about your security requirements: Baseline Personnel Security Standard (BPSS) Latest start date 2025-01-06 Enter the expected contract length: 2 years Extension period: 1 year Are you prepared to show your budget details?: Yes Indicative maximum: 240000 Confirm if you require a contracted out service or supply of resource Contracted out service: the off-payroll rules do not apply Summary of work London Borough of Newham aims to lead in digital experiences for residents, staff and partners by utilising data and technology to enhance council services. To achieve this, the Council requires leveraging its Azure tenant and establishing a centralised, modern data platform. The project is fundamentally about transforming our relationship with data, creating quick and easy access to organisational information and improving our analytical reporting and transformation capabilities. The Council collects vast amounts of data relating to our residents and the services we provide. However, this data is currently held in multiple legacy systems across the organisation, with limited capability in our existing data infrastructure to access and deliver it at an organisational level. This creates several challenges for stakeholders needing this data to conduct business as usual, resulting in time-consuming, repetitive and inefficient manual processes. The new data platform must be at an enterprise level, providing the capability to bring together data from a large and complex legacy data estate in use at the Council. The key goals of the project will be: Pre-requisite configuration of Azure services within our existing Azure tenant as required for a public cloud platform. Although this is an optional requirement at this stage to be confirmed by the Council, the partner should have expertise in Azure infrastructure and be able to offer specialised knowledge and resources to enable the delivery of a high-quality data platform. Any specific configuration of Azure services needed will be confirmed at a later stage. Development and implementation of a Microsoft-based centralised cloud data analytical platform using Microsoft Fabric, Data Factory or a hybrid architecture with full compatibility with Power BI, our chosen BI solution. The proposed solution should also be future-proof and scalable to enable innovative technologies such as AI to be seamlessly embedded at a later stage. Data integration from source systems which will involve a combination of structured and unstructured data from our on premise systems (Oracle, SQL Server), SaaS applications (APIs) and external sources (flat files). As part of this engagement, we expect ingestion of data from these sources to build a minimum of six data products, which should include: • Finance Data • Social Care Data • Housing Data • Highways and Waste Services Data • Revenues and Benefits Data • Property Data As part of the development of these products, there will be a need to review our existing suite of reports driven by the legacy Data Warehouse to understand what data models are needed, either at an organisational or service level, to unify our analytical reporting and reduce reliance on the current Data Warehouse. A Master Data Management (MDM) data-matching exercise must also be undertaken during the transformation phase to ensure that data from disparate systems is reviewed and data quality issues are resolved to ensure that only the most current and accurate data is displayed. This should allow the Council to build and understand a single view of residents, households, and properties and identify any additional data sources required to create a complete view. The next goal is to configure Azure Purview to classify, catalogue, assign responsibilities, and track the lineage of data assets. Throughout the development of the data platform, there must be strict adherence to organisational data governance policies to ensure data is managed effectively, securely and in compliance with all relevant regulations. This includes conducting a Data Protection Impact Assessment (DPIA) and other security assessments for data processing activities. We also want to enable notebooks in Azure to create real-time insights into our resident data, understand trends and predict issues before they occur. Please note there is an existing small data team with whom the partner is expected to collaborate to develop their in-house capabilities and capacity to take ownership of both development and support tasks. The final requirement is to manage the new data platform for the first year, after which it will be handed over to an internal managed service provider in late 2025. Please note that there may be a need to migrate this platform to a new Azure tenant, which will be confirmed at a later stage. Where the supplied staff will work No specific location (for example they can work remotely) Who the organisation using the products or services is London Borough of Newham Why the work is being done London Borough of Newham is in the early stages of developing a data strategy as part of its Data Modernisation programme. An enterprise level cloud-based data platform is a key component of this wider strategy. The Council is data rich and aims to provide a self-service approach for its stakeholders to forecast business needs, understand performance, identify issues and set priorities. Better, well-managed, trusted and easy-to-use data will enable us to provide the best possible services to those who rely on us, save money, help our residents, provide more support and better target our resources. Currently, our data infrastructure relies on an on premise data warehouse running SQL Server 2016, which serves as our core analytics system. This warehouse consolidates data from over 62 different source systems, including the matching of people, properties and businesses. While this data is accessible to staff via enquiry screens, the underlying technology is outdated and lacks the scalability and flexibility needed to handle the increasing volume and variety of data generated by the Council. Additionally, there are several on premise data marts designed for single-source data storage, supported by legacy SSIS. These data marts are challenging to maintain and often suffer from degradation, leading to operational inefficiencies and unreliable data outputs. Consequently, maintaining and upgrading these older systems presents significant challenges and limits our ability to integrate new data sources and adopt modern analytics tools effectively. The Council aims to transition to a modern cloud architecture that capitalises on advanced features for data processing and analytics. This move will provide flexible structures for organising and interpreting data that can adapt to changes in source systems, suitable connectors facilitate the integration of data from different sources, optimise data storage and enhance system robustness by reducing our on premise server footprint. By adopting this cloud-based approach, the Council wants to remain agile and responsive to evolving organisational needs and technological advancements. Additionally, this transition should help us address current issues with duplicate records, which vary in consistency and accuracy often leading to potentially incorrect and unreliable information that requires manual intervention. The shift to the cloud will also strengthen our security measures, improve our ability to protect sensitive data, and ensure compliance with relevant regulations. Enhancing our approach to data will lead to better resident-facing services, support more informed decision-making for both policy and operational matters and deliver insights that add value and drive efficiencies. The Council aims to improve its ability to use and reuse data effectively, particularly through initiatives like “Tell Us Once” services for our residents and communities and strengthen our business intelligence capabilities at an organisational level to drive the necessary transformation. To achieve these goals, we aim to have a partner in place by January 2025 to begin mobilising and to have the core enterprise data platform configured by April 2025. Full integration of data sources is expected to be completed over the 2025/2026 financial year. The business problem you need to solve Newham Council is collecting increasing volumes of data daily but our current architecture is outdated and lacks the scalability and flexibility needed to manage this growing volume and variety of information effectively. As our business environment rapidly changes, our legacy system is unable to adapt to shifts in operations, resulting in highly manual, time-consuming data analytics processes with limited AI and ML capabilities. Our stakeholders lack a clear view of the data we hold and there is low confidence in the quality and reliability of the available information. Our aim is to address these data silos and eliminate inefficiencies in how we manage and utilise data. We seek to transition to a modern cloud data platform that reduces maintenance costs, provides a flexible architecture to meet the Council's future needs and enhances productivity by allowing us to focus on innovation and growth. This project fundamentally seeks to transform how we work with data across all levels of the organisation. We intend to create a centralised, cloud-based data repository that offers quick and secure access to well-managed data across all council services, while maintaining top-tier security measures that are continuously updated to address potential threats proactively. We are looking for a solution that addresses data quality issues to detect errors or inconsistencies at the point of entry and help us correct problems with the source data, reducing our reliance on time-consuming manual corrections. Additionally, we aim to improve repetitive tasks related to data extraction and transformation from legacy data marts, which currently cause inefficiencies and errors. The new data platform should incorporate modern technologies and tools, ranging from advanced analytics using Power BI to AI solutions and predictive algorithms. We require an organised approach to governing, protecting and managing data effectively, resulting in a more efficient, scalable and reliable data management system that enhances analytical capabilities, supports informed decision-making, and integrates seamlessly with various technologies while allowing data sharing with key partners. Building the technical capability of our internal data team to effectively manage and utilise the Azure platform is crucial. Currently, our IT and data infrastructure are managed as a shared service between Newham and another council. However, both boroughs are in the process of separating their ICT departments and disaggregating the shared infrastructure. This creates additional challenges, requiring us to navigate the complexities and uncertainties around setting up Azure services. You will be expected to work closely with an interim Managed Service Provider to ensure that the correct services are configured within the Azure tenant to support the development of the new data platform. First user type: Data Analysts First user type: Performance leads First user type: Senior leadership First user type: Chief officers First user type: Council officers First user type: Elected members First user type: Azure administrators Enter more details about this user type: Data Analysts across the organisation need to access data from our multiple legacy systems and various sources and ensure it is in a usable format. They require instant and continuous access to this data, in its most up-to-date availability to produce analytical reports for their various service requirements. Access to data from systems managed by other services will allow analysts to produce detailed reports and visualisation on relatable data to provide a clear and full picture to the effectiveness of the services we provide to our citizens. Data analysts require access to good quality data to ensure that their focus can be on delivering insights into our data as opposed to time-consuming and manual transformation of the data to prepare it for analytical use. Enter more details about this user type: View live dashboards, which are fed by constant streams of data from our various systems, to gain insights to council performance in relation to defined metrics and performance indicators. The ability to filter dashboards to view historic data and highlight potential emerging trends in our performance data will allow performance leads to make data-driven decisions and address potential issues before they arise. Enter more details about this user type: Senior leadership must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Senior leadership to make data-driven decisions. Enter more details about this user type: Chief officers must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. This will enable Chief officers to make data-driven decisions. Enter more details about this user type: View live dashboards, which are fed by constant streams of data from our various systems, to obtain a single view of our citizens, combining data from multiple sources to visualise all data we hold on a citizen. This will allow for efficiencies in multiple processes, through the removal of manual data matching and sourcing of relatable data from other services and teams. Confidence in the data platform’s ability to present real-time data ensures that any decision made with citizen data is done so well-informed. Council officers should also have access to relevant data sources, through managed-access levels, to support a self-service first approach to data analytics within the council. Enter more details about this user type: Elected members must be able to interface with detailed and focused reporting fed by up-to-date, good quality data from the proposed solution to allow them to gain valuable insight into our data, through the identification or historical trends and potential future trends through predictive analytics. Having access to these interactive dashboards will allow elected members clear and concise access to relevant data to support decision making. Enter more details about this user type: Azure administrators must be involved in the development of this solution to understand the resources enabled and their configuration. Defined roles to grant and revoke access to specific datasets must be granted to these users to allow for management of the solution beyond the end of the project.