Development Partner to implement a Data Platform

Description

Summary of work The aims of the project are to: 1. Establish a data platform within the Combined Authority’s Microsoft Azure tenant that is capable of consuming and sharing data internally and externally. 2. Appoint a development partner to scope and plan for the data requirements of a franchised bus network. 3. Establish a data platform as a single source of truth for all bus data, ready for managing a franchised bus network by 2027. 4. Establish a data platform as a ‘data integration hub’ for all Mass Transit data throughout the programme life cycle. 5. Have well trained analytical and technical Combined Authority staff to manage and further develop the data platform. The objective of this project are to: • Commission a development partner to lead and manage a programme of work to implement a data platform. The development partner must offer a ‘full solution’, being significantly operationally involved in service delivery. • Commission services to design and setup a secure Microsoft Data Platform (or latest Microsoft equivalent e.g. MS Fabric) in the Combined Authority Azure environment. • Commission Technical Support to the Combined Authority for the data platform for at least 3 years with an option to extend at the Combined Authority’s sole discretion for 2 years. • The budget for this work is approximately £500k. • Commission services to ‘on-board’ a wide range of data sources using various methods (API’s, manual upload, etc.) • Commission data transformation services for onboarded data, to enable the Combined Authority staff to creation of PowerBi Dashboards consuming data from the data platform. • Commission data platform training to allow Combined Authority staff to onboard, manage, catalogue, and transform data within the data platform as well as training to manage security and optimise data storage. The project has 4 key requirements: 1. A Data Platform Development Partner A data platform development partner will be required to: • Refine the scope of requirements for a data platform based on the needs of the Combined Authority in relation to bus franchising and Mass Transit and suppliers previous experience of implementing similar systems in a public sector setting. • Act in the capacity of a range of roles such as project manager and systems architect to deliver the data platform in the context of the needs of bus franchising and Mass Transit. The development partner must offer a ‘full solution’, being significantly operationally involved in service delivery. • Develop a comprehensive plan for development of the data platform based on the emerging needs data needs of Bus franchising which is likely to cover: o Passenger data – detailed information on the number of passengers boarding at stop/station/service/corridor level, demographic data of bus passengers and their travel patterns and fares data including details of ticket pricing structures, ticket types and, payment methods and channel shift. o Operational data – including vehicle tracking data for location and speed across the network (real-time and historic), schedule adherence calculated from actual departure and arrival times vs scheduled times including aspects of disruption/delay, and vehicle performance such as maintenance records, fuel efficiency and milage. o Network data – including service configuration, network performance (e.g. service efficiency, coverage and optimisation), as well as demand forecasting to estimate future demand based on demographics, population growth and historical usage. o Service quality data – including perceptions surveys and complaints, service disruptions, breakdowns and incidents and accessibility data which may influence passenger satisfaction. o Financial data – including transitions related to fare collection, subsidies, operating/maintenance expenses, and revenue generated from advertising or ancillary services, as well as budgeting and financial planning data to support resource planning, and cost-benefit analysis for future tranches of bus franchising operations. o Regulatory and compliance data – including compliance with safety regulations, environmental standards, accessibility laws, and other regulatory obligations imposed on public transportation operators, as well as contractual agreements (terms, conditions, and performance indicators) specified in contracts between the Combined Authority (the franchising authority) and bus operators, including penalties for non-compliance. o Environmental data – including records of carbon emissions, air pollutants, and other environmental impacts, as well as metrics related to fuel consumption, energy usage, and the adoption of alternative fuels or environmentally-friendly technologies to reduce environmental footprint. • Develop a comprehensive plan for development of the data platform based on the emerging needs data needs of a Mass Transit system which is likely to cover: o Demand data – including origin-destination (OD) data for analysis of travel pattern Where the supplied staff will work Yorkshire and the Humber Where the supplied staff will work No specific location (for example they can work remotely) Who the organisation using the products or services is West Yorkshire Combined Authority Why the work is being done This work is an extension of the Combined Authorities MCA Digital Programe which aimed to address key priorities such as data and reporting, regional collaboration, social media, mobile workforce, information governance, security, and visitors. This has led to the creation of a technology vision "Delivering Confidence, Value and Empowerment." The vision aims to provide secure systems, quick and reliable basic ICT services, and be open and accountable. It also aims to implement solutions on time, reduce recurring problems, and empower customers. This work is being carried out to address key data challenges in preparation for bus reform and mass transit: - Multiple versions of the truth​ - Creating reports is complex, time-consuming, and inconsistent across the organisation​ - Lack of skills and general data literacy inhibit work and quality of insights​ - ICT Services spend too much time storing and maintaining data​ - Data has sprawled everywhere, and much is poor quality.​ - Lack of integration between systems prevents seeing the bigger picture To support bus reform and mass transit work must be completed by the following dates: - Project scope refined and confirmed in January 2025 - Development plan confirmed by April 2025 - Data platform setup complete by December 2025 - All data platform training complete by April 2026 - Continous data onboarding from December 2025 - Provide techncial support up to October 2027 The business problem The Combined Authority does not have an approved data/information management strategy. A draft Data Strategy exists, an output of an independent assessment conducted by Risual as part of the CTP in 2020. This assessment involved cross-directorate staff engagement and analysis of current technology and business applications. The report found that whilst the majority of staff recognised that the Combined Authority has a wealth of data, the current data infrastructure, lack of access to datasets, and lack of training, were barriers to being able to unlock that value. The Combined Authority has a Data and Information Management Strategy, specifically for Mass Transit, and a complementary strategy is currently being commissioned for Bus Reform (to be completed by Autumn 2024). Additionally, the Combined Authority are developing a Corporate Data, Information and Knowledge Strategy which will be completed by Autumn 2024. The Combined Authority seeks to improve its transport data insights to support Bus Reform and Mass Transit as well as future large capital programmes, but faces challenges due to a shortage of data, poor data governance, and lack of analytical capabilities. For Bus Reform, accurate and timely insights into bus usage (such as passenger counts, ticket preferences, purchase methods) and travel behaviour (such as frequency of use and multi-modal use) are needed, which will require all bus data to be collated, stored, catalogued and analysed in a centralised permission-driven data platform. For the Mass Transit programme, a data management platform is required for data storage, reporting and analytics. Mass Transit programme data will be managed in a variety of different systems and tools, some of which will be managed by the Combined Authority and others by delivery partners. As such the data platform is needed to act as a ‘data integration hub’ which would facilitate provision and management of data from a range of different systems (internal and external) ensuring data security is maintained and establishing a ‘single source of truth’ approach. The Combined Authority is a Microsoft first organisation and therefore this project must provide a Microsoft based solution that fits within the existing architecture. Non-Microsoft technology can supplement the solution where there is a strong business case but the Combined Authority’s preference is not to use propriety non-Microsoft software. All staff have a Microsoft E5 licence. Some members of the Combined Authority have PowerBi Professional licences and have attended training courses on its use. Development Partner • Essential - Must have extensive experience of implementing and managing Data Platforms (or latest Microsoft equivalent) in a Microsoft environment. • Essential - Must have the ability to undertake directed onboarding, pipeline building, transformations, and output creations in an agile way. • Essential - Must have the ability to scale services for example by running multiple workstreams in parallel. • Desirable - the successful supplier should be a Microsoft Solutions Partner for Data & AI (Azure), or demonstrate their proven capability in delivering comprehensive data solutions using the Microsoft cloud environment. • Desirable - Experience of designing and implementing a data platform with a focus on transport would be beneficial. The people who will use the product or service User type: ICT Services Technician Definition: someone who manages systems including setting up user permissions, is able to configure/control Azure service costs efficiently and monitor usage and costs, ensure security, and add new users to the security roles created within the Combined Authority's environment. User type: Data Analyst Definition: someone who can store, manage, and access anonymised data for strategic reporting purposes, manually upload new or updated data to the Data Platform, set up regular automatic uploads of data, able to connect to internal and external data sources, transform data and record metadata for each data set User type: non-technical user Definition: a member of Combined Authority staff who wants to see what data exists within the Data Platform, as well as who is responsible for it and how often its updated, as well as consume dashboards created by data analysts. Any pre-market engagement done Pre-tender market engagement was undertaken in August 2024. The feedback received is summarised below:  The specification - Most suppliers agreed that the spec provided enough information for them to bid confidently.  Data Sources - Some suppliers wanted more detail about the data sources include the types of data, the expected volumes, the frequency of update (generally complexity). This has now been included. See attached document 'Appendix 1E - Data Sources'  Azure and Alternative solutions – Some suppliers asked whether the platform should be entirely custom-built within Azure environment, or if there were opportunities to integrate existing proprietary solutions. They mentioned that integrating a pre-existing solution into Azure could reduce costs/risks vs a fully custom build. This approach would be acceptable to the CA providing full costs for on-going support are provided  Data quality expectations – some suppliers referenced the need for clear expectations on data quality standards and processes for data cleansing and validation. Some suggested that data quality management should be a key part of the project scope – i.e simply copying datasets onto the platform without a robust data quality strategy would perpetuate issues like "multiple versions of the truth".  Transport focus - Some suppliers seemed a little unsure about focus of the project (e.g., transportation data or general corporate data) and wanted the scope made clearer. This commission is purely related to transport data.  Resource – Some suppliers referenced achieving the milestones depends heavily on the availability and engagement of stakeholders and CA resource. Suppliers should be clear on any assumptions they make with regards to the support from the CA  Timescale - Most suppliers said that project milestones and timelines were achievable - but emphasised it depends on refining the project scope during the discovery phase. The Project Team has then addressed the raised gaps in best way possible, Work done so far Risual were comissioned to produce a data strategy and create a specification of requirements for procurement of a data platform 2019-2020. Risual has worked with the CA to help so we could move into a position where we can procure the services needed to modernise our technology and consolidate to a single data platform. Risual conducted an extensive period of business analysis (with 65 stakeholders completing workshops, questionnaires and interviews) to understand how the Combined Authority use data (in 2019). The sessions were used to build up a picture of how each directorate (reflecting the structure at the time) operates today and what objectives they have for data and analysis in the future. ANS conducted a proof of concept in 2021 confirming that a MS Azure data platform could be used on the CAs Azure Tenant. Which phase the project is in Not started Existing team The suppliers will be working with a DTS project manager, infrastructure technicians in DTS services, data analysts in Research & Intelligence, policy officers in Bus Reform, and project managers in Mass Transit. Address where the work will be done West Yorkshire Combined Authority, Wellington House, 40-50 Wellington St, Leeds, LS1 2DE Working arrangements The supplier: • Must provide support during normal business hours 08:30 – 17:00 Monday to Friday excluding bank holidays. • Must provide various contact methods to allow support for different levels of urgency. • Must have a single point of contact and a secondary contact (which may include generic mailbox/call number) if this person is not available. • Must provide an Account Manager for escalation and other matters of importance with availability between 09:00 – 17:00 Monday to Friday excluding bank holidays. • Must provide an architect responsible for architectural validation and platform efficiency focused on continuous improvement with availability between 09:00 – 17:00 Monday to Friday excluding bank holidays. The architect/platform manager will have oversight of the workstreams involved in this commission. • Must have the capacity to manage multiple workstreams running in parallel. • Must have a defined escalation process. • Must provide written update on any critical failures. • Must discuss with the Combined Authority at set up stage whether Azure services should run; o 24/7/365 o during business hours o on demand. The impact on running costs must be presented and regularly reviewed. • Must provide continuous overview and recommendations on how to optimise the Azure hosting included controlling and managing Azure costs. • Must highlight Microsoft changes that will affect the Data Platform (or latest Microsoft equivalent) with recommendations on how to mitigate them so no/little downtime is experienced. • During the first 12 months of the contact must provide monthly reviews of the Data Platform (or latest Microsoft equivalent), providing recommendations for improving its use (e.g., making use of the range of services within a Microsoft Platform to maximise demonstration of value). Security and vetting requirements Baseline Personnel Security Standard (BPSS) More information about the Security requirements: in an instance when supplier's personnel visits the WYCA site. Latest start date 6 January 2025 Expected contract length Contract length: 2 years 0 months 0 days Optional extension: 1 years 0 months 0 days Budget Indicative maximum: £450000 Indicative minimum: The contract value is not specified by the buyer Further information: The maximum budget for this project over the entire contract life (2 years + 1 year) is £450,000. Contracted out service or supply of resource? Contracted out service: the off-payroll rules do not apply Terms and acronyms Term or acronym: CTP Definition: Corporate Technology Programme Term or acronym: MCA Digital Definition: Mayoral Combined Authority Digital Programme which sought to deliver the projects to allow the organisation to become Digital First. Term or acronym: RTI Definition: Real Time Information Term or acronym: OD Definition: Origin-Destination Term or acronym: GIS Definition: Geographical Information Systems Term or acronym: BIM Definition: Building Information Modelling Term or acronym: MS Definition: Microsoft Term or acronym: ETL Definition: Extract Transform Load Term or acronym: FME Definition: Feature Manipulation Engine