GLA London Datastore Rebuild

Description

Summary of the work GLA is developing a cloud-based data platform to better support data sharing across London’s complex data environment. In addition to an enhanced role as a central register of the city’s key datasets, it will deliver the core functionality on top of which multiple advanced services can be built by others. Expected Contract Length 18-24 months Latest start date Monday 13 June 2022 Budget Range We expect to spend up to £400,000 + VAT on this phase of work. We will collaboratively agree smaller phases through Statements of Work to ensure we continue to meet user needs. Why the Work is Being Done The GLA’s current data sharing platform, the London Datastore, is being used to meet user needs outside of its original scope. We are looking to redevelop the website enable us to better meet the needs of our users and iterate in future. DaaS – Increasingly, the GLA and other organisations are providing ‘data services’ as a means to address city challenges including improving air quality, reducing violence and protecting cultural assets. A Central Register of London’s key datasets - City governance in London is complex, with a wide array of organisations delivering different public services and developing their datasets. Whilst it isn’t practical to create a single ‘data lake’ with all of London’s data, it is important to make key datasets more easily discoverable (including deep metadata to support advanced analytical techniques). Private data sharing – Complex matrices of users, groups and different cuts of datasets can quickly emerge. We need very fine level management of access control and user management. Big data – as the availability of new data sources increases, including from IoT devices / sensor networks, the nature and scale of the data to be accessed, stored and manged is changing, including growing demands on API provision. Problem to Be Solved We have already undertaken an initial “high-level” discovery exercise to help us establish our vision for the website and how it fits in to our wider ambitions for data sharing in London. This was followed by a detailed discovery exercise in summer 2021. We are now looking to assemble a team to build the new London Data-sharing Platform. During the first phase (0-12 months), we will need to build the MVP. This will deliver lots of functionality in a single release to allow users to switch over to the new platform from the current London DataStore (which is procured as SaaS and will continue to run in parallel). In addition to reproducing existing functionality, we will need to meet higher security standards, deliver larger data sets via API, develop the schema for richer metadata, allow more interactive content on data pages and ensure searches look for results across page content, blogs, metadata and data. In the following phases, we will look to take on more complex challenges including single sign-on, supporting digital-first publication and creating a configurable landing page for users, so that they can create a more customised experience. Who Are the Users During the detailed discovery, we undertook 13 workshops and followed up with 10 interviews, and validation sessions. From this, we have user personas, 91 user stories and 7 user journeys, with the work organised into epics. An example user story is: “As a Data Publisher, I want to bulk import new users from a spreadsheet, so that I can save effort and expedite the process”. Six key user roles emerged: Consumer — wants to discover and gain access to data/insights 1. Consumer - wants to analyse data to provide new insights 2. Decision Maker - wants to answer a question Publisher — wants to share data or insights With Others 3. Data Sharer — Wants to share data (files in a number of formats) 4. Analyst - wants to share insights and analysis (which might be supported by data. but the raw data isn’t the key thing) 5. Data Curator - wants to curate a collection of data & analysis to solve a city problem, (data may have originated from different disciplines) and then share the collection Data Collector 6. wants to bring together data from different organisations into one place for analysis (often updated on a regular basis) Work Already Done High-level discovery: https://theodi.org/article/discovering-the-future-of-the-london-datastore/ Outputs from the detailed discovery will be available to the successful supplier, summary recommendations here https://drive.google.com/file/d/1j3fG5bdEj_NEJUNNpglS2yNeRBOBoKXP/view?usp=sharing and https://data.london.gov.uk/blog/a-new-city-data-sharing-platform-for-london/ . For key components, requiring high levels of customisation and iteration we are proposing to invest significant developer time (Catalogue, Search, Access control and Front End). For other components (such as, Identity and Access Management, event monitoring) we would lease or buy 3rd party services or products to shortcut delivery lead times. Post detailed discovery, we’ve installed two front-running open-source data catalogues and carried out interviews to identify commonly recurring technologies and software used in data platforms in London. Existing Team You will be embedded alongside GLA staff in the City Intelligence Unit which is working closely with a growing GLA Digital Transformation Unit. The GLA will provide Product Manager, Delivery Manager, Senior Software Developer, Stakeholder Engagement Officer and Data Curator. You will have some access to a Technical Architect, Test Engineer and Security adviser to ensure integration with the GLA’s other digital products, including the Mayor’s website https://www.london.gov.uk/ . Current Phase Alpha Skills & Experience • Experience of data catalogues and sharing large, complex data sets • Experience of developing modular systems using highly performant, robust, accessible and secure micro-services and data driven APIs • Experience of systems that are a pragmatic combination of adopt (i.e. open-source), build and buy • Experience of building cloud native applications, including ability to scale and containerisation • Experience of developing production environments and managing large, complex code bases, including continuous deployment • Demonstrate a track record of user-centred design and Agile approaches to delivery • Demonstrate a track record of working as part of a mixed inhouse/agency team • Experience of front-end technologies including JavaScript, React • Experience of Elastic Search, Python, Node, Java, Kubernetes • Experience of PostgreSQL and graph databases • Experience of developing ‘in the open’ – working to GDS standards and using tools such as Github • Experience of content design to enable users to perform technical tasks Nice to Haves • Demonstrate an understanding of some of the key services, datasets and analytical capacity within a local authority or national government • Demonstrate active participation in an open-source community, contributing code as well as making use of open-source software Work Location Staff at the GLA have a distributed working pattern at the moment with some time working remotely and some time working in the office when face-to-face working is required. The office is London Fire Brigade Head Office, 169 Union Street, London, SE1 0LL. Working Arrangments The chosen agency would need to fully integrate with our team, share their knowledge and insight, work in an open and transparent way and demonstrate good agile product delivery practice. Regular meetings, workshops, show and tells should be held on suitable platforms such as Microsoft Teams. Security Clearance Bidders will be expected to meet the Baseline Personal Security Standard Additional T&Cs The GLA will be free to use the deliverables arising from the contract without further reference to the contractor. All development work (and all materials to be published online) must as a minimum meet the Web Content Accessibility Guidelines (WCAG) version 2.1 to AA standard. We will be testing against the above WCAG guidelines and you will be required to cover the cost of any resulting code fixes for new products. Compliance with standards and regulation applicable to security and data protection (with the Supplier being the data processor). No. of Suppliers to Evaluate 5 Proposal Criteria • Understanding of the brief • Completeness of the proposed approach • Experience from a similar project • Team structure, including skills, experiences and relevance of individuals • Identification of risks and plan to mitigate them Cultural Fit Criteria • Evidence of working in an Agile way • Approach to working with the GLA as part of an integrated team • Evidence that the supplier will be transparent and collaborative when making decisions • Evidence the supplier will challenge GLA assumptions when a better solution is available – particularly when reviewing the options for each of the modules • How the supplier will share knowledge and experience with other team members • GLA is a diverse organisation, reflecting the diversity of London. Describe how you would successfully work as a team with us and other suppliers in a diverse environment. Payment Approach Capped time and materials Assessment Method Presentation Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers 1. Note to all suppliers: Deadline for supplier response change due to Easter half term The deadline for supplier response is being changed:Old timeline advertised: 18 April 2022New timeline due to Easter Holidays: 22 April 2022 (23:59) 2. On the blog referenced for the project, you mention that ODI did the initial discovery work. You also mention that there was a second discovery project. Who completed that work? The Open Data Institute did the high-level discovery, and Glue Reply did the detailed discovery. See links to information about both in 'About the Work' section. 3. Who did the Discovery for this please? The Open Data Institute did the high-level discovery, and Glue Reply did the detailed discovery. See links to information about both in 'About the Work' section. 4. GLA mentions the current data store is SaaS. Is the preferred approach for the GLA existing team to run the new platform, or is SaaS or any aspect of managed services provision expected from the vendor for the core platform? There are several ways the new platform could be delivered, including open source with the option to customise and build, vs lease. We are planning to adopt a pragmatic approach, using off-the-shelf components where they meet our needs, and prioritising development time on those parts of the new city data platform where full control over the architecture, technical design and operating model are key priorities for users and areas of future innovation and iteration. This includes the catalogue, search, and user management functionalities. When built the platform will be managed and operated by the GLA. 5. Will GLA be republishing the opportunity on Friday to allow 2 weeks response? The DOS Portal automatically closes after two weeks of being live, so suppliers currently will have to submit on Monday, 18th April The GLA is in contact with CCS to amend the timeline, or for CCS to provide a workaround. As soon as the GLA get's an official response back from CCS, we will update the answer to this question. 6. Will GLA be republishing the opportunity on Friday to allow 2 weeks response? The DOS Portal automatically closes after two weeks of being live, so suppliers currently will have to submit on Monday, 18th April Update: After consulting with CCS, we will withdraw and re-advertise this tender tomorrow, to ensure the deadline carries over to the 22nd April. Please note: All questions received to date, will be published again, and associated answers provided.