GLA London Datastore Rebuild copy

Description

Summary of the work GLA is developing a cloud-based data platform to better support data sharing across London’s complex data environment. In addition to an enhanced role as a central register of the city’s key datasets, it will deliver the core functionality on top of which multiple advanced services can be built by others. Expected Contract Length 18-24 months Latest start date Monday 13 June 2022 Budget Range We expect to spend up to £400,000 + VAT on this phase of work. We will collaboratively agree smaller phases through Statements of Work to ensure we continue to meet user needs. Why the Work is Being Done The GLA’s current data sharing platform, the London Datastore, is being used to meet user needs outside of its original scope. We are looking to redevelop the website enable us to better meet the needs of our users and iterate in future. DaaS – Increasingly, the GLA and other organisations are providing ‘data services’ as a means to address city challenges including improving air quality, reducing violence and protecting cultural assets. A Central Register of London’s key datasets - City governance in London is complex, with a wide array of organisations delivering different public services and developing their datasets. Whilst it isn’t practical to create a single ‘data lake’ with all of London’s data, it is important to make key datasets more easily discoverable (including deep metadata to support advanced analytical techniques). Private data sharing – Complex matrices of users, groups and different cuts of datasets can quickly emerge. We need very fine level management of access control and user management. Big data – as the availability of new data sources increases, including from IoT devices / sensor networks, the nature and scale of the data to be accessed, stored and manged is changing, including growing demands on API provision. Problem to Be Solved We have already undertaken an initial “high-level” discovery exercise to help us establish our vision for the website and how it fits in to our wider ambitions for data sharing in London. This was followed by a detailed discovery exercise in summer 2021. We are now looking to assemble a team to build the new London Data-sharing Platform. During the first phase (0-12 months), we will need to build the MVP. This will deliver lots of functionality in a single release to allow users to switch over to the new platform from the current London DataStore (which is procured as SaaS and will continue to run in parallel). In addition to reproducing existing functionality, we will need to meet higher security standards, deliver larger data sets via API, develop the schema for richer metadata, allow more interactive content on data pages and ensure searches look for results across page content, blogs, metadata and data. In the following phases, we will look to take on more complex challenges including single sign-on, supporting digital-first publication and creating a configurable landing page for users, so that they can create a more customised experience. Who Are the Users During the detailed discovery, we undertook 13 workshops and followed up with 10 interviews, and validation sessions. From this, we have user personas, 91 user stories and 7 user journeys, with the work organised into epics. An example user story is: “As a Data Publisher, I want to bulk import new users from a spreadsheet, so that I can save effort and expedite the process”. Six key user roles emerged: Consumer — wants to discover and gain access to data/insights 1. Consumer - wants to analyse data to provide new insights 2. Decision Maker - wants to answer a question Publisher — wants to share data or insights With Others 3. Data Sharer — Wants to share data (files in a number of formats) 4. Analyst - wants to share insights and analysis (which might be supported by data. but the raw data isn’t the key thing) 5. Data Curator - wants to curate a collection of data & analysis to solve a city problem, (data may have originated from different disciplines) and then share the collection Data Collector 6. wants to bring together data from different organisations into one place for analysis (often updated on a regular basis) Work Already Done High-level discovery: https://theodi.org/article/discovering-the-future-of-the-london-datastore/ Outputs from the detailed discovery will be available to the successful supplier, summary recommendations here https://drive.google.com/file/d/1j3fG5bdEj_NEJUNNpglS2yNeRBOBoKXP/view?usp=sharing and https://data.london.gov.uk/blog/a-new-city-data-sharing-platform-for-london/ . For key components, requiring high levels of customisation and iteration we are proposing to invest significant developer time (Catalogue, Search, Access control and Front End). For other components (such as, Identity and Access Management, event monitoring) we would lease or buy 3rd party services or products to shortcut delivery lead times. Post detailed discovery, we’ve installed two front-running open-source data catalogues and carried out interviews to identify commonly recurring technologies and software used in data platforms in London. Existing Team You will be embedded alongside GLA staff in the City Intelligence Unit which is working closely with a growing GLA Digital Transformation Unit. The GLA will provide Product Manager, Delivery Manager, Senior Software Developer, Stakeholder Engagement Officer and Data Curator. You will have some access to a Technical Architect, Test Engineer and Security adviser to ensure integration with the GLA’s other digital products, including the Mayor’s website https://www.london.gov.uk/ . Current Phase Alpha Skills & Experience • Experience of data catalogues and sharing large, complex data sets • Experience of developing modular systems using highly performant, robust, accessible and secure micro-services and data driven APIs • Experience of systems that are a pragmatic combination of adopt (i.e. open-source), build and buy • Experience of building cloud native applications, including ability to scale and containerisation • Experience of developing production environments and managing large, complex code bases, including continuous deployment • Demonstrate a track record of user-centred design and Agile approaches to delivery • Demonstrate a track record of working as part of a mixed inhouse/agency team • Experience of front-end technologies including JavaScript, React • Experience of Elastic Search, Python, Node, Java, Kubernetes • Experience of PostgreSQL and graph databases • Experience of developing ‘in the open’ – working to GDS standards and using tools such as Github • Experience of content design to enable users to perform technical tasks Nice to Haves • Demonstrate an understanding of some of the key services, datasets and analytical capacity within a local authority or national government • Demonstrate active participation in an open-source community, contributing code as well as making use of open-source software Work Location Staff at the GLA have a distributed working pattern at the moment with some time working remotely and some time working in the office when face-to-face working is required. The office is London Fire Brigade Head Office, 169 Union Street, London, SE1 0LL. Working Arrangments The chosen agency would need to fully integrate with our team, share their knowledge and insight, work in an open and transparent way and demonstrate good agile product delivery practice. Regular meetings, workshops, show and tells should be held on suitable platforms such as Microsoft Teams. Security Clearance Bidders will be expected to meet the Baseline Personal Security Standard Additional T&Cs The GLA will be free to use the deliverables arising from the contract without further reference to the contractor. All development work (and all materials to be published online) must as a minimum meet the Web Content Accessibility Guidelines (WCAG) version 2.1 to AA standard. We will be testing against the above WCAG guidelines and you will be required to cover the cost of any resulting code fixes for new products. Compliance with standards and regulation applicable to security and data protection (with the Supplier being the data processor). No. of Suppliers to Evaluate 5 Proposal Criteria • Understanding of the brief • Completeness of the proposed approach • Experience from a similar project • Team structure, including skills, experiences and relevance of individuals • Identification of risks and plan to mitigate them Cultural Fit Criteria • Evidence of working in an Agile way • Approach to working with the GLA as part of an integrated team • Evidence that the supplier will be transparent and collaborative when making decisions • Evidence the supplier will challenge GLA assumptions when a better solution is available – particularly when reviewing the options for each of the modules • How the supplier will share knowledge and experience with other team members • GLA is a diverse organisation, reflecting the diversity of London. Describe how you would successfully work as a team with us and other suppliers in a diverse environment. Payment Approach Capped time and materials Assessment Method Presentation Evaluation Weighting Technical competence 60% Cultural fit 20% Price 20% Questions from Suppliers 1. On the blog referenced for the project, you mention that ODI did the initial discovery work. You also mention that there was a second discovery project. Who completed that work? The Open Data Institute did the high level discovery, and Glue Reply did the detailed discovery. See links to information about both in 'About the Work' section. 2. Who did the Discovery for this please? The Open Data Institute did the high level discovery, and Glue Reply did the detailed discovery. See links to information about both in 'About the Work' section. 3. Please outline the scope a vendor will be expected to cover for an expected £400k spend? Is this to cover implementation of the MVP, and is it expected that roadmap r2 and r3 features will be covered under separate stamens of work and additional budget? If possible, please outline expected scope with reference to Discovery document conceptual architecture and/or delivery roadmap epics and features An indicative roadmap is detailed in the google drive doc linked in the 'about the work' section above. The budget is for delivery of the MVP and releases 2 and 3. Note, we intend to invest developer time in key components (catalogue, search, access control and front end), while for other components we would lease or buy 3rd party services or products to shortcut delivery lead times. We expect to spend up to £400,000 + VAT on this phase of work. We will collaboratively agree smaller phases through Statements of Work to ensure we are able to meet user needs. 4. GLA mentions the current data store is SaaS. Is the preferred approach for the GLA existing team to run the new platform, or is SaaS or any aspect of managed services provision expected from the vendor for the core platform? There are several ways the new platform could be delivered, including open source with the option to customise and build, vs lease. We are planning to adopt a pragmatic approach, using off-the-shelf components where they meet our needs, and prioritising development time on those parts of the new city data platform where full control over the architecture, technical design and operating model are key priorities for users and areas of future innovation and iteration. This includes the catalogue, search, and user management functionalities. When built the platform will be managed and operated by the GLA. 5. Under existing team description an embedded working arrangement is described. This opportunity is categorised as an outcomes-based delivery. Is it correct to assume the vendor will be expected to assign a multi-disciple delivery team that includes a vendor Delivery Manager? The vendor Delivery Manager would work closely with the GLA Delivery Manager to ensure close collaboration between the vendor and GLA teams. I.e., This will not be purely staff augmentation The GLA project team is set out in the 'about the work' section. The vendors are asked to set out the team they propose to assign to the project, including the skills, experiences and relevance of individuals. 6. The new platform will be cloud-based. Has a specific cloud vendor been selected for the platform? No, a specific cloud vendor hasn't been selected yet. Ideally the platform should be 'cloud neutral', however, we recognise that in its first iterations at least, it may benefit from some of the tooling specific to certain cloud vendors. 7. Can you please expand on this question as we’re not sure exactly what you’re asking here...’Experience of content design to enable users to perform technical tasks’. Thank you In contrast to a platform aimed solely at the general public, we also have data scientists, data engineers and modellers as a user group. Examples of their requirements include ensuring that they are well served by APIs and extending the metadata to include more about the provenance of the data. 8. How settled are you on specific technology choices – in the descriptive material there is mention of a number of options to build from (as preferred) of these are they examples or the narrowing options? There are areas where a choice of technology is available and we do not want to be prescriptive at this stage. We expect the supplier to review these options with us and bring their knowledge and insight to our decision making. 9. Can you confirm how many days a week the supplier team would be expected to work on site please? We will work with the supplier to agree the most effective working patterns for the project. Most GLA staff are working from home, with some days in the office. 10. Can you confirm the cloud provider please? A specific cloud vendor hasn't been selected yet. Ideally the platform should be 'cloud neutral', however, we recognise that in its first iterations at least, it may benefit from some of the tooling specific to certain cloud vendors. 11. Can you confirm the tech stack please? There are areas where a choice of technology is available and we do not want to be prescriptive at this stage. We expect the supplier to review these options with us and bring their knowledge and insight to our decision making. 12. Could you provide greater clarity around this question in terms of what experience you are expecting the supplier to demonstrate please: Experience of Elastic Search, Python, Node, Java, Kubernetes There are areas where a choice of technology is available and we do not want to be prescriptive at this stage. We expect the supplier to review these options with us and bring their knowledge and insight to our decision making. 13. "will need to meet higher security standards,” Can you please advise on the level for this? We will work with key users (particularly the London Boroughs) to agree a common approach to the level of security required. The new platform will need to be suitable for sharing personal data where there is a suitable legal gateway and ethical assessment. 14. Can you please clarify the date bidders should expect stage 2 documentation if shortlisted? When the presentation is due to take place and also the award date? Dates provided here are a guide. Notification of shortlisting 5th May. Shortlisted suppliers to submit written responses by 19 May, with presentations held week commencing 23 May. Award date by 13th June. 15. Can you advise the security clearance level for all roles? No security clearance is required. We will require the chosen supplier to be compliant with GLA IT and data policies and they will be asked to sign relevant documents to confirm their adherence to these.