Case Summarisation Tool

Description

Pre-market engagement We’ve spoken to several suppliers who are involved in building stand-alone tools and/or offer services delivering them into organisations. We’ve done some internal scoping of a potential ‘summarisation’ tool and we’ve drawn up some technical designs demonstrating how the architecture should work. But we are looking for the right solution and not just one that fits with our existing plans. Work done so far We’ve completed research and requirements gathering for certain features with internal colleagues (caseworkers) and we’ve worked with 3rd parties to explore what a summarisation tool might look like. We have also defined the HLD and LLDs for the summarisation tool and have started the development work with initial deployment into DEV completed. However, as much as we would want to expand on our existing foundations, we are keen and open to understand the range of options to meet our ambitions. Which phase the project is in Discovery Existing team Head of Digital Transformation –Sponsor and commissioner of the project and proposed transformational solution. Accountable for the business case with the responsibility to define the key project outcome and scope. Platform Owner - Empowered through delegated authority from the sponsor to build the vision, blueprint and roadmap for intelligent automation and artificial intelligence tools to enhance customer and employee journeys. Project Manager – Responsible for planning and driving the project delivery, as well as managing risks, issues, dependencies and tracking progress against scope, budget and time. Business Analyst - Applies Business Analysis Services to the project to translate business desires and vision into technical user stories that are understandable to developers. Customer Journey Owner – champion of the customer experience who aligns business processes, technology, and team efforts to deliver a seamless, engaging, and customer-centric journey. Machine Learning Lead - responsible for leading the team of data scientists, machine learning engineers to develop, implement and optimise machine learning models and solutions. Service Designer – responsible for creating and improving services to make them user-friendly, efficient, and aligned with the needs of both customers and the business. Chief Architect - responsible for guiding and overseeing the technical architecture of solutions developed by the supplier. They define the technical vision, guide architectural decisions, and ensure that the technology landscape is robust, scalable, and secure. Change Lead – responsible for ensuring organisational changes are effectively adopted and utilised by the organisation. Address where the work will be done Remotely and at Exchange Tower, London, E14 9SR. Working arrangements The supplier team will be required to come to the FOS offices in Exchange Tower, London, once a week. The agreed office day will be an opportunity for the supplier and FOS teams to meet in person for closer collaboration. The supplier will run daily stand-ups that will involve supplier dev team and some key FOS team members to ensure timely delivery. The supplier will produce status reports on a bi-weekly basis, that will be shared with FOS Transformation Senior Leaders. The supplier will attend and present in the Project Board meetings, and will be accountable for the technical delivery. The supplier will have to comply with the FOS Consultant Delivery framework and Software development Life Cycle (SDLC) that provide governance about delivery method, communication management, risk management, performance monitoring, key performance indicators, technical delivery approach and expected technical development standards and best practices. Provide more information about your security requirements: No security clearance needed Latest start date 2025-04-30 Enter the expected contract length: 9 months Special terms and conditions Provided within the order form to shortlisted suppliers, Are you prepared to show your budget details?: No Indicative maximum: n/a Confirm if you require a contracted out service or supply of resource Contracted out service: the off-payroll rules do not apply Summary of work We're building a modern Financial Ombudsman Service to better serve our customers, our people, and the Financial Services Sector. As part of our transformational programme, we are looking to implement new tools for our caseworkers which help them get to the heart of the complaint quicker - by using data from the case file combined with historical outcomes, and our knowledge base. We’ve recently built new data infrastructure which extracts and stores the unstructured data from our case files in a database – ready for consumption by tools delivering advanced analytics and AI use-cases. Building on that work, the new solution will provide caseworkers with a detailed summary of the complaint, with references to the relevant data in the specific case file, our historical outcomes on similar cases and our knowledge base. The summary should suggest to the user the key questions and evidence they should focus to reach a fair and reasonable opinion on the dispute. The solution is likely to use an LLM, powering Generative AI outputs, and it should meet the Ombudsman Service’s required standards for Data protection and privacy. We are looking for an agile development approach, with a user-centred design, to accelerate innovation of the tool. This approach should include the provision of a Beta testing and Pilot phase for thorough testing and quality assurance checks before wider roll-out. The development approach should also allow for further iterations to be made easily to ensure the solution delivers on the expected benefit to increase caseworker productivity. Where the supplied staff will work London Where the supplied staff will work No specific location (for example they can work remotely) Why the work is being done As part of the Financial Ombudsman Service’s continuous improvement journey, we are constantly challenging ourselves to look for new ways to serve our customers better; while ensuring we’re a financially sustainable and forward-thinking organisation, with the ability to predict and the agility to adapt to changing demands and new issues within the Financial Services sector. We want to improve the efficiency and timeliness of our service to customers. We plan to do this by developing tools to support our caseworkers, enabling them to see what a complaint is about and which factors they need to consider as soon as they pick up the file, and helping them to find the key information on a file and relevant know-how more quickly. We want to have the production-ready solution by 30th November 2025 at the latest. Suppliers should be available to start work by 30th April 2025. The business problem you need to solve As an informal alternative to the courts, we receive information and documents from complainants and financial businesses to explain and support their case on whether a fair outcome has been reached. Caseworkers presently need to review numerous documents on a case before they get an understanding of what the case is about. They then need to reference our knowledge repository (‘Discovery’ – built in SharePoint Online) to put the evidence in the context of the relevant law, rules and regulations and our historical approach to similar cases. We want to speed up the time it takes for our people to get to the heart of a case, enabling them to deliver fair outcome, quicker. In doing so we need to mitigate against information being summarised incorrectly or missed and ensure the new GenAI-powered solution provides an accurate summary and appropriate guidance on what to look at in more detail. The solution should also give the precise location of the original information in the casefile. We also need to ensure we take care of our customers’ data throughout the development process – so having an effective approach to protecting data as we build the tool is essential. First user type: Caseworker Questions and Clarifications 0. Is there an incumbent? There is no incumbent supplier. Last Updated: <strong>2024-11-27T15:05:57.886411Z</strong> 1. Can we please clarify the character count limit - is this applicable per question in Essential skills and experience (5 questions at 750 characters each or 750 characters for all combined) - and Nice-to-have skills and experience (3 questions at 750 characters each or 750 characters for all 3 combined) Character limit is set by the platform and is per question. Last Updated: <strong>2024-11-27T15:03:07.074887Z</strong> 2. Would you consider a commercial off-the-shelf (COTS) solution? We would be happy to consider off the shelf if it fits with our requirements Last Updated: <strong>2024-11-27T16:17:27.362307Z</strong> 3. Can you provide any further guidance on what you would like from an answer to question 2. The list of technologies is substantial and it's impossible to reference all of those in an answer within the character limit. Would you like a representative selection in the answer, or do you have anything specific that you would like reference to? Please provide a statement or summary of your overall experience with the listed technical stack and/or identify areas that you do not have technical experience in. Providing examples for each is not required. Last Updated: <strong>2024-11-27T16:24:42.90813Z</strong> 4. Please can you share an indicative contract value/budget range to help bidders ascertain whether they are able to deliver within the available budget? We are not providing budgetary information at this stage in order to consider all potential solutions. Last Updated: <strong>2024-11-27T16:26:58.133974Z</strong> 5. Clarification Responses Batch 1 Q1) Please can you confirm who built the new data infrastructure that extracts and stores the unstructured data from your case files into a database – ready for consumption by tools delivering advanced analytics and AI use-cases. A1) This was built by our internal teams in collaboration with Baringa Q2) What is the expected interaction between the supplier and the current FO team? (e.g. will all development of the deliverable be responsibility of the supplier?) A2) Supplier will be responsible for delivery, however we would expect knowledge transfer and insight into the execution phase. Q3) Is there an existing LLM utilised in the FOS? If not, do you have any preference about the LLM to be used? A3) We have implemented ML solutions using BERT and we are also starting to use LLM's available in the Azure portal. We are open to options but currently have intentions to use Azure Open AI service . Q4) Is it correct that bidders’ response to the Essential questions will account for 10% of Technical evaluation, but Nice-to-Have responses will not count towards Technical evaluation at all? A4) Correct. 10% at stage 2. Q5) Could you confirm the names of the third parties you’ve worked with on this requirement to date, please? A5) Baringa, Advantiq Q6) Whilst you won't share info on budget at this stage, will suppliers shortlisted for Stage 2 receive this info? A6) Our intention is not to share budget information, in order to consider all all potential solutions. Q7) Are you able to share any indication of expected contract length? A7) As per the advert the expected contract duration is 9 months. Q8) Q1 asks for a project where we've delivered 'tools' with GenAI technologies. Will bidders score more highly if their answers have more than one example of a tool they delivered or is it enough to evidence delivery of a tool with GenAI technologies? A8) It sufficient to evidence delivery of a tool with GenAI technologies Last Updated: <strong>2024-11-29T16:43:10.422979Z</strong> 6. May you please provide more detail on technologies used for the new data infrastructure (ETL / ELT tool, data storage, etc.)? The solution will integrate with our Dynamics 365 CE-based case handling system. Data generated by the case summarisation tool should auto-populate a ‘Summary’ field in our case handling system. It should be noted our case handling system has been customised and is not an out of the box Dynamics solution.  Our evidence file is the repository of all evidence related to all cases – including email, documents and media – incoming and outgoing. The solution offers listing/sorting of documents, searching of text content, tagging of documents and ability to annotate. The system is an in-house built solution and comprises of React client, Azure Data Lake, Azure Functions and PDFTron. The solution will consume the unstructured case data from our existing Cosmos DB. Appropriate measures should be put in place to obfuscate selected personal information data types to remain compliant with our data protection obligations. The organisation makes use of SharePoint Online as the repository of Knowledge Articles that provide approved guidance for case handling process and procedure. We use Azure Data Factory as ETL tool. Last Updated: <strong>2024-12-04T10:57:45.502184Z</strong> 7. Clarification Question Responses - Batch 2 Q1) What document format(s) are included within in the knowledge base? (e.g. PDF, tables. images etc.) A1) Discovery is Sharepoint Online - So the majority is in pages (HTML) with some links to other docs like pdf, docx and additionally the pages do have controls like paginated grids, tables and images. Q2) Have FOS considered using Graph databases combined with LLM to understand the information in the documents and the relationships between the entities? A2) To our knowledge CosmosDB can function as a Graph database Q3) Are you open to a hybrid team that leverages offshore capability? A3) We are open to hybrid team as long as the team can work UK hours Q4) Are there any specific security requirements needed by the team? A4) Security requirements are part of the non-functional requirements (NFRs) that will be shared at the next stage Q5) Are you already using Gen AI / LLM technologies, which services and for what use cases? A5) We have implemented ML solutions using BERT and we are also starting to use LLM's available in the Azure portal. We are open to options but currently have intentions to use Azure Open AI service. Q6) What type of user interfaces have you envisioned for the case summarization system (chat, search, etc.)? A6) For the user interface, we are keen to get recommendations from third party providers - we want a seamless interface that is simple to use and thought to facilitate adoption. We are open to chatbot and other solutions. Q7) Do you already have agile project management practices in place? If yes, may you please provide more detail? A7) We have agile ceremonies in place and have a development cycle based on 2-week sprints though we would expect the third party providers to help leverage our agile practises Last Updated: <strong>2024-12-04T11:01:46.136205Z</strong> 8. Clarification Question Responses - Batch 3 A1) Have you already envisioned ways to measure the accuracy of the LLM summarization and reference system? Q1) This is something we would expect the third party providers recommendations on. We are dedicated in delivering Responsible AI solutions which means that accuracy including bias and hallucination mitigations will be core to the development of this solution Q2) Are you going to share your HLD / LLD design of the tool during the RfP Process? A2) We will share HLD of the tool during the RfP Process though we don't want the third party providers to come with a preconception of the solution. Instead we want thirs party providers to come with their respective recommendations and we are open to be challenged on what we came with so far. Q3) May you please provide more details on the FOS Consultant Delivery framework and Software development Life Cycle (SDLC)? A3) The FOS Consultant Delivery framework and SLLC will include FOS best practises when it comes to: - Delivery Method and process (inc. definition od Done, CI/CD, technical expectations when it comes to test coverage, code review but also favoured approach to development (e.g. Test-Driven development(TDD), Clean Code, Behaviour-Driven Development(BDD), bug prioritisation,...) - Communications Management - Risk Management - Performance monitoring & KPI Last Updated: <strong>2024-12-04T11:04:37.774225Z</strong> 9. Clarification Question Responses - Batch 4 Q1) Please can you confirm whether you believe any personal data will be processed as part of this engagement? A1) We believe personal data will be processed as part of this solution. It is expected that there will be a level of data obfuscation to be applied using Privacy Enhancing Techniques (PETs) - this will be defined during the project. Part of it, we expect the solution should adhere to relevant data protection regulations, ensuring that personal data is not exposed or processed in ways that would violate privacy laws. In addition, access controls (Case-based permissions) will have to be in place to ensure the solution provides access to case data and respective generated content to end-users who have the permissions only. Q2) To what extent will TUPE apply to this engagement? A2) We do not anticipate TUPE applying this contract. Q3) Please advise whether or not there is currently an incumbent, either an external supplier, or a team working in-house currently supplying any of the services under this Tender. If so, please outline the details of same. A3) There is no current incumbent supplier for these services. Q4) Can you elaborate more on what has been produced, e.g. what % of the features are included in the Dev deployment? Is it a working POC? A4) We have developed a POC using Azure OpenAI in non-rpodcution enviromenet with dummy data. This POC was able torpduce a summary based on the ingested documents. Last Updated: <strong>2024-12-04T11:09:30.361909Z</strong> 10. Clarification Question Responses - Batch 5 Q1) How has the design been tested and does it meets user needs? A1) The POC we have done and the solution we will develop are part of our Trasnformation Programme that aim to digitalise our service to offer better services to our customers. When it comes to the user interface, we are keen to get recommendations from thirs party providers - we want a seamless interface that is simple to use and thought to facilitate adoption. We are open to chatbot and other solutions. Our Service Design Lead and our caseworker representatives and colleagues will be involved in the design process. Q2) Is there anything you know or think needs to be reworked, in terms of the design or architecture? A2) We leave that to the appreciation of the third party provider. We are open to be challenged on what we came with so far. Q3) Are you committed to the Msoft platform? 5. What are the key performance metrics for the summarisation tool? A3) We will use MSplatform licensing - though when it comes to decide on the LLM or SLM offered through MSplatform we are open to understand what will be the provider's recommendations to best fit our purpose. Key performance metrics have to be defined as part of the definition phase of the project. The solution aims to reduce duel time for the case investigation process, providing better outcome and experience to our customers, assisting our caseworkers in the investigation process sand improving productivity. Q4) How will you ensure a level playing field between new suppliers and suppliers who have been involved in earlier stages? A4) Bids will be assessed in an open and transparent way and all previously obtained relevent information will be shared with all bidders where possible. Potential bidders are encouraged to seek clarification or request further information where needed. Last Updated: <strong>2024-12-04T11:13:46.288249Z</strong> 11. Clarification Question Responses - Batch 6 Q1) Regarding the existing team listed in the SoR, will all these roles persist through the project, or will the supplier need to provide some of those? A1) All those existing roles will persist through the project Q2) What are your preferred ways of working? A2) We favour an agile framework. Last Updated: <strong>2024-12-04T11:15:06.75591Z</strong>