Degree

Bachelor of Science (Computer Science)

Department

Department of Computer Science

School

School of Mathematics and Computer Science (SMCS)

Advisor

Ms. Maria Rahim Khowaja, Lecturer, Department of Computer Science

Keywords

Clinical Copilot - using knowledge Graph, Natural language Processing (NLP), Agentic Healthcare AI

Abstract

Clinical Copilot is an all-in-one ecosystem designed to enhance clinical workflow with the help of Artificial Intelligence. The system eases and assists doctors’ workflow by helping them manage their clinical documentations, diagnostic support, and patient data management. It sits at the heart of the doctor’s room, transcribing real time doctor–patient conversation in both English and Urdu languages, and then intelligently extracts facts from the conversation. This helps identify possible misdiagnosis between patient medicines, past illnesses, or even past checkups against the current conversation and then notifies the doctor in real time to prevent any future medical issues. This feature is backed by a Knowledge Graph architecture, a RAG-based approach to provide better accuracy and reasoning, helping the system understand the complex relationships between medications, diseases and patient reports to generate accurate alerts. This is supported by the fact that patients can also upload their medical textual based reports which are processed through OCR and securely ingested into the patient’s database, helping the system for future analysis and diagnostic alerts. The system also allows doctors to automatically generate After Visit Summary (AVS) and customizable SOAP notes tailored to doctor’s preferences. It also provides an integrated conversational assistant, “Shifa Scribe,” which acts as an intelligent interface connected to the entire clinical ecosystem, enabling doctors to retrieve patient insights, summaries, historical data, and contextual recommendations through natural language interaction. By combining speech processing, natural language understanding, OCR, knowledge graphs, and AI driven clinical reasoning, the proposed Clinical Copilot project aims to reduce administrative burden, improve documentation efficiency, support early detection of diagnostic inconsistencies, and enhance continuity of patient care while maintaining HIPAA compliance data management for clinics and hospitals.

Tools and Technologies Used

TypeScript, Python, Next.js, Tailwind CSS, shadcn/ui, Prisma, PostgreSQL, FastAPI, OpenAI Whisper Large-V3, PyanNote, Pinecone, Neo4j, Zod, Clerk, Qwen 3.5 9B, GLM-OCR

Methodology

We identified major pain points in clinical workflows, including excessive documentation time, fragmented patient records, limited contextual diagnostic support, and inefficient data retrieval. Based on these challenges, the platform architecture was divided into interoperable modules, utilizing a modular full-stack methodology to integrate real time transcription, RAG, Agentic Workflow-based reasoning, and automated clinical documentation into a unified workflow. A patient data processing pipeline was developed to consolidate medical records, laboratory reports, and prescription data into structured representations that support efficient querying and analysis. During clinical consultations, a real-time audio processing module captured doctor–patient conversations and generated transcriptions using voice activity detection, speaker diarization, and segmentation techniques was added. Information extracted from these conversations was utilized to generate clinical alerts, SOAP notes, and After Visit Summaries. To support clinician decision-making, a hybrid retrieval framework combines structured database queries with semantic vector-based search, enabling access to both numerical patient data and contextual medical records. Retrieved information was then used to provide grounded responses through a clinician-facing copilot interface called “Shifa-Scribe”. A knowledge graph was incorporated to model relationships between diseases, medications, allergies, symptoms, and patient history. This enabled contextual reasoning for tasks such as drug–allergy conflict detection, drug–disease interaction analysis, and alternative medication recommendations. Finally, schema-constrained generation and validation techniques were employed to ensure consistent documentation outputs, which were validated, formatted, and exported as standardized clinical reports. The overall system was evaluated based on retrieval performance, tool-selection accuracy, and documentation quality.

Document Type

Restricted Access

Submission Type

BSCS Final Year Project

Creative Commons License

Creative Commons Attribution 4.0 International License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Download

Share

COinS