The AI Care Standard
Evaluation Framework™
The AI Care Standard
Evaluation Framework™
Assess AI Solutions Against the Standard
Use this structured evaluation tool to assess the strength of any AI powered patient communication solution provided by a vendor, built by a health system or is an idea for concepts of a solution. This framework will apply a scoring systeme that will indicate how well the solution supports the guidelines for safe, accurate and relevant communication with patients across the core pillars
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
#3 Clinically Reviewed Model Validation
#4 Protecting Patients
#5 Personalization
#6 Patient Agency
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
Every good data scientist knows 95% of the effort is data wrangling and data cleansing. Do more. Put as much effort here as your fancy neural network! Health data is rarely "clean". Each hospital might use slightly different coding standards or reference ranges. Freetext was generated by tired humans who make cut-and-paste errors or contradict themselves. Normalize & sanitize your inputs. Don't make predictions when only 1% of inputs are available. If you take freetext, don't accept 4 word inputs when your system expects a 4 page note.
#3 Clinically Reviewed Model Validation
Not at all. We work with companies at every stage. If you already have a brand system, we’ll refine and extend it. If you’re starting from scratch, we’ll guide you through strategy, naming, identity, and tone of voice before we move into design. Many of our clients came to us with just an idea — by the time we launched, they had a full brand and website they could grow with.
#4 Protecting Patients
Yes. Design and branding don’t end at launch day. We provide post-launch support for small fixes, adjustments, or questions in the first weeks. For brands that want ongoing help — adding new pages, refreshing visuals, optimizing UX — we offer retainer options. This means you’ll always have us as a creative partner when you need to evolve or scale.
#5 Personalization
Your model isn't personalized quite enough. If you support only English, add Spanish in there for US-based health. 13% of the population (about 43M people!) speak Spanish at home. For accessibility (blind, deaf, color-blind, etc.) use the Web Content Accessibility Guidelines (WCAG) Level AA standards. Make sure the model knows more than the patient's age and name. Patients expect that the AI is fully connected to all or most of your EHR data.
#6 Patient Agency
Citing high-quality sources is a crucial scientific principle. Sources may include the patient's results, findings, or notes, or publications from reputable sources. Without citing sources, you're asking patients to accept information on "faith" or authority. That may work for religion, but medical information given to patients should be based on primary sources and empirical studies.
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
Every good data scientist knows 95% of the effort is data wrangling and data cleansing. Do more. Put as much effort here as your fancy neural network! Health data is rarely "clean". Each hospital might use slightly different coding standards or reference ranges. Freetext was generated by tired humans who make cut-and-paste errors or contradict themselves. Normalize & sanitize your inputs. Don't make predictions when only 1% of inputs are available. If you take freetext, don't accept 4 word inputs when your system expects a 4 page note.
#3 Clinically Reviewed Model Validation
Not at all. We work with companies at every stage. If you already have a brand system, we’ll refine and extend it. If you’re starting from scratch, we’ll guide you through strategy, naming, identity, and tone of voice before we move into design. Many of our clients came to us with just an idea — by the time we launched, they had a full brand and website they could grow with.
#4 Protecting Patients
Yes. Design and branding don’t end at launch day. We provide post-launch support for small fixes, adjustments, or questions in the first weeks. For brands that want ongoing help — adding new pages, refreshing visuals, optimizing UX — we offer retainer options. This means you’ll always have us as a creative partner when you need to evolve or scale.
#5 Personalization
Your model isn't personalized quite enough. If you support only English, add Spanish in there for US-based health. 13% of the population (about 43M people!) speak Spanish at home. For accessibility (blind, deaf, color-blind, etc.) use the Web Content Accessibility Guidelines (WCAG) Level AA standards. Make sure the model knows more than the patient's age and name. Patients expect that the AI is fully connected to all or most of your EHR data.
#6 Patient Agency
Citing high-quality sources is a crucial scientific principle. Sources may include the patient's results, findings, or notes, or publications from reputable sources. Without citing sources, you're asking patients to accept information on "faith" or authority. That may work for religion, but medical information given to patients should be based on primary sources and empirical studies.
The AI Care Standard
Evaluation Framework™
Assess AI Solutions Against the Standard
Use this structured evaluation tool to assess the strength of any AI powered patient communication solution provided by a vendor, built by a health system or is an idea for concepts of a solution. This framework will apply a scoring systeme that will indicate how well the solution supports the guidelines for safe, accurate and relevant communication with patients across the core pillars
Access the Framework
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
Every good data scientist knows 95% of the effort is data wrangling and data cleansing. Do more. Put as much effort here as your fancy neural network! Health data is rarely "clean". Each hospital might use slightly different coding standards or reference ranges. Freetext was generated by tired humans who make cut-and-paste errors or contradict themselves. Normalize & sanitize your inputs. Don't make predictions when only 1% of inputs are available. If you take freetext, don't accept 4 word inputs when your system expects a 4 page note.
#3 Clinically Reviewed Model Validation
Not at all. We work with companies at every stage. If you already have a brand system, we’ll refine and extend it. If you’re starting from scratch, we’ll guide you through strategy, naming, identity, and tone of voice before we move into design. Many of our clients came to us with just an idea — by the time we launched, they had a full brand and website they could grow with.
#4 Protecting Patients
Yes. Design and branding don’t end at launch day. We provide post-launch support for small fixes, adjustments, or questions in the first weeks. For brands that want ongoing help — adding new pages, refreshing visuals, optimizing UX — we offer retainer options. This means you’ll always have us as a creative partner when you need to evolve or scale.
#5 Personalization
Your model isn't personalized quite enough. If you support only English, add Spanish in there for US-based health. 13% of the population (about 43M people!) speak Spanish at home. For accessibility (blind, deaf, color-blind, etc.) use the Web Content Accessibility Guidelines (WCAG) Level AA standards. Make sure the model knows more than the patient's age and name. Patients expect that the AI is fully connected to all or most of your EHR data.
#6 Patient Agency
Citing high-quality sources is a crucial scientific principle. Sources may include the patient's results, findings, or notes, or publications from reputable sources. Without citing sources, you're asking patients to accept information on "faith" or authority. That may work for religion, but medical information given to patients should be based on primary sources and empirical studies.
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
Every good data scientist knows 95% of the effort is data wrangling and data cleansing. Do more. Put as much effort here as your fancy neural network! Health data is rarely "clean". Each hospital might use slightly different coding standards or reference ranges. Freetext was generated by tired humans who make cut-and-paste errors or contradict themselves. Normalize & sanitize your inputs. Don't make predictions when only 1% of inputs are available. If you take freetext, don't accept 4 word inputs when your system expects a 4 page note.
#3 Clinically Reviewed Model Validation
Not at all. We work with companies at every stage. If you already have a brand system, we’ll refine and extend it. If you’re starting from scratch, we’ll guide you through strategy, naming, identity, and tone of voice before we move into design. Many of our clients came to us with just an idea — by the time we launched, they had a full brand and website they could grow with.
#4 Protecting Patients
Yes. Design and branding don’t end at launch day. We provide post-launch support for small fixes, adjustments, or questions in the first weeks. For brands that want ongoing help — adding new pages, refreshing visuals, optimizing UX — we offer retainer options. This means you’ll always have us as a creative partner when you need to evolve or scale.
#5 Personalization
Your model isn't personalized quite enough. If you support only English, add Spanish in there for US-based health. 13% of the population (about 43M people!) speak Spanish at home. For accessibility (blind, deaf, color-blind, etc.) use the Web Content Accessibility Guidelines (WCAG) Level AA standards. Make sure the model knows more than the patient's age and name. Patients expect that the AI is fully connected to all or most of your EHR data.
#6 Patient Agency
Citing high-quality sources is a crucial scientific principle. Sources may include the patient's results, findings, or notes, or publications from reputable sources. Without citing sources, you're asking patients to accept information on "faith" or authority. That may work for religion, but medical information given to patients should be based on primary sources and empirical studies.
#1 Comprenhensive model trainning
#2 Clinically verified Model Prediction
#3 Clinically Reviewed Model Validation
#4 Protecting Patients
#5 Personalization
#6 Patient Agency
Assess AI Solutions Against the Standard
Use this structured evaluation tool to assess the strength of any AI powered patient communication solution provided by a vendor, built by a health system or is an idea for concepts of a solution. This framework will apply a scoring systeme that will indicate how well the solution supports the guidelines for safe, accurate and relevant communication with patients across the core pillars
Access the Framework
#1 Comprehensive Model Training
Consider utilizing data from multiple health systems across multiple regions. This may require licensing PHI-redacted data or using public data sets like MIMIC-IV. Use diverse data across age, ethnicity, disease state, and treatment setting (e.g. ER data is very different from Primary Care).
#2 Clinically verified Model Prediction
Every good data scientist knows 95% of the effort is data wrangling and data cleansing. Do more. Put as much effort here as your fancy neural network! Health data is rarely "clean". Each hospital might use slightly different coding standards or reference ranges. Freetext was generated by tired humans who make cut-and-paste errors or contradict themselves. Normalize & sanitize your inputs. Don't make predictions when only 1% of inputs are available. If you take freetext, don't accept 4 word inputs when your system expects a 4 page note.
#3 Clinically Reviewed Model Validation
You need appropriately-trained and licensed providers (i.e. MD/DO/RN/RD/PT/etc.) checking your AI periodically. We suggest an independent panel validate N >1000 at least twice a year. Deploy in "shadow" mode for a few weeks before launching to patients. Retrain and/or update vector databases at least annually. The AI system should check its own work. You might be thinking, “if the AI can check its work, it wouldn't make mistakes.” Partly true. A few techniques to consider: use a second AI to check the work (AI judge). In practice this eliminates ~80% of hallucinations and many other types of errors. Most AI also has confidence metrics. While 0.51 might map to "1" or "yes", it's an indication that the model isn't as confident as a 0.98 score!
#4 Protecting Patients
Detecting suicidal ideation is crucial for any system that allows patient input. But good systems go beyond that basic. Avoid the psychological harm and isolation from learning of a new cancer or loss of pregnancy without human empathy. Check for abuse. Look for patterns (words, phrases, response time) in patient input that may indicate degradation of agency or ability. Check for patient manipulation of the AI system itself.
#5 Personalization
Your model isn't personalized quite enough. If you support only English, add Spanish in there for US-based health. 13% of the population (about 43M people!) speak Spanish at home. For accessibility (blind, deaf, color-blind, etc.) use the Web Content Accessibility Guidelines (WCAG) Level AA standards. Make sure the model knows more than the patient's age and name. Patients expect that the AI is fully connected to all or most of your EHR data.
#6 Patient Agency
Citing high-quality sources is a crucial scientific principle. Sources may include the patient's results, findings, or notes, or publications from reputable sources. Without citing sources, you're asking patients to accept information on "faith" or authority. That may work for religion, but medical information given to patients should be based on primary sources and empirical studies.