Before any smart feature in a health app: the six guardrails we never negotiate

We have built a range of health platforms: Reayah for home care, Women Health for holistic wellness, PharmaStore as an online pharmacy, and Qaweny for coaching and nutrition. On every one we were asked for “an AI feature”. On every one we started with the guardrails first. This article is about the guardrails.

1. AI guides, it does not diagnose

In Women Health the assistant answers “what are the common symptoms of…” and politely declines “do I have…”. The difference is not linguistic; it is a boundary of responsibility. Every health answer ends with a clear path to a medical consultation inside the app itself.

2. A prescription is a documented human decision

In PharmaStore an uploaded prescription is read automatically to speed up entry, but the pharmacist approves. The AI says “I believe this is drug X at dose Y — please confirm”. There is no path to dispensing medicine without a human touch, whatever the model’s confidence.

In healthcare, a feature that saves two seconds and is wrong once in a thousand is not a feature.

3. Critical cases bypass the model

Words like “chest pain”, “bleeding” or “not breathing” trigger a fixed, hand-written path: emergency numbers, the nearest facility, and a supervisor notification. We do not give the model creative freedom in the moment that cannot tolerate creativity.

4. Transparency of source

Every piece of medical information the assistant offers carries its source (an approved guideline, the drug leaflet, the facility’s protocol). In Qaweny, generated nutrition plans are reviewed by a certified specialist before the user sees them, and the specialist’s name appears on the plan.

5. Privacy by default

• Patient data is not used to improve models without explicit, separate consent.
• A log of who saw what, available to the patient themselves.
• Deletion is real, not “hiding”.

6. Measuring after launch matters more than testing before it

In Reayah we review a weekly sample of assistant conversations with caregivers: did it overstep? Did it miss a handover? Did it misread a dialect? The findings feed a rulebook that grows every month — and written rules are applied before the model, not after.

Why we insist

Because the user of a health app is not a “user”; they are a person who is scared, tired or ill. AI can be kinder and faster than any previous system — provided it knows exactly where its role ends and the doctor’s begins.