Reseach Article
Wiki-Constrained Offline Reinforcement Learning for Strategy Recommendation in Health Support Dialogue: A Proof-of-Concept Study
| Journal of Advanced Artificial Intelligence |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 2 - Number 7 |
| Year of Publication: 2026 |
| Authors: Bailing Zhang, Ruofan Zhang |
Bailing Zhang, Ruofan Zhang . Wiki-Constrained Offline Reinforcement Learning for Strategy Recommendation in Health Support Dialogue: A Proof-of-Concept Study. Journal of Advanced Artificial Intelligence. 2, 7 ( Oct 2026), 45-52. DOI=None
Abstract
Aim: To develop and evaluate a proof-of-concept framework, Wiki-Constrained Conservative Hyperbolic Implicit Q-Learning (Wiki-CH-IQL), that integrates clinical strategy constraints from a persistently maintained Therapeutic Constraint Wiki (TC-Wiki) into an offline reinforcement learning (RL) system for strategy recommendation in health support dialogues, with particular focus on the Dynamic Constraint Update (DCU) mechanism that enables guideline revision without model retraining. Materials and Methods: TC-Wiki stores stage-binding, sequence, contraindication, and frequency constraints as structured pages with evidence scores and conflict flags, extracted from clinical counselling texts using a large-language-model pipeline (W2C). Wiki-derived penalty terms modify the offline RL reward without architectural change. Experiments used the ESConv emotional support dialogue dataset (1,300 dialogues; 11,899 training, 2,518 test transitions). A controlled ablation compared the full system (penalty weight λc = 0.5) against the unconstrained baseline (λc = 0). A DCU experiment simulated guideline revision by upgrading a single constraint from disputed to confirmed status. Results: Wiki-CH-IQL (λc = 0.5) achieved a 24.48% match rate with expert behaviour and 16.47% macro recall. The ablation (λc = 0) yielded 24.32% and 16.23%, confirming that wiki penalties contribute consistently but marginally (+0.16 pp match rate, +0.24 pp macro recall). Strategy shifts toward questioning (+49.9 pp) and suggestions (+11.7 pp) were consistent with Helping Skills Theory. The DCU experiment demonstrated a 2.3× amplification of the safety gap without retraining. Policy concentration on two strategies was present in both conditions, confirming collapse as a base RL training issue. Conclusions: The proof-of-concept demonstrates that clinical counselling guidance can be partially formalised as an updatable constraint wiki and integrated into offline RL. The DCU mechanism is the primary practical contribution, enabling knowledge base revisions to propagate to policy evaluation without retraining. Limitations include simulated constraint sources and policy collapse from the base RL training. These define concrete next steps toward clinician-validated health dialogue decision support systems.
References
- Hill CE. Helping Skills: Facilitating Exploration, Insight, and Action. 4th ed. Washington, DC: American Psychological Association; 2014.
- Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. 2017;4(2):e19. doi:10.2196/mental.7785
- Grodniewicz JP, Hohol M. Waiting for a digital therapist: three challenges on the path to psychotherapy delivered by artificial intelligence. Front Psychiatry. 2023;14:1190084. doi:10.3389/fpsyt.2023.1190084
- Cheng Y, Liu D, Li S, Chen Z, Wang Q, Zhang Y. Improving multi-turn emotional support dialogue generation with lookahead strategy planning. In: Proceedings of EMNLP 2022. 2022:3014–3026.
- Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit Q-learning. In: International Conference on Learning Representations; 2022.
- Kumar A, Zhou A, Tucker G, Levine S. Conservative Q-learning for offline reinforcement learning. Adv Neural Inf Process Syst. 2020;33:1179–1191.
- Zhang B, Zhang R. Hierarchy-aware hyperbolic representation with conservative offline policy learning for interpretable strategy recommendation in mental health dialogues. Preprint. 2025.
- Zhang B. LLM wiki as a living constraint oracle for safe offline RL: evidence from type 1 diabetes glucose management. Transactions on Artificial Intelligence. 2026; 2 (1), 178–189.
- Beck AT, Rush AJ, Shaw BF, Emery G. Cognitive Therapy of Depression. New York: Guilford Press; 1979.
- Miller WR, Rollnick S. Motivational Interviewing: Helping People Change. 3rd ed. New York: Guilford Press; 2013.
- Liu S, Zheng C, Demasi O, Sabour S, Li Y, Yu Z, et al. Towards emotional support dialog systems. In: Proceedings of the 59th Annual Meeting of the ACL; 2021. p. 3469–3483. doi:10.18653/v1/2021.acl-long.269
- Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459–9474.
- Nickel M, Kiela D. Poincaré embeddings for learning hierarchical representations. Adv Neural Inf Process Syst. 2017;30.
- Levine S, Kumar A, Tucker G, Fu J. Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv:2005.01643; 2020.
- Kang D, Kim S, Kwon T, Moon S, Cho H, Yu Y, et al. Can large language models be good emotional supporters? Mitigating preference bias on emotional support conversation. In: Proceedings of ACL 2024; 2024. p. 15232–15261. doi:10.18653/v1/2024.acl-long.813
Index Terms
Keywords