Methods: Foundations of Responsible NLP Use for Maternal Health Equity

Context

During the May 2023 Maternal Health Equity Workshop: From Story to Data to Action hosted by the AAMC Center for Health Justice, workshop participants – including members of the public, maternal health care workers, medical providers, and allied health professionals – were introduced to the basics of Natural Language Processing (NLP), heard presentations about the applications of NLP methods to maternal health outcomes, and learned about the biases built into NLP tools. After these foundational sessions, an interactive session allowed the workshop participants to interact with a demonstration chatbot, and share their experiences with the demo and their overall perspectives about artificial intelligence and NLP.

Data Collection

Workshop participants voluntarily engaged with the chatbot demonstration. The activity created a structured interface around a chatbot, which included warnings about privacy and the errors these chatbots can make. Within the interface, the participants drew on their professional and personal experiences to write descriptions of situations in which a person might use a chatbot to get answers to any types of questions related to maternal health (medical/clinical or nonmedical). They could then query the chatbot and rate its responses.

Following this demonstration, a survey and breakout discussions asked participants to express their perceptions of NLP tools for maternal health care. The survey and discussion included questions about how NLP tools could affect maternal health care, health care and maternal support workers (doulas, midwives, etc.), the participant’s trust in these tools, their information seeking needs, and the values that should be considered and prioritized when using NLP methods for maternal health care.

Analysis Process

The workshop session included 236 participants, and 39 of these participants responded to the survey. After the workshop, this survey and user study were repeated with a set of 53 women and birthing persons who had given birth in the last five years, and the research team compared the results.

The research team, comprised of Maria Antoniak¹ Aakanksha Naik², Irene Y. Chen³, Lucy Lu Wang⁴, and Carla S. Alvarado⁵ identified the foundations through a multistep process that involved analyzing the survey responses and the queries posted that constituted the chatbot data using mixed methods. These include qualitative and quantitative methods like open-coding of the chatbot situations and queries, thematic analysis of the discussion recordings, and summary figures of the survey results.

The team also drew on its members’ expertise in maternal health, NLP methods and their applications to different areas of health care (including information extraction from medical notes and scholarly documents), modeling of interactions in online health care support communities, and ethical frameworks for the use of machine learning in health care. The team also conducted a literature review of maternal health care research that has used NLP methods, characterized the focus of past research, and identified challenges and areas of opportunity for future work.

Resources

1. Andalibi N. Symbolic annihilation through design: Pregnancy loss in pregnancy-related mobile apps. New Media & Society. 2021;23(3):146144482098447. doi:10.1177/1461444820984473.
2. Association for Computing Machinery US Public Policy Council. Statement on Algorithmic Transparency and Accountability; 2017. Accessed Oct. 25, 2023. https://www.acm.org/binaries/content/assets/public-policy/2017_usacm_statement_algorithms.pdf.
3. Blodgett SL, Barocas S, Daumé III H, Wallach H. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. ACLWeb. doi:10.18653/v1/2020.acl-main.485.
4. Chen IY, Pierson E, Rose S, Joshi S, Ferryman K, Ghassemi M. Ethical Machine Learning in Healthcare. Annual Review of Biomedical Data Science. 2021;4(1):123-144. doi:10.1146/annurev-biodatasci-092820-114757.
5. Field A, Blodgett SL, Waseem Z, Tsvetkov Y. A Survey of Race, Racism, and Anti-Racism in NLP. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021;1. doi:10.18653/v1/2021.acl-long.149.
6. Hovy D, Spruit SL. The Social Impact of Natural Language Processing. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2016;2. doi:10.18653/v1/p16-2096.
7. Jakesch M, Buçinca Z, Amershi S, Olteanu A. How Different Groups Prioritize Ethical Values for Responsible AI. 2022 ACM Conference on Fairness, Accountability, and Transparency. Published online June 20, 2022. doi:10.1145/3531146.3533097.
8. Klein LF, D’Ignazio C. Data Feminism. The MIT Press; 2020.
9. McMahon C, Johnson I, Hecht B. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. Proceedings of the International AAAI Conference on Web and Social Media. 2017;11(1):142-151. doi:10.1609/icwsm.v11i1.14883.
10. Oprescu AM, Miró-amarante G, García-Díaz L, Beltrán LM, Rey VE, Romero-Ternero Mc. Artificial Intelligence in Pregnancy: A Scoping Review. Institute of Electrical and Electronics Engineers Access. 2020;8:181450-181484. doi:10.1109/ACCESS.2020.3028333.
11. Yang Q, Hao Y, Quan K, et al. Harnessing Biomedical Literature to Calibrate Clinicians’ Trust in AI Decision Support Systems. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Published online April 19, 2023:1-14. doi:10.1145/3544548.3581393.