Fred Milacci, Jennifer Smith Ramey and Fred Volk
Liberty University and Horizon Behavioral Health (US)
Transforming Your Approach to Formative Program Evaluation: Harnessing the Power of Qualitative Methods for Effective Implementation
Theme: This engaging workshop will provide an immersive experience in understanding how qualitative methodological approaches can significantly enhance the process of formative program evaluation. Participants will gain valuable insights into assessing program effectiveness while it is actively in implementation. This method not only identifies areas for early improvements but also critically assesses the quality of the program’s design and its implementation strategies to ensure they are aligned with the intended goals.
Grounded in adult learning theory, the workshop will spotlight the essential role of qualitative interviews in formative evaluation, focusing on how these conversations can uncover the program’s strengths, needs, and gaps.
Objective: The primary goal is to equip participants with practical tools, such as a customizable interview guide template, that can be tailored to suit a range of program evaluations, ensuring versatility and practical application in various contexts.
Strategy: The workshop will seamlessly blend didactic instruction with experiential learning. It will feature a practical example of a qualitative evaluation, allowing participants to see theory in action. Small group practice will provide an interactive environment for participants to develop key components of an evaluation, including crafting an evaluation purpose statement, formulating evaluation research questions, and creating a tailored interview guide.
Expected Results: Participants will emerge equipped with essential techniques and skills to conduct a robust formative program evaluation using qualitative methodologies. They will leave with a deeper understanding of how to effectively gather and analyze qualitative data to enhance program outcomes, ready to implement these practices in their own work.
Keywords: formative program evaluation, implementation, qualitative interviews
Online Workshop | See the full program
Judita Kasperiuniene
Vytautas Magnus University (LT)
Pablo Burneo
CATENA-USFQ (EC) and University of Aveiro (PT)
António Pedro Costa
University of Aveiro (PT)
Exploration of AI-Powered Mixed-Methods Approach as Methodological Literature Reviews
Abstract. Methodological literature reviews often involve challenges like managing large datasets, identifying patterns, and ensuring rigour. Most methodologically based literature reviews focus on either quantitative or qualitative approaches only. The integration of the two approaches with the triangulation of Generative AI tools is still quite recent and requires Human-AI collaboration skills. For example, the Bibliometric analysis (quantitative) allows researchers to map trends and clusters in academic publications, while the framework analysis technique (qualitative) allows researchers to systematically organise, structure, and interpret qualitative data. The Large Language Models (LLMs) such as Gemini and ChatGPT, through chain-of-thought prompting, researchers can perform exploratory analyses on datasets, for example, to identify the first categories. The articulation of these different dimensions requires proposed theoretical models that can be applied in real contexts.
The proposed workshop will engage participants in participatory tasks. We will present on a practical QUAL-quant (qualitative-dominant) mixed-methods approach integrating the bibliometric analysis in the framework analysis technique. The participants will be writing a Chain-of-Thought prompting and use it with tools like Gemini or ChatGPT to explore datasets and full papers that will be shared with participants.
The workshop will allow participants to understand the importance of human-AI collaboration, to triangulate different types of tools and do teamwork tasks. Through hands-on activities and dynamic discussions, participants will leave with actionable insights to integrate AI into their own research.
Keywords: Bibliometric analysis, Chain-of-Thought Prompting, Mixed-Methods Research, Generative AI, webQDA.
Necessary resources: Access to ChatGPT (preferably GPT-4). Workshop materials will be provided, including sample datasets and practical guides.
Online Workshop | See the full program