Cartoon Versus Traditional Self-study Handouts for Medical Students

Cartoon versus traditional self-study handouts for medical students: CARTOON randomized controlled trial


Abstract

Objective: The objective of this study is to compare the effectiveness of a “cartoon-style” handout with a “traditional-style” handout in a self-study assignment for preclinical medical students.

Methods: Third-year medical students (n = 93) at the Faculty Of Medicine Ramathibodi Hospital, Mahidol University, took a pre-learning assessment of their knowledge of intercostal chest drainage. They were then randomly allocated to receive either a “cartoon-style” or a “traditional-style” handout on the same topic. After studying these over a 2-week period, students completed a post-learning assessment and estimated their levels of reading completion.

Results: Of the 79 participants completing the post-learning test, those in the cartoon-style group achieved a score 13.8% higher than the traditional-style group (p = 0.018). A higher proportion of students in the cartoon-style group reported reading ≥75% of the handout content (70.7% versus 42.1%). In post-hoc analyses, students whose cumulative grade point averages (GPA) from previous academic assessments were in the middle and lower range achieved higher scores with the cartoon-style handout than with the traditional one. In the lower-GPA group, the use of a cartoon-style handout was independently associated with a higher score.

Conclusions: Students given a cartoon-style handout reported reading more of the material and achieved higher post-learning test scores than students given a traditional handout.


Materials and methods

Study settings and participants

The cartoon versus traditional self-study handouts for medical students (CARTOON) trial was a parallel-group RCT undertaken at the Faculty Of Medicine Ramathibodi Hospital, Mahidol University in Bangkok, Thailand. All third-year medical students in 2015, in their final year of pre-clinical placements, were eligible for the study. Although not previously given any learning exercise concerning intercostal drainage, they had studied the physiology necessary to understand the topic. Students volunteered to participate after receiving an explanation of the study in class. The study was approved by the Ethics Committee of the Faculty Of Medicine Ramathibodi Hospital, Mahidol University.

Intervention and comparison

The two types of handout on the topic “Basics of intercostal chest drain (ICD) system: one and two-bottle ICD” were developed by the first author (D. J.). These handouts described the basic anatomy and physiology of the pleurae, the causes of pneumothorax, the working principles of intercostal drains, and the basic circuits of one and two bottle ICD systems. Sample pages from both handouts can be seen in Figure 1 and full versions can be accessed via the following link: http://tiny.cc/cartoon_trial (available in Thai and English).

Figure 1. Sample pages from the “Cartoon-style” handout (left) and the “Traditional-style” handout (right).




The study intervention was the “cartoon-style” handout. This was a series of hand-drawn pictures, 23 pages in length. These represented anthropomorphic animal characters which interacted with each other following a storyline. In the story, the characters encountered a pneumothorax, developed a solution using basic physiologic principles, and managed additional problems occurring afterwards. Physiological aspects were described by using cartoon pictures with metaphors, e.g. using a stream and a length of tube to illustrate the principle of an underwater seal and comparing the fluctuation of water in a tube with a straw sucking water from a glass. The knowledge content was conveyed through diagrams, narrative, conversations between the characters (speech balloons) and by using additional text beneath each picture. The artwork was created on paper and scanned into a computer for coloring with Adobe Photoshop CC (Adobe Systems, Inc., San Jose, CA). The language and fonts were of informal style, some being hand-written, to match the nature of the cartoon. Thai was used as the main language, except for specific medical terms which were in English.

For comparison, a “traditional-style” handout was created as a 10-page text-based document with computer-generated color figures using Word 2013 and PowerPoint 2013 (Microsoft, Corp., Redmond, WA). Content was presented by mainly using text and diagrams. Unlike the cartoon-style version, no metaphoric images were used and comparisons were described in the text. Formal language and fonts were used throughout this form of the handout.

Quality and comparability of handouts
To confirm face and content validity of the two handouts and to verify that they were comparable, an independent expert panel (consisting of two respiratory physicians, a cardiothoracic surgeon, an emergency medicine physician and a family physician) checked that both handouts were equivalent in terms of information accuracy and comparability of contents. This included ensuring that all content presented in the cartoon version was also present either in the text or as figures or diagrams in the traditional version. Each of the experts qualitatively assessed the equivalence of both versions, resulting only in minor suggestions about formatting e.g. choice of coloring and clarity of the lines drawn. We also asked a group of third year medical students (n = 21) from the Praboromarajchanok Medicine Programme, Mahidol University, who were not involved in the study phase, to give qualitative feedback on the clarity of the handouts in conveying key concepts and on any issues raised from reading them. This feedback was used to improve the final version of both handouts. Additionally, we conducted a pilot RCT with 152 second-year medical students in a controlled-environment self-study assignment. There were no significant differences in post-learning assessment scores between the two groups using different handouts (see eMethods in Supplemental material).

Study procedure
After giving written informed and signed consent, participants took a pre-learning multiple-choice question (MCQ) test. All participants were advised that no part of the test would be used for their formal academic assessment. A lapse period of 2 weeks was then allowed to minimize any carry-over effect. Students were then allocated to two groups using a computer-generated simple randomization program and were given either a cartoon-style or traditional-style handout in a sealed envelope. Participants were told they had two weeks to study the material individually, whenever it suited them. The use of other learning materials was discouraged and was to be reported. After the learning period, students returned to take a post-learning test using the same MCQ used in the pre-learning test.

Data collection
We collected baseline characteristics of the study participants i.e. age, sex, and cumulative grade point average (GPA). Student preferences for reading cartoons in daily life were rated using a 5-point Likert scale (strongly unfavored, unfavored, neutral, favored, strongly favored).

Outcome measurements
Primary outcome
The primary outcome was the post-learning MCQ test scores. The pre-test and post-test scores from the newly created 20-question MCQ were used to assess participants’ recall of knowledge, their understanding of the information, and their ability to apply knowledge about ICD (stated as learning objectives on the first page of both handouts). The expert panel were asked how well our MCQs sampled the range of content knowledge and the various cognitive levels that students were supposed to acquire. They were also directed to ensure that the multiple choice questions were related to topics included in our handouts or could be answered by applying concepts stated or implied in the learning materials. The MCQ paper was then tested by the group of students from the Praboromarajchanok Medicine Programme previously described. After further adjustments, the MCQs were piloted in a study with second-year medical students (see eMethods in Supplemental material) before being used in the current RCT. Cronbach’s alpha was calculated from post-learning test scores as a measure of the internal consistency reliability of the test. An English version of the MCQ can be seen in eMaterials (Supplemental material).

Secondary outcome
We wanted to compare the attention given to the study materials by assessing levels of reading completion in the two groups of students. Each participant was asked to self-report his or her level of reading completion using a 5-point Likert scale (0%,<25%,<50%,<75%, and ≥75% of all handout content). Attention differences between the two arms of the trial were measured by comparing the proportion of students who reported having read ≥75% of the assigned handout contents.

Sample size calculation
We calculated the sample size from the mean post-learning test score of the control group in the pilot trial, i.e. 15.69 (SD 2.26). To detect a 10% difference between the post-learning test scores of the groups, with a 2-sided type I error rate of 0.05 and power of 80%, we needed 72 participants (i.e. at least 90 students, allowing for an estimated loss-to-follow-up rate of 20%).

Data analysis
Continuous parameters were presented by mean (with standard deviation) and categorical data were presented by frequency (percentage). For primary outcomes, we used the unpaired T-test to compare the differences between the post-learning test scores of the groups. For secondary outcomes, attention differences were analyzed with Chi-square. A two-sided test with p value <0.05 was considered to be statistically significant. Data were analyzed using SPSS version 16.0 for Windows (SPSS, Inc., Chicago, IL).

Analysis was performed on an intention-to-treat basis. We also asked participants about “cross-group” reading of un-assigned handouts (contamination), measured as the amount of read content using a self-reported 5-point Likert scale similar to the one used to assess the secondary outcome. If any participant reported having read >25% of unauthorized content, additional per-protocol analysis would be performed with the exclusion of the participant.

To discover whether or not our handouts were equally effective among students with different levels of academic achievement, we categorized students into three groups based on quartiles of their previous GPAs, namely an upper-GPA group (GPA above the third quartile), lower-GPA group (GPA below the first quartile), and middle group (GPA within the interquartile range). Post-hoc analyses were undertaken to compare post-learning test scores and attention differences between students in the same GPA group.

Results
Baseline characteristics

We recruited 93 volunteers from 179 eligible students. Seventy-nine participants attended the post-learning test session and were included in the analysis. There was no statistically significant difference in baseline characteristics between the two groups





Primary outcome: the post-learning test score

Before the self-learning session, there was no significant difference between the test scores of either group. After their learning sessions, all students had significant score improvements (paired T-test p values of <0.001 and 0.002 for the cartoon-style and traditional-style groups respectively). For the primary outcome, students using the cartoon-style handout performed better than those using the traditional-style handout, with a mean post-learning test score of 1.69 marks (13.8%) higher (p = 0.018) (Figure 3). Cronbach’s alpha for post-learning test scores was 0.663. One participant in the traditional-style group declared also reading the cartoon-style handout but only to an extent <25% of total content, lower than our predefined threshold for contamination. None of the students reported using additional learning materials related to our topic.

Figure 3. Pre-learning test and post-learning test scores of students receiving cartoon-style and traditional-style handouts. The numbers shown are mean (± SD) and the error bars demarcate the range of ± 1SD (Total test score: 20 marks).




Secondary outcomes: students’ attention and level of reading completion

More students in the cartoon-style group (70.7%) reported having read ≥75% of the assigned handout content compared with the traditional-style group (42.1%); p = 0.01 by Chi-square. The level of reading completion correlated significantly with the post-learning test score, i.e. the more content read, the higher the score achieved (Spearman’s rho = 0.553, p < 0.001). This was true for students receiving either type of material.

Figure 4. A histogram showing detailed levels of reading completion in each group.



Figure 5. Correlation between level of reading completion for all participants and the post-learning test score (Spearman’s rho = 0.553, p < 0.001).



Multivariate linear regression analysis of two factors associated with the post-learning test score (i.e. the type of handout being cartoon-style and the level of reading completion) showed only the level of reading completion to be independently associated with the post-learning test score with beta-coefficient of 1.07 (95% CI: 0.65, 1.50)

Outcomes for students with different previous academic achievements

There were no significant differences in baseline characteristic or pre-learning test scores for any of the student groups categorized by quartiles of GPAs (eTable 2 in Supplemental material). However, students in the lower and middle GPA groups who had been given the cartoon handout achieved significantly higher post-learning test scores than those who had used the traditional handout (Figure 6). Moreover, in these two groups, the number of students who reported having read ≥75% of the handout content tended to be higher for the cartoon material. The level of reading completion also correlated with the post-learning test score.

Figure 6. The post-learning test score of the students who read the cartoon-style and the traditional materials, categorized by the quartile of their GPAs. The numbers shown are mean (± SD) and the error bars demarcate the range of ± 1SD.



After adjusting for the level of reading completion by multiple linear regression, the use of the cartoon-style material by students in the lower-GPA group was independently associated with higher post-learning test scores (beta coefficient 2.47, 95% CI: 0.69-4.24). For the same level of reading completion, the use of cartoon-style material in this group of students would result in a post-learning test score 2.47 marks higher than would be achieved using the traditional handout. However, this result was not observed for students in the middle and upper-GPA groups, in which, the level of reading completion was solely an independent factor for the post-learning test score

Discussion

Our study found that students given a cartoon-style handout for self-study achieved higher post-learning test scores and reported higher levels of reading completion of the material compared with students given a traditional handout. The level of reading completion was positively associated with post-learning test scores. Interestingly, in the post-hoc analyses, students with previous GPAs in the lower and middle range achieved significantly higher post-learning test scores using the cartoon-style handout than those assigned to read the traditional-style handout. This effect was strongest in students in the lower-GPA group and independent of the reported level of reading completion.

The results of our RCT support the findings of previous observation and quasi-experimental studies showing that cartoon-style handouts can help learners to understand complex medical content. Evaluation of anatomy comic strips for further production and applications. Improving health Professional's knowledge of hepatitis B using cartoon based learning tools: a retrospective analysis of pre and post tests. Using comics for pre-class preparation. The use of educational comics in learning anatomy among multiple student groups. This suggests that using carefully designed cartoons can equal or better the results achieved with traditional materials. There could be two explanations for our observed outcomes. First, cartoons may be better able to attract and sustain a reader’s attention throughout a reading session. In our research, students who received a cartoon handout reported higher levels of reading completion than students using traditional materials. Furthermore, linear regression analysis showed an association between levels of reading completion and higher post-learning test scores. This supports the results of previous studies demonstrating an increase in motivation and interest when cartoon-based educational materials were used. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Improving health Professional's knowledge of hepatitis B using cartoon based learning tools: a retrospective analysis of pre and post tests. Second, a cartoon-style handout may be better able to aid student understanding and retention of complex medical concepts through three mechanisms.

“Metaphors and analogy”. These help students to connect the concrete with the abstract, prior knowledge with unfamiliar concepts, and language with images. Metaphors can also promote assimilation and “structurizing”. This provides a scaffold to aid memorization of content, with cartoon characters making the material more concrete. Stories not only stimulate cognition but also engage the readers’ emotions, including imagination and the recall of past experiences. Emotions provoked during academic sessions have been shown to be related to motivation, self-regulation, and achievement of learning. Cartoon characters and a storyline may help students to engage emotionally with educational content, motivating them to continue reading and perhaps aiding memorization.

The benefits of the cartoon handout seen in our main RCT were not duplicated in our pilot study, which used the same set of handouts but resulted in similar post-learning test scores for both groups. However, the two studies had important differences. The pilot study was performed in a classroom setting with a fixed period of learning time whereas our main study allowed students to use the materials out-of-class and in their own time. We conclude that cartoons may be more effective when used for flexible home study than in time-limited classroom settings.

When exploring how different handout styles affected students with different levels of academic achievement, we found that students in the lower and middle GPA levels learned significantly more when reading a cartoon-style handout while students in the upper GPA group performed equally well with cartoons or traditional material. Our results imply that the cartoon-style handout particularly helped students in the lower academic achievement group. Some students in this group may have difficulties dealing with the complexity and quantity of information taught in medical school when this is presented only in a traditional way. As mentioned previously, a well-designed cartoon-style handout may be able to aid understanding in ways that a traditional text handout cannot.

To the best of our knowledge, this is the first RCT to comprehensively address the effectiveness of a cartoon-style handout in the area of medical education. Students with no pre-existing knowledge of ICD were randomly allocated to intervention and control groups, thus minimizing selection and confounding bias. As self-study sessions can be confounded by multiple and uncontrollable environmental factors, we designed and conducted a pilot RCT in the controlled environment of a classroom in order to test the equivalence of efficacy of the two types of handout before using these in our main study.

Despite these strengths, our study has some important limitations. First, participants were drawn from a single medical school, thus limiting generalizability for students at other levels and in other settings. Second, we were not able to blind participants to their assigned group and the novelty of the cartoon handout may have resulted in it receiving greater student attention. Third, all participants were Thai and their perceptions of cartoon-based material may differ from those in other countries and cultures. According to a survey of the Publishers and Booksellers Association of Thailand, cartoons and graphic novels are the most widely read categories of book for Thai adult reading (PUBAT 2015PUBAT. 2015. The study of books reading and buying behaviors in Thai adults. [Internet]. Thailand: The publishers and booksellers association of Thailand. Available from:http://www.pubat.or.th/index/topic/14a28607f879b38e6d64578ee26a46b2 ). Although the relevance of our trial results to other parts of the world and to different reading cultures is at present unknown, the potential for suitably adapted cartoons to be an effective cross-cultural tool remains. Lastly, our MCQs may have limitations when used to assess higher learning objectives such as comprehension, application and analysis. Evaluating the translation of learning into practice may require a different assessment tool but this is beyond the design of our present study. Given that our multiple choice questions tested different aspects of the topic and different levels of learning, inherent heterogeneity in the test led to relatively low Cronbach’s alpha internal consistency reliability, especially with the low sample size of the study. Nevertheless, Cronbach’s alpha for the main study was sufficient (Nunnally 1967Nunnally JC. 1967. Psychometric theory. 1st ed. New York: McGraw-Hill. [Google Scholar]). In addition, as we had not pre-defined GPA subgroups at the outset of our study, differences in the effectiveness of cartoons among students in different quartiles of GPA should be interpreted with care. However, our post-hoc analyses raise the possibility that cartoon material may achieve different responses from students of varying academic ability. Differences in outcome could also relate to learning-styles, which were not explored in this study. These factors should be probed in future research, with pre-planned subgroup analyses and larger sample sizes.

The traditional academic style currently in use to present information may not be appropriate for every student. This study gives a new and positive message for the use of cartoon materials in teaching and learning. For most medical students, carefully designed cartoon-style handouts can transfer core medical knowledge as well as, if not better than, traditional handouts. Developing innovative approaches to learning may result in better outcomes, especially for those deemed previously to have below average academic ability. Creating high-quality cartoon material may seem daunting but, in our view, the most important issue is not necessarily the need for an artist with medical knowledge. Any medical teacher with the ability to tell a story can work alongside an artist to produce cartoon lessons. Our experience also shows us that talented medical students with a passion for drawing cartoons represent an untapped resource. From an academic perspective, future research is needed to evaluate the use of cartoons for a variety of learning outcomes and to establish long-term results. The influence of preferred learning styles and the impact of geographically different reading cultures should also be taken into account. Finally, we do not know what style of cartoon or what sort of characters is likely to achieve the best results.

Conclusions

Our RCT found that a cartoon-style handout was able to attract more of a student’s attention and result in a higher post-learning test score when compared with a traditional handout. This effect appeared to be most marked in students with previously low academic evaluations and was independent of other factors. We hope that our study will encourage medical educators to consider using cartoon-style materials and to undertake further research into their effectiveness and popularity with students.

Source