1

5

Quantitative Research Study Report

Insert Your Name Here

School of Public Service and Education, Capella University

EDD8040: Research Design for Practitioners

Insert the Instructor’s Name Here

Insert the Due Date Here (Month, Day, Year)

Introduction

Research Theory Framework

1.    What were the key concepts of the research framework that supported the development of the research questions?

2.    What are your reflections on the connections between theoretical or conceptual frameworks and research questions as they relate to developing an Applied Improvement Project (AIP)?

Methodology

3.   Describe the intervention?

 

Results

 

4. From the results, describe the answers to each of the four research questions and the rationale or supporting evidence for those answers.

 

Discussion/Conclusion

5. What ideas, concepts, or processes from this study did you find interesting or useful and how might those ideas, concepts, or processes be incorporated into an applied research project.

References

Chen, M., Hwang, G., & Chang, Y. (2019). A reflective thinking-promoting approach to enhancing graduate students’ flipped learning engagement, participation behaviors, reflective thinking and project learning outcomes. British Journal of Educational Technology, 50(5), 2288-2307.https://doi-org.library.capella.edu/10.1111/bjet.12823

Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.

1

2

Quantitative Research Study Report

Insert your Name Here

School of Public Service and Education, Capella University

EDD8040: Research Design for Practitioners

Insert the Instructor’s Name Here

Insert the Due Date Here (Month, Day, Year)

[Important Writing Instructions]

[This assignment needs be written in the third person voice. Do not write in the first-person voice (I . . .). There should be none of you and your voice in this assignment or the course project. However, for those questions that ask you your opinion or how something applies to your Applied Improvement Project, you can answer in the first-person voice. Do not use awkward language such as The researcher . . . or The learner when referring to yourself. Do not refer to yourself unless you are answering those questions that ask you your opinion or how something applies to your potential Applied Improvement Project (AIP). Do not write in the second person voice (writing that uses or implies the language you or your).]

[Always present the study and other literature with past tense verbs (APA 7th ed. section 4.12 pp. 117-118); for example, Chen et al. (2019) conducted . . .]

[Scholarly writing is meant to be read and interpreted literally. Therefore, please avoid slang, colloquialisms, anthropomorphisms, and conversational writing (refer to APA 7th ed. pp. 113-125). Instead, be clear, precise, and accurate in your writing.]

[At the doctoral level, most of your writing should involve summarizing or paraphrasing the literature. However, for an assignment like this one in which you conduct an in-depth review and analysis of a single study, there will be instances when you will likely use direct quotes. For direct quotes with fewer than 40 words, put quotation marks around the quoted text and include within the in-text citation, the author’s name, year, and page or para. number from which the quote came. For direct quotes with 40 or more words, put in block format (See APA 7th ed. pp. 272-273 for guidance and examples) and include within the in-text citation, the author’s name, year, and page or para. number from which the quote came.]

[There might be instances in which you use a direct quote that came from the article’s literature review. If the article’s authors use a quote or cite another author and you want to use that text as a direct quote, be sure to quote your article’s authors as the secondary sources (see APA 7th ed. p. 258).]

[Do not write with bullet points. Instead use complete sentences developed within coherent paragraphs. Use transitional language to smoothly move the flow of the thought along throughout.]

[Apply APA formatting rules and adhere to APA writing style guidelines.]

[Here are two important self-assessment final steps to help ensure you do as well as you can with the assignment: When you complete your draft, read it aloud to yourself. This step can catch typos, grammar errors, awkward writing, etc. For a final step – self-assess your assignment by reviewing the corresponding scoring guide and compare the proficient and distinguished columns criteria to your draft and revise as necessary.]

Introduction

[Briefly identify the quantitative study by following APA writing style, which means citing the author’s last name and year of publication. When identifying and discussing the study do not include the article’s title in your text as that is not how APA style writing is done. The title can be found in the reference citations. Instead follow APA writing style and include only the author’s last name and the year the article was published when referring to the study.]

[When describing the study do not write with anthropomorphisms. An anthropomorphism is when you assign human capabilities to a study or other non-human entities. Here is an example of anthropomorphism: The study found . . . Please note that studies are incapable of taking human action. Instead write, for example: Jones (2018) found . . . Avoid writing like this: The authors found . . . or The researchers found . . . Instead write, for example: Smith and Jones (2019) found . . . Smith and Jones (2019) explained . . . Note that in APA writing style you use past tense verbs to present or describe a study (see APA 7th ed. pp 117-118).]

[Follow APA writing style and identify the authors each time you refer to the study. Within a paragraph in which you cite the same author or authors more than once, include the year in the first citation and omit the year in subsequent citations within the paragraph (refer to APA 7th ed. section 8.16 pp. 265-266). ]

[Briefly describe the type of quantitative study; that is, what specific quantitative design was used (e.g., quasi-experimental, correlational, predictive correlational, comparative, descriptive survey, etc.). Describe the purpose of the study (what was the study designed to accomplish?). Identify the research question(s) and variables. Use direct quotes sparingly; however, if there is a research question, copy it and provide attribution via quotation marks and an in-text citation that includes the author, year and page or para. number. If there is a purpose statement, you can copy that as well provide attribution via quotation marks and an in-text citation that includes the author, year and page number. At the doctoral level your writing should be comprised primarily of summarizing and paraphrasing. If you must use a direct quote ensure the quoted text is in quotation marks followed by an in-text citation that includes the author’s name, year and page (refer to your APA manual 7th ed. Ch. 8 pp. 269-278). Conclude this section by analyzing how the Chen et al. (2019) study represents a quantitative design.]

Research Theory Framework

[Briefly describe the study’s theoretical or conceptual framework. What is theoretical or conceptual foundation for the study? In other words, what is the theory or previous literature that provides the basis or foundation for the study? Did the researcher test a hypothesis that stems from a theory or a model or a body of research or a combination of these components? What construct or constructs were measured? “A construct is a hypothetical concept that is typically developed from a theoretical framework. Constructs are names for things that cannot be seen (intelligence, motivation, self-esteem)” (Lodico, Spaulding, & Voegtle, 2010, p. 13). Constructs, concepts, and variables are derived from theories or models. Describe in detail the relationship among the theoretical of conceptual framework and the formation of measurement of variables? Please do not refer to the researcher’s hypothesis or expectation as a theory. To be a theory a theory must be published as a theory.]

[In addition, answer the following questions: Be sure to leave these questions (prompts) in the assignment. Within your assignment answer each question below the question.]

1.    What were the key concepts of the research framework that supported the development of the research questions? [Keep in mind that your answer to this question can be repetitive with the research theory framework described above. Ensure your response includes all components of the research theory framework and describe the support and development relationship between these key components and the research questions.]

2.    What are your reflections on the connections between theoretical or conceptual frameworks and research questions as they relate to developing an Applied Improvement Project (AIP)?

Methodology

[Identify the research design (e.g., correlational, quasi-experimental, etc.) and briefly describe the study’s sample (number of participants, where they were studied, and their demographics), the study's instruments, and the procedures used. Note how threats to validity and any legal or ethical issues or considerations were addressed, referring to the Creswell and Creswell (2018) text and/or your CITI training. Typically, researchers as authors will not state assertions in their journal articles reporting studies such as: Here is how I countered threats to validity or Here is what I did to address ethical considerations. Instead, they describe how they countered threat of validity and ethical considerations as a part of their methodology descriptions. Therefore, you need to be able to recognize how threats to validity were countered and ethical considerations addressed. Creswell and Creswell’s (2018) threats to internal and external validity (Ch. 8 Tables 8.5 and 8.6) apply to experimental studies only and not to other forms of quantitative research. In other non-experimental quantitative studies, internal validity is the summation of the reasons why the researcher/author was confident that any conclusions the researcher made were warranted from the data collected and the method used. Did the researcher(s) demonstrate that the methods used to collect and analyze the data lead to the trustworthiness of conclusions drawn? One potential threat to validity to consider for quantitative research is the validity of the instrument or instruments used (see Creswell & Creswell p. 153). How did the Kim describe the validity of the instrument? How do we know the instrument is valid (that is, it measures what it purports to measure)? If the validity and ethical issues and considerations were not discussed, that omission is a weakness and limitation in the study and indicate the omission and which of these components were missing. In reflecting demographics ensure you use properly formatted in text citations. When you cite figures from a study, include an in-text citation.]

[In addition, answer the following question: Leave this prompt (question) in your assignment and answer below it. ]

3.    Describe the intervention??

Results

[Include a comprehensive summary of the major findings of the study. Include key details. Remember – at the doctoral level you should use direct quotes sparingly because the bulk of your writing should consist of summarizing or paraphrasing. Describe the results that were found.]

[In addition, answer the following question: Leave this prompt (question) in the assignment and answer below it.]

4. From the results, describe the answers to each of the four research questions and the rationale or supporting evidence for those answers.

Discussion/Conclusion

[Describe how the Chen et al. (2019) connected the study's results to the literature presented in the literature review (the term systems literature refers to the related relevant literature presented in the study’s literature review). In other words, what concepts, assumptions and or expectations from the literature review were confirmed by the study’s results? Describe and analyze the strengths and weaknesses of the findings (results). How Chen et al.’ study be improved? Include your ideas for improving the study. Describe and analyze Chen et al.’s recommendations for future research and implications for practice. If Chen et al.’s recommendations and implications are missing or not sufficiently addressed in the study, indicate what is missing or not sufficiently addressed as part of your review and critique. Conclude by describing how Chen et al.’s findings could be applied to practice. In other words, what did your learn from Chen et al.’s study that could be applied by practitioners (organizational leaders)?]

[In addition, answer the following question: Leave this prompt (question) in the assignment and answer below it. ]

5. What ideas, concepts, or processes from this study did you find interesting or useful and how might those ideas, concepts, or processes be incorporated into an applied research project.

References

[Include a properly formatted list of references cited in this assignment. References go on a separate page. Include all references cited. Ensure references are in the hanging indent format and are properly APA formatted; refer to APA Publication Manual 7th edition (2020) Chapters 9 and 10 for guidance and examples. Do not include references contained with the Chen et al. article unless you have read these sources.]

Chen, M., Hwang, G., & Chang, Y. (2019). A reflective thinking-promoting approach to enhancing graduate students’ flipped learning engagement, participation behaviors, reflective thinking and project learning outcomes. British Journal of Educational Technology, 50(5), 2288-2307.https://doi-org.library.capella.edu/10.1111/bjet.12823

Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.

British Journal of Educational Technology doi:10.1111/bjet.12823

Vol 50 No 5 2019 2288–2307

© 2019 British Educational Research Association

A reflective thinking-promoting approach to enhancing graduate students' flipped learning engagement, participation behaviors, reflective thinking and project learning outcomes

Mei-Rong Alice Chen, Gwo-Jen Hwang and Yu-Ying Chang

Mei-Rong Alice Chen is a PhD candidate at the Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology. Her research interests include flipped learning and digital game-based learning. Gwo-Jen Hwang is a chair professor at the Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology. His research interests include mobile learning, digital game-based learning, flipped classroom and AI in education. Yu-Ying Chang is an assistant professor at the Department of English, Tamkang University, Taiwan. Her research interests include  EFL learning, flipped learning, and digital learning. Address for correspondence: Gwo-Jen Hwang, Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology, 43, Sec.4, Keelung Rd., Taipei, 106, Taiwan. Email: [email protected]

Introduction Flipped learning is an evolutionary educational approach that combines lecture-based and in- teractive teaching methods (Johnson & Renner, 2012; Strayer, 2012). It reverses the instruc- tional focus of teachers, helping students to foster lower level thinking skills (ie, remembering and understanding) at home and developing their higher order thinking skills (ie, analyzing, evaluating and creating) in the classroom (Anderson, Krathwohl, & Bloom, 2001; Bergmann & Sams, 2012). In recent years, millions of videos of online educational resources from subject experts on a wide range of topics have been made freely available online. This ever-increasing

Abstract Although flipped learning has been recognized as being a potential approach enabling students to learn at their own pace before the class and facilitating in-depth peer- to-peer and student-to-teacher interactions in the class, it remains a challenge to promote students’ active learning in the before-class stage, which could significantly affect their in-class engagement and learning performance. In this study, a reflective thinking-promoting approach is proposed to facilitate students’ learning design project performance, technology-enhanced active engagement, and their reflective thinking and participation in the before-class stage of flipped learning. A quasi-experiment was conducted on a flipped Digital Learning course of a Master’s program in a university to evaluate the effects of the approach on students’ learning design performance, engagement, reflective thinking and participation. A total of 19 students (7 male and 12 female) were in the experimental group learning with the reflective thinking- promoting approach, while 19 (4 male and 15 female) were in the control group learning with the conventional flipped learning approach. The results indicated that the proposed approach significantly enhanced not only the students’ learning design project outcomes and reflective thinking, but also their engagement and participation in the before-class stage of flipped learning.

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2289

access to lecture capture technology has made flipped learning much easier. Flipped learning is no longer a buzzword in education but a tendency and even a must (Bishop & Verleger, 2013; Hoffman, 2014; Lage, Platt, & Treglia, 2000). Many agree that it enhances learning practice and makes full use of the class time, focusing on student-centered interactive activities and individ- ual scaffolding (Bergmann & Sams, 2012; Tam, 2000).

Among various educational objectives, reflective thinking skills involve complex judgmental skills such as critical thinking and problem solving (Burton, 2010). These are indispensable skills in today’s world of technologies and information abundance. Therefore, it is believed that flipped learning must be able to help students attain these skills in all classrooms, at all education levels and in all subjects. To acquire reflective thinking in flipped learning, students need to fully par- ticipate and be engaged in designed pre-class and in-class activities and experience the process of remembering, understanding, analysis, synthesis, evaluation and finally conceptual change. This process is recognized as the development of reflective thinking (Atkins & Murphy, 1993;

Practitioner Notes

What is already known about this topic

• Flipping learning is an effective teaching approach that shifts the lecture time to the before-class stage and hence teachers have more time to conduct learning activities to promote students’ higher order thinking as well as to deal with individual students learning problems.

• Students’ learning experience, motivation and belief could be the factors that guide students towards engagement and participation in content and help them learn new skills.

• Engaging students in reflective thinking is an important and challenging issue. It provides students with an opportunity to scrutinize their own learning and hence make progress.

What this paper adds

• A reflective thinking-promoting approach into flipped learning is proposed to facil- itate students’ flipped learning engagement and participation behaviors as well as their project performance and reflective thinking.

• In addition to promoting students’ learning outcomes, the results indicated that the proposed approach provides promising results on the technology-enhanced active learning experience and participation in online learning in the before-class stage of flipped learning.

Implications for practice and/or policy

• Via monitoring students’ online before-class progress, instructors can recognize the factors that affect students’ learning, adjust or differentiate their instruction and even provide students with more opportunities or with additional support to meet students’ needs for learning.

• The link between the video lectures and the classroom activities can be examined in future research to perceive the influence of video lectures on students’ participation behaviors in-class activities.

• Forming reflective thinking skill is important, but attainable; it needs students’ en- gagement and participation in time and effort.

© 2019 British Educational Research Association

2290 British Journal of Educational Technology Vol 50 No 5 2019

Hong & Choi, 2015), a natural form of learning from experience and reflection. Moreover, it is an active, persistent and careful consideration (Porntaweekul, Raksasataya, & Nethanomsak, 2016). If students can be aware of and govern their learning by actively participating in reflective thinking, that is, if they can assess what they know, what they need to know and how they bridge that gap in learning contexts, learning will occur. Therefore, in this study, we aimed to examine the effect of a reflective thinking-promoting approach on students’ learning design performance and reflective thinking compared with conventional flipped learning. Meanwhile, students’ tech- nology-enhanced active engagement and participation in the learning management system (hereafter LMS) were scrutinized. Further, we hoped to build up a systematic module, leading to more replicable and sustained flipped classrooms. Four research questions are addressed:

1. Does the reflective thinking-promoting approach enhance students’ learning design performance?

2. Does the reflective thinking-promoting approach strengthen students’ engagement? 3. Does the reflective thinking-promoting approach boost the students’ reflective thinking? 4. How does the reflective thinking-promoting approach boost the students’ participation in the

before-class stage of flipped learning?

Literature review Flipped learning The flipped classroom refers to the teaching mode which reverses the traditional instruction by delivering teachers’ lectures at the pre-class stage in the form of instructional videos or other media to enable teachers to have more time in the class to help students do exercises and solve the learning problems they encounter (Bergmann & Sams, 2012). In addition, the Flipped Learning Network (https ://flipp edlea rning.org/) has stated the importance of conducting effec- tive “flipped learning” by taking four components (ie, flexible environment, learning culture, intentional content and professional educator) into account. Researchers have further indicated the need to employ effective learning strategies in flipped classrooms rather than only paying at- tention to the development of pre-class instructional videos to draw on learners’ active learning and engagement (Bishop & Verleger, 2013). Jensen, Kummer, and Godoy (2015) defined that ac- tive learning is using additional technology, teaching materials and peer instruction for mean- ingful learning experience. Instead of teaching, the focus should be on how to involve students in autonomous learning, that is “the involvement of students in activities and thinking about activ- ities.” In addition, Hung’s study (2015) indicated that active learning involves students in doing activities and in thinking about the information, they are learning. In her study, the experimen- tal group participants had also invested more out-of-class study time in demonstrating their learning engagement. Many educators and teachers have supported this revolution (Bergmann & Sams, 2012; Butt, 2014; Hamdan, McKnight, McKnight, & Artfstrom, 2013; Hwang, Lai, & Wang, 2015), whereas some have raised doubts about its effects (Hung, 2017; Johnson & Renner, 2012; Strayer, 2012), and some consider it time-consuming and tedious to prepare video record- ings and in-class activities. Nevertheless, it is believed that two major aspects can make flipped learning effective and practical: one is in-class activities that can trigger students’ engagement and the other is top-quality educational videos of an optimal duration that suits students’ atten- tion span.

The premise of flipping a class is the combination of digitally based lectures as pre-class tasks and learner-centered activities in reserved class time (Hoffman, 2014; Tam, 2000). Because of its blended features, the flipped classroom approach involves digital platforms, that is, learning

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2291

management systems, also known as LMSs (eg, Moodle and 1know) and real face-to-face human interaction (Bishop & Verleger, 2013). In the traditional teacher-centered learning theories, learners watch the instructional videos or read the assigned materials via the LMS out of class without any limitation of time and space; in contrast, in class, aside from assessing learners’ retention and understanding of the subject matter, teachers, based on learners’ diversity and on their knowledge and experiences, implement a learner-centered learning approach to have inter- active classroom activities (Soliman, 2016).

However, flipping a class does not really contribute to flipped learning. Flipped learning requires certain kinds of action (Hamdan et al., 2013; Hwang, Lai, & Wang, 2015). The Flipped Learning Network presented an effective flipped learning model with four components: flexible environ- ment, learning culture, intentional content and professional educator (Hung, 2017). There are still some implications worth restating. First of all, the learning environment has to remain flexible enough to create individual work areas, small group work spaces, and whole class mingling and demonstration stations (Bergmann & Sams, 2012). Next, a wide variety of learning experiences, approaches and academic-support strategies shift from the teacher-centered to learner-centered orientation, focusing on one-to-one differentiated instruction and immediate feedback in a com- fortable learning culture (Bergmann & Sams, 2012). A scaffolding effect on cognitive and social development occurs at any moment (Hamdan et al., 2013). Moreover, the video content is not simply “add-on” homework; it helps learners to build up a solid foundation for in-class activities and to develop their conceptual understanding and procedural fluency (Seaboyer, 2013).

Over the past decade, numerous studies have shown a sequence of positive results, including learn- ers’ academic performance (Bergmann & Sams, 2012; Sergis, Sampson, & Pelliccione, 2018), per- ceptions of engagement and learning skills (Elmaadaway, 2018), satisfaction (Bergmann & Sams, 2012), self-regulation (Lai & Hwang, 2016; Sun, Wu, & Lee, 2017), preferences (Bates & Galloway, 2012), technology acceptance (Kissi, Nat, & Armah, 2018) and so on. Nevertheless, the core value of flipped learning, that is, higher-order thinking skills, is less discussed and less satisfac- torily resolved. Flipped learning, not just revolving around lecture-based learning to attain lower thinking skills, that is, understanding and memorization, also encourages learners to develop higher order thinking skills in class: to apply, synthesize and even create knowledge (Seaboyer, 2013). Therefore, a sound, comprehensive flipped learning module will be outlined in this paper.

Technology-enhanced active engagement and participation Learning effectiveness is determined by students’ optimal engagement, and this engage- ment is achieved by factors that promote learning (Chuang, Weng, & Chen, 2018; Shernoff, Csikszentmihalyi, Schneider, & Shernoff, 2003). Students’ optimal engagement is their self-awareness of and their commitment to their own learning (Andrusyszyn & Yankou, 2004). Chen and Wu (2012), for instance, indicated that in learners’ experience, motivation and be- lief are the factors that guide learners towards engagement in content and help them learn new skills. The former is derived from the motive for learners’ actions, willingness and goals, whereas the latter is a firmly held opinion related to their learning strategies, academic perfor- mance and motivation (Chen & Pajares, 2010). In fact, two personal traits, namely motivation and belief, were examined and were found to be influential in learning in a flipped classroom context (Chuang et al., 2018). On the other hand, researchers (Ainley & Ainley, 2011; Shernoff & Csikszentmihalyi, 2009), have particularly pointed out that learning engagement is char- acterized by crucial factors such as learners’ concentration, interest and enjoyment. That is, in a powerful learning engagement, learners will be at a high energy level to engage in their learning and ultimately they have positive outcomes or results (Fink, 2003). However, with the

© 2019 British Educational Research Association

2292 British Journal of Educational Technology Vol 50 No 5 2019

rapid development and popularization of Web 2.0 and computer technology, these digital native students are faced with an unprecedented impact on their learning experience. According to Thompson (2013) and Fong and Wang (2007), digital natives’ learning motivation drops very fast when they are involved in reading. Their attention and interest reduces even more in ac- ademic subjects (Koutropoulos, 2011; Prensky, 2001). Therefore, technology-enhanced active learning has become particularly important.

The concept of active learning refers to students’ active engagement and participation in the learning process (Peng, Wang, & Sampson, 2017). It seizes the idea of learning by doing and eventually leads students to knowledge construction and continuous learning (Argote & Miron- Spektor, 2011; Pahl & Kenny, 2008). Technology is an effective tool that can facilitate the learn- ing process and consecutively create an active environment for learners to build their knowledge, skills, experience and engagement. For this reason, a reflective analysis of existing technology-en- hanced active learning engagement becomes crucial in this present study.

Learning engagement determines whether learners have learned throughout the course, but it is an individual attribute and needs to be examined (Felder & Brent, 2005; Ventura & Moscoloni, 2015). Based on the theory of Flow, proposed by Csikszentmihalyi (1990), Schwarzenberg, Navon, Nussbaum, Pérez-Sanagustín, and Caballero (2018) set up a more thorough, compre- hensive assessment model to measure learning experience. In this model, the constructs of learn- ing experience were adopted from Shernoff, Csikszentmihalyi, Schneider, and Shernoff ’s (2003) engagement, which consists of feedback, challenge, peer instruction, choice and enjoyment. In the construct of feedback, three dimensions are focused on: (1) the objectives and success, (2) self-current performance and (3) task completeness. Challenge, slightly beyond one’s current level of ability, has a discernible effect on academic performance. Peer instruction provides learn- ers with an opportunity to interact and learn from each other; it has a positive impact on learn- ing achievement. The perceived choice is related to one’s satisfaction and autonomous learning. Enjoyment refers to the satisfaction with the expected outcomes of the task.

In this study, we utilized Schwarzenberg and his colleagues’ (2018) experience model as a refer- ence to investigate learners’ engagement because it combines theories that describe the factors motivating learners and the conditions needed to generate the optimum engagement. With the help of the LMSs that monitor learners’ participation in the before-class stage of flipped learning, it is believed that technology-enhanced active engagement most likely represents a learner’s con- ceptions of how, when and where flipped learning does and can take place.

Reflective thinking Reflective thinking (also known as RT) is rational thinking realized by mental discipline (Kok, 2002). It is often used synonymously with critical thinking, but unlike critical thinking which includes various thinking skills leading to satisfying outcomes, RT puts more emphasis on the process of making decisions or stating opinions about what has happened (Evans, 2010; Schön, 2017). RT provides learners with a structured opportunity to scrutinize their own learning (Verpoorten, Westera, & Specht, 2011). During a reflective activity, learners can develop reflec- tive thinking skills by (1) relating new knowledge to previous understandings, (2) thinking in abstract and conceptual ways, (3) applying specific strategies in new tasks and (4) understand- ing their own ideas and thoughts (Hwang, Wu, & Ke, 2011).

RT can help learners to employ thinking skills such as analysis, synthesis and evaluation to reach a conceptual change (Atkins & Murphy, 1993; Hong & Choi, 2015). It is especially a critical factor in problem solving (Kok, 2002; Wang, Yuan, Kirschner, Kushniruk, & Peng, 2018). Although previous studies pointed out that RT is often associated with post-practice methods of experience

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2293

recapture through self-assessment, such as portfolios or learning diaries (Evans, 2010) or inter- active activities (Hwang et al., 2011), whether flipping learning activates RT skills and engages learners in active reflection needs to be further explored.

Many researchers privilege RT and believe it can promote reflection upon practice (Atkins & Murphy, 1993; Evans, 2010; Hong & Choi, 2015; Hwang et al., 2011; Kok, 2002), but not many of them can actually measure RT and tell whether people are meeting their goal of developing RT and further give explanations (Kember et al., 2000). Fortunately, Kember and his colleagues offered a prominent, detailed questionnaire for assessing different levels of reflective thinking. The questionnaire consists of four constructs, namely habitual action, understanding, reflection and critical reflection. Habitual action refers to learners’ automatic performance with little con- scious thought, whereas understanding means that learners can understand and comprehend a concept in academic learning. As for reflection, based on Dewey’s definition (1933, p. 9), learners engage in intellectual and affective activities, raise questions, test the assumptions during the process of problem solving, and finally gain new understandings and appreciations. The last is a critical reflection, part of a higher order of thinking. It is a reasoning process which finally reaches a perspective transformation. Having the identities of scales, the next step is to draw up the effect of a reflective thinking-promoting approach in flipped learning.

Ref lective thinking-promoting approach for f lipped learning The “reflective thinking-promoting approach” proposed in this study emphasizes the guided “reflection” in a project after engaging students in the learning process of watch-annotate, sum- marize-question, discuss-give feedback and reflect-project by referring to Rath’s (2014) WSQ (Watch-Summary-Question) framework and Bloom’s Taxonomy Matrix (Anderson et al., 2001) to serve as the theoretical framework of the flipped learning activities. The proposed approach is divided into watch-annotate and summarize-question in the pre-class stage, discuss-give feed- back in the in-class stage, and reflect-project in the after-class stage, with six cognitive process dimensions (remember, understand, apply, analyze, evaluate and create) and four knowledge di- mensions (factual, conceptual procedural, knowledge and metacognitive) of Bloom’s Taxonomy Matrix (Anderson et al., 2001), as shown in Figure 1. This is intended to enhance learners’ reflec- tive thinking with a positive learning experience.

Pardo et al. (2015) stated that video annotation could be a beneficial strategy to help learners achieve their learning outcomes. In conventional flipped learning, video watching as a pre-class activity is a less interactive and one-way form of learning (Lai & Hwang, 2016). Fu and Hwang (2018) stated that the adoption of technologies could promote learners’ learning engagement, reflection and reflective thinking. Because of more recent advanced technologies, considering the “1know” system (http://www.1know.net) as an LMS example as mentioned earlier, it can offer time-stamped annotation features, whereby learners can watch course videos and take notes. Meanwhile, learners can also pause, rewind, re-watch the videos. In addition, a reflective thinking-promoting approach was adopted not only to watch-annotate but also to summarize question as a series of tasks before class, and then the learners discussed the question selected by the instructor and then provided feedback in class. The purpose is to help the learners engage in reflective thinking based on Bloom’s Taxonomy Matrix. Accordingly, they were given a chance to develop reflective thinking skills (see Figure 2).

To evaluate the effect of incorporating the reflective thinking-promoting approach into flipped learning on learner’ learning performance, technology-enhanced active engagement, reflec- tive thinking and participation in the before-class stage of flipped learning, the reflective think- ing-promoting approach was implemented in an online learning management system (LMS), the

© 2019 British Educational Research Association

2294 British Journal of Educational Technology Vol 50 No 5 2019

1know system, to monitor learners’ progress, as shown in the color-coded task list in Figure 2. The colors help to keep learners alert and more interested in their own learning progress as well as their peers’, encouraging them to learn.

One of the most substantial advantages of using the LMS is that it can monitor learners’ learning progress on pre-class tasks, note taking and behavior (time spent on each task), but it can also gather data on students’ online learning behavior. It can also offer time-stamped annotation fea- tures whereby students can watch course videos and take notes. In addition, it allows students to propose questions (as shown in Figure 3).

Figure 1: Structure of a reflective thinking-promoting approach for flipped learning [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 2: Illustrative example of the flipped learning management system [Colour figure can be viewed at wileyonlinelibrary.com]

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2295

Experimental design To evaluate the effectiveness of the innovative flipped learning approach, an experiment was conducted on a digital learning course to evaluate the learning design performance, learner en- gagement, reflective thinking and participation in the before-class stage of flipped learning with two different flipped learning approaches. The objectives of the Analysis and Applying Strategies of Digital Learning Literature course are to foster students’ literature reading and analyzing competences as well as their abilities of applying digital learning strategies to educational set- tings and analyzing the learning effectiveness.

Participants A quasi-experimental design with a digital survey was carried out in an Analysis and Applying Strategies of Digital Learning Literature course at a technology university in northern Taiwan. A total of 38 graduate students (11 male and 27 female) participated in this study, of whom 19 (7 male and 12 female) were in the experimental group learning with the reflective think- ing-promoting approach and 19 (4 male and 15 female) were in the control group learning with the conventional flipped learning approach. In order to avoid the influence of different teachers on the experimental outcomes, the two classes were instructed by the same senior professor. The students in the experimental group learned with the reflective thinking-promoting approach incorporated into the flipped learning approach whereas those in the control group learned with the conventional flipped learning approach. During the pre-class learning activity, both groups were assigned to watch seven videos, take notes, write a summary and post a question, while the control group students were assigned similar activities, but note-taking and question-posing were not obligatory, so it was an option for them.

Experimental process In this study, the Analysis and Applying Strategies of Digital Learning Literature course was held for 3 hours a week over a period of 18 weeks. According to the aim of the graduate course, it is to prepare graduate students for advanced research, particularly for the graduate thesis and

Figure 3: Interface of question-posing after watching the instructional video lecture [Colour figure can be viewed at wileyonlinelibrary.com]

© 2019 British Educational Research Association

2296 British Journal of Educational Technology Vol 50 No 5 2019

doctoral dissertation (Austin, 2002; Boote & Beile, 2005). Based on the objective of Analysis and Applying Strategies of Digital Learning Literature course, 10 journal papers were assigned as course materials and seven instructional videos (as in Table 1) providing a starting point for students to cultivate their reading and research skills. Students were instructed to study vari- ous digital learning strategies (eg, peer assessment, project-based learning and problem-based learning) via literature review and analysis, and to analyze the subjects’ learning performances and patterns from various aspects by applying the strategies to practical educational settings. Journal readings were examples related to the learning design and research design methods introduced in the instructional videos, such as learning strategies and pedagogical theories. In class, the students were guided to implement learning design and research design as well as to analyze the data (eg, learning logs) provided by the instructor using the learning strategies and analysis methods learned in the pre-class stage. The instructional videos were developed by an experienced professor who had had more than 20  years’ experience teaching the course, and were reviewed by another experienced educational technology instructor and two flipped learn- ing experts recognized by FLGlobal.

In this course, the participants gave graded reports during the class to display their knowledge and to present all aspects of their original research ideas to fellow researchers in the field. The class was carried out in the read-and-present format. That is, the professor had a list of journal articles on the syllabus for each week related to the strategies of digital learning; each partici- pant studied one of the articles and then gave an in-class report. Each participant was asked to summarize the objectives and findings of the journal article during his or her report, and to dis- cuss ideas and provide their own reflections on the readings with other participants. During the

Table 1: Example of course content and videos

Week Journal papers and instructional video links Time

11 Paper 5: Improving learning achievements, motivations and problem-solving skills through a peer assessment-based game development approach

 

Video1: Peer assessment https ://www.youtu be.com/watch ?v=tAUqR wRgdE I&featu re=youtu.be

7:34

12 Paper 6: Effects of different online peer-feedback approaches on students' performance skills, motivation and self-efficacy in a dance course

 

Paper 7: A concept map-embedded educational computer game for improving stu- dents' learning performance in natural science courses

Video 2: Video sharing https ://www.youtu be.com/watch ?v=9d-EHp8q gbc&featu re=youtu.be

9:32

13 Paper 8: Effects of an integrated concept mapping and web-based problem-solving ap- proach on students' learning achievements, perceptions and cognitive loads

 

Video 3: Digital Media Design Principle of Design—cognitive load 3-1 https ://www. youtu be.com/watch ?v=gLj7Z-InR8M &featu re=youtu.be

16:32

Cognitive Theory of Multimedia Learning 3-2 https ://www.youtu be.com/watch ?v=a7-eEDVR pvs&featu re=youtu.be

11:28

14 Paper 9: Influences of an inquiry-based ubiquitous gaming design on students' learn- ing achievements, motivation, behavioral patterns, and tendency towards critical thinking and problem solving

 

Video 4: Problem-based learning 1 https ://youtu.be/BJv_eNe74I0 08:49 Problem-based learning 2 https ://www.youtu be.com/watch ?v=qB_6rFK2-sg&featu

re=youtu.be 16:42

Inquiry-based learning https ://youtu.be/RnwSK zl9PiE 14:04

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2297

3-hour class, a report was conducted by one or two participants for around 30 minutes, and then the professor provided feedback on the students’ reports and discussed some other issues during the remaining time. After each report, the participants were assigned to discuss and answer the questions from the pre-class tasks while 1  hour was allotted to answer a question in class. The participants were given a week to complete the pre-class tasks. Both the experimental and control groups watched the pre-class video lectures before class and finished the assigned tasks prior to the in-class group discussion sessions. The students in the experimental group generated open- ended questions based on the instructional videos related to the topics of the lecture, such as, “What kind of teaching strategies can be used to increase germane cognitive load and promote schema properties?” and “What teaching methods can be used to transform short-term memory into long-term memory?” These questions were posted by the students of the experimental group after they watched the instructional videos, and were then selected and edited by the professor. On some occasions (eg, when some important issues were overlooked by the students), the pro- fessor could propose questions as well. Then, both the experimental group and the control group students discussed these questions in class. The professor explained and clarified any confusion about the video lectures after their discussion.

Figure 4 shows the experimental design of this study. The experiment was conducted on the four units of an Analysis and Applying Strategies of Digital Learning Literature course, which aims to teach the graduate students the analysis and applying strategies of digital learning literature. Before the experiment, the two groups of students spent 3 weeks on the concepts of flipped class- room strategies. Pre-learning assessments for evaluating their knowledge of collecting digital learning literature and reporting on a digital learning paper was administered. Following that, the participants in the experimental group were required to watch seven videos, take notes, write a summary, and post a question before class meetings; on the other hand, those in the control group were required to do similar activities except for taking notes and posting a question before class meetings. In the pre-class tasks, the students in the experimental group and control group were required to watch videos and then write a summary on the LMS platform before the com- mencement of the class. The videos were embedded with a few short questions to enhance stu- dents’ understanding of that topic. The duration for each pre-class video was kept to a maximum of 16:42 minutes and an average of 12:05 minutes because of the constraint of students’ con- centration time (Brame, 2016). Researchers have suggested that the duration of the video lec- tures in flipped learning should be class grade multiplied by 1.5 in minutes (Atwa, Din, & Hussin, 2018; Bergmann & Sams, 2012), with a maximum video length of around 20 minutes for older students (Handke & Franke, 2013; Phillips & Trainor, 2014).

The pre-class video was uploaded, and the pre-class tasks were assigned each week in the LMS platform. The pre-class tasks are the summary of the assigned journal papers, and summary and annotation of the video lecture, as shown in Figure 5. The students in the experimental group and control group were required to summarize and submit what they had remembered and com- prehended concerning the video’s content. However, in the experimental group, the students were required to post a question regarding what they had watched from the video lectures. The instructor would select 10 frequently asked questions and have students in both the experimental group and the control group discuss them as one of the in-class tasks. After all the pre-class and in-class learning activities, the students designed an e-learning research project, presented it and filled out the post-questionnaires for measuring their learning outcomes. The process was carried out to activate the students’ metacognitive knowledge for more learning to take place. It helps students experience the cognitive process of Bloom Taxonomy Matrix’s create level.

© 2019 British Educational Research Association

2298 British Journal of Educational Technology Vol 50 No 5 2019

Measuring instruments In this study, the students’ learning design performance, technology-enhanced active engage- ment, reflective thinking and participation in the before-class stage of flipped learning were assessed. Learning design performance was evaluated based on the students’ learning design project reports. The pre-project report aimed to evaluate the students’ basic knowledge of de- signing digital learning activities. The students were required to present reports during the class to display their knowledge and perspectives of their learning design, including what they had learned from the literature and their own ideas of learning design. The post-learning design report aimed to assess the students’ concepts of applying and innovative and effective strate- gies for digital learning. The total score of both reports was 100. Two experienced instructors rated the students’ reports based on a rubric consisting of four dimensions, that is, correctness, completeness, innovation and effectiveness. The two instructors’ ratings were found to have the substantial agreement of inter-rater consistency .77 Cohen’s Kappa (Cohen, 1960).

The technology-enhanced active student engagement questionnaire was adopted from the learn- ing experience assessment scale developed by Schwarzenberg et al. (2018). The questionnaire consisted of 18 items in five subscales, namely feedback (six items), challenge (three items), peer instruction (three items), choice (three items) and enjoyment (three items), with a 5-point Likert scale. The Cronbach’s alpha values of the subscales are, .90, .76, .84, .73 and .60 respectively.

The questionnaire of reflective thinking was adopted based on the questionnaire to measure the level of reflective thinking developed by Kember et al. (2000). It consists of 16 items, includ- ing four dimensions: habitual action, reflection, critical reflections and understanding. The Cronbach’s alpha values of the four dimensions were .80, .87, .91 and .81 respectively.

The time spent viewing video lectures and the words posted in the assigned tasks were collected for analysis of participation behaviors in the before-class stage of flipped learning. There were seven video lectures (total 5,081 seconds, average 726 seconds) and 17 tasks published in LMS. During the 4-week flipped learning, participants in the experimental group and control group

Figure 4: Experiment procedure [Colour figure can be viewed at wileyonlinelibrary.com]

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2299

watched video lectures and posted their summaries and reflections on the video lectures as the pre-class tasks. In order to identify students’ participation behaviors in the before-class stage of flipped learning, the data of all participants’ time spent (in seconds) watching the video lectures and words (Chinese characters) were analyzed.

Experimental results There were three kinds of data collected, namely (1) the students’ scores of the pre-reports and post-reports, (2) the results of the technology-enhanced active learning engagement question- naire, (3) the results of the reflective thinking questionnaire and (4) Data in LMS of participants’ time spent watching video lectures and words posted. SPSS software was used to analyze the data. The dependent variables were checked for normal distribution and homogeneity of vari- ance before analysis.

Learning design performances The one-way ANCOVA was used to compare the two groups’ learning design performances by adopting the reflective thinking-promoting approach into the flipped learning model as an in- dependent variable, while the posttest and pretest scores were respectively the dependent vari- able and covariate. After confirming the assumption of homogeneity of regression with F = .023 (p  =  .88  >  .05), ANCOVA was performed, as shown in Table 2. A significant difference was

Figure 5: A list of pre-class tasks [Colour figure can be viewed at wileyonlinelibrary.com]

© 2019 British Educational Research Association

2300 British Journal of Educational Technology Vol 50 No 5 2019

found with F = 4.29 (p < .05), showing that the learning design performances of the two groups were significantly different because of the different flipped learning models. The result showed that the participants (adjusted mean  =  89.05, Std. error  =  1.48) who learned with the reflec- tive thinking-promoting approach incorporated into the flipped learning model outperformed the participants (adjusted mean = 84.73, Std. error = 1.48) who learned with the conventional flipped learning mode.

Technology-enhanced active learning engagement Table 3 shows the t-test result for the technology-enhanced active learning engagement ratings of the two groups. The means and standard deviations of the ratings were 4.59 and .30 for the experimental group, and 4.36 and .35 for the control group. The ratings of the two groups were significantly different with t  =  2.21 (p  <  .05). There was a significant difference between the two groups with d = .71 (Cohen, 1988). This implies that there was more positive effect on the learning engagement of the participants in the experimental group than on those in the control group.

We also performed t-tests to assess the impact of specific dimensions. As shown in Table 4, it is recognized that peer instruction t = 2.65 (p < .05, d = .86) had more positive effect, with a sig- nificant difference, on the experimental group (M  =  4.77 SD  =  .33) than on the control group (M = 4.33 SD = .64).

Reflective thinking The t-test outcome of the comparison of reflective thinking for the two learning approaches is shown in Table 5. The mean of reflective thinking ratings for the experimental group was 4.57 (t = 3.34, p < .01, d = 1.11), which was significantly higher than that of the control group (mean = 4.16). Furthermore, the result of reflective thinking for the two groups represents a good effect size according to Cohen’s (1988) criteria, where a Cohen’s d value greater than .8 is con- sidered as a large effect size. This means that with the reflective thinking-promoting approach incorporated into the flipped learning approach, the students achieved significantly higher re- flective thinking than those in the conventional flipped learning approach. In other words, the reflective thinking-promoting approach can successfully promote students’ reflective thinking.

In addition, it was found that three of four dimensions showed significant differences between the two groups, as shown in Table 6, in particular, for habitual action (t = 2.65, p < .05, d = .85), reflection (t = 3.09, p < .001, d = 1.50) and critical reflection (t = 2.51, p < .0001, d = .92). Such a finding could be further evidence that the students who learned with the reflective thinking-pro- moting approach in the flipped learning approach showed better habitual action, reflection and critical thinking than those who learned with the conventional flipped learning approach.

Participation behaviors in the before-class stage of flipped learning In order to clearly understand the students’ participation behaviors in the before-class stage of flipped learning, students’ average time spent (in seconds) on online video lectures and the

Table 2: ANCOVA result of the post scores on students' learning performance

Group N Mean SD Adjusted mean Std. error F η 2

Experimental group 19 89.05 5.411 89.05 1.48 4.29* .106 Control group 19 84.73 7.327 84.73 1.48    

*p < .05.

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2301

words (Chinese characters) they posted was analyzed. Table 7 shows the t-test results of the com- parison of the experimental group (Mean  =  1217.72, t  =  3.14, p  <  .01, d  =  1.01) which spent significantly more time than the control group (Mean  =  885.99) watching video lectures. In addition, the t-test results also reveal that the experimental group (Mean = 1344.26, t = 5.66, p < .001, d = 1.84) posted significantly more words than the control group (Mean = 1005.68).

Discussion and conclusions The study aimed to explore students’ engagement and participation in active learning with learning management systems and to examine the effect of whether a reflective thinking-pro- moting approach would enhance students’ flipped learning design performance and participa- tion. The implications of the findings are discussed as follows.

In general, the findings provide evidence to support the reflective thinking-promoting approach. Concerning research questions 1 and 3: This study aimed to measure the impact of the inte- grated reflective thinking-promoting approach in flipped learning on the students’ learning design performance and reflective thinking of graduate students in the fall 2018 Analysis and Applying Strategies of Digital Learning Literature course. In the experimental group, the partici- pants showed a significantly higher level of learning design performance and reflective thinking,

Table 3: Independent sample t-test results of the technology-enhanced active learning experience

Group N Mean SD t Cohen’s d

Experimental group 19 4.59 .30 2.21* .71 Control group 19 4.36 .35 19  

*p < .05.

Table 4: Independent sample t-test results of the technology-enhanced active learning experience in each dimension

Dimensions Group N Mean SD t Cohen's d

Feedback Experimental group 19 4.67 .39 .86 .30   Control group 19 4.55 .43     Challenge Experimental group 19 4.31 .44 1.64 .51   Control group 19 4.04 .61     Peer instruction Experimental group 19 4.77 .33 2.65* .86   Control group 19 4.33 .64     Choice Experimental group 19 4.70 .43 .86 .27   Control group 19 4.58 .46     Enjoyment Experimental group 19 4.39 .47 1.42 .46   Control group 19 4.18 .44    

*p < .05.

Table 5: Independent sample t-test results of reflective thinking

Group N Mean SD t Cohen's d

Experimental group 19 4.57 .35 3.34** 1.11 Control group 19 4.16 .39    

**p < .01.

© 2019 British Educational Research Association

2302 British Journal of Educational Technology Vol 50 No 5 2019

indicating that integrating the reflective thinking-promoting approach in flipped learning has a positive impact on students’ learning design performance and reflective thinking capacity. In par- ticular, the results of three of four dimensions: habitual action, reflection and critical reflection, demonstrated significant differences between the experimental group and the control group.

One of the most important pre-class tasks in this study was question-posing, and this task provided an instruction way for both teaching and learning. Participants in the experimental group were required to pose a question after watching each video lecture, and then the instruc- tors would select 10 suitable questions and have participants in the experimental group and the control group discuss them in the class. Lan and Lin (2011) mentioned that good quality question-posing requires more reflective thinking, and suggested that engaging in the practice of question-posing can foster students’ cognitive development. In this study, reflective think- ing tasks in flipped learning offered participants an organized approach to examine their own learning, so they had an opportunity to develop reflective thinking skills. Various studies have shown that promoting students’ reflective thinking via activities is valuable for learning design performance (Bigge & Shermis, 1999; Cheng & Chau, 2013; Hwang, Wu, & Ke, 2011; Phan, 2009; Verpoorten, Westera, & Specht, 2011). This is consistent with a considerable amount of literature demonstrating the correlation of higher order skills with students’ success in learning (Ghanizadeh, 2017; Kealey et al., 2005). Phan (2009) further demonstrated that the effective- ness of the learning process is influenced by reflective thinking; likewise, learning achievement is influenced by reflective thinking. Reflective thinking fosters knowledge that can be applied to a variety of situations and contexts. Students have the capability to reflect initially, to acquire and then to generate the new knowledge, so they tend to be more successful academically.

Flipped learning combines video lectures on LMSs as pre-class content instruction and in-class hands-on activities that encourage students to practice, apply, reflect and master the content that

Table 6: Independent sample t-test results of the reflection between the two groups

Dimensions Group N Mean SD t Cohen's d

Habitual action Experimental group 19 4.67 .35 2.65* .85   Control group 19 4.24 .62     Reflection Experimental group 19 4.61 .38 3.09** 1.50   Control group 19 3.92 .53     Critical reflection Experimental group 19 4.46 .38 2.51*** .92   Control group 19 4.05 .50     Understanding Experimental group 19 4.53 .53 .47 .16   Control group 19 4.43 .72    

*p < .05; **p < .01; ***p < .001.

Table 7: Independent sample t-test results for students' participation behaviors in the before-class stage of flipped learning

Variable Group N Mean SD t Cohen's d

Time spent on videos

Experimental group 19 1217.72 4.38.62 3.14** 1.02 Control group 19 885.99 139.01    

Words posted Experimental group 19 1344.26 236.33 5.66*** 1.84   Control group 19 1005.68 110.39    

**p < .01; ***p < .001.

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2303

they have learned from the pre-class requirements (Hoffman, 2014; Tam, 2000). This blended approach provides flexibility for students who can work at their own pace and take on the respon- sibility for learning; however, it may not be easy for teachers to keep students more engaged with learning, which could lead to dissatisfaction in the end (Johnson & Renner, 2012; Strayer, 2012). Therefore, students’ engagement and participation in active learning with the LMS becomes more important. The results in Table 3 for question 2 present that, in the technology-enhanced active engagement, the students with a reflective thinking-promoting approach overall outper- formed the students in the conventional flipped learning environment. As Chuang, Weng, and Chen (2018) mentioned, students’ optimal engagement that promotes learning depends on certain factors. In this study, taking notes and posting questions are such behaviors of learning engagement.

In terms of detailed constructs of the engagement, the students with a reflective thinking- promoting approach had a statistically significant difference in peer instruction, compared with the students in the conventional flipped learning environment. Students’ learning engage- ment and active learning were examined in this study. Gilboy et al. (2015) claimed that learn- ing engagement is students’ commitment to and responsibility for learning as they contribute their time and effort to their learning production. Jensen, Kummer, and Godoy (2015) defined that active learning is using additional technology, teaching materials and peer instruction for a meaningful learning experience. Instead of teaching, the focus should be on how to involve students in autonomous learning, that is, the involvement of students in activities and think- ing about activities. In addition, Hung (2015) indicated that active learning involves students in doing activities and thinking about the information they are learning. In her study, the partici- pants of the experimental group also invested more out-of-class study time to demonstrate their learning engagement. In this study, the participants in the experimental group increased their participation time in the before-class stage of flipped learning, where they needed to take notes and reflect when raising questions. In class, they had deeper discussions on their own questions with peers and the instructor. The process of student engagement was the reason for their active learning (Pahl & Kenny, 2008). Lastly, it was found that the students in both groups had high values for feedback, challenge, peer instruction, choice and enjoyment (with means of more than four on the 5-point Likert scale). Perhaps that is because the LMS can show students’ color-coded progress on each task list, which resulted in their commitment and enhanced their attentiveness to their own and their peers’ learning progress.

The present study examined students’ engagement in active learning with learning manage- ment, and the findings were positive. Meanwhile, this is evidence that a reflective thinking- promoting approach can enhance students’ flipped learning design performance and participa- tion. A reflective-thinking promoting approach amplifies the core value of flipped learning, that is, higher-order thinking. Students can activate their intuitive reflection and gain learning expe- rience skills by analyzing, synthesizing and evaluating what they have learned when interacting with the video content instruction and discussions from their peers and instructor. In addition, with the LMSs that monitor students’ progress, instructors can recognize the factors that affect students’ learning, adjust or differentiate their instruction and even provide students with more opportunities or with additional support to meet students’ needs for learning. Flipped learning with a reflective thinking-promoting approach is one of the examples that encourages students to become more responsible for their learning and that aids them in learning their subject matter.

On the other hand, there are several limitations and issues that need to be considered for future research. First, the sample size was insufficient, making it difficult to make generalizations. However, it was not easy to have a great number of graduate students in this present study.

© 2019 British Educational Research Association

2304 British Journal of Educational Technology Vol 50 No 5 2019

Therefore, we suggest that in future research, a variety of subjects can be considered as partic- ipants. Second, the time of the study was not long enough. If the time were longer, the results should be more substantial because reflective thinking and active learning are important, but they take time to become habits. Third, the activities carried out in the class, such as students’ discussion and feedback, were not entirely assessed in this study. Overall, the most significant purpose of this study is to illustrate the fact that forming reflective thinking skills is both essential and attainable. Most importantly, it needs students’ engagement, participation, time and effort.

Acknowledgements This study is supported in part by the Ministry of Science and Technology of the Republic of China under contract number MOST-105-2511-S-011-008-MY3.

Statements on open data, ethics and conf lict of interest The data can be obtained by sending request e-mails to the corresponding author.

The participants were protected by hiding their personal information during the research pro- cess. They knew that the participation was voluntary and they could withdraw from the study at any time.

There is no potential conflict of interest in this study.

References Ainley, M., & Ainley, J. (2011). Student engagement with science in early adolescence: The contribution

of enjoyment to students' continuing interest in learning about science. Contemporary Educational Psychology, 36(1), 4–12.

Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York, NY: Longman.

Andrusyszyn, M.-A., & Yankou, D. (2004). Attention fatigue and the effect of debriefing in a web-enhanced graduate nursing course. International Journal of Instructional Technology and Distance Learning, 1(4), 1–7.

Argote, L., & Miron-Spektor, E. (2011). Organizational learning: From experience to knowledge. Organization Science, 22(5), 1123–1137.

Atkins, S., & Murphy, K. (1993). Reflections: A review of the literature. Journal of Advanced Nursing, 18, 1188–1192.

Atwa, Z. M., Din, R., & Hussin, M. (2018). Effectiveness of flipped learning in physics education on Palestinian high school students' achievement. Journal of Personalized Learning, 2(1), 73–85.

Austin, A. E. (2002). Preparing the next generation of faculty: Graduate school as socialization to the aca- demic career. The Journal of Higher Education, 73(1), 94–122.

Bates, S., & Galloway, R. (2012). The inverted classroom in a large enrolment introductory physics course: A case study. Paper presented at the Proceedings of the HEA STEM learning and teaching conference.

Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach every student in every class every day. Washington, DC: International Society for Technology in Education.

Bigge, M. L., & Shermis, S. S. (1999). Learning theories for teachers (6th ed.). New York: Longman. Bishop, J. L., & Verleger, M. A. (2013). The flipped classroom: A survey of the research. Paper presented at

the ASEE national conference proceedings, Atlanta, GA. Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature

review in research preparation. Educational Researcher, 34(6), 3–15. Brame, C. J. (2016). Effective educational videos: Principles and guidelines for maximizing student learn-

ing from video content. CBE—Life Sciences Education, 15(4), es6.

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2305

Burton, E. (2010). High level thinking and questioning strategies. Cross Cultural Management, 17(1), 5–9. Butt, A. (2014). Student views on the use of a flipped classroom approach: Evidence from Australia. Business

Education & Accreditation, 6(1), 33–43. Chen, C. H., & Wu, I. C. (2012). The interplay between cognitive and motivational variables in a supportive

online learning system for secondary physical education. Computers & Education, 58, 542–550. Chen, J. A., & Pajares, F. (2010). Implicit theories of ability of grade 6 science students: Relation to epis-

temological beliefs and academic motivation and achievement in science. Contemporary Educational Psychology, 35, 75–87.

Cheng, G., & Chau, J. (2013). Exploring the relationship between students' self-regulated learning ability and their ePortfolio achievement. Internet and Higher Education, 17, 9–15.

Chuang, H., Weng, C., & Chen, C. (2018). Which students benefit most from a flipped classroom approach to language learning? British Journal of Educational Technology, 49(1), 56–68.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

Cohen, J. (1988). Set correlation and contingency tables. Applied Psychological Measurement, 12(4), 425–434.

Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York, NY: Harper Perennial. Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to the educative process.

Boston, MA: D.C. Heath & Co Publishers. Elmaadaway, M. A. N. (2018). The effects of a flipped classroom approach on class engagement and skill

performance in a Blackboard course. British Journal of Educational Technology, 49(3), 479–491. Evans, J. S. B. (2010). Intuition and reasoning: A dual-process perspective. Psychological Inquiry, 21(4),

313–326. Felder, R. M., & Brent, R. (2005). Understanding student differences. Journal of Engineering Education, 94(1),

57–72. Fink, L. D. (2003). Creating significant learning experiences: An integrated approach to designing college courses.

San Francisco, CA: John Wiley & Sons. Fong, J., & Wang, F. L. (Eds.). (2007). Blended learning. Edinburgh, United Kingdom 15–17 August, 2007

Proceedings. City University of Hong Kong. Fu, Q.-K., & Hwang, G.-J. (2018). Trends in mobile technology-supported collaborative learning: A system-

atic review of journal publications from 2007 to 2016. Computers & Education, 119, 129–143. Ghanizadeh, A. (2017). The interplay between reflective thinking, critical thinking, self-monitoring, and

academic achievement in higher education. Higher Education, 74(1), 101–114. Gilboy, M. B., Heinerichs, S., & Pazzaglia, G. (2015). Enhancing student engagement using the flipped

classroom. Journal of Nutrition Education and Behavior, 47(1), 109–114. Hamdan, N., McKnight, P. E., McKnight, K., & Artfstrom, K. M. (2013). The flipped learning model: A white

paper based on the literature review. Retrieved from the Flipped Learning Network website. Handke, J., & Franke, P. (2013). xMOOCs im virtual linguistics campus. In: Schulmeister R, Herausgeber.

(Ed.), MOOCs-massive open online courses: Offene Bildung oder Geschäftsmodell (pp. 101–126). Münster: Waxmann Verlag.

Hoffman, E. S. (2014). Beyond the flipped classroom: Redesigning a research methods course for e-instruc- tion. Contemporary Issues in Education Research, 7, 51–62.

Hong, Y., & Choi, I. (2015). Assessing reflective thinking in solving design problems: The development of a questionnaire. British Journal of Educational Technology, 46(4), 848–863.

Hung, H.-T. (2015). Flipping the classroom for English language learners to foster active learning. Computer Assisted Language Learning, 28(1), 81–96.

Hung, H. (2017). Design-based research: Redesign of an English language course using a flipped class- room approach. TESOL, 51(1), 180–192.

Hwang, G., Lai, C., & Wang, S. (2015). Seamless flipped learning: A mobile technology-enhanced flipped classroom with effective learning strategies. Journal of Computer in Education, 2(4), 449–473.

Hwang, G.-J., Wu, P.-H., & Ke, H.-R. (2011). An interactive concept map approach to supporting mobile learning activities for natural science courses. Computers & Education, 57(4), 2272–2280.

© 2019 British Educational Research Association

2306 British Journal of Educational Technology Vol 50 No 5 2019

Jensen, J. L., Kummer, T. A., & Godoy, P. D. D. M. (2015). Improvements from a flipped classroom may sim- ply be the fruits of active learning. CBE—Life Sciences Education, 14(1), ar5.

Johnson, L. W., & Renner, J. D. (2012). Effect of the flipped classroom model on a secondary computer applica- tions course: Student and teacher perceptions, questions, and student achievement. Louisville, KY: University of Louisville.

Kealey, B. T., Holland, J., & Watson, M. (2005). Preliminary evidence on the association between critical thinking and performance in principles of accounting. Issues in Accounting Education, 20(1), 33–49.

Kember, D., Leung, D. Y. P., Jones, A., Loke, A. Y., McKay, J., Sinclair, K., … Yeung, E. (2000). Development of a questionnaire to measure the level of reflective thinking. Assessment & Evaluation in Higher Education, 25(4), 381–395.

Kissi, P. S., Nat, M., & Armah, R. B. (2018). The effects of learning-family conflict, perceived control over time and task-fit technology factors on urban-rural high school students' acceptance of video-based instruction in flipped learning approach. Educational Technology Research and Development, 66(6), 1547–1569.

Kok, J. (2002). Reflective journal writing: How it promotes reflective thinking in clinical nursing educa- tion: A students' perspective. Curationis, 25(3), 35–42.

Koutropoulos, A. (2011). Digital natives: Ten years after. MERLOT Journal of Online Learning and Teaching, 7(4), 525–538.

Lage, M. J., Platt, G. J., & Treglia, M. (2000). Inverting the classroom: A gateway to creating an inclusive learning environment. The Journal of Economic Education, 31(1), 30–43.

Lan, Y.-F., & Lin, P.-C. (2011). Evaluation and improvement of student's question-posing ability in a web- based learning environment. Australasian Journal of Educational Technology, 27(4), 581–599.

Lai, C., & Hwang, G. (2016). A self-regulated flipped classroom approach to improving students' learning performance in a mathematics course. Computers & Education, 100, 126–140.

Pahl, C. & Kenny, C. (2008). The future of technology enhanced active learning – A roadmap. In M. L. Lytras, D. Gasevic, P. Ordonez de Pablos, & W. Huang (Eds.), Technology enhanced learning (pp. 348–375). Hershey, PA: IGI Global.

Pardo, A., Mirriahi, N., Dawson, S., Zhao, Y., Zhao, A., & Gašević, D. (2015). Identifying learning strate- gies associated with active use of video annotation software. Paper presented at the Proceedings of the Fifth International Conference on Learning Analytics and Knowledge.

Peng, J., Wang, M., & Sampson, D. (2017). Visualizing the complex process for deep learning with an au- thentic programming project. Journal of Educational Technology & Society, 20(4), 275–287.

Phan, H. P. (2009). Exploring students' reflective thinking practice, deep processing strategies, effort, and achievement goal orientations. Educational Psychology, 29(3), 297–313.

Phillips, C. R., & Trainor, J. E. (2014). Millennial students and the flipped classroom. ASBBS Proceedings, 21(1), 519.

Porntaweekul, S., Raksasataya, S., & Nethanomsak, T. (2016). Developing reflective thinking instructional model for enhancing students' desirable learning outcomes. Educational Research and Reviews, 11(6), 238–251.

Prensky, M. (2001). Digital natives, digital immigrants part 1. The Horizon, 9(5), 1–6. https ://doi. org/10.1108/10748 12011 0424816

Raths, D. (2014). Nine video tips for a better flipped classroom. The Education Digest, 79(6), 15. Schön, D. A. (2017). The reflective practitioner: How professionals think in action. Abingdon: Routledge. Schwarzenberg, P., Navon, J., Nussbaum, M., Pérez-Sanagustín, M., & Caballero, D. (2018). Learning expe-

rience assessment of flipped courses. Journal of Computing in Higher Education, 30(2), 237–258. Seaboyer, J. (2013). The role of technology-assisted assessment in fostering critical reading in undergradu-

ate literary studies. International Computer Assisted Assessment Conference. UK: Computer Assisted Assessment.

Sergis, S., Sampson, D. G., & Pelliccione, L. (2018). Investigating the impact of flipped classroom on stu- dents' learning experiences: A self-determination theory approach. Computers in Human Behavior, 78, 368–378.

© 2019 British Educational Research Association

Reflective thinking-promoting for flipped learning 2307

Shernoff, D. J., & Csikszentmihalyi, M. (2009). Flow in schools: Cultivating engaged learners and optimal learning environments. In R. Gilman, E. S. Heubner, & M. J. Furlong (Eds.), Handbook of positive psychol- ogy in schools (pp. 131–145). New York, NY: Routledge.

Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. S. (2003). Student engagement in high school classrooms from the perspective of flow theory. School Psychology Quarterly, 18(2), 158–176.

Soliman, N. A. (2016). Teaching English for academic purposes via the flipped learning approach. Procedia - Social and Behavioral Sciences, 232, 122–129.

Strayer, J. (2012). The flipped classroom: Turning the traditional classroom on its head. Retrieved from http:// www.knewt on.com/flipp ed-class room/

Sun, J. C. Y., Wu, Y. T., & Lee, W. I. (2017). The effect of the flipped classroom approach to Open Course Ware instruction on students' self-regulation. British Journal of Educational Technology, 48(3), 713–729.

Tam, M. (2000). Constructivism, instructional design, and technology: Implications for transforming dis- tance learning. Educational Technology & Society, 3(2), 50–60.

Thompson, P. (2013). The digital natives as learners: Technology use patterns and approaches to learning. Computers & Education, 65, 12–33.

Ventura, A. C., & Moscoloni, N. (2015). Learning styles and disciplinary differences: A cross-sectional study of undergraduate students. International Journal of Learning and Teaching, 1(2), 88–93.

Verpoorten, D., Westera, W., & Specht, M. (2011). Infusing reflective practice in eLearning courses—Can widgets help? International Journal of Technology Enhanced Learning, 3(1), 93–109.

Wang, M., Yuan, B., Kirschner, P. A., Kushniruk, A. W., & Peng, J. (2018). Reflective learning with com- plex problems in a visualization-based learning environment with expert support. Computers in Human Behavior, 87, 406–415.

Copyright of British Journal of Educational Technology is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.

Get help from top-rated tutors in any subject.

Efficiently complete your homework and academic assignments by getting help from the experts at homeworkarchive.com