Author: Ed Nuhfer

Ed Nuhfer received his PhD in geology from University of New Mexico, and served as a geologist and researcher in industry and government before starting an academic career. He held tenure as a full professor at four different universities, authored publications on environmental geology, sedimentary geology, geochemistry, petrology and geoscience education, served as a mentor for hundreds of geology and reclamation students. He served as a regional/national officer for the American Society for Surface Mining and Reclamation, the American Institute of Mining Engineers and as national editor for The American Institute of Professional Geologists from which he received three presidential certificates of merit and the John Galey Sr Pubic Service Award. His book, The Citizens' Guide to Geologic Hazards, won a Choice award for "outstanding academic books" from the Association of College and Research Libraries. While on sabbatical on 1988-1989 in Colorado, he discovered faculty development and returned to found one the first faculty development centers in Wisconsin. He subsequently served as Director of Faculty Development for University of Wisconsin at Platteville, University of Colorado at Denver, and Idaho State University, as Director of Faculty Development and Assessment of Student Learning at California State University Channel Islands, founded the one-week faculty development program "Boot Camp for Profs," which he directed for nearly twenty years, received the national Innovation Award Finalist and the Faculty Development Innovation Award, from POD and served in his last full-time job as Director of Educational Effectiveness at Humboldt State University "years beyond when I thought I would want to retire" before finally retiring in 2014. He has authored over a hundred publications in faculty development, and served as an invited presenter and featured speaker of workshops for The Geological Society of America, POD, AAC&U, WASC, Lilly Conferences, and as an invited presenter of workshops and keynotes on faculty development and assessment for many universities and conferences. He continues to work from as a writer and researcher, as a columnist for National Teaching and Learning Forum for which he has written Developers' Diary for over twelve years --a column based on the unique theme of using fractals and chaos as a key to understanding teaching and learning. Ed remains on as a member of the editorial review boards for several journals and publishers and is winding up a seven-year project with colleagues as principal investigator in developing and testing the Science Literacy Concept Inventory.

Guest Edited Series on Self-Assessment: Synthesis

by Ed Nuhfer, California State Universities (retired)

Self-assessment is a metacognitive skill that employs both cognitive competence and affective feelings. After over two decades of scholars’ misunderstanding, misrepresenting, and deprecating self-assessment’s value, recognizing self-assessment as valid, measurable, valuable, and connected to a variety of other beneficial behavioral and educational properties is finally happening. The opportune time for educating to strengthen that ability is now. We synthesize this series into four concepts to address when teaching self-assessment.

Image of a face silhouette watching a schematic of a man interfacing with mathematical symbols and a human brain — Image by Gerd Altmann from Pixabay

Teach the nature of self-assessment

Until recently, decades of peer-reviewed research popularized a misunderstanding of self-assessment as described by the Dunning-Kruger effect. The effect portrayed the natural human condition as most people overestimating their abilities, lacking the ability to recognize they do so, the most incompetent being the most egregious offenders, and only the most competent possessing the ability to self-assess themselves accurately.

From founding to the present, that promotion relied on mathematics that statisticians and mathematicians now recognize as specious. Behavioral scientists can no longer argue for “the effect” by invoking the unorthodox quantitative reasoning used to propose it. Any salvaging of “the effect” requires different mathematical arguments to support it.

Quantitative approaches confirm that a few percent of the populace are “unskilled and unaware of it,” as described by “the effect.” However, these same approaches affirm that most adults, even when untrained for self-assessment accuracy, are generally capable of recognizing their competence or lack thereof. Further, they overestimate and underestimate with about the same frequency.

Like the development of higher-order or “critical” thinking, the capacity for self-assessment accuracy develops slowly with practice, more slowly than required to learn specific content, and through more practice than a single course can provide. Proficiency in higher-order thinking and self-assessment accuracy seem best achieved through prolonged experiences in several courses.

During pre-college years, a deficit of relevant experiences produced by conditions of lesser privilege disadvantages many new college entrants relative to those raised in privilege. However, both the Dunning-Kruger studies and our own (https://books.aosis.co.za/index.php/ob/catalog/book/279 Chapter 6) confirm that self-assessment accuracy is indeed learnable. Those undeveloped in self-assessment accuracy can become much more proficient through mentoring and practice.

Teach the importance of self-assessment

As a nation that must act to address severe threats to well-being, such as healthcare, homelessness, and climate change, we have rarely been so incapacitated by polarization and bias. Two early entries on bias in this guest-edited series explained bias as a ubiquitous survival mechanism in which individuals relinquish self-assessment to engage in modern forms of tribalism that marginalize others in our workplaces, institutions, and societal cultures. Marginalizing others prevents holding the needed consensus-building conversations between diverse groups that bring creative solutions and needed action.

Relinquishing metacognitive self-assessment to engage in bias obscures perceiving the impacts and consequences of what one does. Developing the skill to exercise self-assessment and use evidence, even under peer pressure not to do so, seems a way to retain one’s perception and ability to act wisely.

Teach the consequences of devaluing self-assessment

The credibility “the effect” garnered as “peer-reviewed fact” helped rationalize the public’s tolerating bias and supporting hierarchies of privilege. A quick Google® search of the “Dunning Kruger effect” reveals widespread misuse to devalue and taunt diverse groups of people as ignorant, unskilled, and inept at recognizing their deficiency.

Underestimating and disrespecting other peoples’ abilities is not simply innumerate and dismal; it cripples learning. Subscribing to the misconception disposes the general populace to avoid trusting in themselves, in others who merit trust, and to dismiss implementing or even respecting effective practices developed by others presumed to be inferiors. It discourages reasoning from evidence and promotes unfounded deference to “authority.” Devaluing self-assessment encourages individuals to relinquish their autonomy to self-assess, which weakens their ability to resist being polarized by demagogues to embrace bias.

Teach self-assessment accuracy

As faculty, we have frequently heard the proclamation “Students can’t self-assess.” Sadly, we have yet to hear that statement confronted by, “So, what are we going to do about it?”

Opportunities exist to design learning experiences that develop self-assessment accuracy in every course and subject area. Knowledge surveys, assignments with required self-assessments, and post-evaluation tools like exam wrappers offer straightforward ways to design instruction to develop this accuracy.

Given the current emphasis on the active learning structures of groups and teams, teachers easily mistake these as the sole domains for active learning and deprecate study alone. The interactive engagements are generally superior to the conventional structure of lecture-based classes for cognitive mastery of content and skills. However, these structures seldom empower learners to develop affect or recognize the personal feelings of knowing that come with genuine understanding. Those feelings differ from those that rest on shallow knowledge and often launch the survival mechanism of bias at critically inopportune times.

Interactive engagement for developing cognitive expertise differs from the active engagement in self-assessment needed to empower individuals to direct their lifelong learning. When students employ quiet reflection time alone to practice self-assessment by enlisting understanding for content for engaging in knowing self, this too is active learning. Ability to distinguish the feeling of deep understanding requires repeated practices in such reflection. We contend that active learning design that attends to both cognition and affect is superior to design that attends only to one of these.

To us, John Draeger was particularly spot-on in his IwM entry, recognizing that instilling cognitive knowledge alone is insufficient as an approach for educating students or stakeholders within higher education institutions. Achievement of successful outcomes depends on educating for proficiency in both cognitive expertise and metacognition. In becoming proficient in controlling bias, “thinking about thinking” must include attention to affect to recognize the reactive feelings of dislike that often arise when confronting the unfamiliar. These reactive feelings are probably unhelpful to the further engagement required to achieve understanding.

The ideal educational environment seems one in which stakeholders experience the happiness that comes from valuing one another during their journey to increase content expertise while extending the knowing of self.

Knowledge Surveys Part 2 — Twenty Years of Learning Guiding More Creative Uses

by Ed Nuhfer, California State Universities (retired)
Karl Wirth, Macalester College
Christopher Cogan, Memorial University
McKensie Kay Phillips, University of Wyoming
Matthew Rowe, University of Oklahoma

Early adopters of knowledge surveys (KSs) recognized the dual benefits of the instrument to support and assess student learning produced by a course or program. Here, we focus on a third benefit: developing students’ metacognitive awareness through self-assessment accuracy.

Communicating self-assessed competence

Initially, we just authored test and quiz questions as the KS items. After the importance of the affective domain became more accepted, we began stressing affect’s role in learning and self-assessment by writing each knowledge survey item with an overt affective self-assessment root such as “I can…” or “I am able to…” followed by a cognitive content outcome challenge. When explaining the knowledge survey to students, we focus their attention on the importance of these affective roots for when they rate their self-assessed competence and write their own items later.

We retain the original three-item response scale expressing relative competence as no competence, partial competence, and high competence. Research reveals three-item scales as valid and reliable as longer ones, but our attraction to the shorter scale remains because it promotes addressing KS items well. Once participants comprehend the meaning of the three items and realize that the choices are identical for every item, they can focus on each item and rate their authentic feeling about meeting the cognitive challenge without distraction by more complex response choices.

photo of woman facing a black board with the words "trust yourself" — Image by Gerd Altmann from Pixabay

We find the most crucial illumination for a student’s self-assessment dilemma: “How do I know when I can rate that I can do this well?” is “When I know that I can teach how to meet this challenge to another person.”

Backward design

We favor backward design to construct topical sections within a knowledge survey by starting with the primary concept students must master when finally understanding that topic. Then, we work backward to build successive items that support that understanding by constantly considering, “What do students need to know to address the item above?” and filling in the detail needed. Sometimes we do this down to the definitions of terms needed to address the preceding items.

Such building of more detail and structure than we sensed might be necessary, especially for introductory level undergraduates, is not “handing out the test questions in advance.” Instead, this KS structure uses examples to show that deceptively disconnected observations and facts allow understanding of the unifying meaning of “concept” through reaching to make connections. Conceptual thinking enables transferability and creativity when habits of mind develop that dare to attempt to make “outrageous connections.”

The feeling of knowing and awareness of metadisciplinary learning

Students learn that convergent challenges that demand right versus wrong answers feel different from divergent challenges that require reasonable versus unreasonable responses. Consider learning “What is the composition of pyrite?” and “Calculate the area of a triangle of 50 meters in length and a base of 10 meters?” Then, contrast the feeling required to learn, “What is a concept?” or “What is science?”

The “What is science?” query is especially poignant. Teaching specialty content in units of courses and the courses’ accompanying college textbooks essentially bypass teaching the significant metadisciplinary ways of knowing of science, humanities, social science, technology, arts, and numeracy. Instructors like Matt Rowe design courses to overcome the bypassing and strive to focus on this crucial conceptual understanding (see video section at times 25.01 – 29.05).

Knowledge surveys written to overtly provoke metadisciplinary awareness aid in designing and delivering such courses. For example, ten metadisciplinary KS items for a 300-item general geology KS appeared at its start, two of which follow.

I can describe the basic methods of science (methods of repeated experimentation, historical science, and modeling) and provide one example each of its application in geological science.
I can provide two examples of testable hypotheses statements, and one example of an untestable hypothesis.

Students learned that they would develop the understanding needed to address the ten throughout the course. The presence of the items in the KS ensured that the instructor did not forget to support that understanding. For ideas about varied metadisciplinary outcomes, examine this poster.

Illuminating temporal qualities

Because knowledge surveys establish baseline data and collect detailed information through an entire course or program, they are practical tools from which students and instructors can gain an understanding of qualities they seldom consider. Temporal qualities include magnitudes (How great?), rates (How quickly?), duration (How long?), order (What sequence?), frequency (How often?), and patterns (What kind?).

More specifically, knowledge surveys reveal magnitude (How great were changes in learning?), rates (How quickly we cover material relative to how well we learned it?), duration (How long was needed to gain an understanding of specific content?), order (What learning should precede other learning?), and patterns (Does all understanding come slowly and gradually or does some come in time as punctuated “Aha moments?”).

Knowledge survey patterns reveal how easily we underestimate the effort needed to do the teaching that makes significant learning change. A typical pattern from item-by-item arrays of pre-post knowledge surveys reveals a high correlation. Instructors may find it challenging to produce the changes where troughs of pre-course knowledge surveys revealing areas of lowest confidence become peak areas in post-course knowledge surveys showing high confidence. Success requires attention to frequency (repetition with take-home drills), duration (extending assignments addressing difficult contents with more time), order (giving attention to optimizing sequences of learning material), and likely switching to more active learning modalities, including students authoring their own drills, quizzes, and KS items.

Studies in progress by author McKensie Phillips showed that students were more confident with the material at the end of the semester rather than each individual unit. This observation even held for early units where researchers expected confidence would decrease given the time elapsed between the end of the unit and when the student took the post-semester KS. The results indicate that certain knowledge mastery is cumulative, and students are intertwining material from unit to unit and practicing metacognition by re-engaging with the KS to deepen understanding over time.

Student-authored knowledge surveys

Introducing students to the KS authoring must start with a class knowledge survey authored by the instructor so that they have an example and disclosure of the kinds of thinking utilized to construct a KS. Author Chris Cogan routinely tasks teams of 4-5 students to summarize the content at the end of the hour (or week) by writing their own survey items for the content. Typically, this requires about 10 minutes at the end of class. The instructor compiles the student drafts, looks for potential misconceptions, and posts the edited summary version back to the class.

Beginners’ student-authored items often tend to be brief, too vague to answer, or too focused on the lowest Bloom levels. However, feedback from the instructor each week has an impact, and students become more able to write helpful survey items and – more importantly – better acquire knowledge from the class sessions. The authoring of items begins to improve thinking, self-assessment, and justified confidence.

Recalibrating for self-assessment accuracy

Students with large miscalibrations in self-assessment accuracy should wonder, “What can I do about this?” The pre-exam knowledge survey data enables some sophisticated post-exam reflection through exam wrappers (Lovett, 2013). With the responses to their pre-exam knowledge survey and the graded exam in hand, students can do a “deep dive” into the two artifacts to understand what they can do.

Instructors can coach students to gain awareness of what their KS responses indicate about their mastery of the content. If large discrepancies between the responses to the knowledge survey and the graded exam exist, instructors query for some introspection on how these arose. Did students use their KS results to inform their actions (e.g., additional study) before the exam? Did different topics or sections of the exam produce different degrees of miscalibration? Were there discrepancies in self-assessed accuracy by Bloom levels?

Most importantly, after conducting the exam wrapper analysis, students with significant miscalibration errors should each articulate doing one thing differently to improve performance. Reminding students to revisit their post-exam analysis well before the next exam is helpful. IwM editor Lauren Scharff noted that her knowledge surveys and tests reveal that most psychology students gradually improved their self-assessment accuracy across the semester and more consistently used them as an ongoing learning tool rather than just a last-minute knowledge check.

Takeaways

We construct and use surveys differently than when we began two decades ago. For readers, we provide a downloadable example of a contemporary knowledge survey that covers this guest-edited blog series and an active Google® Forms online version.

We have learned that mentoring for metacognition can measurably increase students’ self-assessment accuracy as it supports growing their knowledge, skills, and capacity for higher-order thinking. Knowledge surveys offer a powerful tool for instructors who aim to direct students toward understanding the meaning of becoming educated, becoming learning experts, and understanding themselves through metacognitive self-assessment. There remains much to learn.

Knowledge Surveys Part 1 — Benefits of Knowledge Surveys to Student Learning and Development

by Karl Wirth, Macalester College,
Ed Nuhfer, California State Universities (retired)
Christopher Cogan, Memorial University
McKensie Kay Phillips, University of Wyoming

Introduction

Knowledge surveys (KSs) present challenges like exam questions or assignments, but respondents do not answer these. Instead, they express their felt ability to address the challenges with present knowledge. Knowledge surveys focus on self-assessment, which is a special kind of metacognition.

Overall, metacognition is a self-imposed internal dialogue that is a distinguishing feature of “expert learners” regardless of the discipline (e.g., Ertmer & Newby, 1996). Because all students do not begin college as equally aware and capable of thinking about their learning, instructors must direct students to keep them in constant contact with their metacognition. Paul Pintrich, a pioneer in metacognition, stressed that “instruction about metacognition must be explicit.” Knowledge surveys enable what Ertmer & Newby and Pintrich advocate in any class in any subject.

road sign with words "data" pointing to words "information" pointing to word "knowledge" with the word "learning above — Image by Gerd Altmann from Pixabay

Knowledge surveys began in 1992 during a conversation about annual reviews between the guest editor and a faculty member who stated: “They never ask about what I teach.” Upon hearing the faculty member, the guest editor decided to create a 200-item form to survey student ratings of their mastery of detailed content for his geology course at the start and end of the class. The items were simply an array of test and quiz questions, ordered in the sequence the students would encounter during the course. The students responded to each item through a 3-point response at the start and end of the course.

The information from this first knowledge survey proved so valuable that the guest editor described this in 1996 in a geology journal as a formative assessment. As a result, geoscience faculty elsewhere started taking the lead in researching them and describing more benefits.

In 2003, U.S. Air Force Academy’s physics professor Delores Knipp and the guest editor published the first peer-reviewed paper (Nuhfer and Knipp, 2003) for multiple disciplines. If new to knowledge surveys, click the hotlink to that paper now and read at least the first page to gain a conceptual understanding of the instrument.

Self-assessment, Metacognition, and Knowledge Surveys

Becoming educated is a process of understanding self and the phenomena that one experiences. Knowledge surveys structure practices in understanding both.

Our series’ earlier entries revealed the measurable influence of self-assessment on dispositions such as self-efficacy, mindset, and intellectual and ethical development that prove indispensable to the lifelong process of becoming educated. The entries on bias and privilege revealed that the privilege of having the kind of education that renders the unconscious conscious may determine the collective quality of a society and how well we treat one another within it.

Knowledge surveys prompt self-assessment reflections during learning every aspect of the content. Over a baccalaureate education, cumulative, repetitive practice can significantly improve understanding of one’s present knowledge and self-assessing accuracy.

Improving Learning

Knowledge surveys’ original purpose was to improve student learning (e.g., Nuhfer & Knipp, 2003; Wirth et al., 2016, 2021). Providing students with a knowledge survey at the beginning of a course or unit of instruction offered an interactive roadmap for an entire course that overtly disclosed the instructor’s intentions for learning to students.

Early on, users recognized that knowledge surveys might offer a measure of changes in learning produced by a unit of instruction. Demonstrating the validity of such self-assessed competence measures was crucial but was finally achieved in 2016 and 2017.

Deeper Reading

Students quickly learned the value of prioritizing knowledge through engaging with the knowledge survey prior to and during engaging in reading. The structure of the KSs enabled reading with the purpose of illuminating known learning objectives. The structure also primed students to understand concepts by using the reading to clarify the connectedness between knowledge survey items.

Rather than just sitting down to “complete a reading,” students began reading assignments with appropriate goals and strategies; a characteristic of “expert readers” (Paris et al., 1996). When they encountered difficult concepts, they displayed increasing effort to improve their understanding of the topics identified as being essential to understand the concept. Further, knowledge surveys facilitated mentoring. When students did not understand the material, they proved more likely to follow up with a colleague or instructor to complete their understanding.

Facilitating Acquiring Self-Regulation

Well-constructed knowledge surveys are detailed products of instructor planning and thinking. They communicate instructor priorities and coordinate the entire class to focus on specific material in unison. That students’ comments expressing they “didn’t know that would be on the exam” nearly disappeared from classroom conversations cannot be overly appreciated.

Replacing scattered class-wide guessing of what to study allowed a collective focus on “How will we learn this material?” That reframing led to adopting learning strategies that expert learners employ when they have achieved self-regulation. Students increasingly consulted with each other or the instructor when they sensed or realized their current response to a knowledge survey item was probably inadequate.

Levels and Degrees of Understanding

In preparing a knowledge survey for a course, the instructor carefully writes each survey item and learning objective so that learning addresses the desired mastery at the intended Bloom level (Krathwohl, 2002). Providing awareness of Bloom levels to students and reinforcing this throughout a course clarifies student awareness of the deep understanding required to teach the content at the required Bloom level to another person. Whereas it may be sufficient to remember or comprehend some content, demonstrating higher cognitive processes by having to explain to another how to apply, synthesize or evaluate central concepts and content of a course feels different because it is different.

Knowledge surveys can address all Bloom levels and provide the practices needed to enable the paired understanding of knowing and “feeling of knowing” like no other instrument. Including the higher Bloom levels, combined with the explicitly stated advanced degree of understanding as the level of “teaching” or “explaining” to others, builds self-assessment skills and fosters the development of well–justified self-confidence. A student with such awareness can better focus efforts on extending the knowledge in which they recognize their weakness.

Building Skills with Feedback

The blog entries by Fleisher et al. in this series stressed the value of feedback in developing healthy self-assessments. Knowledge survey items that address the same learning outcomes as quizzes, exams, assignments, and projects promote instructional alignment. Such alignment allows explicit feedback from the demonstrated competence measures to calibrate the accuracy of self-assessments of understanding. Over time, knowledge surveys confer awareness that appropriate feedback builds both content mastery and better self-assessment skills.

A robust implementation directs students to complete the relevant portions of a knowledge survey after studying for an exam but before taking it. After the teacher grades the exams, students receive their self-assessed (knowledge survey score) and demonstrated (graded exam score) competence in a single package. From this information, the instructor can direct students to compare their two scores and to receive mentoring from the instructor when there is a large discrepancy (>10 points) between the two scores.

Generally, a significant discrepancy from a single knowledge survey-exam pair comparison is not as meaningful as longer-term trends illuminated by cumulative data. Instructors who use KSs skillfully mentor students to become familiar with their trends and tendencies. When student knowledge survey responses consistently over- or under-estimate their mastery of the content, the paired data reveal this tendency to the student and instructor and open the opportunity for conversations about the student’s habitually favored learning strategies.

A variant implementation adds an easy opportunity for self-assessment feedback. Here, instructors assign students to estimate their score on an assignment or exam at the start of engaging the project and after completing the test or assignment prior to submission. These paired pre-post self-assessments help students to focus on their feelings of knowing and to further adjust toward greater self-assessment accuracy.

Takeaways

Knowledge surveys are unique in their utility for supporting student mastery of disciplinary knowledge, developing their affect toward accurate feelings of knowing, and improving their skills as expert learners. Extensive data show that instructors’ skillful construction of knowledge surveys as part of class design elicits deeper thinking and produces higher quality classes. After construction, class use facilitates mutual monitoring of progress and success by students and instructors. In addition to supporting student learning of disciplinary content, knowledge surveys keep students in constant contact with their metacognition and develop their capacity for lifelong learning.

In Part 2, we follow from our more recent investigations on (1) more robust knowledge survey design, (2) learning about temporal qualities of becoming educated, (3) student authoring of knowledge surveys, and (4) mentoring students with large mis-calibrations in self-assessed competence toward greater self-assessment accuracy.

Metacognition and Mindset for Growth and Success: Part 2 – Documenting Self-Assessment and Mindset as Connected

by Steven Fleisher, California State University
Michael Roberts, DePauw University
Michelle Mason, University of Wyoming
Lauren Scharff, U. S. Air Force Academy
Ed Nuhfer, Guest Editor, California State University (retired)

Self-assessment measures and categorizing mindset preference both employ self-reported metacognitive responses that produce noisy data. Interpreting noisy data poses difficulties and generates peer-reviewed papers with conflicting results. Some published peer-review works question the legitimacy and value of self-assessment and mindset.

Yeager and Dweck (2020) communicate frustration when other scholars deprecate mindset and claim it makes no difference under what mindset students pursue education. Indeed, that seems similar to arguing that enjoyment of education and students’ attitudes toward it makes no difference in the quality of their education.

We empathize with that frustration when we recall our own from seeing in class after class that our students were not “unskilled and unaware of it” and reporting those observations while a dominant consensus that “Students can’t self-assess” proliferated. The fallout that followed from our advocacy in our workplaces (mentioned in Part 2 of the entries on privilege) came with opinions that since “the empiricists have spoken,” there was no reason we should study self-assessment further. Nevertheless, we found good reason to do so. Some of our findings might serve as an analogy to demonstrating the value of mindsets despite the criticisms being leveled against them.

How self-assessment research became a study of mindset

In the summer of 2019, the guest editor and the first author of this entry taught two summer workshops on metacognition and learning at CSU Channel Islands to nearly 60 Bridge students about to begin their college experience. We employed a knowledge survey for the weeklong program, and the students also took the paired-measures Science Literacy Concept Inventory (SLCI). Students had the option of furnishing an email address if they wanted a feedback letter. About 20% declined feedback, and their mean score was 14 points lower (significant at the 99.9% confidence level) than those who requested feedback.

In revisiting our national database, we found that every campus revealed a similar significant split in performance. It mattered not whether the institution was open admissions or highly selective; the mean score of the majority who requested feedback (about 75%) was always significantly higher than those who declined feedback. We wondered if the responses served as an unconventional diagnosis of Dweck’s mindset preference.

Conventional mindset diagnosis employs a battery of agree-disagree queries to determine mindset inclination. Co-author Michael Roberts suggested we add a few mindset items on the SLCI, and Steven Fleisher selected three items from Dweck’s survey battery. After a few hundred student participants revealed only a marginal definitive relationship between mindset diagnosed by these items and SLCI scores, Steve increased our items to five.

Who operates in fixed, and who operates in growth mindsets?

The personal act of choosing to receive or avoid feedback to a concept inventory offers a delineator to classify mindset preference that differs from the usual method of doing so through a survey of agree-disagree queries. We compare here the mindset preferences of 1734 undergraduates from ten institutions using (a) feedback choice and (b) the five agree-disagree mindset survey items that are now part of Version 7.1a of the SLCI. That version has been in use for about two years.

We start by comparing the two groups’ demonstrable competence measured by the SLCI. Both methods of sorting participants into fixed or growth mindset preferences confirmed a highly significant (99.9% confidence) greater cognitive competence in the growth mindset disposition (Figure 1A). As shown in the Figure, feedback choice created two groups of fixed and growth mindsets whose mean SLCI competency scores differed by 12 percentage points (ppts). In contrast, the agree-disagree survey items defined the two groups’ means as separated by only 4 ppts. However, the two methods split the student populace differently, with the feedback choice determining that about 20% of the students operated in the fixed mindset. In contrast, the agree-disagree items approach determined that nearly 50% were operating in that mindset.

We next compare the mean self-assessment accuracy of the two mindsets. In a graph, it is easy to compare mean skills between groups by comparing the scatter shown by one standard deviation (1 Sigma) above and below the means of each group (Figure 1B). The group members’ scatter in overestimating or underestimating their actual scores reveals a group’s developed capacity for self-assessment accuracy. Groups of novices show a larger scatter in their group’s miscalibrations than do groups of those with better self-assessment skills (see Figure 3 of resource at this link).

Figure 1. A. Comparisons of competence (SLCI scores) of 1734 undergraduates between growth mindset participants (color-coded blue) and fixed mindset participants (color-coded red) mindsets as deduced by two methods: (left) agree-disagree survey items and (right) acceptance or opting-out or receiving feedback. “B” displays the measures of demonstrated competence spreads of one standard deviation (1 Sigma) in growth (blue) and fixed mindset (red) groups as deduced by the two methods. The thin black line at 0 marks a perfect self-assessment rating of 0, above which lie overconfident estimates and below which lie underconfident estimates in miscalibrations of self-assessed accuracy. The smaller the standard deviation revealed by the height of the rectangles in 2B, the better the group’s ability to self-assess accurately. Differences shown in A of 4 and 12 ppts and B of 2.3 and 3.5 ppts are differences between means.

On average, students classified as operating in a growth mindset have better-calibrated self-assessment skills (less spread of over- and underconfidence) than those operating in a fixed mindset by either classification method (Figure 1B). However, the difference between fixed and growth was greater and more statistically significant when mindset was classified by feedback choice (99% confidence) rather than by the agree-disagree questions (95% confidence).

Overall, Figure 1 supports Dweck and others advocating for the value of a growth mindset as an asset to learning. We urge contextual awareness by referring readers to Figure 1 of Part 1 of this two-part thematic blog on self-assessment and mindset. We have demonstrated that choosing to receive or decline feedback is a powerful indicator of cognitive competence and at least a moderate indicator of metacognitive self-assessment skills. Still, classifying people into mindset categories by feedback choice addresses only one of the four tendencies of mindset shown in that Figure. Nevertheless, employing a more focused delineator of mindset preference (e.g., choice of feedback) may help to resolve the contradictory findings reported between mindset type and learning achievement.

At this point, we have developed the connections between self-assessment, mindset, and feedback we believe are most valuable to the readers of the IwM blog. Going deeper is primarily of value to those researching mindset. For them, we include an online link to an Appendix to this Part 2 after the References, and the guest editor offers access to SLCI Version 7.1a to researchers who would like to use it in parallel with their investigations.

Takeaways and future direction

Studies of self-assessment and mindset inform one another. The discovery of one’s mindset and gaining self-assessment accuracy require knowing self, and knowing self requires metacognitive reflection. Content learning provides the opportunity for developing the understanding of self by practicing for self-assessment accuracy and acquiring the feeling of knowing while struggling to master the content. Learning content without using it to know self squanders immense opportunities.

The authors of this entry have nearly completed a separate stand-alone article for a follow-up in IwM that focuses on using metacognitive reflection by instructors and students to develop a growth mindset.

References

Dweck, C. S. (2006). Mindset: The new psychology of success. New York: Random House.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Metacognition and Mindset for Growth and Success: APPENDIX to Part 2 – Documenting Self-Assessment and Mindset as Connected

by Ed Nuhfer, Guest Editor, California State University (retired)
Steven Fleisher, California State University
Michael Roberts, DePauw University
Michelle Mason, University of Wyoming
Lauren Scharff, U. S. Air Force Academy
Ed Nuhfer, Guest Editor, California State University (retired)

This Appendix stresses numeracy and employs a dataset of 1734 participants from ten institutions to produce measures of cognitive competence, self-assessed competence, self-assessment accuracy, and mindset categorization. The database is sufficient to address essential issues introduced in our blogs.

Finding replicable relationships in noisy data employs groups from a database collected from instruments proven to produce high-reliability measures. (See Figure 10 at this link.). If we assemble groups, say, groups of 50, as shown in Figure 1 B, we can attenuate the random noise in individuals’ responses (Fig. 1A) and produce a clearer picture of the signal hidden within the noise (Fig. 1B).

Figure 1 Raw data person-by-person on over 9800 participants (Fig. 1 A) shows a highly significant correlation between measures of actual competence from SLCI scores and postdicted self-assessed competence ratings. Aggregating the data into over 180 groups of 50 (Fig. 1 B) reduces random noise and clarifies the relationship.

Random noise is not simply an inconvenience. In certain graphic types, random noise generates patterns that do not intuitively appear random. Researchers easily interpret these noise patterns as products of a human behavior signal. The “Dunning-Kruger effect” appears built on many researchers doing that for over twenty years.

Preventing confusing noise with signal requires knowing what randomness looks like. Researchers can achieve this by ensuring that the surveys and test instruments used in any behavioral science study have high reliability and then constructing a simulated dataset by completing these instruments with random number responses. The simulated population should equal that of the participants in the research study, and graphing the simulated study should employ the same graphics researchers intend to present the participants’ data in a publication.

The 1734 participants addressed in Parts 1 and 2 of this blog’s theme pair on mindset are part of the larger dataset represented in Figure 1. The number is smaller than 9800 because we only recently added mindset questions.

The blog containing this Appendix link showed the two methods of classifying mindset as consistent in designating growth mindset as associated with higher scores on cognitive measures and more accurate self-assessments. However, this finding does not directly test how the two classification methods are related to one another. The fact noted in the blog that the two methods classified people differently indicated a reason to anticipate that the two may not prove to be directly statistically related.

We need to employ groups to attenuate noise, and ideally, we want large groups with good prospects of a spread of values. We first picked the groups associated with furnishing information about privilege (Table 1) because these are groups large enough to attenuate random noise. Further, the groups displayed highly significant statistical spreads when we looked at self-assessed and demonstrable competence within these categories. Note well: we are not trying to study privilege aspects here. Our objective, for now, is to understand the relationship between mindset defined by agree-disagree items and mindset defined by requests for feedback.

We have aggregated our data in Table 1 from four parameters to yield eight paired measures and are ready to test for relationships. Because we already know the relationship between self-assessed competence and demonstrated competence, we can verify whether our existing dataset of 1734 participants presented in 8 paired measures groups is sufficient to deduce the relationship we already know. Looking at self-assessment serves as a calibration to help answer, “How good is our dataset likely going to be for distinguishing the unknown relationships we seek about mindset?”

Table 1. Mindset and self-assessment indicators by large groups. The table reveals each group’s mindset composition derived from both survey items and feedback and the populace size of each group.

Figure 2 shows that our dataset in Table 1 proved adequate in capturing the known significant relationship between self-assessed competence and demonstrated competence (Fig. 2A). The fit-line slope and intercept in Figure 2A reproduce the relationship established from much larger amounts of data (Fig. 1 B). However, the dataset did not confirm a significant relationship between the results generated by the two methods of categorizing people into mindsets (Fig. 2B).

In Figure 2B, there is little spread. The plotted points and the correlation are close to significant. Nevertheless, the spread clustered so tightly that we are apprehensive that the linear relationship would replicate in a future study of a different populace. Because we chose categories with a large populace and large spreads, more data entered into these categories probably would not change the relationships in Figure 2A or 2B. More data might bump the correlation in Figure 2B into significance. However, this could be more a consequence of the spread of the categories chosen for Table 1 than a product of a tight direct relationship between the two methods employed to categorize mindset. However, we can resolve this by doing something analogous to producing the graph in Figure 1B above.

Figure 2. Relationships between self-assessed competence and demonstrated competence (A) and growth mindset diagnosed by survey items and requests for feedback (B). The data graphed is from Table 1.

We next place the same participants from Table 1 into different groups and thereby remove the spread advantages conferred by the groups in Table 1. We randomize the participants to get a good mix of the populace from the ten schools, sort the randomized data by class rank to be consistent with the process used to produce Figure 1B and aggregate them into groups of 100 (Table 2).

Table 2. 1700 students are randomized into groups of 100, and the means are shown for four categories for each group.

The results employing different participant groupings appear in Figure 3. Figure 3A confirms that the different groupings in Table 2 attenuate the spread introduced by the groups in Table 1.

Figure 3. The data graphed is from Table 2. Relationships between self-assessed competence and demonstrated competence appear in (A). In (B), plotting classified by agree-disagree survey items versus mindset classified by requesting or opting out of feedback fails to replicate the pattern shown in Figure 2 B

The matched pairs of self-assessed competence and demonstrable competence continue in Figure 3A to reproduce a consistent line-fit that despite diminished correlation that still attains significance like Figures 1B and 2A.

In contrast, the ability to show replication between the two methods for categorizing mindsets has completely broken down. Figure 2B shows a very different relationship from that displayed in 1B. Deducing the direct relationship between the two methods of categorizing mindset proves not replicable across different groups.

To allow readers who may wish to try different groupings, we have provided the raw dataset used for this Appendix that can be downloaded from https://profcamp.tripod.com/iwmmindsetblogdata.xls.

Takeaways

The two methods of categorizing mindset, in general, designate growth mindset as associated with higher scores on tests of cognitive competence and, to a lesser extent, better self-assessment accuracy. However, the two methods do not show a direct relationship with each other. This indicates the two are addressing different dimensions of the multidimensional character of “mindsets.”

Metacognition and Mindset for Growth and Success: Part 1 – Understanding the Metacognitive Connections between Self-Assessment and Mindset

When I first entered graduate school, I was flourishing. I was a flower in full bloom. My roots were strong with confidence, the supportive light from my advisor gave me motivation, and my funding situation made me finally understand the meaning of “make it rain.” But somewhere along the way, my advisor’s support became only criticism; where there was once warmth, there was now a chill, and the only light I received came from bolts of vindictive denigration. I felt myself slowly beginning to wilt. So, finally, when he told me I did not have what it takes to thrive in academia, that I wasn’t cut out for graduate school, I believed him… and I withered away. (actual co-author experience)

schematic of person with band aid and flowers growing who is facing other people — Image by Moondance from Pixabay

After reading the entirety of this two-part blog entry, return and read the shared experience above once more. You should find that you have an increased ability to see the connections there between seven elements: (1) affect, (2) cognitive development, (3) metacognition, (4) self-assessment, (5) feedback, (6) privilege, and (7) mindset.

The study of self-assessment as a valid component of learning, educating, and understanding opens up fascinating areas of scholarship for new exploration. This entry draws on the same paired-measures research described in the previous blog entries of this series. Here we explain how measuring self-assessment informs understanding of mindset and feedback. Few studies connect self-assessment with mindset, and almost none rest on a sizeable validated data set.

Mindset, self-assessment, and privilege

Mindset theory proposes that individuals lean toward one of two mindsets (Dweck, 2006) that differ based on internalized beliefs about intelligence, learning, and academics. According to Dweck and others, people fall along a continuum that ranges from having a fixed mindset defined by a core belief that their intelligence and thinking abilities remain fixed, and effort cannot change them. In contrast, having a growth mindset comes with the belief that, through their effort, people can expand and improve their abilities to think and perform (Figure 1).

Indeed, a growth mindset has support in the stages of intellectual, ethical, and affective development discovered by Bloom & Krathwohl and William Perry mentioned earlier in this series. However, mindset theory has evolved into making broader claims and advocating that being in a state of growth mindset also enhances performance in high-stakes functions such as leadership, teaching, and athletics.

Figure 1. Fixed – growth mindset tendencies. (From https://trainugly.com/portfolio/growth-mindset/)

Do people choose their mindset or do their experiences place them in their positions on the mindset continuum? Our Introduction to this series disclosed that people’s experiences from degrees of privilege influence their positioning along the self-assessment accuracy continuum, and self-assessment has some commonalities with mindset. However, a focused, evidence-based study of privilege on determining mindset inclination seems lacking.

Our Introduction to this series indicated that people do not choose their positions along the self-assessment continuum. People’s cumulative experiences place them there. Their positions result from their individual developmental histories, where degrees of privilege influence the placement through how many experiences an individual has that are relevant and helpful to building self-assessment accuracy. The same seems likely for determining positions along the mindset continuum.

Acting to improve equity in educational success

Because the development during pre-college years primarily occurs spontaneously by chance rather than by design, people are rarely conscious of how everyday experiences form their dispositions. College students are unlikely even to know their positions on either continuum unless they receive a diagnostic measure of their self-assessment accuracy or their tendency toward a growth or a fixed mindset. Few get either diagnosis anywhere during their education.

Adapting a more robust growth mindset and acquiring better self-assessment accuracy first requires recognizing that these dispositions exist. After that, devoting systematic effort to consciously enlisting metacognition during learning disciplinary content seems essential. Changing the dispositions takes longer than just learning some factual content. However, the time required to see measurable progress can be significantly reduced by a mentor/coach who directs metacognitive reflection and provides feedback.

Teaching self-assessment to lower-division undergraduates by providing numerous relevant experiences and prompt feedback is a way to alleviate some of the inequity produced by differential privilege in pre-college years. The reason to do this early is to allow students time in upper-level courses to ultimately achieve healthy self-efficacy and graduate with the capacity for lifelong learning. A similar reason exists for teaching students the value of affect and growth mindset by providing awareness, coaching, and feedback. Dweck describes how achieving a growth mindset can mitigate the adverse effects of inequity in privilege.

Recognizing good feedback

Dweck places high value on feedback for achieving the growth mindset. The Figure 1 in our guest series’ Introduction also emphasizes the importance of feedback in developing self-assessment accuracy and self-efficacy during college.

Depending on a person’s beliefs about their particular skill to address a challenge, they will respond in predictable ways when a skill requires effort, when it seems challenging, when effort affects performance, and when feedback informs performance. Those with a fixed mindset realize that feedback will indicate imperfections, which they take as indicative of their fixed ability rather than as applicable to growing their ability. To them, feedback shames them for their imperfections, and it hurts. They see learning environments as places where stressful competitions occur between their own and others’ fixed abilities. Affirmations of success rest in grades rather than growing intellectual ability.

Those with a growth mindset value feedback as illuminating the opportunities for advancing quickly in mastery during learning. Sharing feedback with peers in their learning community is a way to gain pleasurable support from a network that encourages additional effort. There is little doubt which mindset promotes the most enjoyment, happiness, and lasting friendships and generates the least stress during the extended learning process of higher education.

Dweck further stresses the importance of distinguishing feedback that is helpful from feedback that is damaging. Our lead paragraph above revealed a devastating experience that would influence any person to fear feedback and seek to avoid it. A formative influence that disposes us to accept or reject feedback likely lies in the nature of feedback that we received in the past. A tour through traits of Dweck’s mindsets suggests many areas where self-perceptions can form through just a single meaningful feedback event.

Australia’s John Hattie has devoted his career to improving education, and feedback is his specialty area. Hattie concluded that feedback is “…the most powerful single moderator that enhances achievement” and noted in this University of Auckland newsletter “…arguably the most critical and powerful aspect of teaching and learning.”

Hattie and Timperley (2007) synthesized many years of studies to determine what constitutes feedback helpful to achievement. In summary, valuable feedback focuses on the work process, but feedback that is not useful focuses on the student as a person or their abilities and communicates evaluative statements about the learner rather than the work. Hattie and Dweck independently arrived at the same surprising conclusion: even praise directed at the person, rather than focusing on the effort and process that led to the specific performance, reinforces a fixed mindset and is detrimental to achievement.

Professors seldom receive mentoring on how to provide feedback that would promote growth mindsets. Likewise, few students receive mentoring on how to use peer feedback in constructive ways to enhance one another’s learning.

Takeaways

Scholars visualize both mindset and self-assessment as linear continuums with two respective dispositions at each of the ends: growth and fixed mindsets and perfectly accurate and wildly inaccurate self-assessments. In this Part 1, we suggest that self-assessment and mindset have surprisingly close connections that scholars have scarcely explored.

Increasing metacognitive awareness seems key to tapping the benefits of skillful self-assessment, mindset, and feedback and allowing effective use of the opportunities they offer. Feedback seems critical in developing self-assessment accuracy and learning through the benefits of a growth mindset. We further suggest that gaining benefit from feedback is a learnable skill that can influence the success of individuals and communities. (See Using Metacognition to Scaffold the Development of a Growth Mindset, Nov 2022.)

In Part 2, we share findings from our paired measures data that partially explain the inconsistent results that researchers have obtained between mindset and learning achievement. Our work supports the validity of mindset and its relationship to cognitive competence. It allows us to make recommendations for faculty and students to apply this understanding to their advantage.

References

Dweck, C. S. (2006). Mindset: The new psychology of success. New York: Random House.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Heft, I. & Scharff, L. (July 2017). Aligning best practices to develop targeted critical thinking skills and habits. Journal of the Scholarship of Teaching and Learning, Vol 17(3), pp. 48-67. http://josotl.indiana.edu/article/view/22600

Isaacson, Randy M., and Frank Fujita. 2006. “Metacognitive Knowledge Monitoring and Self-Regulated Learning: Academic Success and Reflections on Learning.” Journal of Scholarship of Teaching and learning6, no. 1: 39–55. Retrieved from https://eric.ed.gov/?id=EJ854910

Yeager, D. S., & Dweck, C. S. (2020). What can be learned from growth mindset controversies? American Psychologist, 75(9), 1269–1284. https://doi.org/10.1037/amp0000794

Metacognitive Self-assessment in Privilege and Equity – Part 2: Majority Privilege in Scientific Thinking

Ed Nuhfer, California State University (Retired)
Rachel Watson, University of Wyoming
Cinzia Cervato, Iowa State University
Ami Wangeline, Laramie County Community College

Being in the majority carries the privilege of empowerment to set the norms for acceptable beliefs. Minority status for any group invites marginalization by the majority simply because the group appears different from the familiar majority. Here, we explore why this survival mechanism (bias) also operates when a majority perceives an idea as different and potentially threatening established norms.

Young adult learners achieve comfort in ways of thinking and explaining the world from their experiences obtained during acculturation. Our Introduction stressed how these experiences differ in the majority and minority cultures and produce measurable effects. Education disrupts established states of comfort by introducing ideas that force reexaminations that contradict earlier beliefs established from experiences.

Even the kind of college training that promotes only growing cognitive expertise is disruptive but more critical; research verifies that the disruptions are felt. While discovering the stages of intellectual development, William Perry Jr. found that, for some learners, the feelings experienced during transitions toward certain higher stages of thinking were so discomforting that the students ceased trying to learn and withdrew. Currently, about a third of first-year college students drop out before their sophomore year.

Educating for self-assessment accuracy to gain control over bias

We believe that the same survival mechanisms that promote prejudice and suppress empathizing and understanding different demographic groups also cripple understanding in encounters with unfamiliar or contrarian ideas. In moments that introduce ideas disruptive to beliefs or norms, unfamiliar ideas become analogous to unfamiliar groups—easily marginalized and thoughtlessly devalued in snap judgments. Practice in doing self-assessment when new learning surprises us should be valuable for gaining control over the mechanism that triggers our own polarizing bias.

Earlier (Part 2 entry on bias), we recommended teaching students to frequently self-assess, “What am I feeling that I want to be true, and why do I have that feeling?” That assignment ensures that students encounter disruptive surprises mindfully by becoming aware of affective feelings involved in triggering their bias. Awareness gives the greater control over self needed to prevent being captured by a reflex to reject unfamiliar ideas out of hand or to marginalize those who are different.

Teaching by employing self-assessment routinely for educating provides the prolonged relevant practice with feedback required for understanding self. Educating for self-assessment accuracy constitutes a change from training students to “know stuff” to educating students to know how they can think to understand both “stuff” and self.

When the first encounter with something or someone produces apprehension, those who gain a capacity for self-assessment accuracy from practice can exercise more control over their learning through recognizing the feeling that accompanies incipient activation of bias in reaction to discomfort. Such self-awareness allows a pause for reflecting on whether enlisting this vestigial survival mechanism serves understanding and can prevent bias from terminating our learning and inducing us to speak or act in ways that do not serve to understand.

Affect, metacognition, and self-assessment: minority views of contrarian scholars

We address three areas of scholarship relevant to this guest-edited series to show how brain survival mechanisms act to marginalize ideas that contradict an established majority consensus.

Our first example area involves the marginalization of the importance of affect by the majority of behavioral scientists. Antonio Damasio (1999, p. 39) briefly described this collective marginalization:

“There would have been good reason to expect that, as the new century started, the expanding brain sciences would make emotion part of their agenda…. But that…never came to pass. …Twentieth Century science…moved emotion back into the brain, but relegated it to the lower neural strata associated with ancestors whom no one worshipped. In the end, not only was emotion not rational, even studying it was probably not rational.“

A past entry in Improve with Metacognition (IwM) also noted the chilling prejudice against valuing affect during the 20th Century. Benjamin Bloom’s Taxonomy of the Affective Domain (Krathwohl et al. 1964) received an underwhelming reception from educators who had given unprecedented accolades to the team’s earlier volume on Taxonomy of the Cognitive Domain (Bloom, 1956). Also noted in that entry was William G. Perry’s purposeful avoidance of referring to affect in his landmark book on intellectual and ethical development (Perry, 1999). The Taxonomy of the Affective Domain also describes a developmental model that maps onto the Perry model of development much better than Bloom’s Taxonomy of the Cognitive Domain.

Our second example involved resistance against valuing metacognition. Dunlosky and Metcalfe (2009) traced this resistence to French philosopher Auguste Comte (1798-1854), who held that an observer trying to observe self was engaged in an impossible task like an eye trying to see itself by looking inwardly. In the 20th Century, the behaviorist school of psychology gave new life to Comte’s views by professing that individuals’ ability to do metacognition, if such an ability existed, held little value. According to Dunlosky and Metcalfe (2009, p. 20), the behaviorists held “…a stranglehold on psychology for nearly 40 years….” until the mid-1970s, when the work of John Flavell (see Flavell, 1979) made the term and concept of metacognition acceptable in academic circles.

Our third example area involves people’s ability to self-assess. “The Dunning-Kruger effect” holds that most people habitually overestimate their competence, with those least competent holding the most overly inflated views of their abilities and those with real expertise revealing more humility by consistently underestimating their abilities by modest amounts. Belief in “the effect” permeated many disciplines and became popular among the general public. As of this writing, a Google search brought up 1.5 million hits for the “Dunning Kruger effect.” It still constitutes the majority view of American behavioral scientists about human self-assessment, even after recent work revealed that the original mathematical arguments for “the effect” were untenable.

Living a scholars’ minority experience

Considering prejudice against people and bias against new ideas as manifestations of a common, innate survival mechanism obviates fragmentation of these into separate problems addressed through unrelated educational approaches. Perceiving that all biases are related makes evident that the tendency to marginalize a new idea will certainly marginalize the proponents of an idea.

Seeing all bias as related through a common mechanism supports using metacognition, particularly self-assessment, for gaining personal awareness and control over the thoughts and feelings produced as the survival mechanism starts to trigger them. Thus, every learning experience providing discomfort in every subject offers an opportunity for self-assessment practice to gain conscious control over the instinct to react with bias.

Some of the current blog series authors experienced firsthand the need for higher education professionals to acquire such control. When publishing early primary research in the early 1990s, we were naively unaware of majority consensus, had not yet considered bias as a survival reaction, and we had not anticipated marginalization. Suggesting frequent self-assessments as worthwhile teaching practices in the peer-reviewed literature brought reactions that jolted us from complacency into a new awareness.

Scholars around the nation, several of them other authors of this blog series, read the guest editor’s early work, introduced self-assessment in classes and launched self-assessment research of their own. Soon after, many of us discovered disparagements at the departmental, college, and university levels, and even at professional meetings followed for doing so. Some disparagements led to damaged careers and work environments.

The bias imparted by marginalization led to our doubting ourselves. Our feelings for a time were like those of the non-binary gender group presented in the earlier Figure 1 in the previous Part 1 on privilege: We “knew our stuff,” but our feelings of competence in our knowledge lagged. Thanks to the feedback from the journal peer-reviewers of Numeracy, we now live with less doubt in ourselves. For those of us who weathered the storm, we emerged with greater empathy for minority status and minority feelings and greater valuing of self-assessment.

Self-assessment, a type of metacognition employing affect, seems in a paradigm change that recapitulates the history of affect and metacognition. Our Numeracy articles have achieved over 10,000 downloads, and psychologists in Europe, Asia, and Australia now openly question “the effect” (Magnus and Peresetsky, 2021; Kramer et al., 2022; Hofer et al., 2022; Gignac, 2022) in psychology journals. The Office of Science and Society at McGill University in Canada reached out to the lay public (Jarry, 2020) to warn how new findings require reevaluating “the effect.” We recently discovered that paired measures could even unearth unanticipated stress indicators among students (view section at time 21.38 to 24.58) during the turbulent times of COVID and civil disruption.

Takeaways

Accepting teaching self-assessment as good practice for educating and self-assessment measures as valid assessments open avenues for research that are indeed rational to study. After one perceives bias as having a common source, developing self-assessment accuracy seems a way to gain control over personal bias that triggers hostility against people and ideas that are not threatening, just different.

“Accept the person you are speaking with as someone who has done amazing things” is an outstanding practice stressed at the University of Wyoming’s LAMP program. Consciously setting one’s cognition and affect to that practice erases all opportunities for marking anyone or their ideas for inferiority.

References

Bloom, B.S. (Ed.). (1956). Taxonomy of educational objectives, handbook 1: Cognitive domain. New York, NY: Longman.

Damasio, A. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. New York: Harcourt.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. American Psychologist 34, 906-911.

Gignac, Gilles E. (2022). The association between objective and subjective financial literacy: Failure to observe the Dunning-Kruger effect. Personality and Individual Differences 184: 111224. https://doi.org/10.1016/j.paid.2021.111224

Hofer, G., Mraulak, V., Grinschgl, S., & Neubauer, A.C. (2022). Less-Intelligent and Unaware? Accuracy and Dunning–Kruger Effects for Self-Estimates of Different Aspects of Intelligence. Journal of Intelligence, 10(1). https://doi.org/10.3390/jintelligence10010010

Kramer, R. S. S., Gous, G., Mireku, M. O., & Ward, R. (2022). Metacognition during unfamiliar face matching. British Journal of Psychology, 00, 1– 22. https://doi.org/10.1111/bjop.12553

Krathwohl, D.R., Bloom, B.S. and Masia, B.B. (1964) Taxonomy of Educational Objectives: The Affective Domain. New York: McKay.

Magnus, Jan R., and Peresetsky, A. (October 04, 2021). A statistical explanation of the Dunning-Kruger effect. Tinbergen Institute Discussion Paper 2021-092/III, http://dx.doi.org/10.2139/ssrn.3951845

Nicholas-Moon, Kali. (2018). “Examining Science Literacy Levels and Self-Assessment Ability of University of Wyoming Students in Surveyed Science Courses Using the Science Literacy Concept Inventory with Expanded Inclusive Demographics.” Master’s thesis, University of Wyoming.

Perry, W. G. Jr. (1999). Forms of Ethical and Intellectual Development in the College Years. San Francisco, CA: Jossey-Bass (a reprint of the original 1968 work with minor updating).

Tarricone, P. (2011). The Taxonomy of Metacognition (1st ed.). Psychology Press. 288p. https://doi.org/10.4324/9780203830529

Metacognitive Self-assessment in Privilege and Equity – Part 1 Conceptualizing Privilege and its Consequences

by Rachel Watson, University of Wyoming
Ed Nuhfer, California State University (Retired)
Cinzia Cervato, Iowa State University
Ami Wangeline, Laramie County Community College

Demographics of metacognition and privilege

The Introduction to this series asserted that lives of privilege in the K-12 years confer relevant experiences advantageous to acquire the competence required for lifelong learning and entry into professions that require college degrees. Healthy self-efficacy is necessary to succeed in college. Such self-efficacy comes only after acquiring self-assessment accuracy through practice in using the relevant experiences for attuning the feelings of competence with demonstrable competence. We concur with Tarricone (2011) in her recognition of affect as an essential component of the self-assessment (or awareness) component of metacognition: the “‘feeling of knowing’ that accompanies problem-solving, the ability to distinguish ideas about which we are confident….”

A surprising finding from our paired measures is how closely the mean self-assessments of performance of groups of people track with their actual mean performances. According to the prevailing consensus of psychologists, mean self-assessments of knowledge are supposed to confirm that people, on average, overestimate their demonstrable knowledge. According to a few educators, self-reported knowledge is supposed to be just random noise with no meaningful relationship to demonstrable knowledge. Data published in 2016 and 2017 in Numeracy from two reliable, well-aligned instruments revealed that such is not the case. Our reports in Numeracy shared earlier on this blog (see Figures 2 and 3 at this link) confirm that people, on average, self-assess reasonably well.

In 2019, by employing the paired measures, we found that particular groups of peoples’ average competence varied measurably, and their average self-assessed competence closely tracked their demonstrable competence. In brief, different demographic groups, on average, not only performed differently but also felt differently about their performance, and their feelings were accurate.

Conceptualizing privilege and its consequences

Multiple systems (structural, attitudinal, institutional, economic, racial, cultural, etc.) produce privilege, and all individuals and groups experience privilege and disadvantage in some aspects of their lives. We visualize each system as a hierarchical continuum, along which at one end lie those systematically marginalized/minoritized, and those afforded the most advantages lie at the other. Because people live and work within multiple systems, each person likely operates at different positions along different continuums.

Those favored by privilege are often unaware of their part in maintaining a hierarchy that exerts its toll on those of lesser privilege. As part of our studies of the effects on those with different statuses of privilege, we discovered that instruments that can measure cognitive competence and self-assessments of their competence offer richer assessments than competency scores. They also inform us about how students feel and how accurately they self-assess their competence. Students’ histories of privilege seem to influence how effectively they can initially do the kinds of metacognition conducive to furthering intellectual development when they enter college.

Sometimes a group’s hierarchy results from a lopsided division into some criterion-based majority/minority split. There, advantages, benefits, status, and even acceptance, deference, and respect often become inequitably and systematically conferred by identity on the majority group but not on the underrepresented minority groups.

Being a minority can invite being marked as “inferior,” with an unwarranted majority negative bias toward the minority, presuming the latter have inferior cognitive competence and even lower capacity for feeling than the majority. Perpetual exposure to such bias can influence the minority group to doubt themselves and unjustifiably underestimate their competence and capacity to perform. By employing paired measures, Wirth et al. (2021, p. 152 Figs.6.7 & 6.8) found recently that undergraduate women, who are the less represented binary gender in science, consistently underestimated their actual abilities relative to men (the majority) in science literacy.

We found that in the majority ethnic group (white Caucasians), both binary genders, on average, significantly outperformed their counterparts in the minority group (all other self-identified ethnicities combined) in both the competence scores of science literacy and the mean self-assessed competency ratings (Figure 1).

Figure 1. Graph of gender performance on measures of self-assessed competence ratings and demonstrated competence scores across ethnic majority/minority categories. This graph represents ten years of data collection of paired measures, but we only recently began to collect non-binary gender data within the last year, so this group is sparsely represented. Horizontal colored lines coded to the colored circles’ legend mark the positions of the means of scores and ratings in percent at the 95% confidence level.

Notably, in Figure 1, the non-binary gender groups, majority or minority, were the strongest academic group of the three gender categories based on SLCI scores. Still, relative to their performance, the non-binary groups felt that they performed less well than they actually did.

On a different SLCI dataset with a survey item on sexual preference rather than gender, researcher Kali Nicholas Moon (2018) found the same degree of diminished self-assessed competence relative to demonstrated competence for the small LGBT group (see Fig. 7 p. 24 of this link). Simply being a minority may predispose a group to doubt their competence, even if they “know their stuff” better than most.

These mean differences in performance shown in Figure 1 are immense. For perspective, pre-post measures in a GE college course or two in science rarely produce more than mean differences of more than a couple of percentage points on the SLCI. In both majority and minority groups, females, on average, underestimated their performance, whereas males overestimated theirs.

If a group receives constant messages that their thinking may be inferior, it is hardly surprising that they internalize feelings of inferiority that are damaging. Our evidence above from several groups verifies such a tendency. We showed that lower feelings of competence parallel significant deficit performance on a test of understanding science, an area relevant to achieving intellectual growth and meeting academic aspirations. Whether this signifies a general tendency of underconfidence in minority groups for meeting their aspirations in other areas remains undetermined.

Perpetuating privilege in higher education

Academe nurtures many hierarchies. Across institutions, “Best Colleges” rating lists perpetuate a myth that institutions that make the list are, in all ways, for all students “better than” those not on the list. Some state institutions actively promote a “flagship reputation,” implying the state’s other schools as “inferior.” Being in a community of peers that reinforces such hierarchical valuing confers damaging messaging of inferiority to those attending the “inferior” institutions, much as an ethnic majority casts negative messages to the minority.

Within institutions, different disciplines are valued differently, and people experience differential privileges across the departments and programs that focus on educating to support different disciplines. The degrees of consequences of stress, alienation, and physical endangerment are minor compared to those experienced by socially marginalized/minoritized groups. Nevertheless, advocating for any change in an established hierarchy in any community is perceived as disruptive by some and can provide consequences of diminished privilege. National communities of academic research often prove no exception.

Takeaways

Hierarchies usually define privilege, and the majority group often supports hierarchies detrimental to the well-being of minority groups. Although test scores are the prevalent measures used to measure learning mastery, paired measures of cognitive competence and self-assessed competence provide additional information about students’ affective feelings about content mastery and their developing capacity for accurate self-assessment. This information helps reveal the inequity across groups and monitors how well students can employ the higher education environment for advancing their understanding of specialty content and understanding of self. Paired measures confirm that groups of varied privilege fare differently in employing that environment for meeting their aspirations.

Understanding Bias in the Disciplines: Part 2 – the Physical and Quantitative Sciences

by Ed Nuhfer, California State University (Retired)
Eric Gaze, Bowdoin College
Paul Walter, St Edwards University
Simone Mcknight (Simone Erchov), Global Systems Technology

In Part 1, we summarized psychologists’ current understanding of bias. In Part 2, we connect conceptual reasoning and metacognition and show how bias challenges clear reasoning even in “objective” fields like science and math.

Science as conceptual

College catalogs’ explanations of general education (GE) requirements almost universally indicate that the desired learning outcome of the required introductory science course is to produce a conceptual understanding of the nature of science and how it operates. Focusing only on learning disciplinary content in GE courses squeezes out stakeholders’ awareness that a unifying outcome even exists.

Wherever a GE metadisciplinary requirement (for example, science) specifies a choice of a course from among the metadiscipline’s different content disciplines (for example, biology, chemistry, physics, geology), each course must communicate an understanding of the way of knowing established in the metadiscipline. That outcome is what the various content disciplines share in common. A student can then understand how different courses emphasizing different content can effectively teach the same GE outcome.

The guest editor led a team of ten investigators from four institutions and separate science disciplines (biology, chemistry, environmental science, geology, geography, and physics). Their original proposal was to investigate ways to improve the learning in the GE science courses. While articulating what they held in common as professing the metadiscipline of “science,” the investigators soon recognized that the GE courses they took as students had focused on disciplinary content but scarcely used that content to develop an understanding of science as a way of knowing. After confronting the issue of teaching with such a unifying emphasis, they later turned to the problem of assessing success in producing this different kind of understanding.

Upon discovering no suitable off-the-shelf assessment instrument to meet this need, they constructed the Science Literacy Concept Inventory (SLCI). This instrument later made possible this guest-edited series and the confirmation of knowledge surveys as valid assessments of student learning.

Concept inventories test understanding the concepts that are the supporting framework for larger overarching blocks of knowledge or thematic ways of thinking or doing. The SLCI tests nine concepts specific to science and three more related to the practice of science and connecting science’s way of knowing with contributions from other requisite GE metadisciplines.

Self-assessment’s essential role in becoming educated

Self-assessment is partly cognitive (the knowledge one has) and partly affective (what one feels about the sufficiency of that knowledge to address a present challenge). Self-assessment accuracy confirms how well a person can align both when confronting a challenge.

Developing good self-assessment accuracy begins with an awareness that having a deeper understanding starts to feel different from merely having surface knowledge needed to pass a multiple-choice test. The ability to accurately feel when deep learning has occurred reveals to the individual when sufficient preparation for a challenge has, in fact, been achieved. We can increase learners’ capacity for metacognition by requiring frequent self-assessments that give them the practice needed to develop self-assessment accuracy. No place needs teaching such metacognition more than the introductory GE courses.

Regarding our example of science, the 25 items on the SLCI that test understanding of the twelve concepts derive from actual cases and events in science. Their connection to bias lies in learning that when things go wrong when doing or learning science, some concept is unconsciously being ignored or violated. Violations are often traceable to bias that hijacked the ability to use available evidence.

We often say: “Metacognition is thinking about thinking.” When encountering science, we seek to teach students to “think about” (1) “What am I feeling that I want to be true and why do I have that feeling?” and (2) “When I encounter a scientific topic in popular media, can I articulate what concept of science’s way of knowing was involved in creating the knowledge addressed in the article?”

Examples of bias in physical science

“Misconceptions research” constitutes a block of science education scholarship. Schools do not teach the misconceptions. Instead, people develop preferred explanations for the physical world from conversations that mostly occur in pre-college years. One such explanation addresses why summers are warm and winters are cold. The explanation that Earth is closer to the sun in summer is common and acquired by hearing it as a child. The explanation is affectively comfortable because it is easy, with the ease coming from repeatedly using the neural network that contains the explanation to explain the seasonal temperatures we experience. We eventually come to believe that it is true. However, it is not true. It is a misconception.

When a misconception becomes ingrained in our brain neurology over many years of repeated use, we cannot easily break our habit of invoking the neural network that holds the misconception until we can bypass it by constructing a new network that holds the correct explanation. Still, the latter will not yield a network that is more comfortable to invoke until usage sufficiently ingrains it. Our bias tendency is to invoke the most ingrained explanation because doing so is easy.

Even when individuals learn better, they often revert to invoking the older, ingrained misconception. After physicists developed the Force Concept Inventory (FCI) to assess students’ understanding of conceptual relationships about force and motion, they discovered that GE physics courses only temporarily dislodged students’ misconceptions. Many students soon reverted to invoking their previous misconceptions. The same investigators revolutionized physics education by confirming that active learning instruction better promoted overcoming misconceptions than did traditional lecturing.

The pedagogy that succeeds seemingly activates a more extensive neural network (through interactive discussing, individual and team work on problem challenges, writing, visualizing through drawing, etc.) than was activated to initially install the misconception (learning it through a brief encounter).

Biases that add wanting to believe something as true or untrue are especially difficult to dislodge. An example of the power of bias with emotional attachment comes from geoscience.

Nearly all school children in America today are familiar with the plate tectonics model, moving continents, and ephemeral ocean basins. Yet, few realize that the central ideas of plate tectonics once were scorned as “Germanic pseudoscience” in the United States. That happened because a few prominent American geoscientists so much wanted to believe their established explanations as true that their affect hijacked these experts’ ability to perceive the evidence. These geoscientists also exercised enough influence in the U. S. to keep plate tectonics out of American introductory level textbooks. American universities introduced plate tectonics in introductory GE courses only years later than did Europe.

Example of Bias in Quantitative Reasoning

People usually cite mathematics as the most dispassionate discipline and the least likely for bias to corrupt. However, researchers Dan Kahan and colleagues demonstrated that bias also disrupts peoples’ ability to use quantitative data and think clearly.

Researchers asked participants to resolve whether a skin cream effectively treated a skin rash. Participants received data for subjects who did or did not use the skin cream. Among users, the rash got better in 223 cases and got worse in 75 cases. Of subjects who did not use the skin cream, the rash got better in 107 cases and worse in 21 cases.

Participants then used the data to select from two choices: (A) People who used the cream were more likely to get better or (B) People who used the cream were more likely to get worse. More than half of the participants (59%) selected the answer not supported by the data. This query was primarily a numeracy test in deducing the meaning of numbers.

Then, using the same numbers, the researchers added affective bait. They replaced the skin cream query with a query about the effects of gun control on crime in two cities. One city allowed concealed gun carry, and another banned concealed gun carry. Participants had to decide whether the data showed that concealed carry bans increased or decreased crime.

Self-identified conservative Republicans and liberal Democrats responded with a desire to believe acquired from their party affiliations. The result was even more erroneous than the skin cream case participants. Republicans greatly overestimated increased crime from gun bans, but no more than Democrats overestimated decreased crime from gun bans (Figure 1). When operating from “my-side” bias planted by either party, citizens significantly lost their ability to think critically and use numerical evidence. This was true whether the self-identified partisans had low or high numeracy skills.

Figure 1. Effect of bias on interpreting simple quantitative information (from Kahan et al. 2013, Fig. 8). Numerical data needed to answer whether a cream effectively treated a rash triggered low bias responses. When researchers employed the same data to determine whether gun control effectively changed crime, polarizing emotions triggered by partisanship significantly subverted the use of evidence toward what one wanted to believe.

Takeaway

Decisions and conclusions that appear based on solely objective data rarely are. Increasing metacognitive capacity produces awareness of the prevalence of bias.

Understanding Bias in the Disciplines: Part 1 – the Behavioral Sciences

by Simone Mcknight (Simone Erchov), Global Systems Technology
Ed Nuhfer, California State University (Retired)
Eric Gaze, Bowdoin College
Paul Walter, St Edwards University

Bias as conceptual

Bias arises from human brain mechanisms that process information in ways that make decision-making quicker and more efficient at the cognitive/neural level. Bias is an innate human survival mechanism, and we all employ it.

Bias is a widely known and commonly understood psychological construct. The common understanding of bias is “an inclination or predisposition for or against something.” People recognize bias by its outcome—the preference to accept specific explanations or attributions as true.

In everyday conversation, discussions about bias occur in preferences and notions people have on various topics. For example, people know that biases may influence the development of prejudice (e.g., ageism, sexism, racism, tribalism, nationalism), political, or religious beliefs.

A deeper look reveals that some of these preferences are unconscious. Nevertheless, they derive from a related process called cognitive bias, a propensity to use preferential reasoning to assess objective data in a biased way. This entry introduces the concept of bias, provides an example from the behavioral sciences, and explains why metacognition can be a valuable tool to counteract bias. In Part 2, which follows this entry, we provide further examples from hard science, field science, and mathematics.

Where bias comes from

Biases develop from the mechanisms by which the human brain processes information as efficiently as possible. These unconscious and automatic mechanisms make decision-making more efficient at the cognitive/neural level. Most mechanisms that help the human brain make fast decisions are credited to adaptive survival. Like other survival mechanisms, bias loses value and can be a detriment in a modern civilized world where threats to our survival are infrequent challenges. Cognitive biases are subconscious errors in thinking that lead to misinterpreting future information from the environment. These errors, in turn, impact the rationality and accuracy of decisions and judgments.

When we frame unconscious bias within the context of cognitive bias and survival, it is easier to understand how all of us have inclinations to employ bias and why any discipline that humans manage is subject to bias. Knowing this makes it easier to account for the frequent biases affecting the understanding and interpreting of diverse kinds of data.

People easily believe that bias only exists in “subjective” disciplines or contexts where opinions and beliefs seem to guide decisions and behavior. However, bias manifests in how humans process information at the cognitive level. Although it is easier to understand bias as a subjective tendency, the typical way we process information means that bias can pervade all of our cognition.

Intuitively, disciplines relying on tangible evidence, logical arguments, and natural laws of the physical universe would seem factually based and less influenced by feelings and opinion. After all, “objective disciplines” do not predicate their findings on beliefs about what “should be.” Instead, they measure tangible entities and gather data. However, even in the “hard science” disciplines, the development of a research question, the data collected, and the interpretations of data are vulnerable to bias. Tangible entities such as matter and energy are subject to biases as simple as differences in perception of the measured readings on the same instrument. In the behavioral sciences, where investigative findings are not constrained by natural law, bias can be even harder to detect. Thus, all scientists carry bias into their practice of science, and students carry bias into their learning of it.

Metacognition can help counter our tendencies toward bias because it involves bringing relevant information about a process (e.g., conducting research, learning, or teaching) into awareness and then using that awareness to guide subsequent behaviors.

Consequences of bias

Bias impacts individual understanding of the world, the self, and how the self navigates the world – our schemas. These perceptions may impact elements of identity or characterological elements that influence the likelihood of behaving in one way versus another.

Bias should be assumed as a potentially influential factor in any human endeavor. Sometimes bias develops for an explanation after hearing it in childhood and then invoking that explanation for years. Even after seeing the evidence against that bias, our initial explanations are difficult to replace with ones better supported by evidence because we remain anchored to that initial knowledge. Adding a personal emotional attachment to an erroneous explanation makes replacing it even more difficult. Scientists can have emotional attachments to particular explanations of phenomena, especially their own explanations. Then, it becomes easy to selectively block out or undervalue evidence that modifies or contradicts the favored explanation (also known as confirmation bias).

Self-assessment, an example of long-standing bias in behavioral science

As noted in the introduction, this blog series focuses on our team’s work related to self-assessment. Our findings countered results from scores of researchers who replicated and verified the testing done in a seminal paper by Kruger and Dunning (1999). Their research asserted that most people were overconfident about their abilities, and the least competent people had the most overly optimistic perceptions of their competence. Researchers later named the phenomenon the “Dunning-Kruger effect,” and the public frequently deployed “the effect” as a label to disparage targeted groups as incompetent. “The effect” held attraction because it seemed logical that people who lacked competence also lacked the skills needed to recognize their deficits. Quite simply, people wanted to believe it, and replication created a consensus with high confidence in concluding that people, in general, cannot accurately self-assess.

While a few researchers did warn about likely weaknesses in the seminal paper, most behavioral scientists selectively ignored the warnings and repeatedly employed the original methodology. This trend of replication continued in peer-reviewed behavioral science publications through at least 2021.

Fortunately, the robust information storage and retrieval system that characterizes the metadiscipline of science (which is a characteristic distinguishing science from technology as ways of knowing) makes it possible to challenge a bias established in one discipline by researchers from another. Through publications and open-access databases, the arguments that challenge an established bias then become available. In this case, the validity of “the effect” resided mainly in mathematical arguments and not, as presumed, arguments that resided solely within the expertise of behavioral scientists.

No mathematics journal had ever hosted arguments addressing the numeracy of arguments that established and perpetuated the belief in “the effect.” However, mathematics journals offered the benefit of reviewers who specialized in quantitative reasoning and were not emotionally attached to any consensus established in behavioral science journals. These reviewers agreed that the long-standing arguments for supporting the Dunning-Kruger effect were mathematically flawed.

In 2016 and 2017, Numeracy published two articles from our group that detailed the mathematical arguments that established the Dunning-Kruger effect conclusions and why these arguments are untenable. When examined by methods the mathematics reviewers verified as valid, our data indicated that people were generally good at self-assessing their competence and confirmed that there were no marked tendencies toward overconfidence. Experts and novices proved as likely to underestimate their abilities as to overestimate them. Further, the percentage of those who egregiously overestimated their abilities was small, in the range of about 5% to 6% of participants. However, our findings confirmed a vital conclusion of Kruger and Dunning (1999): experts self-assess better than novices (variance decreases as expertise increases), and self-assessment accuracy is attainable through training and practice.

By 2021, the information released in Numeracy began to penetrate the behavioral science journals. This blog series, our earlier posts on this site, and archived presentations to various audiences (e.g., the National Numeracy Network, the Geological Society of America) further broadened awareness of our findings.

Interim takeaways

Humans construct their learning from mentally processing life experiences. During such processing, we simultaneously construct some misconceptions and biases. The habit of drawing on a misconception or bias to explain phenomena ingrains it and makes it difficult to replace with correct reasoning. Affective attachments to any bias make overcoming the bias extremely challenging, even for the most accomplished scholars.

It is essential to realize that we can reduce bias by employing metacognition to recognize bias originating from within us at the individual level and by considering bias that influences us but is originated from or encouraged by groups. In the case above, we were able to explain the bias within the Behavioral Sciences disciplines by showing how repeatedly mistaking mathematical artifacts as products of human behavior produced a consensus that held understanding self-assessment captive for over two decades.

Metacognitive self-assessment seems necessary for initially knowing self and later for recognizing one’s own personal biases. Self-assessment accuracy is valuable in using available evidence well and reducing the opportunity for bias to hijack our ability to reason. Developing better self-assessment accuracy appears to be a very worthy objective of becoming educated.

Introduction: Why self-assessment matters and how we determined its validity

By Ed Nuhfer, Guest Editor, California State University (retired)

There are few exercises of thinking more metacognitive than self-assessment. For over twenty years, behavioral scientists accepted that the “Dunning-Kruger effect,” which portrays most people as “unskilled and unaware of it,” correctly described the general nature of human self-assessment. Only people with significant expertise in a topic were capable of self-assessing themselves accurately, while those with the least expertise supposedly held highly overinflated views of their abilities.

The authors of this guest series have engaged in a collaborative effort to understand self-assessment for over a decade. They documented how the “Dunning-Kruger effect,” from its start, rested on specious mathematical arguments. Unlike what the “effect” asserts, most people do not hold overly inflated views of their competence, regardless of their level of expertise. We summarized some of our peer-reviewed work in earlier articles in “Improve with Metacognition (IwM).” These are discoverable by using “Dunning-Kruger effect” in IwM’s search window.

Confirming that people, in general, are capable of self-assessing their competence affirms the validity of self-assessment measures. The measures inform efforts in guiding students to improve their self-assessment accuracy.

This introduction presents commonalities that unify the series’ entries to follow. In the entries, we hotlink the references available as open-source within the blogs’ text and place all other references cited at the end.

Why self-assessment matters

After an educator becomes aware of metacognition’s importance, teaching practice should evolve beyond finding the best pedagogical techniques for teaching content and assessing student learning. The “place beyond” focuses on teaching the student how to develop a personal association with content as a basis for understanding self and exercising higher-order thinking. Capturing the changes in developing content expertise together with self in a written teaching/learning philosophy expedites understanding how to achieve both. Self-assessment could be the most valuable of all the varieties of metacognition that we employ to deepen our understanding.

Visualization is conducive to connecting essential themes in this series of blogs that stress becoming better educated through self-assessment. Figure 1 depicts the role and value of self-assessment from birth at the top of the figure to becoming a competent, autonomous lifelong learner by graduation from college at the bottom.

Figure 1. Relationship of self-assessment to developing self-regulation in learning.

Let us walk through this figure, beginning with early life Stage #1 at the top. This stage occurs throughout the K-12 years, when our home, local communities, and schools provide the opportunities for choices and efforts that lead to experiences that prepare us to learn. In studies of Stage 1, John A. Ross made the vital distinction between self-assessment (estimating immediate competence to meet a challenge) and self-efficacy (perceiving one’s personal capacity to acquire competence through future learning). Developing healthy self-efficacy requires considerable practice in self-assessment to develop consistent self-assessment accuracy.

Stage 1 is a time that confers much inequity of privilege. Growing up in a home with a college-educated parent, attending schools that support rich opportunities taught in one’s native language, and living in a community of peers from homes of the well-educated provide choices, opportunities, and experiences relevant to preparing for higher education. Over 17 or 18 years, these relevant self-assessments sum to significant advantages for those living in privilege when they enter college.

However, these early-stage self-assessments occur by chance. The one-directional black arrows through Stage 2 communicate that nearly all the self-assessments are occurring without any intentional feedback from a mentor to deliberately improve self-assessment accuracy. Sadly, this state of non-feedback continues for nearly all students experiencing college-level learning too. Thereby, higher education largely fails to mitigate the inequities of being raised in a privileged environment.

The red two-directional arrows at Stage 3 begin what the guest editor and authors of this series advocate as a very different kind of educating to that commonly practiced in American institutions of education. We believe education could and should provide self-assessments by design, hundreds in each course, all followed by prompt feedback, to utilize the disciplinary content for intentionally improving self-assessment accuracy. Prompt feedback begins to allow the internal calibration needed for improving self-assessment accuracy (Stage #4).

One reason to deliberately incorporate self-assessment practice and feedback is to educate for social justice. Our work indicates that we can enable the healthy self-efficacy needed to succeed in the kinds of thinking and professions that require a college education by strengthening the self-assessment accuracy of students and thus make up for the lack of years of accumulated relevant self-assessments in the backgrounds of those lesser privileged.

By encouraging attention to self-assessment accuracy, we seek to develop students’ felt awareness of surface learning changing toward the higher competence characterized by deep understanding (Stage #5). Awareness of the feeling characteristic when one attains the competence of deep understanding enables better judgment for when one has adequately prepared for a test or produced an assignment of high quality and ready for submission.

People attain Stage #6, self-regulation, when they understand how they learn, can articulate it, and can begin to coach others on how to learn through effort, using available resources, and accurately doing self-assessment. At that stage, a person has not only developed the capacity for lifelong learning, but has developed the capacity to spread good habits of mind by mentoring others. Thus the arrows on each side of Figure 1 lead back to the top and signify both the reflection needed to realize how one’s privileges were relevant to their learning success and cycling that awareness to a younger generation in home, school, and community.

A critical point to recognize is that programs that do not develop students’ self-assessment accuracy are less likely to produce graduates with healthy self-efficacy or the capacity for lifelong learning than programs that do. We should not just be training people to grow in content skills and expertise but also educating them to grow in knowing themselves. The authors of this series have engaged for years in designing and doing such educating.

The common basis of investigations

The aspirations expressed above have a basis in hard data from assessing the science literacy of over 30,000 students and “paired measures” on about 9,000 students with peer-reviewed validated instruments. These paired measures allowed us to compare self-assessed competence ratings on a task and actual performance measures of competence on that same task.

Knowledge surveys serve as the primary tool through which we can give “…self-assessments by design, hundreds in each course all followed by prompt feedback.” Well-designed knowledge surveys develop each concept with detailed challenges that align well with the assessment of actual mastery of the concept. Ratings (measures of self-assessed competence) expressed on knowledge surveys, and scores (measures of demonstrated competence) expressed on tests and assignments are scaled from 0 to 100 percentage points and are directly comparable.

When the difference between the paired measures is zero, there is zero error in self-assessment. When the difference (self-assessed minus demonstrated) is a positive number, the participant tends toward overconfidence. When the difference is negative, the participant has a tendency toward under-confidence.

In our studies that established the validity of self-assessment, our demonstrated competence data in our paired measures came mainly from the validated instrument, the Science Literacy Concept Inventory or “SLCI.” Our self-assessed competence data comes from knowledge surveys and global single-queries tightly aligned with the SLCI. Our team members incorporate self-created knowledge surveys of course content into their higher education courses. Knowledge surveys have proven to be powerful research tools and classroom tools for developing self-assessment accuracy.

Summary overview of this blog series

IwM is one of the few places where the connection between bias and metacognition has directly been addressed (e.g., see a fine entry by Dana Melone). The initial two entries of this series will address metacognitive self-assessment’s relation to the concept of bias.

Later contributions to this series consider privilege and understanding the roles of affect, self-assessment, and metacognition when educating to mitigate the disadvantages of lesser privilege. Other entries will explore the connection between self-assessment, participant use of feedback, mindset, and metacognition’s role in supporting the development of a growth mindset. Near the end of this series, we will address knowledge surveys, the instruments that incorporate the disciplinary content of any college course to improve learning and develop self-assessment accuracy.

We will conclude with a final wrap-up entry of this series to aid readers’ awareness that what students should “think about” when they “think about thinking” ought to provide a map for reaching a deeper understanding of what it means to become educated and to acquire the capacity for lifelong learning.

Learning Philosophies of Teachers and Students: Two Neglected Metacognitive Catalysts for Success

by Dr. Ed Nuhfer, California State Universities (retired)

Hmmm… COVID year… what a trip. If I slept through it, there were nightmares—lots of ’em. Where did I leave off last April (Nuhfer, 2020)? Oh yeah, we ended with a question: “How might teaching students to write their learning philosophies improve their learning?” Well, OK…let’s continue that… by first realizing that professors’ teaching philosophies are their learning philosophies, and those who do write them come to recognize how keeping such a written record enhances success in teaching.

Philosophies are reflective; they record the results of a metacognitive conversation with oneself. The results articulate the plan for practice disclosing what one wants to do, how one chooses to do it, and how to know the chosen practices’ impact. Philosophies focus on learning about process, which is too rarely the stuff of college education, where the emphasis is on learning content—the disciplines’ products. Even “student-centered-learning” structures too seldom involve directly teaching students to be reflective about how to learn.

$Illustration of components (thinking, teaching, learning) in the fractal generator for faculty and students (by Ed Nuhfer)$

Six critical components of a teaching philosophy appeared in the graphic of the fractal generator above (and also shared my April 2020 blog). An informed teaching/learning philosophy considers all six components. Three crucial components describe internal strengths enlisted during learning: affect, levels of thinking, and metacognition, and three more are competencies mainly built from external sources: content, pedagogy, and assessment.

For nearly twenty years, I led week-long faculty development retreats in which each faculty arrived with a written one-page document that they had constructed as their teaching philosophy. A faculty member rarely arrived at a retreat with a written philosophy that addressed more than three of the six components.

Through the retreats, participants revisited and edited their philosophy a bit each day. Our final exercise of the retreat was sharing the revised philosophies in groups and polishing them more for use. No participant left without awareness of the vital role that each of the six components played. Participants’ written philosophies of practice were probably their most valuable tangible takeaway.

Why written philosophies?

Consider the advantage a written plan confers to constructing anything complex. Architectural design requires written blueprints and strategies because the challenge is just too complex to address well by acting spontaneously from what one can carry around in one’s head. Construction contractors avoid working without a written plan because doing so produces disappointing results.

Learning and teaching are challenges as complex as any construction project. A teaching/learning philosophy acts as the equivalent of an architectural design plan and encompasses the big picture of what we intend to do. Students need learning philosophies for the same reason professors need them, but acting spontaneously without any written philosophy is probably the norm in higher education. How many of your students approach learning with a written plan?

The six components that are so essential for professors to consider also confer similar value to students. It really is up to professors to mentor students to craft their first informed learning philosophies. An excellent way to start students toward constructing their personal learning philosophies is to give each a nearly blank paper with the six components’ names at the top.

Beginners must begin to incorporate the six as a checklist by asking self: “Where are awareness of affect, levels of thinking, etc. in my practice?” After they internalize these six through at least a year of practice, they become cognizant of how all six are interconnected, and awareness occurs that developing one component awakens new insights about the others’ roles. Some authors of philosophies later employ visualization and supplement their philosophies with graphics.

Years ago, the fractal generator shown above became my graphical philosophy. I produced fifty articles under the title “Educating in Fractal Patterns…” for The National Teaching and Learning Forum, as that graphic philosophy helped me more deeply understand and grow from my experiences. Those who maintain a written philosophy and reflect on it regularly will almost surely have similar “Aha moments.”

Essential Components

Let’s see how we can help students become reflective and increase their capacity to learn from the six components.

Affect

We can help students to appreciate the importance of affect by reflecting upon whether the affective mode in which they find themselves is “I have to do this” or “I want to do this.” (See assignment shared in Nuhfer 2014.) Wanting to learn enlists more brainpower to drive learning. Finding ways to make learning fun for ourselves, to want to do it—such as making a party of it by studying with others, can become our most valuable asset. Quantitative courses elicit the most negative affective responses from students (Uttl, White, Morin, 2013). Yet, the book I recommend as the most inspirational book ever written to learn to reverse negative feelings for specific content is Francis Su’s Mathematics for Human Flourishing. A particular quote I like from p. 11 follows: “When some people ask, ‘When am I ever going to use this?’ what they are really asking is ‘When am I ever going to value this?‘”

Pedagogy

Teachers employ “active learning” pedagogies (Univ Wyoming resources) to increase learning through engagement. One principle underlies all “active learning:” the more of the brain invoked during learning, the better the learning. But another principle is seldom addressed: the longer the time spent in learning with significant portions of the brain activated, the greater the understanding.

Students can apply the same principles to enlist more of their brains. Writing to learn along with reading invokes more of the brain than only reading to learn. Revision of written products, multiple revisions, is one of the most powerful learning strategies known, at least as powerful as any modality on the “Active Learning Spectrum” linked above. For developing their learning philosophies, instructors should assign students to write, revise, and record at the end of each revision how doing revision improved their understanding. Learning to enlist active learning by writing to learn for oneself does not require doing a thesis. Simply writing and revising how to solve a word problem or an evaluative assignment offers sufficient capital through which to develop an appreciation for the power of writing to learn.

Content

Writing (Didn’t we just mention its power?) a knowledge survey builds understanding of the course content that faculty quickly appreciate (Nuhfer & Knipp, 2003). Watch these two very short videos to get a sense of what knowledge surveys are and their impact.

Screen shot of Dr. Ed Nuhfer talking about what knowledge surveys are and their benefits — https://www.youtube.com/watch?v=ENW51282Bwk

screenshot of opening slide stating "Faculty reflect on the value of knowledge surveys" — https://www.youtube.com/watch?v=XWbw8buSXIo

With each knowledge survey item that a professor writes, they should follow that with the thought, “What do students need to know to respond with understanding?” That will lead to writing several additional items to build the scaffold needed to address the former item. While students lack the expertise to construct course-based knowledge surveys from scratch, instructors can direct students to share in scaffolding their learning. They can let students build a knowledge survey in small pieces by frequent assignments such as: “Given the content we covered today, replace the knowledge survey’s items about that content with your own authored items to address the equivalent content.” An additional brief helpful exercise to help students to build their own learning philosophy can be: “How did writing your own knowledge survey items change your understanding of the content material? Why might writing such items be a useful learning strategy for learning in other subjects and courses?”

Levels of Thinking

We have already covered developmental levels in a prior post (Nuhfer, 2014), and we showed the connection between levels of thinking and affect (Nuhfer, 2018). Nearly all students misperceive becoming educated as learning skills and content. They have heard of “critical thinking” or “higher-order thinking,” but almost none can really articulate what either looks like. They will not know about such developmental levels of thinking unless the faculty teach them. We earlier provided a module with which instructors can do so (Nuhfer & Bhavsar, 2014). If you are unfamiliar with such developmental models and levels of thinking, which is likely, go through the same module for yourself. Then, guide students through it.

Metacognition

Is it surprising that this component should receive a mention on this blog? What would be truly surprising is if any personal learning philosophy sentence was not a product of metacognitive reflection. A written philosophy archives the portions its writer valued that came out of purposefully directed conversations with self. It expresses the current state of the author’s focus at a specific stage of development. It will change with additional experiences and reflections.

Assessment

The assessment portion of the philosophy likely considers three questions. The first is “Did I really practice my philosophy—did I do what I planned to do?” The second is “What happened as a result?” The third is “Based on what I learned from what happened, what will I focus on next?”

When one develops the ability for “fractal thinking,” one is constantly considering what would result if a pattern of action taken at one scale were enacted at different scales. If these three assessment queries give valuable substance to building an individual’s expertise, what might result if we nurtured such happening at the scale of a program or an institution?

For years, I had hoped to convince an institution to replace the practice of using student ratings scores as the highest stakes criterion for the annual review of instructors with professors’ written philosophies instead. Review committees would examine each philosophy and ask correlatives to the three questions above. It seems easy to see why encouraging metacognitive reflection on improving practice offers a superior option to thinking instead of raising student ratings’ scores. Sigh! I’m still hoping for just one institution to try it.

In summary, writing, revisiting, and revising learning philosophies scaffold us to higher proficiency. Addressing the same six components offers professors and students common ground on which to come together to understand learning and the process of becoming educated.

Fractals and Teaching Philosophies (Part 2): Some Reflection on the Value of Metacognition

by Dr. Ed Nuhfer, California State Universities (retired)

Our previous blog contribution introduced the nature of fractals and explained why the products of intellectual development have fractal qualities. Our brain neurology is fractal, so fractal qualities saturate the entire process of intellectual development. Read the previous blog now, to refresh any needed awareness.

The acts of drafting and using a written philosophy are metacognitive by design. Rare use of philosophies seems symptomatic of undervaluing metacognition. When we operate from a written philosophy, each day offers a practice of metacognition through asking: “Did I practice my philosophy?” That involves considering where we might not have exercised one of the fractal generator’s six components (Figure 1) and then resolve to do so at the next opportunity. Doing so instills the habits of mind needed to do what we intended. Such metacognitive practice is very different from engaging challenges as separately packaged events, without using the “thinking about thinking” needed to understand how our practice was consistent with what we wanted to do. In teaching, we find some of our most significant difficulties appear when we find ourselves doing the opposite of what we most wanted to do. We get into those difficulties by not being aware of the decisions that brought us there.

One possible application of metacognition lies in a large-scale challenge that affects all schools – the annual evaluation of faculty for retention and promotion often reveals chronic problems. How might the standard practices faculty typically experience differ from a practice in which faculty employed written teaching philosophies as a way to address this annual challenge?

WHY Do We Do Annual Review of Faculty?

A metacognitive approach would start with a reflection of the reasons WHY schools go through the prickly annual ritual of evaluating faculty and the outcomes that they hope to attain from doing it. A recent discussion on a faculty development listserv showed that almost no institutions have satisfying answers to “WHY?” For many, an unreflective approach to annual review commonly defaulted to ranking the faculty according to their scores from student ratings forms, sometimes from just one global item on the forms. Asking “WHY?” resulted in the following personal email from an accomplished faculty member: “The main rationale seems in practice often to be simply ‘We have to determine annual merit scores to determine salary increases, and so we have to generate a merit score for teaching (and for research and for service).’”

Given such an annual review process, the faculty will focus on becoming “better teachers” by focusing on raising their student ratings scores, but is that the primary outcome that institutions want? Would we write that “to obtain high student ratings” as a reason that we teach in our teaching philosophies? If we sincerely want effective teaching and student learning, is there a better answer to “WHY?”

Employing Written Philosophies – An Alternate Approach

More specifically, consider which outcome of the following you would choose to expend efforts for yourself or your colleagues: 1) to try to achieve higher student ratings or 2) to improve their mastery of some things labeled in Figure 1 that are known to increase student learning? For example, if a faculty member chose for one year to produce better learning by expanding his knowledge of pedagogy to permit the matching of different kinds of instruction to specific types of content, could that be preferable? Suppose another faculty member discovered that particular stages of adult thinking existed. What if she aspired in the coming year to gain an understanding of this literature, and she focused in the coming year on designing some lessons that helped students to discover the level of thinking they had reached and what their next higher stage might be? Might that be preferable to trying to achieve higher ratings?

Illustration of components (thinking, teaching, learning) in the fractal generator for faculty and students (by Ed Nuhfer)

Figure 1. We repeat the graphic from Part 1, Figure 2 here. This representation of a philosophy as a fractal generator is somewhat analogous to a stem cell in that it contains all the essential components to produce whatever we need. Metacognition allows us to identify something of value to our current practice. Then for a year, we articulate a philosophy that includes a focus to develop that area.

When we begin to be metacognitive concerning WHY we should want to do annual evaluations and how we should use student input, things should emerge that differ from merely sorting faculty into categories in order to dispense rewards and penalties. Some positive outcomes might be enhancing awareness of how we could design our annual evaluations to help make our institutions more fit places in which to teach and learn, or to provide our graduates with better capacity for life-long learning. In such cases, the nature of annual review changes from an inspection of each faculty member’s popularity with her or his students at the end of their courses into a metacognitive process designed to produce valued outcomes. Management expert Edwards Deming warned particularly about trying to “inspect in” quality at the end of an event or process. Deming’s 14 principles can be condensed into just one concept: “Be metacognitive.” Remain aware of what you most wanted to do when you take any actions to do it.

Changing the Annual Review Format: Embed Metacognition

“Be metacognitive” represents a significant change in most institutional thinking. So, how might we enact this change? One approach would be to design the annual review more like a self-directed contract for practice. Faculty write the philosophies that they intend to practice. A graphic generator like Figure 1 can assist understanding what one now does lots of and too little of. They pick a specific area that they want to do more of and articulate their intent to develop some additional strength in that area. They also articulate WHY they chose this emphasis and what outcome they seek to achieve.

When faculty start their term, they share with their students the emphasis and the outcome through the written syllabus of each class. During the semester, their practice now achieves a metacognitive quality. They regularly reflect on their practice and monitor themselves on whether they are practicing as intended. Their annual review of teaching then becomes a report with parts somewhat like the following.

Did they practice their stated philosophy?
How did their students respond?
How did their practice change, and did that contribute to revisions in their philosophy?
What is their written contractual plan and philosophy for the coming year?

Weighing the Alternatives

Of the two models of annual evaluations shared above, over-reliance on student ratings for faculty evaluation answers the WHY question with: “We maintain universities so that students can rate the faculty and so that faculty will strive to be rewarded for higher ratings.” Such absurdities arise whenever we practice with no better answers to “WHY?”

As a final thought, consider how an end-of-the-course grade for a student is analogous to annual evaluation for a faculty member. How might teaching students to write their learning philosophies improve their design for learning?

Awareness of Fractals Strengthens Metacognition Needed for Enacting Informed Teaching Philosophies

by Dr. Ed Nuhfer, California State Universities (retired)

Since 2002, I’ve written a theme-based column, “Developers’ Diary,” for The National Teaching and Learning Forum (NTLF). The central theme through all of these columns is “educating in fractal patterns.” Additionally, I facilitated week-long retreats from 1993 to 2010, and still run workshops, both of which employ visualization of a fractal generator as an aid to understanding concepts of teaching and learning. The wonderful “LAMP” (Learning Actively Mentoring Program) program at the University of Wyoming, where I serve as a mentor, continues to incorporate this aid in participants’ development of informed teaching philosophies.

In writing involved with our academic professions, perhaps no documents are so much the products of metacognition as our written teaching philosophies. These come from within us, which may account for their being so challenging to write. The information-gathering and evidence-based kinds of education through which we mastered most of our own education rarely gave us much practice for metacognitive self-assessment and deep self-reflection.

When properly used, the value of a teaching philosophy lies in “shaping” and nurturing the continuous growth of its author’s expertise. Rather than just a statement, the document serves to direct the author’s intention to enact the practices espoused in the philosophy. In this column, I seek to infuse readers’ already developed metacognitive capacities with an added dimension of “fractal awareness.”

Fractals: Why “Y” Why?

A fractal form is one that develops through growing from a “seed” called a generator (Fig. 1). Development involves repeatedly connecting additional generators to the growing structure. Thus, the character of the full form depends on the characteristics of the generator. A generator consists of simple Euclidean parts, perhaps the simplest being a straight-line segment. We enlist Figure 1 to clarify how initiators form generators, and fractal forms grow through recursively adding more and more generators.

Four levels of fractal development: initiator, generator, fractal form, complex fractal form

Figure 1. Development of a branching fractal form from a “Y-shaped” generator and its precursor initiator (from Nuhfer, 2007). Fractal shapes are the most common of all natural forms. Plants, mountains, clouds, coastlines, patterns of natural events in time like rainfall and floods, blood vessels, and the neural networks in our brains are examples of natural fractal forms.

The concept of fractal form is more than an abstract visualization that inspires creatively thinking about the process of becoming educated. The neural connections that develop in our brains through learning really are fractal forms. When we learn, we connect and stabilize fractal neural networks, so a good deal of our thinking and behavior almost surely has fractal qualities. We can enhance our understanding of educating and becoming educated by discovering the fractal qualities that these endeavors exhibit. One of the most important to recognize is that healthy final forms grow from robust generators. In practice, we can build a sturdy generator from a “blueprint” established by writing a well-informed teaching philosophy. If we mindfully practice this philosophy, the strengths and omissions of our “generator” grow into the strengths and blind spots that characterize our practice.

The branching fractals that develop in our brains are certainly more complex than the model in Figure 1, but even that simple figure helps us to understand and explain countless aspects of the process of learning and, over time, developing higher level thinking capacities.

The Philosophy as a Fractal Generator for Teaching, Learning, and Thinking

The statement, “Metacognition is thinking about thinking” always triggers the question, “What do we think about?” The fractal generator (Fig. 2) in use by me for about the past two decades tends to trigger six items for consideration in what to “think about” to build an informed philosophy. The meaning of “informed philosophy” extends beyond a document informed by a solid base of research on teaching, learning, and thinking. The term “informed philosophy,” as used here, is a document that reflects the growing understanding of ourselves in concert with our growth in knowledge, skills, and evidence-based practices.

Three components in blue (Fig. 2) are mostly components of skills and knowledge. Development of strengths in these three areas comes mainly (not wholly) from external sources. These include the research provided from the literature and from our network of colleagues who help us to build our content expertise and our awareness of varied pedagogical approaches and assessment practices. These originate primarily from resources from outside self, and we mostly develop our practice by drawing on these contributions.

Figure 2. A fractal generator model for higher education begins with an initiator that is affect. No deliberate efforts to teach or learn are devoid of affective qualities. Without affective desire to learn to value any of the six areas, such areas will not develop. Practice will then grow from a stunted generator.

The components in red that we call “internal strengths” (Fig. 2) require understanding that develops primarily from within us. The initiator for our generator (Fig. 2) is the red line segment at the base of the generator, which represents our affective feelings. Strong affective interest and enthusiasm may be our most valuable assets for guiding learning efforts to success. We needed to want to do something such as attend college, major in an area that felt attractive and to continue acting to achieve expertise by persevering to develop. That desire comes from within. When our affective passion and cognitive focus align for learning, we are unlikely to fail.

Finding Our Initiators from Within

In starting to write a teaching philosophy, a valuable awareness occurs when we query ourselves about how we obtained our present affective desires for what we aspire to do. Recalling an influential mentor often reveals from whom, when, and where that initial desire occurred. Recollecting a mentor’s valued qualities often reveals that how a teacher now hopes to be remembered began to form with learning to appreciate the power and validity of a particular mentor’s qualities. These recollections usually carry strong emotional ties, and early ideas that produced our conceptions of what constitutes good teaching can be beneficial if they really fit us. They can also be limiting if we unconsciously attempt to imitate a revered mentor rather than advance to develop the teaching that arises from our unique experiences and values.

Cultivating the habit of regular metacognitive conversations with ourselves allows us to confront a query of great importance: “Is what I am doing in the present truly what I most intended to do?” If not, the revised philosophy serves to direct our efforts back to regain doing what we intended to do. That practice allows us to tap the optimal power of affect by doing what a plan of deep introspection revealed that we most wanted to do in our practice. When a troubling event starts to occur, a valuable first reflection is, “Am I actually practicing my philosophy through how I am engaging with this challenge?” Often, we will find that troublesome events occur from a brief moment of inattention that sidetracks us into doing something other than what we intended to do.

Fractals and Uniqueness

In the neural networks that store the well-developed expertise within our brains, the separate neural components are in communication with one another, and they enlist one another to engage successfully with challenges or unexpected changes. Thus, the six areas of the generator (Fig. 2) that grow through our experience should grow to work simultaneously in active practice. Although I’ve found no contributions to research in faculty development that cannot be addressed from within the components of Figure 2, the fractal model is not one of prescriptive development. It does not lead to producing instructors in cookie-cutter fashion who all think alike and teach alike. Indeed, it cannot.

For the same reason that there are neither two trees nor two rainstorms that are alike, there can be no two brains that wire alike. Small differences between individuals’ generators occur through the unique experiences of each person. As these differences influence the replication through the repeated exercise of one’s practice, they guarantee the development of diversity and uniqueness of every teacher, every student, and thus every teaching moment experienced within a class. An internalized awareness that these will never occur again leads to consciously respecting others and valuing the present moment deeply.

We have seen in this brief entry how becoming aware of the pervasiveness of fractals in the physical world and understanding the role of the generator helps the author appreciate the utility of a written teaching philosophy for illuminating one’s own generator. Through the recursive process of repeated implementation, robust generators significantly strengthen one’s practice through time. In our next blog entry, we will examine metacognition’s specific roles in developing each of the six individual components.

Nuhfer, E. B. (2007). “The ABCs of fractal thinking in higher education.” To Improve the Academy (25) 70-89.

Paired Self-Assessment—Competence Measures of Academic Ranks Offer a Unique Assessment of Education

by Dr. Ed Nuhfer, California State Universities (retired)

What if you could do an assessment that simultaneously revealed the student content mastery and intellectual development of your entire institution, and you could do so without taking either class time or costing your institution money? This blog offers a way to do this.

We know that metacognitive skills are tied directly to successful learning, yet metacognition is rarely taught in content courses, even though it is fairly easy to do. Self-assessment is neither the whole of metacognition nor of self-efficacy, but self-assessment is an essential component to both. Direct measures of students’ self-assessment skills are very good proxy measures for metacognitive skill and intellectual development. A school developing measurable self-assessment skill is likely to be developing self-efficacy and metacognition in its students.

This installment comes with lots of artwork, so enjoy the cartoons! We start with Figure 1A, which is only a drawing, not a portrayal of actual data. It depicts an “Ideal” pattern for a university educational experience in which students progress up the academic ranks and grow in content knowledge and skills (abscissa) and in metacognitive ability to self-assess (ordinate). In Figure 1B, we now employ actual paired measures. Postdicted self-assessment ratings are estimated scores that each participant provides immediately after seeing and taking a test in its entirety.

Figure 1. Academic ranks’ (freshman through professor) mean self-assessed ratings of competence (ordinate) versus actual mean scores of competence from the Science Literacy Concept Inventory or SLCI (abscissa). Figure 1A is merely a drawing that depicts the Ideal pattern. Figure 1B registers actual data from many schools collected nationally. The line slopes less steeply than in Fig. 1A and the correlation is r = .99.

The result reveals that reality differs somewhat from the ideal in Figure 1A. The actual lower division undergraduates’ scores (Fig. 1B) do not order on the line in the expected sequence of increasing ranks. Instead, their scores are mixed among those of junior rank. We see a clear jump up in Figure 1B from this cluster to senior ranks, a small jump to graduate student rank and the expected major jump to the rank of professors. Note that Figure 1B displays means of groups, not ratings and scores of individual participants. We sorted over 5000 participants by academic rank to yield the six paired-measures for the ranks in Figure 1B.

We underscore our appreciation for large databases and the power of aggregating confidence-competence paired data into groups. Employment of groups attenuates noise in such data, as we described earlier (Nuhfer et al. 2016), and enables us to perceive clearly the relationship between self-assessed competence and demonstrable competence. Figure 2 employs a database of over 5000 participants but depicts them in 104 randomized (from all institutions) groups of 50 drawn from within each academic rank. The figure confirms the general pattern shown in Figure 1 by showing a general upwards trend from novice (freshmen and sophomores), developing experts (juniors, seniors and graduate students) through experts (professors), but with considerable overlap between novices and developing experts.

Figure 2. Mean postdicted self-assessment ratings (ordinate) versus mean science literacy competency scores by academic rank. Figure 2 comes from selecting random groups of 50 from within each academic rank and plotting paired-measures of 104 groups.

The correlations of r = .99 seen in Figure 1B have come down a bit to r = .83 in Figure 2. Let’s learn next why this occurs. We can understand what is occurring by examining Figure 3 and Table 1. Figure 3 comes from our 2019 database of paired measures, that is now about four times larger than the database used in our earlier papers (Nuhfer et al. 2016, 2017), and these earlier results we reported in this same kind of graph continue to be replicated here in Figure 3A. People generally appear good at self-assessment, and the figure refutes claims that most people are either “unskilled and unaware of it” or “…are typically overly optimistic when evaluating the quality of their performance….” (Ehrlinger, Johnson, Banner, Dunning, & Kruger, 2008).

Figure 3. Distributions of self-assessment accuracy for individuals (Fig. 3A) and of collective self-assessment accuracy of groups of 50 (Fig. 3B).

Note that the range in the abscissa has gone from 200 percentage points in Fig 3A to only 20 percentage points in Fig. 3B. In groups of fifty, 81% of these groups estimate their mean scores within 3 ppts of their actual mean scores. While individuals are generally good at self-assessment, the collective self-assessment means of groups are even more accurate. Thus, the collective averages of classes on detailed course-based knowledge surveys seem to be valid assessments of the mean learning competence achieved by a class.

The larger the groups employed, the more accurately the mean group self-assessment rating is likely to approximate the mean competence test score of the group (Table 1). In Table 1, reading across the three columns from left to right reveals that, as group sizes increase, greater percentages of each group converge on the actual mean competency score of the group.

Table 1. Groups’ self-assessment accuracy by group size. The ratings in ppts of groups’ postdicted self-assessed mean confidence ratings closely approximate the groups’ actual demonstrated competency mean scores (SLCI). In group sizes of 200 participants, the mean self-assessment accuracy for every group is within ±3 ppts. To achieve such results, researchers must use aligned instruments that produce reliable data as described in Nuhfer, 2015 and Nuhfer et al. 2016.

From Table 1 and Figure 3, we can now understand how the very high correlations in Figure 1B are achievable by using sufficiently large numbers of participants in each group. Figure 3A and 3B and Table 1 employ the same database.

Finally, we verified that we could achieve high correlations like those in Figure 2B in single institutions, even when we examined only the four undergraduate ranks within each. We also confirmed that the rank orderings and best-fit line slopes formed patterns that differed measurably by the institution. Two examples appear in Figure 4. The ordering of the undergraduate ranks and the slope of the best-fit line in graphs such as those in Fig. 4 are surprisingly informative.

Figure 4. Institutional profiles from paired measures of undergraduate ranks. Figure 4A is from a primarily undergraduate, public institution. Figure 4B comes from a public research-intensive university. The correlations remain very high, and the best-fit line slopes and the ordering pattern of undergraduate ranks are distinctly different between the two schools.

In general, steeply sloping best-fit lines in graphs like Figures 1B, 2, and 4A indicate when significant metacognitive growth is occurring together with the development of content expertise. In contrast, nearly horizontal best-fit lines (these do exist in our research results but are not shown here) indicate that students in such institutions are gaining content knowledge through their college experience but are not gaining metacognitive skill. We can use such information to guide the assessment stage of “closing the loop.” The information provided does help taking informed actions. In all cases where undergraduate ranks appear ordered out of sequence in such assessments (as in Fig. 1B and Fig. 4B), we should seek understanding why this is true.

In Figure 4A, “School 7” appears to be doing quite well. The steeply sloping line shows clear growth between lower division and upper division undergraduates in both content competence and metacognitive ability. Possibly, the school might want to explore how it could extend gains of the sophomore and senior classes. “School 3” (Fig. 4B) probably should want to steepen its best-fit line by focusing first on increasing self-assessment skill development across the undergraduate curriculum.

We recently used paired measures of competence and confidence to understand the effects of privilege on varied ethnic, gender, and sexual orientation groups within higher education. That work is scheduled for publication by Numeracy in July 2019. We are next developing a peer-reviewed journal article to use the paired self-assessment measures on groups to understand institutions’ educational impacts on students. This blog entry offers a preview of that ongoing work.

Notes. This blog follows on from earlier posts: Measuring Metacognitive Self-Assessment – Can it Help us Assess Higher-Order Thinking? and Collateral Metacognitive Damage, both by Dr. Ed Nuhfer.

The research reported in this blog distills a poster and oral presentation created by Dr. Edward Nuhfer, CSU Channel Islands & Humboldt State University (retired); Dr. Steven Fleisher, California State University Channel Islands; Rachel Watson, University of Wyoming; Kali Nicholas Moon, University of Wyoming; Dr. Karl Wirth, Macalester College; Dr. Christopher Cogan, Memorial University; Dr. Paul Walter, St. Edward’s University; Dr. Ami Wangeline, Laramie County Community College; Dr. Eric Gaze, Bowdoin College, and Dr. Rick Zechman, Humboldt State University. Nuhfer and Fleisher presented these on February 26, 2019 at the American Association of Behavioral and Social Sciences Annual Meeting in Las Vegas, Nevada. The poster and slides from the oral presentation are linked in this blog entry.

Measuring Metacognitive Self-Assessment – Can it Help us Assess Higher-Order Thinking?

by Dr. Ed Nuhfer, California State Universities (retired)

Since 2002, I’ve built my “Developers’ Diary” columns for National Teaching and Learning Forum (NTLF) around the theme of fractals and six essential components in practice of college teaching: (1) affect, (2) levels of thinking (intellectual & ethical development), (3) metacognition. (4) content knowledge & skills, (5) pedagogy and (6) assessment. The first three focus on the internal development of the learner and the last three focus on the knowledge being learned. All six have interconnections through being part of the same complex neural networks employed in practice.

In past blogs, we noted that affect and metacognition, until recently, were deprecated and maligned by behavioral scientists, with the most deprecated aspect of metacognition being self-assessment. The highest levels of thinking discovered by Perry are heavily affective and metacognitive, so some later developmental models shunned these stages when only cognition seemed relevant to education. However, the fractal model advocates for practicing through drawing on all six components. Thus, metacognition is not merely important for its own merits; we instructors rely on metacognitive reflection to monitor whether we are facilitating students’ learning through attending to all six.

The most maligned components, affect and self-assessment may offer a key to measuring the overall quality of education and assessing progress toward highest levels of thinking. Such measurements have been something of a Grail quest for developers. To date, efforts to make such measures have proven to be labor intensive and expensive.

Measuring; What, Who, Why, and How?

The manifestation of affect in the highest Perry stages indicates that cognitive expertise and skills eventually connect to affective networks. At advanced levels of development, experts’ affective feelings are informed feelings that lead to rapid decisions for action that are usually effective. In contrast, novices’ feelings are not informed. Beginners’ approaches are tentative and take a trial-and-error approach rather than an efficient path to a solution. By measuring how well students’ affective feelings of their self-assessed competence have integrated with their cognitive expertise, we should be able to assess their stage of progress toward high-level thinking.

To assess a group’s (a class, class rank or demographic category) state of development, we can obtain the group’s mean self -assessments of competence on an item-by-item basis from a valid, reliable multiple-choice test that requires some conceptual thinking. We have such a test in the 25-item Science Literacy Concept Inventory (SLCI). We can construct a knowledge survey of this Inventory (KSSLCI) to give us 25 item-by-item self-assessed estimates of competence from each participant.

As demonstrated in 2016 and 2017, item-by-item averages of group responses attenuate the random noise present in individuals’ responses. Thus, assessments done by using aggregate information from groups can provide a clear self-assessment signal that allows us to see valid differences between groups.

If affective self-assessed estimates become increasingly informed as higher level thinking capacity develops, then we should see that the aggregate item-by item paired measures correlate with increasing strength as groups gain in participants who possess higher order thinking skill. We can indeed see this trend.

Picture the Results

For clear understanding, it is useful first to see what graphs of paired measures of random noise (meaningless nonsense) look like (Figure 1A) and how paired measures look when they correlate perfectly (Figure 1B). We produce these graphs by inputting simulated data into our SLCI and KSSLCI instruments (Figure 1).

Random nonsense produces a nearly horizontal line along the mean (“regression to the mean”) of 400 random simulated responses to each of the 25 items on both instruments. The best-fit line has values of nearly zero for both correlation (r) and line slope (Figure 1A).

We use a simulated set of data twice to get the pattern of perfect correlation when the participants’ mean SLCI and KSSLCI scores for each item are identical. The best-fit line (Figure 1B has a correlation (r) and a line slope, both of about unity (1). The patterns from actual data (Figure 2) will show slopes and correlations between random noise and perfect order.

Figure 1. Modeling correlational patterns with simulated responses to a measure of competence (SLCI) and a measure of self-assessed competence (KSSLCI). A shows correlational pattern if responses are random noise. B shows the pattern if 400 simulated participants perfectly assessed their competence.

Next, we look at the actual data obtained from 768 novices (freshmen and sophomores—Figure 2 A). Novices’ self-assessed competence and actual competence have a significant positive correlation. The slope is 0.319 and r is .69. The self-assessment measures explain about half of the variance (r²) in SLCI scores. Even novices do not appear to be “unskilled and unaware of it.” Developing experts (juniors, seniors and graduate students, N = 831 in Figure 2B) produce a fit line with a slightly steeper slope of 0.326 and a stronger r of .77. Here, the self-assessment measures explain about 60% of the variance in the Inventory scores.

When we examine experts (109 professors in Figure 2C), the fit line steepens to a slope of 0.472, and a correlation of r = .83 explains nearly 70% of the variance in Inventory scores. The trend from novice to expert is clear.

Final Figure 2D shows the summative mean SLCI scores and KSSLCI ratings for the four undergraduate ranks plus graduate students and professors. The values of KSSLCI/SLCI increase in the order of academic rank. The correlation (r) between the paired measures is close to unity, and the slope of 0.87 produces a pattern very close to that of perfect self-assessment (Figure 1B).

Figure 2: Correlations from novice to expert of item-by-item group means of each of the 25 items addressed on the KSSLCI and the SLCI. Panel A contains the data from 768 novices (freshmen and sophomores). B consists of 831 developing experts (juniors, seniors and graduate students). C comes from 109 experts (professors). Panel D employs all participants and plots the means of paired data by academic rank. We filtered out random guessing by eliminating data from participants with SLCI scores of 32% and lower.

Figure 2 supports: that self-assessments are not random noise, that knowledge surveys reflect actual competence; that affective development occurs with cognitive development, and that a group’s ability to accurately self-assess seems indicative of the group’s general state of intellectual development.

Where might your students might fall on the continuum of measures illustrated above? By using the same instruments we employ, your students can get measures of their science literacy and self-assessment accuracy, and you can get an estimate of your class’s present state of intellectual development. The work that led to this blog is under IRB oversight, and getting these measures is free. Contact enuhfer@earthlink.net for further information.

Developing Affective Abilities through Metacognition Part 4: Exercising Academic Courage in Faculty Development

by Dr. Ed Nuhfer, California State Universities (retired)

Courage is a metacognitive quality that enables not having our actions limited or dictated by fear. Academic courage is a unique category of “moral courage.” Unlike physical acts of courage that occur in a brief time span, moral courage governs a day-to-day way of being and acting in practice, despite recognizing the forces that merit fear. Martin (2011) describes academic courage as perseverance through academic difficulty in the face of fear. The threatening environments of universities and the nature of courage that educators, particularly faculty developers, need in their professional practice are our focus here. With the notable exceptions of Palmer (2017) and Martin (2011), data-based studies of developed courage by teacher-scholars are nearly nonexistent.

“Fear” inevitably enters nearly all discussions about “courage.” While courage does not exist in the absence of fear, courage is a developed metacognitive capacity; fear is not. Acting with courage becomes possible when a person’s learning involves affective development along with intellectual and ethical development. (See the last blog in this series, Part 3.) Hannah, Sweeney, and Lester (in Pury and Lopez, 2011, pp 125-148) portray acts of courage as the products of a developed “Cognitive-Affective Processing System.” In brief, courage enlists a lot of brainpower.

Teachers have courage (image from http://tamaravrussell.com/2015/07/teachers-have-courage/)

Academic Courage versus Suicidal Tendency

Institutions of higher education produce unique threats, and several are increasing for faculty and faculty developers. The unifying theme for most fear in academia is the potential loss of career and livelihood. Because threatening faculty livelihoods for reasons other than incompetence or ethical violations is a detriment to society, tenure developed in the U. S. as a means to shield scholars from being intimidated and having their voices silenced. But tenure has not always worked perfectly. Tenure-track positions are vanishing as academic staff, with no prospects of tenure or its protections, now replace retiring tenured professors. Increased faculty vulnerability naturally accompanies increased apprehension and fear. People involved in threatening scholars’ livelihoods are politicians & regents boards, administrators, students, and colleagues.

Politicians in parts of the federal government, and a few states, now actively suppress or work to eliminate government researchers and college faculty who work on topics where investigations produce (or might produce) evidence that contradicts partisan advocacy. Political power now is wielded to weaken tenure and eliminate programs that produce knowledge that contradicts authoritarian positions and even university benefactors’ interests. Faculty can suffer reprisals for doing politically-charged scholarship. The World Wide Web and social media open today’s college faculty and students to unprecedented surveillance, so even work performed outside universities with scholarly journals or professional organizations produces risks. Sometimes, faculty can wonder whether every group on the campus has power over them. When faculty state that their careers “live and die by student ratings,” this discloses that they also fear their students. To some faculty, it seems that even students have supervisory power to take away faculty livelihoods. All of these sources of fear represent substantial threats. When a faculty member discloses fear, a developer must never think to trivialize or discount it. The reason for fear is likely real.

College faculty have responsibilities as teachers and scholars. In both, a responsibility provides an obligation to remain current about new knowledge, to add to the knowledge base, and not merely act as dispensers of knowledge. As new evidence validates the increased effectiveness of different instructional methods, teacher-scholars have obligations to maintain command of the current literature and to apply the best available knowledge to serve their students. Yet, doing so poses risks.

Introducing unfamiliar teaching structures to the classroom is not always received favorably by students or by colleagues. Striving to enact nontraditional instructional techniques carries risks of becoming unpopular by not satisfying students with teaching as they expected to be taught. Where the retention of livelihoods rests more on popularity with those who have the power to take livelihood away than on the quality of research achieved or the learning promoted, acting from fear of becoming unpopular has potential consequences.

A courageous action is not rash or suicidal; it recognizes and respects actual threats. The temptation to bulldoze ahead with “best practices” by believing that one has superior knowledge disrespects a genuine threat. Acting courageously respects the threat and works to understand it, but without giving into fear. Acting from courage requires more brainpower than does just giving up. Building capacity to enact academic courage involves a lot of work. A courageous approach may strive to find empathy for colleagues and students, seek awareness of the reasons for their dissatisfaction and will work to gain an understanding of what changes will be needed for those in opposition to begin to accept and support the most beneficial actions. A failed first attempt will serve to inform later efforts.

Face management as a Cultural Challenge to Courage

Face management seeks to advance oneself socially through associating and being seen with those who are popular and/ or influential. Face management is a way of life in many organizations. Its dark side appears when advancing self involves marginalizing those perceived as unpopular or just different. Faculty being ostracized can include those who are struggling to make changes and newcomers with new ideas, perhaps controversial to local established practices, who are striving to be accepted and valued. Stanford professor Robert I. Sutton referenced such face-managing actions as “kiss-up; slap-down,” and recognized that the behavior could render toxic the culture of an entire organization–a department, a college, and even an institution. Ostracism is something any individual should fear. Researchers at the University of British Columbia’s Sauder School of Business studied the impact of ostracism on employee health and morale and discovered that it can be even more harmful to one’s mental and physical well-being than harassment or other forms of bullying. For a faculty developer, supporting those ostracized entails risks of also becoming marginalized.

That’s a hazard in faculty development work because faculty development is a helping profession. A faculty developer’s responsibility is to support faculty who are in need of help, and they seldom are people who hold popularity or power. Anyone who aspires to be a faculty developer needs to realize that he/she will likely experience such a threat multiple times. Often, faculty in need are also being marginalized, struggling to succeed and sometimes suffering the consequences of being caught crossways in a toxic, unit-level culture of kiss-up-slap-down. Unfortunately, face-managing cultures are tolerated, even nurtured, in universities, perhaps because courage isn’t taught in college.

While the exercise of academic courage in faculty development requires knowledge and skills, both are external qualities. Acting with academic courage is different. It’s an internal capacity that infuses knowledge and skills with empathy and affect. Internal development is a ceaseless metacognitive reflection, and it is a LOT of work.

References

Martin, A. J. (2011) Courage in the Classroom: Exploring a New Framework Predicting Academic Performance and Engagement. School Psychology Quarterly 26 2 145-160.

Palmer, P. (2017) The Courage to Teach: Exploring the Inner Landscape of a Teacher’s Life. 20th Ed. San Francisco, CA: Wiley.

Pury, C. L. S., and S. Lopez, eds. (2011). The Psychology of Courage. Washington DC: American Psychological Association.

Developing Affective Abilities through Metacognition Part 3: Recognizing Parallel Development of Cognition and Affect

by Dr. Ed Nuhfer, California State Universities (retired)

In Part 1, we showed how the initial views of behavioral scientists toward metacognition and affect led for a time to a view of intellectual development as exclusively cognitive. In Part 2, we showed that established ways of knowing each rest on unique concepts, and gaining a working understanding of any way of knowing requires first becoming aware of its supporting concepts.

In Part 2, we used the way of knowing for reaching ethical decisions to illustrate the practical necessity of understanding the four components of ethics and their relationships to each other. There seems to be no profession in which thought and practice do not involve ethical decisions, so it seems no accident that William Perry chose the title: Forms of Ethical and Intellectual Development in the College Years for his landmark book describing how higher education, when successful, changes students’ abilities to think.

Major ways of knowing, obviously ethics but even heavily objective ways of knowing such as science or quantitative reasoning, require us to commit to decisions that resolve conflicts between what we feel we want to be correct with what new knowledge leads us toward knowing to be correct. When a conflict occurs between feeling and knowing, it often arises from life experiences that we have not critically examined but which new knowledge and/or newly acquired processes of critical examination force us to confront. For part 3, we examine the role of metacognition to help understand how intellectual progress causes us to feel in certain ways as we work to gain a college education.

About a decade ago, I discovered that the Bloom team’s Taxonomy of the Affective Domain mapped so well onto the Perry Model of Intellectual Development (Nuhfer, 2008) that it provided a much-needed map for empowering metacognitive reflection on both affect and cognition. The map, summarized in Figure 1, greatly clarified for me how to better promote metacognitive development in both students and faculty. I hope that readers will find this map equally useful.

The researchers’ named equivalent stages of development appear in Figure 1’s rows, and the affective feelings noted in the middle column were those that I deduced from examining the affective comments of students recorded in Perry’s book and other studies, made within the stages deduced through researchers’ longitudinal interviews. Longitudinal studies were the basis for the Perry stages and also for the studies that followed after Perry (see Journal of Adult Development, 2004).

Figure 1. Parallel development of intellectual and affective capacities through higher education (slightly modified from Nuhfer, 2008). Metacognition must engage with emotions (middle column) if it is to be effective in advancing adult intellectual development. Otherwise, metacognition becomes just an additional tool for increasing absorption of disciplinary content.

When students know that becoming educated involves passing through an established sequence of developmental stages, each with its own defining cognitive and affective traits, they have a map that they can use to discover their present location and to guide them toward what lies ahead on the path to gaining an education. Regarding metacognition’s description as “thinking about thinking,” awareness of the sequential stages with their accompanying emotions allows students to expect, reflect, and then resolve the discomforting affective feelings that arise. Trepidation and even some fear are normal, and they even can serve as important indicators of progress in cognitive growth.

Those who strive to become educated engage in a journey toward the highest Perry Stages of intellectual development through passing through the earlier stages. Achieving resolution of our reactive affective feelings that occur during these transitional stages is often an internal struggle. Metacognition, a reflective internal conversation with self about our thinking, seems indispensable to this growth.

Important Questions when Linking Bloom’s taxonomies and Perry’s stages

Bloom’s Taxonomy of the Cognitive Domain (see Scharff, 2017) is one of the best-known contributions to education, but experts debate the degree to which the Bloom cognitive levels are hierarchical, developmental products. In contrast, the developmental character of both the Perry model and the Taxonomy of the Affective Domain is generally accepted. That both address the sequential development of college students explains why the two map better onto one another than do even the two Bloom team’s taxonomies of the cognitive and affective domains.

The map provided by Figure 1 illuminates a possible deficiency of learning design in higher education. Educators consistently refer to Perry’s highest stages of intellectual development (7, 8 & 9 – see Figure 1) as the stages characterized by metacognitive reflection. The lower stages seldom receive that recognition, so why might that be? Is metacognition just not happening in the preceding stages? If so, why not?

If those who have actually engaged in metacognition throughout their intellectual development are just those few who develop metacognitive ability spontaneously on their own, this accounts for its scarcity in the earlier stages and how few achieve the highest stages. Because intellectual and affective development requires passage through a sequence of stages, we instructors can only increase the proportion of those who attain highest-stage reasoning abilities by infusing metacognitive skills into the earlier stages as a part of our instructional design. Such design would shift all students’ perceptions of gaining an education from absorbing content provided by teachers in classrooms toward developing abilities to understand content in concert with developing understanding of self.

Dangerous Passages

Two dangerous passages in the journey through the stages of intellectual development end the educational aspirations of many students to achieve a true education marked with a celebratory graduation. Figure 1 offers a map that reveals the dangerous passages of our journey, where impactful emotions can urge us to give up on our own development. These are places where metacognition informed by only a little research on adult development can provide valuable assistance.

Many lower-division undergraduate students fail to graduate by getting trapped at the lower Perry stages 2 and 3. Stage 2 students typically view the purpose of education as learning facts rather than as experiencing challenges that develop expanded capacities to think. Further, students in Stage 2 often learn that beliefs and childhood teachings that they revere are, upon examination, flawed and perhaps even untrue. This sends them to Stage 3 and the bankrupt belief that all conclusions and arguments are equally valid. From there, educators’ efforts to move students into higher stages of thinking bring forth students’ affective reactions of frustration and bewilderment. These negative feelings can negate students’ trust in teachers and raise students’ doubts about their own abilities. At this stage, gaining relief by giving up can seem an attractive choice.

Another passage takes a similar toll, but this one manifests later, where it produces attrition of nearly half of our brightest students who gained admission to graduate school to achieve doctorates. Most Baccalaureate graduates are only Stage 4 thinkers, and in graduate school, the barrier to completion is the required dissertation, which is a challenging, open-ended Stage 5 project. Stage 5 challenges cannot be addressed by the same approaches that brought much undergraduate success— demonstrating rote knowledge and ability to perform calculations that arrived at uniquely “right answers.” The transition into Perry’s Stage 5 brings proficiency to evaluate conflicting evidence and arrive, not at “right answers,” but at conclusions that are most reasonable after evaluating all of the relevant, conflicting knowledge currently available. This high-attrition passage, not surprisingly, comes again with strong emotions. Powerful negative feelings of personal inadequacy or “imposter syndrome” often accompany the efforts to advance out of Stage 4, and too many graduate students lose confidence and withdraw before they can make the transition. If these distressed students understood the nature of the situation they were in, they likely would persist, trusting that continued perseverance would bring the necessary punctuated transition to Stage 5. With this transition comes the confidence and awareness necessary to engage ambiguous problems, which include dissertations.

In blog column Part 4, we will look at developing the affective quality of academic courage, which allows one to persist through challenges that bring fear and erosion of confidence.

References

Gigerenzer, G. (2007) Gut Feelings: The Intelligence of the Unconscious. New York. Penguin.

Journal of Adult Development (2004) Special volume of nine papers on the Perry legacy of cognitive development. Journal of Adult Development (11, 2) 59-161 Germantown NY: Periodicals Service Co.

Nuhfer, E. B. (2008) The Feeling of Learning: Intellectual Development and the Affective Domain: Educating in Fractal Patterns XXV. National Teaching and Learning Forum 18 1 7-11.

Developing Affective Abilities through Metacognition Part 2: Going Granular

Ed Nuhfer, California State Universities- retired

In Part 1, we noted that the highest stages of thinking are not merely cognitive, but they require cognitive knowledge and skills with the addition of metacognitive reflection involving affect. We also promised to present some ways to help students increase the capacity for reaching these highest levels of thinking through using metacognition to understand and develop affective reasoning.

Granular components make up a whole shape

This contributed post, Part 2, has three components. The first recognizes that understanding a way of knowing can take two forms, global and granular. The second provides research-based evidence that gaining an understanding of a metadiscipline’s way of knowing (e.g., science) by gaining awareness of the essential interconnections (granular approach) that constitute the metadiscipline is more effective than trying initially to understand the metadiscipline through considering it as a whole (global approach). The third introduces an example of a heavily affective way of knowing—ethics— and its interconnected components.

From describing to understanding

The popular definition of metacognition as “thinking about thinking” invites a universal response: “OK. So, now what do we think about?” No individual invented or discovered any complex way of knowing, such as science or ethics. Instead, these ways of knowing developed over a long time through the collective contributions of many workers. Over centuries, added insights made awareness of new concepts possible, and better understanding allowed an improved global articulation of each specific way of knowing.

In a few years of college education, we strive to produce understanding of bodies of knowledge that took centuries to develop. We believe that an effective sequence of gaining understanding of a metadiscipline usually recapitulates the historical order of its development. This parallel process for understanding a complex way of knowing involves first becoming aware of the essential interconnected concepts. Afterwards, scholars have increased capacity for constructing their global understanding of a way of knowing by learning how each concept contributes to the reasoning process that characterizes that way of knowing. To aid teaching and assessments of major ways of knowing, it is valuable to distinguish how global and granular queries elicit different ways of thinking and understanding.

Global approaches to understanding address complex issues with a single question. Examples are “How do you treat others ethically?” and “How well do you understand science?”

Granular approaches to thinking address the interconnected concepts that enable specific ways of knowing. For example, the Science Literacy Concept Inventory (SLCI) (Nuhfer et al. 2016a) is a granular instrument. It addresses a dozen interconnected concepts that science rests upon through twenty-five multiple-choice challenges. The composite score on all twenty-five items provides the measure of competence to answer the global challenge of “How well do you understand science as a way of knowing?” It achieves this measure without either directly asking participants the global question or asking them to name any of the specific concepts.

An example query from the SLCI follows.

Which of the following statements presents a hypothesis that science can now easily resolve?

Warts can be cured by holding quartz crystals on them daily for a week.
A classmate sitting in the room can see the auras of other students.
Radio City Music Hall in New York is haunted by several spirits.
People with chronic illnesses have them as punishment for past misdeeds.

The query tests for a granular understanding of science as a way of knowing the physical world through testable hypotheses. The query seeks to see if a student can recognize which of the statements is testable and addresses the physical world. All four options present possible hypotheses, but only one option offers a testable hypothesis and addresses physical phenomena. Note that the query tests for understanding, not for a memorized definition of “hypothesis” or “science.” Answers to twenty-five such questions that address a dozen concepts give a highly reliable assessment of understanding science as a way of knowing.

Now comes the rub. Experts can perform effective metacognition of their understanding in direct response to a single complex global question because their understanding has already assimilated the essential granular concepts that underlie science. Their knowing “what to think about” now comes intuitively from long experience. Novices (students) who directly try to address a global question about a complex issue don’t yet have the experiences that enable experts to respond quickly by unconsciously incorporating the most essential granular concepts in their informed response.

Novices need to methodically consider each of the granular concepts as checkpoints before they can reach a well-informed response. With practice in doing so over time, they can internalize the concepts and intuitively employ them more holistically. An early start in recognizing that granular-to-global-understanding process helps to achieve internalizing earlier in one’s career or education. Without instruction, the process will not begin until a challenge makes the need for the skill apparent, and an inept response can prove costly if the challenge involves a high-stakes decision.

Granular disclosure deepens understanding quickly — the evidence from science

As noted, experts have the advantage of experience. However, their traditional educational experiences rarely included metacognitive reflection, so few of our current experts had the privilege of early understanding that might have resulted from undergraduate instruction on how to achieve an understanding of an ambiguous problem through metacognitive reflection on the most relevant underlying checkpoints of a relevant way of knowing. Many experts achieved this only after high-stakes challenges forced them to adopt more appropriate thinking.

If instructors explicitly engaged in relevant metacognitive instruction, might we be able to produce better future experts than exist now? Research says “yes” by showing that minds gain an increased global understanding of science simply from responding to a granular spectrum of queries that address the interconnected concepts that underlie science (Nuhfer et al., 2016b; 2017).

These research measures started with a global query that honestly disclosed the nature of the SLCI and asked students to estimate their anticipated scores. Our current dataset consists of 1576 participants, and the correlation between their estimates from this initial global self-assessment and their actual test scores was r = .28.

Following the global query, participants completed the SLCI knowledge survey. Knowledge surveys are granular self-assessment instruments that direct students to reflect metacognitively on the interconnected, granular components underlying a comprehensive topic. The SLCI contains 25 test items. For this research, participants first rate their competency on each item and then they answer all the questions. The correlation between the cumulative self-assessment on all 25 items on the entire knowledge survey and participants’ demonstrated competence from their score on the SLCI was r = .6. On later postdicted global queries (recorded after taking the knowledge survey and after taking the Inventory), the correlations between the global self-assessed scores and the actual SLCI scores all remained high at between r = .5 and r = .6.

These results offer a valuable insight: students knew no more content about science after taking the knowledge survey than they did before taking it because no instruction or study was involved. However, taking a knowledge survey provided a granular disclosure of what they must “think about” and conveyed a significantly better understanding of the complexity of the global query than did a detailed global description of the query. Improved metacognitive understanding of the challenge relative to one’s immediate competency is not the same thing as improved content knowledge. Rather, the former clarifies to the learner the specific content learning that one needs to get to improve his or her overall competency.

When we decide to teach a complex way of knowing, conveying an understanding of what the knowing involves (i.e., conveying the granular concepts) will contribute to success. Further, metacognitive exercises are more effective than hearing the key points in lectures, because metacognitive reflection is focused interactive engagement with the problem. The focused conversation with self that is the hallmark of metacognition enlists sufficient parts of the brain to build understanding. Listening alone engages relatively little of the brain’s neural network and produces little understanding that can be retained. Metacognitive exercises will be most effective if we build students’ competence through taking a granular approach from the very start. We want to direct our students to think about and internalize the checkpoints rather than to try to answer the global question directly from unexamined feelings.

From science to ethics

Science focuses on cognitive thinking that uses testable evidence. Instructors are most familiar with developing such thinking, which lies within Perry’s stages 4, 5 and 6. Developing highest level thinking abilities, (stages 7, 8 and 9) requires additional components that allow us to go beyond constructing strong, defendable arguments and enter the realm of using our results for making decisions and acting on them. These highest levels of thinking are metacognitive and affective. Reaching them requires that we develop an awareness of how our own affective feelings are an influence on our decisions, and it further requires that we develop capacity for empathy so that we truly understand how our actions impact others.

Like science, ethics constitutes a complex way of knowing, but the latter is a way of knowing that involves more affect. We treat one another ethically because we feel that we should do so, even when competing feelings and pragmatic arguments may exist to do otherwise in our perceived self-interests. Thus, an understanding of ethics requires understanding a different set of interconnected concepts.

The four granular ethical principles or concepts are, beneficence – “do good;” nonmaleficence – “do no harm;” justice – “treat equals as equals,” and autonomy – “respect others’ control over their own lives.” These provide our checkpoints for granular understanding.

To help readers initiate a global understanding of an ethical decision as experienced through a granular approach, I’ve included a short module exercise with this blog entry. Open it; read it. The text is less than 900 words. Afterwards, confront a few of the reflective exercises at the end of the module.

In Part 3, we can pick up our discussion with deeper exploration of the role of affect and metacognition in making ethical decision. Afterwards, we can explore the role of metacognition in other affective dimensions of thinking.

Developing Affective Abilities through Metacognition: Part 1

by Ed Nuhfer, PhD, California State Universities (retired)

Roman Taraban launched such an important topic for our blog on July 20 with “Hate-Inspired Webforums, PTSD, and Metacognition” that it is surely worth extending his discussion further.

Roman noted that groups develop recognizable vocabularies (discourse) and manners of speaking for set purposes. The purpose of developed vocabulary and manner of speech of hate groups is to enlist support and then empower and activate those with dispositions toward bias and bigotry. Activation in hate groups includes intimidation, shaming, shunning, and physical violence. Affect is the ultimate origin of discourse because the desire to promote such discourse is an affective feeling. Like cognitive thinking and psychomotor activity, affect is essential to human life and function. However, affect can guide us to act in ways that are ineffective, toxic, or destructive.

Learning and education are the processes through which we support and advance civilization. The purpose of civilization may be to elevate effective, beneficial actions and to minimize deleterious ones. Through learning and education, we develop frameworks of reasoning and processes for developing beneficial proficiencies. Examples of a psychomotor framework would be a process through which one learns to hunt for food, play a musical instrument, or to produce a painting. Examples of cognitive frameworks would be the logic of language and the use of testing and verification as a way of knowing through which we understand the physical world. An example of an affective framework is ethics—the way of knowing through which we evaluate the nature of feelings that are directing (or attempting to direct), our choices and decisions through which we act.

It is relatively easy to assess when psychomotor efforts are effective and successful. It is more difficult to see how language presents a fallacious argument or when an accepted cognitive perception about the physical world constitutes a misconception. It is most difficult to determine whether an affective feeling is likely to direct us to actions that are beneficial and healthy or toxic and perverse. We observe our affective state through metacognition, which is a purposefully directed internal awareness. Metacognition has an ineffable quality. In contrast, physical action and cognitive reasoning are easier to assess through their immediate products.

The history of education seems marked by an initial focus on the development of effective psychomotor skills needed for survival, technology, and simple arts. Later educational efforts offered an emphasis on written language, literature, increasingly sophisticated arts, and science. We finally are arriving at a time in Western education when an acceptance is dawning that becoming educated should proceed beyond cognitive and psychomotor development to understanding ourselves and our affective traits. This pattern seems inevitable because it is recapitulated on a smaller scale in our development as individuals.

If we are lucky, we start life acquiring the skills needed for our survival and further development. If we are particularly fortunate, we progress to gaining valid knowledge, valuable skills, and capacity for understanding and appreciating the social and natural realms in which we live. Finally, if we are uncommonly privileged through fortune, we can develop wisdom that promotes our living in an expanded awareness of our reality and increased capacity for nurturing and caring well for our natural world and others around us.

Given the progression outlined above, we should expect that metacognition will be our students’ most challenging and least-developed capacity for learning and becoming educated. As educators, we should also expect struggle and resistance, both individually and collectively, against the legitimacy of affective development efforts and metacognition as essential to becoming educated. We have already seen such resistance to these advances.

In hindsight, it now appears that Benjamin Bloom and his team of educators who worked in the 1950s and 60s seemed decades ahead of their contemporaries by recognizing the indispensable importance of the affective domain to the process of becoming educated. The Bloom team’s contribution on affect took many years before its importance was realized. At the time Bloom published his taxonomy of the cognitive domain, he was producing a second volume on the taxonomy of the affective domain (and still later, the psychomotor domain), the established behavioral sciences were focused solely on cognition. These sciences ridiculed affect, dismissed metacognition (see Dunlosky and Metcalf, 2009) and treated both as nonsense that obstructed objective reasoning and cognitive thinking. Bloom’s first volume on the cognitive domain became the most-cited educational reference in history, but the second volume on the affective domain fell into such obscurity that few college professors even know that it existed. The academic realm so de-legitimatized affective feelings that researchers from the 1960s into the early 1990s were actually afraid to study or write about emotions (see Damasio, 1999).

William Perry’s 1960s landmark work (Perry, 1999) was contemporary with Bloom’s research. Perry presented his discovery of distinct stages of adult intellectual development that he derived from analysis of language patterns (discourse) that manifested during interviews that Perry held over several years with groups of students. This longitudinal study found that students changed their thinking and reasoning process during years of becoming educated. Moreover, the interviews revealed that the highest stages went beyond cognitive thinking by incorporating and regulating metacognitive awareness of one’s affective inclinations. This discovery of the nature of highest-level reasoning arrived with awkward timing, given the regard by scholars for affect and emotions. In Perry’s entire book, reference to “affect” occurs only once (in a brief footnote on page 49) and to “emotions” only once (on p. 140). “Feeling” / “feelings” appear thirty-nine times, but mostly in the quotations of statements made by students during interviews. Perry seemed unable to write openly about these aspects, so the three chapters on his three highest stages are conspicuously brief. Today, a close reading of these chapters indicates that he had probably also discovered the development of emotional intelligence in his interviews, but he seems to have understood the dangers that any emphasis on emotion might pose to his larger discovery.

Another landmark book (King and Kitchener, 1992) that followed Perry’s interview approach refused to venture even that far. These authors restricted their investigation of higher intellectual stages to purely cognitive reasoning. However, by 2004 (Journal of Adult Development, 2004) a synthesis revealed that many investigations and classification schemes that followed Perry all mapped to each other and were essentially describing the same stages.

Bloom’s Taxonomy of the Affective Domain seems to map even better onto the Perry stages than it does to Bloom’s Taxonomy of the Cognitive Domain, (see Nuhfer, 2008) indicating that building affective capacity is indeed a developmental process. Thus, well-designed higher education curricula can build it, providing instructors design the curricula to produce the highest levels of thinking.

As an added benefit, development of metacognitive awareness is probably the best way to curtail the influence of “hate groups,” whether these be minor cults or mainstream establishment organizations. People with metacognitive awareness can perceive when their affect is getting involved from external attempts to direct their abilities toward beneficent or maleficent ends. In part 2, we’ll consider how teaching any discipline presents an opportunity to push thinking to highest levels through using metacognitive awareness to reflect on ethics, respect, courage, and gratitude.

References

Damasio, A. (1999). The Feeling of What Happens: Body and Emotion in the Making of Consciousness. New York: Harcourt.

Dunlosky, J. and Metcalf, J. (2009). Metacognition. Thousand Oaks, CA: Sage.

Journal of Adult Development (2004). Special volume of nine papers on the Perry legacy of cognitive development. Journal of Adult Development (11, 2) 59-161 Germantown NY: Periodicals Service Co.

King, P. M., and Kitchener, K. S. (1994). Developing Reflective Judgment. San Francisco, CA: Jossey-Bass.

Nuhfer, E. B. (2008). The feeling of learning: Intellectual development and the affective domain: Educating in fractal patterns XXVI. National Teaching and Learning Forum, 18 (1) 7-11.

Perry, W. G. Jr. (1999). Forms of Ethical and Intellectual Development in the College Years. San Francisco, CA: Jossey-Bass (a reprint of the original 1968 work with minor updating).