Assessment

The Evolution of ELT: Sharing Our Experiences in the Field

Today concludes the 25th Anniversary of the BA in ELT at the Universidad Autónoma de Aguascalientes. I am sharing the two presentations from the conference that focus on personal learning networks and professional learning and the importance of performance tasks.

Seldom do Teachers Become Lifelong Learners by Accident - September 10, 2018


When it comes to ELT, what makes a worthwhile performance? September 11, 2018



Academic Writing: Self-Assessment


In this fourth-semester class (English language teacher trainers/Spanish L1), I facilitate a group discussion around common writing errors stemming from their first drafts (of a five-paragraph essay).  Trainers were asked to look through their own work and choose if the error type was something they needed to consider or if it was something they already did well.

Common Academic Writing Errors Discussed in Class 

  1. Level I heading (APA)
  2. Thesis statement: transitional phrase
  3. Times New Roman, font size 12
  4. Spacing: Double space and equal spacing between headings and paragraphs
  5. Avoid
    1. To be
    2. It is important, it is necessary, it is vital, it is essential, etc.
    3. Passive voice with non-referential it
    4. Focus more on concepts than the authors
    5. Using modals
    6. Dictionaries and encyclopedia citations and references
  6. APA
    1. Last name and year for citations
    2. Direct quotes no more than 15%
    3. References: capitalization and italics
  7. Punctuation
    1. Serial comma
    2. At least three references and six citations
    3. Five to eight sentences in each paragraph
    4. Subject-verb agreement: check each (main, subordinating, and relative) clause.

Academic Research Podcast 1 (#arpscholar)

Article

Cooper, G. (2008). Assessing international learning experiences: A multi-institutional collaboration. Phi Kappa Phi Forum, 88(1), 8.

Problem

Implicit: relying on standardized testing or one-shot assessment practices that lack a broader understanding of a learners knowledge, skills, and attitudes/dispositions.

Purpose

To apply learning outcomes and assessment rubrics to international learning experiences.

Method

Participants

Undergraduate students from various disciplines that included six different participating institutions: Michigan State University, Portland State University, Dickinson College, Kalamazoo College, Palo Alto Community College, and Kapi'olani Community College.

Instruments

Eportfolios and an online student survey (i.e., Student Portfolio Information Form (SPIF)) was used in this study. The survey included both demographic information as well as information related to eportfolios.

Procedure

Data collection: Students (participants) included a variety of artifacts to be included in each respective learner eportfolio.

Analysis: Assessment rubrics were used to evaluate knowledge, skills, and attitudes of learners using the following scale: inadequate, minimal, moderate, and extensive. Knowledge, skills, and attitudes were then cross-referenced based on subjects, experiences, backgrounds, and fields of study.  A cross-tabulation of data approach was applied.

Results and Discussion

Thirty out of forty students (or 75%) who studied abroad scored extensive on the specified criteria while only fifteen out of sixty students (or 25%) who did not study abroad scored extensive. Possible conclusion: students who study abroad have higher scores in recognizing the importance and validity of others' perspectives.

Conclusions

  • Method and purpose for others to adapt study to local contexts (strengths)

  • Light on theory and results (weakness)

Hashtag: #arpscholar

What are you researching or planning on researching?  Have you conducted a similar study?  Leave your comments below or use the Twitter hashtag, #arpscholar.

Test Scores Can Matter


I read over Couros's (July 13, 2017) Looking Beyond the Score and I believe I agree with his overall thesis, but admit I would frame it in a slightly more nuanced way.  I identify several potential problems to consider when forming possible solutions to how we assess learners in formal education.

Google's Expectations


Google finds that GPAs and test scores are worthless as a criteria for hiring and that they do not predict anything... "after two or three years, your ability to perform at Google is completely unrelated to how yo performed when you were in school..."

To this, I ask the following:
  • What percentage of any student body goes on to work for Google?
  • What do Google's hiring practices say about the valid, reliable, and unbias nature of standardized exams used in schools?
  • To what degree do curriculum, assessment, and instruction align with each other within a particular school?
  • What's the ratio of formative to summative forms of assessment? Do schools favor one over the other?
  • To what degree do standardized exams measure both ability and aptitude?
  • What other types of decisions do school supervisors, principles, etc. make based on standardized exams results?

Couros's Claims


High test scores in school do not equal success in that area.  There are many other factors that need to be considered.

I would say that high test scores could be one of many contributing factors that lead to (some definition) of success.  This is different than saying unequivocally that scores do not equal success.  Just using the word, success, conjures up a vast array of potential meanings that are important to understand when discussing such a complex subject.
I have met many leaders in education who know all the theory but can't connect with people.

When recognizing the difference between theory and practice, I tend to shift the problem from only being about standardized exams to entertaining the possibility that other issues might exist:

  1. Standardized exams (both internal and external) may not be valid, reliable, and unbiased in representing learners' knowledge, understandings, abilities, and dispositions. Do teachers teach to the test (bad) or help prepare learners for the concepts that test items are supposed to measure (good)?  For more on this distinction, see Popham's (2007) Classroom Assessment...
  2. The curriculum, assessment, and instruction may not align with each other.
  3. The curriculum may not explicitly reflect a learner profile that contributes to society.
  4. Formative assessment may take a "back seat" to summative forms of assessment (i.e., standardized exams).
  5. Standardized exams may not measure both ability and aptitude.
  6. All decisions made based on standardized exams results may not lead to improving learning outcomes throughout the entire institution.

Improving test scores is one piece of a complex puzzle designed to prepare learners for improving society.  If attempting to improve test scores fits students into a box, then perhaps the solution is not to disregard the potential value of what exams could measure - in terms of decisions made to improve ongoing student achievement. The solution is to not remove standardized exams or grades altogether, but rather to understand the six points above in recognizing the complexity of establishing the role of assessment in schools.  I agree, there is more to life than test scores, but this does not mean that test scores do not have a place in formal education.

Giving Grades is not All Bad….

This week I read Chiaravalli’s Teachers Going Gradeless along with follow up remarks and felt compelled to share my thoughts on the matter in a slightly longer post. So here it is…

Going Gradeless


Regarding the grade less/gradeless continuum, I’d view it a bit differently. Gradeless (no grades) and a grading system (one or more grades) are dichotomous, an all-or-nothing proposition — teachers either assign a grade for the class or they do not. A true gradeless scenario (like the U. of Michigan’s Residential College) is a completely different context in my view. Presumably, students receive comments (based on criteria-based evidence) related to their progress and outcomes from a pass/fail perspective and are less inclined to be ranked with their classmates (generally a good thing). Because giving grades is only one form of summative assessment, the question becomes then if this is just doing away with grades only or doing away with all forms of summative assessments like badges, rubrics, etc. Even when assigning a rubric score, students are inclined to rank themselves in order to determine how fair they were evaluated.



Implementing Less Grades


Regarding the implementation of less grades (one or more grades), a continuum does exist. But the difference here is that students are (again presumably) more inclined to rank themselves by comparing their grade(s) with their classmates (generally a bad thing). Even by assigning one grade at the end of the course, I feel students will compare themselves with their classmates, which rarely adds to the educative experience. For me it’s not enough to resign to the idea that assigning only one grade (at the end of the term) is inherently better than assigning more grades throughout or is better than a more “traditional” approach where formative assessment is lacking. 1) There are alternatives to using formative and summative assessments that can address this challenge and 2) most agree to the benefits of frequent and timely doses of formative assessment in terms of their benefits to learning.


Sanchez & Dunworth (2014) found that graduate students receiving written feedback from tutors was oftentimes difficult to understand. In a university where grades were assigned, content from written feedback was hard to interpret for students when trying to align formative assessments (from written comments) with summative assessments (from receiving a grade). In other words, students perceived “a lack of consistency between formative and summative feedback” (p. 9). Even though some of the participants of this study were international students, I believe that these results remain generalizable to most students in high education of all levels. The study also discusses different challenges students face in receiving timely feedback such that changes that affect the final grade can be made before specific due dates.


Personally, negotiating with students at the end of the term their final grades is simply too little, too late. If this type of communication is going to exist, then it should exist throughout the term. Students benefit from receiving provisional grades throughout the course with the understanding that changes to these grades can be made with evidence of higher academic achievement.



Conclusion


It’s not enough to compare gradeless or grading less learning designs with more “traditional” designs of saturated amounts of summative assessments, but to tease out the current complexities of what gradeless and grading less actually mean in a variety of contexts. Nor is simply grading less necessarily optimal over a concerted effort to provide timely feedback throughout the educative experience that includes both formative and summative assessments. Teachers rely on a variety of forms of evidence to make better inferences to students’ understandings.


Originally posted to Medium.

How can a diagnostic test determine "fossilization"?

  1. When I read "fossilized error" (term used in quotation marks), I suppose that Jagasia does not really accept the term, but uses it anyway given that others have come to define the term a certain way in the past? Later the term is not in quotation marks that would suggest that it is an accepted term?
  2. The thesis of this piece seems to have less to do with "fossilization" and more to do with 1) feedback and assessment, 2) differentiated instruction (DI), and 3) potential issues with the placement test in terms of validity, reliability, and/or bias. So, to answer the first question from the title would include more to do with feedback, DI, and diagnostic test validity, reliability, and unbias than some notion of "fossilization". These were issues not covered in the piece.
  3. Regarding point #2, in the absence of any real example, it is hard to support the idea that just because a student has a tutor (e.g., italki, Verbling, etc.) that "fossilization" is less likely.  To understand "fossilization" is to observe a language learner longitudinally and not necessarily the individual learning spaces where learning (or lack thereof) takes place.  Another way to state this is that observations (to understand "fossilization") need to be understood from a diachronic versus synchronic lens. 
  4. The problem that this piece sets out to address is not clear.  Jagasia (2016) states, "Almost every student who sits our placement test possesses a significant amount of ingrained or "fossilized" errors" (para. 2). Again, how can a single diagnostic exam (synchronically) measure fossilization presumably before the fact?  Perhaps the assumption is if a language learner is taking a more advanced level class but is still making lower-level errors that this automatically means that "fossilization" occurs.  Intuitively, one can see the weakness of this argument when considering other possibilities: 1) learners acquire the language at different rates, 2) learners had little-to-no exposure to a grammatical structure, 3) learners had little-to-no practice in moving understandings from short term to long term memory, 4) the error could have be a "slip of the tongue" or a mistake that the learner really understands but carelessly overlooked, 5) personal circumstances that would interfere with concentration during the test, 6) simply a poor test taker, 7) poor alignment between diagnostic test and instruction, 8) poor alignment between diagnostic test and assessment, 9) poor alignment between curriculum and diagnostic test, etc. 
Forgetting the term "fossilization" for a moment, how students make mistakes (whether repeatedly or in isolation) cannot be viewed entirely from a single diagnostic test.  It seems for Jagasia (2016) as if the diagnostic test is detecting issues of instruction or assessment?  To understand how students make mistakes requires observations that occur over time in terms of how feedback is given and received and requires test designers to recognize the integrity of the instrument and its purpose.

Assessment-based instruction...always applicable?


Hashtags: #assessment #formativeassessment #summative assessment #testing

Inspired by a Google+ chat on a discussion of assessment and instruction, I felt compelled to discuss how I see the relationship between the two terms.

I’d like to address what Seburn finally concludes in his last post:

I would suggest rather the assessment and instruction are working off each other, not one driving the other so black and white. While I agree assessment should not simply be an afterthought, which really dismisses its value (as though it’s simply tagged on to the end of a course out of necessity), I don’t think it’s what should be made first either, before the instructional content. That lends to the danger of simply teaching to the test and not allow for the flexibility of adapting instruction to emergent student needs.

There are few things in life that I feel are black and white, right or wrong, better or worse, etc. But when it comes to assessment and instruction, my belief is that assessment should come first…I’ll explain.

Formal education

A common reality in formal education, or learning in schools, universities, etc., is that goals, objectives, and values, and usually expressed in terms of a curriculum. The written curriculum states what learning outcomes can be expected. The taught curriculum (which can differ from the written curriculum) involves a great number of variables that influence how the written curriculum is interpreted and employed within each classroom. If a curriculum is evaluated, then some attempt is made to reconcile any possible disconnects between the written and taught curriculum.  Most often this implies a change in the taught curriculum more than a change in the curriculum. So, at the level of institution, the premise that outcomes are planned beforehand is fairly straightforward: instruction has yet to be implemented in this scenario. Instruction, in fact, is designed later as a means for achieving certain ends, ends that are articulated in the written curriculum, interpreted by individuals (teachers) in the taught curriculum, and (hopefully) reflected upon later to evaluate any discrepancy between the written word and human discernment.

Let’s move now to the level of the individual classroom or teacher. Based on the curriculum, a syllabus, scheme of work (e.g., weekly schedule), and more detailed lesson plans are all planning devices that ideally align with each other. The goal however remains the same: to provide a (written) predetermined “roadmap” as to what can be expected in terms of learning outcomes. Lesson plans will include instructional designs, but only after assessments have been established beforehand that articulate the learning outcomes that align with the scheme of work, syllabus, and written curriculum. Assessments are where expectations (of learning outcomes) are revealed and may appear in the lesson plan, scheme of work, syllabus, and the curriculum.

So, lesson plans follow the same logic as the scheme of work, syllabus, and written curriculum. In formal education, planning for expected outcomes drives instruction. Planning for performance tasks, creating academic prompts, and factoring in different types of formative assessments beforehand creates the “blueprint” for instructional design. Taking into consideration both formative and summative assessments collectively guide teachers and students to the end goal. I'll all for heavy formative assessments, but the absent of summative assessments create a lack of goals (expected learning outcomes) which will lead to detours throughout the learning journey.

What happens when assessments are planned after instruction? A teacher begins instruction (with no specific end in mind) and after a few weeks decides to think about assessment. Perhaps formative assessment was employed, but with no clear performance task, academic prompts, etc. being considered, there is more of a likelihood that assessments end up aligning more with instruction than aligning with the scheme of work, syllabus, and/or curriculum.  In other words, summative assessments get tweaked to accommodate prior instruction.  An alternative is planning assessments with students (an option) which can serve as a motivation tactic that allows students to establish an end goal collaboratively before beginning instructional activities. If assessments are planned afterwards, it’s as if the logic behind planning a lesson plan differs from the planning that goes into each week, each course, and each school-wide program.

Assessment-based instruction does not equate to a “cookie-cutter” approach to education. Differentiated instruction (DI) provides opportunities for a more democratic way of learning. DI affords learners choices in what content they interact with, how they decide and ultimately interact with such content and with others, and what products they will create.  Teachers create this learning environment based on students' readiness levels, interests, and learning preferences. Negotiating performance tasks with students (before instruction) is an example of DI which permits students to invest in their own learning. Depending on the context, students can even have a say as to how they will be evaluated. DI works when teachers and students are co-creators within an overall learning design, one which still plans for assessment before instruction.

Today, high-stakes exams (or standardized tests) are a reality. A common misconception is that assessment-based instruction is the same as teaching to the test. Teaching to the test is preparing students for the test items that appear on a standardized test. Assessment-based instruction is not about teaching to the test; it’s about assuring that instruction allows students to gain the understandings, skill sets, and disposition that standardized test items set out to measure.

Informal education (intentional)

Informal education that is intentional, or teaching and learning typically occurring outside of school for a particular purpose, may differ from formal education assessments in how they emerge, but still should be designed before instruction. When one sets out to learn something (intentional learning), goals are set. There may be no restrictions to time and place, but setting goals helps one reflect on one’s progress. Goals align with the overall purpose and can be the driving force behind instruction. It’s important to note that instruction (whether in formal or informal educational contexts) means one person or group imparting understandings, skill sets, and dispositions onto another person or group.

Informal education (incidental)

People are always learning, whether they are aware of it or not. But I would argue that incidental learning alone seldom occurs - for the most part, incidental learning co-exists with intentional learning since humans usually behave around a certain purpose or for a certain reason.  I'll just say that my thesis pertains to intentional learning only; if someone else wishes to build an argument solely around incidental learning and assessment/instruction, have at it.

I agree with Seburn when he says, I would suggest rather the assessment and instruction are working off each other...only as it applies to formative assessment and only if what is meant by "working off each other" means making adjustments to instruction and learning tactics during designated times (as opposed to some fluid exchange happening continuously).  Education should be heavy in formative assessment because it promotes learning, I concur.  But I don't agree with no forms of summative assessment being put to use.  Summative assessment tends to get a bad rap because many associate it with tests, quizzes, etc. where these are the primary forms of assessment (i.e., with little-to-no formative assessment). But measuring learning (i.e., summative assessment) also helps students (and all educational stakeholders) see where they've been and where they've ended up.  It also provides criteria for goal setting and purposeful education (as in the case of performance tasks where rubrics are used).  In other words, both formative and summative assessment complement each other.  In order for them to complement each other, assessment-based instruction needs to replace assessment that emerges after instruction has emerged.  Most enter a car knowing where they will end up and prepared to take an alternative route if necessary.

So, what do you think? What's the relationship between assessment and instruction? 

Photo attribution


English Language Teaching (ELT) Lesson Planning and Assessment (#eltlive, #keltchat)


[youtube https://www.youtube.com/watch?v=ZJtjIpwVqNE]

I enjoyed today's #eltlive discussion on lesson planning.  The main takeaway for me was how to associate lesson planning around the idea of assessment.  One of the questions I posed was how can lesson plans be assessed, which shifted the conversation to the importance of assessing students during the implementation of the lesson plan (i.e., formative assessment).

Assessing students

Comments were made about how we receive feedback from students and how we typically reflect in action, to borrow from Schon (1983). The conversation included the dichotomy of covering content by strictly sticking to the lesson plan and being flexible with the lesson plan based on how students are performing in class.  And although this relates to assessing students indirectly, it doesn't exactly reveal how we plan lessons around the assessment of students.

I tend to think of lesson planning as being a backward design (Wiggins and McTighe, 2005; Popham 2008).  We plan our lessons around assessments first (presumably based on course objectives), then decide on the most appropriate learning sequence.  By deciding on assessments first, we prepare a "road map" that frames the journey students are to take in order to achieve particular outcomes (whether these outcomes are part of the curriculum or pre-determined by the language teacher and students).  This approach to assessment is the opposite of planning a lesson sequence first, then thinking later how to assess students on what was covered in prior classes.  In a backward design, the point is to "uncover" content and not merely cover it.

Assessing the lesson plan


Throughout today's discussion, I also kept thinking about how others might assess their lesson plans (I ask because I don't do nearly enough of this).  Assessing lesson plans can take on three forms: 1) assessing the lesson plan before implementing it, 2) assessing the lesson plan while implementing it, and 3) assessing the lesson plan after implementing it.  Assessing before the class might involve sharing and collaborating around a lesson plan with colleagues, students, admins., or any other education stakeholder around course objectives, materials or technologies used, among others.  Assessing during the class is not necessarily the same as assessing students as mentioned earlier, but rather would include reflection in action in terms of students' actual behavior and how one originally planned students would behave before class.  And finally, assessing after the class would not only be an individual reflection (reflection on action) on students' actual behavior vs. planned behavior, but could also be a shared experience with others (e.g., via social media).  All three ways to assess a lesson plan include distinguishing between intentional and incidental student behaviors that are either favorable or unfavorable.   

One idea I heard repeatedly was that many of us know when students are engaged, on task, etc. which we then can assume to mean that the lesson plan went well...and this quite often might be the case.  But I have oftentimes been surprised to finish a lesson, think that all went well, only to find out (after asking students) that it did not go quite as well as I had originally hoped.  It's not a stretch to acknowledge that misinterpretations can exist when it comes to the signals students provide in class and assumptions we place around those signals.

I always appreciate those who take part in these open, online discussions (like #eltlive), whether they are HOAs, Twitter feeds,  or through some other means because it gives me perspective and awareness that teaching in isolation does not have to be the norm and that professional learning opportunities continue to be at our fingertips.

Applied Linguistics Group Can Hang!

I had the pleasure today to hangout with other EFL/ESL educators (via Google+ Hangouts on Air - HOAs) along with my 7th semester, applied linguistics group to discuss how we feel about portfolios and sharing work openly online.  We also discussed how feedback and error correction might take place and different rationales for doing so.

[youtube https://www.youtube.com/watch?v=jdndIy1f5nY]

After today's HOA, I immediately began thinking about how we might schedule future sessions since my group really enjoyed the experience.

[youtube https://www.youtube.com/watch?v=tyEWKmmsDBU]

I would like my students taking applied linguistics this semester to leave comments below regarding any topics we might discuss in our next session, and anything else about today's HOA that you'd like to share.

Performance-based assessment in action...into action


A very thoughtful post Schneider in Performance-based assessment in action. My response I particularly appreciate the argumentative essay rubric and accompanied the blog post.  My response...





Given the academic prompt above, and thinking in terms of putting this into practice...




1. Are performance tasks standardized throughout the institution?  What flexibility is there for educators to adapt (or differentiate) academic prompts (performance tasks) around a particular classroom context?
2. How do educators share in the planning, implementation, and assessment of their respective outcomes from these performance tasks?  In other words, are performance tasks (planning, implementation, and assessment) shared publicly?
3. How do performance tasks connect learners with the global community?  How do learners interact with individuals outside their own classroom?
4. How are standards (e.g., CCSS) referenced throughout the process of planning, implementing, and assessing a performance task?
5. How are professional development opportunities embedded within the process of planning, implementing, and assessing performance tasks?
6. How much time is spent on a performance task, both inside and outside the classroom?  What other resources are required?




Certainly, the initial blog post above sets the stage for putting performance-based assessment into action, but the actual outcomes of a given performance task will depend in large part how one answers the six questions above.


Five myths when learning an additional language

There are seven days in a week, and someday is not one of them...

http://youtu.be/0x2_kWRB8-A

Five barriers people feel they face (myths) when learning an additional language...

  1. No language gene or talent

  2. Too old to learn a second language

  3. Can't travel to the country right now

  4. Bad memory for all that vocabulary

  5. Frustrate native speakers

Essential Question Discourse Analysis

I was reading Alber's (2013) 5 Powerful Questions Teachers Can Ask Students and I began to think about how discourse emerges behind asking essential questions in the classroom.  Alber (2013) suggests that teachers ask the following types of questions:

  1. What do you think?

  2. Why do you think that?

  3. How do you know this?

  4. Can you tell me more?

  5. What questions do you still have?


When applying the Socratic Method, pace becomes important.  The suggested question types provided above avoid the easier "yes/no" questions that allow for lower order thinking, but what happens when students 1) are not used to answering essential questions (or used to the teacher simply giving the answer), 2) lack the content knowledge to adequately address such questions, or 3) lack the language skills - in the case of English language learners - to provide an adequate response.  Alber (2013) suggests a type of think, pair, share activity to allow for deeper group discussions, but this too affects the overall pace of classroom discourse and an interruption of the Socratic Method.

(Language) Educator Challenge: Record yourself during a class and analyze the way you form questions and how your students reply.  For example, how do you lead up to any one of the five questions above?  Are there lower-thinking questions that occur first, or are you able to jump right to these questions in order to generate thought-provoking discussion?  Conduct a discourse analysis to see how you present questions in a way that maintains good pacing and thought-provoking discussions around big ideas.

 

Essential Question Discourse Analysis

I was reading Alber's (2013) 5 Powerful Questions Teachers Can Ask Students and I began to think about how discourse emerges behind asking essential questions in the classroom.  Alber (2013) suggests that teachers ask the following types of questions:

  1. What do you think?

  2. Why do you think that?

  3. How do you know this?

  4. Can you tell me more?

  5. What questions do you still have?


When applying the Socratic Method, pace becomes important.  The suggested question types provided above avoid the easier "yes/no" questions that allow for lower order thinking, but what happens when students 1) are not used to answering essential questions (or used to the teacher simply giving the answer), 2) lack the content knowledge to adequately address such questions, or 3) lack the language skills - in the case of English language learners - to provide an adequate response.  Alber (2013) suggests a type of think, pair, share activity to allow for deeper group discussions, but this too affects the overall pace of classroom discourse and an interruption of the Socratic Method.

(Language) Educator Challenge: Record yourself during a class and analyze the way you form questions and how your students reply.  For example, how do you lead up to any one of the five questions above?  Are there lower-thinking questions that occur first, or are you able to jump right to these questions in order to generate thought-provoking discussion?  Conduct a discourse analysis to see how you present questions in a way that maintains good pacing and thought-provoking discussions around big ideas.

 

Framework for English Language Proficiency Development Standards

Developed by the Council of Chief State School Officers and the English Language Proficiency

Development Framework Committee in collaboration with the Council of Great City Schools, the

Understanding Language Initiative at Stanford University, and World-Class Instructional Design

and Assessment, with funding support from the Carnegie Corporation of New York (Source)

ELPD Framework Booklet-Final

Formative Assessment that Transforms the Individual

Points to consider from yesterday's Google+ post...



What's the argument for capturing formative assessment?



Summative assessment attempts to measure learning and formative assessment improves learning.  Improving how teachers learn should be a larger part of the evaluative (judgmental) process.  Too often, a teacher's evaluation is limited to a numeric representation used to form judgments on the knowledge, skill, and disposition of an individual.  Instead, teacher evaluations should rely on iterative and reciprocal interactions (among all stakeholders) based on qualitative and quantitative data where assessment and instruction and support cycle through more fluidly.  We need more than a single snapshot of evidence of one's competence, we need an entire photo album.    



If we look at USA educational history, which has employed mostly summative evaluation systems, we see great success in the past. 



Is the USA successful because they relied on mostly summative evaluation systems?  Can we claim great success when there has been virtually no increase in college entrance exam scores yet great advancement in the way we communicate using recent technologies developed over the last 40 years?



 


Business and other institutions seek entrants with "21-century skills" such as critical thinking, collaboration, creativity, communication, etc.?



I question whether relying on summative assessments (as an end measurement) properly allows one to make the accurate inferences on the level of "21-century skills" a learner has.



What proof do we have that formative-heavy evaluation is the right direction.  Pedagogy isn't an exact field of study.  Are we sure?  Are we taking educated guesses based on environmental situations and observations?



Since pedagogy and learning are not exact fields of study, we're better off referring to the current literature on formative assessment and the notion that individuals learn in different ways...it's out there.  Formative assessment does not rely on direct cause-and-effect relationships to teaching and learning, but rather provides the means for greater interaction around the learning process.  Sometimes it looks not only at the act of learning (via a myriad of evidence), but the act of becoming as well.  If properly aligned, formative assessment can impact standardized reporting (Transformative assessment).    



What is the right mix? How do we back it up as sound?



Asking what's the right mix is like asking what's the best way to do it.  It's not about practices or programs, it's about people.  There is no magic formula.  It's about having honest, open, and ongoing discussions where the learner links content, context, and conduits together in a meaningful and relevant way.



Gates is one of the biggest innovators of all time ... could his foundation be wrong? 



I'm questioning the relevance of the report.  And I would question anyone who claimed they knew how others learn best.  Research-based learning principals are fine as long as they are being discussed within a local context. 



...isn't most summative evaluation really just an end result or snapshot of mostly an ongoing formative process which leads to the numerical grading process?



No.  Evaluating one's learning requires the collection of qualitative, quantitative, and relational data in parallel (not serial) that provides a cycle of planning, implementing, and reflecting on and in practice.  This needs to be done at the classroom, school, school-district, state, and federal levels simultaneously where reports like those generated by the Gates Foundation become just one small piece of the puzzle.



For example, at ITESM, teachers are heavily evaluated on the numerical advancement of students between their beginning TOEFL scores and their ending TOEFL scores for the period. While this indeed is summative evaluation .... it really doesn't explicitly highlight the formative assessment that took place between the periods .. it isn't captured by the system at all like summative evaluation is (numbers are number after all, nice and concrete). But, that doesn't mean that formative assessment and learning hasn't happened. It has. Most formative record-keeping is done informally by the teacher, some of which really can't be documented.



Why not evaluate English language teachers by collecting a mass of evidence: TOEFL scores, eportfolios, OER projects, openly shared experiences in online communities, workshops, conferences, most significant change stories, etc.  We should make formative assessment explicit and then judge it along with summative forms of assessment as well.  I would personally place more emphasis on formative than summative assessments, making formative assessment more formal.  Ideally, it's entirely about formative assessment - if quantitative reporting were done in a more timely fashion, it too could be treated as formal assessment (i.e., dynamic assessment).  When quantitative reports are issued months, perhaps years after the actual event, one has to question it's relevancy.  It's like trying to get over a cold by performing an autopsy.  I'd prefer to take care of one's health by taking preventative measures and take one's temperature periodically. 



So even though the Gates Foundation is focusing on the numerical end point, isn't it really an indication of how well the teachers formative evaluation skills are in getting to that point?



No.  It could mean that formative assessment is working extremely well but is not being reflected in the report, or it could mean that formative assessment has no impact on the change process.  Thinking in terms of order, produce the quantitative information first, then immediately follow it up with a lot of formative assessment.  Then continue formative assessments with period quantitative reporting (take the temperature).  Formative assessment needs to be based on past experience (problems) and should be included in the evaluation process.  What's important is that there is not a lot of lag time between the quantitative and qualitative reporting - should be like days or weeks instead of months or years. 



We spend a lot of teacher training time on formative stuff. Either a teacher gets it or does not get it (this has been my observation experience as a teacher trainer). And, I think that mostly gets reflected in the end result ... which should also include some formative project work too. So even though something looks and smells like summative, my feeling is that, to a great extent, the summative is a leading indicator of a teachers formative learning and evaluation skills.



My experience has been different.  I see that teachers understand things by degree - I hardly ever classified it as teachers get it or they don't.  Same goes for students, come to think about it.  Teachers come from different perspectives, have different experiences applying their knowledge, have different levels of empathy or feelings about their craft, etc.  So the learning process is taking them from where they are currently to a new "place".   To measure the degree of understanding teachers have, different types of timely data are required: quantitative, qualitative, and relational.  I would argue that the longer it takes to produce a quantitative report and then act on it, the less formative it becomes.  Another risk quantitative reports have in terms of their formativeness is that the results can be too general.  I have a hard time accepting the report from the Gate Foundation as being formative.  But certainly a school-generated quantitative report has the potential to be formative.



I kind of want to believe that Dan Pink got it RIGHT. Which teachers do you think will do better with students nowadays? Those with left-brained or right-brained predominances? Perhaps this is key to the end measurable result? How does a teacher's cognitive processes lend to productive formative assessment and ultimately summative assessment?



I don't view people as being left or right brained...they're all "full-brained" to mesmiley.  I don't consider learning styles or other personal attributes in isolation.  I watch to see how teachers adapt to their environment by trying to  facilitate particular networked topologies that link information, context, delivery.  Adapting to one's environment not only is cognitive, but physical/material, and affective.  Reporting procedures needs to add value to the process of adaptation.  

Assessment that Transforms the Learner

How Do We Include Students In The Formative Assessment Process? - Classroom Q&A With Larry Ferlazzo - Education Week Teacher.


I was asked to elaborate on a tweet related to how we can include students in the formative assessment process.  But since I had to sign into Educational Week (EW), I opted to post here instead - I don't enter into EW enough to remember my password.


We include students in the formative assessment process by listening to them.  We listen by taking part in informal discussions, instructional conversations, Socratic Method, academic prompts, performance tasks, quizzes and exams, eportfolios, etc.  We listen to them not relying only on a limited number of forms of assessment, but rather through the collection of many different types of evidence that allow us to make better inferences on student achievement.


In one sense, instruction and assessment are separate in that formative assessment should make instruction better (Popham, 2008).  Formative assessment should inform and transform.  How might that happen?  First, after having collected enough evidence from the students, teachers should determine when instructional adjustments are necessary.  This might include spending more time on a particular unit, given students more time to complete a task, or reviewing concepts that students seems to still be missing.  Second, teachers oftentimes need to guide students in making adjustments to learning tactics.  This might mean advising them to better organize their study habits or providing them reading strategies that help them read more effectively and efficiently.  Formative assessment through instructional and learning tactical adjustments then emerge at the classroom level and at the school level; that is, it's oftentimes a top-down and bottom-up change initiative.


In other sense, the line between instruction and assessment becomes harder to define.  As we are constantly assessing learners for understanding, slight instructional or tactical adjustments might be less noticeable or less defined.  An example might be simply checking homework only to find out that students might need a quick review before continuing on to the next unit.  Or perhaps a student was unaware of a change in personal learning tactics that transformed over three or four months.   


Besides relying on the different forms of assessment, we listen to our students by simply talking to them; asking them, perhaps on a weekly basis, the following:



What did you find easy this week?


What was difficult for your this week?


What did you enjoy doing?


What did you not enjoy doing?


How would you prefer to interact this?


What type of additional help do you need?



If these questions are asked frequently enough, there will be time to make minor adjustments throughout the course so that learners receive the support they need.