Guest post by Jim Ryan
Thanks to Andy for inviting me to guest blog. I haven’t blogged much at all, so I apologize in advance if I’m lousy at it. I’m a law professor at the University of Virginia, and I write and teach about law and education. I recently finished a book, which will be published this week by Oxford University Press, which I may talk about later this week. For the first couple of days, though, I want to raise some questions that have puzzled me. I’m hoping readers will have answers.
A front page story in the NYT on July 27 described the recent findings of some education economists regarding the impact of good kindergarten teachers on their students over the long haul. The headline of the article says it all: “The Case for $320,000 Kindergarten Teachers.” The researchers estimated that this was the present value of the additional money a full class of kindergarteners taught by a standout teacher would eventually earn over a class taught by a less talented teacher.
The findings have not been peer reviewed, and they may not hold up. But that’s not what’s interesting to me. What I’m wondering is why more social scientists don’t make an effort to translate their findings into points that resonate with other, non-expert academics (think, say, law professors), policymakers, or the public? The economists who studied the value of kindergarten teachers seem to be following in the footsteps of preschool researchers, who, brilliantly I think, have tried to quantify how much return governments can expect from “investing” in preschool. See, for example, this RAND study about expanding preschool in California.
Yet anyone who regularly reads articles by social scientists would see most findings reported in somewhat arcane and relatively inaccessible terms, like standard deviations or percentile gains over the median, which are difficult for the untutored (including yours truly) to translate into something more meaningful. You know that bigger is better, so a .06 effect is better than a .04 effect, but you (or I, at least) have no real sense of what a .06 effect means in the real world. In another context, I suppose phrases like statistically significant or robust to multiple variations might be evocative, but in these studies, they leave me a little cold. I get writing for other academics, not pandering, maintaining professional standards, being precise, etc., and I recognize that not all findings can be easily translated to plainer terms. But I bet a lot more could.
Ultimately, aren’t social scientists who write about education trying to influence public policy? If so, what would be wrong with translating the findings into terms anyone could understand? Instead of talking (just) about a percentile gain over the current median test score, for example, why not talk about gains in terms of months or years of school work? (And, while we’re at it, why not try more often to compare the costs and benefits of different interventions?) Or is a front page story in the NYT a bad thing for the academic credibility of social scientists?
18 Replies to “Show Me The Money”
In addition to the reasons you list, the reason for the separation between people who conduct methodologically rigorous research and present it as such without translation and people who translate that into policy could be seen as either: (1) specialization for comparative advantage, or (2) a desirable “Chinese Wall” (as you suggest re “credibility” in your last sentence).
You reminded me of your great article in Slate.
If I read you correctly, you basically agree with Mike Petrilli and Diane Ravitch that test results should be used for a type of Consumers Report as opposed to punishing.
I’d be curious also what you think social scientists will say if/when teachers are fired using test score growth models. Once social scientists testify regarding the validities and invalidities of using results from untested statisitcal models, how will those terminations hold up in court? How many judges will allow high school teachers to be fired, for instance, using alghorythms developed in the very different reality of elementary schools? When one year of test scores are used, the average teacher might have a one in three chance of being misidentified as ineffective.
It seems to me that a legal Battle of Verdun will occur if states try to act on their RttT promises. Wouldn’t that money be better spent in high quality preschool?
But getting back to Slate, just using test scores for ranking could cut through this educational civil war, so we could invest directly in helping kids. It shows a way out of this mess.
As a lawyer, I don’t really buy John’s argument that it will be prohibitively expensive to use test scores in dismissals because of potential litigation. Currently, any dismissal is expensive since the unions fight any and all dismissals already, so the threat of litigation is irrelevant. In fact, these cases should be cheaper to litigate, speaking as a defense lawyer, because it’s easier to strip out allegations of age, race and gender discrimination.
The use of test score gains in dismissals is interesting as a policy matter. As a legal matter, using test score gains (or any objective measure) should be cheaper and easier to defend in litigation than a subjective measure, particularly when a union agrees to this in collective bargaining. Then it’s simply a question of whether the math was applied – not whether the math, when applied, led to an unfair outcome.
The reason is pretty quite simple and sad.
The market drives research supply. And, in the academy – particularly in disciplines like Economics, Political Science, and Sociology – young scholars are rewarded with tenure for peer reviewed publications.
Unlike law journals which actually encourage policy relevant stuff (at least some) – think for example about the fact that so many journals in law are titled Law and Public Policy Journal… in the social sciences the top journals reward theory building more than policy implications.
Take my field of political science. I’ve had reviewers tear some of my submissions to shred because they are devoid of theory… or they don’t “build on pre-existing theories about political behavior.” Despite producing methodologically rigorous research that has policy implications and can be articulated in non technical jargon the ruling class in the academy shun this type of work.
Thinks of Freakonomics – Levitt is hated in Econ – the people who explain things in practical terms are thought of as weak scholars – popularizers who have sold out. It’s a shame, but until the academy begins giving tenure and raises for the “public intellectual” aspects of being a professor (e.g. conducting studies that are policy relevant and publishing them in venues outside of arcane journals that are read by policy-makers) then what you observe is likely to continue unabated.
You are the lawyer and I’m just a legal historian but I want to push you on this.
You write “unions fight any and all dismissals already.” What is the basis of that? Maybe the issue comes down to the word “fight,” because I’m not aware of many unions paying the costs to fight many dismissals.
In my experience, it doesn’t require a fight to stop many/most precedings because the dismissals were done so poorly.
You also say, “it’s easier to strip out allegations of age, race and gender discrimination.” In my experience, most cases die because systems are afraid to address anything that approaches those factors. The union has nothing to do with it; we’re just the scapegoat.
Besides, I suspect the prime goal of these systems is to get rid of Baby Boomers to create a more malleable teaching force. I’m expecting a wave of age discrimination issues.
I suspect the use of data for terminations – if or when its tried – will bring more egregious cases because principals and the bosses have no clue about the social science. I doubt they will even attempt to ask whether the growth targets are valid in the cases they want to pursue. In a system like Rhee’s, they’ll just hear a simple message “go git em” and they’ll overreach.
But this is my legal question. How can you hold me accountable for not meeting growth targets, set by an unproven statistical methodology, without showing that it reflects on what a valid growth target would be for my classroom? The idea that VAMs developed and tested in elementary school are valid for high school is preposterous. As an officer of the court, I’d think you would be offended by using statistical models that are more invalid with poorer students, older students, and students that were selectively assigned to high schools where all of those factors are in place.
Add up the total of all experiments with high schools classes and what would your entire evidence base be – negligible or nonexistant?
And this is not a legal argument, but it is the real reason why test scores are so dangerous. Let’s just say VAMs are 95% valid for elementary schools, even though i find that hard to believe and I’m unaware of anyone who has gone on the record with an estimate like that. Then the growth targets produce invalid results for 10%, 15%, 25%, 33% or whatever level for inner city high schools. That gets to George Soros’ question. If you have several bottles of water and only one is poisoned, they are all still useless. If you have a 10 to 15% chance PER YEAR of having your career damaged or destroyed through no fault of your own, who would teach in the inner city.
And getting back to the law, when it comes to firing high school teachers in our segregated system doesn’t it matter whether the models are 10% or 25% invalid? How many evaluators will be capable of making that distinction?
And that always gets me back to the aphorism that any grand jury can indict a ham sandwich. The same will apply to the RttT inspired evaluation models. And real world, being indicted as Ineffective will be no different than being convicted. Its like our old prosecutor loved to say, “everyone I sent to prison was guilty of the crime they were duly convicted of – or some other crime.”
Why would you think that mentality would not be common in any human system?
Regarding union leadership, they have a gun to their heads. They, and we the rank-in-file hope, this craze will settle down as districts find they can’t find replacement teachers. And that is probably what will happen. Remember what happened when NCLB mandated 100% profciency in 12 years? New bogus numbers will replace the current bogus numbers and management will find other ways to throw around it weight and to avoid rigorous evaluations.
I meant to say “And real world, in many situations, being indicted as Ineffective will be no different than being convicted.”
The use of test score gains in dismissals is interesting as a policy matter. As a legal matter, using test score gains (or any objective measure) should be cheaper and easier to defend in litigation than a subjective measure,
Ah…but the idea that test scores themselves are not a subjective measure of teacher performance is completely fallacious. Let’s look at my own circumstances. I teach physics at a large public high school. There are 5 other physics teachers at my school. One might think that test scores would be a legitimate means of comparing physics teachers at my school. One would be wrong.
First, physics is an 11th grade class. Typically students take biology in 9th grade, chemistry in 10th grade, physics in 11th grade and a science elective in 12th grade. When students walk into my class the last time they have had significant physics instruction was back in 8th grade middle school science.
Now let’s look at how the physics courses are distributed at my school.
Teacher #1: Our department head teaches 3 sections of AP physics and 3 sections of pre-AP physics to the best and the brightest 10%. These are the kids bound for top colleges. He has not had a kid fail a state standardized science test in probably 10 years. But the bulk of his students probably could have passed the state science exit exam as 8th graders.
Teacher #2: Another veteran teacher has 3 sections of pre-AP physics and 3 sections of astronomy (a 12th grade elective). Because 12th graders are not tested by the state, only her pre-AP students get tested and they are another select group.
Myself and Teacher #4. We teach general physics to middle of the road 11th grade students. None of our students are science nerds. Some are bright but not particularly interested in science (future English and music majors) and some are just completely unmotivated. Because I am bilingual and have gone to all the special ELL training, I end up with all the Hispanic ELL kids who are mostly recent immigrants from Mexico and Central America. We had a small handful of kids fail the state science test last year. 2 of my 3 kids who failed were recent Hispanic immigrants with language difficulties and questionable science background at their schools in El Salvador.
Teacher #5 teaches physics to what we call “targeted” kids. These are the ones who failed science in 9th or 10th grade or are special ed or otherwise learning disabled. She has class sizes of 15 instead of 24 or 25 like the rest of us. She’s often down to 10 students due to much more casual attendance of her students and the fact that they are often in DAEP or juvenile detention. She has to go twice as slow and reteach a lot of basic concepts like basic algebra before she can even get to physics. She has a lot more kids fail the state science test but a lot of the kids she has that fail also fail all the other state tests (English, math, etc.).
None of the 5 physics teachers at my school have remotely comparable student populations. So comparing us on the basis of test scores would be far from objective. At the same time, none of our students have had any physics since 8th grade so it’s rather difficult to gauge where they were starting from when they entered our class. The statewide science tests are a combination of biology, chemistry, and physics and so their passing rates has as much to do with how good their 9th grade bio and 10th grade chem teachers were than how much I taught them.
So how does one use test scores to compare the science teachers at my school? I defy you to come up with a fair and objective way to do it.
That said, it is fairly easy to identify those teachers who are below average or not performing. All a good administrator has to do is observe classroom management and stay on top of what each teacher is doing. It is pretty obvious to the experienced eye. You just can’t do it with test scores.
Now perhaps things are different at the elementary level. But I’m guessing that a majority of teachers are in situations similar to mine. Where it is extremely difficult or impossible to use standardized test scores as an objective measure of teacher performance.
Kent, these are excellent and well-reasoned, and ought to have been obvious to everyone WHO DOES NOT REFLEXIVELY IMAGINE CHILDREN AS PRODUCTS AND KNOWLEDGE AS A COMMODITY.
Moreover. Next year, your situation and mine (quite like yours) is almost certain to vary yet again. There will be different students, the class balances will be different, there may be a different sense of how to handle science within the department, conversations will have taken place that will have moved any number of valuables (it would be pedantic to list them all) and the entire situation will be substantially changed.
Thus, “value-added” teacher evaluation (measuring how a teacher “adds value” – think about the mind that thought up that expression!) is likewise revealed to be a crock of nonsense. If it were to be valid in any way, first, knowledge itself would have to be a stable commodity (and if you belief that, talk to an epidemiologist as quickly without delay), and second, there would have to be some way to reliably compare one year’s students with that of another. And there is not. That’s because they are living, breathing, and not-the-same-people. And this has important methodological implications from, say, a hospital taking statistics on staph infections or infant mortality.
These differences offend the accountability fetishists. Yes, some of them are sincerely interested in children, but they have a fatal hubris that leads them to (a) cooperate with the forces of late-stage capitalism (think turning to the mythic Dark Side) “for the sake of the children,” and, (b) vastly overrate their knowledge base by simply putting situations like Kent’s above on the Procrustean bed and saying, in effect, “we’ll make one size fit all for the good of all.”
In a word, they are naive. In two words, they are reductive fools.
So now we touch on the post above. As someone who writes educational research, we don’t put .04 in place of a .06 and trumpet the difference because anyone in the field knows that WE DON’T KNOW WHAT THAT DIFFERENCE MEANS.
Those people who claim they do know are (a) lying and (b) trying to sell you something.
Let me give an analogy from history as to where we are now in our field with respect to educational research.
You no doubt know that in October of 1917 there was an historic flu epidemic that killed, worldwide, perhaps 35 million people. In America this was a near apocalyptic event, but not only or even primarily because of the spike in the death rate. America was at the apex of its confidence in science, both on the left and on the right. It was presumed that science would solve all of life’s important problems, and solve them soon. (That was why the obscene futility of World War I was so slowly absorbed by America; it was asking us to believe in something that went against our basic self-understanding.)
As the flu death toll mounted, scientists looked under microscopes and proclaimed bacterium after bacterium to be the cause. There were vaccines prepared and administered. We’ve all seen, I’m sure, the pictures of whole cities with white surgical masks on.
And all of these things were useless, useless, because the flu was caused by a virus, and no one in 1917 had any idea what a virus was, and the electron microscope to see one was quite some time away.
You take my point, I hope. America is frightened about the waning of its onetime perceived superiority in education. This waning is almost certainly brought about by the sharp increase in social inequality – cf. for example, Wilkinson and Pickett *The Spirit Level* for lots of statistics that you can actually count on as supporting evidence.
However, America is addicted to a society that is separate and unequal, and is shocked, shocked, when its children turn out to be unequally prepared for the world.
It must be the teachers! Let’s get ’em!
And, mirabile dictu, this “getting” the teachers is doable, like the bacterial vaccine for the 1917 flu, and it dovetails nicely with the anti-union vector of late-stage capitalism and the general resentment of a working class that knows it has been betrayed, but has no other true information.
A hard rain’s gonna fall.
Which side are you on?
moved any number of variables (not valuables)
talk to an epidemiologist without delay
(not talk to an epidemiologist as quickly without delay)
I’m sorry, by twice now I could swear I’ve tried to type “epistemologist” and I’ve gotten epidemiologist. I think I need an epidemiologist. I may have the flu.
Sorry and solidarity.
These are interesting posts. Just to re-state, though, my point is that measuring teachers via test score gains isn’t going to create greater legal complexities. It may not make sense as a policy matter, but as a legal matter, nothing posted above indicates a legal hurdle since measuring people in “unfair” ways is not in itself a legal issue absent targeting a protected class (i.e. targeting by race/gender/age). Even if the data feels arguably subjective because of differences in class composition etc., that again is a policy matter rather than a legal one.
Tom, I think your post is a bit hyperbolic and over-wrought and I think that about a lot of the comments on education articles and blogs. There seems to be a massive uptick in the number of angry teachers posting comments, and unfortunately, the hyperbole tends to backfire, at least in my own reaction as a public school parent. I don’t think you will find very many non-teachers who think that there is a profit-driven conspiracy to “get” the teachers; most people think there is a policy dispute and, as a parent, it’s reasonable to both want a system that attracts great teachers yet also removes the ones who are lackluster (which, from my viewing as a consumer, is a minority but an inappropriately high number).
Kent and John’s posts were interesting policy posts. I would think that using gains would take away the question of which group of kids are in a class, though it sounds like you still have validity questions. I have two thoughts in response though: 1) It seems like many teachers posting in comments over time have said that they don’t think principals can make valid and competent assessments of their work. So it seems like test scores in evaluations are in part a response to teachers’ distrust of principal evaluations which are viewed as political. Honestly, you have to have SOME way of evaluating effectiveness, and many teachers seem to have issues with each and every proposed measure (other than “peer review” which to me makes no sense; it cannot be the case that only other teachers are capable of knowing what effective looks like). 2) To John’s point that the profession will be harmed if there is a chance of unfair termination: I don’t buy it. That’s pretty much every other job in the country. I am quite sure that 10% of the lawyers I have seen get the boot have a legitimate beef, and yet every year, hundreds more seek to work at the firm. There are no jobs in the country in which everyone fired was fairly fired. None. If our standard is that every single person fired must be legitimate and defensible and non-political, then we will never have a system of removing incompetent people. There will be errors in any system. But I seriously doubt your fears of total randomness, John. You are assuming a world in which 10% of teachers are fired each year, and the system is so atrocious that it is entirely random who gets picked. I don’t think any evidence supports that claim and I am quite sure that the public wouldn’t support those results.
For me, 10% would be a very reasonable error rate because it means that 90% of the teachers leaving are ones who shouldn’t be teaching kids (and when I think that there are 20 kids for every teacher, it means that for every 10 teachers removed, 180 kids were helped and 1 teacher was wronged; I like those numbers). Basically, n a system that removed 10% of teachers a year, if 10% of those teachers were unfairly fired, a teacher would have a 1% chance of being unfairly fired in a given year.
so, if the new evaluation systems removed the bottom 10% of teachers each year, then the question is how to make sure the system is not random but operates with, say, 90% integrity. That seems readily do-able, particularly in a world in which test scores are going to count less than half of an evaluation. I just don’t see this creating the Russian roulette situation that you see; and yet as a parent, I see student assignment each year as Russian roulette in which we all walk over, look at the list on the wall, saying to ourselves, “I hope my child doesn’t get Ms. X” because in most grade levels, there is a Ms. X who shouldn’t be teaching kids. And that is in a “good” school.
Anyway, I appreciate when people’s posts are thoughtful and not vitriolic or making assumptions about motives, so thanks to John and Kent for pushing the discussion.
Kent: Very well put. As a former teacher, I completely agree with you that it is not realistic to compare teachers to one another for several reasons, including the fact that they often teach different populations of students. Another issue I noticed as a middle and high school teacher was that the number of different subjects a teacher was assigned to teach varied from teacher to teacher. To build on your examples above, if one teacher taught 2 periods of physics, 2 periods of astronomy and 2 periods of chemisty, they would have only 1/3 the prep time to prepare for their physics classes, compared to a teacher who taught 6 periods of physics. I’ve taught a straight schedule of the same class and I’ve taught up to three different subjects at once. It’s more difficult to prepare and create high quality lesson plans for three classes at once than for one class at a time.
I know it’s a bit off topic, but did anyone else catch the NYT article yesterday about all the school reform charlatans coming out of the woodwork now that there are federal dollars up for grabs?
Depressing. And to think that that at some point my tax dollars will be paying for some of these folks to come tell me how to do my job.
Since you mentioned science, let’s throw in another wrinkle. In our state, middle school students are tested with “Science” CRTs that were designed a decade ago to be rudimentary and just test facts. Those scores typically have a pass rate of 70%+. Then those scores will go into a VAM for 9th grade Biology test score targets? Those tests measure scientific thinking and have a pass rate in the mid-20%.
The same applies to Algebra teachers whose growth targets will be determined by “Math” scores. How do you make statistical adjustments for such stark differences?
Policy-wise, the alternative would be to redo all of the previous tests, but do you think a principal who will be fired in three years if he doesn’t meet growth targets – or find others to blame – will wait till a fair system is devised?
I’d be curious if the attorney-commenters think that would withstand legal scrutiny.
And here’s another reason why principals alone can’t determine if the models are fair. They may not have the power to enforce discipline but they must take the blame. So, the students on electronic bracelets (as opposed to the run-of-the-mill felons we in neighborhood schools have in every class) in our system are all dumped in the Hispanic high school. And this year because the neighboring school is subject to turnaround, they are dumping hundreds of their toughest schools on that school which is the 2nd lowest in the state.
Similarly, my school which is the lowest in the state, and our neighboring school which is the 3rd lowest in the state, gets more of the SED, homeless, conduct disorders, and kids who should be in labs and alternative school. Then the McKinney Act dumps all of the homeless kids on us, and the transformation of the 3rd lowest school probably will dump their discipline problems on us.
Peer review is not teachers alone evaluating. Under the Grand Bargain, it would be committees of teachers and administrators. And they would be trained by both organizations. That prevents the systematic bias where an administration choreographs a system to design the monoculture of teaching styles it wants. After all, you can’t explain why we would be investing so much in these systems without considering the obvious motive – Many, many, many reformers want to drive Baby Boomers out of the profession. They claim we have “low expectations” and don’t use Smartboards and their other toys.
And counter-intuitively, data-driven evaluations should INCREASE subjectiveity. There is absolutely no way that these toughest schools will meet numerical targets. So, the only way for a principal to survive is to surround himself or herself with loyal followers. Sometimes that will breed teamwork. More often, it will create more yes men.
I think a combination of teachers and administrators is best for a lot of reasons, but diversity is the biggest reason of all.
Yes, unfairly firing 10% of lawyers or cutting 10% of NFL players does not drive talent out of the profession. Pay us like lawyers for a few decades and ease up on the extreme disrespect continually dumped on teachers, and the same would apply to us.
But even before self-respecting teachers leave, this test-driven evaluation system will drive high-quality principals out of inner city schools. In the tough ones, only adrenalin junkies or math illiterates who can’t calculate the odds against them will stay in neighborhood secondary schools.
But I’d be worried if courts allowed property rights of individual teachers to be taken without showing that there is a certain level of confidence that the results from those statistical models say some accurate about that specific teacher. Now, we don’t even have anyone who will go on record about that accuracy. Are we assuming that no amount of inaccuracy would be too much? Otherwise, shouldn’t that accuracy be considered? When I hear the assumption that courts will defer greatly regarding flawed models, without even considering how flawed they are, I wonder if this is more evidence that teachers are being reduced to second class citizens.
I’m wondering about the numbers that are tossed around regarding how many teachers should/would be dismissed (using whatever system). Edconsumer says every kind of job has unfair dismissals, but how many are fired for cause in given professions–engineers, nurses, lawyers, etc.? 1%? 3%? I’m guessing it’s far less than 10%, but does anyone know?
For all of our moaning about the turnover rate in education it is actually quite low when compared with other professional occupations. I’ll look for those numbers and post them when I can.
John, you’re really talking about a larger societal problem as is always the case with education. I teach in suburban Waco Texas. There are about 12 school districts covering this metro area of about 225,000. That’s right. At least 12 different school districts cover parts of the greater Waco metro area. And there are still more far-flung exurban districts where people live and still commute to Waco.
What happens here in Texas is that we just throw away land when it gets used up just like we throw away everything else. One can fly over this area and pick out school district boundaries from the air based on development patterns. All the new subdivisions get built in the 3 or 4 most prestigious districts despite plenty of beautiful land closer in that lies in poorer districts. People literally throw away perfectly good houses and land in the inner city when they move out to the suburbs. Don’t believe me? You can go to the MLS and find properties with houses on them around here for less than the price of a new Toyota Corolla. And it isn’t just housing. Wal-Mart and Target and McDonalds and all the commercial chains repeatedly abandon neighborhoods around here and leapfrog out to the newest strip malls and subdivisions where the consumers with cash are moving.
Back in the inner city the students and families who are left are not stupid. They see society abandoning them. Call it white flight or the suburbanization of America. Or the desire for granite countertops and master-planned communities with dictatorial HOAs. It amounts to the same thing.
We talk about inner city school reform, forcing the “best” teachers back into the inner city and so on. But it’s all just pissing in the wind in the face of these larger societal forces. If we don’t want to leave these kids behind then we as a society need to stop leaving them behind. Waco is a small city yet the center is riddled with abandoned strip malls, big box stores, and crumbling residential districts while around the edges in the preferred school districts we can’t build new strip malls and chain stores fast enough. Even in the recession.
When we start forcing Wal-Mart and McDonalds and real estate developers and soccer moms in Escalades to move back into the neighborhoods that they abandoned, then I’ll know we are finally serious about leaving no child left behind.
Until then it’s easier to use standardized tests to blame the teachers. Until the best ones get fed up and leave and then what?
I believe you. In Oklahoma City, we have 21 school systems! Same story. And those VAMs are only reliable when controled for choice. How do you control for the choice gone wild that you describe. Another Texan described the situation as The Big Sort. Because of the Big Sort, many reformers don’t know how far their theories are removed from actual reality.
By the way, my daughter just left a 100% poverty school in OKC and is considering several 85% poverty schools in the Dallas Metro. Of course, I have mixed feelings.
Which, of course, gets back to The Big Sort. She was a double asset in her last majority Hispanic school because she’s Black and she gets along with everyone. But now with gangs and the prison industrial complex, elementary kids come to school with such extreme racial animosity. She was constantly referreeing one racial conflict after another all through the school. Her brother came back from a second tour in Afganistan and saw the bullet holes and exclaimed, “My sister teaches in Afgansistan!”