Part III: Achievement, Assessment, and Evaluating Learning

Comparative School Achievement
National Case Study Report
International Study of School Achievement
Reflections
The Two Faces of Examinations
Tradeoffs in Examination Policies: An International Comparative Perspective
Secondary School Examinations: International Perspectives on Policies and Practice
An International Perspective on National Standards
A Comparative Assessment of Assessment
An International Comparison of End-of-Secondary School Examinations

          

          Source: Harold J. Noah and Max A. Eckstein,
          "International Study of School Achievement," in
          Notes on Education 4 (1974):  6-9.
          

INTERNATIONAL STUDY OF SCHOOL ACHIEVEMENT

Policy makers in education say they want to make schools "more effective." What knowledge will help them achieve that goal? They need answers to such questions as: How do children's minds work so that they learn, can perform, and continue to learn? (Psychology) How do teachers teach successfully? (Pedagogy) How do home environments facilitate or hinder children's learning and teachers' teaching? (Sociology, anthropology) How do societal factors, school system patterns and internal school arrangements interact with the previous questions? (History, economics, political science, administrative science, systems analysis, etc.)

If we knew all these things, the ways they are patterned and how they interact, we would be in a position to make firmer assertions about pedagogical matters and the probable outcomes of particular educational decisions. Policy makers informed by this knowledge could then perhaps concentrate on their proper function, which is to make moral decisions. (Philosophy)

The research work conducted by the IEA (the International Association for the Evaluation of Educational Achievement) over the past fifteen years represents a massive attempt to answer a few of these questions. Twenty-one countries participated in what was, without doubt, the largest international effort ever made to collect and analyze primary data on schooling and its correlates.

Nine volumes (eleven, if we count the two volumes in 1967 on the mathematics study, or twelve, if we go back to 1962 and the pilot study report); 200 million pieces of information; $5,000,000 of international costs and perhaps half as much again of local, individual country costs; hundreds of thousands of manhours of labor; two international conferences (one just held in November 1973 at Harvard University) all are devoted to trying to find out what principal factors, if any, make a difference to the school achievement and attitudes of young people from different class backgrounds, in different types of schools, in different countries.

Three recent volumes detail the study of education and school achievement in science, literature, and reading comprehension. Six further volumes will appear in the next year or so. Three will deal with teaching, learning and achievement in English and French as foreign languages, and in civic knowledge; a fourth will contain the socioeducational profiles of the several countries; a fifth will be a technical report; and the last will attempt a summary of the entire project.

The conclusions so far are not very startling. No great cornerstone of our understanding about the factors influencing school achievement is either erected or overthrown. The home background of children, as measured by father's education and occupation, mother's education, and number of books in the home, stands out as an internationally strong variable in "explaining" variations in school achievement. Few of the directly school related variables, such as sex of teacher, teachers' experience or training, size of school, quantity of homework, and type of curriculum come through as important in all the nations tested, though here and there, one or another school based variable shows up with a statistically significant coefficient in the regressions. The one consistently important school-related variable is "opportunity to learn": that is, when the curriculum has provided for students to learn a given area of knowledge, when teachers have actually taught it, and when the internationally devised tests included it among the questions asked, students did better. Again, not an earthshaking conclusion.

It could not be established that differences among school related arrangements bear any direct, simple, universal relationship to differences of school achievement especially when home and general environmental factors are held constant. Note that this does not mean that "school makes no difference." To conclude that, IEA would have had to compare test scores of children without any schooling with the scores of children who had had some standard amount. That was not done. What the IEA results do point to is the probability that within countries (and probably even across countries) homes differ from each other vastly more than do schools. Consequently, although most children get a lot out of school, the big variations in achievement observed around that basic level are attributable more to home than to school differences.

The IEA investigation provided three levels of analysis: between-children, between-schools, and between-nations. Hence, a good beginning has been made to answering such difficult questions as: Do the ways in which different countries sort children into different types of schools matter at all? If so, in what particular respects?. Thus, the sorting of children by social class of the home appears dramatically in·the results of between-schools analysis in Scotland and England, in·contrast with Sweden, where·individual school populations are much more heterogeneous. Adding school related factors (after the home factors have been put in) to the equations for Scotland and England does not add much explanatory power to the model; adding them in Sweden made a great deal of difference. Hence, IEA work begins to allow us to specify the school and home contexts in which some factors are important for achievement, and others less so.The IEA results show, too, the degree to which the term "less developed" must be applied to school systems, and not just to economic systems. Four less developed countries (Chile, India, Iran, and Thailand) participated. The gap between mean school achievement levels in the developed and less developed world is very wide. Thus, at the Population I level (10-year olds), the grand mean score for science in the developed countries was twice that for the less developed; in reading comprehension, similarly. These comparative measures of the combined effect of home, school, and environmental factors on school achievement are the first ones of their kind. They underline the enormous tasks facing the poorer nations of the world in raising the level of knowledge and basic educational skills of their young people. It is precisely because of their relatively high levels of school competence that the richer nations of the world can begin to think in terms of some "de-schooling." For the poorer nations, who have hardly reached first base in the organization and operation of their school systems, the notion of "de-schooling" is a cruel joke.

Among other significant questions treated in the present corpus of IEA work are: the differences in achievement that are linked to sex, from subject area to subject area (especially in science), and from country to country; the extent to which comprehensive secondary schooling lowers mean national scores, but leaves virtually unaffected the scores of the top achievers in each country; the relatively high importance of certain school and home factors that are composited under the label, "learning conditions," for explaining achievement in science, and in French and English as foreign languages; and their relatively low importance for reading comprehension and civic knowledge; and the first attempts to measure cross-national differences in students' attitudes to literature and their modes of evaluating the merits of a piece of writing. This list, however, barely scratches the surface of the rich store of data and inferences to be found in IEA publications.

Where does the study of the factors affecting school achievement go from here? Most obviously, subsequent work, designed to replicate the individual subject studies after an interval of five or ten years, can use that opportunity to improve the validity, reliability, and scaling of important variables. In addition, close inspection of the data already collected in each country will identify schools that lie far off the regression plane, so-called "outlier" schools, that appear to produce extraordinarily high (or low) scores in association with extraordinarily unfavorable (or favorable) circumstances. Such schools should become the subjects of detailed on-site studies, to isolate the proximate causes of their "non-predicted" behavior.

Aggregative social science models that rely on data amassed from many thousands of individuals in a score of countries are naturally likely to contain large amounts of error. After every reasonable effort is made to fill in gaps, to check responses, to control coding, keypunching, and data storage, mistakes and gaps are still inevitable. But this first and most obvious difficulty with this kind of work is, in the end, not the most serious. Given time, money, patience, and a few more tries at what is a brand new area in education and social science studies, such problems will be controlled.

More intractable is the problem of adequately modeling the processes at work in the homes, schools, and classrooms of the world. The model used in the present IEA studies is an avowedly simplistic one, designed to facilitate the use of linear multiple regression as the primary analytic tool. Together with linear scaling of variables, this assumes that, for each variable entering the regression, at no matter what quantitative level, its contribution to achievement outcomes will be a constant proportion of that level. Thus, we assume that for a given population there will be a single value of the regression coefficient that can be applied equally to, say, a low level of "home background" factors and to a high level. This is the average value of the association between home background and school achievement. Whether that average value reflects anything of practical importance in the real world depends on close scrutiny and testing of the underlying data.

Because, in general, IEA regression analyses have taken the entire national sample as the universe to be explained, policy makers may find the results much less useful than they might wish. The results of linear multiple regression analyses done on aggregated national samples conceal a great deal of the information policy makers need. The reason for this becomes clear if one reflects on the nature of the information that a multiple regression coefficient conveys. It tells the reader the average value of the strength of the association between the dependent variable and a particular independent variable, all other independent variables being held constant at their average values. Such an explanatory approach is able to clarify the connections between achievement and other factors, and to assign relative weights to the importance of one factor compared with others, on average, over the entire nation.

Policy makers, however, usually require something more detailed than such average indications. They need to know the effect upon achievement of varying a particular item under rather specific circumstances. Policy considerations are directed not at influencing achievement levels in "average," national settings, but typically they deal with the problems of achievement in, say, rural vis-à-vis urban settings; of girls vis-à-vis boys; and of students in poor neighborhoods as distinct from wealthier neighborhoods What provides the largest increments to achievement for low achievers? For average achievers? For high achievers? Thus, the policy makers' questions require analyses that do not hold the values of the other independent variables constant at their average national level, but partition the national sample, in such a way that the associations between a dependent variable (say, achievement) and a particular independent variable (say, current expenditure per student) can be investigated separately for specified groups, e.g., poor children and rich children; urban center children and rural children; poor children and rural children; and so forth.

Some critics may wish to deny totally the validity of the IEA approach. They will point to the essentially small scale nature of the educational process -- teachers and students interacting in a given classroom at a moment of time. They will reject an analytic technique that ignores all these individual interactions and studies instead the circumstances around the teacher-student-home interaction. It appears to us that an either/or choice is not necessary. Rather, our stance should be to encourage diverse, and essentially complementary, modes of analysis. Because education is a small scale, individually based process, techniques of micro-observation, analysis of small group behavior, and observation of classroom interaction and culture are vital. But we know, too, that schooling is a mass enterprise and, as such, there is value in analysing its correlates and outcomes using techniques of mass data collection and analysis. Both techniques have their place and their role. As we develop each of them we should approach a better understanding of schools in their micro- and macro- aspects.

NOTES

IEA Publications

Arthur W. Foshay et al., Educational Achievements of Thirteen- year-olds in Twelve Countries (Hamburg: UNESCO Institute for Education, 1962)

Torsten Husén (ed.), International Study of Achievement in Mathematics: A Comparison of Twelve Countries, 2 vols. (New York: John Wiley and Sons, 1967)

L.C. Comber. and John P. Keeves, Science Education in Nineteen Countries (New York; John Wiley and Sons, 1973)

Alan C. Purves, Literature Education in Ten Countries (New York: John Wiley and Sons, 1973)

Robert L. Thorndike, Reading Comprehension Education in Fifteen Countries (New York: John Wiley and Sons, 1973)

In press:

E. Glyn Lewis, English as a Foreign Language in Ten Countries

John B. Carroll, French as a Foreign Language in Seven Countries

R.F. Farnen, A.N. Oppenheim, and Judith V. Torney, Civic Education in Ten Countries

A. Harry Passow, Harold J. Noah, and Max A. Eckstein, The National Case Study: An Empirical Comparative Study of Twenty-One Educational Systems

G.F. Peaker, An Empirical Study of Education in Twenty-One Countries: A Technical Report

D.A. Walker, The IEA Six-Subject Survey