SummariesIngrid M.E. Munck, 1997: Effect studies of national and international evaluations of schooling: New approaches for qualified analyses at all levels/Effektstudier vid utvärdering av skolan/. Pedagogisk Forskning i Sverige, Vol 2, No 1, Pp 1-24. Stockholm. ISSN 1401-6788.
This article concerns methodological advancements within statistics andpsychometrics which have led or can lead to considerable increases in theyield from quantitative national and international evaluation studies. Thenew approaches are presented in non-technical terms and illustrated withexperiences and content from an IEA-study (IEA - The InternationalAssociation for Educational Achievement) and results from the MALI project(a Swedish research initiative: "Multivariate Analysis of Literacy").
Ever since the publishing of the first IEA study in mathematics in themid-60íes, there has been an embarrassing gap between what can be termed"the Knowledge Olympics"; i.e. the ranking of countries according tomeasures (with concomitantly broad media coverage); and the researchersambition to understand the complex interplay between different factors andtheir roles in relation to the productivity and effectivity of schooling.The main goal of these studies was from the very beginning to conduct anelaborated explanatory analysis of the outcome of education/schooling inorder to point out important factors at all levelshome/class/school/region/nation. What has then been especially difficult tohandle; at least with the analytic tools available, has been the separatingout of the effects of different efforts at different levels in a properway.
For example, in order to fully understand the effects of cutting resourcesat the school level, the implications at the class and student levels haveto be traced and researched simultaneously in the hierarchical system,where a bottom-up strategy in effect studies for the collection andanalysis of school data is called upon both for substantive andmethodological reasons. Students and teachers carry the answers about howdifferent initiatives, investments and or cut-backs have influenced themin different ways and help form the results of school work. The newapproaches have the potential to integrate macro and micro data into aqualified analysis at different levels.
A simple model for effect analysis is developed for the relationshipbetween effort and result, bringing in the so called confounding variablesof initial status and background, and the contextual conditions (see figure1). Referring to Tyler (1984, p. 29), this model is differentiated into athree level model which reflects the social and educational context towhich the student belongs; the home, the area where he/she lives and spendshis/her leisure time, the classroom and the school. That is, it presentsindividuals and groups in a hierarchical/societal system. This is necessarybecause the things which we study are often complex phenomena, such as waysof working or educational programmes which are shaped by each teacher inhis/her professional role. The model must allow for this but should alsotake into account that the school and teacher's efforts will be modifieddepending on the students background and status and that all of this istaking place as an interplay between student and teacher in a class and ina given context. Figure 2 presents the multilevel model illustrated withIEA-variables.
Many prominent educational scholars have reviewed the IEA-project.Härnqvist (1974), Inkeles (1977), and Marklund (1983) are cited in order tomake a résumé of the obstacles seen in relation to the research goals ofthe IEA-project with respect to the kinds of multivariate analysisreported. In short: (1) statistical analyses which mix variables fromdifferent levels school/class/student produce an unclear result which isdifficult to interpret (Härnqvist, 1974), (2) IEA studies have beennotable for the up-to-date methods used including multivariate analysiswhich is central to their approach. The researchers were well aware thatIEA-data are under analysed and that earlier studies left a rich unexploredpotential for the next generation of researchers (Inkeles, l977) and (3),in Sweden there has been a lack of active use of IEA-data, particularlyconcerning the integration of the results with national evaluations inorder to answer policy questions. However, this is essential if we are tounderstand the educational relationships between goals, means and results(Marklund, 1983).
Entering the 90ies, the IEA study improved its survey machinery under theleadership of Neville Postlethwaite and progress were made in many fields,such as tests for international Rasch-scales of Reading Literacy in 28languages administrated in 32 countries, modern survey procedures for datacollection, non-response, quality control and professional documentation inmanuals and code-books. The multivariate analysis couldn't be coveredwithin the IEA project however for technical and economic reasons, but tookplace instead within the separate research project, Multivariate Analysisof Literacy (MALI). This project started in l992 with Department ofEducation and Educational Research at Göteborg University as the hostinstitution, because of its high capacity to handle big data sets.
Towards the end of the 80ies, multivariate methods were developed thatcould approach the multi level problem of school data in a more promisingway. Two such approaches have been applied to IEA-data; (1) the HLM(Hierarchical Linear Models), see for example Williams (1994), and (2) MCA(Multilevel Covariance Structural Analysis), grounded in work by BengtMuthén at UCLA, see for example Muthén (1994). This approach was adaptedfor the MALI analysis and also in STREAMS (Gustafsson & Stahl, 1997), thesoftware which was developed parallel to the project work much influencedby the urgent need within MALI for tools for handling large data sets andmethodological advancements in the field of latent variables and structuralequation modelling (SEM).
The students were given a wide range of reading tasks. These were organisedinto passages and two booklets administrated at two different testsessions. Items from the reading passages have been grouped in the originalIEA study into three categories: Expository, Narrative and Document. Thereading exercises include short simple stories, long narratives, densescientific prose, graphs and tables and the answers were given inmultiple-choice format. Questionnaires were used to collect informationfrom students, teachers and school principles and covered a wide range ofcharacteristics of home and school and of students reading habits andreading resources.
The student level model presented at the bottom of figure 2 illustrates howthe questionnaire data can be more thoroughly explored than in the past byusing the latent variable framework. Measurement models for studentsreading ability and students reading preferences were developed includingfactors of both general and specific nature (see figure 4 for the readingmodel). The approach for elaborated measurement is developed from amultivariate explanatory model of a single answer to a test item (or aquestion in the questionnaire) illustrated in figure 3, where, in contrastto the traditional response model with a true value and an error, thismodel contains a full spectrum of influences. Some of these are rooted insubstantial sources like the general ability to read or different specificreading factors; after having controlled for the general ability (so calledresidual factors); here chosen to be Expository, Narrative and Document.Other sources of influence are observational errors, or measurement errors,which in the case of the IEA reading test are illustrated by a generalmethod factor, Test Booklet 1, a factor effecting all the responses at thefirst test session. Another more specific method factor is the PassageLaser factor which captures what is common in all answers to questionslinked to the text about Laser. Gustafsson & Balke (1993) introduced theconcept "nested factor model" for the measurement model allowing formultiple influences on responses.
This approach to integrating psychometric measurement models withstatistical observational errors makes it possible not only to takemeasurement errors into account, thus much improving the quality of thestructural modelling, but also to estimate the magnitude of certain errorsources. One such interesting method factor is the passage factor, a kindof context factor mirroring the influence on the answers of the chosentexts for the test. As expected the reliability and validity of the passagefactors varied across languages and cultures; an important fact for thecomparability of the test results and when interpreting the results(Gustafsson, in press).
The research strategy has been to develop nested factor models and to startat the item-level before proceeding by aggregating variables up to passagelevel and further, through which process it gradually becomes possible toidentify design and method factors along with general and specificcomponents of abilities. The multi level approach with latent variables inseveral populations then explores stable individual and school factorsacross countries. International comparisons can be made both in achievementlevels on different factors, in the factor-structures at individual and atschool levels and in structural relationships between error-free variables.
In short the MALI-analyses change earlier results and come up with newones. Some of the main findings answer the following critical questions:
Are the postulated factors Document, Narrative and Expository supported by data?Balke(1995), Gustafsson (1995).The postulated factors of the reading test are examined versus models forintelligence.Gustafsson (1995).Are the IEA Rasch-scales fair international scales?Gustafsson (in press).Is the ranking order of the countries changed when the results arecorrected for Speediness?Rosén (l997) Gustafsson (in press).Do earlier results concerning gender differences change when themeasurement of reading ability is differentiated into nested factor models?Rosén (1996)Are teacher effects possible to trace in the test results?Lundberg & Rosén (1995)What is the relationship between reading ability and reading preferencesand how do reading resources at home influence this relationship?Munck (1995)How does the two-level modelling approach operate in different school systems?(Munck, 1995).
Traditional analysis has earlier given a dizzy picture of the relationshipsin our models. This has now changed as the integrated micro and macroanalysis using MCA offers a proper test of the effect hypotheses at bothlevels and takes measurement errors into account. In addition, the featuresof different school- and societal systems are reflected in these models inan adequate way; an important advancement in quantitative evaluationcorresponding to the goals of the IEA study and generally to efficienteffect studies.
The more qualified analysis has been made possible through methodologicaladvancements growing out of a symbiosis between social researchmethodology, statistics and information technology. The software STREAMS,structural equation modelling made simple (Gustafsson & Stahl, l997), cannow assist the researcher to perform advanced multilevel modelling withlatent variables on large data sets using available software for SEM(LISREL, EQS and Amos). Along with the rapid improvement of the technicalprerequisites for multivariate analysis, there is a need for proper designand data collection in order to shape tomorrow's information environmentfor evaluations. There is a lot to gain if the basic principles of effectstudies, of measurement models, of multilevel analysis and of statisticalquality in evaluations (Ribe, l997) guide the design of such informationsystems.
Ingrid M.E. Munck, Statistics Sweden, Box 24 300, S-104 51 Stockholm, Sweden.
pp 72-75
Gunni Kärrby, 1997: Rating of pedagogical quality - Focus on preschool/Bedömning av pedagogisk kvalitet/. Pedagogisk Forskning i Sverige, Vol 2, No 1, Pp 25-42. Stockholm. ISSN 1401-6788.In this article the concept of quality in school and preschool education is discussed. Quality can be seen from different perspectives. Here it is argued that quality has to be seen from the perspective of the learning individual and take into account factors that determine the pedagogical process. Quality in school and preschool is seen in relation to research on long term effects of preschool experience and motivational aspects of school achievment. Results from studies of quality as a pedagogical process in Swedish preschools are presented.
Quality is rated as teacher awareness of physical, interactional and goal aspects of the learning environment.
Quality as a policy issue is discussed. What constitutes quality and how to evaluate aspect of quality in the educational sector has become an important policy issue in our time of economic recession. In Sweden discussions on quality in the educational system started late compared to other public areas like welfare service and public medical care.
In the childcare sector quality as a relative or normative concept is internationally debated. Those who argue for the relativistic approach see quality as a matter of negotiations between different "stake holders" or interest groups. Others see it as a rather stable concept that can be measured in a reliable way.
Within the school sector the concept of quality has to a great extent been taken over from the business world and terms like management, efficiency, customer service, etc are used to indicate quality aspects.
In this paper the concept of quality is seen from the child´s perspective and is focused on educational practise. It is argued that quality in this sense may be measured or rated in a systematic and reliable way.
Quality in preschool practise as well as in school is a complex process. The assumptions are that indicators of quality are based on physical resources, social interaction between teachers and children and teachers´ awareness of the role these play for the learning and developmental processes of the children.
Research on the relationship between childcare conditions and child outcomes shows that motivational and learning conditions seem to have more importance for school progress and life long learning than scholastic attainment. Certain conditions are particularly important for the learning and development of the child like the ability of the teacher to create a stimulating learning environment, the presence of clearly formulated goals and a close relationship to the parents. Longitudinal studies also show that high quality preschool intervention for socially deprived children may have long term effects on the childrens´ lives like completed high school, getting a college education, fewer assignments to special education and less criminality. School management factors like leadership and school climate has been shown to be more crucial for success in school than social background.
A short summary of Swedish childcare policy is presented. Swedish childcare is internationally considered to have high quality. There are, however, few evaluations on educational quality of Swedish preschools or on child outcomes.
Research projects carried out at the Department of Education and Educational Research and National Economis at Gothenburg university are presented. In these studies attempts were made to find instruments for the rating of quality in daycare centers. The Early Childhood Environment Rating Scale (ECERS) was translated and tried out in a number of studies. The ECERS consists of 37 items each constructed as a 7-point scale. The areas for quality are: Personal care, furnishing and inventories for children, language experiences, fine and gross motor development, social and creative activities and adults´needs. The total scale was found to have high inter-rater reliability. In populations of 17, 40, 44 and 20 daycare centers average interrater agreements was between 82 and 95 % for total agreement or the difference of one point. Homogeneity measured with Cronbach´s alpha was shown to be between .87 and .94 in three studies. Dimensions of quality revealed by the use of factor analysis showed a pattern interpreted as the basic pedagogical qualities emphasized in the Swedish the national educational program for preschools. A main factor accounting for 40 % of the variance contained items that all referred to the quality of social and pedagogical interaction between teachers and children.
Comparisons between parental perceptions of quality in their child´s daycare centers and independent ratings of quality can be seen as a validity test of quality ratings. In a study including 40 daycare centers
parental perceptions of the quality in their children´s daycare center were obtained. The results indicated that there was a high consensus between the level of quality in the center rated with the ECERS and the parents´ own evaluation of quality.
The relationship between cost and quality in daycare centers was investigated in two reported studies. No general relationship was found between cost and quality. However, variations in income level in the recruitment area of the daycare center were found to influence the relationship between cost and quality.
ECERS as an instrument for self ratings is discussed. In a community study the aim was to evaluate the effects of in-service training based on the ECERS and self ratings among the teachers. Self ratings were compared to independent ratings of quality. The agreement was found to vary between 55 and 85 % between an independent rater and individual teachers. Teachers in high quality centers tended to underrate themselves while the opposite was true for low quality centers.
The issue of educational policy and quality ratings is discussed. The view that evaluations of quality made with rating scales may have the function of a control instrument in order to impose power is held by some researchers. Others focus on evaluation of quality as a way to change and development of educational practise. In this paper both self ratings and independent observations of quality are seen as opportunities for teachers to interact and discuss their practise in a constructive way.
Gunni Kärrby, Department of Education and Educational Research, Göteborg University, Box 300, S-405 30 Göteborg, Sweden.
pp 75-77
| welcome | previous issues | summaries |
Pedagogisk Forskning i Sverige, Institutionen för pedagogik, Göteborgs universitet, Box 300, SE-405 30 Göteborg
Tel 031 - 773 24 56 Fax 031 - 773 24 62 epost pedagogisk.forskning@ped.gu.se
