Clark Quinn (cnquinn@knowledgeu.com)
Tue, 13 Apr 1999 13:23:37 -0700
Date: Tue, 13 Apr 1999 13:23:37 -0700 From: "Clark Quinn" <cnquinn@knowledgeu.com> Subject: Re: summative evaluation:comments
Mark,
>I am currently researching the role of 'evaluation' of
>software/multimedia resources used in the history classroom.
>
>Current research suggests that the traditional checklist method used
>for evaluating CD-Roms and other software has had its day
>(NCET/BECTa checklist in UK; MicroSIFT in USA; others elsewhere).
Well, I wouldn't use a checklist alone. At Access CMC, we developed
an education technology evaluation methodology that included expert
review (supported by a checklist). However, we developed an
educational theory framework before developing the checklist. So...
>Reasons given for this are:
>1. Checklists do not take into account learning theories
>(i.e. Gardner's multiple intelligence's; constructivist learning
>theories), they tend to concentrate on technical/IT issues rather
>than educational issues.
I would argue that our methodology and checklist DO take into account
learning theories.
>2. Checklists are used in less than rigorous conditions
>resulting in poor evaluation.
We developed a quite rigorous evaluation method, that includes
limited user testing as well (there are issues that any checklist
won't catch). BTW, this method was designed to be commercially
pragmatic, that is it is designed to be affordable and respond in
competitive timeframes.
>3. There appears to be very little consistency in the results
>gained from checklist use (empirical data collected from various
>investigations).
Well, we did formative evaluation, but I must admit that it is an
action research approach, providing results designers need, not
formal validity.
>4. The classroom activity for which the software is intended is
>not appropriate for the software package (poor selection of activity
>giving a bad evaluation).
This doesn't seem related to the checklist issue, but perhaps I don't
understand.
>The list goes on. I would be interested in any comments, feedback,
>examples of checklists or alternative evaluation models used.
I guess this qualifies as one or both of the latter two. Checklists
can be limiting, unless:
they are carefully designed
you carefully structure their use
However, they can be powerful scaffolding to ensure consistency and
thoroughness. For more on our process (acronym LEA for Learning
Effectiveness Assessment), contact John Eklund (johne@accesscmc.com)
who now has responsibility for the service.
Hope this helps, -- Clark
-- Clark Quinn Knowledge Universe Interactive Studio (510) 768-2408 cnquinn@knowledgeu.com
This archive was generated by hypermail 2.0b3 on Tue 13 Apr 1999 - 22:38:51 MET DST