Technology Assessment 101 - Part One

BONUS POST

This Topic of Choice for this week’s post is part of my own discussion for consideration of completing my dissertation and coincidentally relates to the Futuring and Innovation CS855 course I am taking this term, so I have decided to include posts about it here.


Evaluating Evaluation  – Why evaluate? And why learn assessment theory?

The term “evaluation” is used in a myriad of contexts often used to cover many different kinds of judgments, from which it can be generalized to be the act of judging and assessing the worth of an item or as a systematic determination of a subject’s value and merit. In this regard, we all engage in various forms of evaluation on a daily basis. Whether it be a part of an informal and or part of a formal form of a process. In the case of informal evaluation, the process is often implicit and with little or no structure and where we go through a routine activity of determining a value and making a judgment. For example, in making a decision of which gas station to purchase gas at, or which breakfast cereal to buy at a grocery store, we quickly evaluate the various implications of our decisions and determine a solution. Whereas in the form of a formal evaluation, we go through an explicit process where we follow (generally) a set of data driven activities with structured steps during the decision making process. An example of which can be anywhere from simply deciding on a college to attend, or a car to buy, or it can be as complex as determining company policies and or program directives. Never the less, the outcome of our evaluations will be the decisions that we make for one course of action over another.

Evaluation is an essential tool for improving and maintaining quality. From product, to programs, to services, to organizations and or as equally important, to technologies, it is important to continually assess for quality and in some cases for application. By evaluating evaluation methods we can ensure accountability and prepare/build for the future and by understanding the importance for use of best practices in use of assessment, further allows us to strengthen our credibility of our processes. Furthermore, we can begin to base success of the process, based on sound principles that a fundamental feature of almost every type of evaluation and assessment is its complexity and that we should therefore subscribe to the view that the evaluator can help ensure the quality of feedback about the assessment by making certain that evaluation is “future action-directed, carries both scientific and stakeholder credibility, and takes an holistic approach.” (Chen,2005, p. 6)

The approaches and methods of evaluation that have been developed and are in use are plentiful, thus, it is important for evaluators to select an approach or method that is appropriate for use. For example, if one were to take a Universalist approach, a view would be that there is a “best” method for evaluation. This would be hard pressed to hold true, since one best evaluation approach or research method cannot possibly cover every type of evaluation since situations and conditions may vary greatly from one assessment to another. Therefore an alternative and perhaps better approach to a Universalist theory is that of a Contingency View. In contrast, a contingency view is a perspective that there is no single best way to perform evaluation, and the basic principle is that the individual nature and the uniqueness of evaluation require a range of variation within each approach. In general the contingency viewpoint provides for adjustment in evaluation methods which best fit and serve the needs of the assessment. A lack of understanding of various theoretical and paradigmatic positions and the different methods of evaluation could undermine the design, conduct, and outcome of the desired assessment. Alternatively, one of the first forms in which theories and paradigms are considered is the theory about doing evaluation where we familiarize ourselves and begin to understand desired variations based on our personal beliefs and assumptions about evaluation (evaluating evaluation).

In evaluating technology and in particular regarding causal processes, two basic assumptions (or theories) can be generalized and described as follows. A Descriptive assumption concerns the causal processes that lead to whatever problem a technology, or technology-based project or program is aimed at addressing whereas a Prescriptive assumption prescribes those entities and activities (components, resources, systems, people, etc.) that the designers and/or other key stakeholders in a technology, or technology-based project or program deem necessary to its success. (Learning Space, 2012). Furthermore, in evaluating technology it also becomes important to consider opposing positions or perceptions that often occur. Two of which that commonly occur are those of techno-optimism and those of techno-skepticism.

For further entries to this discussion of evaluation and assessment see:

Technology Assessment 101 - Part Two for short discourse on the need for assessment, our assessment abilities, and ideas for an idealized inquiry with an iterative process
  
Technology Assessment 101 - Part Three for discourse on assessment and evaluation, making effective assessments, and understanding the decision making process

References

Clarke,A. (1999) Evaluation Research: An Introduction to Principles, Methods and Practice, London, Sage.

Chen,H.-T. (2005) Practical program evaluation: assessing and improving planning, implementation, and effectiveness. Thousand Oaks, Sage.