Wednesday, September 5, 2012

Technology Assessment 101 - Part Two

BONUS POST

In Technology Assessment 101 Part One there was discussion on what I termed "Evaluating Evaluation" and in particular discussion about why we should evaluate and why we should learn about assessment theory. In part, the discussion was based on our continual need for evaluations and assessment to insure quality. To further this discusion I would like to continue with post of discussion here on our abilities for assessment activity.



I read today in the news that Sir Tim Berners-Lee's World Wide Web Foundation ranked Sweden as tops in a new global index evaluating the state of the web in 61 countries with the US coming in second and the UK coming in third.  http://www.bbc.co.uk/news/entertainment-arts-19492084  Ranking was based on both social and political impact of the web. I found it interesting to learn that one in three people are using the web globally and that 95% of the population of Iceland uses the web, Very impressive figures, and the ability to assess something as large and complex as the internet even further impresses me.

In an article titled Web Science: An Interdisciplinary Approach to Understanding the Web, the author states that the web must be studied as an entitity in its own right to ensure it keeps flourishing and prevent unanticipated social effects. In extension to that however, despite the web's great success as a technology, it remains as an entity suprisingly unstudied. In fact it has been surmissed by many that the web is a technologywhich is changing at a rate greater than what even the most knowledgeable researchers have the ability to observe. Still, it is now inextricably linked to the future of human society, therefore we have a duty to ensure that future development of the web  will make the world a better place.

One may ask how this correlates with our ability for assessments and I digress that it merely stresses a need for the increasing need for finetuning our methods of analyis and our abilities to assess. Furthermore, if I may suggest that considerations be made raising our awareness of the need for assessment and that possibly it could begin with an idealized model for inquiry. This is not to say that the models presented here are the only ideal forms as prescribed by the steps presented and that is not the intention of this post, rather it is to present some of the important aspects of inquiry. It is important to note that variations may be needed and in fact recommended such that appropriate inquiry can be made, and these changes may often lead to new ideas, concepts and theories for further use.

Idealized Inquiry as suggested from http://people.lis.illinois.edu/~chip/pubs/inquiry/process.shtml

a. Formulate problems - dicover real world situations, construct phenomena, specify variables, analyze it
b. Observe - phenomena in real world conditions
c. Invesitgatge - select appropriate research methods, instruments, tools, designs, and sampling procedures
d. Analyze - explore and describe data, relations, and draw conclusions
e. Communicate the findings such as observations, interviews, surveys, etc.

Idealized Inquiry as suggested from http://www.harnessingcollectivewisdom.com/pdf/newagora.pdf

Steps in the definition of Inquiry -

a. complex situation
b. frame and focus on a triggering question,
c. articulat observations,
d. clarify meaning,
e. cluster inductively,
f. develop shared language,
g, vote and rank,
h, structure abductively,
i. intepret learning,
j. evaluate

Idealized Inquiry as suggested by Karl  

a. Uncover and discover – complex situation or phenomena  
b. Acquire a baseline - be a hunter and gatherer
c. Systematic Analyses - Holistic Critical Thinking
d. Baseline – Lay-it-all-out-there
e. Iterate - lean on design to refine (see 5 why’s below as a example)
f. Get Consensus – include stakeholders
g. Present new knowledge - results, ideas, concepts
           Repeat as necessary -
Note: steps are often non-linear and may become concurrent processes

More thoughts on inquiry and process:
For me recently, when attempting to do a causal diagram for my study, the process of diagramming cause-and-effect helped me brainstorm through some of my thoughts and ideas on the concepts and constructs that I wanted to hit upon. Diagrams are a great tool and often provide a good start. Hopefully the process in generating new diagrams will help by requiring you to re-visit your baseline while creating a visual construct of the data (but I am a visual type person, so that is my preference). It's even more applicable when you are hitting on a topic based on model based reasoning. Have you tried the Ishikawa (fishbone) diagrams too? They may help with identifying and listing out key concepts or ideas you may have, and they are a very good "quality control" tool that can be used to help analyze (or qualify) your question(s). 
 It may also help to go through iterations using the 5-whys question-asking technique too.
5-Whys Example
As I was thinking about this the other day, just for interest sake I tried one and thought I would share to help spark some thoughts:

For further entries to this discussion of evaluation and assessment see:

Technology Assessment 101 - Part One for a short discourse on evaluating evaluation, why we evaluate and why we should learn about assessment theory

 Technology Assessment 101 - Part Three for short discourse on assessment and evaluation, making effective assessments, and understanding the decision making process

No comments:

Post a Comment