DEW Program Evaluation of Digital Earth Watch

Digital Earth Watch Program Evaluation 
NASA REASoN Project Survey and Forest Watch Survey

Jeffrey S. Beaudry
University of Southern Maine

A. Mavourneen Thompson
Thompson Consulting

This chapter is composed of two sections: the first explains how one NASA REASoN project, Forest Watch of the Measuring Vegetation Health series, performed a program evaluation of its project, which included an online survey; the second section illustrates how the survey of the Forest Watch project developed into another, more widely comprehensive survey. The Question Bank, which was developed for use by several other NASA REASoN projects. 

The Forest Watch Survey
Forest Watch project managers were interested in obtaining an evaluation tool that allowed them to efficiently gain information about their project activities.  The program managers were looking for feedback from those persons with whom the project engaged: the teachers, students, and administrators ? the “stakeholders”.  This information would help the managers gain insight into the effectiveness of their project activities, thus enabling them to adjust current and future activities in order to fully achieve the overall goals of the project ? in this case, NASA REASoN goals: NASA Operating Principles for Education ?Relevance, Content, Diversity, Evaluation, Continuity, and  Partnerships/Sustainability. Keeping in mind the overall NASA Education goals which their project was theoretically and, hopefully, practically compelled to achieve, the Forest Watch managers chose to initiate a program evaluation.

The Value of Program Evaluations.
Program evaluations serve many purposes; among these, the following:
1. Evaluations advance the state of knowledge about society’s needs.  
2. Evaluations help ascertain if innovative programs have been successful.
3. Evaluations determine how effectively a program can contribute to a solution.
4. Evaluations improve the overall performance of grant making. 
5. Evaluations identify best practices, the results of which can be 
    shared for the benefit of the larger community.  
6. Finally, evaluations improve individual grantor and grantee performance.
7. Evaluations provide grant makers with an understanding of the extent to which the funded programs have fulfilled the established goals and objectives.

The Forest Watch Program Evaluation: A Discovery and Exploration
The Forest Watch evaluation process had three components: interviews, online surveys, and case studies. Both the interviews and surveys were part of a “real-time” feedback system where managers gathered relevant qualitative and quantitative data, useful for assessing their project. For instance, the survey was administered online in an attempt to accommodate teacher who would be able to approach the survey at their own convenience, answer questions, leave the survey, and return to where they had left off.  Thus the survey was easy, accessible and convenient for teachers’ complex classroom environments. The answers were analyzed then transmitted to the program managers immediately for their review.

Evaluating the Forest Watch project stakeholders while they were actively engaged in project activities created an opportunity for each program manager to gain valuable insight into his/her program.  This benefitted managers as well as the larger community of people committed to NASA REASoN program goals. Data gathered by each REASoN program manager provided a means for meeting NASA reporting requirements.  Therefore, NASA administrators obtained data that allowed them to analyze the overall effectiveness of the REASoN Projects.

For instance, managers could be informed about the issue of sustainability, one of the NASA REASoN operating principles for education: If NASA’s goals were reached, non-grant sources were more likely to help sustain the project with entrepreneurial investments, either by supporting infrastructure, subsidizing staff, or simply creating new grant sources.  Not to be forgotten, project managers who demonstrated success in meeting NASA’s goals were more likely to be given high marks for future grant awards.

Components of the Forest Watch Program Evaluation
Interviews
The interviews with selected representatives from the funded program were designed to provide the manager with qualitative data about the program and to increase the understanding of what worked, what didn’t, and why. Interviews were used to (1) formulate survey questions and (2) provide preliminary information for the case studies to follow.

Survey
Evaluators designed and implemented a survey that provided both qualitative and quantitative data about the program. The Forest Watch survey allowed evaluators to access data from a broader range of people than the interviews would practically allow.  The survey, administered online, significantly enriched the evaluation data.

Case Studies
Case studies shed light on factors that impact performance, such as administrative style, employee attitudes and peer culture within the program. Balanced evaluations involve both quantitative and qualitative methods of inquiry.  While there are several qualitative methods of research, three are major:  participant observation, non-participant observation, and ethnographic research.  Case study is a non-participant observation method of qualitative evaluation.  The case study allows the evaluator to gain insight into why a particular program does or does not operate well.

The case study may involve in-depth interviews and observations of people involved in a particular program, as well as a review of program documentation and materials.
Case studies are conducted through in-depth interviews with persons directly involved in the delivery of the funded programs as well as persons outside the agency who interact or collaborate on service delivery. Clients may be interviewed.  Interviews are supplemented with a review of existing program material including program descriptions and progress reports.

A Description of the Forest Watch Online Survey
With questions based on the NASA REASoN Operating Principles for Education, the survey was tailored specifically to the MVH “Forest Watch” Project.  

Samples of Forest Watch Online Survey Questions
Diversity/Demographics: gender, culture, ethnicity, socio-economic status, inclusive, disabilities, equity  

1. Indicate the percentage of students in your school who qualify for free and reduced lunch: ___0-5% ___6-10% ___ 11-15% ___16-25% ___26-40% ___ more than 41%.

2. Indicate the percentage of students in your school who are English Language Learners (ELL): ___0-5% ___6-10% ___ 11-15% ___16-25% ___26-40% ____more than 41%.

3. Indicate the percentage of students in your school who qualify for special education services: ___0-5% ___10% ___ 15% ___25% ___41-60% ____ 61% or more.

Relevance:  current, hands-on, experiential, meaningful, interesting, service learning
(Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree)

1. Forest Watch helps me to develop hands on activities for my students. 
2. Forest Watch activities engage my students.
3. I often teach my students outdoors.
4. I often use classroom lessons to help students understand their community.
5. I often help my students work with community members.

Content/Pedagogy: grade level, specific content areas, local, state, national content and performance standards

The following questions relate to these outdoor Forest Watch activities:
* Outdoor data collecting
* Indoor learning and labs
* Data compilation
* Data submission

(Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree)

1. Forest Watch fits within my school district’s content and performance standards.
2. Forest Watch is mostly suitable for high-achieving students.
3. Forest Watch is mostly suitable for mid-achieving students.
4. Forest Watch is mostly suitable for low-achieving, at-risk students.
5. Forest Watch is suitable for students with learning disabilities.

Evaluation: state assessments, local assessments, program evaluation, reflective assessments

(Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree, Not applicable)

1. Forest Watch helps me teach and assess my students in mathematics content.
2. Forest Watch helps me teach and assess my students in science content.
3. Forest Watch helps me teach and assess my students in technology content area.

(Yes, No)

4. I need help in assessing my students’ learning in relation to Forest Watch.
5. I can assess my students’ learning in relation to Forest Watch.
6. My formal assessment of Forest Watch is:
a. Selected response tests
b. Extended writing tests (essays and written tests)
c. Formal product (lab notebook, written report, etc.)
d. Performance Assessment (presentations, posters, power points, etc.)

Continuity: resources (materials and planning), systemic use and change (transformation, model program), horizontal and vertical alignment

(Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree, Not applicable)

1. Forest Watch reflects what my students need to learn.
2. Forest Watch helps me to connect to my students’ prior learning.
3. Forest Watch activities and lessons are taught in different grade levels in a systematic, planned progression of learning.
4. I have the instructional resources to use Forest Watch (computers, instruments to measure color reflectance, access to plots).
5. I need to acquire more supplies/materials in order to use Forest Watch.
6. My colleagues are very likely to use Forest Watch in their classes.
7. The Forest Watch website has helped me to use Forest Watch tools and resources.
8. The Forest Watch website has helped me to do outdoor activities with my students.
9. The Forest Watch website has helped me to write lesson plans.
10. The Forest Watch website has helped me to do indoor labs with my students.
11. The Forest Watch website has helped me to review Forest Watch data books.

Partnerships/Sustainability: administrative support, leadership, how fits in relation to all else, conflicting demands of other programs, professional development, collaboration with colleagues and experts

(Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree, Not applicable)

1. Forest Watch activities help me teach my students how to learn science and value science learning.                                            
2. Forest Watch activities help me to adopt teaching and assessment strategies that support the development of student understanding of science.                        
3. Forest Watch activities help me to adopt teaching and assessment strategies that nurture a community of science learners.
4. Forest Watch activities help me to adopt teaching and assessment strategies that support the development of student understanding of science.

Some of the Results of the Forest Watch Survey

Content and Pedagogy: Performance Standards
The Forest Watch curriculum was reported by 90 percent of the participating teachers as fitting suitably within each of their school district’s content and performance standards.  However, interviews with the same teachers indicated that it was their familiarity with Forest Watch as participants that led them to agree that the program is aligned with standards.  On the face of it, Forest Watch would not appear to be aligned with state standards to the uninvolved teacher who might be considering using the program.

Content and Pedagogy: Level of Student Suitability
Forty percent of the participating teachers agreed or strongly agreed that the Forest Watch program was suitable for students who had learning disabilities. Associated with this may be the fact that 90 percent of the teachers indicated that Forest Watch helped them to develop hands-on activities for their students.  

Content and Pedagogy: Scientific Inquiry
New science standards heavily emphasize student inquiry.  It is important that 90 percent of the teachers agreed or strongly agreed that the Forest Watch project helped them raise the level of their students’ sensitivity and interest in scientific inquiry.  The fact that this program allowed inquiry in partnership with university researchers may have further helped to motivate students. 
Content and Pedagogy: Teaching Science, Mathematics, and Technology Content

The Forest Watch curriculum was reported by 94 percent of the respondents as being helpful in teaching and assessing students in science content;  65 percent of the participating teachers reported it was helpful in teaching and assessing students in mathematics concepts; and a majority (59 percent) reported that Forest Watch was helpful where technology content was concerned.

Content and Pedagogy: Equity
 Most of the teachers reported that the project also helped them address gender equity issues among their students.   Both boys and girls have a more even playing field with the Forest Watch program.

Continuity and Alignment: Curriculum Alignment
The Forest Watch curriculum was reported by more than 94 percent of the participating teachers that the curriculum both reflected that which their students had to learn and matched their prior learning (78 percent).

Continuity and Alignment: Systematic Progression of Curriculum
Reflective of certain issues with the Forest Watch curriculum, however, is the fact that there was no strong consensus among the teachers that Forest Watch activities and lessons are taught in different grade levels in a systematic, planned progression of learning.  This would likely pose a problem for busy teachers and reduce their commitment to use the Forest Watch curriculum. This raises the question of whether Forest Watch curriculum adheres to content standards.
Continuity and Alignment: Resources

Availability of appropriate resources is very important.  Almost 90 percent of the teachers reported that they had access to computers, instruments to measure color reflectance, and access to plots.

Continuity and Alignment: Forest Watch Website
The results of questions regarding the Forest Watch website indicate a need to reconsider the design, for it was to only two questions that respondents agreed the site was useful and effective.  First, to the question of whether the site was useful in helping teachers use data, there was a relatively strong positive response:  45 percent of respondents reported that they either agreed or strongly agreed; 33 percent offered no opinion.  Since part of the function of the University of New Hampshire is to produce data, it seems that this function of the site was effective.  Second, when asked if they found the Forest Watch website useful in helping them to use Forest Watch tools and resources, 56 percent agreed or strongly agreed.  

On the other hand, 72 percent of respondents were noncommittal when asked if the site helped them to organize and do outdoor activities with their students.  Did the website help teachers to write lesson plans?  Almost a third disagreed and 44 percent were noncommittal.  Only a few of the teachers found that the site helped them to review the Forest Watch Annual Data Books; this could mean that either the teachers did not review the data books or that they did not need the site in order to do so.  Apparently, only a third of the teachers agreed that the site helped them to use Forest Watch power point presentations from the annual meetings.  It appears that the website needs to be reconsidered and redesigned in order to be helpful in driving instructional change.

Continuity and Alignment: Forest Watch Measurements, Protocols and Activities
 In responding to Forest Watch measurements, protocols and activities, teachers reported the following:

• 56 percent of teachers always collected and sent needle samples to UNH.
• 67 percent always measured diameter breast height (DBH).
• 56 percent always measured and recorded needle retention.
• 56 percent always measured and recorded canopy, closure and ground cover.
• 61 percent always reported ozone damage symptomology and needle damage assessment.
• 50 percent always reported tree height/height of live crown.
• 61 percent never completed Global Positioning System/plot center.
• 72 percent never completed Spectronic 20 analysis.
• 50 percent never measured needle anatomy/thin sections.
• 72 percent never did Land Cover classification/remote sensing with Multispec.
• 50 percent never completed the Qualitative Site Analysis activity.
• 44 percent never completed the water content activity.
• 89 percent never completed the ALTA activities.
• 72 percent did not have an adequate number of ALTAs for the reflectance measurement activity.
• 72 percent did not have an adequate number of SPEC 200s for the reflectance measurement activity.

Questions arise about whether teachers had sufficient materials for the specific protocols and activities reported above.  Also, one must ask if class periods allotted enough time for the scientific pursuits of Forest Watch.

Evaluation and Assessment: Summative Assessments
Thirty-five percent of teachers reported that they used Forest Watch to support summative assessments.  Of those, five teachers (30 percent) always or sometimes used selected response tests, extended writing tests, lab notebooks or written reports, and performance assessments (presentations, posters, power points) for the summative assessment of Forest Watch activities.

Evaluation and Assessment: Rubrics
Approximately one third (35 percent) of responding teachers said they developed rubrics to assess their students’ performance of Forest Watch activities.

Evaluation and Assessment: Generating Forest Watch Data
Forty-seven percent of teachers reported that they generated Forest Watch data without assessing their students’ work. This may show that half of the teachers use Forest Watch as an extension activity, not closely related to the science content that is assessed.  

Evaluation and Assessment: Aspirations
Respondents were asked “To what degree has your involvement in Forest Watch influenced you to pursue an advanced degree in Environmental Science, a related science field, or education?”  More than half (53 percent) answered positively. The responses indicated that Forest Watch informs teachers and provides encouragement for professional development.

Evaluation and Assessment: Promoting Technology Development and Environmental Understanding
Forest Watch research activities were reported as having helped students understand the importance technology development and environmental understanding.  All but two teachers reported this, with 41 percent of respondents saying that more than half of their class benefited with a better understanding of technology development and environmental understanding.

Evaluation and Assessment: Environmental Monitoring
Teachers were asked to respond to the following statement: “Forest Watch helped you add or improve your classroom teaching of environmental monitoring.”  Teachers generally agreed and offered the following comments: 
Background understanding, hands-on science activities;                             
Helps kids see the importance of attention to details;                             
I have used the Ozone creation models to teach students about long-range pollution.   
It is fair to say that Forest Watch provides a specific, relevant example of environmental monitoring by taking a single, indicator species and exploring its importance in detail.

Evaluation and Assessment: Professional Development
Teachers were asked to indicate how Forest Watch may have enhanced their professional development.  The following teachers’ comments are presented according to themes:

Professional Development as a Science Teacher: Keeps me in touch with cutting edge scientific research and the scientific community; learning content, learning about how science is done, expanding my understanding, tools and methods of science; summer training; other teachers’ descriptions of their use of FW; updates on current research by Barry Rock and others; annual meetings always provide an interesting selection of professional development topics.                                                                                                                                                                                         

Inspiration of Authentic Science Research: I have been inspired by people doing authentic research; using authentic science in the classroom and outdoors; data collection strategies provide a real life data collection situation; I am aware of ozone level monitoring, white pine health, tip necrosis, etc. I am more observant and knowledgeable when I take my students outdoors into our pine and beech groves.                                                                                                                                                        

Inquiry and Project-Based Science Linking Math and Science: Hands on activities for students; use of technology for environmental assessment.
Specific Example of Data Collection: Data collection on trees has been helpful in teaching students the importance of quality data collection; workshops/meeting with training in various protocols; the technical data on red edge inflection and chlorophyll content. 

Evaluation and Assessment: School-Scientist Partnerships
“The purpose of the Forest Watch program is to partner with schools in evaluating the health of white pine in New England.”  Teachers gave the following suggestions for the improvement of Forest Watch with respect to school-scientist partnerships:
Opportunities to Share Information and Ideas: Keep the annual meetings going.
Cutting edge scientific research and the scientific community!!!!
Keep up the great connection!!  Other teachers’ descriptions of their use of F; visits to school by Forest Watch scientists; updates on current research by Barry Rock and others doing authentic research; using authentic science in the classroom and outdoors; data collection strategies provides a real life data collection situation hands on activities for students. 

Partnership and Sustainability: Inquiring and Valuing Science Learning
Teachers were asked to what degree Forest Watch helped them teach their students science inquiry.  All of the 17 teachers who responded answered that they agreed or strongly agreed.  Similarly, to the question of whether Forest Watch helped the teachers to encourage their students to value science learning, all 17 agreed or strongly agreed. All but one of the teachers agreed or strongly agreed that Forest Watch activities helped students generate their own quest ions about science and that the program elicited authentic science inquiry. 

Partnership and Sustainability: Strategies for Teaching and Assessing
Teachers showed unanimity in reporting that Forest Watch activities helped them adopt teaching strategies that support student understanding of science. All but one of the 17 reporting teachers said that Forest Watch activities helped them adopt teaching and assessment strategies that nurture a community of science learners.

Partnership and Sustainability: Academic Community
Most of the teachers (60 percent) said the connection to University of New Hampshire researchers was important or very important.

Partnership and Sustainability: Reasons for Participating in Forest Watch: Alignment, Outdoor Learning, Global Environmental Issues, and Mathematics and Technology
It was important to all but one of the teachers that the following statement influenced their participation in Forest watch: “There is a close alignment between Forest Watch and our state science standards.”  This survey question presented some ambiguity as to the resulting data.  However, it most likely means that the participating teachers believed that the Forest Watch program would match their state’s standards; whether they did or not is not a subject in this question. Nevertheless, the results show that matching state standards was important to most of the teachers.

With regard to this statement, “Forest Watch creates opportunities for outdoor learning.”-the teachers unanimously agreed that this expectation was important or very important for their participation in the program. Likewise, teachers reported the same for these statements: “There are important connections between Forest Watch activities and global environmental issues.” and “Forest Watch helps me to integrate science with mathematics and technology.”
Partnership and Sustainability: Continued Participation

The teachers all reported that they would participate in Forest Watch professional development opportunities using distance learning technology (Webcasting, Learn Linc, Elluminate, etc.).

NASA Question Bank: Collaboration Among REASoN Project Managers to Assess Projects’ Outcomes as Related to NASA “Operating Principles for Education”

The Question Bank was developed with the goal of furthering NASA’s efforts to provide rich and unique scientific data to every teacher and classroom in the United States. The Bank was developed with the following principles, promoted by the National Aeronautics and Space Administration (NASA), in mind:
NASA Operating Principles for Education:
i.   Relevance
ii.  Content
iii. Diversity
iv. Evaluation
v.  Continuity
vi. Partnerships/Sustainability

These principles are also described as: Customer Focus, NASA Content, the Pipeline, Diversity, Evaluation, and Partnerships/Sustainability.  

NASA encourages education programs with the goals of (1) increasing the number of students pursuing advanced degrees in science, technology, engineering, and mathematics, and (2) making the most effective instructional use of computers, the Internet and other developing technologies. 
Within the structure of a “logic model” (See Appendix II), often used by strategic planners and evaluators, the principles became “outcomes” which NASA REASoN grant products may attain.  Thus, the NASA principles, as desired outcomes, are embedded in the Question Bank.  

The process works like this:  Each Principal Investigator (PI) of a REASoN grant can determine his/her product’s effectiveness by posing questions selected from the “survey bank of questions” (Bank) to his/her various stakeholders.  For instance, a REASoN Formal Education K-12 program manager may want to know if his/her REASoN product ?perhaps a website providing NASA Earth Science data ?encouraged students to consider a NASA-related career.  The PI would select Formal K-12: Students: G. Careers: Question 2:  “The product made me think I want to study science in college.”  

Thus, the PI discovers whether the NASA product is influencing this particular student to pursue a scientific career, or to wish to study in the content area.  The PI’s product has, therefore, demonstrated attainment of one of the NASA principles:  “…develop ongoing relationships between NASA and adult-led groups of students in order to expose students to unique NASA content, careers, and Mission.” 

The Bank has three major areas: Formal K-12, Informal, and Higher Education.  Within each major area there are questions suitable for the different stakeholder groups, as follows:

i. Formal K-12
a. Teachers
b. Students
c. Administrators
ii. Higher Education
a. Faculty
b. Students
c. Administrators
iii. Informal
a. Teachers
b. Virtual Users
c. Individuals/Community Groups

Following are samples of questions from the Formal, K-12 Teachers section of the Question Bank.  The PI interested in discovering teachers’ responses to questions of Content, Diversity and Partnerships/Sustainability (three of the NASA REASoN Operating Principles) would construct his/her survey with selections from the following questions from the Bank: 
Student Ability 1. This _____(product/activity) is mostly suitable for students of high ability.
2. This _____(product/activity) is mostly suitable for students of high ability in grade _____.
3. This _____(product/activity) is mostly suitable for students of moderate ability in grade __.
4. This _____(product/activity) is mostly suitable for students of low ability in grade _____.
5. This _____(product/activity) is suitable for students with learning disabilities.
6. This _____(product/activity) will have to be modified in order to meet the needs of 
     students with learning disabilities.
7. A modification would be the following: (Please explain the modification and the 
     student disability that would require the modification.)

For Teachers Responsible For Science/Technology Content:

1. I have a thorough foundation to teach the science content associated with the __________(product/activity).
2. The science content associated with the __________(product/activity) is aligned with my state’s curriculum standards.
3. The science content associated with the __________(product/activity) is aligned with my school district’s curriculum standards.
4. I have the classroom materials necessary to teach the science content associated with the __________(product/activity).
5. I have the computer technology necessary to teach the science content associated with the __________(product/activity).
6. I have a thorough foundation to teach the technology content associated with the __________(product/activity).
7. The technology content associated with the __________(product/activity) is aligned with my state’s curriculum standards.
8. The technology content associated with the __________(product/activity) is aligned with my school district’s curriculum standards.
9. I have the classroom materials necessary to teach the technology content associated with the __________(product/activity).
10. I have the computer technology necessary to teach the technology content associated with the __________(product/activity).

Content: For Teachers Responsible for Mathematics Content:

1. I have a thorough foundation to teach the mathematics content associated with the __________(product/activity).
2. The mathematics content associated with the __________(product/activity) is aligned with my state’s curriculum standards.
3. The mathematics content associated with the __________(product/activity) is aligned with my school district’s curriculum standards.
4. I have the classroom materials necessary to teach the mathematics content associated with the __________(product/activity).
5. I have the computer technology necessary to teach the mathematics content associated with the __________(product/activity).

Sustainability 

1. Is the ________(product/activity) something you will continue to use?
2. Is the ______(product/activity) easy to use?
3. Can you put the _____(product/activity) into an existing program?
4. Does your course run more smoothly because of this tool?

Institutional Support for Using the _____(Product/Activity)

1. I have easy access to computers for the use of this ____(product/activity).
2. I have sufficient access to computers for the use of this ____(product/activity).
3. I have sufficient access to the Internet for the use of this____(product/activity).
4. My colleagues support my use of this ____(product/activity).
5. My administrators support my use of this ____(product/activity).
6. I have sufficient support from the Computer Lab in order to use this ____(product/activity).

(Note: The item bank contains hundreds of questions, based on the NASA Operating Principles of REASoN grants, from which a PI could choose to construct surveys for several groups of stakeholders, for instance, K-12 and Higher Education Teachers, Students, Administrators.) 


The Logic Model:
Overall Scheme to Determine Short-term and Long-term Outcomes