Education Evaluation Report 2008 - 2009
Executive Summary
The 2008-2009 VSA Arts of Massachusetts (VSAM) educational programming served students who are currently underserved in the arts, cultivated 21st Century Skills for students, fostered collaborative relationships within Artist Residency partnerships, and educated professional learning communities to use the arts as a tool for Universal Design for Learning. VSAM identified the need to increase parental involvement opportunities during Artist Residencies. Doing so will ensure that students are receiving meaningful, inclusive, high quality arts learning opportunities.
Data was collected
from 28 of our 2008-2009 Artist Residency Programs and 3 of our professional
development opportunities. We collected data using Planning Forms, Curriculum
Maps, a collection of annotated student work, conversations from the
Community of Practice website, and participant surveys from our Artist
Residency and Professional Development programs. Understanding the interrelationship
between the school, classroom, student, and professional development
communities provided a rich source of data integral to our evaluation
methods.
Evaluation Methods
VSAM used the 2008-2009
data from Artist Residencies and Professional Development opportunities
to reflect upon three of our goals for this year:
1. Facilitate a partnership
between school administrators and VSAM education staff so that a positive impact is made
on school communities.
2. Inform educators
how the arts can provide multiple options for students and teachers
to represent, express, and engage with academic
content in order to create a positive impact on classroom communities.
3. Ensure that individual
students are provided with a meaningful, inclusive, and high quality
arts experience as a result of participating in one of our Artist Residency
programs.
This evaluation report
describes the impact that VSAM has had on the school communities, classrooms,
individual students, and professional learning communities who have
received our programming. The data for this report was collected from
32 teacher post-residency surveys, 8 artist post-residency surveys,
and 8 principal surveys that represent 19 of our 22 schools. We also
gathered data from the Planning Forms, Curriculum Maps, and student
work from all 22 partner schools. Qualitative data was collected from
conversations documented on the VSAM Communities of Practice website
as well as informal conversations with participants.
VSAM Education Programs
provided arts learning opportunities for students in Pre-Kindergarten
through 12th grade across the state of Massachusetts. Elementary
aged students were the recipients of a majority of our programming with
High School aged students representing the second greatest population
who received our programming. At this time Pre-K and Middle School populations
received the least amount of our programming in comparison to other
populations.
Impact on School Communities
This year the Boston
Foundation and EdVestors supported a study, The Arts Advantage, that
found "schools with larger percentages of Special Education students
among their student body reported lower percentages of students receiving
arts education. Similarly, schools with large percentages of Special
Education students reported offering fewer art disciplines than those
schools with smaller percentages of Special Education students." This
finding reinforces the mission of VSAM Education Programs: To provide
equitable access to high quality arts education programs for students
with and without disabilities. In addition, the finding identifies an
immediate need for more resources to be invested in high quality arts
programming specifically designed to benefit all students -- including
students who receive Special Education services in Boston. The
graph below shows the percentage of students receiving special education
services in each of our partner schools. Many of them exceed the district
average.
A study included in Critical Links, noted that, "arts infused classrooms and arts instruction positively affect the learning, engagement and performance of special needs students and students from low-income backgrounds." (Catterall &Waldorf, 2002). 68% of the schools we partner with qualify for Title I funding.+
The findings from
the Arts Advantage Report and Critical Links study combined with the
demographic information from our partner schools confirm the need for
VSAM to continue to:
1. Facilitate a partnership
between school administrators and VSAM education staff so that a positive
impact is made on school communities.
VSAM hopes to improve
the access of underserved populations to quality arts learning opportunities
through our education programs.
School Community
Outcomes
Information from
eight principal surveys helped us to understand the impact of the artist
residency programs within the whole school community. The comments from
principals suggest that the artists have helped to support individual
school missions, Massachusetts Curriculum Frameworks, inclusive strategies,
and arts integration practices. The graph below illustrates the feedback
from principals.
One area of need
that VSAM has identified is parent engagement. We have outlined a strategy
to address this need in the Learnings and Next Steps section of the
report.
Impact on Classroom Communities
VSAM Education staff
work to ensure that a strong collaboration is formed between each teacher
and VSAM teaching artist thus creating a rich classroom community. Teachers
have content objectives and specific learning goals for individual students,
while artists have aesthetic objectives and specific arts learning goals
for individual students. Each collaborator contributes unique skills
to the classroom community that ultimately impacts individual students
through the learning opportunities that are designed.
Planning Forms, Curriculum
Maps, and documentation of student work were analyzed so that teacher
and artist objectives were aligned and both areas of expertise were
reflected in the curriculum. The formation of this strong collaboration
was integral to achieving our second goal:
2. Inform educators how
the arts can provide multiple options for
students and teachers to represent,
express, and engage with academic content in order to create a positive
impact on classroom communities.
We analyzed data
from teacher post-residency surveys, artist post-residency surveys,
and post-Contours of Inclusion conference surveys (a day long professional
development conference held this year) in order to measure our success
at forming strong collaborations.*
Learning Opportunities in the Classroom
The teacher and artist
in each residency collaborated towards a curriculum goal that could
be taught through the arts. We asked teachers if they felt that their
classroom goals were incorporated into the artists' lessons. 93.8%
of teachers responded "Strongly Agree" (56.3%) or "Agree" (37.5%)
that their classroom goals were incorporated into the artist's activities.
We also asked teachers, "What was
the learning goal you and the artist collaborated towards during this
residency?" to gain an idea of the curriculum connections made in
each of the residency programs. In order to see how the curriculum connections
supported the development of 21st Century Skills we coded
the teacher responses according to 21st Century Themes.
The 21st Century Skills
Framework, developed by the Partnership for 21st Century
Skills, provides VSAM with a system of language that is meaningful to
the mission of our organization and reflects the current dialect that
educational policy makers use when speaking about the future of education
in the Commonwealth. Thinking about our work within this framework provides
VSAM with deeper insight into the type of work we do and how it benefits
those that we serve.
The graph below illustrates the 21st
Century Themes that our 2008-2009 Artist Residencies supported.
Below is an example of how responses were coded:
"[Students will explore] with the medium of clay to make a covered box using geometric shapes." (Mathematics)
"Students will identify traits of a leader (through cartooning)." (Government and Civics)
"Create a story - write it and draw it and tell it in a sequential
order - collaboratively." (English, Reading, Language Arts).
Teacher Learning
To evaluate the impact
of the Artists Residency on the teachers' learning we asked if they
agreed with this statement, "The Artist Residency Program gave me
the opportunity to learn strategies to integrate the arts into my curriculum."
90.7% of teachers responded "Strongly Agree" (43.8%) or "Agree"
(46.9%).
Such opportunities
are documented by the Planning Forms (Appendix A) and Curriculum Maps
(Appendix A) that teachers and teaching artists completed during each
residency.
In order to effectively
complete each form the VSAM Teaching Artist and Teacher must collaborate
and understand each other's content specific learning goals. This
collaboration directly impacts the classroom community by setting the
tone for a rich, high quality, and purposeful arts learning opportunity.
(A completed version of each of these tools can be found in Appendix
A.)
We also wanted to understand what
the teachers have learned from the collaboration. To measure this we
asked, "How did the collaboration with the artist impact you as a
teacher?" Below is a sample from the responses we collected:
"Collaboration with the
artists allowed me to think 'outside the box' and be more creative
in adapting activities (especially integrating second grade student/seeing
them work together."
"[The teaching artist] showed
me creative ways to develop listening skills - connected gross motor development and listening."
"It encourages me to attempt
projects that I may not have tried - thinking of ways to adapt them so that the students can participate
more fully."
The graph illustrates all 32
coded teacher responses taken from the teacher Post-Residency Survey.
VSAM Artist Learning
To identify the impact
of the teaching artist on opportunities for learning we asked artists'
if their residency was effective at providing, "support to help classroom
teachers fulfill Massachusetts Curriculum Frameworks," 87.5% reported
"Very Effective" (50%) or "Effective" (37.5%).
This data was further
supported by the planning materials we received where artists have included
the corresponding curriculum frameworks. This evidence shows a
high level of collaboration between the partners where both partners
identified the points of connection between their learning objectives
that would support greater outcomes for the student.
We also asked VSAM
teaching artists to reflect upon their residency experience and "describe
how you and the classroom teacher(s) collaborated:" An example of
these responses is listed below:
"The lead teacher and I had a detailed planning meeting discussing the goals and students' needs, in order to fashion the projects for the residency. Also, I guided [the classroom teaching team] through an art making experience in order to prepare them to be effective models for the students when we would teach the students."
(Curriculum
Development; Modeling)
From the responses we identified six major areas of collaboration between the VSAM
teaching artist and
the teacher. The six categories were Documentation of Student Work,
Student Motivation, Concept Development, Classroom Management, Modeling,
and Curriculum Development.
The majority of our
Teaching Artists identified Curriculum Development and Modeling as areas
of collaboration. From the data we learned that teaching artists did
not identify Documentation of Student Work as a major area of collaboration.
In the Learnings and Next Steps section of this report you will find
our strategy to increase collaborative documentation.
Impact on Students
The curriculum that
each Teacher and VSAM Teaching Artist produce together directly increases
the students' opportunities for arts learning. The Planning Form and
Curriculum Map provide us with documentation to understand the opportunities
for student learning. The videos, audio recordings, and visual samples
of student work provide us with documentation to understand the student
outcomes. Both sets of data in addition to data collected from
informal teacher and artist interviews allows us to develop a comprehensive
understanding of how the VSAM Education Programs directly impact the
opportunities for student learning. This method supports our third goal
to:
3. Ensure that individual students
are provided with a meaningful, inclusive, and high
quality arts experience as a result of participating in one of our Artist Residency programs.
Student Learning Outcomes
To understand student
outcomes from a holistic perspective we asked teachers, "what are
the most valuable things (skills, processes, concepts, ideas, etc) your
students learned from engaging in the arts?" To synthesize the skills
and processes that teachers reported in the surveys we decided to code
the responses according to 21st Century Skills. The 21st
Century Skills Framework also allows us to articulate the outcomes of
our educational programming in the language that is meaningful to a
broader audience. Teacher responses to this prompt included:
"The most
valuable things that the students learned included working together
as an integrated classroom community"
"Students
learned to express themselves through their work. This carried over
into their writer's workshop and science books (throughout curriculum)."
The results of this
coding are illustrated in the graph below. Most of the skills
taught during the residencies are included in the Learning and Innovation
Skills area. In addition, the educational objectives of our residency
programs align with all three major categories of 21st Century
Skills. Currently, our residency programs do not report Media Literacy
or Information Communications, and Technology (ICT) Literacy skills
as primary outcomes.
Outcomes of student
learning are also documented in student work by VSAM Education staff.
Once we received all of the necessary data collection tools and documentation
of student work we identified the learning objectives stated on the
Curriculum Map and Planning Form within the student work. This method
of annotation allows us to process the data in the context of student
outcomes as well as providing us with a tool to communicate our understanding
of student outcomes with teachers and VSAM teaching artists.
Below is an
example of an annotated piece of student work from an Artist Residency
at the William Henderson School (formally known as the O'Hearn School)
with VSAM teaching artist Tim Archibald. Each annotation connects to
a stated learning objective in the planning materials submitted for
this residency.
Impact on Professional
Learning Communities
Through the VSAM
Partnership for Arts-based Instruction and Inclusion, VSAM creates events
where professionals have the opportunity to join in a dialogue about
the arts and inclusion in schools, classrooms, and cultural venues.
This year VSAM collaborated with university partners to provide professional
development for education professionals across Massachusetts.
A brief description of each event can be found below.
VSA arts of
Massachusetts "Contours of Inclusion: The Intersection Between
the Arts and Universal Design for Learning (UDL)"
Conference at UMass
Boston: VSAM organized a conference in partnership with UMass Boston
that explored the intersection between arts learning and UDL. Presenters
from UMass, MoMA (NY), VSA arts, the Center for Applied Special
Technologies (CAST), and Lesley University were invited to lead workshops.
The 92 conference participants included teaching artists, teachers (general
and special education), principals, district level administrators, and
program administrators from cultural venues.
We asked participants
to reflect on their conference experience. 94.9% rated the overall conference
at a five (59%) or four (35.9%) on a five-point scale. A sample of reflections
are below:
"I thought this was a GREAT
conference. The mix of people was great!! I liked that it wasn't just
artists, higher education, teachers or non-profits. The combination
made for some great dialogue and potential for advocacy."
"Attending this conference
has been an invaluable tool for me and I gained a great deal of information
and insight as well as meeting some very wonderful people. Thank you!
I am telling all my friends and co-workers about the workshops."
"I enjoyed the fact there
were discussion groups in which people from different workshops pooled
their experiences both from the workshops they experienced and their
real-life experiences."
We asked participants
to describe the outcomes of this professional development experience.
The majority of participants reported learning about new strategies
and tools as among the greatest outcomes. The graph below illustrates
the range of outcomes from the conference.
"What do we learn from participating in the arts" Exhibit and Forum
VSAM program managers
and teaching artists explored the learning evident in student work with
a group of participants at the Gutman Library at Harvard Graduate School
of Education (HGSE). The work was part of an exhibit with the same name
which was displayed at Harvard Graduate School of Education, UMass Boston,
Lesley University and the VSAM Gallery. Work from 175 students across
Massachusetts was represented in video, photos, and visual artwork.
42 participants joined VSAM at the forum, where Steve Seidel, Director
of the Arts in Education Program at HGSE, introduced the process of
the "Collaborative Assessment Conference" to assess
student learning. VSAM Education Program Managers introduced two
pieces of student work, which was used to define learning appropriate
for documenting in MCAS-Alt Portfolios. Dan Wiener, from the Department
of Education, joined in the forum and served as an expert on MCAS-Alt portfolio documentation.
While we did not
have a formal evaluation survey for this session we did interview participants
about their experience. Several of the artists reported this being the
first time they dedicated this level of time in order to understand
what students were learning during art classes. One teacher said that
she learned more about her students and their families as a result of
collecting student reflections and sharing their work. She plans to
continue this method of documenting work with student reflections in
upcoming programs.
Cultivating the Field: Teaching Artist Institute
VSA arts, VSAM, Jacobs
Pillow, and Lesley University presented 3 day-long institutes about
assessment and evaluation in the arts. 98 teaching artists participated
in the institute. VSAM presented workshops on curriculum mapping,
communities of practice, and documenting student work. Teaching artists
at the institute utilized skills to document student learning by participating
in the "What do we learn from participating in the arts?" exhibition
by documenting student work to include in the show.
Learnings and Next Steps
School Communities
Convenings held by
the Linde Family Foundation and Mott Philanthropic allow us the rare
opportunity to interact with other organizations doing similar work
and learn from strategies that are effective for them and brainstorm
ideas together. We also benefit from the expertise of professionals
such as Candelaria Silva-Collins, who is currently consulting with us
to develop a parent engagement strategy. Next year we plan to initiate
a new approach to help support schools to increase parent engagement.
We will identify a parent liaison to the program and include parent
in a culminating event at each school. We will include this work with
parents in our evaluations so that we can learn what methods work best
to engage parents in the lives of their child's school though the
arts.
Classroom Communities
It is clear that
each school community and collaboration with a VSAM artist is unique
and requires specific supports. This year we hope to provide access
to professional development that is specific to each community we serve.
To learn about those needs we asked teachers and artists to tell us
the professional development topics that would interest them.
Teacher suggestions
included: linking multi-sensory strategies to mathematics, Arts with
students with cognitive disabilities, preschool music and movement in
content areas, Art in MCAS-Alt, grant writing, and Arts Integration
strategies.
Artist suggestions
include: best practices in working with a wide range of challenges and
disabilities, assessing student learning, collaboration with teachers,
effects of music and dance on the neurological system.
We will offer incentives
to schools for returning surveys in order to improve the evaluation
of our programs and ensure a higher number of surveys are returned.
Students
Documentation of
student work is crucial to support student learning. Artists use this
documentation to reflect on student understandings, building next steps
and inclusive strategies, and communicating the benefits of arts processes
with teachers.
In artist surveys
the artists state a need for more on-site support through VSAM staff
visits and support with documentation of the residencies. Next year
we will schedule VSAM staff visits at the beginning of residencies in
order to ensure adequate support for each artist. We are also currently
working on raising funds to support capacity building initiatives in
the education program.
Professional Learning Communities
Through surveys and
informal conversations we have found that gathering teachers (general
and special education), artists, administrators, cultural organization
staff, university professionals and district level administrators is
a crucial aspect to learn about and advocate for arts-based instruction
and inclusion. We will continue to build this network of professional
communities through the Partnership for Arts-based Instruction and Inclusion
next year.
Special Thanks
VSAM would like to
thank the funders that made this work possible:
Boston Public Schools
BPS Arts Expansion Fund at EdVestors
CAPS Collaborative
The Jessie B. Cox Charitable Trust at the Boston Foundation
CVS Caremark Foundation
EdVestors
Gardner Public Schools
Greater Worcester Community Foundation
Linde Family Foundation
Massachusetts Department of Elementary and Secondary Education
Ohrenberger Elementary School
Pittsfield Public Schools
Samantha's Harvest
Shippy Foundation
Worcester Educational
Development
We would also like
to extend our thanks to our university partners: Lesley University,
University of Massachusetts Boston, and Harvard Graduate School of Education
all of whom have supported the development of our work.
Sources
Hindley, B. (Ed.). (2009).
The arts advantage: Expanding arts education in the Boston public schools.
Boston MA: The Boston Foundation.
Catterall, J.S. (2002), Involvement
in the Arts and Success in Secondary School. In R. Deasy (Ed.), Critical
Links: Learning in the Arts and Student Academic and Social Development.
(pp.68 - 69). Washington, D.C.: Arts Education Partnership.
Partnership for 21st
Century Skills (2008). Transition brief policy recommendations on preparing
Americans for the global skills race.
Tucson, Arizona: Partnership for 21st Century Skills.
U.S. Department of Education
(2009) Title I program description. Retrieved from http://www.ed.gov/ August 7, 2009.
This report was written by Kati Blair, Nicole Agois Hurel and Elena Figueroa.