Choosing the Right Data Collection Method: Understanding the Difference, Study notes of Logistics

Download Choosing the Right Data Collection Method: Understanding the Difference and more Study notes Logistics in PDF only on Docsity!

Quantitative Data Collection Methods

-CASA Session 5-

Session 5 Overview

• Session 5 learning outcomes
• Overview of Qualitative and Quantitative Paradigms
• Quantitative Research & Data Collection Methods
  • Observation
  • Rubrics
  • Existing data
• BREAK
• Surveys
• Session Assessment/Mid-course check-in
• For Next Session…

Assessment Plan Peer Feedback

At this point in the curriculum, you should have the following assessment plan sections fully drafted OR partially outlined/notated:

Assessment Purpose  Assessment Plan Design  Background and Purpose  Assessment Question(s)  Context and Stakeholders

Planned Analysis and Reporting  Planned Analysis  Plan for Reporting  Implications for Improvement

Methods and Implementation  Assessment Method  Resources  Implementation and Design

1) Switch Assessment Plans with a partner

1. Rate each section using the assessment plan rubric.
2. Provide written notes/suggestions in space provided.

THEN…

2) Return Assessment Plans and Feedback Sheet

1. Discuss your ratings with your partner.
2. Identify ways each of you can improve these sections.
3. Authors make notes for themselves regarding next steps.

Where we are, where we are going…

Confidence Barometer

After today, you should be able to:

• Understand the difference between

quantitative and qualitative

approaches

• Identify different possible

quantitative assessment methods

• Understand the logistics involved in

various quantitative assessment

methods

• Select (if applicable) a quantitative

data collection method that

appropriately addresses your

assessment question(s)

Today’s Learning Outcomes

0

3

6

0

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

"I am able to look at my assessment questions and choose an appropriate data collection method."

Strongly Disagree Disagree Agree Strongly Agree

Continuum of methods

More

Qualitative

More

Quantitative

Naturalistic Observation

Open-Ended Survey Questions

Semi-Structured Interviews

Measurements

Close-Ended Survey Questions

How Do I Approach assessment?

Qualitative Methods Quantitative Methods

Sources of Data

Interviews, focus groups, observations, etc. Surveys, counts, pre-existing data, student records

Focus

On meanings and details of experiences (Depth) On generalizable experiences that represents a population (Breadth)

What you Report

Themes, Quotes Numbers, charts, graphs, trends (longitudinal data)

Characteristics

Open-ended questions Close-ended questions, Measurement of specific variables

Fewer participants (resource intensive per participant)

Large number of respondents (with fewer resources, sometimes)

Distance to research: Closer to context Distance to research: further from context

Observation

What is it: Researcher observes ongoing behavior of a person, group, or phenomenon.

• Naturalistic/Non-participant Observation: Researcher is not a part of the action
  • Pro: Can measure what behavior is really like
  • Con: Don’t know reason behind behaviors
• Participant Observation: Researcher becomes a member of the group, a part of the action.
  • Pro: Allows access to information (the why behind behavior)
  • Con: Can lose objectivity, participation influences what you see and measure.

Naturalistic (^) Participant

• How to do Observation (quantitatively):
  • Keep a list of who enters/uses your office or attends a program
    • Uses:
      • Schedule of when the most frequent use occurs
      • % of attendance by students of different characteristics (depending on what is collected)
  • Observe how often you have to replace an item, refill a jar, etc.
    • Uses:
      • Learn about student behavior, needs
      • Help determine questions to ask
  • Note which entrance students use, or the way they access the services (e.g. email, online, phone) - Uses: - Set up information boards, office/desk locations, etc

Observation (cont.)

• How to Use Rubrics (quantitatively)
  • Determine which dimensions were well executed by the individual or group and which dimensions are in need of improvement - Student Organization Event Plans
  • Results can be compiled and disseminated to the individual or for department review - Assessment Curricula “Assessment Plan” Rubric
  • Numeric results can be aggregated to understand participant outcomes, and to assess program effectiveness.

Rubrics (cont.)

Existing data

• Data Audit
• Departmental Data
  • Usage
  • Program Evaluations/Surveys
  • Annual Evaluations
  • Program Review
• Institutional Data
  • Student Surveys
  • Demographic Information (Registrar)

Existing Data

• Session 5 learning outcomes
• Overview of Qualitative and

Quantitative Paradigms

• Quantitative Research & Data

Collection Methods

• Observation
• Rubrics
• Existing data
• BREAK
• Surveys
• Session Assessment/Mid-course

check-in

• For Next Session…

• Any gathering of information from an individual or group of individuals using a pre-specified set of questions and responses
• Formal:
  • Digital survey administered after an event
  • Pen and paper comment cards at a front desk
• Informal:
  • Hands raised in response to questions at an event
  • Index cards collected after an event

Surveys

• Be SPECIFIC
• Focus on ONE concept at a time
• Use simple language
  • General audience, like the newspaper (not NY Times)
  • DUOA (don’t use obscure acronyms!)
• No leading questions
• Ask questions that apply to everyone, or include skip patterns
• Avoid language that can be interpreted in different ways
• Be mindful of response options (mutually exclusive, exhaustive)

Articulating Clear Survey

Questions

• Assessment Question: How satisfied are UCLA undergraduates with the extent that they feel free to express their political beliefs on campus? - Survey questions: - “I feel free to express my political beliefs on campus.” (agreement scale) - “There are safe spaces on campus to have political debate.” (agreement scale) - “How often do you express your political beliefs on campus?” (time scale)

Turning Assessment Questions

into Survey Questions