Mastering Statistical Claims: Observational Studies and Experiments for the SAT

Example 1: Observational Study with Correlation Claim

A researcher collected data on the number of hours 200 randomly selected high school students spent on social media each day and their GPA. The study found that students who spent more time on social media had lower GPAs on average.

Analysis:

• This is an observational study because the researcher is only collecting data without manipulating variables.
• The claim that can be supported is that there is a correlation between social media use and GPA.
• We cannot conclude that social media use causes lower GPAs because there could be confounding variables (e.g., students who spend more time on social media might also spend less time studying for other reasons).
• The random selection of students is a strength of this study as it helps ensure the sample is representative of the population.

Example 2: Well-Designed Experiment with Causal Claim

Researchers wanted to test if a new teaching method improves math scores. They randomly assigned 300 students from diverse backgrounds to either receive instruction with the new method or the traditional method for one semester. All other factors (teacher quality, classroom environment, etc.) were kept constant. At the end, students taught with the new method scored significantly higher on a standardized test.

Analysis:

• This is an experiment because researchers manipulated the teaching method (independent variable) and measured its effect on test scores (dependent variable).
• The random assignment of students to groups helps ensure that the groups are comparable.
• Controlling other factors reduces the influence of confounding variables.
• The claim that the new teaching method causes improved math scores is supported by this experimental design.
• The diverse background of students helps with generalizability of the results.

Example 3: Observational Study with Inappropriate Causal Claim

A health magazine reported that people who drink wine have lower rates of heart disease than those who don't drink alcohol. The magazine claimed that drinking wine prevents heart disease.

Analysis:

• This is an observational study because it only observes existing behaviors without manipulation.
• The claim that wine prevents heart disease is not justified because correlation doesn't imply causation.
• Potential confounding variables include: wine drinkers might have healthier diets overall, higher incomes allowing better healthcare, or different exercise habits.
• An appropriate conclusion would be that wine consumption is associated with lower rates of heart disease, not that it causes lower rates.

Example 4: Experiment with Design Flaws

A company tested its new energy drink by having 20 employees try it for a week. The employees reported feeling more energetic during the week they consumed the drink. The company claimed their drink increases energy levels.

Analysis:

• This is an experiment because the company introduced the energy drink (treatment) and measured its effects.
• However, there are several design flaws:
  • No control group to compare against
  • Small sample size (only 20 employees)
  • Participants weren't randomly selected or assigned
  • Placebo effect not controlled for (participants might report more energy because they expect the drink to have that effect)
  • Subjective measurement (self-reported feelings rather than objective measures)
• The claim that the drink increases energy levels is not well supported by this flawed experimental design.

Example 5: Observational Study with Appropriate Claims

Researchers surveyed 1,500 randomly selected adults across the country about their sleep habits and stress levels. They found that people who reported sleeping less than 6 hours per night also reported higher levels of stress.

Analysis:

• This is an observational study because researchers are only collecting data without manipulation.
• The large, random sample is a strength of this study.
• The appropriate claim is that there is an association between sleep duration and stress levels.
• The researchers appropriately noted that they cannot determine whether lack of sleep causes stress, stress causes sleep problems, or if both are influenced by other factors.
• They correctly stated: "Our findings suggest that sleep duration and stress levels are related, but further experimental research is needed to establish any causal relationship."

Example 6: Well-Designed Experiment with Appropriate Scope

Researchers tested whether a new medication reduces allergy symptoms. They recruited 500 adults with seasonal allergies and randomly assigned them to receive either the medication or a placebo. Neither the participants nor the researchers evaluating symptoms knew which group received which treatment (double-blind). After 30 days, the medication group showed significantly reduced symptoms compared to the placebo group.

Analysis:

• This is a well-designed experiment with:
  • Random assignment to treatment groups
  • Use of a placebo control group
  • Double-blind procedure to prevent bias
  • Adequate sample size
• The claim that the medication causes a reduction in allergy symptoms is supported.
• The researchers appropriately limited their claims to the population studied (adults with seasonal allergies) and didn't overgeneralize to all allergy types or age groups.
• They correctly noted that while the medication was effective in the short term (30 days), longer studies would be needed to evaluate long-term efficacy and safety.

Question 1:

Researchers conducted a study to determine if drinking coffee affects memory. They recruited 200 college students and asked them about their daily coffee consumption and their performance on their most recent exams. The study found that students who reported drinking 1-2 cups of coffee daily scored higher on exams than those who reported drinking no coffee or more than 3 cups daily.

Based on this study, which of the following conclusions is most justified?

A) Drinking 1-2 cups of coffee daily causes improved exam performance.
B) Students should drink 1-2 cups of coffee daily to improve their exam scores.
C) There is an association between moderate coffee consumption and exam performance among the students studied.
D) Coffee consumption has a U-shaped relationship with cognitive performance in all college students.

Answer: C

Explanation: This is an observational study because researchers simply collected data on existing behaviors without manipulating variables. Observational studies can only establish associations, not causation. Option C correctly identifies that an association was found in the specific population studied without making causal claims (like A and B) or overgeneralizing beyond the study population (like D).

Question 2:

A researcher wants to determine if a new reading program improves reading comprehension in elementary school students. Which of the following study designs would provide the strongest evidence about the effectiveness of the program?

A) Survey teachers about whether they think the program would be effective.
B) Compare test scores of students who chose to use the program with those who did not.
C) Randomly assign classrooms to either use the new program or continue with the current program, then compare reading scores.
D) Observe students using the program and note whether they seem engaged with the material.

Answer: C

Explanation: To determine if the reading program causes improvements in reading comprehension, an experimental design with random assignment is needed. Option C describes such an experiment, where classrooms are randomly assigned to either the treatment (new program) or control (current program) group. This design helps control for confounding variables and can establish causation. Options A, B, and D are either based on opinions, subject to selection bias, or don't measure the outcome of interest (reading comprehension).

Question 1:

A study found that people who eat breakfast regularly have lower rates of obesity than those who skip breakfast. A nutritionist claims that eating breakfast prevents weight gain.

Which of the following best evaluates this claim?

A) The claim is justified because the study shows a clear link between breakfast and obesity rates.
B) The claim is not justified because the study design is not described, so we cannot determine if it was an observational study or an experiment.
C) The claim is not justified because an association between breakfast eating and obesity does not prove that eating breakfast prevents weight gain.
D) The claim is justified because nutritionists are experts in their field and understand the relationship between diet and weight.

Answer: C

Explanation: The study described appears to be observational, as it only notes an association between breakfast habits and obesity rates without mentioning any manipulation of variables. Observational studies can only establish correlation, not causation. Therefore, while we can say breakfast eating is associated with lower obesity rates, we cannot claim it prevents weight gain. There could be confounding variables (e.g., people who eat breakfast might have healthier lifestyles overall).

Question 2:

Researchers want to test if a new teaching method improves student performance. Which of the following study designs would be most appropriate?

A) Survey teachers about which teaching method they think works better.
B) Compare test scores from schools that use different teaching methods.
C) Randomly assign students within the same school to classes using either the new method or the traditional method, then compare their test scores.
D) Have all students try the new method for a month and ask if they prefer it to their previous method.

Answer: C

Explanation: To determine if the new teaching method causes improved performance, an experimental design with random assignment is needed. Option C describes such an experiment. Random assignment helps ensure that the groups are comparable before the intervention, so any differences after the intervention can be attributed to the teaching method. Options A and D rely on subjective opinions rather than performance measures. Option B is an observational study that would be subject to confounding variables (schools using different methods might differ in many other ways).