Replication and extension project

In this project, you will work with a partner to reproduce and extend a published research article. The primary goal of this project is to provide the raw materials for a publishable paper, which will be developed and written in the empirical seminar next year. The secondary goals of this project are to 1) give you practice learning new statistical material and computational techniques; 2) give you practice using good software engineering techniques; 3) give you practice doing collaborative research, something you will do throughout your career; 4) show you, in a way that no homework assignment can, the complexity and joy of doing real data analysis. This project not be without pain -- Nightmare after nightmare: students trying to replicate work -- but it will be worth it.

For the project, you must follow the schedule and meet certian technical requirements.

Here are the six steps to the project:

Pick a paper
Get your plan approved
Reproduce the results exactly
Have your work reproduced by a peer and get feedback
Do something new
Have your work reproduced by a peer and get feedback

1) Pick a paper

You should pick a paper in an area that is interesting to you. Some people recommend that the paper be published recently and in a top journal in the field. Also, the paper should use methods related to those that we are going to learn in the course.

Since this is the first time I’ve tried this project, I'm going to require that the paper you select has data already available. I don't want you to spend your time emailing with authors. That is not the kind of learning experience I’m hoping to create with this project. Also, I strongly recommend that you pick a paper where the code is already available. To be clear, you should not use this code. But, having this code available means that you should be able to ultimately reproduce the results in the published paper.

If you are looking for papers to replicate here are some strategies that you could use:

browse examples of papers that have been replicated by students in the past: dataverse for Harvard's Gov2001 course and dataverse for Combridge Replcation Workshop.
start with a large, public data set and look for papers that use it. For example, here are lists of papers that use Add Health and Fragile Families.
browse data archives. Some examples are: dataverse, ISPS data archive, OPR data archive, TESS, and ICPSR.
think about papers that you have read in your classes that you really enjoyed (or thought were wrong).
talk to your advisor, advanced graduate students, and your classmates.

When picking a paper it may be hard for you to tell if the methods will be far beyond what will be covered in class. When in doubt, post a question to Piazza.

2) Get your plan approved

Before getting too far into your project you must get your plan approved. Your project proposal must be written in a structured format and must include each of the following elements:

Citation information for the paper you want to reproduce
Summary of main results of the paper
List of main statistical methods used
List of datasets used
Were these datasets collected via complex sample designs? This will influence the difficulty of the replication.
Summary of data availability and data access plan
Summary of code availability (code availability is not required)
List of ideas for how the paper could be improved or extended (see here for ideas about what you can do new)
Short explanation for why you and your partner picked this paper
pdf copy of the paper

If you do not provide us with all of those elements, it will be impossible for us to give you good advice.

3) Reproduce the results exactly

You should be able to re-create every table and graph in the paper. At this point, however, you should not spend to much time on layout. So, if hte figure has a lengend on top and your legend is on the side that is fine, as long as the results clearly match.

Begin your replication by creating a document with images of each table and each figure. Next, add the parts of the text where the authors describe how the results were generated. Finally, create code that reproduces the results. This document will be highly structured. For example,

Image of Table 1
Text describing how Table 1 was created
Your code to reproduce Table 1

There does not need to be any theory or literature review in this document. It is just about reproducing the results.

4) Have your duplication reproduced by a peer and receive feedback

Someone in the class will clone your repo and reproduce your reproduction. Also, they will send you some helpful feedback.

5) Do something new

You should somehow extend or improve the paper you are reproducing. Here are some examples of how you could do that:

create different visualizations of their data and results.
extend the analysis by using different methods or asking different questions.
re-run the same analysis on completely different data.
take an old paper and update the analysis with newer data.
For other ideas on how to add value, see item 8 in King 2006.

6) Have your work reproduced by a peer and receive feedback

Again, someone in the class will clone your repo and reproduce your work. Also, they will send you some helpful feedback.

Additional information

Schedule

Week 1: look for a paper and a partner
End of Week 2 (Sunday, Feb 15, 2015): post to Piazza a proposal for your project
Week 3: prepare template for reproduction
Week 4: work on reproduction
Week 5: more work on reproduction
Week 6: submit reproduction by Sunday, March 15, 2015 at midnight
SPRING BREAK
Week 7: reproduce reproduction created by a peer, and write a memo offering suggestions for how to improve
Week 8: work on doing something new
Week 9: work on doing something new
Week 10: work on doing something new
Week 11: Submit your replication and extension
Week 12: reproduce reproduction and extension created by a peer, and write a memo offering suggestions for how to improve
Reading period: make improvements based on feedback
Dean’s Date (Tuesday, May 12): submit final product

Tips for providing feedback

Twice in the course you will reproduce a peer's work and provide feedback. Your feedback should be constructive and specific. Here are examples for the kinds of things could be included in the feedback:

Are the variable names clear?
Is the logic of the code clear?
Does the code have lots of copying-and-pasting?
Are there parts of the code that should use loops or functions?
What are things about the code that you particularly like?

Technical requirements

You must work with a partner. Groups of three are also allowed if needed.
You must use RMarkdown or LaTeX + knitr. This ensures that your code and your words are in the same place.
You and your partner(s) must use github for project management and version control.

Presentation

At the end of the semester you will present your results to the class. The goal of your presentation is to teach the class about your replication project and explain what you have learned.

Each group will do a 5 minute presentation; here’s a rough outline (but you can tailor it to your specific project):

summary of the paper you replicated (about 1 minute)
description of your replication and any problems you had (about 1 minute)
description of your extension (about 2 minute)
three biggest things you learned from doing this project (about 1 minute)

You are welcome to use slides.

Using code from others

You are welcome to use code written by other people, including snippets of code you find online and code written by people who are helping you. However, if you are using code from someone else that must be aknowledged. You would not use someone else's sentences in a paper without attribution, and you should not use their code in your code without attribution either. Sometimes there may be a gray area, and if you are in doubt you should attribute.

Inspirations

This project was inspired by similar projects other classes. I would like to thank Gary King, Brandon Stewart, Cristobal Young, Kosuke Imai, and Nicole Janz for their help in shaping this project.