Writing Reviews & Giving Feedback

Earlier today, a friend asked about how I wrote reviews (i.e. academic feedback for manuscripts). In our text exchanges (because we are still practicing social distancing #Covid19), I realized this might have been the first time I verbalized and realized that I even had a process. So, I wanted to share how I think through my feedback and how I write reviews. 

Note: I debated writing this post, given how journals have their own processes and I don't want to make sweeping generalizations. They also usually/hopefully hold orientations for reviewers and an outlined process for authors), so please know that this info is not the end all, be all. Thank you~ 

REVIEWS & REVIEWERS
In the publication process (see my post about that here), after you submit your manuscript and if the editor decides it should be looked at, the manuscript will "go out for review." Review is feedback. The feedback is usually matched as best as possible, by the editor/editorial board with different people who can give relevant, solid comments about your work. Usually, the matches are aligned with content area, theoretical expertise, and/or methodological/method knowledge. Typically reviews are double-anonymized, meaning that reviewers do not know the identity of the author(s), and the author(s) do not know the identity of the reviewers. Some journals are changing this practice given that, a

Reviewers range from faculty, graduate students, practitioners, policy makers, community organizers, depending on the scope/mission of the journal. Most academic journals have faculty reviewers, but a growing list are also accepting graduate students. Some journals do a "Call for Reviewers" so keep an eye out and/or reach out and ask an editor if they need reviewers. This is voluntary work that is in service to the academy so you can imagine, there's always a shortage. 

PROCESS & PLATFORM
Reviewers will give feedback about your manuscript (that you can see) along with a recommendation to the editor/editorial board about what to do with your paper (that you may or may not be able to see). The recommendations range from accept, accept with minor edits, accept with major edits, revise and resubmit (which means you get another set of reviews), or reject completely. From there, the editor/editorial board compiles the reviews and makes a decision. (For more info about the timeline, please refer to this post.)

Different journals use different platforms for feedback. For example, platforms like NowComment allow for reviewers and authors to interact directly (which also means that they are not anonymous to one another). Other journals will anonymize the PDFs or Word documents for reviewers; if the latter, that might also allow for line-by-line commenting. Many journal editors/editorial boards will ask the reviewers to submit bulleted list of concerns for author(s) to address, along with their recommendation.  

---

PERSONAL PROCESS
Again, this is my personal process, but here's my thought process of how I create the bulleted list and craft the recommendation. I try to keep the numbered list at around 10 or less (so that it's not too overwhelming), and try to include my rationale as the reader, along with suggestions of phrasings or resources, citations, links, etc. I also try to stack the biggest concerns at the top with the following tiers in my head: 

• Tier [A]: glaring concerns (like using a deficit or damaging theoretical framework, not using the appropriate method, the content doesn't fit AT ALL with the mission of the journal, etc.)
• Tier [B]: content issues that could be fixed by reworking the paper (like does the RQ match the arguments they’re making at the end; sweeping generalizations; under-conceptualized idea) ;
• Tier [C] structural and organizational issues (like when the argument makes sense with the research but needs to be introduced in a different order); and
• Tier [D] minor grammatical issues (which I usually just comment directly on the doc, but also remembering that reviewers are not copyeditors)

I use my tiered system to help breakdown what recommendation to make: mostly [A]s as rejections; mostly [B]s as revise and resubmit; mostly [C]s as accept with major changes; mostly [D] as accept with minor changes.

When I review a manuscript, I also don't do it at once and instead, have several different stages:

1. For the first round, I go ham. I mark up and nitpick over everything and comment to myself in real time (compared to reading thru the whole paper and then commenting). I choose real-time commenting because that helps me with questions about structure in case, for example, a section of a paper should be moved up because I had a question that was answered later. 
2. For the second, I group them along the tiers and then make them bucketed. I've definitely opened a document with a bloodbath of red comments and remember how demoralizing it was. So instead, I try to find patterns and point to specific (and or several) examples. That's also when I draft my recommendation. 
3. At the third round, I add in the concrete suggestions and revisit my recommendation, asking myself: am I being reasonable? Am I being too harsh? Are there specific empirical research I can point to for citations or examples? 
4. As my final step, I email it to myself and sleep on it. The next day, I open it from my email, download, and read it. I do this to mimic how people will receive the feedback and check my first instincts on opening the review (and recommendation) and reading it with fresh eyes. If I feel good about it, I'll send it over. 

---

THINGS TO KEEP IN MIND
Here are some things to keep in mind, both as an author who reads the reviews and as a reviewer: 

• Reviewing is voluntary; publication timelines are usually pushed back because reviewers are already strapped for time and can't finish reviews in a timely manner (usually being overworked as they already are). If you can, be a reviewer. If you don't have the time, say no. 
• Reviewers don't always have it right. As a reviewer, I encourage you to communicate with your editor/editorial board if you do not feel you are the right fit to review a manuscript; you are allowed to decline. I've declined when I didn't feel like I had enough content expertise and also when (in an anonymized review), I knew who had written it because it was a project where I had been a consultant.
• It's important to look at the scope of work. For example, if a manuscript is qualitative, one of my review comments is not how this work is not generalizable (unless the manuscript itself has that claim). This is especially important to remember with the relationship between racism, research, and how we center whiteness; Dr. Christine A. Stanley has a great article describing her reviewer process that I'd highly  recommend reading here.

​
And lastly, remember what the review is for. Don't be reviewer 2. This is a running joke in academia. "Reviewer 2" is the reviewer that has an almost infinite set of comments, usually harshly written, and usually outside the scope of the work. Reviewer 2 breaks down authors instead of building them up to be excited about and push the work further. Reviewer 2 makes people feel like crap, makes them put away their work and not look at it for months or even years. Reddit has dedicated thread called AITA (am I the asshole?); don't be the asshole; don't be Reviewer 2.

Reviewers and reviewing is about a conversation, about improving the work, about building, and about producing, refining, and pushing the work and ourselves to do better, be better. That isn't mutually exclusive with being compassionate.