Left Image
Main Image

Guide to Reviewing CHI Papers and Notes – What has Changed in 2009

Introduction

The CHI review process has changed this year due to the incorporation of review sub-committees and of contribution types. It is NOT business as usual. Here are the take-away messages.

  1. Your primary criterion for judging a paper is: Does this submission provide a strong contribution to the field of HCI?
  2. Authors indicate a contribution type. As part of judging whether the paper makes a strong contribution, consider the questions about contributions as specified in the description of each contribution type
  3. Authors submit their papers to subcommittees. However, you should not judge a paper by how well it fits the subcommittee theme.
  4. Reviewers rate the paper using a 9-point ranking scale; your written appraisal must support your numeric ranking.

What is the Same

You still have to render judgment on the paper’s acceptability. The main question you should always consider in rendering this judgment is: Does this submission provide a strong contribution to the field of HCI?

The First Difference: Contribution Types

There has been increasing concern over the years that CHI reviewers favor certain styles of papers over others; the perception is that only papers following a certain ‘formula’ can get accepted. To discourage this narrowing of what makes an acceptable paper, each author can now specify a contribution type for their submission (see contribution page). The description of each contribution type includes a set of questions that you, as a reviewer, should consider when evaluating the contribution of the paper. What is important about these questions is that they may push you to think about the contribution in ways that differ from what you may perceive as the CHI formula paper.

However, these questions are not a checklist. The submission(s) you are handling may not cleanly fall within a single contribution type, and the paper should not be penalized for that. When this happens, your primary criteria for judging a paper should still be: Does this submission provide a strong contribution to the field of HCI?

This is important. The contribution type and its associated questions are a guide to assist your thinking; they are not a strict criteria by which you should measure acceptance. They are there to help you think more liberally about the paper’s contribution. If, in your opinion, the paper still makes a contribution outside of the specific contribution type questions, then judge it accordingly.

What you will see on the review form is:

  • A summary of the contribution type questions (i.e., review criteria), followed by a Contribution Type Specific Rating. By filling in your score, you are judging the paper according to its contribution type.
  • Immediately following this is an Overall Rating. By filling this in, you are judging the overall contribution of the paper. While we suggest you use the contribution type as a starting point for appraising the contribution, the paper does not need to exactly match the contribution type criteria
  • The various other boxes ask you to provided details. Again, while we suggest you use the contribution type as a starting point for appraising the contribution, we stress that the paper does not need to exactly match the contribution type criteria.

The Second Difference: Subcommittees

To improve the reviewing process, authors are now asked to submit their papers to sub-committees, where each sub-committee is responsible for various topic areas typically seen at CHI (see the subcommittee page for details). Each sub-committee will comprise of a sub-chair and associate chairs who are knowledgeable in these topics. The idea is that, as specialists in the theme areas, they should be able to find good referees (such as you) for each submission, and that as specialists they should be able to better handle the meta-review process.

However, as a reviewer, you should NOT judge the paper by how well it fits the subcommittee theme(s). Many papers will not cleanly fit into a particularly sub-committee for a variety of reasons, and we do not want to penalize authors for this. Remember, the subcommittee organization is there only to try to improve reviewer matches and to better handle the volume of submissions. If you have a paper that does not fit the subcommittee theme, evaluate it as best you can. Any topic is valid, as long as it fits within the interests of a reasonable fraction of the overall CHI audience.

The Third Difference: 9 Point Scales

The numeric ratings on the review form are now on a 9 point scale, from half steps starting at 1.0 (the lowest ratings) to 5.0 (the highest). Be as accurate with your ratings as possible. As in previous years, it is your written appraisal that is crucial. Make sure your review is balanced, and that its details  reflect the numeric rating. The appraisal of a paper should always indicate why the paper deserves that rating. Your review may be discounted if, for example, you numerically rate a paper highly but only indicate its flaws. As well, be polite to authors. Even if you rate a paper poorly, you can critique it in a positive voice. As part of polite reviewing practice, you should always state what is good about a paper first, followed by your criticisms. If possible, you should offer suggestions for improvement along with your criticism.

The Caveat

Contribution types are just a mechanism to help you, as a reviewer, judge a paper for what it is rather than how well it fits a narrow formula.   However, it is all too easy to fall back on prior expectations of what makes a ‘good’ CHI paper. As an organization, we are implementing contribution types to try to reshape how we as a community judge our papers. But ultimately, it is up to you. As a reviewer, think long and hard about each paper before rendering your judgment.

References

The debate about what makes a good CHI paper has been ongoing for several years, and the contribution types are an attempt to address this debate. If you are interested, the papers below touch upon this debate and contains references to additional papers that concern it.

  • Greenberg, S. and Buxton, B. 2008. Usability evaluation considered harmful (some of the time). In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems. CHI '08. ACM, 111-120. DOI=http://doi.acm.org/10.1145/1357054.1357074
  • Olsen, D. R. 2007. Evaluating user interface systems research. In Proceedings of the 20th Annual ACM Symposium on User interface Software and Technology. UIST '07. ACM, 251-258. DOI=http://doi.acm.org/10.1145/1294211.1294256
  • Dourish, P. 2006. Implications for design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI '06, ACM, 541-550. DOI=http://doi.acm.org/10.1145/1124772.1124855 
  • Newman, W. 1994. A preliminary analysis of the products of HCI research, using pro forma abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating interdependence (Boston, Massachusetts, United States, April 24 - 28, 1994). ACM, New York, NY, 278-284. DOI=http://doi.acm.org/10.1145/191666.191766

Written by Saul Greenberg and Scott Hudson.