I guess that I was asked to be a mentor because of my advanced maturity (I will let you figure out which meaning of “maturity” I mean ☺). I have gone to every RE conference since the first in 1993, and I have been on every RE conference program committee (PC) since the 1994 conference. So I have written and read all kinds of good and bad papers on all kinds of RE-related topics. I am here to help you by providing PC-like feedback on your potential research-paper submissions before you have to submit it to the actual PC. This feedback is to allow you to improve your paper in order to increase its chances of being accepted for publication and presentation at RE’16 in Beijing, PRC.
I now quote, with a few modifications for clarity, some FAQs about submitting papers to conferences and about good and bad research papers that were offered by Don Gause, the previous person in this role of RE Mentor for research papers, for RE’13.
“The following FAQs are literally based on the most frequent shortcomings I have seen in first drafts submitted by authors getting started in their publication of research papers1.
FAQs that never get asked but should
Are reviewers always looking for the worst in submitted research papers?
Believe it or not, many reviewers are looking for the best in papers, even in the “bad” papers. No one wants to abandon a good idea. True, reviewers have the responsibility of maintaining the reputation of the event or journal but they also have the responsibility of making sure that papers are presented and published2. Otherwise, the reviewers would be out of work, as would conference and journal publishers and sponsors not to mention the loss of great ideas and the sharing of knowledge. Writers need to keep in mind that the reviewer can be the author’s best friend in strengthening even the best of papers.
What makes for a bad research paper?
Obviously, bad research [leads to a bad research paper,] but comments on this come later. But, what many early researchers don’t realize is that [a badly written report about] even good research … can mislead a reviewer into a rejection bias …. A common example would be a disorganized paper that looks like it has been written by several authors (and probably has been), no one of which has read the complete paper. Making matters worse, I have reviewed papers in which the reader could easily be convinced that one or more listed authors had not even seen the paper, let alone read it thoroughly before submission. It is hard for a reviewer to feel good about a paper when the reviewer gets the idea that he or she has put more time into understanding the paper than the authors have spent in writing it.
What makes for a good research paper in the first submission?
In Form: The paper is well-written and understandably organized. The figures, tables, and graphs are located appropriately throughout the paper3 and of size to be readable without a magnifying glass.
In Substance4: This is where I can say a few words about good research, or at least abeyance to well accepted research methodologies and documentation. The good research paper addresses each of the [principal] issues:
- Novelty – Does the paper contain new and original ideas and information?
- Relevance – Is the information not only new but is it relevant to an existing problem or opportunity? Do the author’s findings provide good answers to the “so what” question?
- Timeliness – Is the information provided when it is needed or when we can most benefit from it?
- Credibility – Is the information intuitively satisfying and, if not are there plausible explanations? Is the information suitably backed and, when based on experimental results, does the documentation pass the reproducibility test?
- Complete – This is as much in reference to the documentation of the research results as to the resulting information and knowledge. Are the information and its documentation complete to the degree that the work and results can be repeated? Has the author sufficiently described and rationalized the research methodology?
- Qualified – This again refers to the form in which the newly developed information is presented. Has the scope of applicability been effectively defined and, have refutations of the work been offered by the researchers? This is sometime referred to the threats to validity of the presentation and is an effective way of describing the region of admissibility of the study.
I hope you find these very brief comments helpful in getting started. I … wish you the best of luck in the documentation of your research findings.
Don Gause, November 2012
- 1 Glass, Robert L.,Ramesh, V., and Vessey, Iris (2004) “An Analysis of Research in Computing Disciplines”, Communications ACM, 47(6), pp.89-94. June 2004.
- 2 You can read all about the role of the reviewer here: Smith, Alan Jay. (1990) “The Task of the Referee”, IEEE Computer, 23(4), pp.65-71. April 1990.
- 3 The appearance of a figure, table, or graph is normally placed immediately after the paragraph of its first reference in the text.
- 4 Gause D.C. (2004) “Comparative Evaluation in Requirements Engineering: Is it too much too soon, too little to late, or just the right amount at just the right time?“, Keynote Address. Proceedings of the IEEE Conference on Comparative Evaluation in Requirements Engineering. Kyoto, Japan, pp.3-16. September 2004.”
In any case, I am here to offer my advice and opinion at any stage of the writing process, e.g., an early outline, a first draft, a polished draft. I will read what you send as one member of the PC (which I am) would read it.
Please do be aware that I will be as honest, critical, and constructive as I can in assessing the acceptability of your paper. However, I cannot guarantee that if I like your paper and consider it acceptable, it will be accepted. I cannot even guarantee that at least one reviewer will find your paper acceptable. For one thing, about 1/2 of the papers that I have submitted to RE have not be accepted (sigh). For another thing, I will have to declare a conflict of interest that will prevent me from being one of the reviewers of your paper after you submit it.
I look forward to receiving your paper drafts.