10 (Random) Lessons on Paper Writing

Rodolfo Pellizzoni

Introduction
Unfortunately, I only archive the final version of each paper, so I can not show you a step-by-step paper analysis. Luckily, I have been writing research papers for 5 years now, and I had my fair share of ups and downs. This is a collection of 10 (random?) lessons that I picked up and always keep in mind.

Take them with a grain of salt, I am more interested in starting some discussion than proving facts.

Lesson 1: Be sure to pick the right conference / journal

It is very important to submit to a conference / journal whose topic and interests closely match the contribution of your paper. Even within the same community, different conferences can have different focus!
For example, ECRTS is more oriented to theory, RTAS is more oriented to systems. This also reflects in some way differences between the community in Europe and the US.

Whenever possible, try to adjust the focus of the paper based on the conference.
Even just a slight reword of abstract and introduction can make a big difference (see Lesson 3).

Be careful with special tracks: acceptance ratios are typically not higher than main track, submit only if the paper is very focused.

Example
R. Pellizzoni and M. Caccamo: Adaptive Allocation of Software and Hardware Real-Time Tasks for FPGA-based Embedded Systems. RTAS 2006. Algorithms to allocate relocatable tasks on FPGA, the paper is actually quite theoretical. Initially rejected at RTSS 06 with rather negative scores.
Major concern for most reviewers: there is no bus schedulability analysis.

Accepted at RTAS with pretty good scores.

We modify the paper to include more precise implementation details. Note we did not address bus schedulability at all (we had no bus implementation at the time).

Lesson 2: Always check the program committee

Reviewers are people. They all have their different opinions and points of view. Whenever you want to submit to a conference, be sure to check the program committee first and identify possible reviewers! Things you should be careful about:
Related work. Position on controversial issues (see Lesson 6). What they might be interested in seeing in the paper (see Lesson 8). Etc.

Lesson 3: Section 1 is the most important

The introduction is by far the most important section in the entire paper, especially for conferences. Reviewers are always very busy.
Grad students do a lot of reviews and the return on investments in reviewing is poor. For conference papers, the author can not answer back so reviewers are less accountable.

If a reviewer can reject your paper without reading it all, it saves time! The introduction is the first section they read, so make sure your paper does not get killed in Section 1. 5 years ago I used to write the introduction last. Now it is always the first section I write.

Example
Every single paper of mine received comments that could have been avoided with a better introduction. Instead of individual examples, here are some general suggestions: 1. Explain why the problem you solve is relevant and important.
By far the most important point. Also the hardest to properly articulate. Sometimes examples help, sometimes not. No original contribution -> incremental paper -> autorejection. People are skeptic that you found the holy grail. This is an art that comes with experience.

2. Clearly state your contribution and why it is original.

3. Summarize your main assumptions and limitations.

4. Try to add something to encourage the reviewer to read on.

Lesson 4: Intuitions are more important than proofs

I consider myself a 50/50 person. That means I do 50% theory and 50% implementation. It also means I wrote papers that are full of theorems and lemmas. Proofs are a necessary evils.
We are a community of engineers not mathematicians, so reviewers will not like a paper just because proofs are elegant.

Theorem: reviewers do not like to read proofs.

Following Lesson 3, reviewers are busy. Since furthermore reading proofs takes a lot of time, the theorem follows.

Proofs also take a lot of space.

Lesson 4: Intuitions are more important than proofs

Solution: always, always provide a good intuition for your result before you show the proof.
If the reviewer is happy with the intuition, your paper will probably be ok (but see also Lesson 5). I always review a submission without reading any proof first. If I paper is lacking in contribution, there is no further need to read detailed proof.

You can even move all/some proofs to a technical report if you do not have the space.
There is no choice between putting the intuition and putting the detailed proof in the paper. Intuition always win.

Example
R. Pellizzoni and M. Caccamo, Impact of Peripheral-Processor Interference on WCET Analysis of Real-Time Embedded Systems, Submitted to Transactions on Computers (Major Review). Journal version of a previous RTSS conference paper. Both papers are full of proofs. Conference reviewers complained about the complexity of Theorem 4, which proves our main algorithm is correct.
A rather complex proof by induction on a non-intuitive index.

The real problem is that we did not provide a good enough intuition for the main algorithm. No journal reviewer complained about Theorem 4, even if the proof is exactly the same.

Lesson 5: but make sure proofs are correct

Intuition is more important, but make sure all your proofs are formally correct! If a conference reviewer finds any error, your paper is on a fast track to rejection. In journal submissions you usually get a chance to correct the mistake if the result seems reasonable, but do not take that as an excuse to be careless.

Example
Again, too many to mention. Reviewers can be picky about different details, especially regarding notation. Some general suggestions: 1. A good notation goes a long way in making a proof readable.

Try to avoid using too many subscripts, superscripts, stars, apexes, etc. Use notation that is accepted in the community (D is relative deadline, e/c computation time, p/T period, etc.). Repeat multiple times what each symbol means.

2. Break long theorems into multiple lemmas.

People get discouraged when they see a proof three pages long.

Example
3. The simpler the math, the better it is.
If you use anything more than standard algebra and calculus, be sure to provide relevant links. Do not assume that everybody knows X just because every undergrad must study X in your department. Some proof schemes tend to annoy people more than others.

Lesson 6: Strong statements are dangerous

Be very careful when you make strong statements about some research issue: there are people that think otherwise. Be especially careful when taking position on some hotly debated topics in the community, like:
RM vs EDF. P-fair vs multiprocessor EDF. Partitioned vs global multiprocessor scheduling. Hard real-time wireless. Testing vs static analysis. Etc. etc. etc.

Instead of saying X is black, say X is usually black, but in some cases that are not considered in this paper it is white.

Example
R. Pellizzoni and M. Caccamo: M-CASH: A Real-Time Resource Reclaiming Algorithm for Multiprocessor Platforms. Real-Time Systems Journal, 2008, Vol. 40(1): 117-147. We said that we use multiprocessor EDF because while P-fair is optimal, it leads to excessive number of preemptions. Reviewer commented: Some of the statements made about the effects of preemptions are too strong: in fact, there exist real-time systems that use cached data rather seldomly (because new data is continually being processed).

Lesson 7: but if you are confident, go for it!

However, high impact papers are those that successfully challenge existing preconceptions. So do not be shy when you state the main contribution of your paper!
If it is somehow controversial, you might have some troubles getting the paper accepted at first, but it is well worth in term of impact. If it is not, you should still stress your contribution so the reviewer gets more interested in the paper (remember Lesson 3)!

Just be sure to prove your point well enough; the keyword here is successfully challenge.

Example
R. Pellizzoni et al.: Coscheduling of CPU and I/O Transactions in COTS-based Embedded Systems. RTSS 2008. The analysis in the paper computes a pattern of arrival times for cache misses that causes max task WCET increase.
I was scared because the analysis is built on top of my RTSS 07 paper and furthermore the obtained WCET algorithm is pretty similar.

In the introduction, I made the rather strong claim that the result is very counterintuitive because the worst-case pattern is the opposite of the classic real-time critical instant (caches are spread out instead of arriving altogether). Two reviewers out of four commented that the analysis is very interesting and new.

Lesson 8: You can not make everybody happy

Different reviewers are looking for different things (remember Lesson 2). You must accept that it is simply impossible to make all reviewers perfectly happy; as shown in previous lectures, you are forced to make trade-offs.
For the same reason, take all reviewers comments with a grain of salt.

The key: two half glasses of water are better than one full and one empty glass here.
Just one negative review is enough to kill a conference paper. Once your paper is out, if the contribution is solid people will start citing you anyway.

Example
Pellizzoni and P. Meredith and M. Caccamo and G. Rosu: Hardware Runtime Monitoring for Dependable COTS-based Real-Time Embedded Systems. RTSS 2008. Paper is composed by three sections:
Description of the main idea and monitoring theory (main part). Test case (very important to show that this new idea works). FPGA implementation (short because it is not the core of the paper, but required to show that we implemented it for real).

Submitted to RTSS to a new design & verification track. Paper just barely got in.

Example
Two main issues among reviewers. Reviewers 1 & 2 commented that the FPGA implementation details were boring, and we should have expanded the formal method part (PTLTL monitor synthesis). Reviewer 3 wrote: the paper sets out to describe a hardware solution. Yet there is very little actual hardware design details.

Lesson 9: Retry, youll have better luck

What you should get out of the previous lessons: you can do a lot to improve your chances of being accepted, but you also need some luck. If you believe that your contribution is important, you should not get discourage if your paper is rejected!
Out of my 12 original papers, 7 got rejected / major reviewed first! I managed to publish all 7 of them.

If you do not believe that your contribution is important, than you should not have written the paper in the first place!

Lesson 10: but be careful where you resubmit!

Be careful if you submit twice in a row to a conference in the same area, especially if the community is small like in real-time research.
Reviewers are people, and people are affected by negative biases.

If you want to resubmit, be sure to:

Address all reviewers comment to the best of your ability, so the same reviewer can not reject you twice for the same reason. Changing title/abstract/introduction can also help.

Otherwise, just can go directly for journal.

Thats it people!

or it should be, but I have a final Lesson

Lesson 0
It all boils down to the following Prime Directive:

Write for the reviewers, not for yourself!!!