How to Write a Great Research Paper

that's why you here we gotta wait he shot quite finish it off without further ado I'll pass over to Simon because of course I don't have to introduce you yeah okay that's good right okay so uh let's see have you had a good week yeah if any of food anyway so may be food for the brain as well I hope so the ability to write good research papers is absolutely central not only to your career prospect but also to doing good research so what I want to do in this talk is just to make seven very concrete immediately actionable suggestions that I think will make your papers better but the ground rules are the same as in the talk about giving a talk this is slightly less recursive which is this is not a subject on which there are any right answers right so there's no in I have no monopoly on truth here so please ask questions or indeed make comments from your own experience as we go along that would be more fun for me we will still finish at twelve as mentioned at the end of the last talk okay yes definitely although any answers that are definitely wrong probably not actually you know to everything there is a counter example but I'm just going to err on the side of being highly opinionated and saying things that I believe to be true and then you can be free to disagree with them and fight with your supervisor about them okay so here's the first thing just start writing right so here is a typical plan for doing research first of all you have some idea and then you spend some months or even a year or two developing the idea and then you write the paper that sounds logical right that's just good scholarship isn't it utterly wrong here's what you should do right you have an idea you start to write the paper and in writing the paper you think oh I didn't really understand that idea very well that sort of forces you to refine the idea and you you should the act of writing the paper sort of forces out the research as it were what can happen with with this plan is you have the idea you start like you start you do the research and after you you start to like the paper and you think darn six months of that year were completely wasted and there's another six months left to do on bits that I didn't realize that I didn't understand right so I'm a functional program I like lazy evaluation so this is like lazy evaluation right you've got a you start writing the paper use that as a forcing function for doing the research an order Oh drat not only is it a good forcing function but it also it's a good way of trying to make yourself articulate what it is that you're doing so I can't tell you how often I've started to write a paper and I've realized that something I thought I understood I don't understand after all but when you write it down it all seems you know by the time you actually get it in letters on paper it seems more complicated than you thought it was right so it kind of crystallizes your ideas in a very concrete way and that can make it much clearer where you don't understand and also another really good thing about writing a paper is it something you can share with other people while you got all this stuff swirling about in your head it's very difficult for anybody else to comment on it or indeed engage in any dialogue with you you can sometimes do that one-to-one with somebody else but often that's a bit disorganized you know you start conversation with the whiteboard it's all very good but actually you know it's not something you can do in five minutes so the at your act of setting out to your idea in in written form can be a very helpful way to have a do to open a dialogue with somebody there's having sense yeah for you personally astray first oh I'm going to say lots about content in a second I'm just talking at the moment so the question was do you write the abstract first or last actually last but um but I'm going to but here I'm just concentrating on get started with my ting early but almost nobody does yes related to this would you recommend meeting your papers before you do the research oh no one actually going all the way to submitting it to a major conference their workshops are work in progress yes officer for graduate students graduate consortiums absolutely so yes for sort of workshop II things do submit stuff that is in progress say that in progress like if you liked it you think actually this makes no sense at all then you probably don't want to submit it but absolutely one of the nice things about computer science is that it it's just like a fractal it expands like a snowflake in ahead of you so everybody turns out is working in a different part of the snowflake unlike say biochemistry where everybody is converging on a particular the boundary of science is very well-defined so people are kind of in competition in computer science you're much less in competition with other people and so it's much easier to share so yes share half-baked ideas just advertise them a soft baked yes what about majors what about positioning papers yes well you mean I'm trying to work in this kind of area right yeah I have a greater year yeah I just didn't the algorithm but I want to get my ID out there sure so provided the the venue is you know open to that and you're upfront about what you're doing absolutely like just about communication you're trying to say is anybody else interested this idea yeah yeah so you beat be open speak early but I'm I'm almost my main point here is really to do with almost communication with yourself right about starting writing but also the dialogue with others is important okay so but one thing just about having having ideas is I think I've sort of put that pipe way more or less said this that for me writing writing is not an output medium like this this is this is sort of where the paper's like the printer at the end of the algorithm right for me writing is the way that you develop the ideas in the first place it's the it's the it's the machinery of research not the output not the printer for research does that make sense that that's what works for me let's see and so if you're going to do this number two is you need to know what your idea is and that is often not terribly clear in advance but it must be clear when it's finished but I'd like to just articulate what what is this business about an idea so I think of a paper as rather like a talk that I was making about on Tuesday as a kind of vehicle for conveying an idea from your head into what's your name Fraser okay so my paper is a vehicle for conveying my idea from my head into phrases head right so it's it's like a virus I put the paper for me goes into his brain and if it's successful it sort of tape smooth there and you know he becomes a carrier and he you know infects other people so it's a and it it's so it's a the paper is not a sort of you know way of getting research brownie points it's an idea conveying mechanism right and with a middle of phrase it's not I put it in a rather what's the word done dystopian view but but I hope you'll might hope the Fraser is sort of excited about the idea and think that's cool and interesting and maybe I could build on it and do new things that and I put Mozart up here because I think of you know Mozart wrote pieces of music that we are still let's think of them as his papers right and we are still reading his papers in concert halls right we just of course the the interpretation is you know huge huge part of musicianship but the fact is that the ideas that he wrote down was so persuasive and catchy and interesting that people are still looking at 100 years later or hundreds of years later I'll be astonished if anybody's looking at any of my papers in hundred years time but it will be VOC sightings they were so one yeah so so and and the ideas are more durable than anything else you know you may implement stuff you may give talks you may you may do a number of other things in the research but I think in a way that your papers are the most durable part of your output they are the things that future generations of students may read and think you know phrases paper really changed my life you know and I can point of some papers that did change my life so and that's a in a way that implementations sometimes do but not so much so we often focus on tools and implementations actually let's the papers that are the durable piece oh and a bit about communication if you have a great idea because you're brilliant which I'm sure you are and you sit in your cell having the idea and you do not communicate with anybody then what are you a heat generator right not so cool right there however good your idea is if you don't succeed in communicating them they are not benefiting humanity and that's what we're here to do so it's not sort of command communicating ideas it to is a way of discovering whether there good you showed them to people they said that note they sometimes they catch fire and spread and sometimes they don't that's a good winnowing mechanism so and so but but if you don't attempt to communicate in as articulate away as you can and persuasive ways you can then you're you know you're you're hiding your light under a bushel don't worry people will filter them out if they're bad ideas right but if you don't tell them they don't have that chance okay all right so um then we have to wonder about this idea so the the the the worry is if you look at other people's papers you go to any conference or look at any conference between your journal you look at um people that's a really good paper that's a good paper they they have great ideas all these people they must be very intelligent I oh I'm a mere worm my ideas are pitiful and weedy and nobody would be interested in them like this is what everybody feels right we all feel that this stuff that we're doing is not very significant unless you're remarkably strong minded or possibly a good person but I promise you that most computer scientists feel this sense of insignificance by compared to the you know the glorious reality of research in your field but I think the the bottom is is important that even if your idea seems a bit weedy ins and insignificant just write a paper about it anyway but you're not committing in to publishing it at a you know top-tier conference you're just committing to writing it when you write it it may turn out indeed that it was a weedy and insignificant idea and then not much is lost right you've just articulated it clearly but um maybe and very often this happens to me I start writing a paper about something I think is not so cool and then I find it actually takes me 12 pages of double-column eight point font just to explain enough about what it is for to be comprehensible to other people in other words it's turned out to be bigger and more interesting than I expected that's very common right occasionally you discover no it isn't but often you will so that the this is another get started thing don't wait until you have a good idea to write something write stuff about any idea that you've got they got the message there I don't wait to have a good idea because nobody does we only discover that they're good ideas later right there's really important because everybody has this problem and it's very disabling if you wait to you have an idea that you think is cool it cool enough ok let's see last little piece of about ideas is that when you write a paper by the time you are finished at least you must be clear about what the idea is you may not be very clear right at the beginning this goes back to the question about abstracts right so that um you but by the time you finished you really ought to be able to articulate what your idea is um more to say about that later but the the bottom point is the important thing sometimes you find you read a paper and I think oh well that's got several ideas and they've got so shush together they've been sort of squeezed into there you know the compass of a 10 page paper and you haven't done justice to any of them right so if you have lots of ideas you know praise be right lots of papers this is not salami slicing that is writing many papers about one idea that's writing many papers about many ideas ok so resist the temptation to just you know escape lightly over many ideas just try to try to focus and be clear what the idea is sometimes there's a very closely related family this is a kind of you know there's no nothing is completely true but this is a good baseline to start from ok yeah and so this is some Joe touch I gave a nice talk how I was acting what you said he remembered The Hunt for Red October and where captain whoever it is says just one ping they send out a sort of some bounce to another submarine only one light so that's a and the ping is the idea so a good so here's a good exercise when you read somebody else's paper close the paper once and try to say what is the idea that I learned from that paper what is the key thought in that paper used to be surprised how how hard it can be indeed I have found myself as a reviewer writing in my review I believe that a key idea in this paper is de dum de dum de dum but I'm not really sure because you also didn't really tell me and so I like that in my review because I want the author to think well well that wasn't my key idea at all so you know who have you a completely misunderstood they might then reshape it and I think yes that was the key idea I should make it more explicit so you know I would encourage you to be totally explicit and to one to write something that says here is the key idea I sometimes write a section that says you know the main idea that's a section heading right so leave your reader in no doubt when they get to the bit that you matters because papers have a lot of stuff they have some background material some setup that's actually you know it's it's not that's not the new part but it's essential to understand the new part you got a signpost clearly to say oh I we've reached the end of setup right now we're going to tell you the new piece don't leave them in any doubt okay okay so that's so much for the sort of wider white papers and what the main payload is your idea I just want to say a little bit about structure so far so good any other observations or questions yes I mean in general the soup is talking about how to write a good paper there let's not argue about it I mean just as as as a tool to bring your ideas into paper or do something you can present and share with other people do you really think a paper construction of you know up drop them and well some background and then formulation is this really the way to go or yeah and I'll say what I'm like yeah you said so a blog post might be another way to communicate yes but I would think that a blog post often has rather similar structure let's get there let's get there I don't any while Jenny the question what about the painful literature programming like than you just like you know when you're combining code with actually the artists oh listen programming when you're writing a document that sort of somehow interweaves code and exposition right so um so sometimes I have seen papers that are literate programs they will they say you know here's a the source code of the paper and you can run it there can be great and for some you know for some kinds of papers that's perfect if you're doing bioinformatics it might not be so so easy to do that but yeah and the other the other concern there is that if you have to present all the code because you want to execute it then that may overwhelm that the idea in the paper so it's it's a choice but certainly if you if you're presenting code having actually run all that code to make sure there's no bugs in it that would be good thing to do yeah okay let's say a little bit about structure so here's some here's a story so so going back to your question about medium like so my my mental model for a paper or for a blog post all for a personal conversation is I kind of imagined that I'm standing at a whiteboard with a friend and I want to convey my idea so what am I going to do I'm going to say here's the problem I'm trying to solve I'm going to say you know here it's an interesting problem and here's why moreover it's an unsolved problem that is you know there is something to do it's not already been done and now I'm going to tell you about my idea this is that this is how I'm going to do it and then I'm going to give you some evidence to make you believe that the idea actually works in practice you're going to explore some of the detail then I might tell you something about how my idea compares to other people's approaches right this is a kind of storyline if you had you know if you were with somebody for now you might like tell them and I think this is something you might do in a blog post as well like that that narrative story it might also be might all be briefer than in a paper but the storyline might be similar does that does that make sense and it might and it makes sense in a personal conversation as well so how does this turn into into papers well once we're talking about the medium of a paper so I'm now thinking mainly about sort of conference format you know 10 or 12 pages of fairly dense font but hidup like this applies equally well to a journal paper or even to a short form paper but the or even to a thesis actually but so here in the sort of size of a paper is what you what you get title you know abstract introduction introduction trying to fit the introduction on in the first page at least if you're in a format that has a decent amount of text on the first page and then then you say something about what the problem is something about what your idea is so about evidence about why it's good idea something about related work and then we're and then you're finished okay but the look at the visit the write the readership drops off pretty sharply of course I this is not data and just guess these numbers but some people are going to scan the content contents pages and maybe abstracts some people will read the introduction and by the time they finish the first page they're making this mental decision shall I just move on to the next paper in this proceedings you've got to think of those readers right there there you've got to you know that you lose a factor of ten I reckon after page one that's an important part of your strategy for writing a paper is what to do with page one okay so it's my sort of general story I wanted a little bit about each of these parts so I want to start with the beginning introduction the introduction part what are we going to say in the introduction well and this is about contribution so here's my to what I think introductions are good to be like to introduce the problem and to state what your contributions are and that's all you can fit in the introduction I taught early Mumbai introduced the problem so I always try to introduce the problem with an example right so this particular paper this is one of my papers from some time ago and right there after four lines of text I've got some typewriter font with an actual program right there and some and my hope is that somebody reads that and think ah I get I get an idea of what this paper is about now because it's not phrased in vague abstract terms it's raising some executable code so using an example it's very good and it's and it's right up front right very very near the beginning I'm right there saying here is the problem I'm trying to solve okay and then there isn't at this point enough enough space to say here's how other people have solved it and you wonder need might need to say something about wife and interesting and important problem but it can't be to harder problem so I call this mountain's versus mole hills so here's an example of what you see at the beginning of some papers you know computer programs have bugs and bugs are very bad and you know cite reams of papers that say you know have software engineering statistics about the cost of fixing bugs right now when you read a paper like that does your blood begin to race do you begin to think wow I didn't know that bugs were bad I didn't know the lots of programs had bugs or that they were expensive ah this is a paper I absolutely must read is that what you think of course not you think oh yeah yeah yeah yeah yeah let's get to the payload see what I mean it seems worthy it seems you know like good scholarship but it's completely demotivating for your reader instead you want to cut very quickly to something such what I call describing a mountain you know you're describing Mount Everest and it's plain that your paper is not going to conquer Mount Everest because that's like a you know a thousand careers are going to you know still only be in the foothills you want to describe a little hill right a little Foothill that you're going to conquer or at least make a decent assault on so here's a you know consider this program this is actually a bit like this one right I gave a particular program here's a little program it has a particular kind of bug in it you know look you can see the bug here so I'm going to tell you how to find this class of bugs right and I'm going to here and I'm going to give you some guarantees about that or at least I'm going to get interview good stat you see what I mean so you've you've made the problem small enough that it's believable that you might solve it and also that a reader thinks oh actually that's kind of interesting bug if you could really do that that would be pretty cool good name because they know you're not going to do the thing at the top they don't waste their time this is page one right this is the moment to which you have lots of readers and they're all you know falling off the bus okay so that's your sort of motivation part and then so there's motivation what are you trying to solve and then there's contributions now quite a lot of papers do have an explicitly explicitly say the contributions of this paper of this so this is a but a surprisingly large number don't I would say probably about 5050 and so when you're I think of it like this the contributions of your paper are like the specification of a program and the body of the paper is like the implementation so when you read the contributions that's like the spec what you want the reader to do is to think wow if you know that the author could deliver on those promises that would be amazing yeah I want to find out how they can do that what I mean so and so you need to be a bit specific so here's an example from another paper I wrote so blah blah but this is a sort of tail end of some motivational piece and then I say in this paper we put the choice in the firmer basis and then there's a sequence of bullets that describe the specific contributions of the paper I personally I would urge you to use that style to literally say the contributions are these and then give a list of bullets because they're like claims write that in a proof engine they're like the films that you're going to prove so you want to say claim one claim to claim foo you probably don't like that but that's what you're thinking right each one is a claim that you are going to substantiate later and that the reader thinks oh and more of it must be a refutable claim let's see so there are some in irrefutable claims you sometimes see this kind of thing in introductions right we study the properties of this formal system you know we describe the Wiswall system and you know and we've used wizz was in practice and you think we're going to tell you about that and again does that make you think this sounds fantastic they're going to study the properties of a formal system well no it's an irrefutable hypothesis of course you are going to study the properties of your formal system if that's what your papers introducing right you are not telling me we do anything new but if you say instead you know this is a sad this is a type system we are going to prove you know we're going to prove that it's you know type-checking is the tax system is sound you know they's you know progress and preservation and that type checking is decidable well now you said something right and put presumably in an earlier bullet you've said something about that it's a type system with some somewhat ambitious aims do you see the difference right so it must be you must state in your contribution something which could be false you know state in the contributions things that could be false got it I think that's really important and it's surprising how often people don't do that even when they do have contribution sections are you with it any other any observations I want to notice one other thing here is that I've um I brought forward references in this language of these bullets I've got these forward references to sections in the paper so and I did that here do you see here who's been precisely Section four Bravo discussing section five and six in contrasting section seven and what that says is what you should not do is have a often you seen introductions motivation contributions and then the rest of this paper is structured as follows in Section two we do this at section three we do that how many of you read that read that paragraph yes well you know about 50% of papers do this and when you read it are you thinking wow section two is going to be about this I'm so filled right do read that section you read that paragraph very carefully you skip it don't you everybody skips it and it's on page one which is your most precious page so don't waste those precious bytes on the front page with stuff that nobody is going to read instead instead of this stuff you see putting it in the narrative like I was doing here here and here now these have a much different status but now you're making claims and for every claim you're giving a forward reference to we're in the paper you're going to substantiate that claim and that is genuinely helpful to reviewers because they can look at the claims and if you don't have the forward reference they say well I wonder where in the paper I would find substance for this so they leave through it and they miss it right you want to say it's precisely as you can where the evidence is and that then that's actually useful and it comes so naturally into the flow of something like this it's really good strongly recommend this yes this is what's on page one yeah we're still on page one so these contributions can't spread too long either you've got to sort of compress them to say what what yeah we're still on page one yeah right version of these like saying we have a thing that you have to cut down when you are I mean shrinking the paper because most of these things also safe I mean don't ask to down for it ring the people oh you hunk of this to show you the paper this is page one happening in ordi yeah it's tough it's you have eight pages you've got ten pages of text this is the page you cannot shrink one page which is mostly a third of which is title and authors is what you've got to state your contributions and the problem don't shrink that don't try to put your payload on on page one I think because this is the bit that people will read right and so you have to have enough content here that they know whether to carry on you know it's tough for the rest I know but then you have only got seven pages left but it's your specification it's the driving force for the whole paper actually you can use it for garbage collection as well right so if you find that well what do we have right each claim in the introduction is pointed to and then you find section four is not referred to at all from the list of contributions so maybe section four is dead code it could be a good way to shrink your paper right all you think are now section four is really important because I make a contribution in it in which case you might add to your contribution list should I mean this is the universal rule I have papers in which I don't have sections that are playing an important role that I don't refer to from the intro but it's a good first step is to say is every section referred to and if not why not yeah extended abstracts and notes and we only have four or five pages oh yes so if you've only got four or five pages now you're into territory where you know whole pages like 20% of your paper you have to uh it but probably a somewhat different medium you may have to compress it somehow so I suppose I'm thinking maybe I really mean ten percent of your paper rather than exactly one page yeah and I mean the big thing I'm trying to say here is think about that page one think of it is that as a driving force for your paper and think about your readers and the fact that that's the time you've got to get them and that applies to any any media blog post to actually people scan the beginning of a blog post wanna know whether to carry on okay no all right so we talked about this oh and just when I talked about evidence like the body of the paper is the evidence by evidence I don't necessarily mean data right it might be data might be performance measurements it might be theorems that you've proved but it might also just be argumentation right a substantial argument that explains why your claim is holds you know so it's all sorts of things might be a case study might be illustrative examples might be lots of examples that so I don't mean any I don't mean anything toby crisp by evidence i just mean it's the substantiating part okay all right so that's so much so we're still on page one right where to where it's eleven thirty three and we've only got to page one and what comes next on many people's contents list is related work all right it seems that good scholarship what are you going to do you're going to start try to solve a problem the first thing you to do is a scholarly understanding of the state of the art in your in the problem before you began that makes sense doesn't it it's terrible just terrible but this is what happens oh I think you should put I'm going to explain why I think you should put it at the end and you may have to fight with your advisor or supervisor about this but there Hugo so here's why I think it's difficult right remember the goal is to convey your key idea right this related work section is like a sand bar or barrier between your reader and your key idea your goal is to get you know my idea from for my head to focus head as fast as possible and if I put a say before I can tell you about my idea for I first of all have to tell you about you know the notion of transactions from brown and modified for distributing systems and and you know dealing with differential privacy and so forth and you're thinking I have no clear what any of this is about so am I really going to pay attention to Simon so there are several difficulties with this so the first is that it makes them makes your readers think feel stupid all right let's see so let me see how can I put this yeah so you're faced with the compromise you got you're you're writing the paper you're writing this related work section and it keeps getting longer because to do a good job of describing related work it takes a little while all right and then you think but now that you know it's going to be three or four pages before we get to my idea so I'll shrink it so you remove every alternate sentence right and now you sort of squished it now it becomes much harder to understand all right and it's because after you finished your paper your reader will have a lot more intellectual scaffolding they'll have more vocabulary they'll have more examples they'll just have more stuff in their head that will make the the state of the art comprehensible to them moreover by the time you get to the end you also will have explained your idea so that you're it's easy to make a smooth comparison between what you've done and what they've done why was in related work at the beginning you're sort of forced into an uneasy compromise about about saying well this is the state of the art and we're going to improve in the following way but I haven't got to haven't told you yet what we're going to do it all your no clue what I'm going to do so it's very very uneasy and and so so and then your reader is reading this stuff and they feel stupid because they don't understand it why don't they understand it but it's not their territory primarily and because you've compressed it to make it short and they feel tired because they tried to wade through this stuff they thought maybe I need to know this to understand the paper which is probably false so if you're tired so by the time they get to the idea they're a bit worn out they're not their tails are not wagging coming sense yeah so so I just think it's much what works much better to put it at the end right so if you put it back here then you have all the scaffolding you can make smooth comparisons and and you're good to go that is not to say that on this earlier part you should ignore related work right so imagine that you're explaining something just to somebody on the whiteboard right you're going to say well you know here's the problem and here is the first way that you might think to approach it and you know you're probably going to describe you know a pathway to your idea and that pathway is bound to be using ideas that you've you know from the Giants on whose shoulders you are standing and even on the whiteboard you might say this is an idea that you know Mike burrows divert when he you know and he developed B zip but certainly in a talk you could have a little you know square brackets Burroughs 93 just on the side it need an inform part of your soundtrack it just reassures your audience certainly your knowledgeable part of your audience that you do actually know that this is standard work you could say it standard work right but the the goal is for this initial part describe the problem describe your idea it's an exposition in which you're taking the reader to your idea by the most direct path right and the most direct path will involve some mention of related work and you should absolutely mention that describe it in as much detail as is necessary to get to idea and incidentally giving credit where credit is due but your focus is leading your own reader by the hand to the to your idea it is not at that stage on giving credit to the the related work does that make sense so it's where your focus lies now when you get to write the related work section which you absolutely must I mean so in talks that there may not be time in a paper that you you know it really is essential scholarship to have related work to your paper is so here is something which you often find in related work sections that the you know the you you read it and it sounds a bit as if the author is saying well you know brown and white to this but it turned out to be really hopeless and you know green and and blue did this and that was pretty hopeless to it now but my idea is fantastic but so so they're sort of not exactly denigrating but somehow you know the opposition wasn't really very good but life isn't like that so but it is if I have um you know 20 pounds in my pocket here we are well I'm money in my wallet so if I give this to Fraser right there you can take a temporary Fraser so I'm now ten pounds poorer and Fraser is 10 pounds richer what a transaction is taking place it's a sort of zero-sum game but only a hand if I um you know Fraser is friend of mine and I give him love right and friendship am I the poor for it do I have less love to give to other people no love is a sort of infinitely divisible commodity isn't that amazing it's one of the wonderful things about human beings is that we can you know you give love and you get more it's kind of like the exact reverse of money and credit is like that right so if you give credit to other people and you explain how clever they were and how inspiring their work was to you then that doesn't make you look bad at all so obviously don't be sick of Fanta Q no don't say you know you're probably my reviewer so I'm going to say really nice things about you but just just you know express your heart right if you if you feel good about a paper because it really was inspiring to you just take a moment to say it was an inspiring paper you know it was it was seminal it really led to a lot for you say that and and that you know that of course makes them feel good but it makes it it brings credit to the whole field I think okay the other thing of course is when you're talking about related work and comparisons is that computer science is a very multi-dimensional subject right you know up even you're just thinking about programs they may run faster they may be they may be they may run in less space they may be simpler to understand or more complicated to understand they may be written in you know some suitable language or run in certain environments so it's very very multi-dimensional it's very rare to find that your work is sort of better in every dimension than somebody else's right so what you don't want to do is to say is to just focus on the areas where you're better you say well that'll an X and I'm better and why and what Palio that I'm actually not so good on that right so because what you want to avoid is your viewers saying the authors appear to be unaware that their system is completely hopeless when applied to Zed right that's that's that's a bad thing for a reviewer to be able to say it's quite disarming if you say of course our system really doesn't work in situations ed well at all you better use some other system for that it's very difficult to make something that's brilliant at everything okay so it's not just honesty and good scholarship it's actually disarming and and you know it helpful for readers if you explain where your not so good as well as where you're great because how I sense so it's actually it's easy you just gotta tell the way it is I mean yeah when you tear that weaknesses at the end of the related work so in your later work section you're saying here are these other things and you can do a sort of comparative evaluation I mean either as you go along or maybe at the end you could say weaknesses of our approach or something but somewhere towards the end yeah yes and try to avoid the kind of related work section that says brown and white did X green and blue did this and sort of one sentence about each that more or less is no more than a list of citations but doesn't give any value judgment you might not even have read the paper right you try to provide some evidence that you actually know what's in that paper and that you value it yes what about the threats to validity session where do you go down connects to validity tell me a bit about what you mean you have an approach but your approach is not sound it works in particular situations it's not it is not because you're comparing with somebody in yeah but it's better it's just your solution doesn't define jigglers right so you've got an ax which is good for cutting down trees but not very good for chopping vegetables yeah so say this yeah that's sort of weird oh so I would my instinct is always to you're leading to the idea right you might want to say on your path you might like to say you know we're going to focus on building on cutting down trees and we vegetables are important as we discuss in related work right so as you go on your path your forward referencing to your related work section or your evaluation section to say yeah I know I know we don't do everything and I'm going to discuss more there but I don't want to get derailed right here so for referencing all this and then you would have a sort of a slightly because it's easier to do that big picture sort of you know were we in the vast firmament of computer science when you have all the scaffolding that make sense what else okay so related work well so now that was them let's see so we still haven't done the payload of the paper we've done introduction and related work we have done much of the paper but in some ways the the body of the paper kind of writes itself that's not the stuff that's so so hard to write so the only thing I really wanted to say about this middle chunk is think about your readers and this is a bit similar to what they're saying about giving a good talk try not to recapitulate your journey you were wandering in a maze filled with blind alleys and Dragons and you know rotating knives which you wandered into right and it's rather tempting to lead your reader into the you know the alleyway with the rotating knives just to see how they get chopped up as well and I've read papers in which I have plowed through a page and a half of technical material thinking this is essential and then at the end they say does that turned out to be a really bad idea we have a much simpler approach it's really annoying when you do that right really annoying see that's what the paper is for is to avoid the heavy lunch at so don't we capitulated journey just lead them straight to the goodness there is an exception to that if you're wandering through you know you're taking your Vita by the hand and you're leading them through the maze and you go past this door and through the door you see this vista of gardens and flowers and honey bees and lions and lambs lying down together and you just these imposters if you had been noticed and the reader is thinking have you read a paper like that you think why didn't you do the obvious thing like why are you doing this complicated thing so there is sometimes when you really have to stop and say you know you might think that this pathway would be a good one but actually doesn't work at all for the following reason or you know that's elaborated somewhere else again you don't to get to do world we're just acknowledging when there is a very obvious Avenue that you're not taking why you're not taking it is a good thing or you could say somebody else took it and it's described very well in this excellent paper that I'll discuss a bit more but in related work but it turns out that doesn't work very well in the setting that I'm interested in which is this and on you go does that make sense yeah I think that it's also kind of I mean there should be like a trade-off between reproducibility of your paper make your idea I mean like some other people things that is possible to reproduce your work yes because for example if there are like I mean sometimes I agree with you on these that you have to show your idea but maybe it's okay also to put like a paragraph not someone can escape like thank implementation tickets oh I mean they read are kind of sleepy but are you filming but if it's another student like me I mean the student maybe need details in order to continue working oh yes so so when I say you know you want to present your idea I do mean you want to present it it's a paper right you want to present it in enough detail that somebody could actually deploy it and use them use it themselves and that means you know you need a bit of detail right that's what the body of the paper is for so it might be you might say here some implementation details you might say look there's a whole section about implementation you know a little bit later in the paper that gives more detail so I think of it as being somebody might read the papers there might be the introduction they might get to the idea I think that's an interesting idea but actually I'm not interesting up in the details to plow through this so then they jump to the next paper so you're almost wherever they get off the bus they've taken something valuable away with them yeah but don't leave out the meat that's right so I'm not talking about just a high level general idea stuff you want to get to the meat it's a paper it's got content yeah just fine you about just getting all of these details that most of the readers won't be interested in and just sending all of that to the supplementary material because that's someone I've seen several papers doing that yeah yeah so usually you're constrained by length one way and another right so you end up putting as much material in as will fit in the length constraints you're given and the rest in supplementary material and so I mean a classic example as you state a theorem you may give a brief proof sketch and you say you know the full proof is worked out in detail the supplementary material who can see the code in the supplementary material all days you know much yeah so it's a trade-off I would but it's pretty much always constrained by length right what you can fit you probably want anything that's why all papers always end up hard against the length limit do you keep stuffing stuff in but it's good to have a an appendix and reference the appendix even if it doesn't get published you can cite it with a link yeah simplifying something so shouldn't I shoulder either how difficult it was so they can appreciate how oh yes yes yes this is so the the best ideas are the simple ones and sometimes a paper seems so simple right that the reader thinks well that wasn't very hard water but a silly paper of hey that's your that's always your worry right so you might want to show them the awful chopped up limbs of the previous people who wandered into the the alley so it's that's actually quite difficult right because it's um because because it falls straight into this trap of saying well I'm going to show you something simple but he reversed complicated things that I tried so it's probably more convincing if you say he reversed complicated ways that other people tried but you can go and read their paper but I'm going to give you a much simpler way of tackling maybe not exactly the same problem but I found a way to recast the problem so that it does have a simple solution but I I would urge you probably not as a first approximation not to drag them through the complications or maybe illustrate a complication or something say you know this is just a show how bad it could be if we didn't do that something like that yeah ok let's see yes so intuition and examples all right and and the even if you skip the details you take away something valuable this is a good lesson to take from a paper it's also a good lesson to think about when you read somebody else's paper if I you know I've left this paper I've only read half of it have I taken anything away or was it an investment that is essentially yielded me nothing because I didn't actually have enough stamina to get to the real payload doesn't mean you want to do this sort of gradual thing of always offering something conveying intuition is primary I keep talking about examples so instead of starting with let's see what was this yes instead of starting with some general statement that you then specialize and give examples well just like with a talk start with an example so that you get the kind of intuitive idea of what's going on and then you could say here's one example here's another example and now I've managed to extract a general case so even if somebody only gets to the examples and does not even understand the notation in which you've written the general case they've still got some intuitive feel about the problem you're trying to solve and how you're chopping it alright for examples examples examples at examples first same thing in the papers for a talk so here's a here's an example of me doing this in my paper this was some another paper about some while ago this is section 2 at which I'm doing the setup of explaining the problem and again four lines into the setup right I'm off with an example that I then use as an illustration for the whole of that background section to say what problem is that I'm trying to solve and why it's an interesting one yes we talked about recapitulation the journey good right aim is to go and last point okay this is about your readership now of course we've been concentrating on you as an author but trying to think about the audience and but it's good to one kind of use your audience particularly to get help with guinea pigs so a guinea pig is a person who reads your paper and gives you feedback like now of course the ideal guinea pig is already an expert in your field and has been following your work and is really keen to read your next paper these these are hard to find but you have quite a lot of friends probably you know just colleagues other research students in your department other colleagues that you work with in some form or some way or another and so you can ask them to just read your paper and say what they thought of it now so two things about that one is it's like performing an experiment on a you know in a lab on a guinea pig you know you have a guinea pig you would sort of inject the drug and you see if the guinea pig sort of falls over dead or whether it sort of leaps about right but if you have and then you go to refine the drug based on you're going to rewrite the paper in the light of that feedback if you use all your guinea pigs on day one you don't get them all they all fall over dead right then you'll know how many left but when you've done the new version now you might think well I could just ask my friend to read it again but nobody can read your paper for the first time twice stands the reason right and that first time is really important so by the time they read it another time they read it completely differently when you're reading a you know a new draft of the same paper you sort of skip and you you say I think I've read that before it's totally different experience to reading for the first time so just use your guinea pig sparingly you kind of use them one at a time and that also is another thing about writing early right if you started early and you have drafts early you can use your guinea pigs if you haven't started until a week before the deadline well you know you can barely use guinea pigs at all okay so an even non-expert guinea pigs are quite helpful like people who are you know not even expert in your particular field these should be able to make sense of the first few sections right if they're go too motivated because you're they're your friend all right they might not understand all the technical details but it's really important with your guinea pigs is to explain what you want because I tell you it's an invariable quantum property of guinea pigs that they will tell you about spelling and grammar first that's what they will do they'll say you will give you back Babel lots of markup about you know I would have put a comma here you're not interested in that at this stage you are interested in when they ceased understanding right but they will not tell you that unless you look them in the eye Johnny and Anita and Anita right gonna look you in the eye and e2 and say I really want you to tell me when you get lost in this paper you know because it's my fault not yours if you get lost at least in the first few pages right and I've got to make Annie to believe that so that she will tell me because once you know that and then the dialogue then you start a dialogue like so I got lost you know halfway down page two I thought I was I was okay and then I got completely lost and then you have a dialogue you start watching the whiteboard you do diagrams you say what I really meant was this and then he says Oh that was what you meant and you think ah done I just got to write that down it's amazing all you have to do is to literally write down the things that you end up saying to your friend but you will find you explain it much better interactively than you do on paper so it's a it's very important I think not to just give it to somebody and get written feedback but to give it to somebody and say just put it 1 mark where you get lost and then we'll talk about it that's my advice okay of course you know the better qualify they are the more they know the field the better because they can get further through your paper so that's that's all good yeah but the key thing is to get this sort of initial initial feedback you can try and get feedback from people further away who really do know what they're talking about what one wonder you can do is to send people even people you don't know maybe you just met them once or twice at a conference or something you've bowled up and said that was interesting so here's my paper and you know you featuring related work like I just want to would you you know could you have a quick look just check if I um you know if I presented your work fairly people send me a paper like that I think well at least I take a look like after all they they're mentioning me so um it's it's variable how well this works but another nothing you can do is a it's when you're at a conference you are you know there's some important person who's giving a talk that you really think was a good talk you enjoyed and you learnt something from then you know introduce yourself afterwards and say I thought that was a really you know enjoyable talk I learned a lot from it maybe ask them a question you just form a little relationship with them it doesn't necessarily need to lead to anything but um but you know then another nice thing about computer science is that almost everybody is open to conversations with absolutely anybody provided they've got something to talk about and there's something to talk about is the easiest place to start with is something that they've said if you walk up and say can I tell you about my research there then they think well how long is this going to take like if you say you can ask a question about your talk that's that yeah this is all more bounded thing so it's a it's all part of this interaction okay let's see the other thing is that when you get reviews and this is this is sort of more at the end of the pipeline you submitted your paper and it's rejected right and your reviews you know make totally unjustified criticisms and you you're sort of bleeding alright that's really hard or even when you know Anita's criticized my paper right do it do I do I say Anita you're really stupid you know so somehow you have to be to be grateful for critique now why should you be grateful well one is you might be able to use it to improve your paper but another is this remember about love that was infinitely divisible what about time not infinitely divisible in fact absolutely unfun jabal as we were talking about three days ago so if somebody be it a reviewer or you know one of your guinea pigs has given you the gift of an hour of their time to read your paper and in the case of a referee to sit down and write a review that is a free gift that they are giving you that they will never have again that hour for them is gone they're just an hour closer to being dead so they are giving you a gift and you should rejoice in that gift and say thank you right so sometimes you know reviewers are really young you know sometimes reviewers are stupid right sometimes who hears genuinely ill-informed and simply you know are just not qualified to write about it and just say wrong things but that's pretty rare mostly reviewers kind of misunderstand something and so when you finish bleeding from your reviews look at them again and instead of saying stupid reviewer you know you shouldn't have been allowed to review this paper instead say how could I write the paper so that not even this idiot could make that mistake okay that's a constructive approach is not easy this is very not easy but it's important yeah well how do you handle adversarial reviews I mean they they they come back to you though in written you don't know who they're from and well suppose you'll handle them like this you say you know is this criticism justified could I change the paper so that it would be you know is it a misunderstanding or is it simply you know sometimes you get review that just says I just don't agree with this idea it's a it's an it's not a good approach right and you know what can I do about that right so sometimes you just have to ignore it nothing you can do right but um but it's rare that they're actually aggressive usually they're come together with them you know with things that they didn't like or they can make better it's not not invariably yeah oh what did you suspect a review hasn't read your paper well I think that's vanishingly we're actually or maybe maybe not in your experience but I don't think I've ever read a review that I felt was from somebody who had not read the paper by and large in this thing computer science I think people are pretty conscientious about reviewing and yeah I don't know said that have you has that happened to you know is that ever happened to anybody okay so let's not debug that problem it really is where I mean they might've misunderstood they might be informed they almost certainly will read it so thanking that is important right so at the end of your paper it's worth taking those two lines at the end just to say thank you anonymous reviewers if you had guinea pigs who you know who they are mention them my name right it's it's very cheap and it's what that's the spreading the love thing the anonymous reviewers it's just that it's kind of like just polite but then they're not actually gonna pin it on their wall okay oh so that's so that's it here's here's my um summary 1201 very good seven things you can do that will help you write better papers there we go any other questions two observations because yes I'm all that stands between you and lunch really on time but do do do have a go at you know for following some of these I've sort of tried to make them actionable things you can actually do rather than just aspire to it most be in it and know whether you're doing them or not you may have to fight with your advisor a bit about some of them so you know have the fight in the end you have to do what he or she says them

Channel: Microsoft Research

Views: 88,497

Keywords:

Id: VK51E3gHENc

Length: 57min 39sec (3459 seconds)

Published: Tue Jul 26 2016