How does it work? Using process tracing methods to study policy processes - Prof Derek Beach

and at the University of Aarhus Denmark Derrick is political scientist he has a master's from the London School of Economics and MA and PhD in political science from the University of authors he also has visiting positions in the u.s. at Johns Hopkins and Georgetown University Derrick is at the forefront of the development of case study methods for Policy Research and we have been engaging in dialogue with Derrick for a while and he has been great and we we are starting to use the kinds of methods that he's going to be talking about today so we're really delighted that he's able to give this seminar today he was to have been a seminar in person a few weeks ago as you probably know many of you will know and it's really great that Derrick's agreed to give it from his study in his home therein in Denmark and he's going to be talking about process tracing so I'm going to hand over to Derrick I'm going to switch off my video if everyone can keep their microphones mute and their videos off that will help our bandwidth and okay so over to you Derek okay thank you and let me see has this been shared okay yeah and I apologize for I was planning on coming and Denmark locked down right before I was the evening I was leaving and leaving the next day so I simply got spooked and and so I'm glad to be here and I hope to be able to come in and offer a full full course on unprocessed racing at one point so I've been asked to talk a little bit about process tracing and the study of policy and I mean get this okay there we go so talking a little bit about how different different kind of presentation a little bit of existing process evaluation methods for example which also overlap with the broader and a general way of thinking about case studies and and there's I would say a significant lack of real taking process seriously and this is I think the the core goal or the benefit of process tracing is really trying to dig into different for example policy process understanding the interactions between actors and and and trying to figure out what's going on but in figuring out not only the descriptive it also enables you to make some causal inferences and and that would be in particular the the goal would be to to understand how it actually worked in a given given case for those of you that are familiar with the the philosophy of science a lot of what I'm going to be speaking of regarding process tracing is basically within what one would call a realist or a critical realist understanding of science and the kind of underlying ontological assumptions that are that are there I won't spend time talking about that but that was just just omit to put that out there then unpacking what process tracing is understood as well you can see the word process tracing so that's so what is the method how is it being understood right now and then digging into the two elements of process tracing so the first is the more theoretical question of what are we tracing and then the second will be regarding how do we do that and I'll talk frame that in the light of talking about internal validity and the ability to make causal inferences but the and so what we're tracing are mechanisms or processes policy processes for example linking maybe a crisis with a particular type of policy change and then how we trace it is then looking for the observable traces or manifestations of activities that are left that that activities of actors leave in particular cases okay so as an example this is a for example a a a for the UK Medical Research Council talking about process evaluations and and then the different types of methods and tools that that are typically talked about when we talk about analyzing for example policy interventions and for those of you that are familiar with evaluation methods these would be things like you know looking at for example the context within which a given policy intervention is is is put in place this could be using ideas from realist evaluation for example then we would typically have that in the actual white box would be the cause or what is the intervention and its causal assumptions and then we would maybe talk about implementation of the policy and here there is a set of different different methods used there for example and and the mechanisms or or what process tracing actually is would be interested in would be the link between a cause and and an outcome or the intervention a policy intervention and outcome in in in existing evaluation methods for example let's say a theory based evaluation realist evaluation these mechanisms would typically be black boxed and I'll come in and show an example of this in a second and then the actual evidence for example in particular if you look at the the mechanisms of impact we would typically use interviews we would go out and talk to stakeholders and and get their impressions of what actually would actually took place or or what worked and there's a lot of methodological issues regarding just using for example stakeholder and stakeholder interviews as as evidence but let me show you an example of kind of what this this kind of would look like you know the the core problem with with both within evaluation methods like realist evaluation etc or with things like the case study methods that are typically talked about in in political science or in the policy Sciences is is that they do not really answer the how does it work question and in particular what a most existing methods do is that they black box the because of linkages or what actually is is is is doing something our black box as assumptions and the problem with that is that if you don't explicitly theorize like activities people are you know III do this it then leads somebody else to do that I don't theorize that explicitly I cannot then trace it empirically because I do not know what was going on which then means that you're at the end of the day it becomes a descriptive analysis unless you're actually trying to theorize and then trace empirically those causal linkages in a process so this is an example of a from an article I've written with it with an evaluator where we were looking at the particular intervention I and an existing theory the way that people would typically trace think about this and the cause or the intervention is over here the phase one you have this coach engages in an analysis with voluntary participants and then it's supposed to lead to a participant on the right-hand side a participant having increased motivation to change the situation and the question is well how what are they doing how do they engage then and if we look at all these boxes all these boxes in this theory here are just assumptions here are assumptions about that the participant wants to do something here are assumptions about their motivations and the choices they make here are assumptions about motivational levels etc Trust etc but the the this theory there's a lot of boxes but it never tells us what people are actually doing what does engagement how do I engage with for example some participant to make them want to do something what is what are the types of activities that are involved and and and so so basically at this time of this causal linkage then it just just becomes a bunch of assumptions so I never if I was going to use this to analyze how a policy intervention worked I wouldn't have any I just be looking for are the assumptions that I'm assuming are required are present but I'm not actually looking at how it works and and my colleague Benedict and I then then thought about well what how would this look like if we translated this actually into a process model or process theory and just as an example these two parts that I've outlined in red there's a lot of stuff going on here but basically what we tried to do a sequential set of activities so we have a coach using information and then asking the the participant to discuss so there's very clearly theorized activities that this person is using that we can we really think make sense as far as these in part three then we'll lead into part four doing something the person doing something that then you know continues the process along so what we're trying to do in this theory is unpack the linkages the activities understanding how does it work instead of just working with all of these assumptions I'm working right now with with the independent expert group of the World Bank on trying to translate some of their policy intervention models into process tracing one of the things I find there is that they'll talk about indicators for success or or or milestones deliverables but they never actually kind of unpack okay if I give a country this loan why would I what are the activities that I think how is this investment you know what's the process how is this going to lead from A to B so they have these indicators for success but I don't know why those are indicators of success because I don't know what people are actually theorized to be doing in the actual case and so this is the kind of the goal of process tracing is to get that that type of of knowledge is to to answer the question in how it works and this involves two then elements so the core of process tracing the one is that we we have some causal theory a cause and outcome and then we try and unpack the arrow in between so process tracing is not about figuring out what the cause is or what the outcome is it's what links the two together and you unpack it you break it down into parts composed of of entities these are you know people or groups that engage in activities then the activities are what are actually in more physical terms transferring causal forces from one part to the next so so the the core of the process tracing is that we're interested in what people are doing it's the activities that we're trying to to unpack and then we assess this empirically using what are called in medicine they call that mechanistic evidence or traces so basically if if I am lobbying you know a politician that should that that activity of lobbying might leave a trace it in or if it's called corruption it might leave the trace that that that I put money in this person's bank account that would be mechanistic evidence so this is what it processed racing at least in theory you know very very simply put is its unpacking a narrow unpacking it in two parts entities activities its then trying to find the empirical traces of each of the of these activities for each of the parts of course it's a lot easier said than done to date I have not seen anybody actually doing I would say a full full-blown process tracing in the form of the absolute perfect theory and all the evidence that that makes us beyond reasonable doubt confident that this is what happened in a case it's it's an ideal type but we can we can approach that and and I think there is now beginning to come quite a lot of literature for example in the policy Policy Studies field that is that it's really interested in and and and actually has unpacked a bit more policy processes you know what's going on between a cause and an outcome and giving us an understanding of how things work in actual cases okay in in medicine and a lot of my my inspiration for developing these methods and these methods are under development I just recently got a big research for your research grant to continue working on developing these methods I think they're a little bit further along in medicine so there's been a uk-based research project called evidence-based medicine plus involving in particular people like Lori Russell and Williamson Nancy Cartwright was also involved that that actually produced a book this is the final product but where they were they're basically working in the same kind of genre as far as saying that in medicine there's two types of learning about the world in in their opinion the one is what we know from experiments you know controlled assessment of the difference a treatment makes that's one way of learning about the world and answers one particular type of question but there's a lot of question that are left unanswered in particular your experiment does not tell you anything about how a treatment works so so that the process tracing would be this unpacking what's going on in between your treatment and your your outcome and trying to figure out the the the the yeah the mechanistic evidence in the traces and see how it worked so for those of you that are interested in and these methods more broadly this is a this project there's so quite a quite a lot of good good work that they produced that that that is is is relevant okay so what are we tracing in process tracing and we're tracing how things work and this involves the the study of causal mechanisms I'm not actually increasingly a big fan of the term causal mechanisms because I think it's a quite equity vocal term in the form of it has it's so debated what it actually means many people understand causal causal mechanisms as as almost as causes as a synonym for a cause other people understand them as machines but then you don't really have to unpack what's going on it just does something we can treat them as intervening variables I prefer the term causal process and and that's because then it gives us this there's a temporal dimension and that's what we are trying to unpack of course a causal process is also embedded in time and space and in a particular context and these would be the things that we would also be wanting to to pull out but causal mechanisms or processes are what's binding causes and outcomes together to repeat they are not causes they are the arrow in between these are the things that are black boxed in your experiment process tracing of course is only one way of studying causal mechanisms there are people that use things like mediation analysis as well in a second I think process tracing if you really want to take it take mechanism seriously it's probably the only way to do it in my opinion and and actually in medicine there's analogous kind of the process tracing version of Medicine would be using observational data to to for example smoking and cancer then this would involve actually observing how the smoke goes into the lungs irritates tissue produces mucus this then produces cellular changes and you would be actually observing these processes and and and that would be of course something different than if you were doing your experiment this is an example of from from a published study from policy studies journal of what process tracing can look like this is just the theory the evidence is not here and this is a an author that was interested in olga orlova was interested in how an epistemic community here it was within health policy an epistemic community could influence actual policy so you have a community of experts and how could they influence policy and and and her argument in the article was well a lot of people look at epistemic communities and then see that that there is some kind of outcome but they never tell us how it works how does this epistemic community actually gain influence over policy and so her idea was well as a first cut and this is a first cut would would be exploring is there any evidence suggesting that maybe it's through some kind of actually getting in the room kind of process so he or she has her part one which actually i think is the cause is you have an epistemic community that wants something has it has a has a set of goals based upon its its it's beliefs as as you know health policy experts then we have a part two they they are promoting their favorite policy but in particular how the activity that's interesting is they're gaining access to decision-makers so they're actually gaining gaining access in the form of getting hired and getting embedded in different ministries and then because they're in the room this is then part four because they were in the room with decision makers the decision makers do not have all the information they need they rely on the experts the experts or the epistemic community actors so this gives the the epistemic community expert actors in the room influence over how the policy then is developed and then we get the the outcome of a policy in line with what the epistemic community wanted and of course if you look at this there's a lot of unanswered questions epistemic community gains access to decision-makers well how does that work how do you gain access what are what are people actually doing in cases in the part for what is actually going on within the room how does that work and and and and so the you know her article told us something she provided evidence of this process made it plausible that this is what took place and into cases but in some respects her analysis actually in some respects raises more questions than answers so we have we get an idea of how it works but then further research could then probe more about well how does access work how does it work in different contexts she was looking at health policy does it work differently in other types of policy maybe education doesn't work differently in different countries she looked at I believe it was the Czech Republic and Poland etc etc so so in some respects good process tracing it tells us something but it often will actually raise more questions than answers and I think for research that's that's a great thing it's great to know what you don't know and and then you can you can try and and figure some of that out so this is in and on the the references here you can see if you're interested in in what a a good enough to be able to be published in a pretty good journal process tracing can actually look like okay so let's let's dig down a little bit more into what we know from the philosophy of science about mechanisms and process so so again these are the causal processes that bind causes and outcome together they are not series of events a lot of people when they when they are trying to theorize a causal mechanism will just have well this actor did this and then this actor did that and then this actor did that and then the sequence is then actually what they think gives them a causal explanation but just people doing things in sequence is not it gives the appearance of causation but it's not causation you need to see the linkages why did when this person did that how did that influence the next actor and and and that's that linkage that that would turn a descriptive event story into actually a causal process or mechanism so just as you know saying this did this this did this that is not a mechanism so if we look then in the in the literature about well what then can mechanisms be if they're not just events well there's two positions and and and we could spend a whole whole day talking about this I'm not going to the one is a counterfactual and the second isn't productive account so I'll just real briefly come in to what these these these two positions are so a lot of people when they when they think about causation they they think it's all about counterfactuals and this is this comes from a class of quote by david hume where he says if causation is if the first object had not been the second never had existed so the claim is that you can see whether a cause is a cause or a process as a as a causal process or mechanism based upon studying whether its absence results in the absence of the outcome all other things held equal and of course this is exactly the type of causal claim that underlies an experiment so people like woodward and others then argue that causal mechanisms then are just lower-level counterfactuals that each of these parts of a process then should be treated as if it's a counterfactual and then we would have counterfactual dependencies that then can be manipulated for example using an experiment so X to the M and then the mechanism to the to the outcome and then they that M is then some kind of intervening variable in between and then if you're using this kind of understanding of causation and of mechanisms then this means that you need to assess empirically the epistemological implication is that you need to assess empirically the difference that variation of your mechanism makes for values of Y across cases and then controlling for confounders so this could be a mediation analysis large and ideally using experimental manipulation it could be a most similar systems design type of thing we could use matching techniques to figure out what two cases are most similar but the core problem with all of these is that they're comparing across cases so we never actually look into a case and figure out what's going on within that case and if we look at these kind of this literature they never actually tell you how mechanisms are identified the theory about what's going on in between they've used typically more ad hoc case studies so mechanism identification is usually through some kind of with in case they maybe call a pathway analysis but this is not scientific really in this way of thinking the the real science comes in when we're first testing I think there's another way and there's a lot of people in the philosophy of science let's say there's another way of thinking about processes and mechanisms that then has particular type of epidermal aaja chol in plication x' for how we do research and in the questions that were able to answer and this is what could be the productive account of causal mechanisms where we're trying to open up what's going on in between so we're really trying to open up that black box and observe it and actually the the term process tracing originally came from psychology where psychologists when they were doing experiments were tired for example reasoning the the psychologists were tired of a lot of what they were studying would just then disappear into people's heads and they didn't know what was going on so they said well how do we get some clues to to be able to interpret our experimental results about the processes going on the reasoning processes going on and they said well let's let's look for observable traces things like how your eyes move or or your facial gesture or how quick you answer a question as traces of perhaps some things that are mental processes that are going on that then we can infer that that that exists so the the the the where process tracing came from was actually a dissatisfaction with the questions and answers that were coming out of experiments and this would be then kind of the the the where we would continue then with this way of thinking about process tracing and in particular what we're trying to do at the theoretical level is making clear and explicit the causal logics binding parts of a process together instead of just assuming the linkages we're trying to make them explicit theoretically and then try to study them empirically and it results in what could be called how does it work explanation so in a given case when policymakers introduce a particular tax on let's say sugar well the question is well how does that work what how are people responding in the real world to a particular policy intervention that would be the process tracing question at the end of the day you would also want to do the the standard you know statistical covariation analysis seeing whether this intervention actually may the difference your process tracing would answer a different question it would be answering the how does it work question so in this kind of method the causal inference and identification through tracing of fingerprints left by the operation of activities within a case is so we have both an inference and the identification of the process kind of going on at the same time there's then two kind of ways that you can you can think about doing process tracing and in some respects I'm increasingly not very happy with really flagging them as two different types of process tracing because in reality it's always going to be an iteration but let's just take the ideal type so a theory testing process tracing would basically you're able to sit in your office and theorize your your step one is conceptualization of the cause the outcome in policy analysis we often have the cause or we have we know the outcome it's a policy reform and then we want to figure out what that we need to identify then what the cause is it could be a crisis and then we would theoretically try to unpack how do I get from A to B and in the theory testing then after you have that theory conceptualized then you would say okay well if this theory holds if these activities are taking place in this case what are the fingerprints that we would expect it to leave those would then be what you could call propositions or you know propositions about fingerprints or observables terms mean the same thing and then we would actually go out and and and at the empirical level collect empirical material and evaluate its evidential value in relation to being able to infer that the activities that we had theorized as the links actually took place theory building is kind of starting the other way around we don't know what the process is so we go out we start trying to explore soaking and probing in a case and then trying to make a leap up to inferring that okay we're seeing patterns in the empirics maybe this is fingerprints of some activity that we then can make sense of theoretically of course in reality real research is going to be more of what we call abduction it would be a back and forth between we might have an initial we might start with a theory testing we figure out no it didn't actually work exactly that way and then we okay well then what does the empirics tell us actually did take place in this case so in reality process tracing is often an iterative back-and-forth and this slide then illustrates the the kind of the core difference between you could call a variance based or the counterfactual way of doing research so your your your experiment you know RCT on the on the right hand side is basically enabling you in the words of Nancy Cartwright's who make a claim about it works somewhere you don't know of course whether it works in any individual case but your experiment tells you on average it works so so here are we would we would the experiment of course you take a sample of cases out of a total population you give the treatment to some and and and and you know the proxy or the control to others you investigate the average difference it makes in the level of internal validity or the strength of your causal inferences of course here are dependent upon the amount of control that you have in in your experiment and then the external validity questions is do it does this average causal effect hold in other popular Earth's samples or parts of the population and of course the unanswered questions are well local causal effects what's happening in any individual case and you're not answering the how does it work question if you move to the left hand side this is the process tracing this is how it works we take an individual case and we trace a mechanism linking a cause and outcome together it enables us to make a claim about how there's this linkage in the case of course it does not necessarily enable us to say that this is the only this is the only process taking place but we can we can see that there is a linkage and understand how something actually works in a case but because we spent a lot of time working on a case the downside then is well does it work that way other places and and we know of course from Policy Analysis that that often we we extrapolate and and generalize quite heroically a lot of people looked at new Public Management and how it worked or maybe worked well in New Zealand and then everybody else said well I need one of those and it never really worked anywhere else other than New Zealand in the way it was intended and this is this you know when you really dig down into the case you often also find that that it maybe is quite a quite unique or at least it can only be then maybe generalized to a smaller set of cases than then we're typically used to thinking of the unanswered questions in the process tracing are about net causal effects it does not enable you to figure out is this the only cause or or is this what is the actual causal effect between x and y or where does it work the internal valid or an external validity question is typically difficult there just to put up from from the case based so so the the our ability to make strong causal inferences the process tracing would be at the at the kind of the gold standard for a case based way of thinking about how does it work and those kinds of inferences on the right hand side you see the typical evidence hierarchy that we know from variance based research you know experiments the lab experiment is is is the gold standard and then we we move down evidential hierarchy lower lower down okay so what then determines that something has a high internal validity its basically well the strength of the causal inferences and in process tracing this then would be through whether we've actually traced the activities of each part of a causal process linking a cause to an outcome within a case and then the external validity question of course would be does it travel and there's naturally a trade-off between these because the stronger the internal validity of a particular case study is typically when we've really unpacked a process and have a lot of detailed information but then that means that we this theory then that we get out of this is quite case or context sensitive it's not necessarily going to travel very far whereas if we have a much more abstract theory that we lightly evidence so if you think back to the example I showed you with the lebwa and the epistemic community that's a quite abstract theory and that in theory could then travel to many different cases but the internal validity because she hasn't for example really tracked how how they gained access the internal validity or the strength of the causal inference is lower so there's always going to be a that type of trade-off regarding whether you really want to understand how something worked in a particular case or whether you want to have maybe a rougher mechanistic sketch lightly evidenced but that it then can can can work in multiple different cases so the external validity you would also typically have to do multiple case studies to see whether it's similar causal processes going on so if we if we just put on the then kind of what determines and look on the right side the high possibility of the strong causal inference category or column here there's three then things that would would make it a good process tracing case study the one is a theory level you have basically what you could call productive continuity you've unpacked theoretically each of the part of the process in particular you have an unbroken chain of activities linking the cause to the outcome then in theory still you would have them propositions about evidence that are very unique or specific for your theory and are quite direct and then the you have strong sources and full access to the empirical records so in that's the ideal typical situation is that you're really able to evidence each of these parts and have really good good evidence at the theoretical level then this this involves well we you make your cause explicit you unpack each of the parts in the form of activities which are verbs that are doing things that are are linking to the next part your entities then our nouns it could be a government it could be a particular civil servant or whatever is relevant for your for your theory so the parts are factors composed of entities engaging in activities your entities are nouns social objects engaging in activities and the activities are the crucial and I can't emphasize this enough they're the producers have change and then context in this type of theory is also quite important this is a it's a whole whole lecture in and of itself we'll just put it out there for now and to qualify as a mechanistic explanation you have to be able to explain how something works there shouldn't be big holes in your causal mechanism you've told us what the entities are doing the causal logics or made explicit so instead of for example a superficial mechanistic explanation would be you basically you just have well it's mobilization processes okay you haven't really told me what's going on in between that would be superficial an incomplete would be you really don't unpack what are the activities that are linking each part with the next one mechanistic evidence and this is about getting towards the conclusion is then the observable traces left by a particular activity and there could be many different types of evidence that could be relevant not just interviewing people it could be meeting documents it could be statistics or or many different things anything that would tell you something about whether a given activity took place and this figure is a little bit complex I apologize but it's it's linked then into in in the philosophy of science the Bayesian approach where they talk about is this this this two-step between the the very the highest level the the actual entity or activity our theory making propositions about evidence and then having to you have to think about in process tracing for each of these thinking about the propositions about potential evidence to ask yourself questions like do you have to find this fingerprint if you find it are there alternative explanations for finding this particular trace or fingerprint but then you also have to think about well going down to to the actual observational level of actual sources and then this is all of this kind of source critical type of evaluations because you could have in theory maybe a confession from an actor could be in theory very strong evidence but if your confession the actual source for example comes from torture you can't trust it so in theory the confession could be very strong but because your source it cannot be trusted it's not evidence so we do I talk a lot more about this in in the readings I'll share with you this has been a real short you know typically a process tracing when we were trying to teach it for example you teach it over multiple days and and it comes each of these it sounds really easy in theory but like okay well what does a good process Theory actually look like in relation to my own research it's a lot more difficult the process tracing is one way to study mechanisms in theory at least has strong internal validity but it of course can vary a lot depending upon the how much you want to unpack the process and the the quality of the evidence and it's often typically very resource-intensive and you learn a lot about a little you understand okay in in the United Kingdom this policy intervention this is how I think it worked but that doesn't necessarily mean that the same policy intervention is going to work in the same way in Denmark for example because the context is different so so we we will learn about how things work but it can be difficult to have these theories then travel to other cases so you learn a lot about a little about how it works okay and here are just some suggestions I'm not going to for some from further reading there's a couple of things that I've worked on an example of process tracing and policy studies and then there's a book on an edited volume that's come recently talking about making policies work and understanding mechanisms and what you get out of that trying to unpack how things like you know advocacy coalition framework how do those advocacy coalition's actually work in given cases and this is a book then they kind of works through different theoretical paradigms within the study of policy and and really is pleading for more analysis that is basically doing process tracing so I think I will stop there and have time for some questions you

Channel: MRC Epidemiology Unit

Views: 4,883

Keywords:

Id: p22Vo8v2OP4

Length: 43min 46sec (2626 seconds)

Published: Thu May 07 2020