Lecture64 (Data2Decision) Intro to Design of Experiments

hello and welcome to lecture 64 of my class from data to decisions I'm Christmas Mac your instructor and today's lecture is an introduction to design of experiments in fact this is the first a series of lectures to discuss a very important topic designing experiments so far we've been talking about taking data and analyzing it so taking the data and fitting a model to it for example but eventually or earlier somebody had to design the experiment that generated the data that we then collect so we're kind of going backwards of before you even start to make a measurement you have to design the experiment that you want to run well we can think of it this experiment as some kind of a process and this process has inputs and outputs we're gonna try to understand the relationship between inputs and outputs and to do so we have to think about key different kinds of outputs first is the confuse mean different kinds of inputs the first is the controlled inputs these are the things that we're purposely varying in our experiment we wanna Emily what we want to do is try to understand the controlled inputs and how they affect outputs but there are other kinds of inputs as well that are uncontrolled these are things that were not purposely manipulating or we're not able to hold constant and those uncontrolled inputs have two kinds as well we have some uncontrolled inputs that we can observe that is we can measure but we can't control them there are other kinds of uncontrolled inputs that we can that we don't control and we don't even measure them or observe them and we're gonna have to treat these uncontrolled inputs a little bit differently give you an example suppose I'm running an experiment and this experiment is just done and in the lab and the lab has some basic ability to control the temperature and humidity of the atmosphere in the room doesn't have any ability to control the barometric pressure and maybe my every man is a Ellicott optical measurement and picture humidity and barometric pressure X the optical properties of the air the refractive index of the air and so I get slightly different results because of these changes well they're they're sort of controlled in terms of temperature and humidity not perfectly not very well so we can consider them uncontrolled inputs Oh even though they're uncontrolled I can observe them I can measure the temperature humidity the barometric pressure in the room and I can then take advantage of that information is which we'll talk about but there also might be some uncontrolled and unobserved inputs maybe there's some natural vibration of certain frequencies that are wiggling my experimental setup and I don't control it and I don't measure it either well how we think about these three kinds of inputs depends on what we're trying to do with our experiment first of all let me define a term that we'll use again call nuisance inputs nuisance inputs are inputs that we don't really care about I'm not interested in understanding how temperature affects my process or how barometric pressure affects my house process I'm not that I don't care about that that's not my purpose of my experiment but they do affect the outputs so I have to worry about them so that's why we say nuisance right if they don't affect the output then I don't worry about them at all but if someone controlled input does affect the output it's a nuisance input as opposed to a controlled or desired input how we deal with these three classes of inputs depends on what our goals are for for the experiment on goal is modeling or characterization we want to simply understand the process how do input effect outputs but another possibility especially in processes involved in say manufacturing our goal is to optimize and we want to optimize the process so that we get the output that we want what input values produce ask properties of my output usually both the mean by hitting the mean properly or in reducing the variance about that mean another option is a control option it's like the optimization and that you're trying to hit a particular output but in a control problem that target the thing that we're trying to achieve can change so if our target changes what changes in the input allow us to move the output through its new target value that's the process of control and how we deal with characterizing this process depends on which of these types of of characterization and modeling we're trying to achieve that's our goal so how do we deal with these three types of inputs first of all the controlled inputs acts well we use our control over them to perform the experiment vary the inputs we repeat experiments in a systematic way in order to learn about how these inputs affect the outputs the uncontrolled but observe the inputs we have two choices and we're going to talk about both of these in coming lectures one choice is called blocking this is where we group the experiments in the blocks each block having smaller that variation in U or a fixed value of U ideally between blocks from one block to the next block we might have some variation in the input let's go back to the example I said before say barometric pressure what what changes barometric pressure weather low front high front moves through the weather changes we get a different barometric pressure I know a barometric pressure affects this delicate optical measurement and making this experiment that I'm running so what I'd like to do is block up my experiment so all the experiments I want to compare to each other are done over a shorter enough period of time that the barometric pressure is not changing that's usually during one day afternoon something like that and then I another block of experiments on a different day that might have a different barometric pressure but at least within that block they'll have a fixed value uncontrolled but observed input the analysis of covariance is another way of dealing with uncontrolled but observed inputs and that is to model the impact of that input on the output and then subtract out that effect essentially correct my observations for this uncontrolled but observed input and finally we have the uncontrolled and unobserved inputs and those this is the really tough one and the key insight here is Animas ation we're gonna randomized collection of our data performance of our experiment in such a way that on average the uncontrolled and unobserved inputs are can be averaged out the impact I should say those inputs can be averaged out this is zero on average if a sufficient number of measurements of observations to make that happen we use randomization to affect uncontrolled and observed inputs and we use blocking or analysis of covariance for the uncontrolled observed inputs I talk about both of those topics in some humming lectures more detail all right is experimental design we design our experiments for the simple but important goal of getting the most information on the least data principle is simple EDA is expensive collecting data takes time and cost money what we want to do is get maximize the amount of information that we can get from the data that we do collect it always love to have more data everything gets better with more data but sometimes we're limited either because of time or cost so how can we get the most information from that data an experiment is the deliberate variation of one or more variables while observing the effect on one or more responses so we deliberately change an input and observe an output that results this is as opposed to an observation observational study where you observe both the inputs in the outputs and we're not manipulating or controlling them so experiments are the foundation of how we learn things in science the design of experiments is a process for planning experiments who provide valid in the most efficient way possible now I say valid conclusions what I mean by that is myth in the context of this course statistically valid and is we have to have enough power in our experiment so that standard errors around our statistic of interest are sufficiently small that I can draw the conclusion I walk and answer the research questions the purpose of the experiment can be achieved uh and we want to do that efficiently that is I always make these standard errors err bars smaller by acting more and more and more and more data but no matter what there's a limit to the amount of data you collected so how can we get the smallest error bars for the given amount of data we have that's the design of experiments what are we use design of experiments for well there's three major uses first is exploratory work we have a new experimental situation a new environment we're not exactly sure how all the inputs effects the outputs and we want to explore we want to choose between two alternatives maybe process a versus process B we want to look at the key factors that effective response if you think about what is everything that affects the output in this process you might come up with an extremely long list of possibilities so you want to find out what are the key factors the things that affect it most this is a process we call screening then given that you've gotten some set of key factors that that affect the response variables variables you want to study we might use do e to optimize that process this is we're gonna use and talk about of something called response surface model that helps us optimize a process to hit and control target response sometimes we want to maximize or minimize a response we're gonna maximize yield we want to minimize defects for example and sometimes will not simply increase the robustness of our process so that small changes in the inputs only have minimal effect on the outputs this is all part of process optimization we use do e and extensively these kinds of problems and the third way in which we can use design of experiments is for regression or modeling you want to develop a model we want the model to be as good as we can design our experiments to get the best modeling results well since regression and modeling has been the topic of conversation in this class for quite awhile let's talk about that first I'll come back to these other ideas of screening and response surface modeling in some upcoming lectures so what is do e for regression basically I have some given and some number of data points that I'm willing to collect designing for regression means picking the values of the predictor variables in my experiment in order to get best statistical behavior of the thing that I'm interested in an example might be I want the smaller standard errors of the coefficients model or maybe specific coefficients in my model I might want the smaller standard errors of the predictions maybe I want the smallest standard errors of the predictions over the full range of input predictor variables or maybe I want the smallest standard error of the predictions over a particular range of the victor variables and there can be some other go behavior that I'm interested in given a goal that you want to achieve we can find the design that optimizes that goal this process is called optimal design in general it requires some kind of a numerical search for a given model for a few simple cases we can analytically derive what the optimal design is I'm going to talk more about optimal design in the next lecture but let me give you a simple example so that you can understand the basics of it a simple example will be for a simple linear regression in other words I've got a straight line bottle and I want to optimize my design of the experiment for the smallest standard errors of the coefficients slope intercept all right so I've got a straight line model and one predictor variable and I know exactly what the equations are for the variance of the slope and the variance of the estimate of the intercept you see it's a force of function of the standard error of the variance of the residuals it's also provided by this sum of X I minus X bar quantity squared excise the values of the X variables this is what I'm designing I'm deciding what values of X I to use in my experiment X bar anything that minimizes the variance of the slope also minimizes variants of the intercept so how do I do that how can I make this quantity as big as possible Susan yeah as big as possible I want these excise to be as far away from the mean as possible that's what makes this thing large the denominator large therefore the standard error of the slope small so I think you could logically think about it and realize that if I picked half of my data points at the lowest x value that I to explore and the other half of the data points to be at the highest x-value I want to explore then I will get minimum standard error of the slope call this a dumbbell design all the data points are are clumped at the two ends of the range and I'll talk about some of the trade-offs of this but basically it goes exactly the opposite of what we're used to thinking about our designs where we spaced evenly the X values from a min to a max well that spacing of the X values evenly is useful if you want to understand what model is most appropriate if we already have the model that we know is appropriate then it's not the most efficient design show you the example right I'm gonna fit a straight line to a set of data points here's the what's called the space-filling design that is i evenly spaced out to fill the space evenly between a min and the max range and then i fit the line get an r-squared I get a standard error of the slope now suppose instead I did a dumbbell design I piled up half of the data at the minimum value half the data at the maximum value and then I fit a straight line now in this case the standard error of the residuals is exactly the same between these two cases but look at the R square much higher look at the standard error of the slope for this dumbbell design it's much lower fact that dumbbell designs will always minimize standard error of the slope given a certain standard error of residuals now what can't we see from the dumbbell design oh we can't see if our model is a good one or not right I I have no information as to whether or not the true behavior the data looks like an exponential growth or it's leveling off at the top or it's going through some more complicated shape or behavior I can't see that I I'm at the trust an assumption that a linear model is a good one whereas the space-filling design helps me explore possibility of other alternate models so for the case of a straight line model with one predictor I can in fact derive analytical expressions for the standard error as a function of the experimental design so for example for the dumbbell design standard error of the slope or I put half of the data at one extreme and the other half the data at the other extreme is gonna be the standard error of the residuals divided by the square root of the number of data points n divided by half range x max minus X min is the range of the predictor variable divided by two now what if I did the more common space-filling design or my data is spaced evenly apart between the min and the max I can again derive exactly what the standard error is going to be and it turns out that it assuming you know random distribution of errors about that straight line model it's equal to the standard error of the dumbbell design multiplied by something like the square root of three if n is kind of large that turns into the square root of 3 square root of 3 is 1.73 in other words standard error of the slope for my space-filling design is 70% bigger than the standard error I get if I used a dumbbell design it's just like what we saw in the previous slide here's another possible design it's called a quadratic design here I take one-third of the data points I put them at the minimum one-third I put in the middle and one-third of the data points I stick at the maximum value of x for this a graphic design standard error of the slope is the standard error of the dumbbell design times the square root of three halves square root of three halves is about one point two about 20% or standard error now as you might imagine odd Radek design enables me to test the assumption at the the line is straight as compared to an alternative of a quadratic behavior so the quadratic design allows me to check the assumption of a linear model against at least an alternative assumption of a quadratic model I pay the price of about 20% higher standard error in in the slope but not Perry paying nearly the plot by 70% increase in standard error for the space-filling design another goal in experimental design is to equalize the leverage in multiple regression in particular so we've already seen in our previous work on regression that we have to worry about leverage if some points have are more leverage than average those points are much can be much more influential in the fit well that could lead to all kinds of complications it would sure be nice if all of the data points in my design had about the same leverage and they wouldn't have to worry about that well how could we do that one goal of experimental design is to make leverage of a specific point equal to the average leverage of all the points we know the average leverage is P over N where P is the number of parameters in the model n is the number of data points let's take the simple case the case of only one predictor variable I have only one predictor variable here is the exact expression leverage what I want is to make that exactly equal to 2 over N because I only have two parameters how do I do that well here's 1 over N so that may must mean this must also be 1 over n how can I make this 1 over n I can only make this 1 over n if X I and this X bar quantity is squared is a constant the same for every eye that's the case and I will have equal leverage for every data point we go back to our example of the dumbbell design and we see that that in fact is is what it does a dumbbell design makes X I minus X bar quantity squared same for every data point and therefore dumbbell design achieves its minimum standard error the slope by equalizing the leverage of every data point and we can do this a similar kind of thing when we have multiple regression as well all right let me conclude this introduction by discussing the six principles of regression design and we're gonna revisit these principles next time and in coming lectures first of all I got this these six principles from this great mist SEMATECH handbook and statistical methods available online which I refer to frequently there's a link to it on the course webpage as well first of all design experiment or the purpose of regression you want to make sure that experiment has the capacity to do a good job of validating your primary model if if you're trying to fit a straight line generally means having the range of X values to be as large as possible to get the best result you don't want all the X values to be the same value you can't can't get that um but you also want a capacity for an alternate model if you're not sure the prominent primary model is correct you might have an alternate model as well and you'd like the experiment able to allow you to choose between those two so that's called capacity for the alternate model think about straight-line model but maybe as an alternate a graddic model second-order model we would like to have the minimum variance of you the estimated coefficients or predicted values or some similar kind of metric that's what we call the optimal design in everything except very simple cases we have to do a numerical search to find that optimal design we'll talk more about that in the next lecture there's a principle called sample where the variation is when things are constant we don't need to sample very much in that region we need to have repeats and/or replication in order to get some kind of an estimate of the process variation that's independent of our model because we want to both check the variation in the process and check the the goodness of our model and finally we want to employ principles like randomization and packing to allow for the detection of drift influence of of these uncontrolled parameters whether they're known or unknown we'll come back and visit all these principles in the next lecture to go into more detail about design for regression all right have we learned in lecture 64 as always you should be able to quickly and easily answer these questions if not please go back and review the material name the three types of inputs to a process define design of experiments but other three uses of experimental design that we've discussed for regression to a straight line but is the most efficient design and finally when regressing to a straight line what design produces uniform leverage that's it for our lecture today we're going to visit the topic of design for regression more detail in the next lecture till then

Channel: Chris Mack

Views: 15,963

Rating: 4.9619045 out of 5

Keywords: statistics, data analysis, linear regression

Id: UTa-6YCqbBA

Length: 26min 30sec (1590 seconds)

Published: Wed Nov 09 2016