Mar 19, 2024
Do you struggle to meet your goals or targets? Find out how you can change your thinking about goals and your process for setting them so you can keep moving forward. In this episode, John Dues and host Andrew Stotz discuss the first five of John's 10 Key Lessons for Data Analysis.
TRANSCRIPT
0:00:03.0 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode two of four in a mini-series on why goal setting is often an act of desperation. John, take it away.
0:00:32.3 John Dues: Hey, Andrew, it's good to be back. Yeah, in that last episode, that first episode in this mini-series, we talked about why goal setting is often an act of desperation and I basically proposed these four conditions that organizations should understand prior to setting a goal. So it's not the goals in and of themselves that are bad, but it's with this important understanding that's often lacking. So those four things that organizations should understand, one, what's the capability of a system under study? So that's the first thing, how capable is the system or the process? The second thing is what's the variation within that system or process under study? So that's the second thing we talked about last time. The third thing is understanding if that system or process is stable. And then the fourth thing was, if we know all of those things, by what method are we going to approach improvement after we set the goal, basically? So you gotta have those four things, understanding the capability of the system, the variation of the system, the stability of the system, and then by what method, prior to setting a goal. And so I think I've mentioned this before, but absent of an understanding of those conditions, what I see is goals that are, what I call it, arbitrary and capricious.
0:01:48.8 JD: That's a legal characterization. You look that up in the law dictionary. And it basically says that an "arbitrary and capricious law is willful and unreasonable action without consideration or in disregard of facts or law." So I'm just now taking that same characterization from a legal world and applying it to educational organizations and accountability systems, and I just switched it to "a willful and unreasonable goal without consideration or in disregard of system capability, variability, and/or stability." And we see these all over the place for education organizations, for schools, school districts, teachers, that type of thing.
0:02:31.6 JD: And so what I tried to do in the book and tried to do here in my work in Columbus is develop some sort of countermeasures to that type of goal setting and develop the 10 key lessons for data analysis. An antidote to the arbitrary and capricious goals seen throughout our sector. And this process behavior chart tool, looking at data in that format is central to these lessons. So what I thought we would do in this episode and the next is outline those 10 key lessons. So five today and then do another five in the next episode. And in the fourth episode of the series, what we would do is then apply those examples to a real life improvement project from one of our schools. It's helpful, I think too, to sort of, to understand the origin of the key lessons. So there's the lessons that I'll outline are really derived from three primary sources.
0:03:36.0 JD: So the first two come from Dr. Donald Wheeler, who I've mentioned on here before, a lot of Deming folks will, of course, have heard of Dr. Wheeler, who's a statistician in Tennessee, a colleague of Dr. Deming when Dr. Deming was alive and then has carried on that work to this day. The two books, two really great books that he wrote, one is called Understanding Variation, a thin little book, a good primer, a good place to start. And then he's got a thicker textbook called Making Sense of Data, where you get in really into the technical side of using process behavior charts. So I'd highly recommend those. And then the third resource is a book from a gentleman, an engineer named Mark Graban called Measures of Success. And I really like his book because he has applied it, the process behavior chart methodology, to his work and he's really done it in a very contemporary way. So he's got some really nice color-coded charts in the Measures of Success book and I think they're really easy to understand with modern examples, like traffic on my website, for example, in a process behavior chart, really easy to understand modern example. But all three of the books, all three of the resources are built on the foundation of Dr. Deming's work. They're, you know, Graban and Wheeler are fairly similar and I think Graban would say he's a student of Wheeler.
0:05:00.4 JD: He learned of this mindset, this approach to data analysis by finding a Donald Wheeler book on his own dad's bookshelf when he was in college and starting down that path as a young engineer to study this stuff. And basically what I've done is take the information from those three resources and make some modifications so they can be understood by educators, basically. I think it's also worth noting again that process behavior chart methodology is right in the center of this, really for three reasons. One, when you plot your data that way, you can start to understand messages in your data, I think that's really important. Second, you can then start to differentiate between special and common causes, special and common causes, translate that into regular language. I can translate between something that I should pay attention to and something that's not significant basically in my data. And then in so doing, I know the difference between when I'm reacting to noise versus when I'm reacting to signals in my data, so I think that's really important. So the process behavior chart is at the center of all this. So we'll go through five of these lessons, one by one, I'll outline the lesson and then give a little context for why I think that particular lesson is important.
0:06:25.4 AS: That sounds like a plan. So capability, variation, stability and method. You've talked about Donald Wheeler, excellent book on Understanding Variation, that's the one I've seen. And of course, Mark Graban's book, Measures of Success, very well rated on Amazon and a podcaster himself, too.
0:06:49.6 JD: Yeah. And if I was a person studying this and wanting to get into process behavior charts and really knowing how to look at data the right way, I would read Understanding Variation first because it's a good primer, but it's fairly easy to understand. And then I would read Measures of Success 'cause it's got those practical applications now that I have a little bit of a baseline, and then if I wanna go deep into the technical stuff, the Making Sense of Data, that's the textbook that drives everything home. Yeah. So we'll dive into the lesson then.
0:07:19.5 AS: Let's do it.
0:07:20.0 JD: Yeah. Okay. So the first lesson, and I've talked about this in various episodes before, but lesson one, the very first lesson is, "data have no meaning apart from their context." So this seems commonsensical, but I see this all the time where these things aren't taken care of. And what I'm talking about is answering some basic questions. So for anyone looking at my data, they should be able to answer some basic questions, very simply, anybody that looks at my data. First thing is who collected the data? That should be apparent. How were the data collected? When were the data collected? Where were the data collected? And then what do these values represent? So oftentimes I see data either in a chart or in some type of visualization and almost none of those things are known from looking at the data, all important questions.
0:08:18.6 JD: The second question would be, well, that first set goes together. The second question is what's the operational definition of the concept being measured? So we have to be on the same page about what it is exactly being measured in this data that I've collected. I also wanna know how were the values of any computed data derived from the raw inputs? That's important. And then the last thing is, have there been any changes made over time that impact the data set? For example, perhaps the operational definition has changed over time for some reason. Maybe there's been a change in formula being used to compute the data.
0:09:05.4 JD: So an example would be, from my world, high school graduation rates. You know, 20 years ago there was one definition of how you calculated a high school graduation rate, now there's a different definition. So when you compare those two sets of data, you've gotta be careful because you're actually, you're actually working from different definitions and I think that happens all the time. More recently here in Ohio, what it means to be proficient on a state test, that definition changed about 10 years ago. And so if you look at test results from 2024 and try to compare them to 2014, you're really comparing apples and oranges 'cause there's two different definitions of proficiency, but no one remembers those things a decade later. So you have...
0:09:52.3 AS: And then a chart will be presented where the different methodologies are shown as one line that says...
0:10:00.8 JD: Yes.
0:10:00.8 AS: That no one's differentiated the fact that at this point it changed.
0:10:04.6 JD: Yeah, at this point it changed. So first lesson, data have no meaning apart from their context. Second lesson is we don't manage or control the data, the data is the voice of the process. What we control is the system and the processes from which the data come. There's a difference there. Right? So I think this is one of the key conceptions of that system's view, that system's thinking in an organization. When we wanna make improvements in our schools, we need a few things in place. We need the people working in the system. So that would be the students for us, they're working in the system, people that have the authority to work on the system, so that'd be teachers if we're talking about an individual classroom, at the school building level, maybe we're talking about the principal. And those two things are, at least the teacher principal thing is usually in place, the students being a part of improvement projects, definitely less so, but maybe there are places where that's happening. But the third thing is someone with an understanding of the System of Profound Knowledge, I'd say that's almost always lacking in the education sector, at least. And I think the reason the System of Profound Knowledge becomes important, 'cause that's really the theoretical foundation for all the things that we're talking about when we're looking at data in this way.
0:11:38.8 JD: If you lack that conception, then it's hard to bring about any improvement, because you don't understand how to look at that data, how to interpret that data, you don't understand how to run a plan-do-study-act cycle. Because what you're gonna ultimately have to do is change some process in your system and there's some knowledge that you're gonna need to be able to do that, and that's, that third component of an improvement team has to be in place to do that. But I think the most important thing is that we're not in control of the data, we're in control of the processes that ultimately lead to the data. It's a distinction, maybe a fine distinction, but I think it's an important one.
0:12:17.5 AS: The idea of the System of Profound Knowledge and understanding what to do with the data and really understanding the whole thing, I was just thinking what would... An analogy I was thinking about is rain. Everybody understands rain as it comes out of the sky, but not everybody understands how to use that to make a pond, to make an aqueduct, to feed a farm, to, whatever that is. And so having that big picture is key, so, okay. So number...
0:12:57.8 JD: Yeah. Well, and a part of that is something really simple is constantly understanding data is the voice of the process. And so when you're looking at data, what often happens is I'm gonna walk into a meeting with my boss, and I'm looking for some data point, maybe we just got some type of performance data back or survey results or something. I'm gonna pick one of those items where the plot, where the dot from last time has improved when we look at it this time, and I take that and say, "Look how we've improved in this thing." And you need someone to say, "Well, wait a second, while there is a difference between those data points, if I look at the last 12, things are just moving up and down." And there's gotta be someone in the room that constantly points back to that, constantly. And that's where that person with the Profound Knowledge is helpful in improvement work.
0:13:54.5 AS: So the voice of the process is a great way of phrasing it that's been used for a while now and I think it's really good. I remember when I worked at Pepsi as a young supervisor, I saw some problem on the production line and I raised it to the maintenance guys. And they kept coming and fixing it and it would break and they'd fix it and it would break, and I basically got mad at him and I was like, "What the hell?" And he's like, "Bosses won't pay for the things that I need to fix this permanently, so get used to it constantly breaking down."
0:14:33.4 JD: And that's the best I can do.
0:14:34.0 AS: That's the voice of the system, here's what I can produce with what you've given me to produce.
0:14:40.8 JD: Yep. Yep. Yeah. Those guys had a very keen understanding of the system, no doubt in that example. Yeah. Yeah. And that kind of thing happens all the time, I think. That was lesson two. Lesson three is plot the dots for any data that incurs in time order. So a lot of people in this world know Dr. Donald Berwick, he started the Institute for Healthcare Improvement. He was a student of Dr. Deming's, he's done a lot of work in this area. He has a great quote where he says, "Plotting measurements over time turns out, in my view, to be one of the most powerful things we have for systemic learning." And that's what really plot the dots is all about, it's all about turning your data into a visualization that you can learn from. And the National Health Service in England has this #plotthedots. And I think the whole point is that plotting the dots, plotting the data over time helps us understand variation and it leads us to take more appropriate action when we do that. So whether it's a run chart or a process behavior chart, just connecting the consecutive data points with a line makes analysis far more intuitive than if we store that data in a table.
0:16:03.6 AS: Yeah. And I was thinking about if you're a runner and you wanna compete in a marathon, plotting the dots like that is so valuable because you can see when changes happen. For instance, let's just say one night you didn't eat and then you ran the next morning and then your performance was better. Was it just a noise variation or is there something that we can learn from that? And then just watching things over time just give you ideas about what... Of potential impacts of what something could do to change that.
0:16:42.0 JD: Yeah. And we can start with a simple run chart, it doesn't have the limits, it's just a line chart. And then once we have enough of the data collected, enough plotted dots, then we can turn it into the process behavior chart.
0:16:56.3 AS: Some people don't even want to see that, John, like when we looked at your weight chart, remember that?
0:17:03.0 JD: I do remember that. Yeah.
0:17:04.0 AS: So for the people out there that really wanna let's say, control your weight, put a dot plot chart on your wall and measure it each day and just the awareness of doing that is huge.
0:17:18.7 JD: Yep. It is huge. It really is huge. And that works for any data that occurs over time, so almost everything that we're interested in improving occurs in some type of time order, time sequence. So these charts are appropriate for a wide array of data. But the bottom line is that... Oh, yeah, sorry, go ahead.
0:17:33.5 AS: The bottom line?
0:17:35.0 JD: Well, I was just saying the bottom line, whether you're using a run chart or a process behavior chart, it's always gonna tell us more than a list or a table of numbers, basically.
0:17:44.5 AS: I was gonna explain this, a situation I had when I was head of research at a research firm, a broker here in Thailand. I, my goal was to get more output from the analysts, they needed to write more and we needed to get more out. So what I did and I had already learned so much about Deming and stuff at that time. So what I did is I just made a chart showing each person's, what each person wrote each week, and it was a run chart in that sense where people could see over time what they wrote and they could see what other people were writing. And I purposely made no comments on this chart and I'd never really discussed it, I just put it up and updated it every week. And one of the staff that worked for me, an analyst, a really smart Thai woman asked, she said, she went to... She said, "I wanna see you in your office." I was like, "Oh, shit, I'm in trouble." And so she came to my office and said, "You know I went to, so this was maybe six months after I had put that chart up, she said, "I went out to lunch with my counterpart, my competitor, and she's writing research just like me on the same sector, and she asked me how many research reports do you write in a week, and I told her my number, and she was like, "Oh my God, that's a huge number."
0:19:16.6 AS: And she said, "Oh, I didn't really even think about it. But okay." And then she says, "What is Andrew's goal or target for you?" And she had naturally had thought that I had set a target of that amount, that's where she said, "I think I really figured you out." And I was like, "Well, what do you mean?" She said, "You just put that chart up there and you didn't give us any goal, but you knew that we were looking at it, and then it would provide us information and incentive and excitement, and the fact that you said nothing about it, got us to probably a higher level of production than if you had said, "I want everybody to read my reports."
0:19:57.9 JD: Right. Yeah, that's great.
0:20:01.0 AS: The magic of data. What's number four?
0:20:02.4 JD: The magic of data. Number four, so two or three data points are not a trend. So the first thing is, as soon as you've decided to collect some set of data, plot the dots, that should start right away. And again, this really includes all data that we're interested in improving in schools. And I know before I understood this way of thinking, this way of data analysis, I often relied on just comparing two points, that's the most common form of data analysis. What did last month look like, what does month look like? What did last year look like, what does this year look like? What did last week look like, what does this week look like? So that limited comparison is the most typical form of data analysis, especially when you're talking about something like management reports or board reports, revenue over time, those types of things. What was revenue last January? That type of thing. But the problem with looking at just two or three data points is that it tells you nothing about trends, it also doesn't tell you anything about how the data varies naturally.
0:21:17.5 JD: I remember looking at attendance data at one of our schools, and they had up... Last month was 92%, and then had gone up to 94%, but then I just said, well, what did it look like... January is 92, February is 94 in this particular school year, and I just said, well, what did it look like before, and then when you plotted it, what saw very quickly is there was no improvement, the data was literally going like this, up, down, up, down, up, down, up down, right? But no one had that picture, because all you could see was, Here's January and here's February, just numbers written in percentage form, that's almost all the data that I see in schools is in a similar format.
0:22:02.7 AS: On this one, in the stock market, my area of expertise. People always see the up data, the people who have made a lot of money in the stock market, and they see that as evidence that they could make money in the stock market, or they attribute that to skill of that particular person as we want to, with Warren Buffett as an example. And I have, in fact in my class, I asked the students, "Do you think that Warren Buffet outperformed, underperformed, or performed in line with the market over the last 20 years?" And the answer to that is, he performed in line with the market, and I proved that by doing a demonstration through a website that I can do that with, but it was shocking because obviously he's gonna end up with the most amount of money because he let his money compound, and he made huge gains in the beginning years, which compounded over many years.
0:23:02.0 AS: And still he's doing very well, but the point is, is that... The reason why I say this, I also tell the story of, if you had 10,000 people in a stadium and you flipped coins, and asked them if they flipped heads consecutively or tails consecutively to remain standing, and you're gonna end up with 10 people at the end of 10 flips with 10,000, and if you've got a million, you can end up with 20 or 30 or 40 flips that could potentially be heads consecutively or tails consecutively. So my question is, given that long streaks can happen through just plain probability, what if two to three data points are not a trend, can we definitively say, what is a trend?
0:23:50.9 JD: Well, not with certainty, but what this type of data analysis does is it gives you some patterns in the data to look for that are so mathematically improbable that you can be reasonably assured that some changes happened.
0:24:09.4 AS: Right so this is enough of a trend that I'm gonna go with the assumption that there's something significant here.
0:24:21.9 JD: Yeah. I mean it's...well, think back to that attendance example that I just used, so if I went from... If I'm writing this up on, let's say a whiteboard that's in a teacher work room, it says, this month and next month, or last month and this month, and I write those attendance rates up and remember, it's a dry erase board, and I'm gonna erase the last month to put this month's up and so I'm not gonna be able to see that one anymore, I'll have two data points and I'll erase the old one, and so in that example, I used where they went from 92%. It was actually like 92.4% to 94.1%. So it wasn't even two full percentage points. And then you celebrate that as a win, as an improvement, but like I said, you didn't know what happened before, and then you didn't chart after, so you don't really know how things are just bouncing around naturally versus if you had it on a run chart and you did see, let's say, eight points in a row that are above the average attendance for that school, that's one of the patterns that suggest that something different has happened. So you just have increased mathematical probability that there has been meaningful improvement.
0:25:39.4 AS: So it sounds like what you mean in this number four is a little bit more on the end of, Hey, just a couple of data points doesn't have anything, you need to get more rather than somebody looking at a lot of data and trying to understand what is a trend or not?
0:25:56.9 JD: That's exactly right. That's exactly right. And that actually is a segue to Lesson 5, which is "show enough data in your baseline to illustrate the previous level of variation," basically. So this is gonna get a little technical for a second, but the non-technical thing is, we talked about when you have a run chart when you're starting and you have, let's say, three or four or five, six data points at a certain point, you can now have a process behavior chart, which is the addition of that upper and lower natural process limit that defines the bounds of the system, so the limits are not a part of the run chart.
0:26:34.9 JD: In making sense of data what Donald Wheeler basically says is that if you're using an average line, the mean for your central line, then those limits, you begin to have limits that solidify when you have 17 or more values, and then if you're using a median for that central line, that solidification starts to happen when you have 23 or more values. So there's a mathematical theory behind that. But the point is, at a certain point, you start to get enough data to be able to add the limits and feel confident that those limits actually represent the bounds of your current system. But that's getting fairly technical and what Wheeler does go on to say is that, in real life we often have fewer data points to work with.
0:27:31.4 JD: So you can actually compute limits with as few as five or six values, and they can still be meaningful, now they're gonna be not a solid, meaning that each individual data point for a while that you add could potentially shift those limits more than you'd like, because there are a few data points that the limits are based on. But once you get to 17, 18, 19, 20 points, they start to solidify pretty good unless there's some significant change, like one of those patterns I talked about in your data. But an important thing to keep in mind is, is we're using a process behavior chart for continual improvement, so we're taking improvement measurements, not accountability measurements. I'm not trying to paint a certain picture of what my system looks like, I'm not trying to write a fiction about what's happening in my system, I'm actually trying to improve, so I don't really care what the data looks like. I'm not worried about being judged or rated or ranked, it's not an accountability thing, it's an improvement thing. And so I'm just trying to represent the system accurately so that I actually know that what I'm trying is working or not working. It's a completely different mindset. That whole sort of like trying to look better is completely removed from the picture through this type of mindset.
0:28:55.7 AS: I'm just picturing some sort of process where there's a measurement of temperature and the temperature keeps rising, but the worker says, "Boss there's a fire." And the boss said, "There's not enough data yet to confirm that." It only seems like a small fire right now, so I need more data points. Well, sometimes you have to act without thinking about the data and make an assumption that you may be wrong. You turn on the fire sprinklers, boom, and it wasn't a fire, but the damage of letting that go for long and saying I need more data doesn't make sense.
0:29:34.1 JD: Yeah, yeah, that doesn't really work. But the idea with the baseline is, basically, if you wanna improve something, the first thing you do is before you try anything, just gather some baseline data first so you can understand the current conditions. And in that attendance as an example, maybe you don't wanna wait for monthly attendance data, maybe wanna look at daily attendance day, what you have in a school, and just plot that over 12 days, 15 days, two or three weeks, and you can start to get a sense for what this looks like on a daily basis, and then you could try to improve it and see if that improvement has an impact on the data over time.
0:30:15.6 AS: Good, well, let me summarize this, but I have to start off with... My grammar is not particularly great, and since you're more of a school teacher than I am, I may need help with what you said. I think what I got correctly was data have no meaning apart from their context.
0:30:33.6 JD: Yeah, what did I say? Let me see.
0:30:38.5 AS: I always get confused if data is plural or singular.
0:30:41.8 JD: Yeah. Well, it can be either. So in this case, I was using data as a plural, so that's my point. I think technically the singular of data is actually datum. Obviously, nobody uses that 'cause it sounds really weird, but data can be plural, I think so.
0:30:57.4 AS: That sounds awfully Latin of you, alright. Number two, the data is the voice of the process, and that we control the process, not the data, and number 3 we plot the data in time order. Number four, two or three data points are not a trend. And number five is show enough data to illustrate the baseline. Anything you need to say to wrap all this up.
0:31:20.4 JD: Yeah, I just think that... I've mentioned this multiple times. I think when you're talking about continual improvement, primary tool is that process behavior chart, it allows you to visualize your data in a way that makes sense, and then the skill set that you have to learn is how to interpret the process behavior chart. How to use them effectively, how to create useful charts and then underlying... Understanding that underlying logic of process behavior charts. There's other tools, obviously in the improvement tool kit, but I actually think that that particular chart is the most important in my view. And I think with those charts, that tool in hand, we can avoid then those arbitrary and capricious goals that are so pervasive in our sector, basically.
0:32:10.6 AS: Well, that's exciting, and I'm excited for our next session when we talk about the final five. So John, on behalf of everyone at The Deming Institute, I wanna thank you again for this discussion, and for listeners remember to go to deming.org to continue your journey. And you can find John's book, Win-Win: W. Edwards Deming, the System of Profound Knowledge and the Science of Improving Schools on Amazon.com. This is your host, Andrew Stotz and I'll leave you with one of my favorite quotes from Dr. Deming, "People are entitled to joy in work."