tag:blogger.com,1999:blog-14106445668876206722024-02-06T21:18:24.965-08:00Janelle KleinAnonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.comBlogger35125tag:blogger.com,1999:blog-1410644566887620672.post-74373492864099747432014-06-28T13:17:00.006-07:002014-06-29T05:56:02.421-07:00Constant Urgency is a Symptom of a Hazardous EnvironmentWhen we use analogies like land-mines, fires, and various types of hell, we're referring to the hazards in our everyday work. We may not be working with sharp objects or molten metal, but the anxiety, stress and exhaustion from working in a hazardous environment is very real.<br />
<br />
From the outside, these problems are largely invisible. So despite the hazardous work, we are usually expected to work without making mistakes. We never have time to fix the hazards because there's always something more important to do. Building the tools we need for safe development like failure recovery, diagnostic support, adequate logging and reliable deployment are often deferred in favor of more features.<br />
<br />
Then when something explodes, as it inevitably will, high-risk heroics are required to save the day. We work late nights and weekends repairing complex problems by hand and hope that nothing else goes wrong. <br />
<br />
Instead of recognizing the symptoms of a serious problem, the long hours and heroics are often rewarded. Fire-fighting, overtime, and last-minute hacks start to be expected. Constant stress and exhaustion become the norm. <br />
<br />
<h3>
<b>More people just add fuel to the fire.</b></h3>
<br />
Those that don't want to put in the long hours anymore are seen as not pulling their weight. Frustration builds and the team gets burned out and the best developers start to leave.<br />
<br />
The new guys just make things worse. They don't know the software and the hazards to watch out for and they keep messing things up in the code. We try to hold things together, but it's hard to get anything else done. It becomes a full-time job just to keep the system from falling apart. <br />
<br />
Management doesn't understand why productivity is so poor and tries to add more people to the work. This just adds fuel to the fire.<br />
<br />
Once this cycle gets started, it's hard to turn things around. We get sucked into the problems, operating in a mode of constant urgency, and we don't want to see our project fail. So we push ourselves to the limit of stress and exhaustion doing the best we can. However, we're so busy reacting to all the things going wrong, there's no time to stop and fix the problems. One more late night and a few hacks to get things working, but the cycle just doesn't end.<br />
<br />
<h3>
<b>We knew better, but we did it anyway</b></h3>
<br />
The worst part about this is even when we know better, we do it anyway. <br />
<br />
I remember one night in particular after working 60+ hour weeks for several months. I checked in some code without running it at all and deployed my changes so I could test it in production. I was so used to working under constant urgency, I had eventually thrown all my sense of principle out the window.<br />
<br />
We had built out the delivery infrastructure and automated our release process from the beginning. For a while, we were releasing every week; there were challenges, but for the most part things were going fairly well. We had a major deadline coming up to support a new customer on our platform and investors had been promised it would happen by the end of the year.<br />
<br />
The requirements meant drastically changing parts of the architecture and conquering some extremely difficult problems. How long was it going to take? We had no idea, but we did know we had better get to work!<br />
<br />
We broke down the work and started chipping away at it, trying to do just enough unit testing to get by. We paired on the more challenging parts and tried to parallelize the work to get it done as fast as we could. We tried to integrate early, but there were so many problems. The software produced weird results. We just had to work through it.<br />
<br />
<br />
<h3>
</h3>
<h3>
We were caught up in the cycle</h3>
<br />
Some of us worked on testing and fixing, while others kept pushing along with the remaining features. We knew we were headed down the path of a monstrous release, but we didn't seem to have any choice. We worked an insane amount of hours troubleshooting problems just trying to get it stable.<br />
<br />
The end of the year was rolling around and we finally got the software in production. We thought the pain was finally over, but that was just the beginning. We had no time to build out the infrastructure we needed to make changes safely, and our new users had a long list of complaints. The pressure just never let up.<br />
<br />
Every release it seemed like things would go wrong. We'd work all weekend and be up late Sunday night trying to fix deployments that went wrong. The data would be messed up. Reports wouldn't be right. We didn't really have a viable plan B. The system was down, it took too long to restore from backup, we just had to fix it in production. <br />
<br />
<br />
<h3>
</h3>
<h3>
Something had to give...</h3>
<br />
We were so exhausted, but the urgency didn't end. We were yelled at and threatened whenever things went wrong, but expected to continue the high-risk work. How could they possibly give us bandwidth for work that wasn't part of the deliverables, when the project was already several months behind schedule?<br />
<br />
We had poured so much of our time into the software and the people on the team were my friends. We had great developers that had always been disciplined engineers and we all got sucked into the same trap. <br />
<br />
Sometimes you just have to leave. Working under threat and constant urgency makes great people do really stupid things.Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-53235915572204108632014-06-14T07:27:00.001-07:002014-06-14T07:27:11.312-07:00Designing Effective Teams<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">The same things that make for good software make for good team structure. We need high cohesion within a team and low coupling between teams. If people need high bandwidth communication across team structures to do their jobs effectively, the team structures are usually pretty dysfunctional. Likewise if the members of a team don't have a need to talk to each other, they don't really operate as a team either.</span><br />
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
Team structure is a design problem. Developers can be quite good at it, once they start to look at it that way. Designing the team structure around the architecture has a lot of benefits. However, if you have a hairball interdependent architecture, you can't build a good team structure around it. Trying to throw more people at the problem and artificially carve it apart is often where software organizations fail. </div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
Trying to go faster and throw more people at it often results in going *slower*. Teams get stuck in a trap of trying to police the code with reviews and there's no way to keep up. The best resources can no longer be productive because they spend all their time reacting to the system that is busting at the seams. Until the team learns a way to design the system in a way that *communication* can be scaled, leaders need to keep their foot off the accelerator pedal. We need the time to invest in that critical learning.</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">
I don't think it's a hands off, let the team figure it out kind of problem. Organizational design is challenging problem and we need leadership to help figure it out. But we need leaders that listen to their engineers, that know what to look for, and have an appreciation for the challenges of our craft.</div>
Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-34390168864500842452013-10-10T19:41:00.003-07:002013-10-10T19:42:44.507-07:00My new dev coaching...<div style="background-color: white;">
<span style="color: #222222; font-family: arial, sans-serif;">Hey all,</span><br />
<span style="color: #222222; font-family: arial, sans-serif;">Would you mind helping me to send this out to anyone you know that might be interested? And a personalized endorsement of me would help too if you know me. :) </span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">I'm trying to get the word out... trying to build a developer coach-to-get-a-job program. :) My new business experiment.</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">J</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">-------------</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><b>FREE Personal Development Coaching</b></span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">Want to learn how to write cleaner code, or write more effective unit tests that aren't so annoying to work with? How about learning a dev approach that will keep you feeling in control of your code, so the behavior stays predictable?</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">Sign up for personal one on one development coaching with me! I'll meet with you, and teach you, and show you how.</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">The catch:</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">1) I'm only taking 5 people, because I need my sanity. :)</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">2) In exchange for being coached, you have to let me find you an awesome job that will help you grow further. It's the awesomest place I know of to work in fact (other than New Iron of course).</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">3) Java takes preference, because I know it well.</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">4) And you have to be able to pass my interview. :)</span><br />
<span style="color: #222222; font-family: arial, sans-serif;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif;">Send me an email at janelle@newiron.com if you'd like to sign up. Since I have limited time, please say a few words about why this is important to you too, please. And feel free to forward to anyone you think might be interested!</span><br />
<div style="color: #222222; font-family: arial, sans-serif; font-size: 18px;">
<br /></div>
</div>
Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-20565182440169443432013-10-09T18:51:00.001-07:002013-10-09T18:51:37.364-07:00Lovin Life<span style="background-color: white; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 13.333333969116211px; line-height: 18.88888931274414px;">I think my most productive days have been those roll out of bed and work days. Where you're just so into what you're doing it's the the first thing that pops into your head when you wake up. And you can't sleep because ideas keep popping into your head that you have to jot down. I just love the rush from being so excited about what tomorrow will bring, and ideas just coming together in the right way, its almost magical. I have an awesome life.</span><br />
<span style="background-color: white; color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: 13.333333969116211px; line-height: 18.88888931274414px;"><br /></span>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;">Two awesome breakthroughs: </span></span><br />
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;"><br /></span></span>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;">I figured out a working capacity model for shared business resources so that I could map capacity cost per client & revenue per client. Talk about some amazing discovery opportunities. What I've learned over the last week has been incredible.</span></span><br />
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;"><br /></span></span>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;">And the other one, I figured out how to measure understandability and controllability in software so that the metrics matched my subjective evaluation. I'm way excited about that - it's a huge breakthrough and the crux of my book and research effort! I was so close, and could still make some things work ok, but everything seems to be falling into place.</span></span><br />
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;"><br /></span></span>
<span style="color: #333333; font-family: lucida grande, tahoma, verdana, arial, sans-serif; font-size: x-small;"><span style="line-height: 18.875px;">I'm so incredibly excited.</span></span><span style="color: #333333; font-family: 'lucida grande', tahoma, verdana, arial, sans-serif; font-size: x-small; line-height: 18.875px;"> </span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-56589340068725330742013-08-14T13:23:00.002-07:002013-08-14T14:07:06.309-07:00Got Database Pain?I'm doing free brown-bag talks at companies around the community. I have a lot of really great material after dealing with DB struggles in lots of different environments, and learning a lot along the way. We'll go through patterns of common mistakes and how to reduce them, and strategies for making mistakes less costly when they do happen. If you're interested, feel free to send me an email and we can schedule it!<br />
<br />
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
<b><span style="font-size: medium;">Database CI: Practical Strategies for Reducing Database Pain</span></b></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
It's a common challenge - the database gets in the wrong state, the release is delayed, and the entire team is blocked and waiting for the centrally shared resource to be fixed. Recovering from database mistakes can be quite painful. Most of the tools available don't really solve this problem either - migrations focus on automating the <i>deployment</i> of the scripts, but don't help much in <i>developing</i> scripts that work to begin with.</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
But how can we reduce the number of mistakes? And how can we detect our mistakes as early and cheaply as possible?</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
<br /></div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
In this presentation, we'll discuss strategies for reducing database costs with mistake-prone systems. By adapting continuous integration principles to database work, we can drastically reduce the costs involved with database changes. With practical examples and patterns for organizing SQL and database build automation, we'll cover strategies for many challenging issues:</div>
<div style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13.333333969116211px;">
<ul>
<li style="margin-left: 15px;">Managing packages, procedures and views in the database</li>
<li style="margin-left: 15px;">Changes (or mistakes) that can't be rolled back</li>
<li style="margin-left: 15px;">High data volumes that make everything take longer</li>
<li style="margin-left: 15px;">Systems that are hard to keep in your head</li>
</ul>
<div>
To register for this free brown-bag lunch presentation at your company, send us an email at <a href="mailto:contact@newiron.com" style="color: #1155cc;" target="_blank">contact@newiron.com</a> and schedule a date! </div>
</div>
<b><br /></b>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-30190863022097395972013-08-10T07:09:00.000-07:002013-08-10T07:09:02.992-07:00Factors in DB Development CostI've been working on developing a new database CI tool suite, and was talking with a friend about his DB woes. We talked about what problems he was having, where they occurred, and how long they were taking to resolve. His database was really simple. The scripts were pretty much always correct. But some scripts might be forgotten, and his problems usually revolved around tracking down missing changes.<br />
<br />
I realized my solution didn't fit his problems at all. I had never worked in an environment that was all that simple. There were always mistakes in the scripts. And it was always painful to correct them. When things went wrong, it was like the DB became the black hole of engineering hours, sucking away everyone's time. We'd try to repair the mistake by hand. Or if we couldn't figure out how to repair it, we'd have to restore from a snapshot of production. And while all these repairs were going on, the engineering department was pretty much down.<br />
<br />
Making mistakes was so painful. And trying to use any of the database migration tools out there didn't seem to solve my problems. In some cases, it even made them worse. I always ended up resorting to rolling my own custom database tools.<br />
<br />
So that got me thinking... how do you decide what kind of solution you need? <br />
<br />
While there are many complex factors that drive DB development costs, there are two that seem to characterize the problem space quite well: frequency of mistakes, and cost of recovery.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzM-vRRUlOxiQpaAUxFAfqnFffWQIQQObt9_bX556fJmpN-ylrpsa9bpR_dNv7fUo15SeOvicTx5MuRbdxvuKdaCnU7dl9gQz382FYT84fx9WWsM1qckPJMxLq09CEIAhzZapFNFO6T-MO/s1600/Screen+Shot+2013-08-10+at+8.30.33+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="245" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzM-vRRUlOxiQpaAUxFAfqnFffWQIQQObt9_bX556fJmpN-ylrpsa9bpR_dNv7fUo15SeOvicTx5MuRbdxvuKdaCnU7dl9gQz382FYT84fx9WWsM1qckPJMxLq09CEIAhzZapFNFO6T-MO/s400/Screen+Shot+2013-08-10+at+8.30.33+AM.png" width="400" /></a></div>
<br />
When mistakes are rare, and recovery costs are cheap, existing migration tools are the perfect solution. Most of the effort is spent in keeping databases up to date, and making sure all the scripts are deployed as the changes evolve. Migrations are excellent at solving that problem, and handle it beautifully.<br />
<br />
When mistakes are costly, migration tools can still be very helpful, but depending on how rarely mistakes occur, and how costly they are, migrations might not be enough. Augmenting a simple migration tool might be a good option.<br />
<br />
When mistakes are frequent, there's an entirely different kind of problem going on. Developers aren't struggling with deploying the scripts, they're struggling to create correctly working scripts. Migration tools tend to be intolerant of deploying and recovering from broken scripts. And the tools can make recovery even more cumbersome by imposing additional constraints. <br />
<br />
When frequent mistakes are also expensive to resolve... well, life is pain. There's really only two strategies to a less painful life--figure out how to make fewer mistakes, or figure out how to make mistakes easier to recover from. <br />
<br />
I haven't found much help in this space in either the open source or commercial market, which is why I set out to try to fill that gap. And I've built custom tools for solving similar problems several times over now, and had the opportunity to make a lot of mistakes. Now I get to do something with all that learning, and hopefully help reduce some of the pain out there. :)<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-22263238256316091082013-08-09T17:13:00.000-07:002013-08-09T17:13:22.554-07:00Wow how time flies...Wow, has it ever been a while. It was January, February, then suddenly it's August. How the hell did that happen?<br />
<br />
Some challenges came up at work that I had to jump in and get involved with, and most everything in my life started going on hold... including my book, and all the community stuff I wanted to do. I got through writing chapter 4 back in February, and that's still where I'm at. But I'm excited to announce having 2 uninterrupted days per week to focus just on going after making this happen. I couldn't be more excited!<br />
<br />
Since then the ideas started whizzing around in my head again. After going to lunch with an old friend of mine, I was reminded of an idea I had been struggling with. It had been at least a month, trying to make sense of something I didn't quite get, without words to describe what I thought was there. But like a flash in my mind, he gave me a piece to the puzzle I was missing. It was beautiful. <br />
<br />
He was reading Thinking Fast and Slow, a book I read half of, and actually put down. He said, "Association triggers from specific to general." And that it had really struck him, "specific to general."<br />
<br />
After reading On Intelligence, and working through the Art of Learning myself, I had been thinking about memory sequencing, and how it might impact our ability to recognize patterns. Like when you see some messed up code, why is that sometimes an idea pops into your head about how to solve it, and other times it doesn't? Why does this seem to happen more in some developer's heads than others? Is this something that can be learned? And can we learn how to do it faster? This was the puzzle I was working on.<br />
<br />
Specific to General. <br />
<br />
We teach developers design patterns by handing them a book of design patterns. A collection of "aha" moments from our predecessors. But then armed with our new knowledge, we don't seem to have an ability to apply them. When we see our own code, with our own problems, that flash of insight just doesn't happen. Then for some, with experience, it happens.<br />
<br />
Well, you could just say it's experience. But couldn't we tailor the creation of the right experiences so that you would learn what you need to learn faster?<br />
<br />
My friend had the key to my puzzle. Specific to General. The memory sequencing is critical, and the specific sequence of recognition is the opposite of what we teach.<br />
<br />
If I want to have insight that leads to a design pattern, I need to experience specific problem instances that map to the more general pattern. When I scan the code, the similarity of structural pattern to my specific memory should trigger recognition.<br />
<br />
Going to have to pick up that book again I think. :)Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-62501224263188050702012-12-03T06:18:00.001-08:002012-12-03T06:36:15.994-08:00My book is live!I finally got my book effort officially kicked off now. I'm planning on iterating through chapter development and getting the first early release done in January. Also kicking off my new community group in January for Austin DIG and the Idea Flow project, a lot to do!<br />
<br />
My awesome friend Wiley also helped me design the book cover. :)<br />
<br />
<a href="http://leanpub.com/ideaflow">http://leanpub.com/ideaflow</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_ncWr3QQ2vtXfI-P5HuBLNQgKhUAbIDAXXWLPDxsjCl8o9tW967-f4ybaqsrAZoGcM4cDrwoQbs8o8mumJ18o32WuqhLT0ZFgCQH0JJKDLfzUYGp_HGaUk3EhEU2ZH8L3fiQxWxDjAwEp/s1600/idea+flow_blue-01.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_ncWr3QQ2vtXfI-P5HuBLNQgKhUAbIDAXXWLPDxsjCl8o9tW967-f4ybaqsrAZoGcM4cDrwoQbs8o8mumJ18o32WuqhLT0ZFgCQH0JJKDLfzUYGp_HGaUk3EhEU2ZH8L3fiQxWxDjAwEp/s320/idea+flow_blue-01.png" width="228" /></a></div>
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-62909272626283631152012-07-21T18:37:00.001-07:002012-07-21T18:37:51.599-07:00Breaking Things That WorkI went to NFJS today, and went to a talk on "Complexity Theory and Software Development" by Tim Berglund. It was a great presentation, but one idea in particular stuck out to me. Tim described the concept of "thinking about a problem as a surface." Imagine yourself surrounded by mountains, everywhere you look - you want to reach as high as you can. But from where you are, you can only see potential peaks near by. Beyond the clouds and the mountain range obstructing your view, might be the highest peak of all.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgj1Ful6hUAacH95SScQ32nwzv5EqGOOqkEyDWBGmHL3Oq1f0U_WFDPu63mWWwPxRC1hwWWFS1TdoFPfltiPg6W7S2uahAUBRPXX02H2cpCdmhPP6194x-Eoy8OeP8j3UIKjnRv8-fCGC7t/s1600/slide2.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="135" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgj1Ful6hUAacH95SScQ32nwzv5EqGOOqkEyDWBGmHL3Oq1f0U_WFDPu63mWWwPxRC1hwWWFS1TdoFPfltiPg6W7S2uahAUBRPXX02H2cpCdmhPP6194x-Eoy8OeP8j3UIKjnRv8-fCGC7t/s320/slide2.jpeg" width="320" /></a></div>
<br />
With agile and continuous improvement come a concept of tweaking the system toward perfection. But a tweak always takes me to somewhere nearby - take a step, if it wasn't up, step back to where I was. But what if my problem is a surface, and the solution is some peak out there I can't see? Or even if it is a peak I <i>can</i> see... if I only ever take one small step at a time, I'll never discover that other mountain...<br />
<br />
Maybe sometimes we need to leap. Maybe sometimes we need to break the things that are working just fine. Maybe we should do exactly what we're not "supposed to do", and see what happens. <br />
<br />
Now imagine a still pool of water... I drop in a stone, and watch the rings of ripples growing outward in response. I can see the reaction of the system and gain insight into the interaction. But what if I cast my stone into a raging river? I can certainly see changes in waves, but which waves are in response to my stone? It seems like I'll likely guess wrong. Or come to completely wrong conclusions about how the system works.<br />
<br />
With all to variance in movement of the system - maybe it takes a big splash to in order to improve our understanding of how it works? Step away from everything we know and make a leap for a far away peak?<br />
<br />
Here's one experiment. <span style="background-color: white;">We've noticed that tests we write during development provide a different benefit when you write them vs when they fail and you need to fix them later. How you interact with them and think about them totally changes. So maybe the tests you write first and the ones you keep, shouldn't be the same tests? We started deleting tests. If we didn't think a test would keep us from making a mistake later, it was gone. We worked at optimizing the tests we kept for catching mistakes. But this made me wonder about a pretty drastic step - what if you designed all code with TDD, then deleted all your tests? What if you added tests back that were only specifically optimized for the purpose of coming back to later? If you had a clean slate, and you were positive your tests worked already, what would you slip in the time capsule to communicate to your future self?</span><br />
<span style="background-color: white;"><br /></span><br />
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-76855928539504043962012-06-08T09:05:00.000-07:002012-06-08T09:05:00.957-07:00What Makes a "Better" Design?<span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">An observation... there are order of magnitude differences in developer productivity, and a big gap in between - like a place where people get stuck and those that make a huge leap. </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">Of the people I've observed, it seems like there's a substantial difference in the idea of what makes a better design, as well as an ability to create more options. Those that don't make the leap tend to be bound to a set of operating rules and practices that heavily constrain their thinking. Think about how software practices are taught... I see the focus on behavior-focused "best practices" without thinking tools as something that has stunted the learning and development of our industry. </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">Is it possible to learn a mental model such that we can evaluate "better" that doesn't rely on heuristics and best practice tricks? If we have such a model, does it allow us to see more options, connect more ideas? </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">This has been my focus with mentoring - to see if I could teach this "model." More specifically a definition of "better" that means optimizing for cognitive flow. But since its not anything static, I've focused on tools of observation. By building awareness of how the design affects that flow, we can learn to optimize it for the humans. </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">A "better" software design is one that allows ideas to flow out of the software, and into the software more easily.</span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-31872208082992038382012-06-04T16:36:00.001-07:002012-06-04T16:36:18.088-07:00Effects of Measuring<span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">As long as measurements are used responsibly, not for performance reviews or the like, it doesn't affect anything, right?</span><br />
<span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;"><br /></span><br />
<span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">It's not just the measurements being used irresponsibly - the act of measuring effects the system, our understanding, and our actions. Like a metaphor - metrics highlight certain aspects of the system, but likewise hide others. We are less likely to see and understand the influencers of the system that we don't measure... and in software the most important things, the stuff we need to understand better, we can't really put a number on. </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">Rather than trying to come up with a measurement, I think we should try and come up with a mental model for understanding software productivity. Once we have an understanding of the system, maybe there is hope for a measurement. Until then, sustaining productivity is left to an invisible mystic art - with the effects of productivity problems being so latent, by the time we make the discovery, its usually way too late and expensive to do much about it. </span><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><br style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;" /><span style="background-color: white; font-family: Arial, Helvetica, 'Nimbus Sans L', sans-serif; font-size: 13px; line-height: 16px;">Productivity understanding, unlike productivity measuring, I believe is WAY more worth the investment. A good starting point is looking at idea flow.</span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-20065502596834567582012-05-24T16:05:00.003-07:002012-05-24T16:05:44.453-07:00Humans as Part of the SystemI think about every software process diagram that I've ever seen, and every one seems to focus on the work items and how they flow - through requirements, design, implementation, testing and deployment. Whether short cycles or long, discreet handoffs or a collapsed 'do the work' stage, the work item is the center piece of the flow.<br />
<br />
But then over time, something happens. The work items take longer, defects become more common and the system deteriorates. We have a nebulous term to bucket these deterioration effects - technical debt. The design is 'ugly', and making it 'pretty' is sort of a mystic art. And likewise keeping a software system on the rails is dependent on this mystic art - that seems quite unfortunate. So why aren't the humans part of our process diagram - if we recognized the underlying system at work, could we learn how to better keep it in check?<br />
<br />
What effect does this 'ugly' code really have on us? How does it change the interactions with the human? What is really happening?<br />
<br />
If we start focusing our attention on thinking processes instead of work item processes, how ideas flow instead of how work items flow... the real impact of these problems may actually be visible. Ideas flow between humans. Ideas flow from humans to software. Ideas flow from software to humans. What are these ideas? What does this interaction look like?<br />
<br />
Mapping this out even for one work item is enlightening. It highlights our thinking process. It highlights our cognitive missteps that lead us to make mistakes. It highlights the effects of technical debt. And it opens a whole new world of learning.<br />
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-78771701299686265342012-04-12T12:20:00.002-07:002012-04-12T12:24:22.730-07:00Addressing the 90% Problem<div>
If I were to try to measure the time that I spent thinking, analyzing and communicating versus actually typing in code, how much of the time would it be? If I were to guess, I'd say something at least 90%. I wonder what other people would say? Especially without being biased by the other opinions in the room?</div>
<div>
<br /></div>
<div>
We spend so much of our time trying to understand... even putting a dent in improvement would mean -huge- gains in productivity. So...</div>
<div>
<br /></div>
<div>
How can we improve our efficiency at understanding?</div>
<div>
How can we avoid misunderstanding, forgetting, or lack of understanding?</div>
<div>
<div>
How can we improve our ability and efficiency at communicating understanding?</div>
</div>
<div>
How might we reduce the amount of stuff that we need to understand?</div>
<div>
<br /></div>
<div>
These are the questions that I want to focus on... its where the answers and solutions will make all the difference.</div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-73927060913258468972012-04-12T10:56:00.000-07:002012-04-12T10:57:10.865-07:00Mistakes in a World of GradientsI've been working on material for avoiding software mistakes, and have been searching for clarity on how to actually define "mistake."<br />
<br />
I've been struggling with common definitions that are very black and white about what is wrong and considered "an error" or "incorrect". Reality often seems more of a gradient than that, and likewise avoiding mistakes should maybe be more a matter of avoiding poor decisions in favor of better ones?<br />
<br />
I like this definition better, because it accounts for the gradient in outcome, without missing the point.<br />
<br />
<div class="p1">
"A human action that produces an incorrect or inadequate result. Note: The fault tolerance discipline distinguishes between the human action (a mistake), its manifestation (a hardware or software fault), the result of the fault (a failure), and the amount by which the result is incorrect (the error)."</div>
<div class="p1">
<br /></div>
<div class="p1">
GE Russell & Associates Glossary - <a href="http://www.ge-russell.com/ref_swe_glossary_m.htm">http://www.ge-russell.com/ref_swe_glossary_m.htm</a></div>
<div class="p1">
<br /></div>
<div class="p1">
<br /></div>
<br />
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-3912013593849326162012-04-10T11:01:00.002-07:002012-04-10T11:02:50.199-07:00What is a Mistake?I've been trying to come up with a good definition for "mistake" in the context of developing software.<br />
<br />
It's easy to see defects as caused by mistakes, but what about other kinds of poor choices? Choices that led to massive work inefficiencies? And what if you did everything you were "supposed to" do, but still missed something, is that caused by a mistake? What if the problem is caused by the system and no person is responsible, is that a mistake?<br />
<br />
All of these, I think should be considered mistakes. If we look at the system and the cause, we can work to prevent them. The problem with the word "mistake", is it's quickly associated with blaming the who responsible for whatever went wrong. Mistake triggers fear, avoidance, and guilt. Which is the exact opposite of the kind of response that can lead somewhere positive.<br />
<br />
Here's the best definition I found from dictionary.com:<br />
<br />
<span id="hotword" style="background-color: white; color: #333333; font-family: verdana; font-size: x-small; position: static; text-align: left;"><span id="hotword" name="hotword" style="cursor: default; position: static;">"an</span> </span><a href="http://dictionary.reference.com/browse/error" style="background-color: white; color: #333333; font-family: verdana; font-size: small; text-align: left;">error</a><span id="hotword" style="background-color: white; color: #333333; font-family: verdana; font-size: x-small; position: static; text-align: left;"> <span id="hotword" name="hotword" style="position: static;">in</span> <span id="hotword" name="hotword" style="cursor: default; position: static;">action,</span> <span id="hotword" name="hotword" style="position: static;">calculation,</span> <span id="hotword" name="hotword" style="position: static;">opinion,</span> <span id="hotword" name="hotword" style="position: static;">or</span> <span id="hotword" name="hotword" style="position: static;">judgment</span> <span id="hotword" name="hotword" style="position: static;">caused </span><span id="hotword" name="hotword" style="cursor: default; position: static;">by</span> <span id="hotword" name="hotword" style="cursor: default; position: static;">poor</span> <span id="hotword" name="hotword" style="cursor: default; position: static;">reasoning,</span> <span id="hotword" name="hotword" style="position: static;">carelessness,</span> <span id="hotword" name="hotword" style="position: static;">insufficient</span> <span id="hotword" name="hotword" style="position: static;">knowledge,</span><span id="hotword" name="hotword" style="cursor: default; position: static;">etc.</span></span> "<br />
<br />
From this definition, even if you failed to do something that you didn't know you were supposed to do (having insufficient knowledge), its still a mistake. Even if it was an action triggered by interacting parts of the system, but no one thing, its still a mistake.<br />
<br />
But choices that cause inefficiencies? That seems to fall under the gradient of an "error of action or judgement". If we could have made a better choice, was the choice we made an error? Hmm.<br />
<br />Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-90874964788932023922012-04-08T14:10:00.004-07:002012-04-15T12:37:55.870-07:00A Humbling ExperienceAbout 7 years ago, I was working on a custom SPC system project. Our software ran in a semiconductor fab, and was basically responsible for reading in all the measurement data off of the tools and detecting processing errors. Our users would write thousands of little mini programs that would gather data across the process, do some analysis, and then if they found a problem, could shutdown the tool responsible or stop the lot from further processing.<br />
<br />
It was my first release on the project. We had just finished up a 3 month development cycle, and worked through all of our regression and performance tests. Everything looked good to go, so we tied a bow on it and shipped it to production.<br />
<br />
That night at about three in the morning, I got a phone call from my team lead. And I could hear a guy just screaming in the background. Apparently, we had shut down <b>every</b> tool in the fab. Our system ground to a screeching halt, and everyone was in a panic. <br />
<br />
Fortunately, we were able to rollback to the prior release and get things running again. But we still had to figure out what happened. We spent weeks verifying configuration, profiling performance, and testing with different data. Then finally, we found a bad slow down that we didn't see before. Relieved to find the issue, we fixed it quickly, and assured our customers that everything would be ok this time. <br />
<br />
Fifteen minutes after installing the new release... the same thing happened. <br />
<br />
At this point, our customers were just pissed at us. They didn't trust us. And what can you say to that? Oops?<br />
<br />
We went back to our performance test, but couldn't reproduce the problem. And after spending weeks trying to figure it out, and about 15 people on the team sitting pretty much idle, management decided to move ahead with the next release. But we couldn't ship...<br />
<br />
<div class="p1">
There's an overwhelming feeling that hits you when something like this happens. A feeling that most of us will instinctively do anything to avoid. The feeling of failure. <br />
<br />
We cope with it and avoid the feeling with blame and anger. I didn't go and point fingers or yell at anyone, but on the inside, I told myself that I wasn't the one that introduced the defect, that it was someone else that had messed it up for our team. <br />
<br />
We did eventually figure it out and get it fixed, but by that time it was already time for the next release, so we just rolled in the patch. We were extra careful and disciplined about our testing and performance testing, we didn't want the same thing to happen.<br />
<br />
At first everything looked ok, but we had a different kind of problem. It was a latent failure, that didn't manifest until the DBA ran a stats job on a table that crashed our system... again. But this time, it was my code, my changes, and my fault.<br />
<br /></div>
<div class="p1">
There was nobody else I could blame but myself... I felt completely crushed.<br />
<br />
I remember sitting in a dark meeting room with my boss, trying to hold it in. I didn't want to cry at work, but that only lasted so long. I sat there sniffling, while he gave me some of the best advice of my life.<br />
<br />
"I know it sucks... but it's what you do now that matters. You can put it behind you, and try to let it go... or face the failure with courage, and learn everything that it has to teach you."<br />
<br />
<div class="p1">
Our tests didn't catch our bugs. Our code took forever to change. When we'd try to fix something, sometimes we'd break five other things in the process. Our customers were <b>scared</b> to install our software. And nothing says failure more than bringing down production the last 3 times that we tried to ship! <br />
<br />
That's where we started...</div>
<div class="p2">
<br /></div>
<div class="p1">
After 3 years we went from chaos, brittleness and fear to predictable, quality releases. We did it. The key to making it all happen, wasn't the process, the tools, or the automation. It was about facing our failures. Understanding our mistakes. Understanding ourselves. We spent those 3 years learning and working to prevent the causes of our mistakes.</div>
</div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-33352826201687113152012-03-28T17:23:00.003-07:002012-03-29T00:22:28.458-07:00What we REALLY Value is the Cost...<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Today, someone in the community mentioned the idea of measuring "value points". And the light went on... could this finally highlight our productivity problems? It could be a totally dead-end idea, but its a hypothesis that needs testing.</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;"><br />
</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">When I thought "value points", I imagined a bunch of product folks sitting </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">around playing planning poker judging the relative value of features. Using </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">stable reference stories and choosing whether one story was more or less </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">valuable than others. Might seem goofy, but its an interesting idea. My </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">initial thought was that this would be way more stable over time than cost since </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">cost varies dramatically over the lifetime of a project. And if it is truly </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">stable, it might provide the missing link when trying to understand changes in </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">cost over time.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">For this to make sense, you gotta think about long term trends. Suppose</span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;"> our team can deliver 20 points of cost per </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">sprint. But our codebase gets more complex, bigger, uglier and more costly to </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">change. Early on, we can do 10 stories at 2 points each. But 2 years later, very similar features on the more complex code base require more effort to implement, so maybe these similar stories now take 5 points each and we can do 4 of them. </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Our capacity is still 20 story points, but our ability to delivery value has REALLY decreased.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">We often use story points as a proxy for value delivered per sprint, but think </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">about that... We get "credit" for the -cost- of the story as opposed to the </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">-value- of the story. If our costs go up, we get MORE credit for the same work!</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">How can we ever hope to improve productivity if we measure our value in terms of </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">our costs? How can we tell if a story that has a cost of 5 could have been a </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">cost of 1? Looking at story points as value delivered makes the productivity </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">problems INVISIBLE. It's no wonder that it's so hard to get buy in for </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">technical debt...</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">What if we aimed for value point delivery? If you improved productivity, or your </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">productivity tanked, would it actually be visible then? On that same project, </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">with 20 cost points per sprint, suppose that equates to 10 value points early </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">on, and 4 value points later. Clearly something is different. Maybe we </span><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">should talk about how to improve? Productivity, anyone? Innovation?</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">At least it would seem to encourage the right conversations...</span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-74531731579101682012-03-08T15:37:00.001-08:002012-03-08T15:37:50.102-08:00Does Agile process actually discourage collaboration and innovation?<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Before everyone freaks out at that assertion, give me a sec to explain. :)</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">In the Dev SIG today, we were discussing our challenges with integrating UX into</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">development, and had an awesome discussion. I think Kerry will be posting some</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">notes. Most of the discussion though, went to ideas and challenges with</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">creating and understanding requirements, and the processes that we use to scale</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">destroying a lot of our effectiveness. The question we all left with, via Greg</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Symons, was how do we scale our efforts while preserving this close connection</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">in understanding between the actual customer and those that aim to serve them?</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">In thinking about this, our recent discussions about backlog, and recalling past</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">projects, I realized some crucial skills that we seem to have largely lost. In</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">the days of waterfall, we were actually much more effective at it.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">My first agile project was with XP, living in Oregon, and fortunate enough to</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">have Kent Beck provide a little guidance on our implementation. Sitting face to</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">face with me, on the other side of a half wall cube, was an actual customer of</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">our system, who had used it and things like it for more than 20 years. I could</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">sit and watch how they used it, ask them questions, find out exactly what they</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">were trying to accomplish and exchange ideas. From this experience I came away</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">with a great appreciation for the power of a direct collaborative exchange</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">between developers and real customers.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">My next project, was waterfall. One of the guys on my team, wickedly smart, his</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">background was mainly RUP, and he just -loved- requirements process. What he</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">taught me were techniques for understanding, figuring out the core purpose,</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">figuring out the context of that purpose, and exploring alternatives to build a</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">deeper understanding of what a user really needs. Some of these were</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">documentation techniques, and others were just how you ask questions and</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">respond. I learned a ton. On our team, the customers would make a request, and</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">the developers were responsible for working with the customers to discover the</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">requirements.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">With Scrum-esque Agile process, this understanding process is outsourced to the</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">product owner. As we try to scale, we use a product owner to act as a</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">communication proxy, and with it create a barrier of understanding between</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">developers and actual customers. Developers seldom really understand their</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">customers, and when given the opportunity to connect with them, the number of</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">discoveries of all the things we've been doing that could have been so much</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">better, are astounding.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">I've done agile before on a new project, sitting in the same room with our real</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">users, understanding their problems, taking what they asked for and figuring out</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">what they needed, and also having control of the architecture, design, interface</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">and running the team to build it - the innovation of the project and what we</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">build was incredible. Industry cutting-edge stuff was just spilling out of</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">everything we did. And it all came out of sitting in a room together and</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">building deep understanding of both the goals, and the possibilities. This was</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">agile with no PO proxy. The developers managed the backlog, but really wrote</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">very little down... we did 1 week releases.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Developers seldom have much skill in requirements these days. And are often</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">handed a specification or a problem statement that is usually still quite far</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">from the root problem.</span><br />
<br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">In building in this understanding disconnect, and losing these skills, are we</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">really just building walls that prevent collaboration and tearing down our</span><br />
<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">opportunities to innovate?</span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-48264238635481574102012-03-08T14:25:00.002-08:002012-03-08T14:25:44.803-08:00Manufacturing of a Complex Thought<span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Imagine that the software system is a physical thing. Its shape isn't really</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">concretely describable, its like a physical version of a complex thought. All</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">of the developers sit in a circle, poking and prodding at the physical thought -</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">adding new concepts, and changing existing ones.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Just like we have user interfaces for our application users, the code is the</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">developer's interface to this complex thought. Knowledge processes and tools</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">help us to manipulate and change the thought.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">If I want to make a change, I need to first understand enough of the thought to</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">know how it would need to change. If I can easily control and manipulate parts</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">of the thought, and easily observe the consequences, its easier and faster to</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">build the understanding I need. Once I understand, I can start changing the</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">ideas, and again if I misunderstood something, it would be nice to know as early</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">as possible what exactly my mistake was, so that I can correct my thinking. If</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">there were no misunderstandings, the newly modified complex thought is then</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">complete.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">In order to collaboratively work on evolving this complex thought, we must also</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">maintain a shared understanding - more brains involved increases the likelihood</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">of misunderstandings and likewise mistakes.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">So with that model of development work, then think about all of the ideas and</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">thinking that have to flow through the process in order to support the creation</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">of this physical idea. Inventory in this context is a half-baked idea, either</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">sitting in the shelf, or currently in our minds being processed. These ideas</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">are what we manufacture, but since each idea has to be weaved into a single</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">complex thought - our tools that we use to control and observe the thought, the</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">clarity and organization of the thought, the size of the thought, all have a</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">massive impact on our productivity.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">The tools are not the value, the ideas that get baked into this complex thought</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">are. All of the tooling is just a means to an end. We should strive to do just</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">enough to support the manufacturing of the idea.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">If you think about creating tests from the mindset of supporting the humans and</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">these knowledge processes, a lot of what we do with both automated and manual</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">testing can clearly be seen as waste. An idea that is clear to everyone, for</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">example, is not one likely to cause misunderstandings and mistakes. We should</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">first aim to clarify the idea to prevent these misunderstandings. We should</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">then aim for controllable and observable, as these characteristics allow us to</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">come to understand more quickly. And when misunderstandings and mistakes are</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">still likely, we should then use alarms to alert us when we've made a mistake. </span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">False alarms quickly dilute the effectiveness of the useful alarms... thus</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">taking care in trying to point the human clearly to the mistake, and not raise</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">unnecessary false alarms is what makes effective tests.</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">Now think about things like code coverage metrics in this light. This metric</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">tends to encourage extreme amounts of waste. We forget all about the humans and</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">fill our systems with ever-ringing false alarms. We tend to only think about</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">tests breaking in our CI loop, but their real effect is their constancy in</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">breaking while we try to understand and modify this shared complex thought. </span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">With our test-infected mindsets, we quickly bury the changeability of our ideas</span><br style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;" /><span style="background-color: white; font-family: Georgia; font-size: 14px; text-align: left;">in rigidity, and lose the very agility that we are supposedly aiming for.</span>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-6911938996572822672012-02-17T16:10:00.000-08:002012-02-19T13:26:44.410-08:00Fighting my way to agility - Part 5<b>Shrinking Release Size</b><br />
<div><br />
</div><div>Now that our batches were shrinkable, we could feasibly shrink our iteration and release sizes. The ultimate test of whether you are -really- shippable is to actually ship. If we could actually get our software in production, we could put the risk behind us for the changes we'd done so far. If you don't actually ship, it was hard to -really- know if we were shippable. We were also introducing more risk at a time to production - and troubleshooting production defects would usually take longer to diagnose and fix.</div><div><br />
</div><div>But our customers were just starting to trust us, and were still doing about a month of testing after our testing before they would feel safe enough to install something. We asked if we could do releases more often, and the answer was pretty much... 'hell no.' Rather than give up on trying, we figured out why there was so much pushback. And solving those problems, whatever they might be, became our top priority.</div><div><br />
</div><div>We learned about all kinds of problems that we didn't even know we had. Some were ticket requests that had been sitting in the backlog for years. And others were just stuff that had to be done for every release that isn't really a big deal until we asked them to be done a whole lot more often. There was a whole lot of pain downstream that we weren't even aware of. And most of it was really just our problems rolling downhill - and completely within our power to fix.</div><div><br />
</div><div><b>Reducing the Transaction Cost of a Release</b></div><div><br />
</div><div>There's lots of different kinds of barriers to releasing more often. Regardless of what yours are, getting the right people in the room, working together to understand the whole system goes a long ways. A lot of the stuff that seem like hard timeline constraints actually aren't. Challenge your assumptions. </div><div><br />
</div><div>So how could we relieve our customer's pains?</div><div><br />
</div><div><b>"Everytime we do a release, we lose some data"</b> - we had no idea. The system was designed to do a rolling restart, but there was a major problem. During the roll over we had old component processes communicating with new ones. In general the interfaces were stable, but there was still subtle coupling in the data semantics that caused errors. Rather than trying to test for these conditions, we instead changed the failover logic so all of the data processing would be sticky to either the old version or new version and could never cross over. This prevented us from having to even think about solving cross-talk scenarios. We also created a new kind of test that mimicked a production upgrade while the system was highly live. This turned out to be a great test for backward compatibility bugs as well.</div><div><br />
</div><div><b>"Its not safe to install to production without adequate testing, and we can't afford to do this testing more often" - </b>Whether they found bugs or not while testing, was almost irrelevant. Unless they knew what was tested and felt safe about it, they wouldn't budge. They were doing different testing, with different frameworks, tools and people than we were, and unless it was done that way, it was a no go. So we went to our customer site. We learned about their fears and what was important to them. We learned how they were testing and wanted testing to be done. </div><div><br />
</div><div>We shared our scenario framework with them, code and all, and then worked with them to automate their manual tests in our framework. We made sure their tests passed (and could prove it), before we gave them the release. And likewise, we adopted some of their testing tools and techniques and at release time gave them a list of what we had covered. We also started giving them a heads up about what areas of the system were more at risk based on what we had changed so they didn't feel so much need to test everything. After we helped them to reduce their effort and just built a lot more trust and collaborative spirit with our customers, this was no longer an issue.</div><div><br />
</div><div><b>Editing the Scrum Rule Book - What process tools DID we actually use?</b><br />
<br />
Since we weren't predictable and weren't using time boxes, we also threw out story point estimation, velocity and any estimation-based planning activities. The theory goes that you improve your ability to estimate and therefore your predictability by practicing estimation. I think this is largely a myth. <br />
<br />
"Predictions don't create Predictability." This is one of my favorite quotes from the Poppendieck's book, Implementing Lean Software Development. You create predictability by BEING predictable. The more complex and far in the future your predictions, the more likely you are to be wrong - and way wrong. So wrong that you are likely to make really bad decisions under the illusion that your predictions are accurate. Its an illusion of control when control doesn't actually exist. You can't be in control until you ARE controllable. Predictability doesn't come from any process, its an attribute that exists (or doesn't) in the system. Uncertainty is very uncomfortable. But unless you face reality and focus on solving the root problem, nothing is ever <b>really</b> likely to change.<br />
<br />
Burn downs we did use, but not until we were closer to wrapping up a release. This was helpful in answering the 'are we done yet?' questions, the timing of which we used to synchronize other release activities. There were tickets to submit, customer training to do, customer testing to coordinate etc. We tried doing burn downs for the whole sprint, but since our attempted estimates were so wildly inaccurate - it wasn't helpful and more so harmful as input into any decisions. The better decision input was that we really had no idea, but were trying to do as little as possible so done would come as soon as possible. If a decision had to be made, we would try and provide as much insight as we could to improve the quality of the decision, without hiding the truth of uncertainty. Although management never liked the answers, our customers were unbelievably supportive and thankful for the truth.<br />
<br />
</div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-67254265320457856722012-02-11T13:24:00.000-08:002012-02-12T08:05:36.123-08:00Fighting my way to agility - Part 4<div class="p1"><b><span style="font-family: inherit;">Shrinking your Iterations when you have a Painful Test Burden</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;">The obvious thing, which we quickly jumped on, was automating our existing manual test cases. They were mostly end-to-end, click-through UI and verify tests. We automated most of them with QTP, and tried to run them in our CI loop. This turned out to be a terrible idea… or rather a horrible nightmare. Not only were they terribly painful to maintain and always breaking - we had 2 other problems: Our developers started totally ignoring the CI build since it was always broken, and they weren't even catching most of our bugs. It was a massive cost, with very little benefit. We ended up throwing them all away and going back to manual. Don't get me wrong, this sucked too. We instead focused our efforts on creating better tests. </span><br />
<span style="font-family: inherit;"><br />
</span><br />
<span style="font-family: inherit;">We started looking more deeply at where our bugs were coming from, and why we were making mistakes. Our old manual test suite was built by writing manual tests for each new feature that was implemented. The switch in attention to <b>finding ways to stop our actual bugs</b>, instead of just trying to cover our features, was key to really turning quality around. We ultimately created 3 new test frameworks. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>Performance simulation framework</b> - The SPC engine had some pretty tight performance constraints that seemed to unexpectedly go haywire. Trying to track down the cause of the haywire performance, was a hell of a challenge. The more code that had changed the harder it was to track down. We also discovered that our existing performance test that used simulated data didn't find our problems. The system had very different performance characteristics with highly-variable data. So we got a copy of each production site and made a framework that would reverse engineer outputs as inputs and replay them through the system. We would run a test nightly that did this and send us an email if it detected 'haywire'. We actually used our own SPC product and created an SPC chart against our performance data to do it ;) Catching performance issues immediately was a complete game-changer. It also gave us a way to tune performance, and as a bonus some handy libraries for testing our app.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>SPC 'fingerprinting' Tests -</b> Remember all the bandaids and fragileness I talked about? It was insanely hard to not break. We used the performance test tooling to replay the outputs as inputs to the system, but then we recorded the new output as a fingerprint file. For every single chart in every production environment, we generated one of these fingerprinted scenarios. Then as the system changed, we would compare the old fingerprints with the new ones and fail if there were differences. Even with 8000 generated tests, it was easy to flip the bar green for expected changes by copying all files in 1 folder to another. But the challenge was in telling between expected changes and accidental ones. Again, this greatly increased in complexity with an increased amount of change. If the change was relatively small, you could look at a sample of files and see if behavior was as expected or not. We not only caught a TON of bugs this way, but also found cases where users asked us to implement changes that would break other use cases and we were able to alert them.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>Integration scenario tests -</b> The other area where we had a lot of bugs were with scenarios that crossed system boundaries. For example, SPC would put a lot (container of material) on hold, a user would investigate and then release the hold. These scenarios got quite complicated when the lot was split up, combined with other material, and processed different places. We had to track down all the material that might be affected and put all of it on hold. With remote calls failing on occasion and other activities happening concurrently, there were a lot of ways for things to go wrong. Anyway, we worked with the testers for the other system to create a framework that we could orchestrate and verify state across systems. For our apps, rather than going through the UI, we had an internal test controller that we would use to drive the app from the inside. To verify state we would internally collect and dump all the critical state information to an XML file that we would diff to detect failure. We had about 150 or so end to end integration tests like this that had a much lower maintenance cost.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><span style="font-family: inherit;">The Fate of Our Manual Tests</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">Once we had much better coverage of the error-prone parts of the system, running all of these tests became a lot less important. A few we converted to scenario or unit tests, but for the most part these all stayed manual. It was still expensive to run them all, but we used another strategy to reduce that burden. We categorized them all in a Sharepoint list, then at release time, we would run only tests we thought had a chance at finding a bug based on what we had changed that release. With all the other testing, this was good enough. We found a couple bugs with them, but at this point, they were almost always green.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><span style="font-family: inherit;">But the Work was Still TOO Big!</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">We still suffered some pretty massive productivity problems, but at this point we were back in control. When we changed stuff it wasn't so risky. But it was still too time-consuming. So we focused our team on how could we get the <b>most productivity gain for our effort</b>. </span><br />
<span style="font-family: inherit;"><br />
</span><br />
<span style="font-family: inherit;">We analyzed both our past work for where we were spending time, and where future code changes needed to be. The UI layer was frightful and always time-consuming to change, but we rarely had to change it. Effort here, wouldn't actually buy us more productivity. Most ouf changes were in the core SPC pipeline and it was a major bear to understand and change. Although the effort level was high, we all knew we would see the payoff. The impact we had on productivity from rewriting the SPC engine was HUGE. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">We had all kinds of awesome new features we were able to do, including supporting SPC for a completely different type of facility. Our productivity exploded. We could have never done these features on the system before - the effort was probably beyond the cost to rewrite. Our users were also thrilled because the behavior was so predictable - even for complex scenarios. Their productivity improved in designing and maintaining SPC charts!</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">As we got better and better control of our productivity, we kept shrinking our iteration sizes, and were finally able to have consistent quality. It wasn't perfect, but it was good enough to earn back the trust of our customers. Though we never did end up using timeboxes, at this point, our sprints were 'boxable'. :)</span><br />
<div><span style="font-family: inherit;"><br />
</span></div></div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-85743635540847838482012-02-11T13:23:00.001-08:002012-02-12T07:22:11.865-08:00Fighting my way to agility - Part 3<div class="p1"><b><span style="font-family: inherit;">Breaking down work</span></b></div><div class="p1"><b><span style="font-family: inherit;"><br />
</span></b></div><div class="p1"><span style="font-family: inherit;">On a large complex hairy project, work just gets big. Our work was big. And breaking down work is sort of an art. The think time was usually 10x the coding time, and the test time would explode by just changing 1 line in the SPC engine. When we tried to force big work into a small box, the team kept trying to slice it in ways that didn't independently deliver anything useful - which you're not supposed to do.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">A lot of times in a break down we'd get a requirements story, a design story etc… and not because the team was just hung up on waterfall, they were trying to figure out how to break down 'think'. After all, how do you break down a 1 line code change story when the scope of the work is really thinking through and testing the implications of changing that line of code? Clarifying the exact expectations for a specific SPC behavior might take the developer and customer working together for a couple weeks. Our stories might have looked 'wrong' to some, but when these cases came up, breaking down 'think' work this way was helpful. </span><span style="font-family: inherit;">A more problematic breakdown attempt had stories that checking in the code for story part would leave the system unshippable. When this would come up we would work really hard to find another way - which usually meant creating a story to find another way. In either case, all of the various story parts that didn't independently deliver anything useful we tracked under an epic that did.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">In the process of trial and error with breaking down work, we learned a lot about what mattered and what really didn't. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><b><span style="font-family: inherit;">Lessons Learned on Breaking Down Work</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;"><b>Teach over Force - </b>Given all the pain we had, rather than making a box and forcing work to fit, I would focus teaching break-down work skills. Whether your box is fixed or not, you need to solve this problem. You can aim for a box, but don't mutate the work to a degree that it violates the below constraints. If you can't come up with a way to do that, just leave it as a big item.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>No Invisible WIP - </b>If you have stories that don't deliver anything usable to the customer (e.g. my requirements story), track them under an epic. Not because of some Agile holy gospel, but because if you don't it HIDES work in progress. You have a partial investment in work, and over time you have thoughts rotting away, and code that may not even work sitting in your code base. If its not ALL done and you have to even <b>think</b> about the set of changes again, its WIP. Keep all your WIP in front of you, all of the time. It should remind you of everything that is part way done. Push to finish what you started before starting more stuff. Push to get back to shippable as soon as you can. Don't let yourself feel like the work is all done when you still have WIP. Its not done. That looming manual testing that tells you whats broken in your code? The integration testing with that other system? Your stories are still in progress and if you hide that fact its easy to feel ok about starting more work. </span></div><div class="p1"><b><span style="font-family: inherit;"><br />
</span></b></div><div class="p1"><span style="font-family: inherit;"><b>If you are not shippable, you still have WIP.</b> This is probably the most critical thing I see missed for people in any process - invisible, hidden WIP. I think its one of the main reasons people don't even see what otherwise would be a glaringly obvious problem.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>Vertical Slicing -</b> The typical strategies of trying to break work into vertical slices (sliver of usable functionality traversing all layers of the system) and finding the smallest possible thing that can work - all are generally good advice. Try to break down the implementation of the work by iterating over even smaller vertical slices, even if its totally not useful. You learn a lot in the process, which tends to reduce rework.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;"><b>Don't abandon your requirements format</b> - "User story" format can be helpful, but it's not that important. The key is communication. Whatever you need to do to effectively communicate the purpose and expectations effectively - use what works. If you've got something that is working well for you, don't change it. Don't assume stories are going to work better for you. After some chaos and thrashing, we basically settled on a 'story statement' added to our existing really formal requirements document. Distributed communication, cultural barriers, complex requirements all make communication REALLY hard.</span></div><div class="p1"><br />
</div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-40164312612405764772012-02-11T13:22:00.000-08:002012-02-11T13:25:23.289-08:00Fighting my way to agility - Part 2<div class="p1"><b><span style="font-family: inherit;">Trying to Fix the Timebox... and Failing</span></b></div><br />
<div class="p1"></div><div class="p1"><span style="font-family: inherit;">We tried to fix the time and vary the scope and be shippable at the end of the sprint. Well, that didn't really work. How do you run 1200 manual tests in the sprint? What if I just have a month of code integrated that other people built stuff on top of, and I'm almost done but not quite? What if the app doesn't work at the end? What if the smallest possible work size is just big? If we have to be shippable at the end of the sprint, and the scope of getting to quality is highly variable, how could we possibly fix the timebox? </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">I've seen people try to do testing sprints or hardening sprints or integration sprints but it really fundamentally destroys one of the core principles of Scrum - being shippable at the end of every sprint. Shippable as in - you really could put what you just coded in production. As soon as you make it ok to throw your testing into another sprint, you ignore that fundamental constraint. And even worse, it masks a HUGE leering problem that really should be your priority to solve. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">If you do another sprint, with your testing all delayed, you then violate another core principle - you shouldn't build on top of BROKEN code. It was massively harder to diagnose bugs when there was more code that hadn't been proven yet. It was roughly about 4 to 1. Doubling the amount of code, roughly quadrupled the time of our test/fix cycle. But it didn't seem to take much more to go off the test/fix cliff of no return.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">Both of these are way more important to not violate than time boxes. Yet I see soooo many teams keep the time boxes and throw out the other 2. Until you can actually <b>be</b> predictable, I don't think the timeboxes buy you much anyway. Something had to give, we voted for the box... with a goal of working on becoming more 'boxable'.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;">So we scheduled our 'sprint' for what we thought was about a month-ish of work, but then however long it took to test the app and get it all working and finish anything that we couldn't pull out, we just took that time and didn't work on anything new. It was done whenever we were back to shippable. Nobody was allowed to work on forward development. Nobody did any refactoring that might add more risk. Get the app back to being green. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">It was easy to fall into a trap of worrying about inefficiencies, especially as test/fix cycles dragged out, the business wanted their features. We tried creating a release branch and having a few continue on forward development while we focused on stabilizing. This idea blew up in our face. We were only really thinking about the penalty of managing the branch and merging, and didn't foresee this at all. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">Eventually we got the release out the door and planned out another one month-ish sprint. Then we got to our test/fix phase, but there was an even bigger batch of work in it! The last stabilization had taken ~2 months, and so we actually did about twice as much work. The difference in time that it took to troubleshoot defects in the larger set of changes was HUGE. The troubleshooting complexity shot through the roof. It was another 6 months before we were able to get the changes out the door. </span><br />
<span style="font-family: inherit;"><br />
</span><br />
<span style="font-family: inherit;">Forget getting a head start, and have your spare capacity work on the problem right in front of you - its too damn hard to get code out the door. </span><span style="font-family: inherit;">What good are a bunch of coded features that you can't ship?</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><br />
</b></div><div class="p1"></div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-29695900062934467282012-02-11T13:17:00.000-08:002012-02-11T14:35:15.685-08:00Fighting my way to agility - Part 1<b><span style="font-family: inherit;">Real Projects Have Hard Problems</span></b><br />
<b><span style="font-family: inherit;"><br />
</span></b><br />
<div class="p1"></div><div class="p1"><span style="font-family: inherit;">The challenge with principles is that its really hard to see how they apply in a specific context. The idea of the principle may make perfect sense. How the software process should work in theory may make perfect sense. But the messy world of software reality isn't so kind. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">Concrete messy real world experiences are awesome to learn from because they give us insights on how to take abstract truths (principles) and map them to reality. From all of these examples, we can distill patterns that give us ideas on how to map solutions between different problems. Its far too easy to make a really bad decision if you over-simplify a very complex world. Real software is just hard. </span><br />
<span style="font-family: inherit;"><br />
</span><br />
<span style="font-family: inherit;">For anyone who reads this, <b>please</b> share your stories. Whether it ends in victory or not, its the guts of our learning. By sharing them we broaden all of our experiences, and improve all of our abilities to tackle the problems we face. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">So here's one of my stories. This is the story of a messy real world project in which I personally fought my way to agility.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><span style="font-family: inherit;">My Project</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">A semiconductor factory SPC system responsible for reading and aggregating data coming off of the tools to detect problems (and shut down what was causing the problem). High volume, highly variable incoming data stream, user-defined analysis programs and near real-time charts. A program would usually gather a mix of historical data and current data, and do a bunch of math on the results to make a decision. If we took too long to make that decision, the tool would timeout and shutdown. Deployed in a 24/7 environment with 1 downtime per year.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><span style="font-family: inherit;">Starting point (for me, 2005)</span></b></div><div class="p1"><b><span style="font-family: inherit;"><br />
</span></b></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;">Scary reality. :) 500k lines of code, ~10 years old. The web server was home-grown, the UI flow-control was done with exceptions (seriously!), and the core SPC engine had bandaid over bandaid of hack fixes that it was next to impossible to make a change without having some unintended side-effect on another use case. 1200 formal manual test cases to all be run and made green for every release. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">Half of the team in Austin, half in India, one in Germany, and customers in both Austin and Germany each with different data formats, problems and strategies. Interestingly, the team started with 2 week iterations, but as the transaction cost of delivery went up with the growing test burden, the batches kept getting larger to compensate. When I got there, they were doing ~4 months of development and a couple months of test/fix chaos after that. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">They had recently been having major performance problems and didn't know how to solve them. So the 2 months of chaos 2 months and counting with no end in sight. I moved from an XP shop building financial transaction/banking software, but they had really hired me for my Oracle performance wizardry skills. When I got there it was even more tragic, they had also been having major quality problems. They had just rolled back the production release - again. Our customers were literally scared to install our software. We were sitting on the tipping point of being<b> completely unable to release</b> with critical defects that we couldn't diagnose.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"></div><div class="p1"><b><span style="font-family: inherit;">End point (3 years later)</span></b></div><span style="font-family: inherit;"><br />
</span><br />
<div class="p1"><span style="font-family: inherit;">Consistently delivering releases every ~2 months with high-quality, stable performance, and predictable behavior. Deployed 2 more installations, one of them was a totally different type of processing facility. Happy customers. They even threw a party for me when I went to Germany!</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><b><span style="font-family: inherit;">Going from A to B - Was it Magic?</span></b></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;">It was pain. A lot of pain. A lot of mistakes. A lot of learning and hard work.</span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">We accidentally shutdown every tool in the fab (twice). We accidentally had a sprint that our test and fix churn dragged out for a year before we saw light at the end of the tunnel (way scary). We threw away massive investments in test automation with QTP and started over. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><span style="font-family: inherit;">We were aiming to do Scrum. We all read a Scrum book, and Bryan (still my awesome boss) was then our manager and Scrum Master. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"><span style="font-family: inherit;">It took a long and hard journey to accomplish real change. </span></div><div class="p1"><span style="font-family: inherit;"><br />
</span></div><div class="p1"></div><div class="p1"><b><br />
</b></div>Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0tag:blogger.com,1999:blog-1410644566887620672.post-27453666848280942932011-08-02T20:24:00.000-07:002011-08-02T20:34:36.650-07:00Real MentorshipI was interviewing for a team I was building and found someone just phenomenal. He had a couple years experience and had quite a few technical gaps, but was just smart smart smart and just had a natural knack for design. Like all the natural aptitude that makes a developer great - this guy had what it took to be a rockstar. I was so excited to be able to hire this guy and coach him, I just couldn't wait!<br />
<br />
And then... my project fell through, or at least was significantly delayed. And I was so bummed. I tried to make something work, because I saw the potential, but I couldn't make it work with clients that only wanted senior people. And so I tried to let it go, but it just kept nagging at me...<br />
<br />
A couple days ago, I met someone else pretty amazing. But she had a different problem... she was in an environment with great people that she loved, but silo'd like in a time machine and had missed the last 10 years. But she had what it took to be great... and was motivated to learn, just needed a little help. At first I felt a little sad... but then I realized that I was the only really holding me back. I didn't need to have an excuse to mentor someone. I could just do it! If she really has all the potential I see in her, shouldn't take her long to work her way up to remarkable, right? Then I can help her to get an awesome job and surround her with talent and then she'll really become amazing!<br />
<br />
We met at a coffee shop on Monday. Talked about software process and design... wrote some unit tests together and now she's doing some reading before we get together again. Pragmatic Programmer. A good top-of-the-list first book, I think. Later that day I got a thank you note from a friend of hers... it was really touching. Made my whole day. :)<br />
<br />
It's the kind of help that really makes a difference and touches someone's life... in an unexpected remarkable kind of way. And thinking, learning, designing, questioning are not just things everyone picks up in their software travels... I find lots of people with lots of years of the wrong experience. <br />
<br />
If we really want to advance knowledge and grow, we really need to help each other to do it. <br />
What if we all gave just a little bit of our time to help someone out?Anonymoushttp://www.blogger.com/profile/15380585954613538355noreply@blogger.com0