Over the years that I have been CTO at HedgeServ, members of our technology group have given me feedback many times about the need for “better communication”. My many early attempts to address this concern were to have status meetings with large groups of people to talk about what they were all doing. I sent email after email describing the details of what was going on. If they wanted communication, dammit, I would give it to them. But the people working in our organization weren’t satisfied, and I wasn’t either.
Clarity Trumps Detail
Having more meetings and sending more emails was my attempt at addressing concern about communication by providing details about what everyone was working on. It turned out that, while these details may or may not have been interesting to some people, it didn’t satisfy the need people had to understand how their work fit in with what we are trying to do in the business.
From The Art of Action: Leadership that Closes the Gaps between Plans, Actions and Results
Providing more detail is a natural response to a demand for clarity. But clarity and detail are not the same thing at all. The pursuit of detail actually increases noise and so makes it less clear what really matters. Details change quickly, so the more details we put in our plans the less robust they will be. Opportunities are missed, and by the time a decision has been made, the situation has changed, prompting further information gathering and analysis. The more detailed we make action plans, the more we constrain what people can do, which increases rigidity. People become demotivated and keep their attention firmly fixed on their KPIs, which become more important than what they were supposed to measure. Commitment is replaced by compliance, energy is sapped, and morale declines. The end result is a slow, expensive robot.
Here at HedgeServ, we have grown pretty rapidly and our technical infrastructure has been growing rapidly at the same time. Many months ago, our technology leaders got together and talked about how we can put ourselves in the best position to continue our growth. In that meeting we talked about DevOps and how DevOps principles might help us. We agreed to begin adoption of some of those principles.
This post is not intended to describe what DevOps “is”. I think of the word “DevOps” like “Agile”. It represents a notion that we should think about *why* we are doing what we are doing and to re-evaluate if what we are doing is the right thing. Here are a couple resources that have helped shape my thinking on the topic: here (money quote from @jeffsussna in the comments, “The ability to tip up 100 Apache servers at the push of a button doesn’t necessarily address the relationship between Dev and Ops”), and here.
We have over 200,000 lines of PowerShell code that we use to automate our infrastructure. This code was written by the individuals on our IT team who took it upon themselves to learn PowerShell and use it to solve their problems. I am amazed by what they have accomplished pretty much on their own. With our shift to DevOps thinking, we have started having design discussions with both Ops and Dev in the room for tasks that were traditionally Ops. It was in a recent one of those meetings that I learned a few lessons about things I had done as a leader to make things more difficult for our teams, and things I could do differently to give our teams more flexibility to solve problems in the future.
We are a Windows shop. Ten years ago, I counted myself as one of those people who said, “If you are on Windows, you have already lost” (“a venture capitalist…[would] have a hard time hiring anyone good to work for him if the project had to be built on [Windows] NT” - http://paulgraham.com/gh.html). But our platform relied on software that runs on only Windows so we had to make it work. And we did. And I now believe more than ever that a successful platform relies much more on the people building it than the specific technology being used.
The task at hand was the implementation of a new server build to enable a more optimal release process. The team has been building the application (formerly known as “scripts”) and we needed to build some test servers to validate some of the riskiest assumptions. The lead on the project, let’s call him Brent (just came into my head), was saying that the application isn’t done yet but they can use the pieces to create these servers. When pressed, Brent said he would manually execute steps where the application doesn’t work.
Brent was approaching his goal as getting our systems to the point where we had two additional servers! The application he was using to do it was secondary! Lesson #1.
Thanks to our collaboration with PraxisFlow (@kevinbehr, @jabe_bloom, and @john_dennin), we have learned that people who consistently have work forced upon them learn to accept that as the state of the world. They become used to a world full of constraints and therefore have difficulty imagining anything else. Brent was so used to having a large queue of high priority tasks that he could not help but to focus on satisfying the immediate “customer” and deliver his two servers. Lesson #2.
Developers approach problems almost exactly the opposite way. When faced with a problem, they like to try to find a solution that expands their own knowledge, to the point where sometimes the non-developers around them get extremely frustrated. I am reminded of a friend of mine from many years ago. I recall sitting with him while he pulled some data into Excel and needed to get the product of a column of numbers. Instead of copying and pasting a formula that iteratively multiplied the numbers, he thought about it a little and came up with the idea of summing the logs of each number, then using the exponent function to achieve the result. A non-developer next to him complained that he could have gotten the answer much more quickly through brute force, but who knows when that one small piece of knowledge might come in handy again when there isn’t a simple option?
I suggested that Brent use this server build as an opportunity to finish the app. If something breaks, he should delete the server he was in the process of creating, then start over from the beginning. This way, when he is done and the build is good, he knows he has an application he can use to re-run the build again. Brent expressed concern that this approach would slow him down tremendously because it takes two hours to run the build. Hmmmm, two hours? No wonder you don’t want to start over from the beginning. Five iterations of tests and you are at a 10 hour day! Without the ability to iterate quickly, you are going to be resistant to any approach that suggest you start over. Developers have powerful workstations for this exact reason. This enables them to iterate at a very fast rate. I realized that we had never put together a development environment for our IT people. We had created a system that actually discouraged them from learning! Lesson #3
I believe that part of the DevOps way of thinking is to enable everyone in the organization to keep learning at their job. This means eliminating the constraints that prevent people from iterating rapidly. This means giving IT people development training so they learn how to leverage iterations to start small and let an application unfold in front of them.
So the meeting, which started as a discussion about testing specific risks in the build, uncovered how the system that our IT team worked in limited our own ability to grow.
This article is about an accidental discovery of a self-organizing team right under my nose. It involves my son and his classmates. In the commentary below, I changed some of the details just in case the school doesn’t approve of the approach the kids are taking. The school my son attends is pretty progressive, and in the right context I want to approach the school administration to see what they think. But not right now. I have done my best to keep the substance the same. Anywhere a detail is completely screwed up is entirely my fault. I have included some screen shots of chats, but they are pretty scattered. When I asked for them, I was told that kids delete chat history very aggressively. Apparently kids say things in chat they don’t want their parents to see. I left the original language in. Sorry if anyone is offended. And finally, the extent to which I have made my son sound more like a 45 year-old than 16 year-old is also my fault.
My son was studying for a test at the kitchen table while we were getting ready for dinner. He was using a document on his laptop, which wasn’t normal. Traditionally he had written material he used. When I asked what he was doing, he said it was Google Docs.
It started innocently enough. (D = Dad, S = Son)
D: “What is that material you are studying?”
S: “It is Google Docs”
D: “When did you start using Google Docs to study?”
S: “A couple months ago. A couple of us started using it”
D: “What do you mean, a couple of you?”
S: “Dad, it’s Google Docs. People can share it”
D: “How do you guys share it”
S: “We all take notes into the same document so we make sure we get everything”
Now I am interested. I put down the dishes and sit down.
D: “Wait a second, you take notes into the same document? How does that work? When do you do this? How did it get started?”
D: “Ok. Maybe you can describe how you guys use Google Docs to take notes”
S: “Well, at the beginning of each chapter in the book is a series of key items and questions that we are supposed to get out of that chapter. So one of the kids creates a Google Doc with those items and questions in it. Then, during class…”
D: “Wait, you do this *during class*?”
S: “Yeah, we all have laptops with our Facebook Group and the Google Doc open”
D: “Wait, where does Facebook come in?”
S: “Well, at the beginning of the year, one of the kids in class created a Facebook Group and sent us all invites. We use it to talk about what is going on in class, ask about homework or help each other with upcoming tests.”
D: “Ok, but how do you use the FB Group in class?”
S: “We primarily use the chat feature in case someone has a question or comment.”
D: “So you have the FB Group and Google Doc open. How does it work?”
S: “We each take responsibility for certain items and/or questions. The person responsible for a particular item edits the doc during class to define the item or answer the question.”
D: “How do address the difference in abilities of different kids? What if one kid isn’t very good at taking notes?”
S: “People can comment on the notes a person is taking. It happens sometimes. Everyone has a color so you know who did what.”
D: “Do kids resent having to support kids who don’t take good notes.”
S: “Not that I know of. We have settled on the rule that, if you don’t think something is right, you write ‘(I don’t think this is right)’ and we figure it out.”
D: “What about the kids who take the notes that get comments? Do they feel bad or like they aren’t contributing?”
S: “Not that I know of.”
D: “Has there ever been a time when someone might go a little too far? Like make a comment on a note that is mean?”
S: “Not really. Early on one kid edited other people’s notes a lot. A couple of us posted on FB that he needed to chill and he stopped.”
D: “How did this get started?”
S: “One kid has been using Google Docs since the beginning of the year to take notes in class. A couple other kids asked him for help one time and he gave them his doc. One of those kids posted the doc on the FB Group in response to a question from someone else. When we got to the next chapter, the kid who created the doc posted the empty doc on the FB Group in advance. A couple of us in class just started taking notes right in it. As they were working, they noticed other kids in the class logged into the doc. One of us posted on the FB Group that we should all just edit it together. So we did.”
D: “Wow, how did you manage who was going to do what?”
S: “At first it was a mess. Everyone was just editing the same thing so people were all typing the same thing. On the FB Group, someone posted that we should take turns and each work on something different. We all generally sit in the same place around the table, so we used our seat locations to identify which items we were responsible for”
D: “Did everyone know how to use Google Docs?”
S: “It is pretty easy, so yeah”
D: “When did you start using this approach for notes? How is it working?”
S: “A couple months ago. We’ve had two tests since then. I got a B- on my last test before we were doing this. In the test we just had a got a B+”
Who is the manager of this team? Is it the teacher? The school?
Esther Derby has said (http://www.estherderby.com/2011/07/misconceptions-about-self-organizing-teams-2.html) that self-organizing teams require managers to “create the conditions that enable teams to thrive and continue to self-organize”.
The structure the school provides gives them boundaries for their work - regular times, periodic tests for review, the school network for connectivity, and the requirement that each student have a laptop.
The tests provide them feedback on how they are doing.
Mike Cohn says (http://blog.mountaingoatsoftware.com/the-role-of-leaders-on-a-self-organizing-team) “self-organizing teams are not free from management control.” This is certainly true of the students. They have to be in class on time. They don’t get to pick what time they have class (although what if they could?). They don’t get to choose their tests or other homework assignments. In the same blog, Mike points out that management chooses the product to build. I will stretch that to say that management chooses the goal and in this case, the goal is to learn the subject matter.
For my son and his classmates, they have mostly been together for three years so the team has had time to gel.
Not only am I impressed by the students’ abilities to very quickly take this approach from concept to implementation, but I am also impressed by the way these kids leveraged their tools to create a better way to accomplish their goals. I have recently read articles about “Millenials” and their impact on the workplace. (For example, http://paul4innovating.com/2013/01/25/millennials-see-innovation-differently-from-todays-leaders/). My experience with my son confirms for me that they can be real sources of innovation if we can give them an opportunity.
Correction on Feb 3, 2013: I originally referred to Esther Derby as “ED”. I apologize to Esther and have since changed to list her full name.
We recently began a new project at work. The team decided at the beginning of the project that they wanted to use the opportunity to learn new techniques. I would like to share a recent example that demonstrates two of them.
The first is the difference between Dialogue and Discussion. I was first introduced to this distinction from Peter Senge’s The Fifth Discipline: The Art & Practice of The Learning Organization. Per Senge:
The discipline of team learning involves mastering the practices of dialogue and discussion, the two distinct ways that teams converse. In dialogue, there is the free and creative exploration of complex and subtle issues, a deep “listening” to one another and suspending of one’s own views. By contrast, in discussion different views are presented and defended and there is a search for the best view to support decisions that must be made at this time. Dialogue and discussion and potentially complementary, but most teams lack the ability to distinguish between the two and to move consciously between them.
I learned the second technique we are using from Gerry Weinberg in The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully. Gerry introduces the notion of “The Rule of Three”. According to Gerry:
If you can’t think of three things that might go wrong with your plans, then there’s something wrong with your thinking.
When I use this rule, I think of it as:
If you can think of only two things, you haven’t really thought that hard.
It is easy to know when to apply the Rule of Three. Almost invariably in the course of a discussion, someone will say, “we have two options…”. When everyone on the team knows the Rule of Three, a comment like that is met with a call for other options.
The following example shows how the team used these two techniques to help them solve a thorny problem.
Many financial systems have an “Instrument” class. This class has the potential to be involved in quite a bit of the system’s functionality and, without care and attention it could get bloated and overwhelming. Over the last couple of years as we have been adding new services, this has happened to us. The class grew to over 2100 lines long, it directly imported over 30 other classes, and it had over 100 instance variables directly named in addition to supporting unnamed and uncounted variable arguments. In order for developers to work around this behemoth, they started creating separate, independent classes like “LightWeightInstrument” and “TradeCaptureInstrument”, further confusing the functionality related to Instrument. The team needed to leverage the Instrument class in their project and they decided they couldn’t use it the way it was.
The team set aside time for everyone to participate in a discussion on the design. One of the concerns they discussed was how best to handle those situations where the code needed to call out to an external resource for information. The current implementation used a combination of LazyLoading with caching to direct lookups whenever the resource was needed. As they dove into the concept of lazy loading, one of the team members felt like he suffered when working with a class that lazily loaded values. He worked on an application that had a high performance requirement and he couldn’t provide a predictable experience for his users. Someone else suggested that lazy loading is required to enable someone who doesn’t all of the possible values the class supports. The fact that the team understood they were having a design discussionhelped them to respond to challenges in a productive way instead of taking things personally.
At one point in the design discussion, someone said, “There are two ways we can do this”. The team immediately all set about trying to identify alternatives. In the course of the follow-up, one of the team members referenced the Interface Segregation Principle from Uncle Bob’s Clean Coder videos. He mentioned how, in the ATM example, Uncle Bob pointed out how you can wrap the interfaces of many independent but related classes together into one master interface for users. This strategy allowed the team to support discrete, “eagerly loaded” instances of a detailed class if they desire, while also supporting a higher level generic interface that will lazily load instances as they are needed. The team began digging into this idea and all agreed it addressed their needs.
The final solution the team agreed on would not have been found were it not for their understanding of Senge’s “Discussion”, Weinberg’s “Rule of Three”, and of course Uncle Bob’s Clean Coder videos.
Given the way many technical people seem to “fall into” a consulting role, it is not hard to imagine there are many things about being a consultant people don’t know. Jerry Weinberg’s “The Secrets of Consulting” is a treasure trove of useful information for consultants and employees alike.
The book relies heavily on anecdotes that show different perspectives on consulting challenges. The anecdotes are used to establish sets of practical, understandable rules for how to be a successful consultant, including such activities as how to interact with a client, and marketing and pricing your services. The rules are given names, like “Charlotte’s Law” to describe the importance of marketing oneself for quality positions instead of the quantity of work, and “Pandora’s Pox” to describe the desire for people to keep trying each new fad that comes along in the hopes that ‘this time it will be different’.
While the title of the book is about consulting, the wisdom it contains applies to everyone. Jerry refers to employees as “internal consultants” and frequently specifically mentions how to apply the rules in that case. For example, when he talks about the need for a consultant to work with many clients, he encourages employees to not get stuck for too long in a single job. This is consistent with a theme I have seen from Jerry in his books and workshops - a person is in control of his or her own destiny.
A minor quibble - I read the ebook and this particular one suffers from some early ebook issues like layout issues with chapters that blend together and a high number of typographical errors. It would be nice if these could be cleaned up.
Jerry’s books belong on every technical person’s bookshelf. This one is no exception. That said, if you are just getting started with your Weinberg collection, I personally prefer others like Becoming a Technical Leader: An Organic Problem-Solving Approach and the Quality Software Management series. Start with Quality Software Management: Systems Thinking.
One thing to note - this book is like one of those movies where you need to stay to the end of the credits. The last pages include a list of recommended readings. There is only so much time one can devote to reading and having a book list laid out by Jerry Weinberg is priceless.
I talk to a lot of people who ask for input on how they can get better at what they do. This is something I think about. So I want to share how I think about it and perhaps people can add comments on what works for them (since that is how *I* get better).
How can someone get better at being a programmer? I think of the following things that contribute to being “good” (with the idea that improving them makes one “better”):
For now, I want to talk about how we can get better at writing code. The quality of our code directly influences everyone else in the organization. In “QSM, Vol 1” Jerry Weinberg says “Quality is value to some person”. Off the top of my head I can think of a couple ways a person might value code:
I am sure there are more. The many different types of value that exist contribute to the challenge of building software. Even worse, if you try to build code to satisfy a particular person’s value, you may end up making the code worse for someone else! This leads to inevitable conversations around quality trade-offs - “If I build it the ‘right’ way, it will take longer”, “This code does what the user wants but is impossible to test”, “We are done with the code but component XXX requires manual deployment”, etc.
Attempting to address all known (and unknown!) interpretations of quality is a herculean feat. So what we do instead is try to come up with techniques and processes, which if followed, allow us to write code that delivers a consistent level of quality to known parties and provides a solid foundation for those people who have yet to give their input on the value they are looking for.
The specific code that you write has a direct impact on many notions of quality. How do you decide when you introduce a new class? How do you determine which method you should change when you need to introduce new logic? What is the best data structure for storing a particular entity? These are just a couple of the thousands of decisions that are made every time you touch source code. How do you get better at making those decisions? You read books, you search Google and you learn about techniques for doing things. You may even try those techniques as you code each day. But then you end up with different techniques and approaches sprinkled all over the code making it harder for other developers and even yourself to go back and look at something weeks or months after you wrote it.
To get better at something, you need to practice it. How do you practice coding?
One technique I’ve used is code katas. A small number of people at work have taken me up on the offer to do the bowling game code kata.
Another technique for practicing coding is something called a coderetreat. Code Retreats are focused, one-day sessions, for programmers to practice the craft of programming. You can read more about coderetreats at http://coderetreat.ning.com/profiles/blogs/how-a-coderetreat-works.
We have code retreats scheduled for the 18th and 19th of this month with Corey Haines, one of the founders of this technique. Here is a link to learn about code retreats as well as some of the other interesting aspects, such as “TDD as if you meant it”, which is at http://gojko.net/2009/02/27/thought-provoking-tdd-exercise-at-the-software-craftsmanship-conference/
I think this is some pretty cool stuff. Let me know what you think.