On the Big Ball of Mud, a reading on the paper by Foote and Yodder

I found the Foote and Yodder paper "Big Ball of Mud" most interesting. I have spent much of my working career firstly, testing said balls of mud, then managing the building and deployment of the mud. I'm now a product manager, making sure we no longer build mud :)

For reasons of financial necessity or just disorganisation a lot of development teams have short sighted goals, where any kind of working delivery is viewed as a far better outcome than taking longer to make a more reliable or easier to maintain solution. By taking this approach, these short-sighted teams will build up technical debt. That is, they will get their working delivery on a high-interest lease program, with a large payment due in the future as their system is difficult to  expand and upgrade.

Think of an ERP system. In one of our previous modules, we were told a number of large investors used a company announcing the roll out of an ERP system as a merger and acquisition strategy. These investors considered ERP systems so complex & expensive a number of companies investing into getting them in place will run into problems and cripple the company. You often see and hear of companies having these systems in place, and they get someone to customise them, to suit their practices & desires. This makes this easier day to day, which is good. On the flip side this is a fast track to ball of mudland!! I managed an instance of SalesForce [a CRM system] for a large finance company. Their SalesForce instance was *very* heavily customised. What this meant was that when SalesForce did their system wide upgrades 3 times a year our system fell apart, and once took three weeks to fix. When I left that company SalesForce were about to retire two features we were heavily dependent on. We had never taken the time to upgrade them [3 years after they first said they would replace them]. Management simply put the update on the long finger, and left them alone as we had spent so much time and money customising the system. That it was deemed too expensive, it did not add any value. So then, down the road, they were faced with a major upgrade project, with a gun to their heads, and half the time and necessary resources available. Insert more mud here!!

I do like the focus, so far on this course, in thinking about how a product manager would think. As a product manager you need to think about getting your "thing" out the door. But you also must have an eye on medium and longer term goals. So there is very little benefit in getting a version 1 out there, if its unstable or impossible to maintain, as by the time you deliver it, plans are well underway to your next set of enhancements. Consider this article, from an industry blog I often read. This article describes how you break down complex items into easy to digest user stories which your development teams can then consume. They can build their solutions accordingly. But it is very much implied [and I would reinforce in person], that in the example on that website example, I want the ability to later add currencies, change prices etc. You build your system with that in mind from day one, it takes longer out the door, yes - but later enhancements are easy [as is testing and eventual refactor & redesign]

Iona..

I'm going to enjoy reading & watching the Iona case studies. Not only am I old enough to remember when Iona had massive hype as "the next big thing" and one of the leading lights of the Irish tech scene.

I also work for the current incarnation of what was a company founded when Iona was broken up. Cape Clear software was a spin-off company from Iona, and was formed in 1999. It was based in Ireland and Californa. Cape Clear became one of the world's leading providers of enterprise service bus software. In 2008 Workday Inc bought Cape Clear, and what was Cape Clear became Workday Europe. Read this interesting blog post, posted by Annrai O'Toole in 2013. You will notice the name, as Annrai is listed in this week's case study as an Iona co-founder, and is now CTO of Workday Europe. The blog covers the first 5 years of the partnership between Cape Clear and workday.

Working on the move..

I was at the rugby in London over the weekend. The flight was delayed on my return. What is brilliant is that I don't really miss much in terms of work.

I have full email and calendar access from my phone here. I can chat with the various teams in work with on Slack. I was able to dial into one meeting from the Gatwick express train (we had folks on the call from 3 different countries). This was made easier as we use a phone service with local numbers in each country. So I'm 2 hours late to the actual office,  but thanks to some good technology solutions the world keeps turning & no one is stressed :)

Not only that,  my multi tasking extends to doing this blog post for college on my phone,  with the blogger app!

Fieldwork Research, Ticketmaster

For my research, I looked at the ticket selling site, Ticketmaster. In days of old when any big event went onsale there were invariably backend issues, resulting in front end timeouts or full-on crashes. A couple of weeks ago, U2 put tickets for Dublin concerts on sale, and TicketMaster unveiled a new process to prevent such issues reoccuring.

Im going to look at this progression from old system, to the new system with the help of two different protocols. First, I am going to do a 5 whys on the process of site going down and what could be done around that. Second, I'm try the site, and be my own customer.

Ask - 5 Whys

• Ask why in response to 5 consecutive answers
• Ticketmaster crashed under high load
• Large amounts of users overload web server and backend systems
• Ticketmaster front end systems flood the backend with more users than it can handle
• Web servers handling high user load is very easy, but one point of failure in backend would show a failure in front end
• Backend failures are often costly, difficult and time consuming, they also take the entire site down, to overall lost revenue is extreme, and not just limited to one set of concerts :)

Oops, you need to start again - and by that stage you know the concert will be sold out :)

Ticketmaster's solution is a clever one, in times of high load they have a queueing system. I could imagine a thread based queuing system. The system is designed to handle a specific number of simultaneous transactions, and above that you simply go into a number of queues, and each person is addressed in turn.

Still not perfect, but waiting in line for a chance to buy a ticket is far better user experience than the site full on crashing :)

Try - Be your customers

• Outline typical customer experiences

Previous customer experience was that in the event of a busy concert the website eventually would grind to a halt, and it was a matter of when and not if the site would crash. Knowing how these kinds of backend systems work, there is likely to have been at least one failure point.  I have seen many systems where one part of the system does not perform under load, and when a specific point is reached, it either slowed down or crashed full stop. I worked on one system where when tested in isolation each end point was perfect under load of tens of thousands of users. But when together on the same set of servers they would crash with 100 users doing regular actions. The items put each other under stress, and importantly put their host machine under stress.

Ah, you nearly got to the end, but our payment gateway is down, sorry - start again and hope for the best :)

Under this new system at ticketmaster, they are clearly regulating what customers reach the critical points of the backend systems. So at a basic level, they have a funnel. So they have a trickle of users reaching the choke points in the backend. So their site stays up, and is performant for users buying tickets for the busy concert, and also importantly, other users of the site.

Another good idea, in terms of keeping the site up, was that for the U2 concert one had many ticket options. By having many different classes of ticket, everyone was not hitting the same end point in the backend, you had ordinary tickets, student tickets, VIP tickets, Premium tickets, I cant believe its not butter tickets, the list goes on. This means you have 5 or 6 main queues and not one epic queues. This is good traffic management, if a little confusing for the end user.

Activity Checklist

Activity theory is in effect a way in which you, as a product manager, can build a checklist by which you can analyse and break down complex issues and problems into different sections. It is important in that it mixes what actions are computer controlled and which actions are driven by human interaction.

"The five basic principles of Activity Theory:

1. Object-orientedness
2. Hierarchical structure of human activity
3. Internalisation and externalisation
4. Mediation
5. Development"

We can break down these activities to analyse an existing system, or indeed to start to design a new system.

It makes me think of a book I was given as a gift when I was first promoted to the role of product manager. That book is called "The checklist manifesto". The book breaks down how the most complex activities can be made more easily repeatable and safer by making people who are involved follow a checklist. It shows examples how mortality rates around medical procedures fell dramatically when staff were made follow a checklist, even around simple things like ensuring the correct person was being operated on, the correct procedures being performed and so on. I had occasion to see this in action this summer, as I broke both my arms. When I went into the first surgery, they had down that my left wrist was to have pins and plates inserted into it [this was wrong, my right wrist was the broken one!!!]. This was only caught in pre-operation checks, as I was about to be put under an anesthetic. You can draw a correct parallel with an IT system. You begin with simple questions like "what problem are we trying to solve" and you then ask the questions which logically follow on from that. You can easily make this into a checklist or set questions, and thus follow it for all changes.

The Soul of a New Machine

We had assigned reading for this class of The Soul of a New Machine. I cheated on this a tiny bit, in that I was away last week, and the book I had with me was not brilliant [to say the least]. So I decided to be good, and get this one and check it out. I was sitting in a bar in Lisbon, Portugal. I launched my Kindle, and found the book in 2 minutes, cost was only €4.50, and 2 minutes later I had the book. On its own this is a brilliant use case for the kindle and its Whisper sync functionality. I can get a book basically anywhere on earth, in seconds. You cannot but be amazed and admire that, regardless of what you think of other aspects of Amazon's business practices.

On the subject of the book, many wont have read it yet. Without giving anything away, its interesting that at the very start of my career [a long 16 or 17 years ago] I worked in a couple of software companies with a culture and environment a lot like what the book describes. Overall at the time I did not mind, but I look back and think that I learned a lot from those experiences, but many of those learnings are things not to do again and places where we were disgustingly inefficient in terms of planning or process. I think many of the people in the book would say the same thing.

Anyway, more on that whole subject in a few weeks :)

First post..

This is a blog for the Managing Design and Development class. We are doing this class as part of the MSc in iBusiness class at the Smurfit school of business. I am in the second year of this class, and this is my seventh module.

By day, I am a senior product manager with a large cloud based software company. My focus with that company is  on their internal monitoring and development tools. I have around 15 or 16 years experience working with development teams, from a range of different points of view and perspectives. 

The blog title also refers to the fact that during this semester I shall become a dad for the first time. It's therefore inevitable that the odd baby reference shall creep in here. Indeed, we have already have a planning meeting, complete with post-it notes in different swim lanes, as you would plan a software project :)