How confident are you about your product quality?

A story of how I handled a critical quality and tight deadline project

During my tenure at HSBC Investment Bank (Southern Africa) in the early 2000s as the Architect, I encountered numerous challenges, the most formidable of which emerged when the Johannesburg Stock Exchange decided to transition its trading platform, SETS, to the London Stock Exchange platform. We were alerted to this significant change early in 2001. The technical implications for this are massive due to the profound impacts on the business model and systems. This entailed a comprehensive overhaul, including the trading platform, live market feeds, and all downstream systems such as client portfolios, market risk analysis, corporate trading systems, market research tooling, data warehousing, live tickers, and CRM.

Design principles

A fundamental design principle I've always adhered to is creating high cohesion and loosely coupled systems. Equally important was devising an architecture flexible enough to evolve with the business. In the day, I called it a plug-and-play architecture; today, it is best described as an emerging and evolving architecture. This approach significantly contributed to our success; most systems required only minor changes due to separation of concern and loose coupling.

Nevertheless, we faced two substantial risks. The first was our proprietary trading system, which necessitated a complete rewrite due to the vast differences in trading models—a feat impossible to achieve within the tight 16-month deadline. After presenting facts to the executives, a collective decision was to procure a new trading system. Building our own new system would be too risky. The second risk involved our live market feed, integral to numerous dependent systems. Failure here meant jeopardizing the entire business operation.

Crazy deadlines

The London Stock Exchange imposed a stringent compliance deadline of 31st March 2002, beyond which any uncertified system would be barred from connecting for six months—effectively crippling our trading capability. The pressure was immense, and a real “deadline” for failure closed the business, and many people lost their jobs.

The pressure escalated as the JSE took their sweet time releasing any technical specifications. When I first received the initial technical specifications from the JSE on 21st December 2001, a document riddled with errors, yet a starting point, nonetheless. We had 3 months to pull off a miracle.

Commitment to Quality

My commitment to quality is unwavering. I reject subpar standards as vehemently as I do unnecessary complexity. I advocate for a professionally appropriate level of quality that mitigates risk. I assembled and led a dedicated team of ten, including contractors, supported by management, tasked with this critical project. Given the urgency, we worked in seclusion to avoid disruptions.

In this case, quality was essential because defects meant we could not fix them for six months, and the business failed if a P0 or P1 defect occurred. Time was also a factor, as the looming 31 March was only three months away.

The work

I had full support from the executives and got everything I asked for. I insisted on working closer to an XP model without project managers; I needed some extra technical skills, and we hired some contractors we previously used. I asked that we work in isolation, and they gave us access to their disaster recovery site, where we were the only ones.

The final stretch from late December to 31st March was grueling. My routine comprised 16–18-hour workdays, seven days a week, often sleeping under my desk or the sick bay if not occupied. We showered onsite. We were oblivious to the time of day.

Through relentless effort, we successfully implemented both the new trading system and the market feed system, completing our final conformance test on 30th March. With the code digitally signed to prevent alterations, we deployed it onto the production servers for launch. Any failure meant waiting until October for the next compliance opportunity, putting my career and many others' jobs at risk.

In April, we focused on tweaking some internal systems that received data from the feed; these systems were exempt from LSE compliance. By then, the go-live date was fast approaching, yet confidence in our product's quality was high among the team and me.

D-Day

On the morning of going live, pre-trade started at 6 AM, and the market opened at 9 AM. So what do you think I did?

I asked the team at the office at 7:30 AM; we then went to the local mall and had breakfast and a surprise of tickets to the new Spiderman 1 movie starting at 9 AM. We had breakfast and enjoyed the movie. After the movie, we returned to the office to find nobody had missed us. The system was a triumph, and the traders were doing their thing.

All good does not end well

Despite this success, I ultimately left HSBC Investment Bank South Africa due to them breaking their promises of remuneration for my critical role in "saving the company." I and all staff were basically met with, “We are so grateful for you getting us out of a pickle and contributing to a really profitable year, but sorry, no bonus, no increases, and no, you cannot claim overtime”. It was all a greedy money grab from the executives despite them having their biggest profitable year with a huge margin. Naïve Brett learnt a lesson that day: get all promises in writing.

Years Later, I learned that the live market feed system we developed remained in production for 9 years with only a two-hour downtime during a cloud migration of servers, boasting impeccable reliability without a single defect.

This is important.

This narrative exemplifies what we should all do: refuse to compromise on minimum quality standards. Do not compromise quality, but also don’t over-engineer it. Do what is professionally right. Ask yourself how confident you are in your quality that when you say it is “done”, you know there will be no defect? Do you push back when people try to compromise that agreed quality?

Back then, I had never just heard rumblings of Scrum but had not paid attention to it. But “Done” and “Definition of Done” resonated with me. It has held me in good stead for my career, and I have seen it work in past clients with the same attitude, but I also see failure when they don’t value quality. Unless you enjoy the pain, learn what “Done” means.

When you think of Done, think of a cocky Brett being so confident in the quality that it’s okay to do something else; like go to the movies on go-live day. Done means “DONE”!

Using Scrum? Have Quality issues? Let's fix them. Could you book a consultation now and let's turn that around? Connect with me, and let’s arrange it.