2 posts tagged with "change management"

The Problem​

Clients had the ability to press a Sync Now button to sync data from their systems to ours.

We received feedback early on that it was difficult to find and use, in it's current location on a sub-page, so we moved it to the sidenav where it was always on display. Innocuous decision, right? Simple, elegant. Easy-to-access from anywhere.

Growing Complexity​

One day, we ran into our first use-case where a specific client had two systems we needed to sync from.

Shoot, now what? Do we put one Sync Now button for each client integration? Do we display a modal popup and ask which system to sync?

We opted to move the Sync Now button to the integration settings page.

The next morning a new Slack message:

Clients are furious we removed the Sync Now button from the sidenav, we need to hotfix it back. Highest priority.

Multiple Attempts At A Fix​

Round 1​

So we hotfixed it back while figuring out our game-plan and that's when things became not-so-innocuous. At first we tried to implement a strategy to ask the user what they wanted to sync from a list of integrations that were enabled.

We're getting feedback from most of our customers, who have one integration per property that they are upset they need to take that extra step.

Round 2​

When each page loads (because it's on the sidenav), return a list of integrations and so we know if 2 or more are enabled. If so, ask them to select an integration to sync from, otherwise just show the toast popup for "Sync started"

...That worked for a while but over time new problems arose:

• We said Sync started (processes that can sometimes take 10 minutes or more) when we didn't know if it started because we couldn't keep network requests open for that long. Sometimes it failed due to invalid credentials or misconfiguration, leading to support requests
• There was no feedback around what synced, if anything. Sometimes it would connect to the integration and re-sync data, but no new data was synced, and furthermore we didn't have a place we could easily showcase how much data we were able to pull over

The Solution​

The idea behind the final UI iteration was to move this button to any page where users are checking if their data is synced or needs to be synced. If multiple integrations were enabled that supported syncing data, show a dropdown and have the end-user select the integration they want to sync from.

First, this cuts down on data-fetching on every page load since it's no longer in the side-nav.

Second, with the ability to refetch the data to see if new data was synced in, there is a feedback loop where clients can see if the Sync Now click worked or not. The clients are also happy that the ability to Sync Now happens on the page where they see stale or missing data.

Third, utilizing websockets notify the button clicker when the sync completed or that there was a problem with a Toast popup or Alert. If there is a problem, include a link to where they can triage the problem and then resume their data syncs.

Lessons Learned​

• Design and UI/UX are a critical facet of the SDLC and when building new products or systems, early design decisions can become quickly solidified in Product Marketing content and Knowledgebase How-Tos. The cost of changing Design decisions later is very high.
• End-users can become resistant to change, especially if the product is a SaaS offering that is meant to improve their productivity or help them perform critical tasks. Redesigns must be properly communicated and in some cases, it's beneficial to let have both experiences in parallel (or to toggle between them) so that they can get familiar with the new design patterns when it's not interrupting their daily workflows.
• Whenever a relationship between two entities exist (organizations and data-sync integrations, for example), building with a many-to-many relationship instead of a one-to-one relationship in mind can cut down on refactoring costs later, even if there aren't immediate use-cases for many-to-many relationships present

The Problem​

I rolled out of bed on a brisk summer morning in Phoenix (95F) to walk my dogs before the heat would force us into our daily hibernation routine (seek shade; stay inside; dream of a home in Greenland).

Checking my texts as my eyes adjust to the aggregiously flamboyant sun peaking over the McDowell Mountains, I see a frantic, somewhat hyberbolic SMS from a director saying:

Whelp, I've poured myself a glass of Belvenie on ice and I'm tendering my resignation.

We made breaking changes to the org. reports. It broke for Client X because we didn't give them enough notice aannd their entire operation will be halted at 9AM EST (east coast based client) due to inability to process the new reports, costing them a huge amount of wasted time and money.

They emailed our CEO and I [the director] was cc'd. Nice knowing you, Jason. It's been a good run.

That's not how we wanted to start the day. The rest of the timeline:

• 4:15AM - Give doggos special-contingency frozen marrow bones. Sorry, doggos, no walk today. (They were pleased with their consolation prize)
• 4:20AM - Instruct director not to jump off any nearby tall hills
• 4:30AM to 5:00AM - Upload previous version of org. reports to every client's SFTP servers before 8AM EST
• 5:00AM - 5:30AM - Share post-mortem with team and implement new SDLC process specific to Change Management around the org-reports system
• 5:45AM - Drive to said director's home and have them pour me a glass of Balvenie such that we might lament together (and so they don't feel tempted to do anything irrational around surrounding, aforementioned tall hills)
• 6:00AM - Chin up and continue iterating on SDLC whilst building epic products

The Solution​

• Now, organization reports have a 90-day communication period with clients for breaking changes
• Any new fields added to these reports are optional for pre-existing clients who are subscribed, but can be enabled at their discretion. For new clients, all fields are enabled by default

Lessons Learned​

• Change Management processes around Production systems can be a critical step as an organization grows
• Ensure you have someone who can supervise and ensure those Change Management processes are being followed
• Heavily focus on documentation and training so that Product and Engineering teams are familiar with Change Management for these systems
• Use a CODEOWNERS to ensure the right supervisors are notified anytime these systems change and ensure there are clear policies around merging code in these systems.