2 posts tagged with "planning"

"It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity, it was the season of light, it was the season of darkness, it was the spring of hope, it was the winter of despair." ― Charles Dickens, A Tale of Two Cities

Intro​

The product was called Self-Guided Tours (SGT).

Using smartlocks, property owners could allow prospective tenants (prospects) to tour an apartment unit or a single-family home for rent before leasing it out.

SGT was a big success. But as time passed, the system's complexity grew.

Property owners began asking for subtle ways to tweak their SGT experiences.

For example, to mitigate fraud and theft of these unoccupied units, a deterrent subsystem was implemented called geofencing. The idea: Don't show the entry code to the property or unit until the prospect is on premises.

On the Property Settings page, property owners could interact with this dropdown: I want Geofencing to be enabled: [Always | Never]

A Feature Request​

One day a client emailed us a new request:

Can you please add a dropdown to disable geofencing on the Tour page. If the prospect calls in indicating they are having trouble seeing the entry code due to geofence / cell phone / cell tower issues, there is nothing we can do.

The problem seemed valid.

On the surface the request seemed simple and the involved teams skipped Planning and Analysis, two critical Phases of the SDLC. A Feature Request was entered into JIRA where the email verbiage was copied into the story.

It made it into the next sprint and the change was in production in the blink of an eye.

While auditing that system and reviewing work that had been completed, I discovered a new dropdown on the Tour detail page that was always visible and it two options: [Enable geofencing | Disable Geofencing].

Within the scope of that Feature Request ticket, on the surface it makes sense, but let's consider now some of the challenges that we later faced.

We introduced cascading logic that had four possible outcomes (it's actually more because we didn't smartly set cascading-defaults in the initial implementation, but I'll spare you the gory details):

1. "Always enable geofencing" (Property Settings) + "Enable geofencing" (Tour Detail)
2. "Always enable geofencing" (Property Settings) + "Disable geofencing" (Tour Detail)
3. "Never enable geofencing" (Property Settings) + "Enable geofencing" (Tour Detail)
4. "Never enable geofencing" (Property Settings) + "Disable geofencing" (Tour Detail)

Subsequent Months​

What came next was a series of Support Tickets, over time, about problems with these two dropdowns, mostly around confusion on the cascading business logic, but there were also Implementation errors that added to the confusion.

For example, if "Never enable geofencing" (Property Settings), the code never even looked at if the tour-specific override was set to "Enable Geofence" (Tour Detail). 😅

For the most part, the Support team dealt with confusion for a while.

But then the following transpires:

1. Big Client A emailed us arguing that the cascading logic wasn't wrong and that "Tour settings should never supercede Property Settings." What they meant was "We want additional authorization checks in place before allowing any one of our staff to be able to modify geofence settings on the Tour Details". There was lack of Planning and perhaps understanding around that complaint, a bug ticket was filed to reverse the cascading business logic, an engineer implemented the bug report as written, and it was deployed. Happy customer.
2. 1-2 months later, Big Client B came forward stating that "disabling the geofence on the tour page didn't work. The original "bugfix" was already a distant memory. A new bug ticket was submitted and engineers patched it back to the original behavior, without discussing this in detail or reviewing the history of this feature. Oh no. 🫠
3. The system, again, no longer worked how Big Client A wanted: now their property staff could freely disable geofencing again. Here we found ourselves hotfixing back to the behavior from Step 1 above, and only now does Planning start, which ultimately led to a lot of arguing, a lot of refactor, and a solution that worked for both clients.

The Solution​

1. In Property Settings, keep the Enable / Disable geofencing for all tours (specifying this is the default behavior, but can be overridden on a tour-by-tour basis with proper permission).
2. On the Tour page, if geofencing was enabled at the Property Settings and if I, the user, had proper permissions, show a button that says "Disable geofence for this tour (Overrides Property Settings)". When clicked, show a button that says "Re-enable geofence for this tour" (just in case).

This allowed:

• Organizations who wanted to prevent geofence overrides could simply avoid assigning the new permission to their users
• Organizations who wanted to allow geofence overrides could assign that permission to their trusted staff members and the cascading business logic worked as expected (and was clearly documented on screen now)

Support requests in this area dropped substantially and we didn't hear any additional feedback around this system again. 🎉

Summary​

Churn and lax policies around following the SDLC resulted in scenarios with numerous frustrated customers (including some repeat frustration), worsened brand reputation, and a lot of extra effort put in by Support, Engineering, QA, Devops, and Product Management.

• Breakdowns in the SDLC process often happen as a way to "be more agile". This is a pitfall.
• Work that seems simple on the surface can actually underlying complexities, especially with pre-existing Production systems

Lessons Learned​

1. Plan and Analyze all changes / feature requests, regardless of size
2. Create some parameters around when Design should be looped in
3. Parse what the client wants from the ask, but don't let them steer the ship. It's our product to build, sell, and maintain
4. Put 1 SME (subject matter expert) in Planning / Analysis, someone who knows the pre-existing systems well and can play through the various scenarios in their head? The Design Phase having an SME also likely would have caught the overarching UX issues.
5. Ensure Tests Cases are written in the story. A quote from an email isn't sufficient for a story because it doesn't tell our Testers about what the expected behaviors are
6. Automated testing can catch subtle implementation issues preventing further bugfixes or hotfixes

The Problem​

I rolled out of bed on a brisk summer morning in Phoenix (95F) to walk my dogs before the heat would force us into our daily hibernation routine (seek shade; stay inside; dream of a home in Greenland).

Checking my texts as my eyes adjust to the aggregiously flamboyant sun peaking over the McDowell Mountains, I see a frantic, somewhat hyberbolic SMS from a director saying:

Whelp, I've poured myself a glass of Belvenie on ice and I'm tendering my resignation.

We made breaking changes to the org. reports. It broke for Client X because we didn't give them enough notice aannd their entire operation will be halted at 9AM EST (east coast based client) due to inability to process the new reports, costing them a huge amount of wasted time and money.

They emailed our CEO and I [the director] was cc'd. Nice knowing you, Jason. It's been a good run.

That's not how we wanted to start the day. The rest of the timeline:

• 4:15AM - Give doggos special-contingency frozen marrow bones. Sorry, doggos, no walk today. (They were pleased with their consolation prize)
• 4:20AM - Instruct director not to jump off any nearby tall hills
• 4:30AM to 5:00AM - Upload previous version of org. reports to every client's SFTP servers before 8AM EST
• 5:00AM - 5:30AM - Share post-mortem with team and implement new SDLC process specific to Change Management around the org-reports system
• 5:45AM - Drive to said director's home and have them pour me a glass of Balvenie such that we might lament together (and so they don't feel tempted to do anything irrational around surrounding, aforementioned tall hills)
• 6:00AM - Chin up and continue iterating on SDLC whilst building epic products

The Solution​

• Now, organization reports have a 90-day communication period with clients for breaking changes
• Any new fields added to these reports are optional for pre-existing clients who are subscribed, but can be enabled at their discretion. For new clients, all fields are enabled by default

Lessons Learned​

• Change Management processes around Production systems can be a critical step as an organization grows
• Ensure you have someone who can supervise and ensure those Change Management processes are being followed
• Heavily focus on documentation and training so that Product and Engineering teams are familiar with Change Management for these systems
• Use a CODEOWNERS to ensure the right supervisors are notified anytime these systems change and ensure there are clear policies around merging code in these systems.