Category: Management

Do the riskiest thing first

Sachin Dhamdhere

Today’s blog is inspired by my experience delivering works of software in what most teams would describe as ‘aggressive timelines‘, even by typical high-flying software shop standards. A meeting at work triggered me to blog about it.

So the general theme is this: at the outset, list out all the things that scare you about a project, list out all the things you don’t know, list out all the things that’ll break your customers experience – once you have those, stack rank them and pick the riskiest thing(s) to do first – you’ll have much better outcome than otherwise. You’ll have de-risked the future. It is true you will have inaccurate estimates in the beginning making your stakeholders questions ability of the team to deliver – but with open and transparent communication you should be able to persuade them about this approach. Other advantage is at the beginning you’ll have lot more control over schedule slips and dealing with downstream impact. As you make progress you will deliver the necessary features on time and with high certainty because the riskiest items are either done or no longer unknown. Avoid the trap of doing things that you can, are known to you at the beginning, reason is simple – you know them, you can execute later.

Now, to the specific trigger: we are re-architecting and releasing a service where a few new components are in the mix and will be serving traffic from all customers by routing them appropriately (think API Gateway-like services for routing, authentication and authorizing traffic). In the existing version of the service, each cluster of software is isolated, thus there’s no “noisy neighbor” problem. Given current incarnation of service is available for many years now, there’s robust tooling available for conducting performance testing – on existing software stack. The engineers working on performance testing the new version of service had fallen in the track of thinking: “this is how its done in existing service”, “this is something we can do now”, “we don’t know how to write tests, harnesses for new service & it’ll take time”, “we don’t know how to project traffic patterns for the new common components, it is 10K requests/second or 100K or a million” … you see the trap, right? The customer experience will largely be determined by scalability of new components & we were punting them and favoring to do things we can & do them now.

This will lead to rush and stress later on when team figures out how to write those tests, how to measure performance, how to simulate traffic to find out the new components don’t scale well or need a lot of tuning or need a lot more hardware behind them or worse re-design some – putting the go-live date at risk or outright missing it. If it turns out we need to redesign a few things – it’ll lead to even more costlier delays – it’d be best to invest time in unknowns, riskiest things – right now Vs doing them later, yes it’ll be uncomfortable to write tests with unknowns but the discovery phase of this will help not only performance engineering team but overall org by bringing to the front questions we may not have event thought about.

Management lessons from trenches: Part 1 – Remote Meetings

Sachin DhamdhereLeave a comment

This is first in the series I intend to write about “management lessons from trenches”. I’ve worked in small and large software companies – building, shipping software, running in-the-cloud services and being responsible for large geographically distributed teams. These are the lessons I learned, some the easy way & some the hard way – paying in terms of missed deadlines, less-than-beautiful software, that dreaded call about your micro-service not working at 2 am (why is this call, real or proverbial, always at 2 am??) but hey! learning never ends…

So, here’s a typical scenario in most software development shops – you are setup with multiple teams in different geographies, across timezones & across cultures, most of the times there are cross-dependencies between teams across functions (dev, QA, SRE etc.) The upstairs people (bosses, yes, me included!) of course want things built with reliability of Trump-tweets that cause a storm on the left and (&&) the right & of course, all of it must be done yesterday! Your teams are always looking to make progress, get things done & more often than not, you end up doing meetings, many meetings – using a audio & screen-share conferencing tool so everyone can join, listen, participate & you all can move the ball forward – what could be noble-r than this? Indeed, it is good to collaborate, to make everyone on the team see the world same way – so all arrive to promised release/go-live date looking somewhat sane and still friends with each other.

This is where I’ve seen, time & again, things go off-the-rails, in the enthusiasm of meetings, rapid-progress – we completely the miss the point that people we pull-in to the meetings are remote as not-in-same-room, they can not hear all the side conversations going on the conf room, they can not see the facial expressions of people, they can not read the non-verbal cues – all they have to go by is the words they hear on phone or the images they see on the screen – if you are lucky a video conferencing call. You will find this a trivial point, an obvious observation – but the impacts of this are far-reaching – continue this habit or this way or running meetings without understanding the remote-ness, without paying conscious attention to it – you’ll start alienating the remote teams, they will lose interest, they will not understand the designs, they will come to regard those implementations as something forced-upon them rather than being their own – a toxic team environment is forming & soon spreads like a disease … so don’t be that jerk who runs meeting with remote attendees the same way as in-person, everyone in the room type meetings. Here are some simple rules (which are good to follow for any meetings, really) …

  • Precise list of attendees: Make sure all the right players & ONLY the right players are invited to the meeting – if someone is not going to derive value from meeting or someone is not going to provide value to the meeting – spare them, trust me, they have productive things to do & will be thankful! Bosses tend to tag-along for many meetings – the rule is same for everyone, you either derive value from the meeting or you provide some – so ask boss which one is it? (ok, not all bosses may take this in right spirit, just because I do – but you can be crafty about this)
  • It’s not 10 AM everywhere in the world: People work in different time-zones, they have outside-of-work commitments – so be mindful of time you are scheduling the meeting at – make sure it doesn’t overlap with family dinner time for your India or China or Eastern Europe teams. Just because you work at HQ (wherever it might be) – doesn’t mean all other remote locations need to adjust their schedule every time there’s a remote meeting, may be take turns – you join late night or early morning call at times & at times expect others to do so, this fosters trust & team-spirit, sends a message “we are all the same & in this together”.
  • Publish an agenda: In the meeting invite, state the goals of the meeting, be explicit with words about what it is you are expecting to achieve by end of the meeting, it could be a decision the team needs to arrive at by considering alternatives, it could a design pattern being proposed, critiqued – but in the end voted up or down – something that is tangible & can be referred to after the meeting is over.
  • Stick to the agenda: Its natural to get drifted in many different directions when people are sitting in same room, across tables & side conversations are a natural thing – but as dude (or dudette) leading the meeting, be firm & stop the side conversations.
  • Require participation: For every item discussed require local as well as remote attendees participate in discussions, decisions. There’ll always be people (statistically speaking at least 1 person who wants to dominate the talking part of meetings!) – either on the phone or in the room – who simply love to talk, but as dude (or dudette) leading the meeting – intervene, politely tell people to let others speak up. Make sure everyone participates, this is NOT “encouraging” participation, its active, mandatory participation.
  • Share the running notepad or meeting progress updates: As dude (or dudette) leading the meeting – either take live notes yourself or appoint a scribe to take notes & most important share those notes – real-time (Zoom, GoToMeeting – whichever tool you use), this gives a visual orientation for everyone involved, an orientation about what is being discussed and how is it relevant to the agenda.
  • One speaker at a time: As dude (or dudette) leading the meeting, take charge, allow one person to talk at any given point and be respectful for time
  • Followup with post-meeting notes: These could be in form of minutes-of-meeting, decisions-we-took, activities-we-agreed-to-do but be specific about who, what & when (& if needed how) if you have shared workspace, Wiki, shared-doc, even better.
  • Beware of these phrases:
    • “Let’s take this offline“: if this is said about one of the agenda items, intervene vehemently – the time to discuss the matter is here & it is now.
    • “We are working xxx team on this“: These can be vapor-words either suggesting a meaningful collaboration is underway or nothing is being done, we are simply talking or wanting to to talk to team xxx – so insist for details, specific steps, concrete actions that be traced, viewed, reviewed.
    • “I’m looking into zzzz“: What does it mean? what’s there to look into? Again, insist for concrete steps, actions, outcomes – this makes tracking, asking questions, followup, setting goals easy Vs nebulous “looking into it”.
  • Be wary of recurring meetings: If you have good (remote) meeting practices in place, in all likelihood, you’d not need recurring meetings, if you do – please, please ask participants if they find these meeting productive & if not course-correct in time. Too often such recurring meetings become a boring time sink where people end up checking their emails or engage in other social media. Think of it this way – you are inviting “N” people, for say an hour-long meeting for “X” weeks & assuming the average cost spent per employee per hour is $100 (pick any number that makes sense to your situation) – is that meeting producing “100 times N times X” worth of value? (measure value whichever you chose, tangible, intangible …)

You must be thinking – “geez, this is too much“, well, it is, no one said building great software across continents and oceans is walk-in-the-park! But the good news is, with 3 to 4 such meetings it gets easy & everyone involved will follow along & understand the rules of the game and you will build an efficient, effective team.