LogRocket Interviews: Is scoping more trouble than it’s worth?

Virtually all software development projects begin with a period of tedious and detailed planning. Every stakeholder wants to maximize the organization’s engineering resources, and company leadership wants to know when they can put their products and services on the market. Wrong turns in the development lifecycle impact bottom lines, and yearly budgets require accurate predictions around timelines, headcount, and more.

The goal of most scoping exercises is to minimize these pitfalls in the early stages of development through meticulous and prescriptive planning of deadlines, checklists, and project blueprints. But there’s an opportunity cost associated with overplanning, and failure to strike the right balance could limit engineers’ ability to think critically and solve problems creatively.

Given these risks, are scoping exercises — at least, in the traditional, waterfall sense — more trouble than they’re worth?

Whether your organization errs on the side of detailed scoping or takes a more agile, plan-as-you-go approach, communication at all levels is paramount for engineering and product team managers. That means understanding your developers’ individual skills and goals, managing the expectations of company leadership, and, most importantly, articulating the desired outcome from the end user’s perspective.

In many ways, managing a software project has as much to do with building relationships, juggling egos, and navigating conflicting priorities as it does with actually building software. We talked to four engineering and product team managers to learn more about the art behind the shaky science of predicting the future — the dreaded exercise known in the developer world as scoping.

TL;DR

Krishtha Spuglio, senior product manager at HubSpot. She has managed a wide range of products, including sales enablement tools and customer-driven solutions, at her current company. She now oversees internal-facing antiabuse systems designed to identify product abuse and suspicious behavior.

Jordan Smith, an engineering manager who previously worked at PillPack, a full-service online pharmacy that was acquired by Amazon in 2018, during its early growth stages. At his current company, he manages a team that primarily works on order systems — specifically, changes made to customer orders before they ship.

Kate Green, a self-described “unorthodox problem solver” who recently started working as a software engineering manager at WireWheel, which helps companies achieve compliance with data privacy regulations. Her previous experience includes working as a lead software engineer at Till.

Mari-Megan Moore, customer experience manager and head of design for B2C commerce at SAP. She spends most of her time overseeing the design of SAP Upscale Commerce, a SaaS-based unified commerce platform for mid-market retailers.

Scoping is simple, but not easy

“If the problem is that you’re stubbing your toe on a piece of furniture, your solution is going to be isolated to either that piece of furniture or your toe.”

The primary objective of scoping is simple: to determine what you can accomplish within a given period of time. But how you arrive at that estimate and to what extent it should influence the development lifecycle are much less straightforward.

There are practical and logistical factors to consider, such as the organization’s engineering budget and available resources, predetermined go-live dates, promises made to stakeholders and customers — the list goes on. Managing all these moving parts requires a certain level of focus on things like alignment, prioritization, and accountability — all of which we’ll get to later — and the specific steps of any scoping process depend first and foremost on the desired outcome from the end user’s perspective.

No matter what type of project you’re scoping, the first step is to answer a single, fundamental question: what is the problem you’re trying to solve?

“The scope of a project almost always depends on the scope of the problem,” says Krishtha Spuglio, a senior product manager who works primarily on in-house antiabuse systems for HubSpot. “To give a silly example, if the problem is that you’re stubbing your toe on a piece of furniture, your solution is going to be isolated to either that piece of furniture or your toe.”

But for most software projects, the problem is broader and the stakes much higher. Business leaders make promises — whether to customers, investors, or the board — that engineers are tasked with keeping. Suboptimal internal-facing programs impact productivity and, ultimately, the business’ bottom line.

Without mutual understanding and open communication between business and technical leadership, conflicting priorities drive unrealistic expectations and erode trust on both sides. When engineers are more concerned with meeting deadlines than actually building, they’re less likely to produce exceptional software. And when business leaders see a shoddy finished product, they’ll want to know where the process broke down.

Why engineers lie about timelines

“It’s not that we’re unwilling to do this planning or that we’re lazy. It’s just that you shouldn’t want us to spend this much time doing estimation; you should want us to spend time doing work.”

That’s where accountability comes in — and where things get tricky. The traditional scoping process revolves around business stakeholders’ desire to minimize the aforementioned risks. But at what point does scrupulous planning become self-defeating?

“It’s not the nicest phrase, but it should be a plan, not a suicide pact,” said Jordan Smith, a software engineering manager who formerly worked at PillPack. “Just because something was a good idea six months ago, it doesn’t mean it’s a good idea now. You’re hiring effective engineering managers, not fortune tellers.”

According to Jordan, the conventional steps associated with scoping — biweekly sprints, retrospectives, planning poker, etc. — serve more to assuage engineering and business leaders’ anxieties than to optimize the development process.

“I have my playbook, there’s a certain set of prescribed activities, and it’s really comforting to know that if I do these things and I do them well, then I have succeeded in some sense because I did the things that the book told me to do. And the org told me to read the book, so I’m good.”

The trouble with this model is that the solution to one problem often reveals a web of additional problems. If forced to address these issues within the time frame that was initially scoped, developers may resort to cutting corners to meet deadlines.

In the long run, Jordan says, the classic waterfall approach to scoping could also influence engineering managers to deliberately overestimate the time and resources required to complete a given task.

“People wonder why every project that is estimated at nine months takes 18 months. Well, it’s because when I told you nine months, I secretly thought it was going to be four. I blew out my estimate just to be safe, but then I learned halfway through a bunch of new shit that I hadn’t thought about. And that’s basically the reason why 50 percent of projects fail, because they’re running in this waterfall manner.”

Furthermore, there’s an opportunity cost associated with the sheer amount of time it takes to complete a thorough scoping exercise.

“It’s not that we’re unwilling to do this planning or that we’re lazy. It’s just that you shouldn’t want us to spend this much time doing estimation; you should want us to spend time doing work,” Jordan continues. “Sometimes I say, yeah, sure, I can spend the next two weeks really trying to plan this super thoroughly, or instead I can just start building and I’ll learn more from starting to build. We will get to the end faster and, along the way, there will be less risk.”

If you’re not careful, front-loading the development process with strict frameworks and hard deadlines can lead to a lose-lose scenario: either you stick to the playbook and limit your flexibility to pursue new opportunities as they arise, or you throw out the playbook, pivot, and eat the time spent planning.

Kate Green, a software engineering manager at WireWheel, is a self-professed “overplanner” when it comes to scoping. But she acknowledges that plans change all the time, especially if you work at a fast-moving startup, and engineering teams risk painting themselves into a corner if they don’t leave wiggle room to adjust on the fly.

“In this sprint I’m on one thing, the next sprint I’m on something else entirely, and then by the third one, again, I’ve moved on.” said Kate. “So sometimes that long-term planning process isn’t worth it. Sometimes you just have to keep moving. And in those cases, that’s usually when things kind of go a little wild.”

Like it or not, scoping is ubiquitous

“A development cycle without scoping gets you something big and probably a bit haphazard … I think it’s the absolute wrong way. But then again, I’m a planner.”

In an ideal world, time would stop so developers could examine every angle of how to solve a problem. But the wildcard factor Kate referenced is precisely why scoping exercises exist. From a business perspective, budgets simply need to be hashed out and goals must be met. The business world does not stop to accommodate the software development lifecycle.

And, certainly, scoping has benefits from an engineering standpoint that even the most deadline-weary developers can appreciate. Let’s start with the most obvious: scoping helps businesses focus their engineering resources on tasks that will help solve a given problem or achieve a goal as efficiently as possible.

A world without scoping, argues Mari-Megan Moore, customer experience manager for SAP Upscale Commerce, is a world full of janky software and bankrupt tech companies.

“Without scoping and the right process, you would spend four times the amount of time on something that probably wouldn’t deliver the right value to the customer,” she says. “As a result, your product will probably fail. Or you’ll spend all your money. One of them is going to come first.”

Careful, upfront planning is also conducive to long-term flexibility. It may seem counterintuitive, but limiting the scope in the early stages of development actually enables teams to expand their focus as new opportunities present themselves.

Krishtha likens it to a fishing expedition. Set before you is a vast ocean of possibilities, but sinking all your resources into an area devoid of viable solutions limits your ability to adapt to changing conditions and pursue new discoveries.

“If we didn’t spend time scoping things down, there would be more risk in doing any single piece of work. But I also think you’re less likely to be able to pivot to what the direction ultimately should be,” she explains. “When you start with something pretty big, you’d like to be able to take creative license, but it’s much harder for you to change course compared to when you’re working on smaller iterations.”

It’s a delicate balance: you want to empower you engineers to think critically and approach problems creatively, but when that freedom compromises the team’s productivity, it impedes their flexibility to do just that. The result, again, is suboptimal software.

“A development cycle without scoping gets you something big and probably a bit haphazard,” offered Kate. “It probably will work, but it’ll be a little clunky — and I’m being generous here. I think it’s the absolute wrong way. But then again, I’m a planner.”

The self-accountability trap

“Judging performance is always going to be personal to the person receiving it and to you giving it. And I think trying to take yourself out of the loop in this clinical fashion — I’m sorry, but it’s sad.”

So how do successful engineering and product managers navigate the cold, harsh realities of the business while also letting developers be developers? The magic bullet is simple to understand but easier said than done: have empathy.

In Jordan’s experience, scoping efforts are least effective when leaders lack empathy. Pressure from executives leads engineering managers to evade responsibility for accurately predicting timelines and outcomes. In environments plagued by poor communication and lack of trust, the buck is often passed from business stakeholders to engineering managers before trickling all the way down to rank-and-file developers.

“I think a lot of times there’s this push for managers or directors who are nervous about their team’s ability to execute to let them do their own estimates. And then if they don’t hit their own estimates, you can get them in trouble for that,” he said. “If your primary accountability mechanism for your team is their own estimates, it’s inevitably going to poison the well.”

What he means is that, when subject to a culture of self-accountability, it’s natural for engineers to inflate their estimates so they can work at their own pace. Programmers are born problem solvers and, by and large, intrinsically motivated. For many developers, completing a task in the most efficient and effective manner is a reward unto itself.

Part of the engineering manager’s job, Jordan argues, is to treat engineers like human beings. To keep them engaged and motivated, it’s critical to avoid tying their job performance to the type of goal- and deadline-oriented metrics that business stakeholders are used to.

“If you, as a manager, are using velocity and estimates and sprint accountability to help you manage the performance of the engineers on your team, I think you’re dodging the hard part of your job,” said Jordan. “Judging performance is always going to be personal to the person receiving it and to you giving it. And I think trying to take yourself out of the loop in this clinical fashion — I’m sorry, but it’s sad.”

Tips to help take the edge off

“If you can understand their needs, maybe there’s a way to help them solve their problem without creating a problem for engineering.”

How do you manage business leaders’ expectations in a world where budgets, launch dates, and project-based estimation are inescapable realities? Is there a happy medium between flying blind and planning yourself into oblivion?

We asked our interview subjects to share some of their go-to strategies for fostering alignment, trust, and open communication across the organization throughout the development lifecycle.

Document everything

For Krishtha, the best way to align engineering, product, and business priorities is to put everything in writing.

She recalls spending a lot of time early in her career repeating the same information to various stakeholders involved in the scoping process. It wasn’t long before she realized that attending wall-to-wall meetings wasn’t the best use of her time — or anybody else’s, for that matter. Very often, detailed documentation will suffice in place of synchronous updating.

“To me, effective communication is the art of writing effective documentation. That includes making sure that projects have directly responsible individuals — DRIs, which is a term we use — that the stakeholders are acknowledged, and that you have timelines and you’re able to effectively explain the relative priority of a single piece.”

The practice of breaking projects into smaller pieces is what Krishtha says helps her team foster a culture of transparency and interdepartmental collaboration. Dividing the development cycle into steps and diligently documenting the progress of each initiative helps her demonstrate to nontechnical colleagues how the sausage is made. Best of all, she doesn’t need to repeat herself endlessly whenever there’s a production setback or opportunity to pivot.

Krishtha recalls one project in which documentation proved particularly crucial in aligning the goals and responsibilities of stakeholders from disparate corners of the organization. The project affected HubSpot’s email integration and involved roughly a quarter of the teams at the company.

“The thing that ended up making that project successful was being able to distill out what those individual projects were and who was responsible for those projects and then actually show the stuff we’re doing, what success looks like in these situations, and the date we’re trying to accomplish this by,” she explains. “Being able to have that in a single doc and shoot it off to anybody who is interested in understanding the progress, it ended up being really helpful.”

Stand your ground

Mari-Megan says her team doesn’t have a typical product roadmap when working on a project. Instead, it builds customer journeys that are high-level, user-focused, and business- and goal-oriented. This type of documentation enables her team to account for pivots and production delays, and business stakeholders can more easily digest these nontechnical details in the context of the desired customer outcome.

“We work from customer journeys when it comes to scheduling and priority, and we surface that to our external stakeholders — all of our customers, all of our higher-ups at SAP — they all have access to these things,” Mari-Megan explains. “So they can see our journeys and from that get a clear, clear idea of the value they’re going to get by the end of the year.”

This strategy only works if there’s mutual respect and transparency across teams. Scoping can become confusing and messy without clear delineations of who owns what. And no matter how thorough the documentation, there are bound to be confrontations and conflicts of interest.

That’s where soft skills enter the picture: product designers and engineers should know how to stand up for their work when another function creeps into their domain.

“If you’re a designer, the product owner shouldn’t be owning your design. That needs to be a collaboration, and same for development,” Mari-Megan says. “If they think you’re designing something nuts, they should be able to tell you why it is they haven’t adopted your methodology or they’re just stuck in the traditional way of doing things.”

Overthrow the tyranny of timelines

As averse as he is to “the traditional way of doing things,” Jordan says he understands why company leadership needs concrete benchmarks and timelines to run the business effectively. That’s fine, he says, as long as budget planning doesn’t bleed over into accountability.

“The thing with quarterly planning or big project planning is that you’re trying to stack rank bang for buck. You’re going to have to do this, but you want to keep it super vague and you want to avoid it turning into calendar delivery times. You also want to avoid accountability to those timelines.”

When scoping projects at his current company, Jordan’s team employs a version of the objective and key results (OKR) methodology, a goal-setting framework that aligns stakeholders around measurable objectives. This gives engineers some autonomy to change course and respond to discoveries during development because they aren’t held accountable for completing every single goal outlined in scoping.

“The nice thing about OKRs is that you’re only supposed to hit like 70 percent of them,” says Jordan. “I use that wiggle factor as an excuse, basically, to not estimate super precisely. The fact that I don’t have to hit all of them means that if some of them turn out to be super, super hard, I can scrap them. If some of them are surprisingly easy, we check those off.

“I actually think estimating super precisely for OKRs is a waste of your time. The process works as long as your org has the discipline to understand that you’re not actually going to hit every OKR, you’re going to hit, say, two-thirds.”

Estimates are only estimates

Whatever approach to goal-setting and estimation your team employs, it’s the engineering manager’s job to contextualize these benchmarks in terms the rest of the organization can understand. You want to give company leadership the reassurance it seeks in hard deadlines and strict roadmaps without committing your engineers to anything they can’t deliver — or anything they might find during the development process isn’t worth pursuing.

“I think the relationship is more about understanding what they want out of the estimate and then trying to get them that thing without tethering yourself to something that you don’t want, which is generally predictable accountability that’s far away,” Jordan explains. “And that’s where you want the relationship to be strong so that they can compromise. I’m going to try to give you estimates, but understand that they’re vague and please don’t hold me to them in a year. If you want me to do that, I can, but you’re not going to like the result.”

For Jordan, it’s all about keeping estimates in the appropriate realm. One benchmark may apply to head count planning, for example, while another metric applies to bang-for-buck estimation, and yet another to a big marketing launch.

Separating goals from developer accountability requires an unwavering commitment to transparency. Jordan says it’s best not to spend too much time on scoping upfront and to be honest with leadership about the limitations of a given estimate.

As for specific communication strategies, he recommends using “weird metaphors” that strike closer to business leaders’ wheelhouse.

“Sometimes someone will ask, ‘How long will it take to fix these three bugs that are blocking our launch?’ Well, how long does it take the police to solve three murders? They know generally how long it takes to solve a domestic-related thing, but sometimes it looks domestic-related and then you find out like, oh, it was a home invasion robbery, or it involves bitcoin and people from Eastern Europe — what the hell?”

Jordan is partial to the crime-fighting metaphor, but sometimes it’s even better to appeal to personal experience. If you’re talking to an executive, for example, you might liken software creation to building a house or moving the company into a new office space. If they’ve overseen an ERP rollout at some point in their career, this can be a good lens through which to contextualize the development process.

But what happens if all your metaphors fall on deaf ears and executives push you for hard estimates despite your best efforts to be transparent about the process?

“Someone once told me a great phrase: a date for a date. If someone tries to corner me — ‘How long is it going to take to move the service from app store X to another store?’ — I’ll say, ‘I don’t know, but I can find out in two weeks,’” Jordan explains. “Saying ‘I don’t know, but I will know by X date’ is a great strategy to buy yourself some time.”

Have empathy and ditch your ego

At the end of the day, none of the communication strategies our experts discussed are possible without empathy from the top down.

According to Jordan, it starts with engineering managers.

“As a manager, you have to have empathy for everyone in the org, not just the people you manage,” he says. “You have to have empathy for your boss and your boss’ boss. Why are they telling you to do this thing that is frustrating to you? If you can understand their needs, maybe there’s a way to help them solve their problem without creating a problem for engineering.”

That same spirit of goodwill should extend to the developers on your team. As mentioned earlier, engineers tend to be intrinsically motivated, but there’s a ton of diversity within that category as it relates to how they approach a given task.

“Some people just want to get in and solve and look at a discrete problem, and some people really do think about the big picture,” says Kate.

The engineering manager must consider how their own skills and those of their staff level up to the agreed-upon goal. When leading a team of developers, especially on a project that requires broad buy-in throughout the organization, there’s no room for egos.

“Each team has its way of doing things, and you evolve over time,” Kate explains. “Whenever I settle into a new team, I’m going to bring some of the stuff I know, but I’m also going to be ready to shift on a moment’s notice if they work a better way.”

What alignment looks like

“The more eyes on a problem, usually, the better the solution.”

So what does scoping look like when healthy communication is firing on all cylinders, stakeholders across the organization understand the scoping process, and engineering and product leaders are sympathetic to business goals and priorities?

Kate describes a system in which big-picture scoping dovetails effortlessly into more focused initiatives that add up to a streamlined, transparent development cycle. Engineering and product leaders start on the same page and execute a unified strategy to align business stakeholders. With all hands on deck and all minds open to changing course for the good of the project, the best solution wins out.

The finished product is fully optimized software that drives positive outcomes for the business and satisfies engineers’ insatiable desire to solve problems as efficiently as possible.

“When I’m meeting with stakeholders, I’m thinking more big picture — I’m thinking of the whole scope. And then when I get back to my team, I’m going to bring a product representative with me to answer the more people-sided questions. From there, we start thinking about the actual technical solution.”

Whether the roadmap to that solution is rigorously scoped, charted along the way, or the product of a hybrid approach, the best direction will depend on a multitude of factors that vary widely from project to project. The surest way to minimize bumps along the way is to respect the needs and goals of everyone involved in the project.

Engineering and product managers are uniquely positioned to champion cooperation and free exchange of ideas across the organization. When stakeholders bring good intentions to the table, they create the ideal conditions for the best ideas to coalesce into first-rate software. Open dialogue engenders mutual trust, which leads to happy business leaders and motivated engineers.

“I’ll have some ideas of what I think is the right answer, but then what I really want is my team to slow down my solution and tell me what they have,” Kate says. “It’s better because the more eyes on a problem, usually, the better the solution.”