Structured Skepticism Builds Stronger Engineering Teams | Eric Lubow, CTO, Mapp
This week’s Dialog guest is Eric Lubow, Chief Technology Officer at Mapp. He leads the company’s engineering and product org, with a focus on driving faster innovation and business impact for clients.
Before Mapp, he spent three years at Thrasio as VP of Engineering, helping the company absorb over 75 acquisitions. He co-founded SimpleReach, scaled it to process billions of social media events, before exiting in 2019.
He’s also the co-author of Practical Cassandra, a hands-on guide to building production-grade distributed data systems.
Takeaways
Giving developers visibility into how their changes affect production, and having ops walk them through it, closes the knowledge gap faster than any monitoring tool. The real barrier was never data, it was context.
When leading a reorg, the hardest part is convincing people at every level that you actually understand the problems from their vantage point, not just yours.
In distributed teams, the managers who build the deepest trust are the ones who learn how each person communicates and adapt to it, rather than expecting the team to adapt to them.
Yassine: At SimpleReach, you worked on improving visibility between what developers built and how it performed in production. What helped bridge that gap beyond just adding monitoring tools?
Eric: For context, I started my professional career as a SysOp, what they now call devOps. But I did that after many years of being a mediocre dev. I got to Sysops because I thought that setting stuff up was more fun than coding it up (though I enjoyed the mix). The closest thing to being responsible for setting up systems at the time was SysOps and I searched for jobs that let me focus on that but gave me the leeway to code when necessary.
Through this time, I found that the tooling for developers to be “close” to the production environment was … lacking. To be honest, it still kinda is and this has made the idea of SysOps/DevOps kind of a dirty word. Everything was about throwing things over the fence. Devs created something and then threw it over the fence to Ops to deploy. If it broke during off-hours, then Ops brought it back up (if possible) and threw it back over the fence to devs to harden/fix or do something more permanently useful.
So absent great tooling, I focused on what I had access to, process and procedure. Basically, we aimed to have devs understand the impact of their changes on production by picking a key metric (or metrics) that they would follow or aim to have impacted by their changes. We basically put the burden of knowledge on the ops folks to help the devs understand what the impact of their changes was. The more information that they (often backend) devs had, the closer they were able to get to the ops work until it eventually blurred the line.
At the same time, the knowledge sharing was happening in the other direction. The devs would talk through their changes with the ops people to help them understand the change, the purpose, and the expected outcome(s). This absolutely took longer and slowed the process down. But it mitigated a lot of the finger pointing and helped bring the team closer together (when it worked).
The last bit of this was the injection of Tech Talks and Lightning Talks. We had a dedicated time every few weeks to let someone from the team, it was a tech talk, deep dive into something about the system or something they learned recently that they were planning on implementing to improve things. If it was a little smaller, they’d do a 5ish-minute lightning talk.
This is the kind of thing that seems obvious or common now, but wasn’t as common in the early 2010s when we kicked this process off. Continuous learning has always been a core value for me. So imparting that onto the team and facilitating the environment for everyone to learn and pass along knowledge, was key to bridging that gap more than just throwing dashboards and monitoring tools at people and expecting them to understand things and magically know what to do with them.
Yassine: Looking back, what would you approach differently with the tools and practices available today?
Eric: Tracing tools have come a VERY long way since then. Vendors like Honeycomb and Datadog have revolutionized system introspection, APMs, and democratizing information about the system making it more accessible for people up and down the stack. The majority of messaging and database systems now also have their own suite of introspection tools that are not only very capable in their own right, they are able to plugin to things like Honeycomb and Datadog and overlay their information onto other useful things like performance monitoring, alerting, code changes, etc.
I would definitely implement more introspection tools and be better about the types of anomaly detection and bounding boxes/thresholds used in those detections. I would also be a bit better about how to facilitate documentation knowing what LLMs are now capable of synthesizing into useful information.
The ability to correlate events is also MUCH better now than it was years ago. You can mark off deploys at timestamps and see how metrics/KPIs changed at deploy times. Ensuring people use these correlation tools would also be a key part of any strategy that I would have given what’s available today. When you ask an LLM to look at the overall system, it can just take in more context than a human can and will often find anomalies in ways that humans struggle to do.
Yassine: At Thrasio, you managed integrations of over 75 acquisitions into the technology architecture. What’s one unexpected lesson from integrating that many businesses that fast?
Eric: The biggest thing is to look for patterns in order to minimize the overall footprint. Basically create a focused list of vendors to minimize tool sprawl. Vendor consolidation is incredibly important. Not just to minimize the workload on the IT team, but also, fewer tools to manage and log in to means that the teams using those tools can be power users. You also get the ability to negotiate down costs the more “accounts” you have with a vendor as well as potentially influencing their Roadmap if your use-cases are sufficiently portable.
Most of the time at Thrasio, it wasn’t integrating teams. We acquired “mom and pop” shops mostly for their product(s), notoriety, inventory, and ideas. They typically had limited “tech” let alone a tech team. What they always had were “tricks” to make their setup or minimal resources outperform their available capacity. What was important to the transition team(s) weren’t just getting the access and making it shareable/access through our internal SSO, but documenting the things that made the stack and workflows successful. For example:
1. How did they tie their inventory tool to their ads tool in a unique way?
2. Can we/Should we be doing something similar?
3. Did they learn anything here about the operational pipelines that we didn’t know?
4. What questions/problems were they solving for that were obvious to them but beyond the standard problem set?
In order to get themselves to the point of being acquirable, the people and companies had already differentiated themselves. My teams were looking for the in-between the lines stuff beyond just standard sales/marketing/inventory/social that they did to set themselves apart to add it to our overall toolkit.
Yassine: Mapp CEO, Steve Warren, said you “played a crucial role in improving solution delivery to customers and unifying teams under a common mission.” How would you describe your way of bringing teams together around a shared direction? And what’s a challenge that’s less obvious until you’re in the middle of it?
Eric: A little context is useful. I was brought in to clean up years of leadership challenges at Mapp – ranging from bad strategy and senior leadership to technical and organizational debt built up from poorly integrated acquisitions. There was a LOT of organizational trauma built up around all of these things, and they take time and trust building to unpack.
Every organization has their own flavor of dysfunction. It’s just a matter of figuring out what it is and what caused it in order to build confidence in everyone that you understand the problems. And one of the most important things to me when bringing teams together is getting everyone on the same page. Everyone has seen different levels of dysfunction and comes with different baggage depending on their backgrounds, their place in the organization, their time in service, and many other factors. So the problems and solutions typically need to be described in multiple ways, multiple times, and at multiple granularity levels.
I typically do this with a bit of over communication. If I’m doing a re-org, then I tell the org that we are going to make a change. I explain the issues and how it manifests at the various levels. Then I explain the change and how it will impact the organization itself. This usually also covers why something as drastic as a re-org is necessary to remedy these issues. Then I explain what success will hopefully look like. I remind everyone that we won’t get everything right on the first try even though we’re doing our best and tell them that they should speak up if something can be improved. Then I show them the schedule of changes including when they’ll receive progress updates. I will have already gone through this with my leadership team and discussed all the issues that we are aware of.
This is to ensure that there is support for the changes or that the people against it are already at the disagree and commit stage. I will announce all this at a Town Hall and then begin the trainings for leaders at people manager trainings. I will reinforce the success metrics using OKRs where relevant and do regular checkins with leaders. At the next Town Hall, I’ll show progress and reinforce the goal outcomes. If something significant was missed, I’ll also let everyone know that so people are reminded of the fact that they have psychological safety to speak up.
The challenge that’s less obvious is getting everyone to know that you understand the problems from THEIR perspective. Just because I see the problems clearly and I see a (mostly) clear path to solving them or at least improving the situation, doesn’t mean that everyone is on board. People will mostly do what they are told because that’s how the professional “game” is played. But to really get people on board, they need to think that you understand what they think are the problems.
Most of the time, this is restating the problem with a different perspective. Sometimes it’s a problem that is getting solved unintentionally simply because there are better processes being instituted. But either way, getting buy in at all levels is often harder than we, as leaders, want it to be. That doesn’t make it any less important though. While it isn’t always necessary to do this step, it does help you to get more people on your side. This is especially important in times of change.
Another challenge that’s not as obvious until you are in the middle of it, is how many times you need to repeat at varying levels of granularity, the strategy and how teams fit into it, contribute to it. As a leader, you are thinking about the strategy and organizational stuff all the time. You are spending dozens, if not hundreds of hours thinking about the strategy, communication, execution, feedback, etc. Then you spend only a few hours communicating this to people.
There is no way they can absorb and understand all this context (nor should they) in addition to their day-to-day workload. Strategy isn’t strategy unless repeated, and repeated a lot. This is another way you get people on board. You need to tell them what’s going on often enough for them to connect the dots between their day-to-day and the change in strategy otherwise the buy-in doesn’t land the way you need it to land.
Yassine: You actively push your teams to focus on “what is important for the business, not only what sounds cool.” When an engineer proposes a solution you’re skeptical of, what’s your process for evaluating it fairly while keeping the team’s focus on business impact?
Eric: I prioritize keeping team motivation and morale high. My go-to tactic is typically a compromise, facilitated by a structured process that ensures the engineer sees other points of view while giving their idea a fair chance to succeed. For starters, RFCs are typically where these things start. An engineer wants to solve a problem creatively and the first thing they need to do in my org, assuming the project is large enough, is to write an RFC. Now their solution is going to be reviewed by their lead, VP and the other engineering leads. When their solution is presented at an architecture meeting, is typically when I and everyone else gets to hear and comment on it.
First thing I’m going to see how everyone else reacts to the recommendation. Usually people will speak up and ask the hard questions. Who is going to maintain this if no one else in the org knows language X? What kind of organizational support are we bringing in to the org to ensure that this system is properly maintained/upgraded? Does this new system make sense for a single-purpose solution? etc. If there are all good answers and it makes sense, then we might go ahead with it. Usually the question of, who will maintain this if you leave the org, gets a healthy conversation about this going.
If we’ve gotten positive reactions to this point and I’m still skeptical about this, I often try to figure out how a system can “upgrade/improve” other parts of our infrastructure. For example, if someone wants to bring Apache Airflow to solve a single problem and I’m unsure of this, they can show me how other parts of our async job infrastructure can be redeployed more effectively as DAGs. Then it makes more sense to move ahead with an (initially) single-purpose solution. There are often ways to prove that an idea makes sense for the business in terms of KPI improvements (usually KPIs in the OKRs). Engineers should know what these important KPIs are as part of the regular strategic context communications (quarterly/trimester OKRs, strategy/KPI updates in Town Halls, etc).
And if all this work has been done, RFC, architecture meeting presentation, etc, and I’m still not sure, I try to have the engineer timebox the solution. I ask them how quickly a PoC (Proof of Concept) or MVP (Minimum Viable Product) can be implemented for everyone, including the engineer to see if it makes sense to push ahead.
Sometimes things are just more complex than everyone thinks they’ll be. Letting people try and see how far they get shows that you trust them and willing to take risks and even be wrong. That kind of trusting environment really helps foster the culture of innovation even when sometimes the answer to a request has to be “no”.
Yassine: You co-authored “Practical Cassandra: A Developer’s Approach,” focused on building scalable distributed systems. What’s one lesson from that work that you still apply to how you structure engineering teams today?
Eric: The most enduring lesson I still apply is the need to treat the organization itself like a distributed system. The principles of designing scalable software apply directly to structuring engineering teams and their interactions.
Every system should be designed with being able to answer the following questions:
What are the roles and responsibilities for each part of the system? Who is the responsible party for that part of the system?
What does (barely) functioning look like? What does normal actually look like? What does optimized look like? How do we evaluate or measure these states or state transitions?
What do the common failure cases look like and how are they being addressed (either gracefully or otherwise)? Be intentional about failure and its handling.
What are the pressure release valves on the system? What causes them to activate? What causes them to turn off? What pressure are each of them supposed to relieve?
What does the safety net look like? What causes it to activate? When (and how) does it get shut off in order to return to “normal” operations?
How do you handle when things get out of sync? What type of resynchronization options exist in the system? How do they get applied? And how do you know/what do you do when they don’t work as expected?
What happens when things take too long? What does the backpressure look like? How do notifications traverse the system?
What does monitoring look like? How do you trace something through the system? How can you introspect or observe operations without disturbing operations?
How do the right people get alerted in the right order and with the right information when something isn’t performing as it should be?
Obviously there are a lot more things to consider than this. But I still apply these kinds of ideas to pretty much every system I work with. This applies to distributed software systems just as much as human systems/organizations. I often look at the organization just like a distributed system to ensure that it “should” work in a good state before applying the “people” factor. How does the data flow from component to component? Can the teams operate relatively independently or are they extremely dependent on other teams to be able to move fast? Lastly, but not less importantly, the people factor is where you figure out how well the humans in those teams work together and work with the adjacent and complementary teams.
This might seem like more than a single lesson. But it amounts to being as prepared as you can be, at least in the planning stages, to how you expect your system to function or not function. It’s basically ensuring that you always attempt to have a mental model for what degradation, failure, ownership and recovery looks like.
Yassine: Looking back at your time at Thrasio, where you held roles including CPO, Interim CTO, and VP Engineering as the company scaled, how do you provide stability for your teams when the org structure is evolving?
Eric: Communication is the most important thing. You have to keep your leaders informed and up to date. The goal is to keep everyone operating in as much BAU (business as usual) mode as possible. In this case, BAU means that everyone knows their short and medium term goals, roles and responsibilities, OKRs/KPIs, business strategy/strategies, priorities, and how their roles and responsibilities impact all of that.
I go about ensuring this massive effort in a few ways. Having very clear roles and responsibilities definitions is a good starting point. This way, if the org grows/changes outside of Tech and there isn’t a clear internal mapping/representation of that support, the leaders inside of Tech can look at their roles/responsibilities and identify the gap and propose how it should be filled (assuming there is capacity for that support). If there is no capacity, just identifying the gap is a good starting point in case requests come in.
Having as clear as possible definitions of roles and responsibilities allows for clear prioritization. When everything is a priority, then nothing is a priority. And when nothing is a priority, then people can’t focus. If they can’t focus, then they can’t get anything done, let alone the right business impacting work. In order to keep people focused and avoid being interrupt driven, I want to ensure that everyone knows the priorities and can make as many decisions as possible without me in the loop (although I’m always around to support as needed). I do a lot of this prioritization with OKRs.
OKRs are one of the main facilities I use to push decisions down to the lowest level in the organization and remove decision paralysis as a bottleneck for getting work done. Well-written and agreed-upon OKRs facilitate prioritization decisions in a way that the people at all levels of the organization can justify their prioritization and decision making.
Keeping people updated with regular communication is also important. Every week, I ensure that my leadership team gets the latest updates from the SLT (Senior Leadership Team) and strategy so they can filter those things down. I repeat important macro changes to everyone in the org at the monthly Town Halls that I do. I ensure understanding of these things on a more individual level during 1:1s. And I do regularly async updates to the various levels of the org when something important happens out of band.
One other important tactic for keeping stability is to ensure that people know what’s occurred and its impact. At least once/month, there is company-wide communication from Tech. This could be from me, from the head of product or someone else senior in the org. The goal is to ensure that everyone knows there is valuable work coming out of Tech even if it doesn’t necessarily impact their team directly.
This kind of constant and regular communication keeps people feeling as though business as usual is occurring even when an org is growing and changing and allows them to stay as focused on their day to day as possible. I also do weekly updates to the SLT about what’s been accomplished and status updates in Tech to ensure that there is, at the very least, awareness, at the SLT level of what’s going on inside Tech.
Yassine: You work with distributed teams from Berlin, and you’ve once led an org spread across 20 countries. When you’re creating manager training content, what do you emphasize for managers of remote and distributed teams?
Eric: Communication is even more important in distributed teams than it is in in-person teams. It’s nearly impossible to create the “water-cooler” moments that exist in an office. You don’t want to burn people out with meeting fatigue either trying to recreate those situations. So my focus is giving people more tools for the important things like relationship building, asking good questions, being a good listener, and other important and often overlooked or undertaught “soft” skills.
I do this with a lot of examples and self-check questions. Here is an example: In one of the recent trainings, I walked through the value of micromanagement when applied correctly and the damage that’s done when applied incorrectly. I gave a few examples of how to look over someone’s shoulder in a healthy way vs an unhealthy way. When you are in a hybrid or fully remote culture, you probably can’t physically look over someone’s shoulder. So I provide strategies that function well for both in-person and remote.
I also talk a lot about authenticity. Being authentic helps to build rapport whether you are remote, hybrid, or in-person. I help people with strategies to show their authenticity and be more human. We’ve all had robotic managers before. No one wants to work for someone like that or be someone like that. In striving to help people be more authentically themselves, I try to give them really specific ways in which they can bring out their authenticity and not suppress it in other people. I’m not someone who thinks people can (or should) always bring their “whole selves” to work. But there is a HUGE gap between showing up and being authentically who you are and being a corporate robot. Helping managers and leaders connect with their people through more authenticity lies in that in-between.
Another relevant point here is to be aware of culture. While it’s not great to think about people or offices in their country’s stereotype, knowing cultural behaviors and their drivers is useful. For example, Germans often find small talk less of a thing than Americans. When you ask a German, “How are you,” you can expect to find out how they are as opposed to most Americans who treat “How are you” as a greeting. Giving a nod to culture, especially when working with teams of people from many countries, helps people interact in a more aware way. Using the occasional example like this helps remind people that there are different ways to communicate.
One of the other big ticket items is around mastering the async communication tools. Everyone operates differently and I expect managers to learn their people well enough to figure out how to best work with them and integrate that into how they, themselves work as well. Some people like chat, some email, some Confluence or Jira. Knowing how to best engage with your people in a way that works for you will pay dividends as a manager in a remote team. This also helps to ensure that people in different time-zones don’t get penalized for not being around during the working hours of the manager.
Yassine: As you continue to build a solid content library around engineering leadership and management, what do you hope other engineering leaders take away from your work?
Eric: Generally, I just want people to not make so many of the same mistakes that I did. I’m painfully aware that many learn by doing and simply need to make those mistakes. I think I am more often one of those people than I want to be. But I at least want to put some of the stuff out there that’s worked for me over the years. I’ve built up a rather large library of things that have worked for me.
As I continue to do more executive coaching, I keep being told how often I’ve got a unique or interesting approach to a common problem or situation. So I wanted to ensure that I take the time to document the things I do. If everything I write, record, or create only ends up being read by the people I work directly with, that’s fine with me. But if the stuff I put together can at least help one other person, then I think it will be worth the effort.
Specifically, as a neurodivergent person, or someone whose brain works a bit differently than most people’s, I want everyone to know that this is ok. Not only is it ok, but as someone for whom so much of the “advice” that I would read felt like it was missing specific details, transitions, or explanations that my brain couldn’t just “fill in,” I want to write the pieces for the in-between brain. It’s not that I think I am different. I’ve just been told enough times over my professional career that I have a different way of handling situations. So I want to take some time to document how I function.
I also want people to know that it’s ok if their style is different. At the beginning of every people manager training that I do, I remind everyone that all of these things that I pass on are just information and tools in a toolkit. They actually need to make these things their own if they are going to be successful as leaders and managers.
I’ve been managing people for over 20 years. This has been in the military, in business, and coaching people in business and martial arts. Inside of Tech, I’ve been managing engineers, Product Managers, DevOps, Designers, Project/Program Managers, IT/Security/Compliance, and a handful of other focuses.
So many of the principles are translatable between management focuses and departments. It’s also very helpful to learn what adjacencies do in order to help people see other ways of approaching problems. There are so many great resources out there now in people, articles, videos, etc. Having directly managed so many types of technical teams and leaders, I want to pass on to people as many of the transferable lessons as I can.
Above all, I hope (engineering) leaders focus on systems, not heroes. I want them to realize that their highest-leverage work is not in solving the hardest technical problem (though sometimes that’s the case), but in architecting human systems– communication, process, and culture, to allow their people to solve problems sustainably. The core takeaway should be to treat the organization as a complex, scalable system where empathy, safety, authenticity, and clarity are the primary protocols for mitigating failure and achieving scale.
📚 Eric Lubow’s Go-To Resources:
2 people you follow and recommend?
My kinda famous long-time friend, Charity Majors
Non-Tech person (also long-time skydiving friend), Melanie Curtis
Podcasts you make time for?
I don’t have a single podcast I make time for regularly. I’m usually working on my own, Beyond the Belt, where I interview other BJJ black belts on their life and passions outside of Jiujitsu.
I do make time for the occasional deep dive episode on stuff like Startalk, First Principles Podcast, or anything else very space sciency or deeply technically nerdy.
A book that shaped your thinking?
Thank you Eric for your time and insights!
This interview is part of the “Exec Engineering Dialog” series where I interview seasoned tech leaders on the topics of talent, product, management and culture.
If you liked the insights shared in this interview, consider giving feedback and/or sharing it with your network, it’s the best way to help this segment improve and grow.
Yassine.