By Bill Lamie
President/CEO, PX5
The origin of the phrase “penny wise and pound foolish” comes from the 1621 work The Anatomy of Melancholy by Robert Burton. The idea behind this phrase is that we sometimes concentrate so hard on small costs that we can overlook larger opportunities to reduce costs or increase gains. This mindset has beset embedded development for quite some time and continues to do so.
According to Glassdoor (as of last September) the median annual salary for an embedded software engineer in North America is $129,478. With a typical overhead rate of 40%, the total company cost of an embedded software engineer is $181,269 per year. According to a recent 2023 survey, the average development team consists of nearly eight software engineers, so the total labor cost associated with embedded software development team is a whopping $1,450,152 per year. Of course, this is just an average amount, larger and perhaps more experienced teams will easily surpass this.
Given this nearly $1.5M annual cost for an embedded software engineering team, a company might choose to reduce budget by minimizing the infrastructure costs associated with the team. For example, using “free” embedded software tools (free compiler, free RTOS, and free middleware components) for a team of eight software engineers might initially save the company $60K. However, relative to the $1.5M development cost, this savings isn’t significant, which begs the question: is forcing your very expensive embedded software development team to use free RTOS tools “penny wise and pound foolish?” I think so.
Anecdotally, I recently had a development tool experience myself that helps make this point. I was porting my new RTOS to a semiconductor company’s “free” tool chain. The first problem I ran into was trying to figure out what version of the tool suite to use (there were many different downloads for the many different processor families offered by this particular semiconductor company). Without support, it took me some time to just figure out what to download. Once I finally found and installed the correct tools, things surprisingly got worse. In order to port my new RTOS to this new tool chain, I needed to update one assembly language file and one C include file. Sounds pretty simple, right? Well, in this case nothing could be further from the truth. The compiler and assembler were very primitive – seems like they hadn’t been updated in many years. Even more problematic, the tool’s documentation didn’t match the actual behavior of the tools. What’s worse than no documentation – wrong documentation! I eventually had to use trial-and-error to figure out how to use some of the assembly directives and instruction syntax. The compiler wasn’t any better – limited features and few useful intrinsics. All in all, the quality of these tools and erroneous documentation resulted in multiple lost days. What should have taken one day turned into four. I couldn’t help but feel sorry for the professional developers having to use these embedded software tools on a daily basis.
As for time-to-market, nearly every device manufacturer is under constant pressure to improve quality, add features, and beat their competition to market. A failure in any area can result in serious financial loss. Conversely, a success in these areas can result in great financial gain. I remember a study some time ago that showed substantial financial loss for every month of schedule delay. This same study showed that companies using a particular commercial RTOS enjoyed a significantly better chance of meeting their target release date. It was clear that the commercial RTOS more than paid for itself with improved time-to-market.
So, is it “penny wise” to use free RTOS tools? At the highest level, it seems quite obvious. Why would a company invest $1.5M per year on an embedded software development team and not equip them with the proper software tools? To illustrate this point in a more granular way, let’s look at the per day cost of the software engineering team (assuming 250 working days per year). The $1.5M annual cost translates to $6K per day. At this daily rate, the commercial RTOS tools option (compiler plus RTOS) only has to save 10 days of development effort to economically outperform the “free RTOS” alternative. I think this is a very low bar.
My direct experience leads me to believe the number of development days saved by using commercial RTOS tools is on the order of 30 days per year – easily making up for their initial cost. This doesn’t even account for savings/profit associated with greater schedule certainty, improved product quality, and the potential of lower bill of materials (BOM) costs associated with using purpose-built commercial RTOS tools. In my opinion, not giving your expensive embedded software team proper resources is a great example of being “penny wise and pound foolish.”