By Bill Lamie
President/CEO, PX5
It’s midnight the day before the product release and your system just crashed. As you follow the execution back from the point of the fault, you find yourself deep inside the real-time operating system (RTOS) code. If you don’t have professional RTOS support, your only option is to try and figure out why the RTOS is failing yourself. Of course, you have to first figure out how the RTOS works to even have a chance at understanding what is going wrong. If you are fortunate enough to figure this all out, you are then tasked with trying to fix the issue – and then ensure the fix is propagated back into the next release of the RTOS.
Fixing an issue in an RTOS takes considerable time, and embedded software developer time is quite expensive. The median, fully-loaded embedded software engineer salary last year was roughly $181,269 (based on Glassdoor). If an RTOS bug takes you two days to fix, your direct cost is roughly $1.5K. Of course, things get much worse if the whole project is blocked.
For a typical development team of eight software engineers, each day the project is blocked costs on the order of $6K (not including indirect costs associated with schedule delays). Also remember that these costs are just for one RTOS bug with a fairly rapid two-day resolution. Things don’t always go so smoothly in the real world!
We embedded developers know there are bugs lurking in every piece of software. No exceptions. That said, the exact number of defects varies greatly on the quality and maturity of a specific RTOS. According to the 2014 Coverity Scan Open-Source Report, the defect density of open-source software was roughly 0.6 defects per 1,000 lines of code (LOC), which means there is more than 1 bug for every 2,000 LOC.
Most RTOS distributions contain many thousands of lines of code. Given that, it is likely there are bugs in the RTOS you are using. It’s also worth mentioning that this defect density number is in the context of static analysis and therefore doesn’t account for timing and other complex run-time interaction classes of bugs, which are most prevalent in RTOS software.
We know that there are bugs in all software and fixing them is expensive. What can we do? Investing in professional RTOS support (for both commercial and open-source software), offers many valuable benefits:
Professional RTOS support provides access to experienced professionals who specialize in RTOS software. Their expertise can be crucial in resolving complex issues, optimizing configurations, and providing guidance on best practices. In some cases, the guidance alone prevents costly problems down the road.
Most professional support packages come with a service level agreement (SLA) that guarantees response times for issue resolution. This ensures that if you encounter problems or have queries, you'll receive prompt assistance, minimizing downtime and potential disruptions. Remember the average $6K per-day cost if the project is stalled on an RTOS issue – so timely response is all important.
RTOS support services can provide customized solutions and recommendations tailored to your specific needs and use cases. They can assist in optimizing configurations, implementing features, or resolving issues specific to your environment.
Many professional RTOS support services offer timely updates, patches, CVE notifications, and security fixes – even for open-source RTOS software. This ensures that you're using the latest stable version of the RTOS and are protected against security vulnerabilities.
Some open-source licenses come with specific obligations or restrictions. Professional RTOS support can help ensure that your use of the software complies with licensing terms, minimizing legal risks.
RTOS support services often have options for training programs, supplemental documentation, and resources to help your team understand and utilize the software more effectively. This can improve your team's efficiency and reduce the learning curve and help improve your product’s quality as well as your time-to-market.
By having a dedicated RTOS support team behind the software, you gain confidence in its stability and reliability. Knowing that you have experts to rely on can provide peace of mind when using critical open-source tools. Professional RTOS support serves as a development insurance policy.
The cost of most professional RTOS support packages is less than the cost of having your development team stalled for just one day. Professional RTOS support also helps your team manage the RTOS source code integration, including notification of new updates, patches and CVEs. There are also indirect financial benefits in helping your team create a better-quality product with reduced time-to-market. In most situations, purchasing professional RTOS support saves money!