Debugging Large Embedded Memories in Embedded Programming
It is common in the embedded space to have small, simple storage devices accessed address-wise.
Chips can be as small as a few kB, leaving insufficient space for a file system. On the other hand, we may encounter situations where large storage devices are used in the design, but no file system is included because it was not necessary for the intended purpose.
Software testing validation and debugging in embedded systems can be long and complicated in such cases. For instance, we worked on a product which would generate a few hundred bytes of storage a few times a week. The design would have to be tested when the chip is full, which by design should take some weeks
Challenges in Software Testing Validation and Debugging in Embedded Systems That We Overcame
The first solution is to abstract away data accesses through a dummy file. It , which doesn’t do much. Its true purpose is to act as a barrier between the rest of the code and the memory accesses.
When running code for the MCU target, we compile the code which uses its access methods. However, when running on the desktop, the code interacts with file system accesses. For example, in a system that reads a single byte from a specific address, the access method will differ depending on whether the code is executed on the MCU or the desktop environment.
We could directly code the register access sequence into the method, or we could have a
HAL function which passes it through to another. Both signatures look identical:
int HAL_ReadMemory(uint8_t * buffer, uint32_t address, uint32_t
count)
It doesn’t matter to the function calling it if the underlying memory access is through an MCU or the file system. It wants a memory area to hold the result, a return code, a start address, and the number of bytes to read.
When we need to know on the desktop that we can read it back, we can use f_read
to read from a file. For many systems, it doesn’t even need to be that complicated – we can store the whole thing in a uint8_t array.
And it’s fast. It’s RAM, after all!
We can run pass/fail tests much more quickly than if we need to wait for an embedded
erase cycle, or for the product to generate the required amount of data naturally.
Embedded Programming Key Methods
Think back to our system that generates a few hundred bytes per week. How do we test whether the memory accesses are good?
- First, we simulate the maths on the desktop. We mock out the accesses and substitute our desktop methods.
- Now we can fill the dummy memory chip simulator with anything we want, under our control, as fast as the desktop can do it.
- We can “fill” the chip and check it rolls over correctly. We can test the erase routines get called as expected.
- We can see what happens when the data contents change, if that’s what’s important to us. It gives us significant test capability.
- We can easily generate coverage reports, and ensure we have good feature coverage in a short time
Debugging Outcomes in Small Embedded OS for Large Memory Devices
Here, we have presented a way of speeding debugging large memory devices in small embedded operating systems.
Sometimes it means we expand our test coverage cheaply. Sometimes we use it to expose integration test failures in the final product.
Either way, we learn something new about our product and how it behaves under stressful conditions.
Don’t just take our word for it. Hear from our customers.
An Alternative Solution
The second answer is a little more devious. What if we could do all of that, but on the real target?
After all, the desktop doesn’t have to deal with the same embedded memory limitations. It doesn’t have to deal with the same sector sizes or erase times. It is best to do that on the target. But how?
The answer here lies in integration test commands. We require a back channel onto the target. We send the target a specially crafted message, and it runs its “store some data” method in a loop. We have to deal with things like the power gulp during sector erase.
We can’t hide from erase times on the chip. If there’s a problem in the layout, and it interferes with the address lines, this is one way to find it.
Maybe interrupt priorities matter? Maybe you can expose a race condition?
How to Debug the Target
Here is one way to debug it. If we have a set of integration tests running on the target, we can design tests to expose timing dependencies. We can link as much of the application as we want. With an appropriate bidirectional communication link, we can collect results and treat the runtime like a unit testing environment.
For example, referring back to the original 128MB storage IC, filling this memory “naturally” would have taken ~21 months. Using a desktop-centric unit testing approach, the test runs in around 13 seconds.
As an integration test on the real target, the memory fills in around 4 ¾ minutes. We must wait for the memory to be written to the chip in real-time, with various pages write times. We must wait for the chip to carry out sector erases in real-time. All of this happens in the background while the device executes its normal scheduler, so it is dependent on ISRs and conflicts just like anything else.
Modern Embedded Programming Tools
As an aside, we now find tools are rarely the problem. Yes, historically we used to see toolchain problems – the compiler wouldn’t always generate the same code, and copious compiler-specific directives and quirks of the devices worked around in unclear ways.
Modern tools (particularly Cortex tools) are reliable. What you get on the embedded target is almost always what happens on the desktop. The argument is that there are significant differences between the desktop and embedded runtimes is largely now discredited.
Beware the vendors pointing out all the clever ways their tools are “good with C” – there’s probably a reason the marketing department chooses that wording.
Contact Us
At Bermondsey, we are specialists in a range of areas, including embedded software and firmware, embedded IoT and embedded Linux. Our expertise extends to software testing validation and debugging in embedded systems, ensuring that your products perform optimally under all conditions. If you have questions about how we can help your business, please complete the form below, and we will be in touch shortly.
Alternatively, please call us on +44 (0)208 0650 162
Email : [email protected]
FAQs on embedded programming and debugging
What is debugging in an embedded system?
Debugging is a vital part of embedded systems development. Since these systems handle critical tasks, even a small error or malfunction can lead to serious consequences. Effective debugging allows us to identify and correct issues, and ensures the system operates as it’s intended.
What are some common memory-related issues in embedded operating systems?
Common issues include:
- Memory leak – a program that allocates memory but does not release it when it’s no longer needed
- Fragmentation – when memory is allocated and freed in a way that creates unusable gaps, making it harder to use the available memory efficiently
- Crosstalk – crosstalk is interference caused when the electric or magnetic fields of one telecommunication signal disrupt a signal in a nearby circuit
