I dread writing embedded GUIs(blog.benjamin-cabe.com)
blog.benjamin-cabe.com
I dread writing embedded GUIs
https://blog.benjamin-cabe.com/2021/10/21/5-reasons-why-i-dread-writing-embedded-guis
20 comments
For small systems, LVGL address all of these pain points. It works well with small amounts of memory, maintains a responsive, animatable UI that works over slower display interfaces, and is easy to port with only a small amount of driver code needed to get up and running.
I second that. After decades of rolling stuff by hand I recently did a project based on LVGL. It's highly configurable, easy on the resources, properly documented and all. I had some troubles with getting the layout to do what I want, but with the latest release that has been solved with the new CSS-like layouting mechanism.
Recommend, would use again.
https://lvgl.io/
Recommend, would use again.
https://lvgl.io/
That looks amazing. Thanks for the resource.
OP mentioned that along with Azure RTOS GUIX deeper in the article.
On an instrustial machine I worked on we got around this by hosting a node.js website on a Linux touch screen panel. There was a redis cache which was updated by a small python script that communicated with the embedded stuff via a serial interface.
If you have the budget for it, this is a great architecture since creating a website GUI is much easier than an embedded GUI. Also, you could very easily pull up the machine control panel remotely from any web browser (with proper remote vs local security enabled of course).
If you have the budget for it, this is a great architecture since creating a website GUI is much easier than an embedded GUI. Also, you could very easily pull up the machine control panel remotely from any web browser (with proper remote vs local security enabled of course).
If you can install Linux on your device you won't have any of the problems he is talking about. He's talking about microcontroller GUIs.
Honestly running a node.js website with a Redis cache and a Python script for an embedded GUI sounds like a completely insane and fragile mess!
Qt or Flutter are clearly the best options for embedded touchscreen devices where you can run Linux.
Honestly running a node.js website with a Redis cache and a Python script for an embedded GUI sounds like a completely insane and fragile mess!
Qt or Flutter are clearly the best options for embedded touchscreen devices where you can run Linux.
My point was at a hardware cost it may save you software development cost by adding a Linux machine to an otherwise embedded platform
Similar things happen on every industrial machine. Typically, it's a VxWorks-based PLC (implementation detail, never ever exposed to the end user) and a Windows CE touchscreen HMI. It is remarkable, though, how much this separation makes sense.
You end up with some ancient 68000 or PowerPC processor running all the performance-critical logic with - if you're lucky - a megabyte of SRAM. It's expected to hit one millisecond scan times every single millisecond for ten years straight. And it does so, until you miss your SRAM battery maintenance interval or drive a forklift through the panel; it will probably keep running through any lesser hiccup.
The PLC communicates over that serial interface (or an Ethernet socket which emulates a serial interface) with the HMI, which probably has a multicore x86 CPU and literally a thousand times more memory. It runs a pared-down desktop operating system. It's not made to hit one-millisecond scan times, it probably has visible lag while merely changing the background color of an indicator. Good luck creating a usable GUI with the IDE, which might be decades old.
An Arm Cortex microcontroller for the embedded system, and a Linux touchscreen displaying a NodeJS website would be a convergent way to develop such a machine if you didn't have 50 years of legacy code and trundling along behind you....
You end up with some ancient 68000 or PowerPC processor running all the performance-critical logic with - if you're lucky - a megabyte of SRAM. It's expected to hit one millisecond scan times every single millisecond for ten years straight. And it does so, until you miss your SRAM battery maintenance interval or drive a forklift through the panel; it will probably keep running through any lesser hiccup.
The PLC communicates over that serial interface (or an Ethernet socket which emulates a serial interface) with the HMI, which probably has a multicore x86 CPU and literally a thousand times more memory. It runs a pared-down desktop operating system. It's not made to hit one-millisecond scan times, it probably has visible lag while merely changing the background color of an indicator. Good luck creating a usable GUI with the IDE, which might be decades old.
An Arm Cortex microcontroller for the embedded system, and a Linux touchscreen displaying a NodeJS website would be a convergent way to develop such a machine if you didn't have 50 years of legacy code and trundling along behind you....
That last sentence is what we're mostly standardizing on, except for using QT.
I've pushed for web interfaces but it's weird how even embedded developers with broad backgrounds get when you bring it up. I think there's a psychological barrier some people have to using webpages on an embedded system.
I've pushed for web interfaces but it's weird how even embedded developers with broad backgrounds get when you bring it up. I think there's a psychological barrier some people have to using webpages on an embedded system.
It’s because the web is a moving target and embedded systems are static. What’s your reaction when you see a web browser or web page that is 10 years old? 20? The timeframes of industrial machines is incompatible with web anything.
You need to make a separation in your mind between "the web" and "web technologies." I said nothing about putting the device online.
> You need to make a separation in your mind between "the web" and "web technologies." I said nothing about putting the device online.
I wish so, but Google kind-of wrecks that proposition, introducing new stuff to the point that some developers might (unknowingly) use something that is not available on the embedded browser.
I wish so, but Google kind-of wrecks that proposition, introducing new stuff to the point that some developers might (unknowingly) use something that is not available on the embedded browser.
That's kind of impossible when the server and the browser are on the same machine :-)
Bear in mind that what I'm talking about here is making UI's for embedded systems easier to design by using web technologies. That means that we're only leveraging web tech for local use: there is no connectivity farther than (maybe) to another device on the same circuit board.
Something like BoneScript (javascript for the BeagleBone) takes this step farther by building I/O extensions into the language so your system controller and webserver can be the same process.
Bear in mind that what I'm talking about here is making UI's for embedded systems easier to design by using web technologies. That means that we're only leveraging web tech for local use: there is no connectivity farther than (maybe) to another device on the same circuit board.
Something like BoneScript (javascript for the BeagleBone) takes this step farther by building I/O extensions into the language so your system controller and webserver can be the same process.
On the contrary, I find writing embedded GUIs enjoyable. Sure, it's harder, but also very rewarding. You need an event queue, so you can handle internal and external events the same way. The rest is "just" programming.
Same here.
The constraints he talks about (hardware and memory) I find to be a lot easier to program for because they really put a hard stop to useless abstractions - I find myself just wiring straightforward code and getting things up and running immediately instead of wasting lots of time on design and architecture.
The constraints he talks about (hardware and memory) I find to be a lot easier to program for because they really put a hard stop to useless abstractions - I find myself just wiring straightforward code and getting things up and running immediately instead of wasting lots of time on design and architecture.
I think this is mostly a problem of finding good tools for your workflow. Admittedly, there are fewer of them in the space of embedded development than for the usual "apps", but there are some. Besides the already mentioned LVGL [1] for the actual GUI you'll need a proper build tool that can handle all that conversion stuff OP's talking about reliably. I can only recommend Bob [2] here. Unfortunately, it's very unknown but brings everything you need for embedded development (w/ or w/o GUI, higher-level stuff like custom Linux images, too). Using it for years professionally and for personal projects and will never look back.
[1] https://lvgl.io [2] https://bobbuildtool.dev
[1] https://lvgl.io [2] https://bobbuildtool.dev
A better hardware architecture for running a gui with some real-time requirements is often to move the real-time stuff into a separate microprocessor entirely. I’ve run into some tricky problems trying to do both on the same core, as you have zero control over when the UI will suddenly start consuming more resources, and it becomes a game of whack-a-mole to keep your ui responsive without overtaxing your event loop. I think this is one reason TI put a Cortex M processor on die with the AM335X. It enables you to offload your real-time logic onto a separate processor and dedicate hardware control to that so the main application processor could focus on rendering and responding to user inputs.
That post really resonated with me. I attempted making my own ESP32-based remote for controlling my media centre while I'm in my kitchen, or generally away from my laptop.
The nature of embedded with interrupts (e.g. from buttons) and trying to mix in MQTT events and nice multi-layered UI made me completely question my programming abilities.
I really wanted to have modals/pop-ups (e.g. to show current volume on change) and it ended up so messy. Not only it is about control flow but about memory too (as the author mentioned). I couldn't just store a stack of "screens" to fall back to.
I'm watching this library recently: https://github.com/peterhinch/micropython-micro-gui
It seems promising to solve many of the common issues that you get, once you move from simple static information display.
The nature of embedded with interrupts (e.g. from buttons) and trying to mix in MQTT events and nice multi-layered UI made me completely question my programming abilities.
I really wanted to have modals/pop-ups (e.g. to show current volume on change) and it ended up so messy. Not only it is about control flow but about memory too (as the author mentioned). I couldn't just store a stack of "screens" to fall back to.
I'm watching this library recently: https://github.com/peterhinch/micropython-micro-gui
It seems promising to solve many of the common issues that you get, once you move from simple static information display.
We are definitely spoiled on modern systems. But imo, this convenience is unreasonably costly. It is baffling to me that modern devices with ghz clock rates can hardly render guis and websites responsibly. At the moment everyone swoons over the new M1 processors. Not too long and those performance gains will also be lost to costly abstractions and badly written code. I'm not very optimistic this will change in the near future.
We are already at a point where a cpu (!) based raytracer can outrender some user interfaces. The amount of cycles lost on the equivalent of paper pushing is incredible these days.
Source: my own raytracer vs microsoft teams and 90% of all other electron apps or otherwise very javascript heavy frontends.
Source: my own raytracer vs microsoft teams and 90% of all other electron apps or otherwise very javascript heavy frontends.
Unless you're building many thousands of units for a price-sensitive market, it's almost always a better use of your time to run a nice GUI on a separate PC like a Raspberry pi and do your real-time stuff on a $4 arduino or $1.50 MCU.
I suppose I'll be the outlier here, but on more powerful devices like Raspberry Pi or other Cortex-based devices, I have had good luck using LispWorks and their CAPI UI library. Plus, Lisp!
I recently built a machine with a couple of beams, a Hall effect switch, heater, thermocouple and a couple of badass stepper motors with external controllers.
Wrote my whole stack in C# / Winforms and deploy to a Mono running on a Raspberry Pi 4.
UI runs on main thread and is purely viewing and event generation. On a background thread, I have a homegrown cooperative multitasking state machine driven kernel that handles all interaction with the machinery. Works awesome. Plenty of CPU to spare. And event response is < 5ms typically, which is based on the timer interval.
Only two downsides, only printf debugging. I gave up after a couple of days of following SO and various tutorials on how to get the remote Mono / .NET debugger working. Maybe if I switch to WPF…but I avoided that because the Interface Designer for VS was half-baked (and sucked) at the time.
Second downside, some weird point map scaling variation between the Interface Designer and Mono placing with the window under the WM. I worked around by incorporating a build script to parse the source of the form and write out an initialization routine to fix the size and location of all screen elements. It’s definitely sketchy but it works.
The end result says it all though. My UI and responsiveness beats the snot out of machines we’ve spent multiple $100K on that run Win Embedded + PLC architecture.
Wrote my whole stack in C# / Winforms and deploy to a Mono running on a Raspberry Pi 4.
UI runs on main thread and is purely viewing and event generation. On a background thread, I have a homegrown cooperative multitasking state machine driven kernel that handles all interaction with the machinery. Works awesome. Plenty of CPU to spare. And event response is < 5ms typically, which is based on the timer interval.
Only two downsides, only printf debugging. I gave up after a couple of days of following SO and various tutorials on how to get the remote Mono / .NET debugger working. Maybe if I switch to WPF…but I avoided that because the Interface Designer for VS was half-baked (and sucked) at the time.
Second downside, some weird point map scaling variation between the Interface Designer and Mono placing with the window under the WM. I worked around by incorporating a build script to parse the source of the form and write out an initialization routine to fix the size and location of all screen elements. It’s definitely sketchy but it works.
The end result says it all though. My UI and responsiveness beats the snot out of machines we’ve spent multiple $100K on that run Win Embedded + PLC architecture.
A Raspberry Pi 4 is a good bit more powerful than most embedded systems.
Unless the use cases are very simple or there's a special reason not to for most embedded devices I think I'd rather communicate largely with an app (e.g. by bluetooth).
I'm thinking for washing machines, dishwashers, aircon, things like that.
Unless simple, embedded UIs are usually bad and usually make complex interactions (e.g. setting timers) especially horrible. Whereas android/apple/the web have been optimized for UX.
It's oddly not as common as I'd have expected, though.
I'm thinking for washing machines, dishwashers, aircon, things like that.
Unless simple, embedded UIs are usually bad and usually make complex interactions (e.g. setting timers) especially horrible. Whereas android/apple/the web have been optimized for UX.
It's oddly not as common as I'd have expected, though.
It's oddly not as common as I'd have expected, though.
If you put Apple on your hardware product's critical path, they may capriciously change their app store rules at some point to invalidate your whole product or business model.
If you put Google on your hardware product's critical path, they may just get bored of supporting whatever Google product(s) you are using, and kill your whole product or business model without even realizing they're doing it.
These companies should be treated as forces to be reckoned with, not (just) as friends to partner with.
If you put Apple on your hardware product's critical path, they may capriciously change their app store rules at some point to invalidate your whole product or business model.
If you put Google on your hardware product's critical path, they may just get bored of supporting whatever Google product(s) you are using, and kill your whole product or business model without even realizing they're doing it.
These companies should be treated as forces to be reckoned with, not (just) as friends to partner with.
Not to mention they may ask for money, if not now then perhaps later.
We definitely don't want an app, but a standard API of some kind. 3rd parties can then write apps, or web apps, etc, spawning a 3rd party market to create great UIs. Manufacturers can focus on what they should be good at, manufacturing ergonomic, efficient, reliable, long-lasting and repairable devices and appliances.
Personally I'd rather avoid apps. I already have like seven dozen apps from all the smarthome devices my partner buys. It clutters up my phone, I have no idea how much they're spying on me, and it's a pain to pull out my phone to change some hardware right in front of me.
Let me configure it physically, or do it over a web browser, but preferably the later. Especially for when the servers are shutdown because the product is no longer profitable (looking at you sonos)
Another thing, has anyone seen a well designed software made by a hardware company? They seem very rare. Just look at any printers.
Let me configure it physically, or do it over a web browser, but preferably the later. Especially for when the servers are shutdown because the product is no longer profitable (looking at you sonos)
Another thing, has anyone seen a well designed software made by a hardware company? They seem very rare. Just look at any printers.
This is true. If embedded device makers do make an app they seemingly can't resist doing it without vacuuming up your personal data / device telemetry or proxying device connections via their servers.
It's not even that. I'm one of those people who never has their phone nearby so it would be a chore to have to get my phone out of the bedroom every time I wanted to use an appliance.
We've done real-time JavaScript UIs for several projects where the hardware is slick enough to host a REST API or a WebSockets server. It's worked well, making it easy to access devices from a kiosk browser or remotely. It also enables communication between devices across the same API layer, and by decoupling the UI from the application and control planes, it allows us to iterate super quickly with familiar tools.
[deleted]
I've written quick and dirty GUIs in Racket on the Raspberry Pi 4 and 3. I've found the 1gb memory on the PI3 with an editor open, Raspbian running, and a pdf, I have about 300mb free for another app. It can be a little tight with a 150-175mb Gui written in Racket, but on a 2gb PI4 the extra breathing room helps a lot.
Please just stop!
There is one area where gadgets and appliances are the most cumbersome to use, and that is their electronic UI. They have always been terrible (eg programming VCRs).
Graphical UIs won't make them better.
And all those screens and buttons will be the first things to fail, will be very hard to source replacements for, and will be too expensive to bother repairing anyway, not to mention, an expensive addition to The Thing in the first place.
Please, just add a bluetooth LE interface, and a web app, for control and programming with cheap, ubiquitous, and designed-for-UI mobile phones, and open-source it.
The Thing itself should only have the most minimal physical interface, eg on/off button and led. Keep the embedded mcu (and embedded programmers) for low-level device control mechanisms and some policy enforcement for safety.
There is one area where gadgets and appliances are the most cumbersome to use, and that is their electronic UI. They have always been terrible (eg programming VCRs).
Graphical UIs won't make them better.
And all those screens and buttons will be the first things to fail, will be very hard to source replacements for, and will be too expensive to bother repairing anyway, not to mention, an expensive addition to The Thing in the first place.
Please, just add a bluetooth LE interface, and a web app, for control and programming with cheap, ubiquitous, and designed-for-UI mobile phones, and open-source it.
The Thing itself should only have the most minimal physical interface, eg on/off button and led. Keep the embedded mcu (and embedded programmers) for low-level device control mechanisms and some policy enforcement for safety.
There is a lot to discuss here, but if I had to choose I would reject your idea.
This is mostly because, in reality, manufacturers not only will they never open source those apps, most probably they will never provide APIs and the Thing and associated apps will depend on some server. Such a dependency I find unacceptable, but it is the trend of all things IoT.
If this were not the case, if by law all IoT Things had to provide open APIs, I would be all for it.
This is mostly because, in reality, manufacturers not only will they never open source those apps, most probably they will never provide APIs and the Thing and associated apps will depend on some server. Such a dependency I find unacceptable, but it is the trend of all things IoT.
If this were not the case, if by law all IoT Things had to provide open APIs, I would be all for it.
I agree, please no server, no cloud, no auto-update, the interface should be local.
'Right to repair' laws should extend to 'right to program' , thus forcing open APIs of some kind.
There should be some kind of standard, defacto, or official.
'Right to repair' laws should extend to 'right to program' , thus forcing open APIs of some kind.
There should be some kind of standard, defacto, or official.
In my Utopia, I could see a vending machine, open its web app UI in my phone, pay via UPI (Indian interoperable payment system) app, and get the item.
Its HTTPS over Bluetooth. I could list Bluetooth devices like ACs and TVs in range, and open paired devices as a web app. It could be server rendered with JS sprinkled, a PWA I could install like an app, or even streaming video.
Its HTTPS over Bluetooth. I could list Bluetooth devices like ACs and TVs in range, and open paired devices as a web app. It could be server rendered with JS sprinkled, a PWA I could install like an app, or even streaming video.
I agree with him that Renode is by far the best tool around. But I haven't yet managed to add support for my boards yet. Neither to qemu ditto. Renode has much more boards supported than qemu though.
Also, lvgl is a very good toolkit.
Also, lvgl is a very good toolkit.
[deleted]
I can imagine a generic smartphone app that has configurable UI and is designed to talk to serial devices over BLE. Add a BLE module to your IoT, send a json config to this app, and away you go!
[deleted]
my takeaway from the article is I hope OP isn't responsible for that lag fest of an animation[0]. I hate embedded stuff that takes ages to animate while the hardware obviously just can't handle it. Skip the animation, and just make it snappy, damn it! There's nothing more infuriating than a slow interface that makes you wait needlessly.
[0] https://blog.benjamin-cabe.com/wp-content/uploads/2021/10/re...
[0] https://blog.benjamin-cabe.com/wp-content/uploads/2021/10/re...
OP here :) That animation is laggy on the Renode emulator (at the moment at least), but definitely snappy and 30fps on the real HW...
Off topic -> but on topic of the original post. Do you know if https://github.com/ocornut/imgui is available for embedded? It might be too much for the very low end but it for a reasonably capable system it's better than web or some high powered UI like Qt.
ah, good to hear! In that case act as if I said nothing :)
The coffee machine at the office has the laggiest animated interface and as a software developer it's just painful. They have complete control over the hardware and software so there is just no excuse.
What do you think happened there? Did the decision-makers never try using the product before it was manufactured? Were the stakeholders shown the "final product" on some high-powered prototype device?
The interface is very attractive -- high-resolution pictures of the various coffees, nice fonts, and a clean design. My guess is that graphic designers designed the images and interactivity without any considerations of performance. Then that design was given to developers to be shoe-horned into a machine not capable of handling it.
Possibly different teams contracted out to do both jobs. The design is done by one team, finished, and then handed off to the next.
I'm sure the interface could be tweaked (remove some animations, lower the resolution, etc) to make it usable but instead there was a complete failure of management and it was shipped as-is.
Possibly different teams contracted out to do both jobs. The design is done by one team, finished, and then handed off to the next.
I'm sure the interface could be tweaked (remove some animations, lower the resolution, etc) to make it usable but instead there was a complete failure of management and it was shipped as-is.
Been down this road many times:
- UI toolkit is cross platform (cough cough Qt, or Altia, or Crank) and runs great on the UX developer's x86 box. The UX guy loves fancy CSS transitions.
- Graphic assets are delivered in 32bit RGBA and the data cache chokes on the massive bit maps. And the cheap LCD is 565BGR so the CPU is constantly mixing channels. Then we discover the hardware has no alpha blending capability. More software load.
- Hardware gets cost reduced. Or the flagship platform gets the beefy CPU and the consumer versions get a Rockchip.
- RAM gets downsized or the project grows out of bounds and now we're blitting from QSPI flash.
- The whole system runs in JavaScript because that's what they can get out of the overseas development team, or it's too expensive to buy 2 more Qt seat licenses (or whatever the hell they want now. I think we're up to first-born children)
It all repeats, over and over.
- UI toolkit is cross platform (cough cough Qt, or Altia, or Crank) and runs great on the UX developer's x86 box. The UX guy loves fancy CSS transitions.
- Graphic assets are delivered in 32bit RGBA and the data cache chokes on the massive bit maps. And the cheap LCD is 565BGR so the CPU is constantly mixing channels. Then we discover the hardware has no alpha blending capability. More software load.
- Hardware gets cost reduced. Or the flagship platform gets the beefy CPU and the consumer versions get a Rockchip.
- RAM gets downsized or the project grows out of bounds and now we're blitting from QSPI flash.
- The whole system runs in JavaScript because that's what they can get out of the overseas development team, or it's too expensive to buy 2 more Qt seat licenses (or whatever the hell they want now. I think we're up to first-born children)
It all repeats, over and over.
This is even happening in video games now. See Gameface for example.
As the owner of the first company I worked for said when he was complaining about a product, "somebody said, 'fuck it, it's good enough' and shipped crap out the door."
So how many embedded GUIs did the author actually write?
He seems to have compiled an arbitrary list of 'hard things' or 'problems you may encounter'. But these are presented so shallow, I find it hard to believe he ever struggled with them.
Although he seems to push the opensource Renode at the end, I feel it is mostly an Azure submarine.
He seems to have compiled an arbitrary list of 'hard things' or 'problems you may encounter'. But these are presented so shallow, I find it hard to believe he ever struggled with them.
Although he seems to push the opensource Renode at the end, I feel it is mostly an Azure submarine.