Support for other devices than a Raspberry Pi

[rhvarrier]

After watching the following video, I tried replicating the same installation process on a mini PC. However it seems that many of the requirements of the IoTStack are hard coded for a Raspberry Pi such as:

• requirement to have a user called 'pi'
• Generated Docker compose file, references to some devices that are not present in an x86 system in node red for example) :
  ` devices:
  • "/dev/ttyAMA0:/dev/ttyAMA0"
  • "/dev/vcio:/dev/vcio"
  • "/dev/gpiomem:/dev/gpiomem"
    `
    Can the installation process be modified to detect automatically on which device the containers are getting installed so that it could adapt accordingly

[marcelstoer]

Same here but for a Mac Mini. I was working my way through the "Getting Started" a few minutes ago when I decided to check the issues list for existing questions on that topic.

I wouldn't mind tweaking the generated configuration manually if I didn't have to do this line-by-line in trial & error mode.

[Paraphraser]

I'll split this into two parts.

First, pi. Historically, IOTstack did assume username pi, userID 1000, and home directory /home/pi. Those dependencies have steadily been removed and I'm reasonably sure they're now all gone. I can't spot anything in a git grep.

In my own case, I've built bare-metal systems starting from:

• Raspberry Pi Imager using its GUI tools to select a username other than pi; and
• Balena Etcher plus /boot/userconf.txt to select a username other than pi,

followed by:

• a PiBuilder run to construct an IOTstack-ready system, and then,
• an iotstack_restore (from IOTstackBackup) to restore a backup taken on a system where the username was pi.

That all worked. If any remaining pi dependencies were going to show up, I think a round-trip like that would've exposed them so I'm reasonably comfortable in saying that this is all ancient history.

There may be scripts out there that assume pi but I don't think there are any in IOTstack itself (nor in PiBuilder, nor in IOTstackBackup). If you spot any, please file an issue or a pull request.

I agree that the basic setup - requirements documentation still implies that pi is the only game in town. That wording actually dates from when there were a fair number of hard-coded pi dependencies. The paragraph beginning "Please don't read …" relaxes that to some extent but it is still the simple truth that most testing occurs on Raspberry Pis running Raspberry Pi OS where ID=1000 is username pi.

---

Second, the issue with devices. This is a lot more tricky. The IOTstack menu is not "smart". It's basically a concatenation engine with a bit of string substitution for passwords, plus a special case for Node-RED which is, itself, a concatenation engine with a bit of list substitution.

In IOTstack, most of the "hard work" gets done, by hand, long before the menu gets involved. Adding a new container means searching the web and studying existing docker run commands or suggested docker-compose fragments (what we refer to as "service definitions"), and adapting what we find to use IOTstack conventions. Those conventions are:

1. Service definitions mostly start with:

   «service»:
     container_name: «container»
     image: «relevantImage»
     restart: unless-stopped
   

   where:

   • unless there's a good reason to do otherwise, the «service» name and «container» name will be the same; and
   • «relevantImage» will be an image with architecture(s) that run on the Raspberry Pi.

   That last bit can also be a hidden trap for anyone trying to run on different architecture. If the image mentioned in the IOTstack template doesn't support the architecture the user is running on, everything turns to custard.

2. Persistent storage in a volumes directory, relative to the directory containing docker-compose.yml. In other words, the usual pattern is:

   ~/IOTstack/
    docker-compose.yml
    volumes/
     «service»/
   

   with whatever discrete files or directories that are needed by the container being stored below «service»/.

   By the way, this will work perfectly well if the top-level directory is something else like:

   ~/my/own/path/for/eye-oh-tee-stack/
   

   It's just like pi and 1000. The ~/IOTstack gets most of the testing.

3. Best-efforts port-conflict avoidance. In general, you won't find multiple containers using the same external ports unless it makes no sense to do otherwise.

   An example of the latter is PiHole and AdGuardHome. Both are DNS servers so both need to respond on port 53. It makes no sense to try to run both containers on the same machine at the same time so, if anyone tries to do that, it is better if docker-compose says "hey" and moans about the port conflict.

   We generally resolve potential port conflicts on a first-come, first-served basis. For example, if "fred" already has 8080:80 then "jack" is going to get 8081:80, and so-on.

   Avoiding port conflicts is why we try to minimise containers running in host mode. Resolving port conflicts between two host-mode containers means digging into the controls for either/both containers and is well beyond the menu's abilities.

With the foregoing as background, the problem with devices needs to be split into two parts.

First, there are containers like Node-RED where the service definition includes a standard set of devices. This implicitly assumes the hardware is a Raspberry Pi running Raspberry Pi OS. The devices are there because the default set of add-on nodes added via Dockerfile include those that a user of a Raspberry Pi running Raspberry Pi OS would expect to be there and those, in turn, depend on the device mappings. I realise it's a kind of circular logic but that's the reason.

Second, there are containers like Zigbee2MQTT and Octoprint where the left-hand-side of the device mapping uses /dev/ttyAMA0 as a placeholder. That device is chosen because it's guaranteed to exist on a Raspberry Pi running Raspberry Pi OS.

In the first case, we could resolve it by changing the default set of add-on nodes and omitting the devices but then, of course, users with a "real" Pi would have to put everything back if they wanted to use those add-on nodes. I'm not saying that's a compelling argument against doing it but it's still a consideration (the needs of the many vs the needs of the few stuff).

In the second case, not providing a left-hand-side device at all means the container goes into a restart loop. That's bad. The problem then becomes how to identify a suitable placeholder device on arbitrary hardware running an arbitrary OS.

The two sides of this problem go together. Any solution needs to address both. But, for now, the menu has no ability to deal with any of this. The best we can do is what we've ready done: document the problem - device errors.

---

This is probably not what you wanted to hear but, for now, it's the best we can do. Sorry.

I've tried to give you an explanation-in-depth because, with Andreas' latest video, there are likely to be a lot of people with these same questions.

[Paraphraser]

To flesh this out a bit more, I don't believe device discovery is a particularly easy problem to solve.

I'll use Zigbee2MQTT as my example but the same principles/problems apply to pretty much anything that connects to a host over USB.

As shipped, the Zigbee2MQTT template at:

~/IOTstack/.templates/zigbee2mqtt/service.yml

contains the following device mapping:

devices:
- /dev/ttyAMA0:/dev/ttyACM0

The right hand side of that device mapping meets the expectation set by the process running inside the container that a Zigbee adapter of some kind will mount as /dev/ttyACM0.

The right hand side is the container's /dev, as distinct from the host's /dev which is on the left hand side. The left-to-right mapping provides the glue between the container's world and the host.

As mentioned before, the left hand side uses /dev/ttyAMA0 as a placeholder. It's the Raspberry Pi's built-in serial port and is guaranteed to be present on a Raspberry Pi running Raspberry Pi OS.

If you're wondering "why not use /dev/null because that's guaranteed on any Unix system?" the answer is: "it sends the container into a restart loop." In that sense, /dev/ttyAMA0 is a least-worst compromise.

Assume a Raspberry Pi running Raspberry Pi OS with nothing connected to its USB ports other than (perhaps) an SSD. At that point, there is no /dev/ttyACM0 on the host side. So, if the default service definition had been written as:

- /dev/ttyACM0:/dev/ttyACM0

docker-compose would complain and refuse to start the container, with:

Error response from daemon: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory

As you've realised, that's not a particularly useful message because you have to infer that "ttyACM0" might be the problem, hunt for it in your compose file, and then figure out what the heck to do next.

Assume a CC2531 Zigbee adapter has been connected. All other things being equal, the adapter will mount as /dev/ttyACM0. Telling the container to come up at that point would succeed.

So far, so good. But suppose you have other USB devices connected which manifest as serial ports. Now the order of enumeration comes into play. The Zigbee adapter might be /dev/ttyACM0 today, /dev/ttyACM1 tomorrow. Similarly, not all Zigbee adapters mount as ttyACMx. The Sonoff adapter, for example, mounts as ttyUSBx.

docker-compose can't cope with that kind of variability so, faced with all these moving goal-posts, you either read the IOTstack wiki or Google the topic, then ferret about to identify your adapter by its ID, and you come up with a device mapping like:

- /dev/serial/by-id/usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00125A00183F06C5-if00:/dev/ttyACM0

Alternatively, you can do what I do and drill into the how-to of writing a UDEV rule so you can use something far more meaningful to mere humans like:

- /dev/Zigbee_CC2531:/dev/ttyACM0

As an aside, if you don't see the sense in going that extra step, consider these three error messages, all of which mean "your Zigbee adapter is not connected":

• Error response from daemon: error gathering device information while adding custom device "/dev/ttyACM0": no such file or directory
• Error response from daemon: error gathering device information while adding custom device "/dev/serial/by-id/usb-Texas_Instruments_TI_CC2531_USB_CDC___0X00125A00183F06C5-if00": no such file or directory
• Error response from daemon: error gathering device information while adding custom device "/dev/Zigbee_CC2531": no such file or directory

True, "CC2531" is buried in the second message but the third style directs your attention to your Zigbee adapter. Well, I think so anyway.

Now let's bring macOS into the picture. If I connect my Zigbee CC2531 to my iMac, it mounts as /dev/cu.usbmodem14301; the Sonoff adapter mounts as /dev/cu.SLAB_USBtoUART. If there's a macOS equivalent of "by id" or UDEV for reliable device disambiguation, I've never found it. A quick Google suggests there isn't a solution, which means you're pretty much stuck with whatever enumeration variability macOS sends your way. It looks to me like the Sonoff adapter would be particularly prone to changing its device name because SLAB_USBtoUART is pretty common. I believe it means a Silicon Labs CP2102 chip and I see that on many ESP32 development boards.

For the IOTstack menu to sort that out means it needs to know what your adapter is, whether it's physically connected or not, the conventions of how devices mount on the host OS, how they're likely to be enumerated, and how to disambiguate things that mount with the same name.

I reckon that's a bit of a tough ask.

And that's just Zigbee2MQTT…

Bottom line: any scheme to avoid having to work through designed-for-Pi service definitions, by hand, to adapt them to arbitrary host and OS combinations, would need to be pretty sophisticated.

[marcelstoer]

I now published my Mac Mini installation notes on my blog: https://frightanic.com/computers/install-iotstack-on-a-mac-mini/

[Paraphraser]

Very interesting. Can I ask if you considered installing Debian and running IOTstack there?

When Andreas did his recent "Pi alternatives" video, I spun up Debian in Parallels and, aside from the device issues, it just worked. I'd be curious to know what the experience would be from booting a Mac with Debian.

---

"opinionated", huh? I like it! 🤪

[marcelstoer]

I did indeed consider running Debian on the Mac Mini for a brief moment. However, we've only got Macs and Pis in my household and I am so happy with Macs in general that wanted to use the Mini as-is. (Ok, I did replace its old hard disk with an SSD I scraped from an old laptop.)

As for "opinionated", yes, IOTStack is opinionated and that's most of the added value for me. I get your opinion by default but can change anything I like - the best of both worlds.

[MichaelB4711]

I did indeed consider running Debian on the Mac Mini for a brief moment. However, we've only got Macs and Pis in my household and I am so happy with Macs in general that wanted to use the Mini as-is. (Ok, I did replace its old hard disk with an SSD I scraped from an old laptop.)

As for "opinionated", yes, IOTStack is opinionated and that's most of the added value for me. I get your opinion by default but can change anything I like - the best of both worlds.

IOTstack runs fine on debian 11. I use it there for a while.
But there are compatibility problems on the brandnew debian 12.