As part of working for an engineering firm, I sometimes need to integrate machines into a system designed by one of the firm’s mechanical engineers. This could be part of a POC, so we can move things a bit quicker. Or it could be part of our final product - integrating proven products into our designs means we can focus on getting the final machines built quicker, and less prone to errors. This is done on a regular basis in different disciplines of engineering. In mechanical engineering, a lot of designs integrate standard parts such as bearings or motors, for example. In electronics engineering, relying on pre-built modules in designs could save a lot of hassle.

But in this one machine we’re building, I needed to integrate a couple of different machines that required a different set of skills, since they were (definitely) not meant to be included in any larger machine - and that’s what I want to write about. In this particular machine, we needed to incorporate a consumer-level machine into our bigger system. We ended up incorporating two different machines, one for the POC, and a different one for the first real iteration of the machine.

What are my options, anyway?

There are a few different options for controlling machines that have Android interfaces.

I’ve had a bit of a hard time figuring out how to get it working, so I wrote it down. Mostly, to not have to search for answers again in the future.

These options include:

  • Automate the machine through macros I wrote with webhooks, controlling an app designed by the machine manufacturer, running on a phone hidden inside our machine.
  • Automate the machine by recording UART between the controller and the UI device (started, deemed too hard).
  • Automate the machine by reverse engineering the controller firmware (never tried, WAYYYY too hard for the specific machine).
  • Automate the machine by leveraging ADB:
    • Blindly.
    • UIAutomator (deprecated by Google, works only for static content).
    • OpenCV pattern matching (last resort).
    • Appium (UIAutomator2 driver, FOSS, QA tool).

The first machine

The first machine was selected thanks to its ability to be controlled remotely via an Android app that sent commands through Bluetooth.

Initial probing

Since I knew it was Bluetooth-based, I first tried sniffing the communication, using Wireshark and Nordic nRF. I ordered an nRF52840 Feather from Adafruit, and followed Adafruit’s great guide for installing and using that to sniff BLE communication between the machine and the application installed on my phone. I also used Nordic’s own nRF Connect on my phone (there’s a desktop app as well) to see the BLE descriptors as well.

But while looking at the sniffed packets I realized the machine was only advertising its existence through BLE, then (probably) moving onto the more secure Bluetooth Classic communication for actually being controlled through the accompanying smartphone app.

A quick look through the code for the app (apk packages can mostly be unzipped) convinced us it was time to think of a simpler way for controlling this particular machine.

Second best - but winning

As a quicker method to get the POC rolling, a consultant suggested we try installing a macro app for Android. Specifically, he suggested we use Macrodroid, since he’s had good results using it in the past.

The process was quite simple, if a bit tedious. I found a disposable phone from an earlier project, installed the application for controlling the machine as well as Macrodroid, and started setting up a macro that would start the app, wait for a bit, then press the desired button on the second page, then delay again, and finally the third button.

For selecting textual buttons, Macrodroid offers a search functionality. For moving sliders around, I enabled Android’s Pointer Location in the Developer’s options, and set up moving the slider based on the actual display of the device I used.

Finally, I set up a webhook with some query parameters to trigger the macro on the device - and my part of the job was done.

I passed the webhook addresses and the parameters to the guy writing the rest of the machine’s controller, made sure the phone was always on and had its Bluetooth, location, and WiFi, always on (and with power), and went on to the next project.

Stay Tuned

In the next post, I’ll cover the next iteration of the machine, and how I had to use a (very) different approach to get it working. Stay tuned for a detailed walkthrough of the process and the final solution!