Using a Kindle Screen to Monitor ESXi
Posted Thursday, November 26, 2020 in Projects
The Premise
This is a bit of a thanksgiving project that I've been pushing off for a bit. Luckily, I've already had some of the groundwork set out from when I took my old Kindle DX and repurposed it into a literary clock. A big part of that project was reworking the generation of PNGs based off the quotes to make them a bit cleaner and nicer looking on the DX's larger screen. I did this PNG rework in C#, which in hindsight, was a terrible choice for image generation. It seemed like a good idea at the time, because “do what you know,” right? Well, it turns out that the text-to-image manipulation ecosystem in .NET is a nightmare with all kinds of weird issues that require fixes like dynamically loading windows libraries to force Win32 API calls to set opacity (wow!).
Anyway, I decided to take another crack at it, and this time to adapt the Kindle setup towards a status screen for my home server.
The Plan
I have a few things as monitoring targets: ESXi, pfsense, gitlab, various web servers, and some applications. I figured I could boil this down to the bare essentials by grabbing key ESXi stats and maybe a bit of the web stuff from the frontend proxy (nginx in this case). Once I have a few stats, I found that this SVG editing method would be a good method of getting my data from text to png fairly easily. So the general process is:
- Setup an image display service on the Kindle
- Setup some statistics retrieval for Esxi and Nginx
- Create an SVG template to display the relevant statistics
- Create a script to generate the appropriate PNG and ship to the Kindle
Again, I was lucky enough that former-me already setup the display service on the Kindle based on the literary clock project. This also included all the work for jailbreaking the kindle. So for step 1, all that was left was to make some minor modifications to the literary clock scripts to display a single “image.png” that I'd update remotely over the usbnet piece that I installed as part of the jailbreak (my Kindle DX unfortunately does not have wifi capabilities).
Stats from a Hobbyist's ESXi
For your lowly home server uses, ESXi can be pretty handy and has a lot of benefits over hypervisors I've used for experimentation. Benefits like being compatible with more than a handful of operating systems and not falling to bit rot the second you look away. But one thing ESXi doesn't have over hypervisors like proxmox, is an easy avenue to hack away to your heart's content. Easy access to APIs and niceties are locked away behind paid licensing or partner agreements. Which, in retrospect of ESXi being totally free, is not the worst thing in the world. Somebody needs to be a billionaire around here and it certainly isn't going to be me.
One of the APIs that have slipped through the licensing cracks is SNMP. Probably because it's a huge pain to use. You can start down this trail by first configuring the ESXi host for SNMP. For this project, I setup my host with SNMP v3, sha1 auth, and aes128 privacy. You'll want to do a lot of testing here with tools like snmpget, snmpwalk, or whatever the windows equivalent is for testing snmp mibs. There's a few resources at vmware and github that can help you track down relevant mibs.
I decided to collect the snmp data via python, so I used the python module pysnmp. There's a few weird concepts in play with pysnmp that makes walking snmp mibs a bit different than traditional tools, but it just takes a bit of desk-to-head application to get just right. There are a few tutorials that I used to help out here.
I decided to narrow my SNMP data collection down to CPU, memory, and upload/download speeds. You do need to fenagle these values a bit and, unfortunately, it looks like the CPU values are probably way off. I found several forum posts for RedHat, IBM, and some in serverfault for ESXi that seems to point to the SNMP counter values for CPU load to be well off from what the GUI reports. I gave up here and just grabbed the lowest CPU core load value instead of the much less accurate average value. Probably still way off, but oh well. Memory and network usage is a bit tricky as well, so there's a few tricks I threw in to get the proper values.
Pretty Pictures
Once I had my data I would now need to throw that into a PNG. For this part, I went over to Method Draw and crafted an SVG. For each of the text boxes I wanted to replace with real data, I threw in some placeholder text (S1, S1, etc…). After saving off the SVG, I now had what is essentially a text file that I could easily perform a find and replace on before converting to PNG. I also thew in a warning icon in here and set the opacity to 0. I figured I could use this later by changing the opacity of the icon to something viewable for when a statistic hits some threshold. I haven't actually used this, but I figured it's important to plan ahead.
Throwing the code into my update_kindle.py script, I'm now collecting stats, updating my template.svg, and then converting to a Kindle appropriate png (proper grayscale and all that).
Server/Kindle Comms
Maybe this is easier on other Kindles, but for the DX I'm working on, the method of getting the image onto the Kindle's filesystem is a wee bit of a never-ending plumet into madness. The first thing you might find is that the Kindle's framework and powerd services will interfere with screen manipulation, but also are critical for maintaining a usb mass storage drive which WOULD BE NICE for sending my images to. My alternative here is the great usbnet addition provided by the users over on mobileread. Unfortunately, ESXi doesn't provide native support for USB network devices. Fortunately, a “fling” is available to provide usb network drivers as an installable vib package. I did need to update my ESXi image from 7.0.0 to 7.0.1U1 for this to work though. Once I did all the updates, I was able to attach the Kindle as a usb network device to the VM of my choice. The last event here in the series of unfortunate events, is that the usb network device is a bit flaky and prone to crashing. I played around with a few fixes, but what seems to work the best is to simply reduce the speed that scp sends over the image.
The wrap up here is to have the update_kindle.py script scp the generated PNG over to the kindle. This is kicked off by a cronjob every two minutes.
