Weather station woes and fixes

I've set up some Raspberry Pis to export data from various FineOffset weather stations, using WeeWX as the server software periodically downloading records and presenting them graphically in a webpage. Each weather station comes with a dedicated console to receive transmissions from the outdoor sensors every minute or so, and the console has a USB socket to allow a host to configure it and extract data.

This console is known to have a “USB lock-up” bug, whereby it refuses to talk to the host after some random period (from a few days to a couple of months), even though it had been interacting successfully prior to that point. The only robust work-around is to power-cycle the console, which is not easy to automate. Here's what I had to do.

Detection

The lock-up bug now appears as Operation timed out in the WeeWX log, as given by sudo /bin/systemctl status weewx:

fousb: get_records failed: [Errno 110] Operation timed out

You can get essentially the same lines from grep weewx /var/log/syslog. Four of these appear (about 45 seconds apart), and then WeeWX seems to reconnect in vain, and gets another four, and so on. This cycle lasts about 3½ minutes.

Note that the WeeWX documentation on the matter identifies a different error:

could not detach kernel driver from interface

Maybe that means this isn't really the lock-up bug I'm getting, but the symptoms and treatment seem to be the same.

Recovery

You have to take the batteries out of the console, and ensure it is disconnected over USB. You can run the console off USB power alone, so for an unattended power cycle, you just need a USB hub that can depower its sockets. The big disadvantage of leaving the batteries out is that no readings are taken during a power cut; with the batteries in, you could at least pull them off the console when power returned, as it can store several days' worth.

Here are the Pis I'm using with each weather station:

Hostname	Host model	Weather station model	WeeWX version
fish	RPi 3B+	WH3083	3.9.2
ruscoe	RPi 3B	WH3083	3.9.2
kettley	RPi 3B	WH1080	3.9.1

All are running some version of Raspbian.

I've used uhubctl to check for and invoke the power-cycling feature, and can confirm that both the RPi 3B and 3B+ can control the power on their USB sockets. I also tried an RPi Zero W, which would have had the ideal amount of grunt for the task, but it's unable to control power on its sockets. Since I've not seen the problem on the WH1080, it could be used there, or indeed on any similar set-up with a different type of weather station. I was using an older RPi model at some point (with no built-in Wi-Fi); it could power-cycle its entire USB hub, although this included the USB Wi-Fi chip!

The output of sudo uhubctl looks something like this (on a 3B; it's marginally different on the 3B+):

$ sudo uhubctl 
Current status for hub 1-1 [0424:9514]
  Port 1: 0503 power highspeed enable connect [0424:ec00]
  Port 2: 0100 power
  Port 3: 0100 power
  Port 4: 0303 power lowspeed enable connect [1941:8021]
  Port 5: 0100 power

1941:8021 is the weather station console:

$ lsusb 
Bus 001 Device 014: ID 1941:8021 Dream Link WH1080 Weather Station / USB Missile Launcher
Bus 001 Device 013: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

This means that a command of the following form will power-cycle the console:

sudo uhubctl -l 1-1 -p 4 -a 2 -d 30 -R

Update: On fish, I had to set -p to one less than the reported port! sudo uhubctl said it was in port 3, but the command only worked on port 2. It helps to have someone looking at the console to confirm when you're doing it remotely!

• -l 1-1 -p 4 are taken from the output of uhubctl, identifying the hub and port.
  
• -a 2 causes a power cycle, rather than switching on or off.
  
• -d 30 keeps it off for a generous 30 seconds. That maybe could be trimmed a bit.
  
• -R resets the hub, forcing devices to re-associate. I found this to be essential, and wonder if it would be effective without the power cycle. Update: It isn't; you must remove the batteries.
  

Putting it together

A script, in ~/.local/bin/check-weather-station:

#!/bin/bash

count=0
while read line ; do
    if [[ "$line" == *"get_records failed: [Errno 110] Operation timed out"* ]] ; then
        ((count++))
    elif [[ "$line" == *"Stopping LSB: weewx weather system"* ]] ; then
        count=0
    fi
done < <(grep weewx /var/log/syslog | tail -50)

if [ $count -ge 4 ] ; then
    printf >&2 'Fault detected, power-cycling...\n'
    echo >&2 'Stopping station software'
    sudo /bin/systemctl stop weewx
    echo >&2 'Power-cycling hub'
    sudo /usr/sbin/uhubctl -l 1-1 -p 4 -a 2 -d 30 -R
    echo >&2 'Waiting for end of sensor-learning period'
    sleep 180
    echo >&2 'Setting time'
    sudo /usr/bin/wee_device -y --set-time
    echo >&2 'Setting interval'
    sudo /usr/bin/wee_device -y --set-interval=5
    echo >&2 'Restarting station software'
    sudo /bin/systemctl start weewx
fi

A cron job then checks every few minutes:

*/3 * * * * $HOME/.local/bin/check-weather-station

That should pick up the fault within one 3½-minute cycle.

Other aspects of the script:

• Four “timed out” messaged are awaited. Maybe I could get away with two, or even one!
  

• The weewx service is suspended during the reset. This ensures there's no interaction with the console shortly after it comes back on.
  

• While the service is suspended, we don't want overlapping invocations of the script to do anything. This is detected by resetting the message count whenever we see that the service has been stopped. Only “timed out” messages that are not followed by a “stopping” message are counted.
  

  (There's a potential race condition here, but it's not going to happen unless parsing the log and stopping the service take more than 3 minutes.)
  

• Waiting three minutes after the reset ensures that the console's sensor-learning mode is not jeopardized by external activity. The weather-station manual warns about key activity on the console during this time, and I suspect it actually extends to USB activity too. I've been very cautious, so it might be possible to trim the timing a bit.
  

• The console's time is synchronized with the host's. This can only be done while the service is stopped. (Unfortunately, this does not seem to update the clock displayed in the console.)
  

• The logging interval is set to 5 minutes. Apparently, the console can sometimes forget this after a power cycle, but this setting is thought to reduce the likelihood of lock-ups.
  

Other issues

• One Pi wouldn't come back on after a power cut. Changing the power supply fixed that.
  

• Another Pi seems to lose its Internet connection, but continued gathering data from the console. Being headless, the simplest thing for a non-technical person to do is to power-cycle the Pi, but that's a bit drastic, and undermines the goal of unattended operation. I tried the following to prevent the Wi-Fi from going to sleep, but it still happened:

  sudo iw dev wlan0 set power_save off
  

  I've resorted to pinging the router once a day.
  

Results

With a cruder detection mechanism (one that took about four or five log cycles to get lucky), I've seen the script work twice in just a couple of days. I'm trying out this new detection mechanism above, which should be safe to use as often as every three minutes, and so it should be able to detect the first cycle. I'll update this article as things develop.

Removing variable prefixes and suffixes from other variables in Bash

Just been bitten by this…

If you have a variable txt in Bash, you can strip a given prefix or suffix from it like this:

$ txt=a/b.d/c.jpg
$ echo "${txt%.*}"
a/b.d/c
$ echo "${txt##*.}"
jpg
$ echo "${txt%%.*}"
a/b
$ echo "${txt#*.}"
d/c.jpg

The % operator strips of the shortest matching suffix, and .* matches .jpg, so that gets removed. %% strips off the longest matching suffix. Similarly, # and ## strip off the shortest and longest matching prefix, respectively. Asterisks, square brackets and other characters are special, probably following the same rules as Pattern Matching in the Bash manual page.

You can also use literal strings as the patterns, i.e., no special characters:

$ txt=a/b.d/c.jpg
$ echo "${txt%.jpg}"
a/b.d/c
$ echo "${txt%.png}"
a/b.d/c.jpg
$ echo "${txt#a/b.d/}"
c.jpg
$ echo "${txt#c/b.d/}"
a/b.d/c.jpg

Note that, if the prefix or suffix doesn't match (whether you use special characters or not), you get the whole string returned.

These operations are useful for traversing pathnames:

$ path="/home/john/file.jpg"
$ echo Leaf is "${path%%*/}"
Leaf is file.jpg
$ echo Dir is "${path#/*}"
Dir is /home/john

You have to be careful if your input doesn't contain the separator:

$ input1=path/to/file.jpg
$ input2=file.jpg
$ echo Input 1 dir "[${input1%/*}]" leaf "[${input1##*/}]"
Input 1 dir [path/to] leaf [file.jpg]
$ echo Input 2 dir "[${input2%/*}]" leaf "[${input2##*/}]"
Input 2 dir [file.jpg] leaf [file.jpg]

To avoid this special case, I thought I could do this:

input1=path/to/file.jpg
input2=file.jpg
input1leaf="${input1##*/}"
input1dir="${input1%${input1leaf}}"
input2leaf="${input2##*/}"
input2dir="${input2%${input2leaf}}"
echo "[${input1dir}]" "[${input1leaf}]"
echo "[${input2dir}]" "[${input2leaf}]"

…which leads to this:

[path/to/] [file1.jpg]
[] [file1.jpg]

However, I hadn't noticed that special characters are still interpreted after the partial expansion:

input3="path/to/file [2002].jpg"
input3leaf="${input3##*/}"
input3dir="${input3%${input3leaf}}"
echo "[${input3dir}]" "[${input3leaf}]"

The square brackets are taken as a wildcard, and fail to match the literal value:

[path/to/file [2002].jpg] [file1 [2002].jpg]

The trick is to quote again:

input3="path/to/file [2002].jpg"
input3leaf="${input3##*/}"
input3dir="${input3%"${input3leaf}"}"
echo "[${input3dir}]" "[${input3leaf}]"

Now you get the intended result:

[path/to/] [file1 [2002].jpg]

An alternative technique would be to use the length of your prefix/suffix in a substring operations, but it's less convenient and more error-prone if you want to do small adjustments to a prefix or suffix before applying it.

Anyway, in summary, if you're going to use Bash's prefix/suffix removal with a computed pattern, put the result in quotes!

Bash redirection with descriptor in variable, and locking

A recommended way to acquire a lock in Bash is to open the lock file for a group command, and call flock on the open descriptor before doing anything dangerous:

{
  echo waiting
  flock -x 9
  echo in
  sleep 10
  echo done
} 9> /tmp/lock

Try it in two independent terminals. The second command will run only as the first finishes.

However, one should never have to pick an arbitrary file descriptor (9 in this case). Fortunately, you can get Bash to choose an available descriptor, using {var} in place of the literal descriptor number:

unset lfd
{
  echo waiting
  flock -x $lfd
  echo in
  sleep 10
  echo done
} {lfd}> /tmp/lock

Problem solved!

No, wait. The descriptor doesn't get closed at the end of the group command, so your second invocation will hang indefinitely. Once the first terminal has finished, if you manually close the descriptor, the second proceeds:

exec {lfd}>&-

Looks like you have to do things more explicitly (and the group command is no longer useful):

echo waiting
unset lfd
exec {lfd}> /tmp/lock
flock -x $lfd
echo in
sleep 10
echo done
exec {lfd}>&-

This is inconvenient if you want to break or continue out of an enclosing loop:

for i in $(seq 1 10)
do
  {
    echo waiting
    flock -x 9
    echo in
    sleep 5
    if something_went_wrong ; then continue ; fi
    sleep 5
    echo done
  } 9> /tmp/lock
done

Is this a bug, a feature, or a mistake on my part? (Bash version 4.4.20(1)-release.)

procmail to IMAP

My employer provides an IMAP server for email. Historically, it behaved badly with Message-Id headers, overwriting the original (the first sin, breaking all threading), and overwriting it with a globally indistinct value. It also had limited filtering capabilities. So I avoided it, and eventually set up my own home IMAP server (using Dovecot, Postfix and Procmail). I still had to get emails forwarded from the employer-provided server, and was forced to use ‘forward as attachment’ to ensure that the message id was preserved. An earlier post was on how to extract the email attachment to a pipe for interfacing with Procmail, but the latest incarnation of the employer-provided server has a proper redirection facilty, so that's redundant for me now.

Problem

Running your own IMAP server at home is no fun, especially with a dynamic IP. You have to have a Dynamic DNS service, and hope no-one else gets your mail while it's re-syncing. I need the filtering with procmail, but it only delivers to mailboxes in mbox and Maildirs formats directly (and maybe one other…?); everything else has to be handled through an external command. Is there a beast that will deliver to an IMAP server? If so, I could redirect the mail via SMTP to my own host running procmail, and have it deposit the mail in specific folders back on the server via IMAP.

Stack Overflow has one question on procmail and delivering to an IMAP server?, and suggests the use of mailtool, which is part of the Courier mail server. However, I don't want to install all of Courier to get it, and an apt-file search can't seem to find it, so I suspect it's also ‘historical’.

Solution

Python to the rescue! (I hate Python syntax, by the way. Indentation for syntax is a step backwards, in my opinion. Still, it works…) Here's a small program pushimap, which uses imaplib to write a file to an IMAPS server:

#!/usr/bin/env python

import imaplib
import time
import ConfigParser
import os
import sys
import email.message
import email

import getopt
from pprint import pprint

if __name__ == '__main__':
    ## Parse arguments.
    cfg_name = os.environ['PUSHIMAP_CONFIG'] \
               if 'PUSHIMAP_CONFIG' in os.environ \
                  else os.path.expanduser('~/.config/pushimap/conf.ini')
    acc_name = os.environ['PUSHIMAP_ACCOUNT'] \
               if 'PUSHIMAP_ACCOUNT' in os.environ \
                  else 'default'
    msg_file = None
    mb_name = 'INBOX'
    flags = ''
    opts, args = getopt.getopt(sys.argv[1:], "f:a:d:s")
    for opt, val in opts:
        if opt == '-f':
            cfg_name = val
        elif opt == '-d':
            mb_name = val
        elif opt == '-a':
            acc_name = val
        elif opt == '-s':
            flags += ' \Seen'
    flags = flags[1:]
    if len(args) == 0:
        args.append('/dev/stdin')

    ## Read the account details.  Should include port too.
    config = ConfigParser.ConfigParser()
    config.read([os.path.expanduser(cfg_name)])
    hostname = config.get('account %s' % acc_name, 'hostname')
    username = config.get('account %s' % acc_name, 'username')
    password = config.get('account %s' % acc_name, 'password')

    ## Connect to the server.
    c = imaplib.IMAP4_SSL(hostname)
    try:
        ## Authenticate with the server.
        c.login(username, password)

        ## Process the plain arguments as filenames.
        for fn in args:
            if fn is None or fn == '':
                continue

            ## Read in the message.
            fp = open(fn, "r")
            try:
                msg = email.message_from_file(fp)
            finally:
                fp.close()

            ## Attempt to add the file's contents as a message, or
            ## create the folder and try again.
            created = False
            while True:
                typ, erk = c.append(mb_name, flags,
                                    imaplib.Time2Internaldate(time.time()),
                                    str(msg))
                if typ != 'NO':
                    sys.exit()
                if created:
                    sys.exit(1)
                typ, erk = c.create(mb_name)
                created = True
    finally:
        c.logout()

-f file specifies an INI file to hold credentials and the server address, and defaults to $PUSHIMAP_CONFIG, then to ~/.config/pushimap/conf.ini. The file should contain the likes of:

[account default]
hostname = imap.example.org
username = bloggsj
password = mind-your-own-sodding-business

-a acc specifies a section [account acc] to read fields from, and defaults to default.

-d mailbox specifies a mailbox on the server to add a message to, and defaults to INBOX. Some servers seem to use / (U+002F) as a folder-name separator, and others use . (U+002E, full stop). You could check by temporarily modifying the code to run typ, data = c.list() ; pprint(data).

-s says that messages should be flagged as ‘seen’, i.e., read.

Remaining arguments are filenames. If none are given, /dev/stdin is assumed. Each file is read as an mbox-formatted message (what will it do with an mbox with more than one message?), and appended to the specified mailbox. If that fails, an attempt is made to create the mailbox, then the appending operation is tried again. If that fails, the program exits with a non-zero status. This can be used with Procmail's W flag, in which the exit status of the piped command determines whether to continue filtering.

See imaplib - IMAP4 client library - Python Module of the Week for some example uses of imaplib. The c.list() call is useful in determining what the folder-name separator is.

Now let's Procmail it up:

PUSHIMAP_CONFIG = $HOME/.myimapstuff

:0
* ^List-Id:.*<some\.list\.example\.org>
{
  :0 W
  | pushimap -d "Work/My Employer/Mailing list"

  :0
  ".Work.My Employer.Mailing list/"
}

Caveats

• As a precaution, I'm keeping the old Maildirs action, but it's only done if the pushimap command fails.
  

• I used email.message_from_file in solving the earlier problem (extracting an attachment from a pipe), and it seemed to cope okay with fairly big messages, but I'm not sure. Ideally, it would transparently cache them on disc when they went over a threshold. It's quite possible that the script will simply terminate abnormally, so the :0 W flag is being relied upon to ensure some back-up form of delivery.
  

Notes on bug JDK-8162455

I submitted bug JDK-8162455 some time ago (2016, so it says), and it's not a very important bug, so I can reasonably expect its fixing to be at a very low priority. Nevertheless, I'd like to comment on it, especially as it seems my original report wasn't terribly clear.

What's the problem?

As the bug report describes, if you specify an annotation-processor option to the Java compiler, and you correctly provide the corresponding processor to the compiler, you can get still a warning of the form:

The following options were not recognized by any processor: a list including that option

…if a certain part of the processor is never invoked (because none of the annotations it recognizes are present in the source). You also get it if you fail to specify the processor correctly (e.g., you get the classpath or the class's name wrong, or specify the service incorrectly, etc), so it's useful there. And you get it if you spell the option wrongly, so that's useful too. The problem is that either:

• the logic that determines which options are to be reported is faulty, or
  
• the message is faulty.
  

Which fault applies depends on what the intent of the warning is.

What's the intent?

The class com.sun.tools.javac.processing.JavacProcessingEnvironment implements this behaviour. It creates a set of the names of all provided options before annotation processing properly begins. Then, during that processing, it selectively removes entries matching those recognized by processors that it has just called to process a round. When all processing is complete, it generates the warning if the set is not empty.

Assuming its implementation accurately expresses its intent, a more accurate message would be:

The following options were not recognized by any engaged processor: options

…where engaged means having a processing round submitted to it.

But what use is knowing the options of an unengaged processor? If a processor had not been engaged, all of its options would be reported this way, so you're never going to be told about some of its options but not others (unless another processor was engaged, and unusually happened to use some of the same options). If this isn't the real intent, the logic must be faulty.

What if the intent is to inform about unengaged processors?:

• Why not simply identify the processors directly?
  
• If the options of an unengaged processor are also all recognized by an engaged processor, you won't be informed of the unengaged processor.
  
• If an unengaged processor has no options, again you won't be informed.
  

The current logic doesn't robustly achieve this intent either. I have to presume that the message already expresses the intent.

What's the solution?

How should the logic be fixed? In JavacProcessingEnvironment, don't bother gradually eating away at the set of option names each time a processor is engaged. Instead, just before you check whether the set is empty at the end, go through all processors, and remove the options they support. The remainder is the set to report. A private method checks the set of remaining options, and reports if non-empty. You just have to eliminate the recognized ones before checking:

private void warnIfUnmatchedOptions() {
    for (ProcessorState ps : discoveredProcs)
        ps.removeSupportedOptions(unmatchedProcessorOptions);
    if (!unmatchedProcessorOptions.isEmpty())
        ...
}

So, although it's a fairly inconsequential bug, the fix is also pretty trivial, assuming that the iteration over the processors changes no other state, and that the intent is as I've presumed.

Genuinely scalable SVGs with width and height attributes

It seems some software doesn't know what to do with an SVG that has no width and height attributes. These attributes have always bothered me a bit. What's so scalable about forcing your vector graphic to be a specific number of pixels or inches wide or high?

Or is it only to be taken as a hint? Firefox (65.0) doesn't think so. It fixes the image at the specified size, and treats unitless values as pixels (or maybe virtual pixels).

The desktop background under Kubuntu (18.04 as I write) does seem to take it as a hint, and a mandatory one at that. If the icon to be used for a *.desktop file is an SVG, it might display it. If you specify width and height in millimetres (say), it seems to rasterize at the size you specify, then scales the bitmap up or down as required. But if you don't specify width and height, you end up with a carefully labelled blank space on your screen.

How do you satisfy both requirements? The solution seems to be to use percentages. Check the last two fields of the viewBox attribute:

<svg viewBox='68932 -240980 190516 153908'
     ... >

Divide the smaller by the larger, and express as a percentage:

$ bc -lq
153908/190516*100
80.78481597346154653600

If the last number is smaller, use the computed value as the height; otherwise, use it as the width. Set the other attribute to 100%:

<svg viewBox='68932 -240980 190516 153908'
     width="100%" height="80.78%"
     ... >

This seems to allow Firefox to scale according to available space, while the desktop deigns to display a decently detailed icon.

I've updated Mokvino Web to re-include width and height with these computed values. I've updated my earlier article too.

Update 2021-04-06: It now looks like you should set both width and height to 100%, for the Kubuntu desktop and for Firefox at least. Bloody hell. More investigation…

Issues mounting MTP on Kubuntu

I've had difficulty getting Kubuntu to mount a Samsung Galaxy S8 reliably. Using the Device Notifier gets as far as showing directory structure and thumbnails, but the MTP process dies if you try to read a file properly. Also, the phone asks the user whether it should be accessed via USB only after the first mount attempt. If you say “Allow”, it withdraws its current configuration (thereby invalidating the first mount), and re-offers it (thereby causing the Device Notifier to pop up again, and requiring the user to open another window). Perhaps it's a clash between USB and Android requirements: (say) the phone must respond to a mount-triggered USB request at once, but Android also has to wait for user authorization, and has no way to asynchronously inform the host of new files appearing on an existing mount? To get anywhere, I've had to abandon the Device Notifier, install jmtpfs on the host, and run it manually, and twice. I've also had to enable Developer Options on the phone(?!).

Now I'm trying to write an auto-mounting script for a headless machine, so that the latest photos and videos I've recorded on my devices can be automatically moved off the device simply by plugging it in. The files will later be dropped into an ingest process to make them ready for presentation over DLNA. I use this to watch for USB devices being plugged and unplugged:

$ inotifywait -m -r /dev/bus/usb -e CREATE -e DELETE
/dev/bus/usb/001 CREATE 010
/dev/bus/usb/001 DELETE 010
/dev/bus/usb/001 CREATE 011
/dev/bus/usb/001 DELETE 011

CREATE and DELETE events specify the bus number and device number (e.g., 001:010) when the phone is plugged in or unplugged. The output of lsusb -v -s 001:010 provides the vendor id and serial number of the device, and whether an MTP interface is provided, so events for non-MTP devices can be ignored.

On plugging in, the CREATE event is received. The phone lights up, but doesn't yet ask the user if the host has permission to access its files. I ensure a mount point exists, and run jmtpfs on it, specifying the device id:

mkdir -p "/var/mtp/$vendid-$serialno"
jmtpfs -device=001,010 "/var/mtp/$vendid-$serialno"

This triggers the phone to ask for authorization from the user. Although the response is still pending, the mount appears to succeed, so I proceed to scan the mount point for interesting files with find. All attempts to scan or access fail with I/O errors, so there's nothing to do but unmount with:

fusermount -u "/var/mtp/$vendid-$serialno"

Now I tap “Allow” on the phone, and I get a DELETE for 001:010, immediately followed by CREATE for 001:011. The device id has changed, but the vendor id and serial number are the same, so the same mount point is used, and mounts without error as before. This time, a scan of files succeeds, and unmounting can take place when they have been processed.

So, the trick seems to be:

• Expect failure and assume a retry will occur. (If it doesn't, it obviously wasn't that important.)
  
• Use the vendor id and serial number to avoid treating the retry as new device. (You don't actually need to remember that there was an earlier failure, just make sure that your action in the second cycle tries to do exactly what it tried to do before.)