Removing variable prefixes and suffixes from other variables in Bash

Just been bitten by this…

If you have a variable txt in Bash, you can strip a given prefix or suffix from it like this:

$ txt=a/b.d/c.jpg
$ echo "${txt%.*}"
a/b.d/c
$ echo "${txt##*.}"
jpg
$ echo "${txt%%.*}"
a/b
$ echo "${txt#*.}"
d/c.jpg

The % operator strips of the shortest matching suffix, and .* matches .jpg, so that gets removed. %% strips off the longest matching suffix. Similarly, # and ## strip off the shortest and longest matching prefix, respectively. Asterisks, square brackets and other characters are special, probably following the same rules as Pattern Matching in the Bash manual page.

You can also use literal strings as the patterns, i.e., no special characters:

$ txt=a/b.d/c.jpg
$ echo "${txt%.jpg}"
a/b.d/c
$ echo "${txt%.png}"
a/b.d/c.jpg
$ echo "${txt#a/b.d/}"
c.jpg
$ echo "${txt#c/b.d/}"
a/b.d/c.jpg

Note that, if the prefix or suffix doesn't match (whether you use special characters or not), you get the whole string returned.

These operations are useful for traversing pathnames:

$ path="/home/john/file.jpg"
$ echo Leaf is "${path%%*/}"
Leaf is file.jpg
$ echo Dir is "${path#/*}"
Dir is /home/john

You have to be careful if your input doesn't contain the separator:

$ input1=path/to/file.jpg
$ input2=file.jpg
$ echo Input 1 dir "[${input1%/*}]" leaf "[${input1##*/}]"
Input 1 dir [path/to] leaf [file.jpg]
$ echo Input 2 dir "[${input2%/*}]" leaf "[${input2##*/}]"
Input 2 dir [file.jpg] leaf [file.jpg]

To avoid this special case, I thought I could do this:

input1=path/to/file.jpg
input2=file.jpg
input1leaf="${input1##*/}"
input1dir="${input1%${input1leaf}}"
input2leaf="${input2##*/}"
input2dir="${input2%${input2leaf}}"
echo "[${input1dir}]" "[${input1leaf}]"
echo "[${input2dir}]" "[${input2leaf}]"

…which leads to this:

[path/to/] [file1.jpg]
[] [file1.jpg]

However, I hadn't noticed that special characters are still interpreted after the partial expansion:

input3="path/to/file [2002].jpg"
input3leaf="${input3##*/}"
input3dir="${input3%${input3leaf}}"
echo "[${input3dir}]" "[${input3leaf}]"

The square brackets are taken as a wildcard, and fail to match the literal value:

[path/to/file [2002].jpg] [file1 [2002].jpg]

The trick is to quote again:

input3="path/to/file [2002].jpg"
input3leaf="${input3##*/}"
input3dir="${input3%"${input3leaf}"}"
echo "[${input3dir}]" "[${input3leaf}]"

Now you get the intended result:

[path/to/] [file1 [2002].jpg]

An alternative technique would be to use the length of your prefix/suffix in a substring operations, but it's less convenient and more error-prone if you want to do small adjustments to a prefix or suffix before applying it.

Anyway, in summary, if you're going to use Bash's prefix/suffix removal with a computed pattern, put the result in quotes!

Bash redirection with descriptor in variable, and locking

A recommended way to acquire a lock in Bash is to open the lock file for a group command, and call flock on the open descriptor before doing anything dangerous:

{
  echo waiting
  flock -x 9
  echo in
  sleep 10
  echo done
} 9> /tmp/lock

Try it in two independent terminals. The second command will run only as the first finishes.

However, one should never have to pick an arbitrary file descriptor (9 in this case). Fortunately, you can get Bash to choose an available descriptor, using {var} in place of the literal descriptor number:

unset lfd
{
  echo waiting
  flock -x $lfd
  echo in
  sleep 10
  echo done
} {lfd}> /tmp/lock

Problem solved!

No, wait. The descriptor doesn't get closed at the end of the group command, so your second invocation will hang indefinitely. Once the first terminal has finished, if you manually close the descriptor, the second proceeds:

exec {lfd}>&-

Looks like you have to do things more explicitly (and the group command is no longer useful):

echo waiting
unset lfd
exec {lfd}> /tmp/lock
flock -x $lfd
echo in
sleep 10
echo done
exec {lfd}>&-

This is inconvenient if you want to break or continue out of an enclosing loop:

for i in $(seq 1 10)
do
  {
    echo waiting
    flock -x 9
    echo in
    sleep 5
    if something_went_wrong ; then continue ; fi
    sleep 5
    echo done
  } 9> /tmp/lock
done

Is this a bug, a feature, or a mistake on my part? (Bash version 4.4.20(1)-release.)

procmail to IMAP

My employer provides an IMAP server for email. Historically, it behaved badly with Message-Id headers, overwriting the original (the first sin, breaking all threading), and overwriting it with a globally indistinct value. It also had limited filtering capabilities. So I avoided it, and eventually set up my own home IMAP server (using Dovecot, Postfix and Procmail). I still had to get emails forwarded from the employer-provided server, and was forced to use ‘forward as attachment’ to ensure that the message id was preserved. An earlier post was on how to extract the email attachment to a pipe for interfacing with Procmail, but the latest incarnation of the employer-provided server has a proper redirection facilty, so that's redundant for me now.

Problem

Running your own IMAP server at home is no fun, especially with a dynamic IP. You have to have a Dynamic DNS service, and hope no-one else gets your mail while it's re-syncing. I need the filtering with procmail, but it only delivers to mailboxes in mbox and Maildirs formats directly (and maybe one other…?); everything else has to be handled through an external command. Is there a beast that will deliver to an IMAP server? If so, I could redirect the mail via SMTP to my own host running procmail, and have it deposit the mail in specific folders back on the server via IMAP.

Stack Overflow has one question on procmail and delivering to an IMAP server?, and suggests the use of mailtool, which is part of the Courier mail server. However, I don't want to install all of Courier to get it, and an apt-file search can't seem to find it, so I suspect it's also ‘historical’.

Solution

Python to the rescue! (I hate Python syntax, by the way. Indentation for syntax is a step backwards, in my opinion. Still, it works…) Here's a small program pushimap, which uses imaplib to write a file to an IMAPS server:

#!/usr/bin/env python

import imaplib
import time
import ConfigParser
import os
import sys
import email.message
import email

import getopt
from pprint import pprint

if __name__ == '__main__':
    ## Parse arguments.
    cfg_name = os.environ['PUSHIMAP_CONFIG'] \
               if 'PUSHIMAP_CONFIG' in os.environ \
                  else os.path.expanduser('~/.config/pushimap/conf.ini')
    acc_name = os.environ['PUSHIMAP_ACCOUNT'] \
               if 'PUSHIMAP_ACCOUNT' in os.environ \
                  else 'default'
    msg_file = None
    mb_name = 'INBOX'
    flags = ''
    opts, args = getopt.getopt(sys.argv[1:], "f:a:d:s")
    for opt, val in opts:
        if opt == '-f':
            cfg_name = val
        elif opt == '-d':
            mb_name = val
        elif opt == '-a':
            acc_name = val
        elif opt == '-s':
            flags += ' \Seen'
    flags = flags[1:]
    if len(args) == 0:
        args.append('/dev/stdin')

    ## Read the account details.  Should include port too.
    config = ConfigParser.ConfigParser()
    config.read([os.path.expanduser(cfg_name)])
    hostname = config.get('account %s' % acc_name, 'hostname')
    username = config.get('account %s' % acc_name, 'username')
    password = config.get('account %s' % acc_name, 'password')

    ## Connect to the server.
    c = imaplib.IMAP4_SSL(hostname)
    try:
        ## Authenticate with the server.
        c.login(username, password)

        ## Process the plain arguments as filenames.
        for fn in args:
            if fn is None or fn == '':
                continue

            ## Read in the message.
            fp = open(fn, "r")
            try:
                msg = email.message_from_file(fp)
            finally:
                fp.close()

            ## Attempt to add the file's contents as a message, or
            ## create the folder and try again.
            created = False
            while True:
                typ, erk = c.append(mb_name, flags,
                                    imaplib.Time2Internaldate(time.time()),
                                    str(msg))
                if typ != 'NO':
                    sys.exit()
                if created:
                    sys.exit(1)
                typ, erk = c.create(mb_name)
                created = True
    finally:
        c.logout()

-f file specifies an INI file to hold credentials and the server address, and defaults to $PUSHIMAP_CONFIG, then to ~/.config/pushimap/conf.ini. The file should contain the likes of:

[account default]
hostname = imap.example.org
username = bloggsj
password = mind-your-own-sodding-business

-a acc specifies a section [account acc] to read fields from, and defaults to default.

-d mailbox specifies a mailbox on the server to add a message to, and defaults to INBOX. Some servers seem to use / (U+002F) as a folder-name separator, and others use . (U+002E, full stop). You could check by temporarily modifying the code to run typ, data = c.list() ; pprint(data).

-s says that messages should be flagged as ‘seen’, i.e., read.

Remaining arguments are filenames. If none are given, /dev/stdin is assumed. Each file is read as an mbox-formatted message (what will it do with an mbox with more than one message?), and appended to the specified mailbox. If that fails, an attempt is made to create the mailbox, then the appending operation is tried again. If that fails, the program exits with a non-zero status. This can be used with Procmail's W flag, in which the exit status of the piped command determines whether to continue filtering.

See imaplib - IMAP4 client library - Python Module of the Week for some example uses of imaplib. The c.list() call is useful in determining what the folder-name separator is.

Now let's Procmail it up:

PUSHIMAP_CONFIG = $HOME/.myimapstuff

:0
* ^List-Id:.*<some\.list\.example\.org>
{
  :0 W
  | pushimap -d "Work/My Employer/Mailing list"

  :0
  ".Work.My Employer.Mailing list/"
}

Caveats

• As a precaution, I'm keeping the old Maildirs action, but it's only done if the pushimap command fails.
  

• I used email.message_from_file in solving the earlier problem (extracting an attachment from a pipe), and it seemed to cope okay with fairly big messages, but I'm not sure. Ideally, it would transparently cache them on disc when they went over a threshold. It's quite possible that the script will simply terminate abnormally, so the :0 W flag is being relied upon to ensure some back-up form of delivery.
  

Notes on bug JDK-8162455

I submitted bug JDK-8162455 some time ago (2016, so it says), and it's not a very important bug, so I can reasonably expect its fixing to be at a very low priority. Nevertheless, I'd like to comment on it, especially as it seems my original report wasn't terribly clear.

What's the problem?

As the bug report describes, if you specify an annotation-processor option to the Java compiler, and you correctly provide the corresponding processor to the compiler, you can get still a warning of the form:

The following options were not recognized by any processor: a list including that option

…if a certain part of the processor is never invoked (because none of the annotations it recognizes are present in the source). You also get it if you fail to specify the processor correctly (e.g., you get the classpath or the class's name wrong, or specify the service incorrectly, etc), so it's useful there. And you get it if you spell the option wrongly, so that's useful too. The problem is that either:

• the logic that determines which options are to be reported is faulty, or
  
• the message is faulty.
  

Which fault applies depends on what the intent of the warning is.

What's the intent?

The class com.sun.tools.javac.processing.JavacProcessingEnvironment implements this behaviour. It creates a set of the names of all provided options before annotation processing properly begins. Then, during that processing, it selectively removes entries matching those recognized by processors that it has just called to process a round. When all processing is complete, it generates the warning if the set is not empty.

Assuming its implementation accurately expresses its intent, a more accurate message would be:

The following options were not recognized by any engaged processor: options

…where engaged means having a processing round submitted to it.

But what use is knowing the options of an unengaged processor? If a processor had not been engaged, all of its options would be reported this way, so you're never going to be told about some of its options but not others (unless another processor was engaged, and unusually happened to use some of the same options). If this isn't the real intent, the logic must be faulty.

What if the intent is to inform about unengaged processors?:

• Why not simply identify the processors directly?
  
• If the options of an unengaged processor are also all recognized by an engaged processor, you won't be informed of the unengaged processor.
  
• If an unengaged processor has no options, again you won't be informed.
  

The current logic doesn't robustly achieve this intent either. I have to presume that the message already expresses the intent.

What's the solution?

How should the logic be fixed? In JavacProcessingEnvironment, don't bother gradually eating away at the set of option names each time a processor is engaged. Instead, just before you check whether the set is empty at the end, go through all processors, and remove the options they support. The remainder is the set to report. A private method checks the set of remaining options, and reports if non-empty. You just have to eliminate the recognized ones before checking:

private void warnIfUnmatchedOptions() {
    for (ProcessorState ps : discoveredProcs)
        ps.removeSupportedOptions(unmatchedProcessorOptions);
    if (!unmatchedProcessorOptions.isEmpty())
        ...
}

So, although it's a fairly inconsequential bug, the fix is also pretty trivial, assuming that the iteration over the processors changes no other state, and that the intent is as I've presumed.

Genuinely scalable SVGs with width and height attributes

It seems some software doesn't know what to do with an SVG that has no width and height attributes. These attributes have always bothered me a bit. What's so scalable about forcing your vector graphic to be a specific number of pixels or inches wide or high?

Or is it only to be taken as a hint? Firefox (65.0) doesn't think so. It fixes the image at the specified size, and treats unitless values as pixels (or maybe virtual pixels).

The desktop background under Kubuntu (18.04 as I write) does seem to take it as a hint, and a mandatory one at that. If the icon to be used for a *.desktop file is an SVG, it might display it. If you specify width and height in millimetres (say), it seems to rasterize at the size you specify, then scales the bitmap up or down as required. But if you don't specify width and height, you end up with a carefully labelled blank space on your screen.

How do you satisfy both requirements? The solution seems to be to use percentages. Check the last two fields of the viewBox attribute:

<svg viewBox='68932 -240980 190516 153908'
     ... >

Divide the smaller by the larger, and express as a percentage:

$ bc -lq
153908/190516*100
80.78481597346154653600

If the last number is smaller, use the computed value as the height; otherwise, use it as the width. Set the other attribute to 100%:

<svg viewBox='68932 -240980 190516 153908'
     width="100%" height="80.78%"
     ... >

This seems to allow Firefox to scale according to available space, while the desktop deigns to display a decently detailed icon.

I've updated Mokvino Web to re-include width and height with these computed values. I've updated my earlier article too.

Update 2021-04-06: It now looks like you should set both width and height to 100%, for the Kubuntu desktop and for Firefox at least. Bloody hell. More investigation…

Issues mounting MTP on Kubuntu

I've had difficulty getting Kubuntu to mount a Samsung Galaxy S8 reliably. Using the Device Notifier gets as far as showing directory structure and thumbnails, but the MTP process dies if you try to read a file properly. Also, the phone asks the user whether it should be accessed via USB only after the first mount attempt. If you say “Allow”, it withdraws its current configuration (thereby invalidating the first mount), and re-offers it (thereby causing the Device Notifier to pop up again, and requiring the user to open another window). Perhaps it's a clash between USB and Android requirements: (say) the phone must respond to a mount-triggered USB request at once, but Android also has to wait for user authorization, and has no way to asynchronously inform the host of new files appearing on an existing mount? To get anywhere, I've had to abandon the Device Notifier, install jmtpfs on the host, and run it manually, and twice. I've also had to enable Developer Options on the phone(?!).

Now I'm trying to write an auto-mounting script for a headless machine, so that the latest photos and videos I've recorded on my devices can be automatically moved off the device simply by plugging it in. The files will later be dropped into an ingest process to make them ready for presentation over DLNA. I use this to watch for USB devices being plugged and unplugged:

$ inotifywait -m -r /dev/bus/usb -e CREATE -e DELETE
/dev/bus/usb/001 CREATE 010
/dev/bus/usb/001 DELETE 010
/dev/bus/usb/001 CREATE 011
/dev/bus/usb/001 DELETE 011

CREATE and DELETE events specify the bus number and device number (e.g., 001:010) when the phone is plugged in or unplugged. The output of lsusb -v -s 001:010 provides the vendor id and serial number of the device, and whether an MTP interface is provided, so events for non-MTP devices can be ignored.

On plugging in, the CREATE event is received. The phone lights up, but doesn't yet ask the user if the host has permission to access its files. I ensure a mount point exists, and run jmtpfs on it, specifying the device id:

mkdir -p "/var/mtp/$vendid-$serialno"
jmtpfs -device=001,010 "/var/mtp/$vendid-$serialno"

This triggers the phone to ask for authorization from the user. Although the response is still pending, the mount appears to succeed, so I proceed to scan the mount point for interesting files with find. All attempts to scan or access fail with I/O errors, so there's nothing to do but unmount with:

fusermount -u "/var/mtp/$vendid-$serialno"

Now I tap “Allow” on the phone, and I get a DELETE for 001:010, immediately followed by CREATE for 001:011. The device id has changed, but the vendor id and serial number are the same, so the same mount point is used, and mounts without error as before. This time, a scan of files succeeds, and unmounting can take place when they have been processed.

So, the trick seems to be:

• Expect failure and assume a retry will occur. (If it doesn't, it obviously wasn't that important.)
  
• Use the vendor id and serial number to avoid treating the retry as new device. (You don't actually need to remember that there was an earlier failure, just make sure that your action in the second cycle tries to do exactly what it tried to do before.)
  

Two logical interfaces on one physical, on Ubuntu 18.04 without Netplan

If my ISP-provided home gateway allowed DNS aliases to be configured, I'd get it to map foo.home to bar.home, a headless server. foo.home is meant to be present in my home network and in my relatives', to identify a host providing write access at each site to a library of photos, videos and music that are synchronized between sites. The home gateway has no such aliasing feature, so I've done it by adding an interface on the bar.home host. The new interface looks like a different host to the gateway, so it can have a different name. It happens to get a different IP too.

I could have achieved largely the same with the spare wireless interface, but why use up airwaves to travel 1 foot between a static host and the access point? I could have bought a USB Ethernet dongle, but I did it without any extra hardware or using up a socket on the gateway by creating a virtual interface faux0 piggybacked on the physical wired interface enp3s0. Here's what I did on Ubuntu Server 18.04.

From Netplan to Ifupdown

Ubuntu 18.04 uses Netplan by default. Its configuration files match /etc/netplan/*.yaml. Older Ubuntus use ifup and ifdown which read configuration from /etc/network/interfaces, and ifup -a is run at boot to bring up all marked interfaces. I'd hoped to configure Netplan to set up two logical interfaces on one physical one, with different MAC addresses, and each making DHCP requests with different names, but it doesn't seem to have any way to do that. According to Netplan documentation, installing the package ifupdown is sufficient to disable Netplan:

sudo apt-get install ifupdown

Now you need configuration to make ifupdown perform Netplan's duties:

# In /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp3s0
iface enp3s0 inet dhcp

The interface name enp3s0 is the host's sole wired Ethernet device. Yours might have a different name, perhaps the traditional eth0. You can list all interface names with:

ip link show

Just to make sure, I also renamed 01-netcfg.yaml to 01-netcfg.yaml-disabled. That's the only file I found in /etc/netplan/, so that really should render it inert, as Netplan doesn't modify interfaces it does not match in its configuration.

Things that didn't work

I also investigated removing the package netplan.io, but was told that that would also remove ubuntu-minimal. I suspected that might be a bad idea. There's also a package netplan, which also provides the /usr/sbin/netplan binary, but it was not installed.

Creating the second interface

With ifupdown now responsible for interface configuration at boot, define the new interface:

# In /etc/network/interfaces
auto lo
iface lo inet loopback

auto enp3s0
iface enp3s0 inet dhcp

auto faux0
iface faux0 inet dhcp
pre-up ip link add faux0 link enp3s0 address XX:XX:XX:XX:XX:XX type macvlan
pre-up /sbin/sysctl -w net.ipv6.conf.faux0.autoconf=0
post-down ip link delete faux0

I named the new, virtual interface faux0. It's created with an ip link command just before the interface comes up, and similarly deleted just after being taken down, using the pre-up and post-down directives.

The pre-up /sbin/sysctl is not essential, but disables SLAAC on the interface, which is appropriate for virtual interfaces. Don't know whether I'll need it, but the interface seemed to be accumulating a lot of IPv6 addresses, so I'll try it and see.

The new interface has a distinct MAC address XX:XX:XX:XX:XX:XX, specified as it is created. I've borrowed one from a device I know will not be seen on my home network, but there's probably a better strategy, something that a virtualization system employs, perhaps. It's not something I've looked into yet. Maybe someone will explain in a comment, because I get a lot of those. The interfaces file format also has a hwaddress setting, but it seemed to have no effect.

The new interface is configured to request a DHCP lease with the name foo:

# In /etc/dhcp/dhclient.conf
interface "faux0" {
  send host-name "foo";
  send dhcp-client-identifier 1:XX:XX:XX:XX:XX:XX;
}

I've thrown in a dhcp-client-identifier setting, but I'm not sure how vital it is. It seems you can use any string (with quotes if necessary), and it's just to stop the gateway from thinking the two DHCP clients are the same, leading to both interfaces coming up as under the same name in the gateway's web interface, making it less clear what you're port-forwarding to. However, that could have been caused in my case by bad data cached in the gateway, flushed out by leaving the server off while deleting the entries in the gateway. I'm going to leave them the setting in for now, as it seems harmless. I explicitly set an identifier for the main interface too, for completeness.

Things that didn't work

Setting the hostname for the interface with a hostname directive in /etc/network/interfaces didn't work because dhclient doesn't recognize it. Hence, it is set in dhclient's own configuration.

Setting the MAC address with a hwaddress also didn't work.

ARP flux

ARP flux can be a problem. Both interfaces can respond to ARP requests for either of their IPs. My home gateway then detects that both IPs map to the same MAC, and therefore to the same hostname, so both names end up resolving to the same IP. The other address, though it gets properly assigned to the right interface, never gets used. Functionally, this is fine, and actually meets the goal of having DNS aliases. However, it messes up the rendition and editing of port-forwarding rules in the gateway. If you have a rule forwarding to the disused MAC, its IP has no name, so the IP is displayed as the destination, not the hostname. It's also impossible to select that IP as a destination, because you can only select by name on this particular gateway.

To fix this, it's possible to disable the interfaces responding to ARP requests on behalf of each other, and the following seems to the right combination of settings to avoid one of the interfaces going dead (according to this serverfault article):

# In /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.all.rp_filter=2

You can test these temporarily with the likes of:

sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2
sudo sysctl -w net.ipv4.conf.all.rp_filter=2

Things that didn't work

Not setting rp_filter results in one of the interfaces being unable to receive traffic, effectively leaving it dead.

---

That should be it. Rebooting should put that into effect, but without rebooting, this should be enough (from the machine's console, not remotely!):

sudo ifdown enp3s0
sudo ifup enp3s0
sudo ifup faux0

In summary:

• Install ifupdown.
  
• Remove or rename Netplan files matching /etc/netplan/*.yaml to disable Netplan.
  
• Create entries in /etc/network/interfaces to take over Netplan's duties, and augment to set up the extra interface.
  
• Tell dhclient to use foo instead of the machine's hostname.
  
• Take steps to prevent ARP flux.
  

Happy now?

---

[Edited to include notes on ARP flux.]